[Grok-dev] Re: solving buildout problems - pinning down eggs

[Martijn Faassen]

Hey,

Philipp von Weitershausen wrote:
[snip]
>> I've run into several problems now over the last weeks, too.
> 
> Can you be more specific?

Yes. I've had to pin ZODB and zope.lifecycleevent at various points in 
time as they blew up in my face. I didn't investigate the details why on 
the ZODB, I just know it happened and went away as soon as I pinned 
things down.

I also know JW had things blow up on him multiple times. Perhaps JW can 
share some of his frustrations.

I may note that many of us who have installed Grok often don't even find 
out Grok is actually broken right now, because we are using older 
versions of eggs in our buildout-eggs directory. Beginners just trying 
out Grok may get newer versions of eggs and have things break on them.

> I think our problems are mostly due to bad development/alpha releases 
> and the fact we're not using the prefer-final option in buildout yet. 
> But in general I do agree that it's a bad situation if somebody else's 
> screwed up release can bring *our* stuff into jeopardy.

I agree that things like prefer-final would have helped. I also think 
there is great value in having *identical* releases on everybody's box, 
bug for bug compatible. That's what I need to have a stable platform to 
build on in a team setting. I don't want to run into weird bugs at all 
where finally it turns out that somebody else had different versions 
installed.

It also helps with communication: if somebody reports a bug with Grok 
0.10, that should be enough information to go and reproduce the bug. I 
should not have to go and ask him to reproduce a whole versions list.

[snip]
> On the other hand, we shouldn't really get all these 
> development/alpha/beta versions. We should use prefer-final. Right now 
> it won't do much good, obviously, because most of our eggs aren't final 
> yet. Baiju and I have been doing lots of releases, but it's a 
> time-consuming process (that just has to be done once). Help would be 
> appreciated.

Again, I think that these things would help, but the real reliable way 
to solve the problem of version quicksand is to really pin versions down.

[snip]
>> If making this work takes time, I propose we produce a 0.10.1 release 
>> which simply pins down all the dependencies in its setup.py, hard. I 
>> know that breaks the requirement that people should be able to upgrade 
>> to newer versions of dependencies if they so choose, but we do know it 
>> actually works. We need to get rid out of this swamp of buildout 
>> problems.
> 
> It works, but it could just as well lead to another swamp. After all, 
> pinned down versions are a sure recipe for version conflicts (e.g. I'd 
> like to use package X, but package X wants Y=>3.5.0 and grok pins down Y 
> to 3.4.1).

At least *Grok* works out of the box if we pin things down. And doesn't 
break two days later. If package X wants a newer version than the one in 
Grok, the problem is fairly tractable. We should eventually grow the 
flexibility so that developers can override what's in Grok, but first 
Grok needs to work.

> The discussion on zope3-dev didn't really end in any consensus, but I 
> got out the following of it:
> 
> * The versions= stuff in zc.buildout actually works. Today. I like it, I 
> know Martijn doesn't. I don't think we have to jump to that many hoops 
> to make it work with our release process, but we obviously have 
> different views here. So far, the separate versions file that's loaded 
> over HTTP fulfills all *my* use cases best.

It's a new artifact, one more new thing to remember and explain. It 
complicates maintenance, release management, version control and 
testing. It probably complicates them only slightly, but it adds up.

That said, since the list via HTTP works today, that is a strong point 
in its favor. Let's generate such a list for Grok and try to use this 
for a while. It's quite possible I'm too worried about maintenance, and 
it shouldn't be too hard to evolve this to package metadata eventually 
anyway.

> * Jim doens't want to make setuptools any more complicated than it 
> already is. That said, a viable solution would be to create (meta-)eggs 
> with pinned down version dependencies as suggested by Tres and others. 
> Then we add the ability to either setuptools or zc.buildout to 
> explicitly override version conflicts.
> 
> The second solution seems like a fruitful compromise, even though
> 
> - it requires code that we don't yet have
> 
> - it is unclear where this code should go (setuptools, zc.buildout, ...)
> 
> - it doesn't cover all the use cases (e.g. Dieter Maurer's use case of 
> several working sets for the same package).

There are other drawbacks with this solution:

- It requires the creation of meta-eggs. Grok can't have its own 
dependency list, this is in another egg that needs to be maintained 
separately.

- either you have one meta egg (say, for grok), or you see the birth of 
a whole hierarchy of meta eggs for separate packages that depend on each 
other. This is not nice to maintain. I'd prefer to attach the new 
metadata to the existing hierarchy.

Regards,

Martijn