[Grok-dev] Re: solving buildout problems - pinning down eggs

[Philipp von Weitershausen]

Martijn Faassen wrote:
> Jasper yesterday (see earlier message) ran into yet another case where 
> grokproject wouldn't work because of some new egg release.

No, he was using workingenv which sometimes messes up 
setuptools/zc.buildout so badly that it won't consider the find-links 
option (which allows us to download eggs from 
http://download.zope.org/distribution). I talked to Ian Bicking about it 
and he's aware of the interference workingenv causes with setuptools. 
That's why he's abandoning it in favour of virtualenv, which is based on 
the simpler virtual-python.

> I've run into several problems now over the last weeks, too.

Can you be more specific?

I think our problems are mostly due to bad development/alpha releases 
and the fact we're not using the prefer-final option in buildout yet. 
But in general I do agree that it's a bad situation if somebody else's 
screwed up release can bring *our* stuff into jeopardy.

> How do we solve this immediate problem for grok? What has this 
> requirement for zope.traversing that trips us up?

IIRC, Theuni implemented the skin support for XMLRPC. Now some zope.app 
package probably needs this newer zope.traversing egg. The problem is 
that buildout just doesn't follow find-links (see above). Otherwise this 
would Just Work.

On the other hand, we shouldn't really get all these 
development/alpha/beta versions. We should use prefer-final. Right now 
it won't do much good, obviously, because most of our eggs aren't final 
yet. Baiju and I have been doing lots of releases, but it's a 
time-consuming process (that just has to be done once). Help would be 
appreciated.

> We can't keep solving these individual problems over and over again, as 
> it's clear it'll never end. So, on to solving the general problem:
> 
> We need to have, and use, the facility to "pin down" Grok's dependencies 
> to specific version numbers. This means we need to produce a complete 
> list of *all* of the egg versions that grok is using. I want to pin it 
> down exactly - all dependencies are liabilities and we can't allow a 
> single one to break everything.
> 
> We then need to have an easy way to make sure that people who use Grok 
> x.y actually use those versions, by default.
> 
> Ideally we also want the ability for people to use newer versions of 
> eggs, if they so choose. This is a bonus feature in my mind. If this 
> bonus feature takes time to work out, we need to forget about it for 
> now, as this problem is critical.
> 
> Another requirement is that this should be easy for release managers. 
> Not a lot of work to produce, and fit with the current egg release 
> process ("python setup.py bdist_egg upload").

(Side note: sdist is much more important than bdist_egg as it creates 
the tarball).

> I think we shouldn't 
> introduce a separate release artifact that needs separate management - 
> I'd like it to work with eggs.
> 
> I know Philipp had a discussion on Zope3-dev on this topic. Philipp, 
> what is the outcome of this discussion? Any clear path forward? I got 
> the impression the suggestion was to try to use distribution eggs that 
> pin version numbers, but we need to work out the details and see whether 
> this truly fulfills our requirements.
> 
> If making this work takes time, I propose we produce a 0.10.1 release 
> which simply pins down all the dependencies in its setup.py, hard. I 
> know that breaks the requirement that people should be able to upgrade 
> to newer versions of dependencies if they so choose, but we do know it 
> actually works. We need to get rid out of this swamp of buildout problems.

It works, but it could just as well lead to another swamp. After all, 
pinned down versions are a sure recipe for version conflicts (e.g. I'd 
like to use package X, but package X wants Y=>3.5.0 and grok pins down Y 
at 3.4.1).

The discussion on zope3-dev didn't really end in any consensus, but I 
got out the following of it:

* The versions= stuff in zc.buildout actually works. Today. I like it, I 
know Martijn doesn't. I don't think we have to jump to that many hoops 
to make it work with our release process, but we obviously have 
different views here. So far, the separate versions file that's loaded 
over HTTP fulfills all *my* use cases best.

* Jim doens't want to make setuptools any more complicated than it 
already is. That said, a viable solution would be to create (meta-)eggs 
with pinned down version dependencies as suggested by Tres and others. 
Then we add the ability to either setuptools or zc.buildout to 
explicitly override version conflicts.

The second solution seems like a fruitful compromise, even though

- it requires code that we don't yet have

- it is unclear where this code should go (setuptools, zc.buildout, ...)

- it doesn't cover all the use cases (e.g. Dieter Maurer's use case of 
several working sets for the same package).


-- 
http://worldcookery.com -- Professional Zope documentation and training