[Zope-dev] Re: Egg install bot results

[Jim Fulton]

On Nov 14, 2007, at 4:46 AM, Philipp von Weitershausen wrote:
...
>>    The individual eggs typically can't be used
>>     outside of a zope appserver installation (and if they can, they
>>     probably shouldn't be in "zope.app", they should be in "zope" or
>>     they should be their own top-level package), and the
>>     "namespaciness" of zope.app is suspect when it's unlikely that
>>     anyone who is not a Zope committer will release a distribution
>>     which makes use of that namespace package.  Their current
>>     overgranularity makes distributing them as separate eggs and
>>     releasing them to the cheeseshop a form of "cheeseshop  
>> pollution",
>>     especially given that so few of them can actually currently be
>>     installed using easy_install or "setup.py install".  If  
>> cheeseshop
>>     is going to continue to be used as the index, I'd suggest  
>> creating
>>     a zope.app top-level svn module with a single setup.py in
>>     containing all the packages that are meant to go into zope.app.
>>     Version the resulting "zope.app" distribution as necessary  
>> instead
>>     of versioning many more granular "zope.app.*" distros.  It's  
>> OK if
>>     some people don't use some of the functionality in the resulting
>>     egg, just toss everything in.
>
> We can (and should) do that... almost :). For ages, the zope.app  
> package has been treated as a namespace package because it always  
> contained stuff that was releasable and stuff that wasn't (e.g.  
> zope.app.homefolder or zope.app.css were experiments that probably  
> shouldn't have been committed there in the first place, but alas,  
> it happened).

If we put a lot of these packages back together, we should still  
treat zope.app as a namespace package.  Let's not go back on that.

One caution is that distutils and setuptools provide no help with  
multiple eggs that contain the same package.  This means that someone  
can install the combined and separate packages at the same time (like  
they can now install ZODB3 and transaction) leading to subtle bugs.

Probably the biggest mistake we made in this process was moving to  
fast. Whatever we do to "fix" the current situation, let's be very  
careful not to make it worse.

> I don't think we should start releasing stuff that isn't in a  
> releasable state.

You mean we never should have started. :)

...

>>     Yes, you lose the
>>     ability to make a bugfix in one subpackage and release it, but
>>     IIRC the intent is to trim zope.app down anyway, pushing
>>     libraryish things out to top-level or zope.* packages.
>
> That *was* the plan at least. We moved several packages in the Zope  
> 3.2 -> 3.3 upgrade, but a lot of people complained, so we pretty  
> much stopped doing that. Except when recently the Python parts of  
> zope.app.securitypolicy, zope.app.session were moved to  
> zope.securitypolicy, zope.session in an attempt to be able to reuse  
> the Python stuff without getting the browser views.

IOW, we have to balance cleanliness with disruption.  For a while,  
those of us working on Zope 3 revelled in "refactor mercilessly".   
This was fine before Zope 3 was widely adopted.  We have since leaned  
the value of mercy. :)

>>  - Who's in charge?  Whomever you might be, to what extent do you
>>    agree/disagree with the above suggestions?  If you agree with any,
>>    how can I help fix things?
>>  - I'm unsure how anybody is able to install Zope 3 right now using
>>    eggs, unless there's some fundamental difference in the way
>>    easy_install resolves dependencies vs. buildout.  I have not  
>> looked
>>    at how Stephan's KGS works yet, though, so I'm sure I'm just
>>    missing some magic.
>>  - We should consider fixing setuptools install to detect conflicts
>>    before attempting to install anything.
>
> +1

I strongly suspect that this will be impractical.

>
>>    The current regime of "find
>>    conflicts halfway through an install" is IMO untenable in the long
>>    term for using eggs as a distribution mechanism.  This may mean a
>>    very invasive change to both the package index and the client
>>    software, though.
>
> Yup. The problem is that people run custom indexes locally so  
> setuptools would have to be able to deal with different kinds of  
> index implementations.

Huh? There is a defined index API.  The custom indexes conform to  
this afaik.

Things could certainly be made faster if the index provided access to  
more meta data, still resolving conflicts will still be potentially  
computationally very intensive.

I still believe that a setuptools prefer-final switch would go a long  
way towards addressing this problem.

I'll also not that, AFAIK, the current problems weren't due to true  
conflicts, but due to packages in the cheeseshop requiring package  
releases that weren't in the cheeseshop.

Jim

--
Jim Fulton
Zope Corporation