[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like

[Jim Fulton]

Ian Bicking wrote:
> Jim Fulton wrote:
>>> I actually tried to do this once before with zc.buildout, but I didn't
>>> get far -- probably a result of lack of effort and lack of familiarity
>>> with the overall stack.  But I also recognize lots of the questions
>>> about stuff like the zope.conf file and Data.fs that still seem
>>> unresolved.
>>
>> Certainly when you tried this, buildout was very young and we hadn't 
>> written recipes to deal with these issues.  We've made a lot of 
>> progress since then.
> 
> Well, the last time I really used it was early December, and it still 
> felt slow and awkward to me at the time, with several funny quirks.

Hm, It's a bit hard to respond to "awkward" and "quirks".  I'll
respond to performance issues a bit below.


> 
>>> And frankly I like easy_install.  It's
>>> probably 10x faster than buildout.
>>
>> I doubt that that is true now.  Although that probably depends on what 
>> you are doing.  Early versions of buildout did a lot of things 
>> inefficiently as I was still learning setuptools.  Because of the way 
>> that buildout caches index information, I expect that creating a 
>> buildout from scratch that used a lot of eggs would be much faster 
>> than using easy_install.  One difference though is that buildout 
>> checks for the most recent compatible versions of all of the eggs it's 
>> using every time you run it, whereas, as I understand it, with 
>> workingenv, you'd just run easy_install manually when you want a new 
>> egg.  
> 
> Correct.  The basic process with workingenv is:
> 
> 1. Set it up.
> 2. Start installing stuff.
> 3. Try running stuff.
> 4. Realize you got it wrong, missed something, want to do more 
> development, return to 2.
> 
> I actually find myself doing the 2-4 loop pretty often, both in 
> development and when first deploying something.  Just the amount of time 
> to do "bin/buildout -h" was substantial (though I don't really 
> understand why, except that buildout seemed to be working way too hard 
> to update itself).

Ah yes.  This is a good point.  By default, buildout checks for newer versions
of distributions for which there are open-ended requirements.  This can take
frustratingly long -- especially because pypi is so darn slow.

One advantage of buildout over easy_install (and I assume workingenv)
is that the eggs you get are deterministic by default.  They are always
the newest versions that satisfy your requirements.  With easy_install,
you get the most recently installed eggs that satisfy your requirements.
This means that the eggs you have depend a lot on when you installed them.
To achieve this, buildout looks for newer distributions when a requirement
doesn't have an upper bound or when the upper bound isn't satisfied by an
already-installed egg.

I really should add a quick mode that skips looking for newer versions
of requirements are met by what's already installed.  This would make
the iterative style you describe go much faster.  I would certainly
appreciate this myself. I will do this soon.

>> You can bypass the checks by running in offline mode.  Then buildout 
>> runs very fast.  Because of the ability to share eggs accross 
>> buildouts, it is often possible to run a buidout using lots of eggs in 
>> offline mode.
>>
>> It has been suggested that there should be a mode for buildout that 
>> only talks to the network when there isn't a local egg that satisfied 
>> a requirement.  This would make buildout work more like workingenv 
>> when few if any eggs are actually needed.
> 
> Yes; more like easy_install does as well, actually.  Though the way 
> easy_install works is hardly intuitive; I find myself frequently saying 
> "yes, you installed it, but did you -U install it?"

In particular, upgrading a distribution doesn't upgrade it's dependencies.
This makes it harder to control which distributions are used in an environment.
With easy_install, even through distributions are automatically included
by virtue of being dependencies, they aren't automatically updated.
There's no way to say "I want the most recent version of everything".

I wanted to make it easier to get the most recent version of the distributions
used, which is why buildout has a different policy for looking up distributions.

...

>>> Plus buildout's desire to own everything and
>>> destroy everything it does not own ;)
>>
>> I'm not aware that it destroys anything. Could you be more specific?
> 
> Well, it owns parts, and the recipes control that.  Doesn't it also 
> delete and reinstall there? 

Yes.  Buildout tried to make a buildout reflect it's specification.
This is an important feature.  It uninstalls as well as installs.
But it isn't controlling anything it wasn't asked to control.

 > How it treats each area of the buildout I'm
> unclear. 

I can't help that. I've documented how this works in great detail.

 > Simply making the file layout a bit more conventional, and
> describing anything non-obvious, would make buildout feel a lot more 
> comfortable to the new user.

What is conventional?  Python uses different layouts on different systems.
The Unix layout and Windows layout are quite different. When I came up
with the layout for Zope installations, I tried to mimic the layout
that Python used on Unix systems at the time, and then that layout
changed.  We were stuck with lib/python even though we never had
anything else in lib.

I chose a shallow layout in buildout following "flat is better
than nested".

>>> * As a result buildout supports multiple things in the same buildout
>>> that have conflicting version requirements, but where the packages
>>> themselves don't realize this (but the deployer does).  If the packages
>>> know their requirements then setuptools' native machinery allows things
>>> to work fine.
>>
>> Yes.  I expect that usually, packages won't be very specific.  The 
>> buildout configuration file provides a place to be specific.
> 
> workingenv allows this, insofar as you can be specific while installing 
> things, and with the requirements file.  But it doesn't make the 
> individual scripts very specific, if for instance appfoo requires 
> libX>1.0, and appbar requires libX>1.1, but you actually want appfoo to 
> use libX==1.0 and appbar to use libX==1.1 and install them in the same 
> buildout.  That's the only case where buildout seems to be able to 
> express something workingenv can't.

In practice, this can be very important. At least for us at ZC.

>>> * Some see bin/activate as a jail.  Both workingenv and buildout are
>>> deliberately jail-like.  Both Jim and I loathe the non-repeatability of
>>> system-wide installations (at least I think I can speak for him on that
>>> one point ;).  bin/activate lets you into that jail, and lets you work
>>> there.  There is no way into a buildout.
>>
>> I'm not familiar with bin/activate, but it sounds like an interpreter 
>> script created with buildout.
> 
> It's created by workingenv, and you have to source it because basically 
> its only function is to add the workingenv/lib/pythonX.Y to $PYTHONPATH. 
>  Adding that path to $PYTHONPATH is the only thing that really 
> "activates" a workingenv.

Ah, so it modifies the user's environment.  I think that's a reasonable
approach, although not one that I care for myself. To each his own.

The important things here, IMO, is that both activate and buildout interpreter
scripts let you get a Python interactive session or run scripts with a controlled
path.

...

>>>   Neither can be entirely
>>> compatible with a system-wide Python installation, because Python's
>>> standard site.py f**ks up the environment really early in the process,
>>> and avoiding that isn't all that easy.
>>
>> This reminds me of a place where buildout is looser than workenv.  
>> buildout doesn;t try to disable anything in the system python.  It 
>> just augments it.  I always use a clean python, so avoiding 
>> customizations in the Python I use isn't a problem.  If I wanted to 
>> take advantage of something in a system Python, as I occasionally do, 
>> I can do that with buildout.
> 
> I find the isolation useful when testing things for release;

Yup.  I do the same things by always using clean Python installs.
I *never* use a system Python for development.  I always use a
Python that I build myself from sources.

Your approach is certainly a valid alternative.

> I can be 
> sure that I haven't been using any packages that I don't explicitly 
> include in the egg requirements or instructions. 

Yup

> But it can be annoying 
> in other cases, like when there's a library that doesn't install cleanly 
> (of which there's still quite a few). 

Yes


> Anyway, if you do want to include 
> the global packages, --site-packages will change your workingenv to do so.

Cool.

> It could be argued that workingenv's default should be to include 
> site-packages.  Another option would be to have a tool that allows you 
> to easily include something from the system Python (probably just a tool 
> to manage a custom .pth file, which works even when setuptools' fairly 
> heroic attempts to fix broken setup.py's doesn't work).

<shrug>  My remark was not meant to criticize or to suggest a different
policy.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org