[Zope-dev] Re: ploneout - Or how using zc.buildout for a common Zope2 project might look like

[Martin Aspeli]

Ian Bicking wrote:

>   After setting that project aside someone else at TOPP (Luke Tucker) 
> did a buildout for Deliverance because we needed to build some 
> non-Python libraries and that was a feature of buildout; that did end up 
> working eventually (after considerable effort), but it was not a very 
> satisfying experience, and *using* the buildout was itself a real 
> challenge.  Since Deliverance is just a library, to do anything useful I 
> also had to install another package that *uses* that library, and that 
> was surprisingly difficult.  Actually developing those libraries was 
> even more frustrating.

To my mind, using buildout to manage libraries like that is not what 
it's meant for (certainly not where I'd reach for it). Why isn't 
deliverance just an egg with its own dependencies?

Buildout to me is a way of setting up my zope instance (which is not an 
egg, but a bundle of software) and pulling in the necessary eggs and 
products and configuration in a repeatable and mangable way. It's also 
fairly open-ended, so that I could make extensions to the process of 
setting it all up specific to my use case (by making a new recipie and 
wiring it into buildout.cfg, which is as simple as writing a new egg 
with an entry point and a single class from what I've seen so far).

> So then Rob decides to devote some time to deployment.  And because Rob 
> just wants to get this finished, he wrote the whole things from scratch. 
>   You can see the result of that here: 
> https://svn.openplans.org/svn/topp.deploy/trunk -- it's not a generic 
> setup tool, just what works for us.  The end result is something that 
> does everything we want, that we understand, and that we'll understand 
> how to extend.  An important difference from zc.buildout is that all the 
> logic and work is in *our* script, not in a framework driven by a config 
> file. 

I think this is fine for your own needs. But consider that we want to 
document how people should set up their Plone development environment. 
If that involves "first, write a shell script that sets it all up, a bit 
like so, but tailored to your own needs" then that's pretty much a dead 
end for a large number of people who just want to Get Things Done.

At least with ploneout as it is now, you run two commands (boostrap, 
then buildout) and you have a fully functional Plone 3 site configured 
and you have something you can extend (in ways that are quite obvious to 
me, at least) for your own projects.

However, you *didn't* have ploneout, and I suspect that starting from 
the zc.* recipes for Zope 3 would've been frustrating. I also suspect 
that trying to solve the topp.* problem without a generic Plone version 
to build on would've been difficult.

> The deployment script also uses workingenv, somewhat similarly because 
> it is more library-like.  Well, also because I work at TOPP and can 
> easily support their use, which is certainly a nontrivial reason for the 
> choice.  Workingenv in this case is basically a tool that provides the 
> isolation that we need to create something repeatable.  This is the main 
> feature overlap with buildout.  Workingenv is not the framework in which 
> topp.deploy is written, and workingenv is not intended as a framework.

Indeed, and this is an important distinction.

> Note also that topp.deploy does not have the full set of features we'll 
> ultimately need.  You can't tell it to install another egg, or setup 
> another script, or whatever.  And we don't *have* to add those features, 
> because workingenv is compatible with all the other tools.  Where "all 
> the other" is mostly easy_install.  But someday there will be more, even 
> if the progress is slow.  buildout is basically incompatible with 
> easy_install (the script).  And frankly I like easy_install.  It's 
> probably 10x faster than buildout.

I think buildout is mostly slow (at least from ploneout experience) 
because some of the generalisations are a bit optimistic. For example, 
the current zope 2 recipie tries to check it all out from svn, which 
takes forever. I plan to fix that quickly. When I run it in offline 
mode, it takes seconds here at least.

> easy_install is what people use in 
> documentation, and its conventions are the ones people know (why does 
> buildout not use "Pkg==version", for instance?).

No idea, and good point.

I also think it's true that people document easy_install. However, 
easy_install is inherently "global" (it deals with a global 
site-packages). workingenv "localises" this so that it tricks 
easy_install into thinking the instance-specific setup is the global 
one. It does so by fiddling environment variables. It is stateful in 
that you need to activate and deactivate it. To me, this is less elegant 
in relation to the use case of managing a particular project (what if I 
forget to activate and then easy_install and accidentally upgrade 
something...) than having a well-defined single file that explains what 
this project is.

To some extent, working on Zope 2 projects is a bit different from 
working in a plain-python environment. Zope has the notion of instances 
with separate config files, start/stop scripts and datastore files. It's 
not just a script you write that sits in a package and has some 
dependencies. Maybe I'm blinded, though.

> As for the technical reasons they don't work together:
> 
> * workingenv allows and leaves it to setuptools to maintain the package 
> installation database (basically easy-install.pth).  This is not a very 
> good database, but eh.  buildout doesn't really have a database, but 
> instead just enforces what buildout.cfg indicates.
 >
> * workingenv relies on that database to give default versions and to 
> setup the Python path.  The fixup it does of installed scripts is fairly 
> minimal, just setting up sys.path enough to force its site.py to get 
> called.  buildout enumerates all the activated packages, and ignores 
> easy-install.pth.  This is basically what makes it 
> easy_install-incompatible.  Plus buildout's desire to own everything and 
> destroy everything it does not own ;)

That is somewhat down to recipes, though. I certainly want to tie this 
down in the zope 2 recipies to be less destructive (we're already almost 
there).

> * As a result buildout supports multiple things in the same buildout 
> that have conflicting version requirements, but where the packages 
> themselves don't realize this (but the deployer does).  If the packages 
> know their requirements then setuptools' native machinery allows things 
> to work fine.  The solution with workingenv is to create multiple 
> environments.  Since the actual building happens higher up (e.g., 
> topp.deploy), there's nothing stopping you from creating multiple 
> environments from one deployment script.  Anyway, in summary the way 
> scripts are generated is one of the major incompatibilities between 
> buildout and workingenv.  In effect Buildout's jail is too strong for 
> workingenv to penetrate, and buildout doesn't tell anyone else about 
> what it is doing.
> 
> * workingenv allows you to change versions without informing or using 
> workingenv.  Once you've created the environment you mostly stop using 
> workingenv directly.
> 
> * Some see bin/activate as a jail.  Both workingenv and buildout are 
> deliberately jail-like.  Both Jim and I loathe the non-repeatability of 
> system-wide installations (at least I think I can speak for him on that 
> one point ;).  bin/activate lets you into that jail, and lets you work 
> there.  There is no way into a buildout.   Frankly this weirds me out,
> and is a big part of my past frustration with it.  Maybe that's because 
> I'm in the relatively uncommon situation that I actually know what's 
> going on under the hood of Python imports and packaging, and so it 
> bothers me that I can't debug things directly.

Probably. I think it also has something to do with the fact that Zope 
people (me included) think in the context of a Zope instance as the 
ultimate controller of what gets executed. The buildout builds an 
instance plus some other stuff and informs it of what packages it has 
available.

>  Anyway, neither requires 
> activation when using scripts generated in the environment.  And 
> bin/activate is really just something that sets PYTHONPATH and then does 
> other non-essential things like changing the prompt and $PATH -- I 
> should probably document that more clearly.  Neither can be entirely 
> compatible with a system-wide Python installation, because Python's 
> standard site.py f**ks up the environment really early in the process, 
> and avoiding that isn't all that easy.  In some ways virtual-python is 
> the more complete solution to all of this, and sometimes I think I 
> should just use that technique (of a separate Python interpreter, and 
> hence separate prefix) with some of the improvements I could take from 
> workingenv.
> 
> Anyway, this is my very long summary of why we aren't using buildout, 
> and are using workingenv.

I am still trying to learn more about the details of how eggs and 
dependency management work. You certainly understand it better than me, 
and I found some of this quite enlightening. I would love it if Jim or 
someone more involved in the original design of zc.buildout had anything 
to add in answer to your specific points above. :)

Martin