Deploying with buildout (was: buildout, instancemanager, and eggs)
We have a Plone 3.0/Zope 2.10 application that we need to develop on, maintain and deploy to a ZEO server and multiple ZEO clients. Currently we use svn bundles of Zope and Plone and our own packages and products in one big svn:externals definition. We also use some more recent versions of certain products and packages than those included in the upstream svn:externals. For example: Zope svn://svn.zope.org/repos/main/Zope/branches/2.10 Zope/lib/python/zope/app/interface svn://svn.zope.org/repos/main/zope.app.interface/trunk/src/zope/app/interface Though I haven't seen any documentation to this effect, svn seems to treat the second svn:externals as a "svn switch" on "svn up". So one question I have is how would I "override" the versions of some package like this using buildout? Another question I have is that there aren't releases for all the dependencies our application has. I'm wondering what the best way is to define a canonical release for deployment to our cluster. Ideally, we'd have a buildout (or whatever) on a staging server where we're testing the candidate for deployment and then when all tests pass that *exact* source should be deployed to the cluster. How might one do this without upstream releases? Is there a way to get buildout and setuptools to play together so that I can build a source distribution from the test passing buildout on staging? Is there a way I can have a buildout on each cluster node such that a simple ./bin/buildout will get the latest deployment release for that node? Ross
On Aug 27, 2007, at 10:59 PM, Ross Patterson wrote:
We have a Plone 3.0/Zope 2.10 application that we need to develop on, maintain and deploy to a ZEO server and multiple ZEO clients. Currently we use svn bundles of Zope and Plone and our own packages and products in one big svn:externals definition. We also use some more recent versions of certain products and packages than those included in the upstream svn:externals. For example:
Zope svn://svn.zope.org/repos/main/Zope/branches/2.10 Zope/lib/python/zope/app/interface svn://svn.zope.org/repos/main/ zope.app.interface/trunk/src/zope/app/interface
Though I haven't seen any documentation to this effect, svn seems to treat the second svn:externals as a "svn switch" on "svn up".
So one question I have is how would I "override" the versions of some package like this using buildout?
Putting these packages in your Zope instance home's "lib/python" directory should get them found before the others. For Products, you can use an additional "products-path" directive in zope.conf and put files in there. I don't know how to tell buildout to do either, though.
Another question I have is that there aren't releases for all the dependencies our application has. I'm wondering what the best way is to define a canonical release for deployment to our cluster. Ideally, we'd have a buildout (or whatever) on a staging server where we're testing the candidate for deployment and then when all tests pass that *exact* source should be deployed to the cluster.
I always use a vendor import for this (to a separate CVS server, I don't use SVN).
How might one do this without upstream releases? Is there a way to get buildout and setuptools to play together so that I can build a source distribution from the test passing buildout on staging? Is there a way I can have a buildout on each cluster node such that a simple ./bin/buildout will get the latest deployment release for that node?
Personally I use a script that knows about all of the revisions of all of the products and packages I want to deploy, and which creates a "sandbox", downloading and installing all of them (as well as creating instances, etc). It doesn't work very well for upgrades, but for starting a sandbox fresh, it's fine. See http:// agendaless.com/Members/chrism/software/buildit and http:// www.plope.com/Members/chrism/buildit_example . - C
Chris McDonough <chrism@plope.com> writes:
On Aug 27, 2007, at 10:59 PM, Ross Patterson wrote:
We have a Plone 3.0/Zope 2.10 application that we need to develop on, maintain and deploy to a ZEO server and multiple ZEO clients. Currently we use svn bundles of Zope and Plone and our own packages and products in one big svn:externals definition. We also use some more recent versions of certain products and packages than those included in the upstream svn:externals. For example:
Zope svn://svn.zope.org/repos/main/Zope/branches/2.10 Zope/lib/python/zope/app/interface svn://svn.zope.org/repos/main/ zope.app.interface/trunk/src/zope/app/interface
Though I haven't seen any documentation to this effect, svn seems to treat the second svn:externals as a "svn switch" on "svn up".
So one question I have is how would I "override" the versions of some package like this using buildout?
Putting these packages in your Zope instance home's "lib/python" directory should get them found before the others. For Products, you can use an additional "products-path" directive in zope.conf and put files in there. I don't know how to tell buildout to do either, though.
But wouldn't that cause problems in this case since zope.app.interface is a package in two nested-namespace packages. IOW, wouldn't putting zope/app/interface in $INSTANCE_HOME/lib/python mean that it was the only package in the zope or zope.app namespace packages?
Another question I have is that there aren't releases for all the dependencies our application has. I'm wondering what the best way is to define a canonical release for deployment to our cluster. Ideally, we'd have a buildout (or whatever) on a staging server where we're testing the candidate for deployment and then when all tests pass that *exact* source should be deployed to the cluster.
I always use a vendor import for this (to a separate CVS server, I don't use SVN).
Yeah, I guess that what we'll end up doing.
How might one do this without upstream releases? Is there a way to get buildout and setuptools to play together so that I can build a source distribution from the test passing buildout on staging? Is there a way I can have a buildout on each cluster node such that a simple ./bin/buildout will get the latest deployment release for that node?
Personally I use a script that knows about all of the revisions of all of the products and packages I want to deploy, and which creates a "sandbox", downloading and installing all of them (as well as creating instances, etc). It doesn't work very well for upgrades, but for starting a sandbox fresh, it's fine. See http:// agendaless.com/Members/chrism/software/buildit and http:// www.plope.com/Members/chrism/buildit_example .
Thanks, I'll check it out. Ross
On Aug 28, 2007, at 5:55 PM, Ross Patterson wrote:
Putting these packages in your Zope instance home's "lib/python" directory should get them found before the others. For Products, you can use an additional "products-path" directive in zope.conf and put files in there. I don't know how to tell buildout to do either, though.
But wouldn't that cause problems in this case since zope.app.interface is a package in two nested-namespace packages. IOW, wouldn't putting zope/app/interface in $INSTANCE_HOME/lib/python mean that it was the only package in the zope or zope.app namespace packages?
Gah. Yeah. Sorry. You might get away with creating and installing a custom zope.app.interface-myversion.egg into your Python's site- packages dir and using something like: from pkg_resources import require require('zope.app.interface = myversion') ... in a script early on or do the equivalent within a .pth file in your site-packages dir. Even after that, you *might* need to munge the PYTHONPATH, though, to prevent Zope from finding the one in SOFTWARE_HOME first. :-( I find Ian Bicking's virtual_python.py script very useful for trying things like this out. I usually install both a custom version of Python and one or more virtual pythons in my automated scripts (using the --no-site-packages flag to virtual_python.py). This means I can let distutils/setuptools packages install to site-packages "normally" without polluting the main Python installation's site-packages and so I don't need to go through gyrations to munge the PYTHONPATH for them to be found somewhere else. I think buildout tends to prefer creating scripts that munge the path. - C
Chris McDonough <chrism@plope.com> writes:
On Aug 28, 2007, at 5:55 PM, Ross Patterson wrote:
Putting these packages in your Zope instance home's "lib/python" directory should get them found before the others. For Products, you can use an additional "products-path" directive in zope.conf and put files in there. I don't know how to tell buildout to do either, though.
But wouldn't that cause problems in this case since zope.app.interface is a package in two nested-namespace packages. IOW, wouldn't putting zope/app/interface in $INSTANCE_HOME/lib/python mean that it was the only package in the zope or zope.app namespace packages?
Gah. Yeah. Sorry. You might get away with creating and installing a custom zope.app.interface-myversion.egg into your Python's site- packages dir and using something like:
from pkg_resources import require require('zope.app.interface = myversion')
... in a script early on or do the equivalent within a .pth file in your site-packages dir. Even after that, you *might* need to munge the PYTHONPATH, though, to prevent Zope from finding the one in SOFTWARE_HOME first. :-(
I find Ian Bicking's virtual_python.py script very useful for trying things like this out. I usually install both a custom version of Python and one or more virtual pythons in my automated scripts (using the --no-site-packages flag to virtual_python.py). This means I can let distutils/setuptools packages install to site-packages "normally" without polluting the main Python installation's site-packages and so I don't need to go through gyrations to munge the PYTHONPATH for them to be found somewhere else. I think buildout tends to prefer creating scripts that munge the path.
Great. Thanks, I'll check those out. Ross
participants (2)
-
Chris McDonough -
Ross Patterson