Hello, I'd like to extend zc.recipe.cmmi to support shared build directories. Use case example: we use lxml in a lot of our projects, which currently means having to build libxml and libxslt over and over again, since the buildout needs to be standalone and thus can't depend on them being installed on the system. But while that's a necessity for deployment, it's really annoying for development. Plan of attack: Introduce an option "shared" (defaults to False). If that is set, a) calculate HASH as a hash of the recipe's current options (e. g. configure parameters, environment variables) b) perform the cmmi in ${buildout:download-cache}/cmmi/build/<HASH>, if that directory is not present yet. (Probably needs a little thought about how to differentiate between "present and has a complete build" and "present but we errored out") Any thoughts? Thanks, Wolfgang -- Wolfgang Schnerring · ws@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Am Tue, 24 Mar 2009 10:26:08 +0100 schrieb Wolfgang Schnerring:
I'd like to extend zc.recipe.cmmi to support shared build directories.
Use case example: we use lxml in a lot of our projects, which currently means having to build libxml and libxslt over and over again, since the buildout needs to be standalone and thus can't depend on them being installed on the system. But while that's a necessity for deployment, it's really annoying for development.
So build once and use often, we do this i.e. for ldap by compiling in a separate buildout located in /opt/openldap and then have a section in the specific buildout like: [python-ldap] recipe = zc.recipe.egg:custom egg = python-ldap include-dirs = /opt/is/openldap/parts/open-ldap/include library-dirs = /opt/is/openldap/parts/open-ldap/lib rpath = /opt/is/openldap/parts/open-ldap/lib
Plan of attack: Introduce an option "shared" (defaults to False). If that is set, a) calculate HASH as a hash of the recipe's current options (e. g. configure parameters, environment variables) b) perform the cmmi in ${buildout:download-cache}/cmmi/build/<HASH>, if that directory is not present yet. (Probably needs a little thought about how to differentiate between "present and has a complete build" and "present but we errored out")
Any thoughts?
sounds good to me, it would simplify our infrastructure a lot. important is the hash: it needs to include all parameters, environment et al passed to the cmmi. regards Jens -- Jens W. Klein - Klein & Partner KEG - BlueDynamics Alliance
Wolfgang Schnerring <ws@gocept.com> writes:
I'd like to extend zc.recipe.cmmi to support shared build directories.
Use case example: we use lxml in a lot of our projects, which currently means having to build libxml and libxslt over and over again, since the buildout needs to be standalone and thus can't depend on them being installed on the system. But while that's a necessity for deployment, it's really annoying for development.
Plan of attack: Introduce an option "shared" (defaults to False). If that is set, a) calculate HASH as a hash of the recipe's current options (e. g. configure parameters, environment variables) b) perform the cmmi in ${buildout:download-cache}/cmmi/build/<HASH>, if that directory is not present yet. (Probably needs a little thought about how to differentiate between "present and has a complete build" and "present but we errored out")
This would be great. I'm always happy when a build recipe provides this option so I think generalizing it is a good idea. Ross
* Wolfgang Schnerring <ws@gocept.com> [2009-03-24 10:26]:
I'd like to extend zc.recipe.cmmi to support shared build directories.
Implemented in r98334. I'll use this locally over the next few days, and will then try to push out a release if it proves stable. Wolfgang -- Wolfgang Schnerring · ws@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
* Wolfgang Schnerring <ws@gocept.com> [2009-03-24 16:01]:
* Wolfgang Schnerring <ws@gocept.com> [2009-03-24 10:26]:
I'd like to extend zc.recipe.cmmi to support shared build directories.
Implemented in r98334. I'll use this locally over the next few days, and will then try to push out a release if it proves stable.
Looks good so far. Could someone do a release of zc.recipe.cmmi (or grant 'wosc' pypi access so I can do it myself)? Thanks, Wolfgang
On Tue, May 12, 2009 at 10:56 AM, Wolfgang Schnerring <ws@gocept.com> wrote:
Looks good so far. Could someone do a release of zc.recipe.cmmi (or grant 'wosc' pypi access so I can do it myself)?
Done. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
* Fred Drake <fdrake@gmail.com> [2009-05-16 14:09]:
On Tue, May 12, 2009 at 10:56 AM, Wolfgang Schnerring <ws@gocept.com> wrote:
Looks good so far. Could someone do a release of zc.recipe.cmmi (or grant 'wosc' pypi access so I can do it myself)?
Done.
Thanks! I've just cut the 1.2.0 release that includes the shared build directory functionality. Wolfgang
On 2009-03-24, Wolfgang Schnerring <ws@gocept.com> wrote:
I'd like to extend zc.recipe.cmmi to support shared build directories.
Use case example: we use lxml in a lot of our projects, which currently means having to build libxml and libxslt over and over again, since the buildout needs to be standalone and thus can't depend on them being installed on the system. But while that's a necessity for deployment, it's really annoying for development.
I added a feature to http://pypi.python.org/pypi/z3c.recipe.staticlxml last week that skips the expensive compilation step if there's already an lxml egg. I could do that by using a couple of assumptions that were build into the recipe. So if you need the shared build directory just for lxml, this could save you some work. The recipe uses zc.recipe.cmmi, though, if I remember correctly. So your approach could fix it at the source instead of my one-recipe-only solution :-) Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets
participants (5)
-
Fred Drake -
Jens W. Klein -
Reinout van Rees -
Ross Patterson -
Wolfgang Schnerring