environment variable support for zc.buildout, including extends! https://bugs.launchpad.net/zc.buildout/+bug/557769 works for me so far -- Florian Friesdorf <flo@chaoflow.net> GPG FPR: EA5C F2B4 FBBB BA65 3DCD E8ED 82A1 6522 4A1F 4367 Jabber/XMPP: flo@chaoflow.net OTR FPR: 9E191746 213321FE C896B37D 24B118C0 31785700 IRC: chaoflow on freenode,ircnet,blafasel,OFTC
On 04/08/2010 04:27 AM, Florian Friesdorf wrote:
environment variable support for zc.buildout, including extends!
https://bugs.launchpad.net/zc.buildout/+bug/557769
works for me so far
Uhm ... interesting. I never noticed our recipe not to work with extends. Can you give me an example? Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Thu, Apr 08, 2010 at 08:56:18AM +0200, Christian Theune wrote:
On 04/08/2010 04:27 AM, Florian Friesdorf wrote:
environment variable support for zc.buildout, including extends!
https://bugs.launchpad.net/zc.buildout/+bug/557769
works for me so far
Uhm ... interesting. I never noticed our recipe not to work with extends. Can you give me an example?
actually, i just assumed from looking at buildout.py - extends is processed before the whole Buildout ist up. As far as I understand your recipe is not even loaded when buildout would need that variable lookup. But, I saw you recipe after I implemented and never tried it... -- Florian Friesdorf <flo@chaoflow.net> GPG FPR: EA5C F2B4 FBBB BA65 3DCD E8ED 82A1 6522 4A1F 4367 Jabber/XMPP: flo@chaoflow.net OTR FPR: 9E191746 213321FE C896B37D 24B118C0 31785700 IRC: chaoflow on freenode,ircnet,blafasel,OFTC
Hi, On 04/08/2010 12:59 PM, Florian Friesdorf wrote:
On Thu, Apr 08, 2010 at 08:56:18AM +0200, Christian Theune wrote:
On 04/08/2010 04:27 AM, Florian Friesdorf wrote:
environment variable support for zc.buildout, including extends!
https://bugs.launchpad.net/zc.buildout/+bug/557769
works for me so far
Actually the env recipe was more of a hack to get going and then we forgot to propose getting it into buildout. OTOH handling it as a recipe allows for some other nice tricks, e.g. overriding by extensions. Maybe a specialised part-name, like versions would be helpful so that buildout could pre-populate that part during initialisation and then allow configurations to override individual values. On your specific patch: you sure about that direct regex match for the expanding variables in the extends option? I wonder what Jim thinks about the topic.
Uhm ... interesting. I never noticed our recipe not to work with extends. Can you give me an example?
actually, i just assumed from looking at buildout.py - extends is processed before the whole Buildout ist up. As far as I understand your recipe is not even loaded when buildout would need that variable lookup.
But, I saw you recipe after I implemented and never tried it...
I just played with that scenario. Something that doesn't work with our recipe: a.cfg: [buildout] parts = x [x] recipe = zc.recipe.egg eggs = zope.interface foo = ${env:bar} buildout.cfg: [buildout] extends = a.cfg [env] recipe = gocept.recipe.env -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Thu, Apr 8, 2010 at 8:24 AM, Christian Theune <ct@gocept.com> wrote:
Maybe a specialised part-name, like versions would be helpful so that buildout could pre-populate that part during initialisation and then allow configurations to override individual values.
It's probably worth mentioning that "versions" isn't a specialized part name in any sense; the name has to be explicitly mentioned in buildout:versions, and could be named anything. The use of explicit section references instead of a lot of "magic" section names is a good thing about buildout. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On 04/08/2010 02:57 PM, Fred Drake wrote:
On Thu, Apr 8, 2010 at 8:24 AM, Christian Theune<ct@gocept.com> wrote:
Maybe a specialised part-name, like versions would be helpful so that buildout could pre-populate that part during initialisation and then allow configurations to override individual values.
It's probably worth mentioning that "versions" isn't a specialized part name in any sense; the name has to be explicitly mentioned in buildout:versions, and could be named anything.
The use of explicit section references instead of a lot of "magic" section names is a good thing about buildout.
Yup. The "special" that I was referring to was the fact that you don't have to spell a recipe name for that section (like the buildout section). Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Thu, Apr 8, 2010 at 9:03 AM, Christian Theune <ct@gocept.com> wrote:
The "special" that I was referring to was the fact that you don't have to spell a recipe name for that section (like the buildout section).
There are many times you can have a section that isn't a part; the buildout section is just that, as is the section named by buildout:versions. Only parts need to have the recipe specified. *If* buildout should support some special "environment" section, I'd hope that it was explicitly referenced by something like buildout:environment. This would avoid the problem of breaking existing buildouts that already use sections with whatever conventional name arises for this feature. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On 04/08/2010 03:10 PM, Fred Drake wrote:
On Thu, Apr 8, 2010 at 9:03 AM, Christian Theune<ct@gocept.com> wrote:
The "special" that I was referring to was the fact that you don't have to spell a recipe name for that section (like the buildout section).
There are many times you can have a section that isn't a part; the buildout section is just that, as is the section named by buildout:versions. Only parts need to have the recipe specified.
*If* buildout should support some special "environment" section, I'd hope that it was explicitly referenced by something like buildout:environment. This would avoid the problem of breaking existing buildouts that already use sections with whatever conventional name arises for this feature.
Fullack. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Thu, Apr 8, 2010 at 8:24 AM, Christian Theune <ct@gocept.com> wrote:
I wonder what Jim thinks about the topic.
I have mixed feelings. Accessing environment variables seems reasonable on some level, however: - I'd worry about the choice of the virtual section name. "env" or "environment" seems too likely to break existing buildouts. - I worry a bit about making buildout's "programming language" even more sophisticated. Florian, can you describe why you want to use environment variables in extends? Jim -- Jim Fulton
On 04/08/2010 03:01 PM, Jim Fulton wrote:
On Thu, Apr 8, 2010 at 8:24 AM, Christian Theune<ct@gocept.com> wrote:
I wonder what Jim thinks about the topic.
I have mixed feelings. Accessing environment variables seems reasonable on some level, however:
- I'd worry about the choice of the virtual section name. "env" or "environment" seems too likely to break existing buildouts.
- I worry a bit about making buildout's "programming language" even more sophisticated.
Though so - same mixed feelings over here. :)
Florian, can you describe why you want to use environment variables in extends?
The general reason I can see is "it works everywhere else", except that there's some bootstrapping problem involved at some point anyway. Can you use any variable expansion in the values given in the [buildout] section anyway? Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Thu, Apr 8, 2010 at 9:05 AM, Christian Theune <ct@gocept.com> wrote: ...
Can you use any variable expansion in the values given in the [buildout] section anyway?
Only in a limited way that isn't well defined. :) Probably, you can't use expansion for options that are used early -- before the substitutions are performed. Jim -- Jim Fulton
On Thu, Apr 08, 2010 at 09:01:58AM -0400, Jim Fulton wrote:
On Thu, Apr 8, 2010 at 8:24 AM, Christian Theune <ct@gocept.com> wrote:
I wonder what Jim thinks about the topic.
I have mixed feelings. Accessing environment variables seems reasonable on some level, however:
- I'd worry about the choice of the virtual section name. "env" or "environment" seems too likely to break existing buildouts.
- I worry a bit about making buildout's "programming language" even more sophisticated.
Florian, can you describe why you want to use environment variables in extends?
The mail written during lunchtime made it out before I read this one... -- Florian Friesdorf <flo@chaoflow.net> GPG FPR: EA5C F2B4 FBBB BA65 3DCD E8ED 82A1 6522 4A1F 4367 Jabber/XMPP: flo@chaoflow.net OTR FPR: 9E191746 213321FE C896B37D 24B118C0 31785700 IRC: chaoflow on freenode,ircnet,blafasel,OFTC
On Thu, Apr 08, 2010 at 02:24:53PM +0200, Christian Theune wrote:
Hi,
On 04/08/2010 12:59 PM, Florian Friesdorf wrote:
On Thu, Apr 08, 2010 at 08:56:18AM +0200, Christian Theune wrote:
On 04/08/2010 04:27 AM, Florian Friesdorf wrote:
environment variable support for zc.buildout, including extends!
https://bugs.launchpad.net/zc.buildout/+bug/557769
works for me so far
Actually the env recipe was more of a hack to get going and then we forgot to propose getting it into buildout.
OTOH handling it as a recipe allows for some other nice tricks, e.g. overriding by extensions.
Maybe a specialised part-name, like versions would be helpful so that buildout could pre-populate that part during initialisation and then allow configurations to override individual values.
that would be nice indeed, but again would not work for extends We currently use it for 5 very similar sites that share one repository but use 5 checkouts of it: base.cfg [buildout] parts = instance develop = ... eggs = ... zcml = ... [instance] ... site-1.cfg [buildout] eggs += zcml += testenv.cfg [buildout] extends = base.cfg ${env:site}.cfg <stuff for testenv> deploy.cfg [buildout] extends = base.cfg ${env:site}.cfg <stuff for deploy> % site=site-1 ./bin/buildout -c deploy.cfg
On your specific patch: you sure about that direct regex match for the expanding variables in the extends option?
I'd prefer using the logic from Options, but that relies on self.buildout being an actual buildout already. I patched that up and actually succeeded to use Options for extends, but had some failing tests. The direct regex is only used at _open-time on extends. The only thing coming to my mind, why one would avoid a direct regex, is performance and the performance penalty should be minimal like that. However, I'm happy to be told different.
I wonder what Jim thinks about the topic.
me too
Uhm ... interesting. I never noticed our recipe not to work with extends. Can you give me an example?
actually, i just assumed from looking at buildout.py - extends is processed before the whole Buildout ist up. As far as I understand your recipe is not even loaded when buildout would need that variable lookup.
But, I saw you recipe after I implemented and never tried it...
I just played with that scenario. Something that doesn't work with our recipe:
a.cfg: [buildout] parts = x
[x] recipe = zc.recipe.egg eggs = zope.interface foo = ${env:bar}
buildout.cfg: [buildout] extends = a.cfg
[env] recipe = gocept.recipe.env
In the tests contained in the patches are even more sophisticated scenarios :) -- Florian Friesdorf <flo@chaoflow.net> GPG FPR: EA5C F2B4 FBBB BA65 3DCD E8ED 82A1 6522 4A1F 4367 Jabber/XMPP: flo@chaoflow.net OTR FPR: 9E191746 213321FE C896B37D 24B118C0 31785700 IRC: chaoflow on freenode,ircnet,blafasel,OFTC
Hello Florian, Thursday, April 8, 2010, 3:32:22 PM, you wrote: ... FF> We currently use it for 5 very similar sites that share one repository FF> but use 5 checkouts of it: Maybe you should take a look at keas.build. Tho not repo based, but solves such "almost identical sites" cases. -- Best regards, Adam GROSZER mailto:agroszer@gmail.com -- Quote of the day: One of the things that distinguishes man from the other animals is that he wants to know things, wants to find out what reality is like, simply for the sake of knowing. When that desire is completely quenched in anyone, I think he has become less than human. - C.S. Lewis
On Thu, Apr 08, 2010 at 04:45:30PM +0200, Adam GROSZER wrote:
Maybe you should take a look at keas.build. Tho not repo based, but solves such "almost identical sites" cases.
I will take a look into it, but for my use case it seems overkill and depends on svn, without support for git. -- Florian Friesdorf <flo@chaoflow.net> GPG FPR: EA5C F2B4 FBBB BA65 3DCD E8ED 82A1 6522 4A1F 4367 Jabber/XMPP: flo@chaoflow.net OTR FPR: 9E191746 213321FE C896B37D 24B118C0 31785700 IRC: chaoflow on freenode,ircnet,blafasel,OFTC
On Thu, Apr 8, 2010 at 9:32 AM, Florian Friesdorf <flo@chaoflow.net> wrote:
On Thu, Apr 08, 2010 at 02:24:53PM +0200, Christian Theune wrote:
Hi,
On 04/08/2010 12:59 PM, Florian Friesdorf wrote:
On Thu, Apr 08, 2010 at 08:56:18AM +0200, Christian Theune wrote:
On 04/08/2010 04:27 AM, Florian Friesdorf wrote:
environment variable support for zc.buildout, including extends!
https://bugs.launchpad.net/zc.buildout/+bug/557769
works for me so far
Actually the env recipe was more of a hack to get going and then we forgot to propose getting it into buildout.
OTOH handling it as a recipe allows for some other nice tricks, e.g. overriding by extensions.
Maybe a specialised part-name, like versions would be helpful so that buildout could pre-populate that part during initialisation and then allow configurations to override individual values.
that would be nice indeed, but again would not work for extends
We currently use it for 5 very similar sites that share one repository but use 5 checkouts of it:
base.cfg [buildout] parts = instance
develop = ... eggs = ... zcml = ...
[instance] ...
site-1.cfg [buildout] eggs += zcml +=
testenv.cfg [buildout] extends = base.cfg ${env:site}.cfg
<stuff for testenv>
deploy.cfg [buildout] extends = base.cfg ${env:site}.cfg
<stuff for deploy>
% site=site-1 ./bin/buildout -c deploy.cfg
Your extends usage seems a bit convoluted. Why not just have: - deploy.cfg extend base.cfg - site1-1.cfg etc extend deploy.cfg Then you'd just: % ./bin/buildout -c site-1.cfg Jim -- Jim Fulton
On Fri, Apr 09, 2010 at 10:16:51AM -0400, Jim Fulton wrote:
We currently use it for 5 very similar sites that share one repository but use 5 checkouts of it:
base.cfg [buildout] parts = instance
develop = ... eggs = ... zcml = ...
[instance] ...
site-1.cfg [buildout] eggs += zcml +=
testenv.cfg [buildout] extends = base.cfg ${env:site}.cfg
<stuff for testenv>
deploy.cfg [buildout] extends = base.cfg ${env:site}.cfg
<stuff for deploy>
% site=site-1 ./bin/buildout -c deploy.cfg
Your extends usage seems a bit convoluted. Why not just have:
- deploy.cfg extend base.cfg
- site1-1.cfg etc extend deploy.cfg
Then you'd just:
% ./bin/buildout -c site-1.cfg
That would mean: - 1 base.cfg - 1 deploy.cfg - 1 testenv.cfg - 5 site-specific extending deploy.cfg - 5 site-specific extending testenv.cfg total: 13 files Site-specific information reside in two cfgs, which is error-prone. In contrast to our current setup, with env var support, without the need to duplicate information. - 1 base.cfg - 1 deploy.cfg - 1 testenv.cfg - 5 site specific cfgs total: 8 files Site-specific information could be factored out, as in the setup with env var support: testenv-site-1.cfg: [buildout] extends = testenv.cfg site-1.cfg Information would not be duplicated but the amount of files increases to 18. Further we use a dev environment tailored for local individual development in contrast to the testenv which is meant for the customer to be kept up-to-date about the development, without e.g. PDBDebugMode. With that dev environment the numbers are 18 (with site-specific info in three places) or 23 vs. 9 files. This idea can be taken further to mass hosting in dedicated instances but based on one buildout, or branches of one buildout that share the majority of stuff. Other use cases coming to my mind: - ip address for zope to listen on: instead of hard-coded 8080 or 127.0.0.1:8080, you can have ${env:MYPROJECT_LISTEN} and people being involved in several projects can choose in order to have several buildouts running in parallel - location of the Data.fs: I like to have the Data.fs outside of the buildout dir for development, which enables the use of 'git clean -xdf', i.e. wiping everything which is not tracked. I like the idea of the need to configure the name of the environment section: [buildout] environment=env extends = ${env:...} At the time the extends are processed, the whole buildout section is already read into the options dict, so configuration of the name of the environment section is easily possible and can be used to trigger the substitution code: if extends: envsecname = options.get('environment') if envsecname: # do variable substitution like in my patch # if no environment is specified nothing changes # continue normally I would not pop() the options, so the name of the environment section can be queried by other parts of buildout. Likewise, extends could not be pop()'ed - both then would need to go onto a list, which buildout checks, in order not to complain about them being unused. If I could get commit acces to svn.zope.org, I would further work on this in a separate branch, more effective than via patches. Committer agreement I can fax/email. I am not committing to branches that don't belong to me, without prior consultation. So far I have commit access to the plone repo, which might be meaningless or at least a minor sign of credibility. I hope I could clarify the usefulness of env var support. florian -- Florian Friesdorf <flo@chaoflow.net> GPG FPR: EA5C F2B4 FBBB BA65 3DCD E8ED 82A1 6522 4A1F 4367 Jabber/XMPP: flo@chaoflow.net OTR FPR: 9E191746 213321FE C896B37D 24B118C0 31785700 IRC: chaoflow on freenode,ircnet,blafasel,OFTC
Hi Jim, hi Christian, any further thoughts on this? best regards florian On Fri, Apr 09, 2010 at 05:23:36PM +0200, Florian Friesdorf wrote:
On Fri, Apr 09, 2010 at 10:16:51AM -0400, Jim Fulton wrote:
We currently use it for 5 very similar sites that share one repository but use 5 checkouts of it:
base.cfg [buildout] parts = instance
develop = ... eggs = ... zcml = ...
[instance] ...
site-1.cfg [buildout] eggs += zcml +=
testenv.cfg [buildout] extends = base.cfg ${env:site}.cfg
<stuff for testenv>
deploy.cfg [buildout] extends = base.cfg ${env:site}.cfg
<stuff for deploy>
% site=site-1 ./bin/buildout -c deploy.cfg
Your extends usage seems a bit convoluted. Why not just have:
- deploy.cfg extend base.cfg
- site1-1.cfg etc extend deploy.cfg
Then you'd just:
% ./bin/buildout -c site-1.cfg
That would mean: - 1 base.cfg - 1 deploy.cfg - 1 testenv.cfg - 5 site-specific extending deploy.cfg - 5 site-specific extending testenv.cfg total: 13 files
Site-specific information reside in two cfgs, which is error-prone.
In contrast to our current setup, with env var support, without the need to duplicate information. - 1 base.cfg - 1 deploy.cfg - 1 testenv.cfg - 5 site specific cfgs total: 8 files
Site-specific information could be factored out, as in the setup with env var support:
testenv-site-1.cfg: [buildout] extends = testenv.cfg site-1.cfg
Information would not be duplicated but the amount of files increases to 18.
Further we use a dev environment tailored for local individual development in contrast to the testenv which is meant for the customer to be kept up-to-date about the development, without e.g. PDBDebugMode. With that dev environment the numbers are 18 (with site-specific info in three places) or 23 vs. 9 files.
This idea can be taken further to mass hosting in dedicated instances but based on one buildout, or branches of one buildout that share the majority of stuff.
Other use cases coming to my mind: - ip address for zope to listen on: instead of hard-coded 8080 or 127.0.0.1:8080, you can have ${env:MYPROJECT_LISTEN} and people being involved in several projects can choose in order to have several buildouts running in parallel - location of the Data.fs: I like to have the Data.fs outside of the buildout dir for development, which enables the use of 'git clean -xdf', i.e. wiping everything which is not tracked.
I like the idea of the need to configure the name of the environment section: [buildout] environment=env extends = ${env:...}
At the time the extends are processed, the whole buildout section is already read into the options dict, so configuration of the name of the environment section is easily possible and can be used to trigger the substitution code: if extends: envsecname = options.get('environment') if envsecname: # do variable substitution like in my patch # if no environment is specified nothing changes # continue normally
I would not pop() the options, so the name of the environment section can be queried by other parts of buildout. Likewise, extends could not be pop()'ed - both then would need to go onto a list, which buildout checks, in order not to complain about them being unused.
If I could get commit acces to svn.zope.org, I would further work on this in a separate branch, more effective than via patches. Committer agreement I can fax/email. I am not committing to branches that don't belong to me, without prior consultation. So far I have commit access to the plone repo, which might be meaningless or at least a minor sign of credibility.
I hope I could clarify the usefulness of env var support.
florian
-- Florian Friesdorf <flo@chaoflow.net> GPG FPR: EA5C F2B4 FBBB BA65 3DCD E8ED 82A1 6522 4A1F 4367 Jabber/XMPP: flo@chaoflow.net OTR FPR: 9E191746 213321FE C896B37D 24B118C0 31785700 IRC: chaoflow on freenode,ircnet,blafasel,OFTC
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
-- Florian Friesdorf <flo@chaoflow.net> GPG FPR: EA5C F2B4 FBBB BA65 3DCD E8ED 82A1 6522 4A1F 4367 Jabber/XMPP: flo@chaoflow.net OTR FPR: 9E191746 213321FE C896B37D 24B118C0 31785700 IRC: chaoflow on freenode,ircnet,blafasel,OFTC
On Mon, Apr 19, 2010 at 10:00 AM, Florian Friesdorf <flo@chaoflow.net> wrote:
Hi Jim, hi Christian,
any further thoughts on this?
Not at this time. This is on my to-do list to look at. I think your use case would be better handled through normal command-line option specification, as in: buildout -c deploy.cfg buildout:extends+=site1.cfg but that doesn't work. I think that should work and want to look at why it doesn't. Jim -- Jim Fulton
On Mon, Apr 19, 2010 at 10:28:37AM -0400, Jim Fulton wrote:
On Mon, Apr 19, 2010 at 10:00 AM, Florian Friesdorf <flo@chaoflow.net> wrote:
Hi Jim, hi Christian,
any further thoughts on this?
Not at this time. This is on my to-do list to look at.
I think your use case would be better handled through normal command-line option specification, as in:
buildout -c deploy.cfg buildout:extends+=site1.cfg
but that doesn't work. I think that should work and want to look at why it doesn't.
That'd be nice and would be sufficient for my current use case. And the discussion about environment variables we could continue on the base of this being functional. I took a look at it and so far came up with two docstrings (attached). I think I could simplify _open and make extends work as expected, in case you are comfortable with delegating that (see also 'desired behaviour' in the docstring for _open (attached). florian -- Florian Friesdorf <flo@chaoflow.net> GPG FPR: EA5C F2B4 FBBB BA65 3DCD E8ED 82A1 6522 4A1F 4367 Jabber/XMPP: flo@chaoflow.net OTR FPR: 9E191746 213321FE C896B37D 24B118C0 31785700 IRC: chaoflow on freenode,ircnet,blafasel,OFTC
participants (5)
-
Adam GROSZER -
Christian Theune -
Florian Friesdorf -
Fred Drake -
Jim Fulton