zc.buildout template recipes: concerns with [z3c|collective].recipe.template and other issues
I'm concerned about the state of the zc.buildout template recipes. I want one. I want some one-off files, specific to a certain project, for which writing a standalone recipe feels very heavy. Here are the template recipes I found: collective.recipe.template (Wichert Akkerman) iw.recipe.template (Ingeniweb) inquant.recipe.textfile (Stefan Eletzhofer) z3c.recipe.template (Uli Fouquet) buildout_script (Nathan Yergler) z3c.recipe.filetemplate (Philipp von Weitershausen) First, on the basis of this list, it strikes me that a lot of people want this functionality. Putting template support into zc.buildout itself would be nice. I envision some APIs (like the ones in easy_install.py that zc.recipe.egg uses) that other recipes could use easily. I would even want recipes like the zc.recipe.egg script recipe to allow specifying initialization code by identifying a template (i.e., ``initialization_template = buildout_templates/foo``, which would parse the given template and insert the code the same way the ``initialization`` parameter works). Second, there's no clear winner at the moment. I happened to find z3c.recipe.template, buildout_script, and z3c.recipe.filetemplate first. On the basis of those three, and the comments that z3c.recipe.template made about collective.recipe.template, iw.recipe.template, and inquant.recipe.textfile, I decided to settle on z3c.recipe.template to be the package to which I would contribute. It has tests, a variable substitution approach that seems to work well with buildout, and recent development activity. However, on starting to hack on its documentation to sketch the changes I wanted, I gradually realized that this was a fork of collective.recipe.template. Since collective.recipe.template is listed as BSD in setup.py (though I saw no explicit licensing otherwise), placing z3c.recipe.template in the zope.org ZPL-only repository is problematic. Since I don't find the prospect of creating yet another template recipe attractive, and Philipp's z3c.recipe.filetemplate looks like it can take my new features while maintaining backwards compatibility, I'll try that next, I suppose. But meanwhile, I'm concerned about the state of z3c.recipe.template. IMO, Uli and Wichert should try to reconcile the licensing and forking issues (with Uli taking the lead in those discussions, ideally, since he is the one who forked). z3c.recipe.template should be removed from the zope.org repository, unless/until the licensing story is cleaned up. And maybe we can start to settle on a common template approach soon. This should be a problem that is solved for buildout canonically, IMO. I'll be happy to try and see if my features are generally interesting, once I make them. (Note: I could write about this on the distutils list but I'm not already subscribed there, and I suspect I have a better chance of catching some authors of the pertinent recipes over here, and some of this is about the zope.org repo.) Gary
Hi there, Gary Poster wrote:
I'm concerned about the state of the zc.buildout template recipes. I want one. I want some one-off files, specific to a certain project, for which writing a standalone recipe feels very heavy.
Here are the template recipes I found:
collective.recipe.template (Wichert Akkerman) iw.recipe.template (Ingeniweb) inquant.recipe.textfile (Stefan Eletzhofer) z3c.recipe.template (Uli Fouquet) buildout_script (Nathan Yergler) z3c.recipe.filetemplate (Philipp von Weitershausen)
First, on the basis of this list, it strikes me that a lot of people want this functionality. Putting template support into zc.buildout itself would be nice. I envision some APIs (like the ones in easy_install.py that zc.recipe.egg uses) that other recipes could use easily. I would even want recipes like the zc.recipe.egg script recipe to allow specifying initialization code by identifying a template (i.e., ``initialization_template = buildout_templates/foo``, which would parse the given template and insert the code the same way the ``initialization`` parameter works).
I was thinking about something like this for z3c.recipe.template as well. Should be not too difficult. But, well, first the legal stuff ;)
Second, there's no clear winner at the moment. I happened to find z3c.recipe.template, buildout_script, and z3c.recipe.filetemplate first. On the basis of those three, and the comments that z3c.recipe.template made about collective.recipe.template, iw.recipe.template, and inquant.recipe.textfile, I decided to settle on z3c.recipe.template to be the package to which I would contribute. It has tests, a variable substitution approach that seems to work well with buildout, and recent development activity.
Glad you liked it :)
However, on starting to hack on its documentation to sketch the changes I wanted, I gradually realized that this was a fork of collective.recipe.template. Since collective.recipe.template is listed as BSD in setup.py (though I saw no explicit licensing otherwise), placing z3c.recipe.template in the zope.org ZPL-only repository is problematic.
Yes, you're certainly right. Thanks for spotting this! The history: although I started the project from scratch, I also copied some central parts from Wicherts (very clean and straight) recipe. I'd therefore consider it myself to be derivative work (or fork) of BSD licensed code. In the beginning my code should go into collective.recipe.template itself (Wichert agreed), but I wasn't granted committer access to the collective repository yet. Of course I requested to be approval and waited for weeks, but nothing happened. As we needed the extra-functionality in grokproject urgently and Wichert had no time to implement it himself, I created a new project based on Wicherts work.
Since I don't find the prospect of creating yet another template recipe attractive, and Philipp's z3c.recipe.filetemplate looks like it can take my new features while maintaining backwards compatibility, I'll try that next, I suppose.
But meanwhile, I'm concerned about the state of z3c.recipe.template. IMO, Uli and Wichert should try to reconcile the licensing and forking issues (with Uli taking the lead in those discussions, ideally, since he is the one who forked). z3c.recipe.template should be removed from the zope.org repository, unless/until the licensing story is cleaned up.
I removed the source from the repository and asked Wichert for how to proceed. If there's a lawyer in the house: how can one switch from BSD to ZPL for derivative works if both parties agree? I put the sources in a private repository meanwhile. I'd prefer to put the sources into Zope repository again, but if this is legally too difficult, they will stay hosted elsewhere.
And maybe we can start to settle on a common template approach soon. This should be a problem that is solved for buildout canonically, IMO. I'll be happy to try and see if my features are generally interesting, once I make them.
What features do you think of? I already had some requests for features that were too 'special' so I refused to implement them. What I like with Wichert's approach (and which I tried to perpetuate) is, that it does simple things in a clear way and does not promise to do more. This way it does not become overloaded/overengineered. I'd propose to keep it this way. The initialization-code feature you mention above, however, sounds like an attractive extension (to whatever recipe or buildout itself). Best regards, -- Uli
Hi there, Uli Fouquet wrote:
Gary Poster wrote:
I'm concerned about the state of the zc.buildout template recipes. I want one. I want some one-off files, specific to a certain project, for which writing a standalone recipe feels very heavy.
Here are the template recipes I found:
collective.recipe.template (Wichert Akkerman) iw.recipe.template (Ingeniweb) inquant.recipe.textfile (Stefan Eletzhofer) z3c.recipe.template (Uli Fouquet) buildout_script (Nathan Yergler) z3c.recipe.filetemplate (Philipp von Weitershausen)
[snip]
However, on starting to hack on its documentation to sketch the changes I wanted, I gradually realized that this was a fork of collective.recipe.template. Since collective.recipe.template is listed as BSD in setup.py (though I saw no explicit licensing otherwise), placing z3c.recipe.template in the zope.org ZPL-only repository is problematic.
But meanwhile, I'm concerned about the state of z3c.recipe.template. IMO, Uli and Wichert should try to reconcile the licensing and forking issues (with Uli taking the lead in those discussions, ideally, since he is the one who forked). z3c.recipe.template should be removed from the zope.org repository, unless/until the licensing story is cleaned up.
I removed the source from the repository and asked Wichert for how to proceed. If there's a lawyer in the house: how can one switch from BSD to ZPL for derivative works if both parties agree?
I negotiated with Wichert and we will focus on collective.recipe.template from now on. I'll apply all changes, extensions, docs and tests that went into z3c.recipe.template (in fact only a few lines were copied from c.r.template) now to collective.recipe.template (one template generator less). Wichert does not want to switch the license to avoid license proliferation. I can understand that. So, if you want to contribute (I'd be happy to see this), you can use c.r.template from now on. Having template generating support in stock buildout, however, would also be nice. Best regards, -- Uli
Uli Fouquet wrote:
I put the sources in a private repository meanwhile. I'd prefer to put the sources into Zope repository again, but if this is legally too difficult, they will stay hosted elsewhere.
Why not merge it back to the original recipe in the collective now? Just tell me your plone.org username and I'll give you collective access in a matter of a day. Hanno
Hi there, Hanno Schlichting wrote:
Uli Fouquet wrote:
I put the sources in a private repository meanwhile. I'd prefer to put the sources into Zope repository again, but if this is legally too difficult, they will stay hosted elsewhere.
Why not merge it back to the original recipe in the collective now?
That's exactly what's happening right now :)
Just tell me your plone.org username and I'll give you collective access in a matter of a day.
Looks like I was just to dumb to use my collective access correctly. Many thanks to all friendly people that offered help! And my apologies to the collective admins, esp. gotcha! Beside this it would really be nice, if one could share code at least in the Zope-, Plone and collective repositories with less legal hassle. But I guess this was already discussed extensively here, so I'd better shut up ;) Best regards, -- Uli
Hey, Uli Fouquet wrote: [snip]
Beside this it would really be nice, if one could share code at least in the Zope-, Plone and collective repositories with less legal hassle. But I guess this was already discussed extensively here, so I'd better shut up ;)
Normally the way code is shared is not by copying it but simply importing it. :) If you import, there's no problem building on BSD-licensed code from ZPL-ed code. Or LGPL-ed code either. Or vice-versa If you use code in any of those licenses relying on any other of those licenses, you'll just be able to use without concern. Only if you want to modify an underlying library do you need to think about the implications of the license. Only building on GPL-ed code has some issues, but from what I understand the Plone developers have done quite a lot of work on making sure stuff is BSD-ed or LGPL-ed now. I'd recommend against building libraries in svn.zope.org that rely on a GPL-ed library for their operation (though I do not think it's formally disallowed). Only if you copy code into our repository do you need to worry about licenses. For small snippets of about 10 lines or so I think a combination of fair use and surface-rewrites which typically are useful anyway will take the license worries away (but I'm not a lawyher). Larger pieces of code than that, i.e. you're forking, you need to worry about the license when submitting it to svn.zope.org. But that's one of the reasons we *have* our copyright policy! But when you fork you need to worry about other things anyway. I'm glad this fork got unforked! Thanks Gary for pointing it out, Uli and Wichert for unforking, and others for helping Uli getting access and the like. Regards, Martijn
Uli Fouquet wrote:
In the beginning my code should go into collective.recipe.template itself (Wichert agreed), but I wasn't granted committer access to the collective repository yet. Of course I requested to be approval and waited for weeks, but nothing happened.
I'm sorry to hear that! In general, access to the Collective is very unproblematic. Find an op in #plone on irc.freenode.net and they should be able to help you. How did you request the access? Clearly, something's broken down. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
Thanks Uli, Wichert, and Hanno for working out the legal bits! And thanks to Martijn and Martin for the other replies. On Friday I had moved to z3c.recipe.filetemplate, for the reasons I had described then. Philipp said I could run with that package. However, I'd prefer to work with a more active project. If there's quick agreement on my additions to the collective recipe, and I'm given commit access, I'll switch. I don't have time for a bikeshed discussion though, so if it descends much into that I'll stick with extending Philipp's recipe for now, and maybe switch over later when things settle down. If I built on collective.recipe.template, I'd make the following changes. REQUIRED - verify that the BSD licensing rules are followed (headers, license inclusion, copyright statement, etc.), and fix if not. I'd prefer if a foundation (e.g., the Plone Foundation) had the copyright. (TBH, I'm more comfortable with the Zope Foundation repository because the rules for copyright assignment are clearer AIUI, even if they are breached sometimes, as was this case here. But this isn't critical for my usage or contribution.) - Add the ability to specify "eggs" and "extra-paths" the way you can for scripts. These supply the values for three predefined options (available to the template). If "paths" are the non-zip paths, and "all_paths" are all paths, then the options woud be defined roughly as given here: os-paths: (os.pathsep).join(paths) string-paths: ', '.join(repr(p) for p in all_paths) space-paths: ' '.join(paths) - I have a directory of *.in files that will need to be processed, with shared options, and put in another directory. Therefore, I'd like to be able to just specify the input directory and optionally the output directory. globs, for filters, might be a nice-to-have. I came up with a spelling for all this that appealed to me for Philipp's package (which has a constraint of "I only use *.in inputs, and always strip ".in" for the output). WIth my variant of his spelling, I could do everything I wanted with a line like files = * source-directory = templates Then in ``templates`` you would arrange the directory structure you wanted, and make sure that your template files end with ".in". I don't have a spelling I like as much for the "input" "output" pattern of the collective recipe. I'd be OK with "input" and "output" being able to take multiple files: input = templates/etc/foo.in templates/etc/bar.in output = etc/foo etc/bar That seems like it rocks the boat least for collective.recipe.template NICE-TO-HAVE Unless I discover I need this, these are just ideas that I might do on hobby time. - Extend/override other buildout sections. Here's an example, with Philipp's package. Consider the following buildout:: >>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = message ... ... [template_defaults] ... mygreeting = Hi ... myaudience = World ... ... [message] ... recipe = z3c.recipe.filetemplate ... files = helloworld.txt ... extends = template_defaults ... ... myaudience = everybody ... """) The "message" section now has values extended from the "template_defaults" section, and overwritten locally. A template of ``${mygreeting}, ${myaudience}!`` would thus result in ``Hi, everybody! ``. - Define option values in Python. This would have the os and sys modules in the namespace, and would also have the buildout variables, and would have the paths and all_paths from the ``eggs`` section above so you could manipulate them as desired. interpreted-options = path-separator=os.pathsep example-paths=' '.join('-t%s' % p for p in paths) That would put ``path-separator`` and ``example-paths`` in the recipe's options. - Make a script recipe that allows you to specify an ``initialization_template``, as I described in my first email. I'm pretty busy all around right now, and there's always the YAGNI concern, so I probably won't do those, but they seem like potentially good ideas. Gary
Hi Gary, Previously Gary Poster wrote:
Thanks Uli, Wichert, and Hanno for working out the legal bits! And thanks to Martijn and Martin for the other replies.
On Friday I had moved to z3c.recipe.filetemplate, for the reasons I had described then. Philipp said I could run with that package. However, I'd prefer to work with a more active project. If there's quick agreement on my additions to the collective recipe, and I'm given commit access, I'll switch. I don't have time for a bikeshed discussion though, so if it descends much into that I'll stick with extending Philipp's recipe for now, and maybe switch over later when things settle down.
Commit access to the collective is very liberal and generally arranged within 24 hours. Uli has both commit and pypi access for the package and has already done a lot of work on it. Your contributions are very welcome as well!
If I built on collective.recipe.template, I'd make the following changes.
REQUIRED
- verify that the BSD licensing rules are followed (headers, license inclusion, copyright statement, etc.), and fix if not. I'd prefer if a foundation (e.g., the Plone Foundation) had the copyright. (TBH, I'm more comfortable with the Zope Foundation repository because the rules for copyright assignment are clearer AIUI, even if they are breached sometimes, as was this case here. But this isn't critical for my usage or contribution.)
The BSD license has very little restrictions. It certainly does not require licens statementsin every source file or so. Personally I find those to be clutter. If there is a missing license file that should be fixed.
- Add the ability to specify "eggs" and "extra-paths" the way you can for scripts. These supply the values for three predefined options (available to the template). If "paths" are the non-zip paths, and "all_paths" are all paths, then the options woud be defined roughly as given here:
os-paths: (os.pathsep).join(paths) string-paths: ', '.join(repr(p) for p in all_paths) space-paths: ' '.join(paths)
I can see that being useful.
- I have a directory of *.in files that will need to be processed, with shared options, and put in another directory. Therefore, I'd like to be able to just specify the input directory and optionally the output directory. globs, for filters, might be a nice-to-have. I came up with a spelling for all this that appealed to me for Philipp's package (which has a constraint of "I only use *.in inputs, and always strip ".in" for the output). WIth my variant of his spelling, I could do everything I wanted with a line like
files = * source-directory = templates
Then in ``templates`` you would arrange the directory structure you wanted, and make sure that your template files end with ".in".
I don't have a spelling I like as much for the "input" "output" pattern of the collective recipe. I'd be OK with "input" and "output" being able to take multiple files:
input = templates/etc/foo.in templates/etc/bar.in output = etc/foo etc/bar
That seems like it rocks the boat least for collective.recipe.template
+1
NICE-TO-HAVE
Unless I discover I need this, these are just ideas that I might do on hobby time.
- Extend/override other buildout sections. Here's an example, with Philipp's package. Consider the following buildout::
>>> write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = message ... ... [template_defaults] ... mygreeting = Hi ... myaudience = World ... ... [message] ... recipe = z3c.recipe.filetemplate ... files = helloworld.txt ... extends = template_defaults ... ... myaudience = everybody ... """)
The "message" section now has values extended from the "template_defaults" section, and overwritten locally. A template of ``${mygreeting}, ${myaudience}!`` would thus result in ``Hi, everybody! ``.
This feels more like a feature for buildout: I can see this being useful for other recipes as well.
- Define option values in Python. This would have the os and sys modules in the namespace, and would also have the buildout variables, and would have the paths and all_paths from the ``eggs`` section above so you could manipulate them as desired.
interpreted-options = path-separator=os.pathsep example-paths=' '.join('-t%s' % p for p in paths)
That would put ``path-separator`` and ``example-paths`` in the recipe's options.
Again, this sounds more like a buildout extension. I've missed the option to do shell escapes in buildout as well. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
On Apr 27, 2009, at 4:33 PM, Wichert Akkerman wrote:
Hi Gary,
Hey Wichert. Sorry for not replying sooner. I had already gone pretty far down the z3c.recipe.filetemplate road, so I kept going over there. I just made a release. It is http://pypi.python.org/pypi/z3c.recipe.filetemplate if you are interested. I obviously learned a lot from your design. I wanted to get that release out before replying.
Previously Gary Poster wrote:
Thanks Uli, Wichert, and Hanno for working out the legal bits! And thanks to Martijn and Martin for the other replies.
On Friday I had moved to z3c.recipe.filetemplate, for the reasons I had described then. Philipp said I could run with that package. However, I'd prefer to work with a more active project. If there's quick agreement on my additions to the collective recipe, and I'm given commit access, I'll switch. I don't have time for a bikeshed discussion though, so if it descends much into that I'll stick with extending Philipp's recipe for now, and maybe switch over later when things settle down.
Commit access to the collective is very liberal and generally arranged within 24 hours. Uli has both commit and pypi access for the package and has already done a lot of work on it. Your contributions are very welcome as well!
Thank you, excellent. If you want me to help migrating some of the z3c.recipe.filetemplate features over to the collective version, let me know. I'd like to cooperate. It would be strictly hobby-time work, though, and I have a marked reduction in that lately. :-/ Hopefully things will have calmed down by the fall. :-)
If I built on collective.recipe.template, I'd make the following changes.
REQUIRED
- verify that the BSD licensing rules are followed (headers, license inclusion, copyright statement, etc.), and fix if not. I'd prefer if a foundation (e.g., the Plone Foundation) had the copyright. (TBH, I'm more comfortable with the Zope Foundation repository because the rules for copyright assignment are clearer AIUI, even if they are breached sometimes, as was this case here. But this isn't critical for my usage or contribution.)
The BSD license has very little restrictions. It certainly does not require licens statementsin every source file or so. Personally I find those to be clutter. If there is a missing license file that should be fixed.
I don't think the license is actually included in the collective recipe ATM. Maybe I misremember. But beyond that, BSD doesn't need headers, IIUC, so at least that's not necessary, yeah.
- Add the ability to specify "eggs" and "extra-paths" the way you can for scripts. These supply the values for three predefined options (available to the template). If "paths" are the non-zip paths, and "all_paths" are all paths, then the options woud be defined roughly as given here:
os-paths: (os.pathsep).join(paths) string-paths: ', '.join(repr(p) for p in all_paths) space-paths: ' '.join(paths)
I can see that being useful.
Cool.
- I have a directory of *.in files that will need to be processed, with shared options, and put in another directory. Therefore, I'd like to be able to just specify the input directory and optionally the output directory. globs, for filters, might be a nice-to-have. I came up with a spelling for all this that appealed to me for Philipp's package (which has a constraint of "I only use *.in inputs, and always strip ".in" for the output). WIth my variant of his spelling, I could do everything I wanted with a line like
files = * source-directory = templates
Then in ``templates`` you would arrange the directory structure you wanted, and make sure that your template files end with ".in".
I don't have a spelling I like as much for the "input" "output" pattern of the collective recipe. I'd be OK with "input" and "output" being able to take multiple files:
input = templates/etc/foo.in templates/etc/bar.in output = etc/foo etc/bar
That seems like it rocks the boat least for collective.recipe.template
+1
Cool. I like being able to specify a template source too though. Having to do that *and* specify the individual files seems like an unnecessary DRY violation. I like Philipp's approach of enforcing a *.in -> * pattern, because it is a simple, useful, old convention.
NICE-TO-HAVE
Unless I discover I need this, these are just ideas that I might do on hobby time.
- Extend/override other buildout sections. Here's an example, with Philipp's package. Consider the following buildout::
write(sample_buildout, 'buildout.cfg', ... """ ... [buildout] ... parts = message ... ... [template_defaults] ... mygreeting = Hi ... myaudience = World ... ... [message] ... recipe = z3c.recipe.filetemplate ... files = helloworld.txt ... extends = template_defaults ... ... myaudience = everybody ... """)
The "message" section now has values extended from the "template_defaults" section, and overwritten locally. A template of ``${mygreeting}, ${myaudience}!`` would thus result in ``Hi, everybody! ``.
This feels more like a feature for buildout: I can see this being useful for other recipes as well.
It is a standard buildout recipe pattern, but it is usually implemented per recipe. It's trivial to implement. I did so. I'm not sure how buildout would support it external to the recipe.
- Define option values in Python. This would have the os and sys modules in the namespace, and would also have the buildout variables, and would have the paths and all_paths from the ``eggs`` section above so you could manipulate them as desired.
interpreted-options = path-separator=os.pathsep example-paths=' '.join('-t%s' % p for p in paths)
That would put ``path-separator`` and ``example-paths`` in the recipe's options.
Again, this sounds more like a buildout extension. I've missed the option to do shell escapes in buildout as well.
Well, see how I did it, anyway, and see if you like the idea. What you provide in the locals/globals is a decision that usually you want per-recipe, so again, I'm not sure how buildout would do it. Thanks Gary
Gary Poster wrote:
I'm concerned about the state of the zc.buildout template recipes. I want one. I want some one-off files, specific to a certain project, for which writing a standalone recipe feels very heavy.
Here are the template recipes I found:
collective.recipe.template (Wichert Akkerman)
FTR, this seems to be the de-facto choice in the Plone world, and it's quite well production-tested. I didn't even know about the others. :)
z3c.recipe.template (Uli Fouquet)
It's a bit disappointing that Uli created a new package rather than add the relatively minor feature missing from collective.recipe.template according to the PyPI page, although I agree the lack of tests is off-putting and should be fixed. Still, since the package was based on the other one, I don't see why we couldn't have added tests to collective.recipe.template rather than fork it. :-/
Since I don't find the prospect of creating yet another template recipe attractive, and Philipp's z3c.recipe.filetemplate looks like it can take my new features while maintaining backwards compatibility, I'll try that next, I suppose.
What features do you want to add? Why can't they (plus the tests) go into the original collective.recipe.template? Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
participants (6)
-
Gary Poster -
Hanno Schlichting -
Martijn Faassen -
Martin Aspeli -
Uli Fouquet -
Wichert Akkerman