Hi there, Christian Theune wrote: the cited response from Christian did not make it to the list as I didn't reply to all in my first reply. Sorry!

On Sat, 2009-04-18 at 19:25 +0200, Uli Fouquet wrote:

...

We thought about this as well.

...

Some motivators for me, as I saw some of those variants around and some input with my experience from the buildout business:

- As a Unix-Lover, I do appreciate starting services having a SYSV style call, which means: (somecommand) start/stop/restart/...

Thus no calls that require me to remember a config file location and pass it on.

This also means that (somecommand) needs to be configurable as I might want to deploy multiple buildouts into the same machine having the deployment options create the service scripts in /etc/init.d and thus they can't all be called "instance". Having it named "myproject" seems better to me.

Okay, that should be easy with buildout.

...

- Also, I do like my autocomplete to leave me with full command name completions, thus: (somecommand)-ctl (somecommand)-debug doesn't work well for me: I always have to start typing again to complete the command *and* give a parameter. (Remember that I want parameters so I have SYSV compatible init commands right away)

I see. I am not sure, everybody has this requirement, but we should keep it in mind. The question here is: how do you name your commands then? Do you opt for (somecommand) --debug or similar in that case?

I consider failing tab-completion to be an indicator of something being wrong. ;)

The --debug is an interesting case here. Used to zdaemon, I merely use the commands to specify what to do with the (completely) configured service:

- start it - stop it - restart it - reload it - ...

Starting a daemon in --debug mode usually involves manually touching a configuration file - which would definitively be a hassle if you switch back and forth.

Why does it involve touching the configuration?

OTOH Zope has a "debug mode" parameter which you set globally for your server (in the zope.conf) to get debug functionality and then just start the server regularly. The one option we do differently from normal init scripts with zdaemon is the `fg` command which supports the debug/development scenario.

Yes, that's the Zope way. With paster, however, starting in debug mode means using a different .ini file. And it is not limited to debugging. There can be different setups for different purposes with different WSGI pipelines. The Zope "debug mode" parameter cannot easily be injected into this pipeline, at least as long as the publisher does not set an appropriate pipeline variable. This might change with Shanes new publisher implementation but currently "debug mode" means different things with paster and with Zope. I'd simply like to open that up for approaches the paster community is used to.

...

...

- In a buildout environment, the use of .in files breaks the flexibility that the various buildout mechanism deliver to us: extends, variable inclusion, etc. have to be built again and again and again in the recipes. We started out with zope3instance recipes that did that but finally moved away from this.

Interesting. This, at least with Grok, can become a serious problem. We currently have five different configuration files generated by `grokproject`. Admitted, that one might skip one (`debug.ini`), we are left with four, which require three different configuration grammars:

zope.conf, zdaemon.conf: ZConf grammar deploy.ini: ConfigParser grammar site.zcml: ZCML

but this is a different topic.

I'm not saying you shouldn't generete .ini files, but you should avoid .in files.

You can't avoid .ini files with paster (AFAIK). The problem here is to read and edit those files, not to generate them :) In the beginning I was very confused when I looked at buildout.cfgs with embedded site.zcml and zope.conf and thought that there was no other way until I realized that they were files-in-files. But maybe others can cope better with that right from the beginning.

...

Some of those files (namely the .ini files) meanwhile have grown large defining loggers and all that. Putting them into buildout.cfg makes it nearly unreadable. It becomes even difficult to check, which part of the configuration you're currently watching. People with more advanced setups might run into even greater trouble.

So, the pros and cons of external vs. buildout.cfg-embedded config files might read like this (for embedded configs):

* pros:

- supports all buildout mechanisms (namely: var interpolation and extend)

- all configs in one file (you know where to look for them)

* cons:

- harder to read

- confusing syntax (mix of at least three grammars)

I think you only mix two grammars at one point: buildout + the embedded config file. I don't think any of the config files are embedding another language in turn?

Right, every config file has it's own language. But you had to define several config files in buildout.cfg: zope.conf, site.zcml, deploy.ini, ... Each one speaking a different configuration language.

...

- impossible to split permissions on parts of the config

With external configs it's the opposite picture. Can you tell more about the problems with zope3instance?

I do not understand the `impossible to split permissions` part.

Not too important: you could grant different write permissions to different developers/administrators on a filesystem if you have the configuration split over several files. But this is also possible if you split the buildout.cfg-input files. So just forget that :)

The readability gets better when you split your buildout into functional sections and then re-combine using the `extends` mechanism.

The syntax I haven't found that much of a hassle yet, also I do agree that I came to like systems with concise configuration syntax much more than before (see nginx vs Apache).

If you don't want people to write a complete configuration file in the buildout configuration, the solution would be to move the complexity into a recipe (which in turn can assemble what you did already did without it and also rely on other recipes to do actual work).

And that's where you ran into trouble with zope3recipe? I'd really like to learn more about that :) With grokproject we currently use collective.recipe.template (well, a derived recipe that also supports path creation) to do this and we had no problems yet, at least none that block any buildout functionality AFAIK. The recipe gets an .in file, interpolates any variables (so you can refer to other recipes, ${buildout:directory} and friends), and writes that to a target file. Plain and works. The output file path is available for other recipes in buildout.cfg. I wonder, what problems could arise with that approach. [snip: do not force check-in of generated things]

...

If I try to conclude, then I see several decisions to make:

* "paster serve ..." vs. "commandname" approach.

It looks like that the "commandname" approach makes more sense with Zope as we certainly want to use the power of buildout and therefore have to use generated config files in 'exotic' (seen from the casual paster user's point of view) places.

* "commandname-dbg" vs. "commandname --debug"

This means, we have to decide, whether we want several commands performing special tasks (most probably: running everything in a special 'mode'), or one command that accepts options to indicate that special behaviour expected.

* "conffile.in" vs. buildout.cfg-embedded config files

I personally would really prefer the first approach for ease of use. If, however, this leads to problems with parsing the files, we are more or less bound to the latter. I'd like to investigate that further.

It looks like that the 'commandname-w/o-configfile-param' approach suits buildout-based environments best, which might be a good reason to be different in that respect from other paster-frameworks.

From my initial list of invokes (enriched by Hanno's Plone4 approach) then only some remain:

- "instance-fg <options>" and friends

- "mycommand [start|stop|status...] <options>"

The latter is partly already implemented by z3c.recipe.paster. Currently it does not support external config files and might do a bit more than needed (for example it calls ZConfig parser in schemaless mode, but there certainly is a good reason for it; in grokprojects we let zope.app.wsgi do this stuff), but it took me less than half an hour to create a flatter version that accepts both: internal and external config files (where external zope.confs are parsed with schemata) so that one can start a paster served grokproject with generated configuration files in parts/etc by doing::

$ bin/myapp

in foreground, where the script name is built from the buildout.cfg section name. It is actually a wrapper that runs::

paster serve buildout-set-ini-file [cmdline-options]

This means also, you have to do::

$ bin/myapp --daemon

There's zdaemon for this. It provides the ctl commands for any process, turning them into daemons or run in foreground.

...

to start the server in background (which is bad if you prefer SYSV compatible behaviour, i.e. ``bin/myapp start`` here), but you can do::

$ bin/myapp status

$ bin/myapp stop

which does, what one expects.

See above about zdaemon.

...

One can also pass the usual paster options like ``--reload``. There is, however, no possibility to start in debug mode, i.e. with a debug.ini as argument, except you declare another command (at least I don't see any way to do it different).

zdaemon lets you define those commands.

Hm, maybe we haven't paid enough attention to zdaemon yet (with grokproject). In fact we already provide zdaemon wrappers with grokproject: myproject-ctl and myproject-debug. I know that you don't like the names, but one can easily change them (or skip). I have, however, some problems with that approach I might tell: wrapping a wrapper? ------------------- It looks to me like when we invoke zdaemon to invoke paster, we do something like starting a wrapper around a wrapper, both evaluating cmdline arguments, providing daemon mode, etc. This looks strange to me as it makes the below listed things possible (but might be okay anyway). non-intuitive cmdline behaviour? -------------------------------- With zdaemon wrapping 'paster serve ...' you can do the following oneliners:: $ bin/myapp start # Does what one expects $ bin/myapp fg # Does what one expects $ bin/myapp stop # Does what one expects $ bin/myapp restart # Does what one expects $ bin/myapp reload # Error $ bin/myapp start --reload # Starts in file-monitoring mode $ bin/myapp --help # 'No help available' $ bin/myapp fg --help # Puts out paster help and exits $ bin/myapp start --debug # Fails without any notice on cmdline. You can do the following cmd sequences:: $ bin/myapp start --help # Puts paster help into logfile and exits $ bin/myapp stop # 'deamon manager not running' $ bin/myapp start --daemon # Starts daemon as daemon $ bin/myapp stop # 'daemon manager not running' $ bin/myapp start --stop-daemon # Really stops the daemon-daemon. $ bin/myapp fg --daemon # Starts daemonized $ bin/myapp stop # 'daemon manager not running' $ bin/myapp fg --stop-daemon # Stops everything. Some of the above things look really bizarre to me. I wonder whether it wouldn't be cleaner to use paster without a daemonizing wrapper but instead tell paster to daemonize and just set the appropriate options in the wrapper. For this purpose a recipe like z3c.recipe.paster might be better suited. Best regards, -- Uli