Hi there, Christian Theune wrote:

On Thu, 2009-04-16 at 01:01 +0200, Hanno Schlichting wrote:

...

Hi.

Sorry Hanno, your posting got lost on the way to my mailreader. I noticed you replied when reading Theuni's posting.

...

Uli Fouquet wrote:

...

As Grok is now approaching the 1.0 release we (i.e. some Grok developers and me) want to have a 'compliant' way to offer paster-based serving of Grok instances.

I'd love to have the same for Plone, but failed to find any canonical definition of "compliant" the same way you did.

Glad to hear that others have the same problems :)

...

...

Here's a short (probably incomplete) list of some 'full-stack' paster-capable 'frameworks' and how they invoke paster from the commandline (please correct/extend if I am wrong/incomplete here):

================ =================== ============================ Framework Ini-file location Paster call ================ =================== ============================ grokproject parts/etc/ bin/paster serve \ parts/etc/deploy.ini repoze.bfg <project root> paster serve MyProject.ini turbogears2 <project root> paster serve development.ini pylons <project root> paster serve development.ini zopeproject etc/ bin/paster serve \ etc/deploy.ini z3c.r.paster parts/<app>/ bin/app

I'll add the current Plone 4 way:

Plone parts/instance/etc bin/instance-fg

where instance-fg is a wrapper that calls:

bin/paster serve parts/instance/etc/paste.ini

We thought about this as well.

Some motivators for me, as I saw some of those variants around and some input with my experience from the buildout business:

- As a Unix-Lover, I do appreciate starting services having a SYSV style call, which means: (somecommand) start/stop/restart/...

Thus no calls that require me to remember a config file location and pass it on.

This also means that (somecommand) needs to be configurable as I might want to deploy multiple buildouts into the same machine having the deployment options create the service scripts in /etc/init.d and thus they can't all be called "instance". Having it named "myproject" seems better to me.

I see. I am not sure, everybody has this requirement, but we should keep it in mind. The question here is: how do you name your commands then? Do you opt for (somecommand) --debug or similar in that case?

- Also, I do like my autocomplete to leave me with full command name completions, thus: (somecommand)-ctl (somecommand)-debug doesn't work well for me: I always have to start typing again to complete the command *and* give a parameter. (Remember that I want parameters so I have SYSV compatible init commands right away)

- In a buildout environment, the use of .in files breaks the flexibility that the various buildout mechanism deliver to us: extends, variable inclusion, etc. have to be built again and again and again in the recipes. We started out with zope3instance recipes that did that but finally moved away from this.

Interesting. This, at least with Grok, can become a serious problem. We currently have five different configuration files generated by `grokproject`. Admitted, that one might skip one (`debug.ini`), we are left with four, which require three different configuration grammars: zope.conf, zdaemon.conf: ZConf grammar deploy.ini: ConfigParser grammar site.zcml: ZCML but this is a different topic. Some of those files (namely the .ini files) meanwhile have grown large defining loggers and all that. Putting them into buildout.cfg makes it nearly unreadable. It becomes even difficult to check, which part of the configuration you're currently watching. People with more advanced setups might run into even greater trouble. So, the pros and cons of external vs. buildout.cfg-embedded config files might read like this (for embedded configs): * pros: - supports all buildout mechanisms (namely: var interpolation and extend) - all configs in one file (you know where to look for them) * cons: - harder to read - confusing syntax (mix of at least three grammars) - impossible to split permissions on parts of the config With external configs it's the opposite picture. Can you tell more about the problems with zope3instance?

- Do not make me check in generated things.

Of course. As with buildout we _will_ have generated files (opposite to most other frameworks). This means that we cannot put flat ini files in the project/instance root (as they won't be flat files but generated ones).

- If you'd like people that know things like paster to pick up our environment easily, I think, that: make it obvious where the deploy.ini & Co went and as they are generated files, leave a visible note at the beginning of the file, that states something like:

# NOTE: This file is generated by foo.recipe.paster, it will be # overwritten. Please apply changes to your buildout configuration # (located in XXX) instead of changing this file.

Yep. Thanks for all the input! [snip: zopeproject and referencing 'instances'] If I try to conclude, then I see several decisions to make: * "paster serve ..." vs. "commandname" approach. It looks like that the "commandname" approach makes more sense with Zope as we certainly want to use the power of buildout and therefore have to use generated config files in 'exotic' (seen from the casual paster user's point of view) places. * "commandname-dbg" vs. "commandname --debug" This means, we have to decide, whether we want several commands performing special tasks (most probably: running everything in a special 'mode'), or one command that accepts options to indicate that special behaviour expected. * "conffile.in" vs. buildout.cfg-embedded config files I personally would really prefer the first approach for ease of use. If, however, this leads to problems with parsing the files, we are more or less bound to the latter. I'd like to investigate that further. It looks like that the 'commandname-w/o-configfile-param' approach suits buildout-based environments best, which might be a good reason to be different in that respect from other paster-frameworks. From my initial list of invokes (enriched by Hanno's Plone4 approach) then only some remain: - "instance-fg <options>" and friends - "mycommand [start|stop|status...] <options>" The latter is partly already implemented by z3c.recipe.paster. Currently it does not support external config files and might do a bit more than needed (for example it calls ZConfig parser in schemaless mode, but there certainly is a good reason for it; in grokprojects we let zope.app.wsgi do this stuff), but it took me less than half an hour to create a flatter version that accepts both: internal and external config files (where external zope.confs are parsed with schemata) so that one can start a paster served grokproject with generated configuration files in parts/etc by doing:: $ bin/myapp in foreground, where the script name is built from the buildout.cfg section name. It is actually a wrapper that runs:: paster serve buildout-set-ini-file [cmdline-options] This means also, you have to do:: $ bin/myapp --daemon to start the server in background (which is bad if you prefer SYSV compatible behaviour, i.e. ``bin/myapp start`` here), but you can do:: $ bin/myapp status $ bin/myapp stop which does, what one expects. One can also pass the usual paster options like ``--reload``. There is, however, no possibility to start in debug mode, i.e. with a debug.ini as argument, except you declare another command (at least I don't see any way to do it different). It should also be possible to make use of z.r.paster in Plone4, as one can easily generate an ``instance-fg`` script this way. I wonder, however, whether this is still wanted with paster. Hm, all this is confusing (and much too long). Maybe someone nevertheless has an opinion? Best regards, -- Uli

Hi there, Am Sonntag, den 19.04.2009, 09:33 +0200 schrieb Roger Ineichen:

...

Betreff: Re: [Zope-dev] Best way to invoke paster/WSGI apps on cmdline

Hi there,

Christian Theune wrote:

...

On Thu, 2009-04-16 at 01:01 +0200, Hanno Schlichting wrote:

...

Hi.

Sorry Hanno, your posting got lost on the way to my mailreader. I noticed you replied when reading Theuni's posting.

...

...

Uli Fouquet wrote:

...

As Grok is now approaching the 1.0 release we (i.e. some Grok developers and me) want to have a 'compliant' way to offer paster-based serving of Grok instances.

I'd love to have the same for Plone, but failed to find any canonical definition of "compliant" the same way you did.

Glad to hear that others have the same problems :)

...

...

...

Here's a short (probably incomplete) list of some 'full-stack' paster-capable 'frameworks' and how they invoke paster from the commandline (please correct/extend if I am wrong/incomplete here):

================ =================== ============================ Framework Ini-file location Paster call ================ =================== ============================ grokproject parts/etc/ bin/paster serve \ parts/etc/deploy.ini repoze.bfg <project root> paster serve MyProject.ini turbogears2 <project root> paster serve development.ini pylons <project root> paster serve development.ini zopeproject etc/ bin/paster serve \ etc/deploy.ini z3c.r.paster parts/<app>/ bin/app

I'll add the current Plone 4 way:

Plone parts/instance/etc bin/instance-fg

where instance-fg is a wrapper that calls:

bin/paster serve parts/instance/etc/paste.ini

We thought about this as well.

...

Some motivators for me, as I saw some of those variants around and some input with my experience from the buildout business:

- As a Unix-Lover, I do appreciate starting services having a SYSV style call, which means: (somecommand) start/stop/restart/...

Thus no calls that require me to remember a config file location and pass it on.

This also means that (somecommand) needs to be configurable as I might want to deploy multiple buildouts into the same machine having the deployment options create the service scripts in /etc/init.d and thus they can't all be called "instance". Having it named "myproject" seems better to me.

I see. I am not sure, everybody has this requirement, but we should keep it in mind. The question here is: how do you name your commands then? Do you opt for (somecommand) --debug or similar in that case?

...

- Also, I do like my autocomplete to leave me with full command name completions, thus: (somecommand)-ctl (somecommand)-debug doesn't work well for me: I always have to start typing again to complete the command *and* give a parameter. (Remember that I want parameters so I have SYSV compatible init commands right away)

- In a buildout environment, the use of .in files breaks the flexibility that the various buildout mechanism deliver to us: extends, variable inclusion, etc. have to be built again and again and again in the recipes. We started out with zope3instance recipes that did that but finally moved away from this.

Interesting. This, at least with Grok, can become a serious problem. We currently have five different configuration files generated by `grokproject`. Admitted, that one might skip one (`debug.ini`), we are left with four, which require three different configuration grammars:

zope.conf, zdaemon.conf: ZConf grammar deploy.ini: ConfigParser grammar site.zcml: ZCML

but this is a different topic.

Some of those files (namely the .ini files) meanwhile have grown large defining loggers and all that. Putting them into buildout.cfg makes it nearly unreadable. It becomes even difficult to check, which part of the configuration you're currently watching. People with more advanced setups might run into even greater trouble.

So, the pros and cons of external vs. buildout.cfg-embedded config files might read like this (for embedded configs):

* pros:

- supports all buildout mechanisms (namely: var interpolation and extend)

- all configs in one file (you know where to look for them)

* cons:

- harder to read

- confusing syntax (mix of at least three grammars)

- impossible to split permissions on parts of the config

With external configs it's the opposite picture. Can you tell more about the problems with zope3instance?

...

- Do not make me check in generated things.

Of course. As with buildout we _will_ have generated files (opposite to most other frameworks). This means that we cannot put flat ini files in the project/instance root (as they won't be flat files but generated ones).

...

- If you'd like people that know things like paster to pick up our environment easily, I think, that: make it obvious where the deploy.ini & Co went and as they are generated files, leave a visible note at the beginning of the file, that states something like:

# NOTE: This file is generated by foo.recipe.paster, it will be # overwritten. Please apply changes to your buildout configuration # (located in XXX) instead of changing this file.

Yep. Thanks for all the input!

[snip: zopeproject and referencing 'instances']

If I try to conclude, then I see several decisions to make:

* "paster serve ..." vs. "commandname" approach.

It looks like that the "commandname" approach makes more sense with Zope as we certainly want to use the power of buildout and therefore have to use generated config files in 'exotic' (seen from the casual paster user's point of view) places.

* "commandname-dbg" vs. "commandname --debug"

I'll implement this concept in a next z3c.recipe.paster release. The idea behind the z3c.recipe.paster is to define several instances for what you need to have.

This will also prevents to start a Zope instance in a debug mode if no debug option is defined. I'm not sure if this is the right ting for everyone, but I like to be restrictive on production systems.

...

This means, we have to decide, whether we want several commands performing special tasks (most probably: running everything in a special 'mode'), or one command that accepts options to indicate that special behaviour expected.

* "conffile.in" vs. buildout.cfg-embedded config files

I personally would really prefer the first approach for ease of use. If, however, this leads to problems with parsing the files, we are more or less bound to the latter. I'd like to investigate that further.

It looks like that the 'commandname-w/o-configfile-param' approach suits buildout-based environments best, which might be a good reason to be different in that respect from other paster-frameworks.

From my initial list of invokes (enriched by Hanno's Plone4 approach) then only some remain:

- "instance-fg <options>" and friends

- "mycommand [start|stop|status...] <options>"

The latter is partly already implemented by z3c.recipe.paster. Currently it does not support external config files and might do a bit more than needed (for example it calls ZConfig parser in schemaless mode, but there certainly is a good reason for it; in grokprojects we let zope.app.wsgi do this stuff), but it took me less than half an hour to create a flatter version that accepts both: internal and external config files (where external zope.confs are parsed with schemata) so that one can start a paster served grokproject with generated configuration files in parts/etc by doing::

$ bin/myapp

in foreground, where the script name is built from the buildout.cfg section name. It is actually a wrapper that runs::

paster serve buildout-set-ini-file [cmdline-options]

This means also, you have to do::

$ bin/myapp --daemon

to start the server in background (which is bad if you prefer SYSV compatible behaviour, i.e. ``bin/myapp start`` here), but you can do::

$ bin/myapp status

$ bin/myapp stop

I already have a zdaemon recipe which allows to use a zdaemon starting a paster setup. This looks like:

[xpo] recipe = p01.recipe.zdaemon deployment = deploy program = ${buildout:directory}/bin/app zdaemon.conf = <runner> program ${buildout:directory}/bin/app daemon on socket-name /var/run/zdaemon/xpo.sock transcript /var/log/zdaemon/xpo.log # Enable this to run the child process as a different user user zope </runner>

<eventlog> <logfile> path /var/log/zdaemon/xpo.log </logfile> </eventlog>

This allows to point to a paster app which was setup with a z3c.recipe.paster. and is using a deploy buildout like:

[deploy] recipe = zc.recipe.deployment name = zdaemon etc-directory = /etc/zdaemon log-directory = /var/log/zdaemon run-directory = /var/run/zdaemon logrotate-directory = /etc/logrotate.d crontab-directory = /etc/cron.d rc-directory = /etc/init.d user = zope

Let me know if you are interested and like to move that to the zope repos.

Very interesting. Thank you very much, Roger! Right now I have some problems with zdaemon as told in my reply to Christian. But if everybody likes it, also Grok/grokproject will go that way :) May I tell about my problems with z3c.recipe.paster as well? I know this sounds like complaining all the time, but I am also willing to help and contribute code if it helps. In general I am still attracted by z.r.paster :) For what I get from the discussion until now, I notice that everybody seems to agree, that Zope projects usually are different from other paster projects in that they use buildout and have some kind of meta-configuration. This means, that we have to generate for instance .ini files. Overall, this is not a problem but gives us the power of zc.buildout, which might also be attractive for other paster frameworks that still lack meta-configuration of that kind. If we don't want the 'genuine' paster approach ('paster serve ...') (because we want the power of buildout), then we might want to keep our way as open as possible for non-Zope frameworks, making buildout more attractive for others. What I mean by this is: if possible, one should avoid to require Zope-specific eggs (like ZConfig) in recipes installing paster wrappers. This blocks non-Zope pasters from using the recipe and everybody will start to do its own thing again. I think it is not necessary and further complicates setups (well, maybe it _is_ necessary, but I didn't get the concrete reason yet ;). It also means: appropriate recipes might focus on basic tasks and make no assumptions about the location or even the existance of 'zope.conf', 'site.zcml', etc. This should instead be set in the appropriate .ini file. The only thing I can think of, paster wrapper recipes should really be interested in, is an .ini file. So instead of extending z.r.paster to support more use-cases, I'd tend to flatten it and make it more simple, removing dependencies and the like. This is just a general line I'd like to follow and most probably I've overseen important pre-conditions or side-effects, so please correct me here. Best regards, -- Uli