[Zope-dev] Best way to invoke paster/WSGI apps on cmdline

[Uli Fouquet]

Hi there,

Christian Theune wrote:

the cited response from Christian did not make it to the list as I
didn't reply to all in my first reply. Sorry!

> On Sat, 2009-04-18 at 19:25 +0200, Uli Fouquet wrote:
> > We thought about this as well.
> > 
> > > Some motivators for me, as I saw some of those variants around and some
> > > input with my experience from the buildout business:
> > > 
> > > - As a Unix-Lover, I do appreciate starting services having a SYSV style
> > >   call, which means: (somecommand) start/stop/restart/...
> > > 
> > >   Thus no calls that require me to remember a config file location and
> > >   pass it on.
> > > 
> > >   This also means that (somecommand) needs to be configurable as I might
> > >   want to deploy multiple buildouts into the same machine having the
> > >   deployment options create the service scripts in /etc/init.d and thus
> > >   they can't all be called "instance". Having it named "myproject" seems
> > >   better to me.
> > 
> > Okay, that should be easy with buildout.
> > 
> > > - Also, I do like my autocomplete to leave me with full command name
> > >   completions, thus: (somecommand)-ctl
> > >   (somecommand)-debug doesn't work well for me: I always have to start
> > >   typing again to complete the command *and* give a parameter.
> > >   (Remember  that I want parameters so I have SYSV compatible init
> > >   commands right away)
> >
> > I see. I am not sure, everybody has this requirement, but we should keep
> > it in mind. The question here is: how do you name your commands then? Do
> > you opt for (somecommand) --debug or similar in that case?
> 
> I consider failing tab-completion to be an indicator of something being
> wrong. ;)
> 
> The --debug is an interesting case here. Used to zdaemon, I merely use the commands
> to specify what to do with the (completely) configured service:
> 
> - start it
> - stop it
> - restart it
> - reload it
> - ...
> 
> Starting a daemon in --debug mode usually involves manually touching a configuration
> file - which would definitively be a hassle if you switch back and forth.

Why does it involve touching the configuration?

> OTOH Zope has a "debug mode" parameter which you set globally for your server
> (in the zope.conf) to get debug functionality and then just start the server
> regularly. The one option we do differently from normal init scripts with zdaemon
> is the `fg` command which supports the debug/development scenario.

Yes, that's the Zope way. With paster, however, starting in debug mode
means using a different .ini file. And it is not limited to debugging.
There can be different setups for different purposes with different WSGI
pipelines. The Zope "debug mode" parameter cannot easily be injected
into this pipeline, at least as long as the publisher does not set an
appropriate pipeline variable. This might change with Shanes new
publisher implementation but currently "debug mode" means different
things with paster and with Zope.

I'd simply like to open that up for approaches the paster community is
used to. 

> > > - In a buildout environment, the use of .in files breaks the flexibility
> > >   that the various buildout mechanism deliver to us: extends, variable
> > >   inclusion, etc. have to be built again and again and again in the
> > >   recipes. We started out with zope3instance recipes that did that but
> > >   finally moved away from this.
> > 
> > Interesting. This, at least with Grok, can become a serious problem. We
> > currently have five different configuration files generated by
> > `grokproject`. Admitted, that one might skip one (`debug.ini`), we are
> > left with four, which require three different configuration grammars:
> > 
> >  zope.conf, zdaemon.conf: ZConf grammar
> >  deploy.ini: ConfigParser grammar
> >  site.zcml: ZCML
> > 
> > but this is a different topic.
> 
> I'm not saying you shouldn't generete .ini files, but you should
> avoid .in files. 

You can't avoid .ini files with paster (AFAIK). The problem here is to
read and edit those files, not to generate them :) In the beginning I
was very confused when I looked at buildout.cfgs with embedded site.zcml
and zope.conf and thought that there was no other way until I realized
that they were files-in-files. But maybe others can cope better with
that right from the beginning.

> > Some of those files (namely the .ini files) meanwhile have grown large
> > defining loggers and all that. Putting them into buildout.cfg makes it
> > nearly unreadable. It becomes even difficult to check, which part of the
> > configuration you're currently watching. People with more advanced
> > setups might run into even greater trouble.
> > 
> > So, the pros and cons of external vs. buildout.cfg-embedded config files
> > might read like this (for embedded configs):
> > 
> > * pros:
> > 
> >   - supports all buildout mechanisms 
> >     (namely: var interpolation and extend)
> > 
> >   - all configs in one file (you know where to look for them)
> > 
> > * cons:
> > 
> >   - harder to read
> > 
> >   - confusing syntax (mix of at least three grammars)
> 
> I think you only mix two grammars at one point: buildout + the embedded
> config file. I don't think any of the config files are embedding another
> language in turn?

Right, every config file has it's own language. But you had to define
several config files in buildout.cfg: zope.conf, site.zcml,
deploy.ini, ... Each one speaking a different configuration language.

> >   - impossible to split permissions on parts of the config
> > 
> > With external configs it's the opposite picture. Can you tell more about
> > the problems with zope3instance?
> 
> I do not understand the `impossible to split permissions` part.

Not too important: you could grant different write permissions to
different developers/administrators on a filesystem if you have the
configuration split over several files. But this is also possible if you
split the buildout.cfg-input files. So just forget that :)

> The readability gets better when you split your buildout into functional
> sections and then re-combine using the `extends` mechanism.
> 
> The syntax I haven't found that much of a hassle yet, also I do agree
> that I came to like systems with concise configuration syntax much more
> than before (see nginx vs Apache).
> 
> If you don't want people to write a complete configuration file in the
> buildout configuration, the solution would be to move the complexity
> into a recipe (which in turn can assemble what you did already did
> without it and also rely on other recipes to do actual work).

And that's where you ran into trouble with zope3recipe? I'd really like
to learn more about that :)

With grokproject we currently use collective.recipe.template (well, a
derived recipe that also supports path creation) to do this and we had
no problems yet, at least none that block any buildout functionality
AFAIK.

The recipe gets an .in file, interpolates any variables (so you can
refer to other recipes, ${buildout:directory} and friends), and writes
that to a target file. Plain and works. The output file path is
available for other recipes in buildout.cfg. I wonder, what problems
could arise with that approach.

[snip: do not force check-in of generated things]

> > If I try to conclude, then I see several decisions to make:
> > 
> > * "paster serve ..." vs. "commandname" approach.
> > 
> >   It looks like that the "commandname" approach makes more sense with
> >   Zope as we certainly want to use the power of buildout and therefore
> >   have to use generated config files in 'exotic' (seen from the casual
> >   paster user's point of view) places.
> > 
> > * "commandname-dbg" vs. "commandname --debug"
> > 
> >   This means, we have to decide, whether we want several commands
> >   performing special tasks (most probably: running everything in a 
> >   special 'mode'), or one command that accepts options to indicate
> >   that special behaviour expected.
> > 
> > * "conffile.in" vs. buildout.cfg-embedded config files
> > 
> >   I personally would really prefer the first approach for ease of use.
> >   If, however, this leads to problems with parsing the files, we are
> >   more or less bound to the latter. I'd like to investigate that
> >   further.
> > 
> > It looks like that the 'commandname-w/o-configfile-param' approach suits
> > buildout-based environments best, which might be a good reason to be
> > different in that respect from other paster-frameworks.
> > 
> > From my initial list of invokes (enriched by Hanno's Plone4 approach)
> > then only some remain:
> > 
> > - "instance-fg <options>" and friends
> > 
> > - "mycommand [start|stop|status...] <options>"
> > 
> > The latter is partly already implemented by z3c.recipe.paster. Currently
> > it does not support external config files and might do a bit more than 
> > needed (for example it calls ZConfig parser in schemaless mode, but
> > there certainly is a good reason for it; in grokprojects we let
> > zope.app.wsgi do this stuff), but it took me less than half an hour to
> > create a flatter version that accepts both: internal and external config
> > files (where external zope.confs are parsed with schemata) so that one
> > can start a paster served grokproject with generated configuration files
> > in parts/etc by doing::
> > 
> >  $ bin/myapp
> > 
> > in foreground, where the script name is built from the buildout.cfg
> > section name. It is actually a wrapper that runs::
> > 
> >   paster serve buildout-set-ini-file [cmdline-options]
> > 
> > This means also, you have to do::
> > 
> >   $ bin/myapp --daemon
> 
> There's zdaemon for this. It provides the ctl commands for any process,
> turning them into daemons or run in foreground.
> 
> > to start the server in background (which is bad if you prefer SYSV
> > compatible behaviour, i.e. ``bin/myapp start`` here), but you can do::
> > 
> >   $ bin/myapp status
> > 
> >   $ bin/myapp stop
> > 
> > which does, what one expects.
> 
> See above about zdaemon.
> 
> > One can also pass the usual paster options like ``--reload``. There is,
> > however, no possibility to start in debug mode, i.e. with a debug.ini as
> > argument, except you declare another command (at least I don't see any
> > way to do it different).
> 
> zdaemon lets you define those commands.

Hm, maybe we haven't paid enough attention to zdaemon yet (with
grokproject). In fact we already provide zdaemon wrappers with
grokproject: myproject-ctl and myproject-debug. I know that you don't
like the names, but one can easily change them (or skip).

I have, however, some problems with that approach I might tell:

wrapping a wrapper?
-------------------

It looks to me like when we invoke zdaemon to invoke paster, we do
something like starting a wrapper around a wrapper, both evaluating
cmdline arguments, providing daemon mode, etc. This looks strange to me
as it makes the below listed things possible (but might be okay anyway).

non-intuitive cmdline behaviour?
--------------------------------

With zdaemon wrapping 'paster serve ...' you can do the following
oneliners::

  $ bin/myapp start     # Does what one expects
  $ bin/myapp fg        # Does what one expects
  $ bin/myapp stop      # Does what one expects
  $ bin/myapp restart   # Does what one expects
  $ bin/myapp reload    # Error
  $ bin/myapp start --reload  # Starts in file-monitoring mode
  $ bin/myapp --help    # 'No help available'
  $ bin/myapp fg --help # Puts out paster help and exits
  $ bin/myapp start --debug # Fails without any notice on cmdline.

You can do the following cmd sequences::

  $ bin/myapp start --help # Puts paster help into logfile and exits
  $ bin/myapp stop         # 'deamon manager not running'

  $ bin/myapp start --daemon  # Starts daemon as daemon
  $ bin/myapp stop            # 'daemon manager not running'
  $ bin/myapp start --stop-daemon # Really stops the daemon-daemon.

  $ bin/myapp fg --daemon      # Starts daemonized
  $ bin/myapp stop             # 'daemon manager not running'
  $ bin/myapp fg --stop-daemon # Stops everything.

Some of the above things look really bizarre to me.

I wonder whether it wouldn't be cleaner to use paster without a
daemonizing wrapper but instead tell paster to daemonize and just set
the appropriate options in the wrapper. For this purpose a recipe like
z3c.recipe.paster might be better suited.

Best regards,

-- 
Uli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20090419/560070d5/attachment.bin