[Zope-Checkins] CVS: Zope - TODO.txt:1.1.2.1

[Chris McDonough]

Update of /cvs-repository/Zope
In directory cvs.zope.org:/tmp/cvs-serv5939

Added Files:
      Tag: chrism-install-branch
	TODO.txt 
Log Message:
Add TODO file explaining what needs to be done in this branch to
merge it into the HEAD.


=== Added File Zope/TODO.txt ===
MAJOR DELTAS TO TRUNK
---------------------

 - wo_pcgi, w_pcgi, z2.py are gone.

 - lib/python/zope.py is "moral equivalent"
   of old z2.py

 - Zope now doesn't by default need to be
   run from build directory.  "make install"
   creates a distribution dir.

 - instance_home setup is now default
   by way of "./configure; make; make install;
   $prefix/bin/make_instance.py".

 - starting and stopping zope now typically
   performed via "zctl".

 - config file placed in instance_home /etc
   directory.

 - inst directory is completely changed and
   contains skeleton directories for both
   "make install" and "make instance" as
   well as other utility programs.

 - setup.py has moved into inst directory
   and has been changed to deal with branch
   new packages, package moves, and deals
   better with Windows builds.

 - "make rpmdist" makes a Zope RPM distro.

 - "make testinst" makes the build dir into
   both a software home and an instance_home.

TODO
----

Critical review of config file schema

  A config file is a contract with the user.  We want to get the Zope
  config file as right as possible the first time, as it will be very
  hard to change later.  One of the tricks to doing this is to make
  sure that as few keys and sections as possible in the config are
  "required", so that if we *do* need to change the schema for the
  config later it is likely not to impact a majority of users (as most
  of them will be sticking with the defaults).  We provide sane
  defaults for most things via default= in ZConfig schema and
  procedural code in ZConfig "handlers".  We would optimally like the
  default Zope config file to be able to be "empty" and still have
  Zope work.  This is currently not the case (software_home and
  instance_home are required keys).

Move ZServer from root of Zope "home" directory to SOFTWARE_HOME

  Currently, the ZServer package (which contains all network server
  code for Zope) resides in the "root" directory of a Zope checkout on
  the trunk.  This is suboptimal because it requires that we add this
  directory to sys.path, and there really isn't any reason that this
  package shouldn't live in lib/python (SOFTWARE_HOME) like all of the
  other Zope packages.  Unfortunately, the ZServer package is
  symlinked into the Zope home directory via "repolinks".  Changing
  the link to point to lib/python/ZServer would cause older branches
  to fail.  I'm not sure how to handle this.  Maybe punt and keep it
  there, but it's just so ugly.

  I have created a "new" ZServer package on the branch that actually
  does live in lib/python (not via a symlink).  This package needs to
  be blown away in the repository if we decide to not move ZServer.
  Note also that the servers in the "lib/python/ZServer" package have
  been changed on the branch to support the new "DebugLogger" and
  "AccessLogger" objects (currently in zLOG), so these patches may
  need to be reapplied.

"zctl.py" program needs to be renamed to "zopectl.py"

  The "zctl" program that is put into an instance home's "bin"
  directory should be renamed to "zopectl", as I imagine we'll want to
  have a "zeoctl" as well at some point.  The ".py" extension is
  necessary if we want people to be able to run the script without
  specifying the interpreter path on Windows.

"zctl" should have a "logtail" option

  It should be possible to tail the event, trace, and access logs
  using zctl (in a cross-platform way).

Zope.Startup.ZCtlLib needs refactoring

  Zope.Startup.ZctlLib is ugly and could use some refactoring, and
  possibly a whole new approach to "ctl" should be considered.  Maybe
  instead of spawning a separate process, use a strategy like Guido's
  runsrvr.py, which actually constitutes a mainloop.

Test on Windows

  The branch needs testing under Windows (including source build
  and install).  It worked at one point (a few months ago) but hasn't
  been tested since.

"configure" shell scripts need peer review

  I'd like someone to review the "configure" and "configure.bat"
  scripts in the top-level directory and suggest/make changes.  These
  scripts help find a suitable Python interpreter and create a
  Makefile on UNIX and Windows, respectively.

"utilities" top-level directory should be renamed "bin"

  The top-level directory named utilities (which contains zpasswd and some
  other utility programs) should be renamed "bin".

Move pcgi, doc, Extensions, import, utilities, and bin directories to skel dir

  The Makefile "hacklinks" rule links these top-level directories into
  a "skeleton" directory inside "inst".  These directories should be
  moved into inst instead.  They are written wholesale to the target
  directory during the "make install" process.

"make links" step should be reconsidered

  The makefile includes a "links" rule which copies files
  necessary for both "make install" and "make instance" into
  various places in the tree.  This might be made better.

"inst/do.py" must be exorcised.

  do.py is useless but other things depend on it.

Deal with command-line-specified overrides to configuration values

  It is currently not possible to override config file-specifed values
  by using arguments to zctl or zope.py.  This will be required in
  reality.  There is a bunch of code that anticipates this possibility
  in zope.py and Zope.Startup.ZctlLib.py (anything that mentions
  "overrides" or "getSchemaKeys" is likely aiming to deal with this),
  but none of it actually works.

Normalize zLOG/PEP 282 logging severities

  Currently there is an unintuitive mapping between zLOG and PEP 282
  (logging package) severities.  The config file allows you to specify
  log severities using the PEP282 semantics, but you need to dig into
  the source code to find out what the mappings are between zLOG
  severities and PEP 282 severities (inside zLOG.EventLogger).  Find
  some way to make this make sense either via documentation or by
  extending the PEP282 logger with zLOG severities and changing the
  schema to allow the specification of zLOG severities instead of
  PEP282 severities.

Find a place for zLOG.DebugLogger and zLOG.AccessLogger

  These packages are placed here because I've got no better place for
  them.  They really don't belong here, as they are the loggers for
  "detailed" request logging and access logging respectively (not
  event logging, which is really zLOG's only domain).

Zope.Startup.nt.NTService.py and instance_home/bin/ntservice.py need
testing

  These modules allow you to associate an NT service with a Zope
  instance_home.  Success means being able to invoke the ntservice.py
  module to create an NT service which can successfully start and stop
  a Zope instance on a per-instance-home basis.

nteventlogger needs testing

  The logger "nteventlogger" (which is creatable via the config file)
  needs to be tested.

Deal with lack of PCGI

  In the old setup style, "w_pcgi.py" was invoked to install Zope with
  a PCGI ("persistent CGI") socket and files necessary for using Zope
  in conjunction with Apache/IIS.  There are no knobs for installation
  right now to do the same thing.  The "make_resource.py" file in inst
  is the only remnant of PCGI support left in Zope.

Update Zope distribution creation machinery

  The current distribution-creation machinery will almost certainly
  fail if it's not updated with knowledge of the new Zope installation
  regime.

Get rid of ZOPE_HOME

  An environment variable named ZOPE_HOME was added in 2.6.  This
  envvar is not necessary if ZServer is moved into lib/python.  It's
  usage is cruft and should be exorcised.

Fix "make install"

  Make install currently fails.

Write some unit tests

  Write unit tests for Zope.Startup packages.

ZConfig needs to deal with datatype conversion failures more cleanly

  See http://my.zope.com/CPM/CPM/issues/5.

Make sure ZConfig datatype conversion and handler errors are caught
the right way by zope.py

  zope.py is the responsible for causing the zconfig configuration
  file to be parsed, and needs to deal with errors in a sane way.
  This is tricky because it now houses the top-level exception handler
  which needs to do sane things to distinguish between configuration
  errors, true system exit, and unknown errors.

RPM install should recompile .pycs

  The RPM spec file (inst/Zope.spec.in) should delete and recompile
  .pycs/.pyos during an RPM install to make sure tracebacks contain
  real Zope filenames instead of filenames relative to the build
  directory.  Alternately the RPM install should not contain .pycs or
  .pyos.

RPM install versioning scheme needs to be documented

  The RPM builder has a versioning scheme via the generated Makefile
  in order to allow for the install of multiple simultaneous versions
  of Zope on a single system.  This versioning scheme should be
  documented.

Zope.Startup.misc.TextBlockFormatter (used in Startup.ZctlLib) should
be retired

  ... in favor of textwrap.

Config file needs better inline docs

  The Zope zconfig config file has some inline docs.  They need to be
  completed.  To centralize this data, the docs might be done in the
  schema itself.  A schema introspector which generates the default
  config file could be created.

Sanity check help and usage output of command-line programs

  Make sure zope.py, zctl.py, and ZctlLib.py have sane help output
  (make sure it matches the capabilities of the system).

ZConfig defaults

  We deferred several issues that we recognized as areas for
  improvement in ZConfig that might make it possible to avoid writing
  nasty procedural code for default handling unnecessary
  (e.g. Zope.Startup.handlers._RootHandler).  See
  http://my.zope.com/CPM/CPM/issues/4 and
  http://my.zope.com/CPM/CPM/issues/3.  Not necessary for merge, but
  useful to think about for future.

DBTab Integration

  The Zope config file currently allows you to specify multiple
  databases, each of which has a mount point.  Currently, however, if
  you specify a mount point for a database other than '/', an error
  will be raised at startup.  It would be nice to integrate the
  necessary machinery from DBTab into Zope to make it possible to
  mount databases by just monkeying with the config file.  This is not
  strictly necessary, but the config file format should be validated
  to make sure it can be later used for this purpose.

What to do about custom_zodb?

  The database sections in the config file allow us to (theoretically)
  get rid of custom_zodb.  Shane has requested that we leave the
  custom_zodb hook in Zope (for DBTab to use), but if we integrate
  DBTab into Zope, I think we can get rid of custom_zodb entirely.  If
  we don't get rid of custom_zodb, what precedence should
  custom_zodb-defined storages and databases take against those
  specified in a config file?  Currently a custom_zodb
  storage/database will take precedence over the config file-specified
  root database.  Work this out in Zope.App.startup.

Test 'effective user' support

  Great pains have been taken in configuration integration to defer
  database and logger construction until after Zope has had a chance
  to change effective users.  Make sure that no files are written to
  disk (maybe via unit tests) before Zope has had a chance to switch
  effective users.

Logging during startup

  The branch takes the strategy of logging to stdout and buffering log
  messages until "real" loggers can safely be configured.  After
  "real" loggers are configured, the log messages buffered at startup
  are flushed to the "real" loggers.  Ensure this meets everybody's
  satisfaction via zope-coders (or don't, it's a rathole).

Server construction errors need to be better

  When a server is constructed, it attempts to bind to a port.  If the
  port cannot be bound, an error is raised.  The error currently doesn't
  include the port number, and should.

Switch "integer" datatypes to Barry's "suffix-multiplier" datatypes as
necessary

  Some keys in the Zope config schema are "integers" when they really
  should be backed by Barry's "suffix-multiplier" datatype
  (e.g. cache_size for a database).

"constructor" conversion in ZConfig.datatypes is ugly

  For "escape hatches" in some sections of the configuration
  (e.g. "custom_logger"), I've made it possible to use a "constructor"
  datatype.  The constructor datatype parses a string that looks like
  a Python object constructor (e.g. "constructor
  ZODB.FileStorage.FileStorage('Data.fs')").  The actual constructor
  datatype converter in ZConfig.datatypes module is ugly and
  error-prone and there has to be a better way to do it.  Alternate
  suggestions welcome.

os.chdir where?

  When Zope starts, it's been asserted in various maillist threads
  that it should chdir to '/'.  This is not done.

Data.fs.in overwritten by zero-byte file at some point during install

  I'm not sure why, but when the branch is set up during "make testinst"
  (an easy way to set up a branch checkout as both a software_home and
  instance_home), the build dir's var/Data.fs.in is overwritten with
  a zero-byte file.  Fix.

Performing the merge

  I am not at all confident that doing a cvs up -j
  chrism-install-branch on the HEAD will not screw things up royally
  as the process I've used to keep the HEAD and the branch in sync has
  been... painful.  Instead, we should just manually make the changes
  to the HEAD by incoprorating the changes to the /inst directory, the
  /doc directory, the Zope.Startup package, the Zope.App package and
  other necessary files that exist only on the branch in a commit of a
  HEAD checkout.

Notifying owners of automated testers

  Chris Withers and others who do automated tests of the HEAD will
  need to either discontinue the HEAD tests or they will need to make
  changes to their testers to build Zope properly using the new
  method.  This needs to be explained on zope-coders.

Make sure Zope unit tests pass on branch after all integration work

  Duh.