[Zope-dev] [Interested?] New use for Zope+CMF+Archetypes -- startup time improvements

[Dieter Maurer]

Dear Zope developers,

we have used Zope+CMF+Archetypes in a new way -- not
as a Web Application framework but as a framework for
desktop applications that share a large part of their
functionality with online applications (implemented with Zope+CMF+Archetypes).

A major stumbling block has been Zope's incredibly high startup
time. We observerd times in the order of a minute on computers
with either slow CPU or slow IO. This may be acceptable for
a Web server but is prohibitive for a desktop application -- especially
as the predecessor application started within a few seconds.

To overcome this obstacle, we tweaked Zope and fixed Python's import
mechanism such that Zope now starts either out of a ZIP archive
or as a frozen application. These measures had the following results.

  Startup times on a mid range computer (AMD Athon 1.4 GHz; 512 MB memory)
  with a standard IDE disk.

		    Cold start			    Warm start
		(after computer startup)	(most files in OS cache)

    File system	       13s			      5s
    ZIP archive	        8s			      4s
    Frozen		5s			      3s


In more details, we did:

  *  implement a package for a new kind of "url"s "pypackage:"
     for package relative access to resources.

     The package monkey patches Python's "open", "os.listdir",
     "os.stat" to provide transparent access to
     "pypackage:" identified resources.

     It currently support package relative access for
     packages loaded from the file system, from a ZIP
     archive and from the executable itself (i.e. frozen packages).
     In the last case, the resources are in a separate ZIP
     archive.

     This package might be interesting for Python as a whole
     as it is not Zope specific.

  *  implement a shared object importer to be used
     as a Python "meta_path" hook.

     This importer allows to load shared objects into the context
     of a parent package (such as e.g. "ZODB.TimeStamp") although
     the shared object is not located inside the package's source
     (ZIP archive or executable).

  *  fix about 70 occurrences in Zope code where
     package relative access was implemented by "dirname(__file__)"
     to consistenty use "package_home".

  *  modify about a dozen places in Zope+CMF to use
     "pypackage:" and cope with "__path__" not being a list
     for frozen packages

  *  fix a few products (Archetypes and friends, TextIndexNG2,
     PlacelessTranslationService, ...) to use
     "package_home" (rather than "dirname(__file__)") and
     not to change the current working directory (which obviously
     would fail for destinationsbe in a ZIP archive
     or the executable).

  *  implement lazy loading of "ImageFile"s to
     reduce the risk of recursive imports (and reduce startup time).

  *  support lazy product initialization

  *  support configuration from a pickle file (to avoid
     expensive parsing of the schema and configuration files).

     The pickle is used as a configuration cache.

  *  fixed Python's import mechanism not to treat ZIP
     archives as a directory when the archive could not
     find a module.


If you were *really* interested in these startup time improvements,
I could provide patches which might be integrated
in the Zope core for e.g. Zope 2.9.


-- 
Dieter