[Zope-dev] Re: [Interested?] New use for Zope+CMF+Archetypes -- startup time improvements

[Paul Everitt]

This is really interesting!  A year ago I put Python + Zope on a USB 
key, just for fun.  It would barely fit so I looked at zip techniques, 
and never could figure out how to get ZPTs etc. to load from a ZIP.

How did you do that part?

--Paul

Dieter Maurer wrote:
> Dear Zope developers,
> 
> we have used Zope+CMF+Archetypes in a new way -- not
> as a Web Application framework but as a framework for
> desktop applications that share a large part of their
> functionality with online applications (implemented with Zope+CMF+Archetypes).
> 
> A major stumbling block has been Zope's incredibly high startup
> time. We observerd times in the order of a minute on computers
> with either slow CPU or slow IO. This may be acceptable for
> a Web server but is prohibitive for a desktop application -- especially
> as the predecessor application started within a few seconds.
> 
> To overcome this obstacle, we tweaked Zope and fixed Python's import
> mechanism such that Zope now starts either out of a ZIP archive
> or as a frozen application. These measures had the following results.
> 
>   Startup times on a mid range computer (AMD Athon 1.4 GHz; 512 MB memory)
>   with a standard IDE disk.
> 
> 		    Cold start			    Warm start
> 		(after computer startup)	(most files in OS cache)
> 
>     File system	       13s			      5s
>     ZIP archive	        8s			      4s
>     Frozen		5s			      3s
> 
> 
> In more details, we did:
> 
>   *  implement a package for a new kind of "url"s "pypackage:"
>      for package relative access to resources.
> 
>      The package monkey patches Python's "open", "os.listdir",
>      "os.stat" to provide transparent access to
>      "pypackage:" identified resources.
> 
>      It currently support package relative access for
>      packages loaded from the file system, from a ZIP
>      archive and from the executable itself (i.e. frozen packages).
>      In the last case, the resources are in a separate ZIP
>      archive.
> 
>      This package might be interesting for Python as a whole
>      as it is not Zope specific.
> 
>   *  implement a shared object importer to be used
>      as a Python "meta_path" hook.
> 
>      This importer allows to load shared objects into the context
>      of a parent package (such as e.g. "ZODB.TimeStamp") although
>      the shared object is not located inside the package's source
>      (ZIP archive or executable).
> 
>   *  fix about 70 occurrences in Zope code where
>      package relative access was implemented by "dirname(__file__)"
>      to consistenty use "package_home".
> 
>   *  modify about a dozen places in Zope+CMF to use
>      "pypackage:" and cope with "__path__" not being a list
>      for frozen packages
> 
>   *  fix a few products (Archetypes and friends, TextIndexNG2,
>      PlacelessTranslationService, ...) to use
>      "package_home" (rather than "dirname(__file__)") and
>      not to change the current working directory (which obviously
>      would fail for destinationsbe in a ZIP archive
>      or the executable).
> 
>   *  implement lazy loading of "ImageFile"s to
>      reduce the risk of recursive imports (and reduce startup time).
> 
>   *  support lazy product initialization
> 
>   *  support configuration from a pickle file (to avoid
>      expensive parsing of the schema and configuration files).
> 
>      The pickle is used as a configuration cache.
> 
>   *  fixed Python's import mechanism not to treat ZIP
>      archives as a directory when the archive could not
>      find a module.
> 
> 
> If you were *really* interested in these startup time improvements,
> I could provide patches which might be integrated
> in the Zope core for e.g. Zope 2.9.
> 
>