[Interested?] New use for Zope+CMF+Archetypes -- startup time improvements
Dear Zope developers, we have used Zope+CMF+Archetypes in a new way -- not as a Web Application framework but as a framework for desktop applications that share a large part of their functionality with online applications (implemented with Zope+CMF+Archetypes). A major stumbling block has been Zope's incredibly high startup time. We observerd times in the order of a minute on computers with either slow CPU or slow IO. This may be acceptable for a Web server but is prohibitive for a desktop application -- especially as the predecessor application started within a few seconds. To overcome this obstacle, we tweaked Zope and fixed Python's import mechanism such that Zope now starts either out of a ZIP archive or as a frozen application. These measures had the following results. Startup times on a mid range computer (AMD Athon 1.4 GHz; 512 MB memory) with a standard IDE disk. Cold start Warm start (after computer startup) (most files in OS cache) File system 13s 5s ZIP archive 8s 4s Frozen 5s 3s In more details, we did: * implement a package for a new kind of "url"s "pypackage:" for package relative access to resources. The package monkey patches Python's "open", "os.listdir", "os.stat" to provide transparent access to "pypackage:" identified resources. It currently support package relative access for packages loaded from the file system, from a ZIP archive and from the executable itself (i.e. frozen packages). In the last case, the resources are in a separate ZIP archive. This package might be interesting for Python as a whole as it is not Zope specific. * implement a shared object importer to be used as a Python "meta_path" hook. This importer allows to load shared objects into the context of a parent package (such as e.g. "ZODB.TimeStamp") although the shared object is not located inside the package's source (ZIP archive or executable). * fix about 70 occurrences in Zope code where package relative access was implemented by "dirname(__file__)" to consistenty use "package_home". * modify about a dozen places in Zope+CMF to use "pypackage:" and cope with "__path__" not being a list for frozen packages * fix a few products (Archetypes and friends, TextIndexNG2, PlacelessTranslationService, ...) to use "package_home" (rather than "dirname(__file__)") and not to change the current working directory (which obviously would fail for destinationsbe in a ZIP archive or the executable). * implement lazy loading of "ImageFile"s to reduce the risk of recursive imports (and reduce startup time). * support lazy product initialization * support configuration from a pickle file (to avoid expensive parsing of the schema and configuration files). The pickle is used as a configuration cache. * fixed Python's import mechanism not to treat ZIP archives as a directory when the archive could not find a module. If you were *really* interested in these startup time improvements, I could provide patches which might be integrated in the Zope core for e.g. Zope 2.9. -- Dieter
Go Dieter!
On May 14, 2005, at 02:10, Simon Michael wrote:
Go Dieter!
Hey Simon, Dieter isn't looking for a fan club, he is looking for champions in the right places to integrate his work so it benefits a wider audience! ;) jens
This is really interesting! A year ago I put Python + Zope on a USB key, just for fun. It would barely fit so I looked at zip techniques, and never could figure out how to get ZPTs etc. to load from a ZIP. How did you do that part? --Paul Dieter Maurer wrote:
Dear Zope developers,
we have used Zope+CMF+Archetypes in a new way -- not as a Web Application framework but as a framework for desktop applications that share a large part of their functionality with online applications (implemented with Zope+CMF+Archetypes).
A major stumbling block has been Zope's incredibly high startup time. We observerd times in the order of a minute on computers with either slow CPU or slow IO. This may be acceptable for a Web server but is prohibitive for a desktop application -- especially as the predecessor application started within a few seconds.
To overcome this obstacle, we tweaked Zope and fixed Python's import mechanism such that Zope now starts either out of a ZIP archive or as a frozen application. These measures had the following results.
Startup times on a mid range computer (AMD Athon 1.4 GHz; 512 MB memory) with a standard IDE disk.
Cold start Warm start (after computer startup) (most files in OS cache)
File system 13s 5s ZIP archive 8s 4s Frozen 5s 3s
In more details, we did:
* implement a package for a new kind of "url"s "pypackage:" for package relative access to resources.
The package monkey patches Python's "open", "os.listdir", "os.stat" to provide transparent access to "pypackage:" identified resources.
It currently support package relative access for packages loaded from the file system, from a ZIP archive and from the executable itself (i.e. frozen packages). In the last case, the resources are in a separate ZIP archive.
This package might be interesting for Python as a whole as it is not Zope specific.
* implement a shared object importer to be used as a Python "meta_path" hook.
This importer allows to load shared objects into the context of a parent package (such as e.g. "ZODB.TimeStamp") although the shared object is not located inside the package's source (ZIP archive or executable).
* fix about 70 occurrences in Zope code where package relative access was implemented by "dirname(__file__)" to consistenty use "package_home".
* modify about a dozen places in Zope+CMF to use "pypackage:" and cope with "__path__" not being a list for frozen packages
* fix a few products (Archetypes and friends, TextIndexNG2, PlacelessTranslationService, ...) to use "package_home" (rather than "dirname(__file__)") and not to change the current working directory (which obviously would fail for destinationsbe in a ZIP archive or the executable).
* implement lazy loading of "ImageFile"s to reduce the risk of recursive imports (and reduce startup time).
* support lazy product initialization
* support configuration from a pickle file (to avoid expensive parsing of the schema and configuration files).
The pickle is used as a configuration cache.
* fixed Python's import mechanism not to treat ZIP archives as a directory when the archive could not find a module.
If you were *really* interested in these startup time improvements, I could provide patches which might be integrated in the Zope core for e.g. Zope 2.9.
--On Freitag, 13. Mai 2005 19:26 Uhr +0200 Dieter Maurer <dieter@handshake.de> wrote:
If you were *really* interested in these startup time improvements, I could provide patches which might be integrated in the Zope core for e.g. Zope 2.9.
Although I already worked with pypackage my questions: is it an optional feature or will pypackage take control over all and everything? Products using __path__ or changing the current directory need to be fixed, right? I think 2.9 would be a good target. If there is common agreement that it should go into the core (+1 from my side) then please wait with submitting patches until we have cut a release branch for Zope 2.8...then the trunk can be used for new features for Zope 2.9. Andreas
Andreas Jung wrote at 2005-5-14 11:00 +0200:
... Although I already worked with pypackage my questions: is it an optional feature or will pypackage take control over all and everything?
Provided that any access to a package relative resource is converted from "dirname(__file__)" to "package_home" logic to use or not use "pypackage" could be localized in "package_home" and (maybe) "OFS.Application.get_products".
Products using __path__ or changing the current directory need to be fixed, right?
"__path__" is a string in a frozen application. It is likely that any software part that uses "__path__" currently asumes it to be a list. Any product that tries the change the current directory to lie in a ZIP archive or execatable will observe that this is impossible. -- Dieter
Dieter Maurer wrote at 2005-5-14 20:37 +0200:
Andreas Jung wrote at 2005-5-14 11:00 +0200:
... Although I already worked with pypackage my questions: is it an optional feature or will pypackage take control over all and everything?
Provided that any access to a package relative resource is converted from "dirname(__file__)" to "package_home" logic to use or not use "pypackage" could be localized in "package_home" and (maybe) "OFS.Application.get_products".
After a new thought, I doubt this is worth the effort. When "pypackage" is part of Zope why should Zope not depend on it? Of course, this should not apply to third party products. And indeed, as long as they do not want to run from a zip archive or as part of a frozen application, they can ignore "pypackage". When they use "package_home" (rather than "dirname(__file__)" to access package local resources, they are independent of any "pypackage" consideration. -- Dieter
Dieter, this sounds very interesting. The idea of creating such application that are "shared" between desktop and web has always appealed me and I was wondering how to implement it. So I would be very interested - that your patches make their way to the Zope core (I am not a Zope core contributor tough) - to learn more about what you have done. Robert Dieter Maurer wrote:
Dear Zope developers,
we have used Zope+CMF+Archetypes in a new way -- not as a Web Application framework but as a framework for desktop applications that share a large part of their functionality with online applications (implemented with Zope+CMF+Archetypes).
A major stumbling block has been Zope's incredibly high startup time. We observerd times in the order of a minute on computers with either slow CPU or slow IO. This may be acceptable for a Web server but is prohibitive for a desktop application -- especially as the predecessor application started within a few seconds.
To overcome this obstacle, we tweaked Zope and fixed Python's import mechanism such that Zope now starts either out of a ZIP archive or as a frozen application. These measures had the following results.
Startup times on a mid range computer (AMD Athon 1.4 GHz; 512 MB memory) with a standard IDE disk.
Cold start Warm start (after computer startup) (most files in OS cache)
File system 13s 5s ZIP archive 8s 4s Frozen 5s 3s
In more details, we did:
* implement a package for a new kind of "url"s "pypackage:" for package relative access to resources.
The package monkey patches Python's "open", "os.listdir", "os.stat" to provide transparent access to "pypackage:" identified resources.
It currently support package relative access for packages loaded from the file system, from a ZIP archive and from the executable itself (i.e. frozen packages). In the last case, the resources are in a separate ZIP archive.
This package might be interesting for Python as a whole as it is not Zope specific.
* implement a shared object importer to be used as a Python "meta_path" hook.
This importer allows to load shared objects into the context of a parent package (such as e.g. "ZODB.TimeStamp") although the shared object is not located inside the package's source (ZIP archive or executable).
* fix about 70 occurrences in Zope code where package relative access was implemented by "dirname(__file__)" to consistenty use "package_home".
* modify about a dozen places in Zope+CMF to use "pypackage:" and cope with "__path__" not being a list for frozen packages
* fix a few products (Archetypes and friends, TextIndexNG2, PlacelessTranslationService, ...) to use "package_home" (rather than "dirname(__file__)") and not to change the current working directory (which obviously would fail for destinationsbe in a ZIP archive or the executable).
* implement lazy loading of "ImageFile"s to reduce the risk of recursive imports (and reduce startup time).
* support lazy product initialization
* support configuration from a pickle file (to avoid expensive parsing of the schema and configuration files).
The pickle is used as a configuration cache.
* fixed Python's import mechanism not to treat ZIP archives as a directory when the archive could not find a module.
If you were *really* interested in these startup time improvements, I could provide patches which might be integrated in the Zope core for e.g. Zope 2.9.
Dieter Maurer wrote at 2005-5-13 19:26 +0200:
... high Zope startup time ... To overcome this obstacle, we tweaked Zope and fixed Python's import mechanism such that Zope now starts either out of a ZIP archive or as a frozen application. ... In more details, we did:
* implement a package for a new kind of "url"s "pypackage:" for package relative access to resources.
The package monkey patches Python's "open", "os.listdir", "os.stat" to provide transparent access to "pypackage:" identified resources.
It currently support package relative access for packages loaded from the file system, from a ZIP archive and from the executable itself (i.e. frozen packages). In the last case, the resources are in a separate ZIP archive.
This package might be interesting for Python as a whole as it is not Zope specific.
I have now released this part as independent Python packages "dllimporter" and "packageresources". You can find them via <http://www.dieter.handshake.de/pyprojects> In case, they should be included into Zope, I would give them a dual license (my own BSD-like and ZPL). -- Dieter
participants (6)
-
Andreas Jung -
Dieter Maurer -
Jens Vagelpohl -
Paul Everitt -
robert rottermann -
Simon Michael