I'm forwarding a message from limi here, since it makes much more sense here than it does on plone-developers. To summarize it: while analying performance of the Plone 3 codebase they noticed a lot of time was spent trying to figure out which types could be added at a location. Part of that logic uses the TypesTool _queryFactoryMethod function, which goes through the dispatcher to get to the factory method for a type, which is apparently very slow in Zope 2.10, and much faster in Zope 2.9. They came up with http://paste.plone.org/13211 which is an imho somewhat evil approach which introduces a thread-local cache for App.FactoryDispatcher._product_packages. As described below the speedup as a result of that is huge. Since Zope startup is as far as I know the only time the result of _product_packages is change perhaps it would be useful to generate that result on speedup. Or perhaps to make _product_packages cache its output. I certainly do want any monkey patches in Plone for this since it should be fixed in either Zope or CMF. Wichert. ----- Forwarded message from Alexander Limi <limi@plone.org> ----- From: Alexander Limi <limi@plone.org> To: plone-developers@lists.sourceforge.net Subject: [Plone-developers] Plone 3 - now 3x faster for logged-in users Date: Sat, 03 Mar 2007 19:37:54 -0800 Message-ID: <op.tona1gssvhba00@-emo> Hello from the optimization sprint :) Just thought I'd keep you updated on some of the stuff we've been up to here. The main news for the day is that we found a combination of slowness in Zope 2.10 and CMF using that method in a less-than-optimal way. The result was that rendering the content menu (the one that decides what types you can add where etc) was eating an amazing amount of time. Explained in my usual UI designer hand-wavy way, every time we load a page in logged in mode, it checks the list of Products to figure out what you can add. So, by caching this locally in TypesTool so it doesn't get looked up every time (the list of Products doesn't change with every request, after all), we get a significant speedup. Let's see some numbers of loading the front page of a fresh plone : Before types tool optimization (10 requests): Products/CMFPlone/skins/plone_content/document_view.pt 8.54s lib/python/plone/app/contentmenu/contentmenu.pt 6.26s After types tool optimization: Products/CMFPlone/skins/plone_content/document_view.pt 2.55s lib/python/plone/app/contentmenu/contentmenu.pt 0.29s That makes every logged in page over 3 times faster. I like it. :) Patch for CMFCore TypesTool included. -- Alexander Limi · http://limi.net _______________________________________________ Plone-developers mailing list Plone-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/plone-developers ----- End forwarded message ----- -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
--On 4. März 2007 13:57:23 +0100 Wichert Akkerman <wichert@wiggy.net> wrote:
I'm forwarding a message from limi here, since it makes much more sense here than it does on plone-developers. To summarize it: while analying performance of the Plone 3 codebase they noticed a lot of time was spent trying to figure out which types could be added at a location. Part of that logic uses the TypesTool _queryFactoryMethod function, which goes through the dispatcher to get to the factory method for a type, which is apparently very slow in Zope 2.10, and much faster in Zope 2.9.
Any idea why the code is different between 2.9 and 2.10?
They came up with http://paste.plone.org/13211 which is an imho somewhat evil approach which introduces a thread-local cache for App.FactoryDispatcher._product_packages. As described below the speedup as a result of that is huge.
I would not be opposed against such a speedup patch as long as it does not raise other problems. However I don't know App.* enough to be evaluate the patch. Andreas
Previously Andreas Jung wrote:
--On 4. März 2007 13:57:23 +0100 Wichert Akkerman <wichert@wiggy.net> wrote:
I'm forwarding a message from limi here, since it makes much more sense here than it does on plone-developers. To summarize it: while analying performance of the Plone 3 codebase they noticed a lot of time was spent trying to figure out which types could be added at a location. Part of that logic uses the TypesTool _queryFactoryMethod function, which goes through the dispatcher to get to the factory method for a type, which is apparently very slow in Zope 2.10, and much faster in Zope 2.9.
Any idea why the code is different between 2.9 and 2.10?
Rocky modified it in changeset 67869 to be able to support external methods outside of products. The problem with the new code is that it imports all products to figure out some of their details. So for every factory dispatcher access we now have an import for each product, which means lots of filesystem accesses and python interpreter work being done.
They came up with http://paste.plone.org/13211 which is an imho somewhat evil approach which introduces a thread-local cache for App.FactoryDispatcher._product_packages. As described below the speedup as a result of that is huge.
I would not be opposed against such a speedup patch as long as it does not raise other problems. However I don't know App.* enough to be evaluate the patch.
http://paste.plone.org/13217 should do the trick. It makes _product_packages cache its result using the list of products in Control_Panel as a cache key. That makes sure that removing or adding products there will not result in stale data being returned by the dispatcher. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Previously Wichert Akkerman wrote:
http://paste.plone.org/13217 should do the trick. It makes _product_packages cache its result using the list of products in Control_Panel as a cache key. That makes sure that removing or adding products there will not result in stale data being returned by the dispatcher.
Obvious and stupid mistake in that patch, http://paste.plone.org/13218 should be better. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Mutable default values are evil. I would get rid of that. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
Previously Sidnei da Silva wrote:
Mutable default values are evil. I would get rid of that.
I modeled that on how get_module_info worked, so at least the pattern is not new in the Zope2 codebase. It could just as easily be done in a global value as well. I'm more interested in seeing if people agree with the basic approach though. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Looking at the changes rocky made, I don't see why allowing external methods from non-Products should require making the whole process completely dynamic on every call. The old mechanism of storing the product list should be put back in place, along with the addition of non-Product information to these results. Alec
On Mar 4, 1:49 pm, "Alec Mitchell" <a...@columbia.edu> wrote:
Looking at the changes rocky made, I don't see why allowing external methods from non-Products should require making the whole process completely dynamic on every call. The old mechanism of storing the product list should be put back in place, along with the addition of non-Product information to these results.
Following this message are two patches I'm ready to apply. One for Five and one for Zope2. The lookup is still dynamic, but at least it doesn't open a zodb connection everytime anymore. It more closely resembles what was trying to be done originally, and that is to look up factory info from the Products.* package. I'm prepared to commit this but posting it here for review first. Plus ... what is the policy for updating the Five svn:external for Zope ? (ie right now it points at Five 1.5.2 tag) Regards, Rocky Five Patch: Index: fiveconfigure.py =================================================================== --- fiveconfigure.py (revision 72973) +++ fiveconfigure.py (working copy) @@ -218,6 +218,11 @@ if init_func is not None: newContext = ProductContext(product, app, module_) init_func(newContext) + + registered_packages = getattr(Products, '_registered_packages', None) + if registered_packages is None: + registered_packages = Products._registered_packages = [] + registered_packages.append(module_) finally: try: import transaction Index: tests/test_registerpackage.py =================================================================== --- tests/test_registerpackage.py (revision 72973) +++ tests/test_registerpackage.py (working copy) @@ -65,7 +65,11 @@ >>> 'pythonproduct2' in product_listing True + Make sure it also shows up in ``Products._registered_packages``. + >>> [x.__name__ for x in getattr(Products, '_registered_packages', [])] + ['pythonproduct2'] + Clean up: >>> tearDown() Index: CHANGES.txt =================================================================== --- CHANGES.txt (revision 72973) +++ CHANGES.txt (working copy) @@ -2,6 +2,14 @@ Five Changes ============ +Five 1.5.x (svn/unreleased) +=========================== + +* Added change to registerPackage directive so that it stores the newly + registered packages on the Products package object for faster reference. + This means code that looks this up (ie Zope2's FactoryDispatcher) no longer + has to open a zodb connection each time. + Five 1.5.2 (2007-01-10) ======================= Zope2 Patch: Index: lib/python/App/FactoryDispatcher.py =================================================================== --- lib/python/App/FactoryDispatcher.py (revision 72972) +++ lib/python/App/FactoryDispatcher.py (working copy) @@ -26,32 +26,15 @@ zope2 packages and those without the Products namespace package. """ - old_product_packages = {} + packages = {} for x in dir(Products): m = getattr(Products, x) if isinstance(m, types.ModuleType): - old_product_packages[x] = m + packages[x] = m + + for m in getattr(Products, '_registered_packages', []): + packages[m.__name__] = m - packages = {} - app = Zope2.app() - try: - products = app.Control_Panel.Products - - for product_id in products.objectIds(): - product = products[product_id] - if hasattr(product, 'package_name'): - pos = product.package_name.rfind('.') - if pos > -1: - packages[product_id] = __import__(product.package_name, - globals(), {}, - product.package_name[pos+1:]) - else: - packages[product_id] = __import__(product.package_name) - elif old_product_packages.has_key(product_id): - packages[product_id] = old_product_packages[product_id] - finally: - app._p_jar.close() - return packages class ProductDispatcher(Acquisition.Implicit):
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alec Mitchell wrote:
Looking at the changes rocky made, I don't see why allowing external methods from non-Products should require making the whole process completely dynamic on every call. The old mechanism of storing the product list should be put back in place, along with the addition of non-Product information to these results.
Not to mention that the *idea* of having ExternalMethods loaded from non-products is completely silly: it the package is written to function as a Zope2 product (and I defy anybody to offer an example of a non-Zope2-specific package with an 'Extensions' directory!), then make it a product, dangit. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF7AiH+gerLs4ltQ4RAqyBAJ4yMkUo+japLU1tTrWvDcq7skr+cgCg2Vci 3YMGrdESMpqCe8EXwxKIsFA= =DEeK -----END PGP SIGNATURE-----
participants (6)
-
Alec Mitchell -
Andreas Jung -
Rocky -
Sidnei da Silva -
Tres Seaver -
Wichert Akkerman