[Zope2] ZODB load optimization ineffective due to ExtensionClass flaw
The ZODB allows to include the class (a reference to it) into persistent references in order to avoid loading the object state when a ghost object needs to be created for the persistent reference. For this to work, the class must not require arguments to "__new__". The ZODB checks whether the class defines a "__getnewargs__" method. It it does, the ZODB concludes that its "__new__" might need arguments and does not try to optimize. Unfortunately, "ExtensionClass.Base" defines "__getnewargs__". It is the base class for all Zope2 classes. Zope2 persistent references are not optimized... An analysis of our application revealed that the missed optimization opportunity increases the amount of storage loads by a factor of 4 to 5. Storage loads are often the dominating factor -- especially for searches. @Jim: Is the "__getnewargs__" definition in "ExtensionClass.Base" vital? It returns the empty tuple. Thus, there seems to be a chance to get rid of it. I know that some code (especially in "ExtensionClass/picke/pickle.c") needs to be changed as it depends on the existence of "__getnewargs__". If you think, there is a chance to get rid of "__getnewargs__", I can do the work and put the changes into the collector. Zope2 might become a bit faster... -- Dieter
Dieter Maurer wrote:
The ZODB allows to include the class (a reference to it) into persistent references in order to avoid loading the object state when a ghost object needs to be created for the persistent reference.
For this to work, the class must not require arguments to "__new__". The ZODB checks whether the class defines a "__getnewargs__" method. It it does, the ZODB concludes that its "__new__" might need arguments and does not try to optimize.
Unfortunately, "ExtensionClass.Base" defines "__getnewargs__". It is the base class for all Zope2 classes. Zope2 persistent references are not optimized...
An analysis of our application revealed that the missed optimization opportunity increases the amount of storage loads by a factor of 4 to 5. Storage loads are often the dominating factor -- especially for searches.
@Jim: Is the "__getnewargs__" definition in "ExtensionClass.Base" vital? It returns the empty tuple. Thus, there seems to be a chance to get rid of it.
I know that some code (especially in "ExtensionClass/picke/pickle.c") needs to be changed as it depends on the existence of "__getnewargs__".
If you think, there is a chance to get rid of "__getnewargs__", I can do the work and put the changes into the collector.
Zope2 might become a bit faster...
I'd have to review this, but I'm pretty sure that we dealt with this issue, but I could be mistaken. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote at 2006-3-8 10:46 -0500:
Dieter Maurer wrote:
The ZODB allows to include the class (a reference to it) into persistent references in order to avoid loading the object state when a ghost object needs to be created for the persistent reference. ... Unfortunately, "ExtensionClass.Base" defines "__getnewargs__". It is the base class for all Zope2 classes. Zope2 persistent references are not optimized... ...
[Jim]
I'd have to review this, but I'm pretty sure that we dealt with this issue, but I could be mistaken.
After I have checked that the Zope[2] SVN trunk version, too, cannot use the optimization, I filed a feature request with solution: http://www.zope.org/Collectors/Zope/2042 I adapted the "ExtensionClass" tests and verified that all tests in my Zope 2.8.1 installation continue to work despite my change. -- Dieter
Is there a chance that this will make it into the Zope-2.8.7 release? -- Markus On 3/9/06, Dieter Maurer <dieter@handshake.de> wrote:
Jim Fulton wrote at 2006-3-8 10:46 -0500:
Dieter Maurer wrote:
The ZODB allows to include the class (a reference to it) into persistent references in order to avoid loading the object state when a ghost object needs to be created for the persistent reference. ... Unfortunately, "ExtensionClass.Base" defines "__getnewargs__". It is the base class for all Zope2 classes. Zope2 persistent references are not optimized... ...
[Jim]
I'd have to review this, but I'm pretty sure that we dealt with this issue, but I could be mistaken.
After I have checked that the Zope[2] SVN trunk version, too, cannot use the optimization, I filed a feature request with solution:
http://www.zope.org/Collectors/Zope/2042
I adapted the "ExtensionClass" tests and verified that all tests in my Zope 2.8.1 installation continue to work despite my change.
-- Dieter _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
participants (3)
-
Dieter Maurer -
Jim Fulton -
M. Krainer