Re: [Zope-dev] SVN: zope.index/trunk/ - Port okascore.c optimization used in okapiiindex from Zope2 catalog
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris McDonough wrote:
Log message for revision 100775: - Port okascore.c optimization used in okapiiindex from Zope2 catalog implementation. This module is compiled conditionally, based on whether your environment has a working C compiler.
- Don't use ``len(self._docweight)`` in okapiindex _search_wids method (obtaining the length of a BTree is very expensive at scale). Instead use self.documentCount(). Also a Zope2 port.
Crap! this set of performance fixes was added *six years ago* to the Zope2 version from which zope.index was forked! Hasn't anybody been using this package? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKLvkv+gerLs4ltQ4RAqC8AKCjJefCXo3eXBiW5cH1uC+70Qz/lQCcD6t2 o0aJ+MhKZBWUT/CKhXKD9+Y= =gHGc -----END PGP SIGNATURE-----
On Tue, 2009-06-09 at 20:07 -0400, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Chris McDonough wrote:
Log message for revision 100775: - Port okascore.c optimization used in okapiiindex from Zope2 catalog implementation. This module is compiled conditionally, based on whether your environment has a working C compiler.
- Don't use ``len(self._docweight)`` in okapiindex _search_wids method (obtaining the length of a BTree is very expensive at scale). Instead use self.documentCount(). Also a Zope2 port.
Crap! this set of performance fixes was added *six years ago* to the Zope2 version from which zope.index was forked!
Hasn't anybody been using this package?
We're using this package. The text index I can only remember in a not-so-critical piece where performance currently is not an issue. Also, following checkins even today wouldn't have me remember that this code is shared. Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christian Theune wrote:
On Tue, 2009-06-09 at 20:07 -0400, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Chris McDonough wrote:
Log message for revision 100775: - Port okascore.c optimization used in okapiiindex from Zope2 catalog implementation. This module is compiled conditionally, based on whether your environment has a working C compiler.
- Don't use ``len(self._docweight)`` in okapiindex _search_wids method (obtaining the length of a BTree is very expensive at scale). Instead use self.documentCount(). Also a Zope2 port. Crap! this set of performance fixes was added *six years ago* to the Zope2 version from which zope.index was forked!
Hasn't anybody been using this package?
We're using this package. The text index I can only remember in a not-so-critical piece where performance currently is not an issue. Also, following checkins even today wouldn't have me remember that this code is shared.
The code which now lives in 'zope.index' is a copy-and-rename-everything fork of what was in the Zope2 PluggableIndexes / ZCTextIndex products as of around Christmas, 2002. The fix I was replying to is *far* from the first one which didn't get ported from subsequent changes to the Zope2 indexes, which have obviously been in *heavy* production use. I propose that we consider merging the two implementations, along the lines of the following: 1. Review the change history of the two codebases since the original fork, porting relevant changes from the Z2 indexes to the corresponding Z3 implementations. 2. Review / update the tests for the Z3 indexes, with a goal of 100% coverage, as well as improving the "quality" of the tests. 3. Re-implement the Z2 indexes as thin wrappers around the Z3 versions, ensuring that all existing Z2 tests continue to pass. This is somewhat the same pattern as Andreas used to merge the ZPT implementations, I think. This effort could be the focus of a sprint team, if there was interest. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKL9sQ+gerLs4ltQ4RAksCAKDWRMu820PO+oRQNun0VC1LrFjvOACfU6pE uVtYzoliBjQgI5gark3ALTw= =+WAL -----END PGP SIGNATURE-----
+1 Thanks. Jim On Jun 10, 2009, at 12:10 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Christian Theune wrote:
On Tue, 2009-06-09 at 20:07 -0400, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Chris McDonough wrote:
Log message for revision 100775: - Port okascore.c optimization used in okapiiindex from Zope2 catalog implementation. This module is compiled conditionally, based on whether your environment has a working C compiler.
- Don't use ``len(self._docweight)`` in okapiindex _search_wids method (obtaining the length of a BTree is very expensive at scale). Instead use self.documentCount(). Also a Zope2 port. Crap! this set of performance fixes was added *six years ago* to the Zope2 version from which zope.index was forked!
Hasn't anybody been using this package?
We're using this package. The text index I can only remember in a not-so-critical piece where performance currently is not an issue. Also, following checkins even today wouldn't have me remember that this code is shared.
The code which now lives in 'zope.index' is a copy-and-rename- everything fork of what was in the Zope2 PluggableIndexes / ZCTextIndex products as of around Christmas, 2002. The fix I was replying to is *far* from the first one which didn't get ported from subsequent changes to the Zope2 indexes, which have obviously been in *heavy* production use.
I propose that we consider merging the two implementations, along the lines of the following:
1. Review the change history of the two codebases since the original fork, porting relevant changes from the Z2 indexes to the corresponding Z3 implementations.
2. Review / update the tests for the Z3 indexes, with a goal of 100% coverage, as well as improving the "quality" of the tests.
3. Re-implement the Z2 indexes as thin wrappers around the Z3 versions, ensuring that all existing Z2 tests continue to pass.
This is somewhat the same pattern as Andreas used to merge the ZPT implementations, I think. This effort could be the focus of a sprint team, if there was interest.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFKL9sQ+gerLs4ltQ4RAksCAKDWRMu820PO+oRQNun0VC1LrFjvOACfU6pE uVtYzoliBjQgI5gark3ALTw= =+WAL -----END PGP SIGNATURE-----
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Jim Fulton Zope Corporation
2009/6/10 Tres Seaver <tseaver@palladion.com>:
This effort could be the focus of a sprint team, if there was interest.
Indexing fascinates me. I would be up for putting down some effort on some sort of improved indexing. I'm not sure how much of the time I would help and how much I would be in the way, but there you go. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On 10.06.09 18:10, Tres Seaver wrote:
This is somewhat the same pattern as Andreas used to merge the ZPT implementations, I think. This effort could be the focus of a sprint team, if there was interest.
The better example is TXNG3 where the Zope 2 and Zope 3 parts were completely refactored into eggs: http://pypi.python.org/pypi?%3Aaction=search&term=zopyx.txng&submit=search Andreas
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tres Seaver wrote:
I propose that we consider merging the two implementations, along the lines of the following:
1. Review the change history of the two codebases since the original fork, porting relevant changes from the Z2 indexes to the corresponding Z3 implementations.
2. Review / update the tests for the Z3 indexes, with a goal of 100% coverage, as well as improving the "quality" of the tests.
The zope.index package now has 100% unit test coverage under Python 2.4, 2.5, and 2.6 (measured with nose + coverage: I couldn't figure out how to get the z3c.coverage bit to ignore stuff like conditional imports).
3. Re-implement the Z2 indexes as thin wrappers around the Z3 versions, ensuring that all existing Z2 tests continue to pass.
This is somewhat the same pattern as Andreas used to merge the ZPT implementations, I think. This effort could be the focus of a sprint team, if there was interest.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKMM3M+gerLs4ltQ4RAnMAAKCpWqjs+h1gILkYzteXSFqpAYw5jACgnlXy QaVbg+eTgfvPcT3G0tFClKs= =iydI -----END PGP SIGNATURE-----
participants (5)
-
Andreas Jung -
Christian Theune -
Jim Fulton -
Lennart Regebro -
Tres Seaver