[Zope-dev] Weird BTrees/ZODB interaction under Solaris
Andreas Jung
Andreas Jung <lists@andreas-jung.com>
Tue, 26 Nov 2002 11:06:38 +0100
I made the following reproducable observation running Zope 2.5 under
Solaris:
I was indexing a CMF site using TextIndexNG with about 1600 objects.
During the commit phase (after all objects were indexed), the Zope
process grow from about 100MB to more than 1.2GB.
I could track this down to TextIndexNG. There is a function written
in C that inserts the list of splittet words into the BTrees
for the forward and reverse index (by generating the WordIDs
internally). Replacing this method with a pure Python implementation
made Zope behave fine.
Torturing the corresponding C implementation in an isolated environment
did not show any memory leaks (neither under Solaris nor under Linux
i386). To make it short: my C implementation has three args: the forward
index, the reverse index and a list of words to be inserted. The indexes
are an OIBTree and an IOBTree. Internally I call PyObject_SetItem()
to store the word-wid mapping. This seems to work fine isolated but
seems to blow up the ZODB for unknown reasons.
Note that his behaviour is only reproducable under Solaris. I made some
tests by indexing about 100.000 docs under Linux with TextIndexNG and
the Zope memory usage was very fine (up to 150MB).
Cheers,
Andreas
---------------------------------------------------------------------
- Andreas Jung http://www.andreas-jung.com -
- EMail: andreas at andreas-jung.com -
- "Life is too short to (re)write parsers" -
---------------------------------------------------------------------