[Zope-dev] Weird BTrees/ZODB interaction under Solaris

[Guido van Rossum]

> I made the following reproducable observation running Zope 2.5 under
> Solaris:
> 
> I was indexing a CMF site using TextIndexNG with about 1600 objects.
> During the commit phase (after all objects were indexed), the Zope
> process grow from about 100MB to more than 1.2GB.
> 
> I could track this down to TextIndexNG. There is a function written
> in C that inserts the list of splittet words into the BTrees
> for the forward and reverse index (by generating the WordIDs
> internally). Replacing this method with a pure Python implementation
> made Zope behave fine.
> 
> Torturing the corresponding C implementation in an isolated environment
> did not show any memory leaks (neither under Solaris nor under Linux
> i386). To make it short: my C implementation has three args: the forward
> index, the reverse index and a list of words to be inserted. The indexes
> are an OIBTree and an IOBTree. Internally I call PyObject_SetItem()
> to store the word-wid mapping. This seems to work fine isolated but
> seems to blow up the ZODB for unknown reasons.
> 
> Note that his behaviour is only reproducable under Solaris. I made some
> tests by indexing about 100.000 docs under Linux with TextIndexNG and
> the Zope memory usage was very fine (up to 150MB).

Could you try again with the C version, substituting a different
malloc implementation, or Python 2.3 (which has effectively a
different malloc for small blocks)?  It could be a semi-bug in the
platform malloc where the allocation and deallocation pattern of your
test happens to cause worst-case fragmentation.  Using Python code
probably changes the allocation pattern enough to avoid the
fragmentation.

--Guido van Rossum (home page: http://www.python.org/~guido/)