I made the following reproducable observation running Zope 2.5 under Solaris:

I was indexing a CMF site using TextIndexNG with about 1600 objects. During the commit phase (after all objects were indexed), the Zope process grow from about 100MB to more than 1.2GB.

I could track this down to TextIndexNG. There is a function written in C that inserts the list of splittet words into the BTrees for the forward and reverse index (by generating the WordIDs internally). Replacing this method with a pure Python implementation made Zope behave fine.

Torturing the corresponding C implementation in an isolated environment did not show any memory leaks (neither under Solaris nor under Linux i386). To make it short: my C implementation has three args: the forward index, the reverse index and a list of words to be inserted. The indexes are an OIBTree and an IOBTree. Internally I call PyObject_SetItem() to store the word-wid mapping. This seems to work fine isolated but seems to blow up the ZODB for unknown reasons.

Note that his behaviour is only reproducable under Solaris. I made some tests by indexing about 100.000 docs under Linux with TextIndexNG and the Zope memory usage was very fine (up to 150MB).

Could you try again with the C version, substituting a different malloc implementation, or Python 2.3 (which has effectively a different malloc for small blocks)? It could be a semi-bug in the platform malloc where the allocation and deallocation pattern of your test happens to cause worst-case fragmentation. Using Python code probably changes the allocation pattern enough to avoid the fragmentation. --Guido van Rossum (home page: http://www.python.org/~guido/)

Andreas Jung writes:

... Internally I call PyObject_SetItem() to store the word-wid mapping. "PyObject_SetItem" creates new references for key and value. Is this what you expect?

I know, this should affect all platforms and be independent of whether or not the code runs within or outside Zope. Dieter