[ZODB-Dev] ZEO client leaking memory?

[Chris Withers]

Toby Dickenson wrote:
> 
> >Hurm, where can I learn how to do this tuning?
> 
> Read the source, and meditate.

Oh b*?locks, I thought you might say that :-S

> Ahhhh! Yes, BTrees do have a disk space overhead due to their
> optimisation for incremental, not bulk indexing.

By disk-space overhead, I guess you mean lots of object changes when you index
something. I wonder if that could also be chewing through RAM? I don't really
care about disk space usage given that we've had the data.fs in question up to
about 10GB, but I really want to find out how I'm bleeding so much RAM so
quickly...

> Ive got no reason to think that this wouldnt do what was intended.....
> it should control memory usage (although, as I say, in a fairly heavy
> handed way)

Well, I call it after I've indexed 600 docs, right after I commit the
transaction. Surely this call should make the meory usage drop back down to 10MB
or so, at least? Well, it doesn't, at all, so where is that RAM going, I wonder
:-S

> * a size of 400 objects.
> 
> That means incremental garbage collection only seriously kicks in when
> there are at least 400 objects in the cache. In a well tuned system
> under moderate memory pressure (in my experience) I would expect each
> cache to reach an equilibrium size of up to 2 or 3 times larger than
> this.

What do you mean by object here? A whole BTree, a BTree bucket, etc?

> * a time of 60 seconds
> 
> That means that, when scanning the cache for old objects, it will
> remove those that have not been accessed in 60 seconds. This time is
> proportionally reduced as the cache size increases.

I would guess a call to _p_jar.cacheMinimize() would override this and just dump
all objects out of RAM?

> However, these parameters are not having any effect since you are not
> using Zope and currently not calling incrGC manually.

Well no, but I am calling _p_jar.cacheMinimize() quite often and it's not having
any effect on the RAM usage :-(

> If you do switch to using it then I think these values will be roughly
> right for you. If you see cache sizes growing above 1200 then reduce
> the time value. (its not very sensitive; I would try 10s next)

Well, how can I monitor and change these when I don't have a web front end? I'm
guessing this is specific to the ZEO client so starting up another ZEO client so
I can track memory usage won't help much?

> Calling cacheMinimize moves _everything_ out of memory. 

That hasn't been my experience, which would suggest something, somewhere, is
genuinely leaking...

> incrGC is designed to *maintain* a sensible pool of recently used
> objects in memory, and not let that pool grow too large. It can only
> do its job if it is called frequently.

Where does it get called in a normal Zope setup?

> For the sake of this illustration I am assuming that one of these is a
> full text index of a typically 10k document, and the rest are indexes
> of small properties (title, etc). A *very* rough estimate is that
> these may take up 20k of ram per document.

Not quite. the indexer uses three BTrees which each get one entry per document.
There are two TextIndexes. These each use one BTree for words mapping wordid to
a set of documents, and a BTree mapping to a BTree mapping to a Set of document
Ids for adjoining words. There's also a BTree mapping words to a set of sets for
unindexing purposes.
Documents are roughly 2-5k words long.
There's also a Lexicon, which has a couple btrees mapping words to wordids and
vice versa.

> You said that you are calling the garbage collector every 5000
> documents, so that makes 5000*20k = 100M of memory bloat.

Nope, I do a app._p_jar.cacheMinimize(3) every 600 docs.
I used to do a pack after 5000 docs, but I commented that out.

> Damn. thats not enough to explain all of what you are seeing.

Not by a long shot! ;-)

> Have a look at the source for the relevant bits of the zope management
> pages. Finding out exactly where the memory is going will be a big
> clue.

Indeed... have to have a look.

man, and there was me hoping to do a Squishdot release today :-S

Chris