Leonardo Rochael Almeida wrote:
Why? Although the catalog manages those DateTime objects, the DateTimes are in many ZODB records.
The problem is: there should be a more direct relation between the number of allowed objects in the cache and the number of refcounts. As you say, the DateTimes are in ZODB Records, so if I see a DateTime refcount of, say 120k, I should be seeing at least that many objects in the caches, but I was seeing under 3k objects, unless there's something else that could be counted as a refcount multiplying factor, say if something was keeping a reference to the class for every instance besides the instance itself.
Well, DateTimes do not derive from ZODB.Persistent, so they may be part of a ZODB object, but are never an entire ZODB object. If you have 3k objects in the cache and there are 120k DateTimes floating around, you must have an average of 40 DateTimes per loaded object. That's huge... but if you have FieldIndexes that index DateTimes, it wouldn't surprise me. We at Zope Corp. didn't even realize what a problem this could be until recently.
So we set the cache limit to 1000 objects, and to our amazement, Zope sat comfortably at 150MB RSS, and the performance was very good. Some heavy ZCatalog pages still send Zope for a walk for a few seconds (for those and all other requests) but the site becomes responsive again in under 20 seconds. Not only that, but the DateTime refcount isn't even among the top 5 (the BTrees are winning there now).
So we now have breathing space to do the great ZCatalog reform.
Excellent. The average object size sounds very large, though: start with 150 MB, give about 15 MB for non-ZODB code and data, divide by 4 (4 threads, each with independent copies of the same data), and divide by 1000 objects. That comes to 33-35k per object. That's much larger than most Zopes I've seen. BTW, if you're using ZEO, raise ZEO cache size at the same time you lower the ZODB cache size. The ZEO cache can compensate for the performance loss. The default ZEO cache size is 20 MB, but I prefer 200 MB with today's Zope apps. Shane