Controlling caching of objects
In Zope 2.6.2 it's quite easy to put an upper limit on the number of objects in memory per thread. But does this mean that Zope won't swap out objects until it reaches that limit? It looks like this: Target # of objects per thread: 20,000 Actual # of objects per thread: ~15,000 Actual # of ghost objects per thread: ~43,000 Were the ghost objects once loaded in memory and then swapped out? If so, why were they swapped out? I find the current system of controlling # of objects in memory per thread to be a bit difficult to use. Most people, I think, would prefer to put an upper limit on the total MB of memory Zope uses instead. Are, what are the cons/pros? Bye, -- Bjorn
Bjorn Stabell wrote:
Most people, I think, would prefer to put an upper limit on the total MB of memory Zope uses instead. Are, what are the cons/pros?
The #1 con is that knowing the memory usage of an object on the host system is, afaik, nigh unto impossible in Python. The technique used by some of Zope's content cache systems is to pickle the object to be cached and use the size of the pickle as an indication of the relative amount of memory used. That technique is far from perfect though, and probably not suitable for the object cache of a zodb connection. -- Jamie Heilman http://audible.transient.net/~jamie/ "You came all this way, without saying squat, and now you're trying to tell me a '56 Chevy can beat a '47 Buick in a dead quarter mile? I liked you better when you weren't saying squat kid." -Buddy
(I trimmed the crossposts) On Wednesday 22 October 2003 09:05, Bjorn Stabell wrote:
In Zope 2.6.2 it's quite easy to put an upper limit on the number of objects in memory per thread. But does this mean that Zope won't swap out objects until it reaches that limit?
The right term is "deactivate". The main reason for deactivating an object is that the total number of objects exceeds the per-thread limit. There are other reasons.
It looks like this:
Target # of objects per thread: 20,000 Actual # of objects per thread: ~15,000 Actual # of ghost objects per thread: ~43,000
Were the ghost objects once loaded in memory and then swapped out?
No. Ghost objects are those that need a small (100 byte approx) stub in memory because there is an active object that has a reference to it. The difference between a ghost (stub) and a full object is that the ghost object does not have all its attributes loaded into memory. It does has its oid in memory though, so it can ask the storage to load those attributes on demand. This is called "activation". Note that if one of those attributes is a reference to another persistent object (for example, think of folders), then activation will trigger the creation of a ghost for each referenced object. Ghosts are removed from memory based on reference counting, and is the inverse of the process described in the previous paragraph.
I find the current system of controlling # of objects in memory per thread to be a bit difficult to use. Most people, I think, would prefer to put an upper limit on the total MB of memory Zope uses instead. Are, what are the cons/pros?
pro: it would be much better. con: it hasnt been done yet. The current system is much better than in versions before 2.6. Noone has found the current system bad enough to give it the effort needed for improvement. Patches gratefully accepted. -- Toby Dickenson
participants (3)
-
Bjorn Stabell -
Jamie Heilman -
Toby Dickenson