"R. David Murray" wrote:
OK, so just what determines Zope's memory usage? I naively assumed that beyond a certain minimum size it would be the object cache.
Also the database index. This grows a small amount for each object in the database. This is stored entirly in memory.
But my zope process seems to just keep growing. If I go to the' database management page and flush the cache, the size doesn't change.
The I would suspect a memory leak, or circular references. A memory leak means simply that some part of your application is allocating memory and not releasing it. This typically happens in C. Suspect are the Zope core components themselves (possible), any third party database adapters you use (likely), or a core python module (not likely). Circular refs are a common mistake in python. Python works on a ref counting system. When the number of references to an object drops to 1 in Zope, then the object is deactivated and written to disk (if it changed). Why 1? Well, when the ref count is one then that means only one object reffers to your object, and that object *should* be the object cache itself. When the cache sees that an object has only only reference, it knows that that reference is itself and it can deactivate the object. If you have another object A, that refers to B, and B refers to A, then both will have a ref count of 2 and neither object will ever get deactivated. Over time if a lot of these pairs of objects come about then you will run out of memory and flushing the cache will do you no good (because according to the cache they are not unreferenced).
I need to understand at least a little bit more about the memory profile, because the Zope process is periodicaly consuming all available swap space and I have to restart it.
Look at Control_Panel/manage_debug. This shows you the refcounts for all objects. Track these values over the course of running Zope and see if any of them go up and never down. You'll get the idea.
I'm sure the fact that I'm touching a lot of objects in the database (a couple thousand) is part of the problem, but I'm doing it in batches of 250, so I would think Zope's memory management could handle that.
Yes it shouldn't be a problem.
On a possibly related issue, I turned on the stupid file logger, but now I occasionally get an error page with the following traceback:
Error Type: TypeError Error Value: call of non-function (type file)
<!-- Traceback (innermost last): File ...Zope-2.1.2-src/lib/python/Zope/ZLogger/stupidFileLogger.py, line 99, in stupid_log_write TypeError: (see above)
Hmmmm... that's not normal. Line 99 is a call to _stupid_dest.write, how is _stupid_dest.write getting assigned a file object? I know your asking _us_ that, but I personaly have no idea. Perhaps you could track it down a bit further given that information. -Michel