[Zope-dev] Zope memory usage

Michel Pelletier michel@digicool.com
Tue, 14 Mar 2000 09:18:45 -0800


"R. David Murray" wrote:
> 
> OK, so just what determines Zope's memory usage?  I naively assumed
> that beyond a certain minimum size it would be the object cache.

Also the database index.  This grows a small amount for each object in
the database.  This is stored entirly in memory.

> But my zope process seems to just keep growing.  If I go to the'
> database management page and flush the cache, the size doesn't change.

The I would suspect a memory leak, or circular references.  A memory
leak means simply that some part of your application is allocating
memory and not releasing it.  This typically happens in C.  Suspect are
the Zope core components themselves (possible), any third party database
adapters you use (likely), or a core python module (not likely).

Circular refs are a common mistake in python.  Python works on a ref
counting system.  When the number of references to an object drops to 1
in Zope, then the object is deactivated and written to disk (if it
changed).  Why 1?  Well, when the ref count is one then that means only
one object reffers to your object, and that object *should* be the
object cache itself.  When the cache sees that an object has only only
reference, it knows that that reference is itself and it can deactivate
the object.

If you have another object A, that refers to B, and B refers to A, then
both will have a ref count of 2 and neither object will ever get
deactivated.  Over time if a lot of these pairs of objects come about
then you will run out of memory and flushing the cache will do you no
good (because according to the cache they are not unreferenced).

> I need to understand at least a little bit more about the memory
> profile, because the Zope process is periodicaly consuming all
> available swap space and I have to restart it.

Look at Control_Panel/manage_debug.  This shows you the refcounts for
all objects.  Track these values over the course of running Zope and see
if any of them go up and never down.  You'll get the idea.

> I'm sure the
> fact that I'm touching a lot of objects in the database (a couple
> thousand) is part of the problem, but I'm doing it in batches of
> 250, so I would think Zope's memory management could handle that.

Yes it shouldn't be a problem.
 
> On a possibly related issue, I turned on the stupid file logger,
> but now I occasionally get an error page with the following
> traceback:
> 
> Error Type: TypeError
> Error Value: call of non-function (type file)
> 
> <!--
> Traceback (innermost last):
>   File ...Zope-2.1.2-src/lib/python/Zope/ZLogger/stupidFileLogger.py, line 99, in stupid_log_write
> TypeError: (see above)

Hmmmm... that's not normal.  Line 99 is a call to _stupid_dest.write,
how is _stupid_dest.write getting assigned a file object?

I know your asking _us_ that, but I personaly have no idea.  Perhaps you
could track it down a bit further given that information.

-Michel