[Zope] Ghost objects and memory usage

[Tim Peters]

[Etienne Labuschagne]
> I have a very long running process which causes the Zope process to
> eventually eat up all the RAM in my PC.  I have tried changing the
> object cache sizes to no avail.  Zope sticks to the cache parameters
> fairly well (it doesn't overshoot the target by much).  The number of
> ghost objects do keep on increasing, however.  Does a ghost object use
> the full amount of RAM (for that typical type of object) or is it only
> an empty shell?

Ghosts are still Python objects, and still consume non-trivial memory,
but typically much less than a non-ghost object of the same type. 
Exact details depend on exact versions of Zope and Python in use;
here's a typical detailed analysis:

    http://www.python.org/~jeremy/weblog/030423.html

In that version, each ghost consumed 96 bytes.

> How do I get Zope to delete/unload these objects manually?

In general, you cannot force this.  A ghost is a Python object, and
like any other Python object, stays in memory until its reference
count falls to 0.

> B.t.w. I have tried numerous ways to do the long running process including:
> 
> One thread that get_transaction().commit() every now and again
> Get new thread, do a few transactions, commit, close thread, wash,
> rinse, repeat (yes, I know the pitfalls with opening new threads into
> Zope - I end up reusing one connector from the pool over and over
> again).
>
> The above made no difference to the fact that eventually, memory will run out.
>
> The process makes changes to many objects and creates many new
> objects.  It takes a long time to complete (typically a day or more).
>
> If I know the ghost objects does not use up memory,

They do.

> I can hunt for memory leaks in my own code,

That's still a good idea.

> but currently the only thing that increases and increases is the number of
> ghost objects.

Which does account for memory use increasing and increasing.

A typical cause is appending objects to a list, or inserting them in a
dict, where the list or dict stays alive, and not removing them. 
Those objects can never go away then, although it doesn't stop them
from becoming ghosts.  Heh -- Jim Fulton cut the memory use of the
Zope3 test suite by 300MB just last week by fixing one of those.

Such errant code may be inside Zope or inside products you've
installed or inside your own code.  It can be hard to track down. 
It's least likely (but not impossible) to be inside Zope, just because
the odds favor that someone else would have noticed it before.