[Zope] Ghost objects and memory usage

[Etienne Labuschagne]

Tim,

Thanks for the detailed response.  The size of the ghost objects will
then not cause the amount of memory usage that I see (even if my
Zope/Python combination causes the ghost objects to be twice as large
as the 96 bytes mentioned).

I was convinced from the start that most probably the leak is in my
own code, but hoped that the ghost objects may be the cause (not that
I was very optimistic).

It probably is a "objects stored in structure" type leak as explained
below as garbage collecting does not seem to help.  I'll have to hunt
the leak, then.  I will probably use the garbage collector to hunt for
the leak - any good pointer/suggestions?

Thanks
Etienne

On Wed, 23 Feb 2005 08:56:49 -0500, Tim Peters <tim.peters at gmail.com> wrote:
> [Etienne Labuschagne]
> > I have a very long running process which causes the Zope process to
> > eventually eat up all the RAM in my PC.  I have tried changing the
> > object cache sizes to no avail.  Zope sticks to the cache parameters
> > fairly well (it doesn't overshoot the target by much).  The number of
> > ghost objects do keep on increasing, however.  Does a ghost object use
> > the full amount of RAM (for that typical type of object) or is it only
> > an empty shell?
> 
> Ghosts are still Python objects, and still consume non-trivial memory,
> but typically much less than a non-ghost object of the same type.
> Exact details depend on exact versions of Zope and Python in use;
> here's a typical detailed analysis:
> 
>    http://www.python.org/~jeremy/weblog/030423.html
> 
> In that version, each ghost consumed 96 bytes.
> 
> > How do I get Zope to delete/unload these objects manually?
> 
> In general, you cannot force this.  A ghost is a Python object, and
> like any other Python object, stays in memory until its reference
> count falls to 0.
> 
> > B.t.w. I have tried numerous ways to do the long running process including:
> >
> > One thread that get_transaction().commit() every now and again
> > Get new thread, do a few transactions, commit, close thread, wash,
> > rinse, repeat (yes, I know the pitfalls with opening new threads into
> > Zope - I end up reusing one connector from the pool over and over
> > again).
> >
> > The above made no difference to the fact that eventually, memory will run out.
> >
> > The process makes changes to many objects and creates many new
> > objects.  It takes a long time to complete (typically a day or more).
> >
> > If I know the ghost objects does not use up memory,
> 
> They do.
> 
> > I can hunt for memory leaks in my own code,
> 
> That's still a good idea.
> 
> > but currently the only thing that increases and increases is the number of
> > ghost objects.
> 
> Which does account for memory use increasing and increasing.
> 
> A typical cause is appending objects to a list, or inserting them in a
> dict, where the list or dict stays alive, and not removing them.
> Those objects can never go away then, although it doesn't stop them
> from becoming ghosts.  Heh -- Jim Fulton cut the memory use of the
> Zope3 test suite by 300MB just last week by fixing one of those.
> 
> Such errant code may be inside Zope or inside products you've
> installed or inside your own code.  It can be hard to track down.
> It's least likely (but not impossible) to be inside Zope, just because
> the odds favor that someone else would have noticed it before.
>