Tim, Thanks for the detailed response. The size of the ghost objects will then not cause the amount of memory usage that I see (even if my Zope/Python combination causes the ghost objects to be twice as large as the 96 bytes mentioned). I was convinced from the start that most probably the leak is in my own code, but hoped that the ghost objects may be the cause (not that I was very optimistic). It probably is a "objects stored in structure" type leak as explained below as garbage collecting does not seem to help. I'll have to hunt the leak, then. I will probably use the garbage collector to hunt for the leak - any good pointer/suggestions? Thanks Etienne On Wed, 23 Feb 2005 08:56:49 -0500, Tim Peters <tim.peters@gmail.com> wrote:

[Etienne Labuschagne]

...

I have a very long running process which causes the Zope process to eventually eat up all the RAM in my PC. I have tried changing the object cache sizes to no avail. Zope sticks to the cache parameters fairly well (it doesn't overshoot the target by much). The number of ghost objects do keep on increasing, however. Does a ghost object use the full amount of RAM (for that typical type of object) or is it only an empty shell?

Ghosts are still Python objects, and still consume non-trivial memory, but typically much less than a non-ghost object of the same type. Exact details depend on exact versions of Zope and Python in use; here's a typical detailed analysis:

http://www.python.org/~jeremy/weblog/030423.html

In that version, each ghost consumed 96 bytes.

...

How do I get Zope to delete/unload these objects manually?

In general, you cannot force this. A ghost is a Python object, and like any other Python object, stays in memory until its reference count falls to 0.

...

B.t.w. I have tried numerous ways to do the long running process including:

One thread that get_transaction().commit() every now and again Get new thread, do a few transactions, commit, close thread, wash, rinse, repeat (yes, I know the pitfalls with opening new threads into Zope - I end up reusing one connector from the pool over and over again).

The above made no difference to the fact that eventually, memory will run out.

The process makes changes to many objects and creates many new objects. It takes a long time to complete (typically a day or more).

If I know the ghost objects does not use up memory,

They do.

...

I can hunt for memory leaks in my own code,

That's still a good idea.

...

but currently the only thing that increases and increases is the number of ghost objects.

Which does account for memory use increasing and increasing.

A typical cause is appending objects to a list, or inserting them in a dict, where the list or dict stays alive, and not removing them. Those objects can never go away then, although it doesn't stop them from becoming ghosts. Heh -- Jim Fulton cut the memory use of the Zope3 test suite by 300MB just last week by fixing one of those.

Such errant code may be inside Zope or inside products you've installed or inside your own code. It can be hard to track down. It's least likely (but not impossible) to be inside Zope, just because the odds favor that someone else would have noticed it before.