[ZODB-Dev] sticky objects

[Simon Burton]

On Sat, 29 Jan 2005 22:03:27 -0500
"Tim Peters" <tim at zope.com> wrote:

> [Simon Burton]
...
> > Below is a simple example of what I am talking about.
> >
> > Am I doing something stupid ? Is this just too pathological for ZODB ?
> 
> Which version of ZODB, which version of Python, which operating system,
> which C library, and exactly how are you measuring memory use?  All of those
> feed into what you see, although I _suspect_ the platform C malloc/free
> implementation has the biggest influence on what you see in this program.

ZODB 3.3.0final
Python 2.3.4 (whoops, was python2.3.3)
Mandrake Linux, kernel 2.6.3
glibc-2.3.3

I'm using top to measure memory usage. It's pretty good at indicating
when my machine is freaking out when a process is taking all the memory.

... 
> Detail:  because the object x is reused, every PItem instance created here
> refers to exactly the same 10KB string in memory.  You lose this property
> when reading PItem objects back from the database:  2nd-class objects are
> stored and recreated by value, not by reference.  IOW, when you read these
> things in again, each PItem object will refer to a distinct 10KB string.
> That's where the bulk of the memory gets used in the next section (about
> 2*10000 strings at about 10000 bytes each ~= 200MB).

Yes, I see. I tried a few things for x. A list of ints (taking a copy for each PItem) has the same behavior . 
In the real app x is a dict.

...
> In my run, it's obvious that neither ZODB nor Python held on to the memory.
> If your platform C free() doesn't "give the memory back to the operating 
> system" here, I don't know that there's anything you can do about that short
> of using a different C or libc. 
> 

Well, that's good. It sounds like i'll be instrumenting malloc/free to find
the problem here.

Simon.


-- 
Simon Burton, B.Sc.
Licensed PO Box 8066
ANU Canberra 2601
Australia
Ph. 61 02 6249 6940
http://arrowtheory.com