[ZODB-Dev] Cache Query (why doesn't RAM usage ever drop?)

[Tim Peters]

[Jeremy]
> Python 2.2 and up uses pymalloc

No, it's 2.3 and up.  See earlier reply.

> (or is its right name obmalloc?).

The file is named obmalloc, but we *call* it pymalloc.  That's how we
distinguish people who know what they're talking from dilettantes <wink>.

> But pymalloc never returns memory to the system.

Not so:  it never returns its own arenas to the platform free, but only
requests for < 256 bytes are satisfied from its own areans.  All other
requests are passed on to the platform malloc(), and all free()s of such
things are passed on to the platform free().

> As a rule, I'd say its practically impossible to control when memory
> is returned to the system.

This is very much so, and because passing a thing on to free() is the only
thing Python *can* do, and passing a thing on to free() has no direct
relationship with whether "memory is returned to the system" -- that's up to
the guts of the platform free().

> Does any have evidence of harmful effects caused by large process
> size?

On Win9x, yes.

> After a balloon event (memory balloons to a large size and then
> deflates when the transaction commits), most of the pages aren't
> touched so the working set size is smaller.

pymalloc is also *very* careful never to touch address space unless it
actually needs to.  Indeed, one of the reasons Vladimir wrote pymalloc *not*
to return its own arenas to free() appears to be that the bookkeeping
required to do so would end up periodically touching "extra" address space.
If a pymalloc arena happens not to contain an active object, nothing ever
touches its address space, and it will simply page out.