[ZODB-Dev] Cache Query (why doesn't RAM usage ever drop?)

[Shane Hathaway]

Chris Withers wrote:
> Just got one of our bigger apps up to 2.6.0 and have been sniffing 
> around Toby's new cache. Looks cool, apart from the online help now 
> being out of date ;-)
> 
> However, I noticed something weird. I did a top sorted by memory usage. 
> This Zope instance is using 225Mb 'cos I re-indexed all its content 
> after the upgrade.
> 
> Now, I tried flushing the cache, but the memory usage didn't drop.
> I'm also concerned that with the target number of objects set to only 
> 400, this instance still managed to grow to 225Mb. Hmm, that said, that 
> would mean that at some stage I had 400 0.5Mb objects in memory. On this 
> server, that's perfectly believeable...
> 
> Anyway, does anyone know why memory isn't being returned to the OS by 
> python and in there anything that can be doen to make that happen?

I'm glad you brought this up, since I've been working on memory usage 
the past few days.  I noticed that after a cache flush, memory usage 
never dropped, but it also took a long time before it increased again. 
So I wrote a C extension that lets Python get memory statistics via 
"mallinfo()", and it confirmed what I suspected: after a cache flush, 
Zope only uses a fraction of the process size.  But the heap is sparse. 
  Since there's an object at the end of the address space and you can't 
move objects around, the memory can't be returned to the OS.

Brian suggested that PyMalloc might alleviate this problem.

http://www.python.org/dev/doc/devel/whatsnew/section-pymalloc.html

I learned a bit about Linux and mmap through this exercise.  I 
discovered that when you allocate a block of at least 128KB (or whatever 
you set your threshold to be), malloc actually returns a memory-mapped 
file instead of space from the heap.  I'm not quite sure where the 
"file" exists :-) but like obmalloc.c says, instead of wasting memory 
you end up only wasting address space.  I think of it as many small 
heaps rather than a single big heap.

ExtensionClass does not use PyMalloc.  I wonder if it would be worth the 
effort to change it.  (To the group) Does Python itself use PyMalloc?

Shane