[ZODB-Dev] Cache Query (why doesn't RAM usage ever drop?)

[Shane Hathaway]

Tim Peters wrote:
> [Shane Hathaway, to Chris Withers]
> (snip)
>>I learned a bit about Linux and mmap through this exercise.  I
>>discovered that when you allocate a block of at least 128KB (or whatever
>>you set your threshold to be), malloc actually returns a memory-mapped
>>file instead of space from the heap.  I'm not quite sure where the
>>"file" exists :-) but like obmalloc.c says, instead of wasting memory
>>you end up only wasting address space.  I think of it as many small
>>heaps rather than a single big heap.
> 
> 
> That's a good way to think of it!  Under the covers, Windows ends up
> allocating many distinct *heaps*, which the 9x flavors do a much poorer job
> of than the NT+ flavors.  On 9x I can construct little C programs that cause
> the OS to crash due to fatal fragmentation of the entire address space
> despite having only about 1% of the total address space in use.
> 
> So like obmalloc.c says <wink>, while it's just a waste of address space, it
> can be fatal nevertheless, given a bad enough OS.

This could be a great counterexample for an operating system design 
course. :-)

>>ExtensionClass does not use PyMalloc.  I wonder if it would be worth the
>>effort to change it.  (To the group) Does Python itself use PyMalloc?
> 
> 
> In 2.3 it's enabled, but not before.  I'm afraid I have to advise against
> enabling pymalloc before 2.3:  I spent about a month bulletproofing the
> pymalloc code for 2.3, which means making Python's memory API wholly
> backward compatible in its presence, and plugging security holes (if you dig
> thru the Python-Dev archives, you'll eventually find a pure-Python program
> that can crash the interpreter, overwrite bytecode, etc etc, by exploiting
> holes in the pre-2.3 version of pymalloc; those holes are well & truly
> plugged in the 2.3 version).  The pre-2.3 pymalloc also performs poorly with
> new-style classes, because pymalloc was written long before those, and
> needed tuning and new mini-algorithms to work well with new-style classes.

Where would Python be without you, Tim? :-)  Thanks for the advice not 
to pursue the pymalloc course for now.  We'll just have to rely on the 
ZODB cache settings for now, I guess.  We did manage to reduce memory 
consumption a lot by reducing the database connection pool size from 7 
to 2. Chris, you might want to look into this option.

Shane