[ZODB-Dev] BTree memory bomb

[Simon Burton]

On Tue, 18 Jan 2005 14:35:41 -0500
"Tim Peters" <tim at zope.com> wrote:

> 
> Here's the test driver again, using a fixed amount of new data per
> iteration, and avoiding len(BTree) calls in the loops.  It does grow
> steadily over time (probably due to various in-memory indexing structures),
> to about 20MB RAM on my box after 300 outer-loop iterations (at which point
> it was still running fine, I just got tired of tying up the machine with
> it):

Aha! yes, it was the len(BTree) that kept the thing in memory.
Now when I run it (the original script without the len), the DB file quickly grows to 700Mb,
but memory usage only gets to around 50Mb. 

I guess I should have mentioned, the application is for a web cache, which
I foresee growing easily to the gigabyte range. I don't particularly need to know
the len of it, and if I did I could store that in a counter. But, it was important to test
useing big and distinct values, not just 'abc', as this does not "memory bomb".

I see there are still finer issues to consider, such as index size (??) and Storage backend,
but now at least the cache can grow much bigger than memory available, so that's great.

Thanks alot for the detailed analysis.

Simon.

-- 
Simon Burton, B.Sc.
Licensed PO Box 8066
ANU Canberra 2601
Australia
Ph. 61 02 6249 6940
http://arrowtheory.com