[ZODB-Dev] sticky objects

[Jeremy Hylton]

On Sun, 30 Jan 2005 11:10:51 +0000, Simon Burton <simon at arrowtheory.com> wrote:
> Following on from my "memory bomb" quandry, I have been further
> stress testing my computer with ZODB :)

I hope it's been fun.  ZODB provides a lot of interesting new ways to
stress your computer that are hard to accomplish with just Python :-).

> I am needing to scan through a PersistantList of Persistant objects
> each of which has non-persistant (2nd class) data.
> As the list is built, the memory usage is low, but after a scan it pulls the whole DB into memory.

Note that if you want to keep an efficient list of many objects, you'd
be better off using an OOSet.  A PersistentList has direct references
to each of its contained objects.  All of your contained objects are
persistent objects, which means they can be turned into ghosts -- but
even a ghost use some memory (maybe 100 bytes?).  The list itself will
have a pretty large state, too.  Each reference has at least an oid of
8 bytes, so a 10,000 element list will have at least an 80k pickle.

An OOSet is organized like a BTree (but with single elements instead
of key, value pairs).  It only needs to load the buckets you are
using, so it can allow more efficient memory access.

>   # Memory = 10 Mb
>   raw_input( 'enter to scan' )
>   for item in data:
>     s = item.s
>   # Memory = 200 Mb
>   raw_input( 'enter to invalidate all' )
>   for item in data:
>     item._p_invalidate()
>   data._p_invalidate()
>   connection.cacheMinimize()
> 
>   # Memory = 200 Mb
>   raw_input( 'enter to quit' )

How are you measuring memory?  Python is not returned memory to the
OS, so any os-level tool like top or ps will report the high water
mark for the process (200MB) rather than what is currently allocated. 
It might be more useful to call something like cacheExtremeDetail()
(method on the DB object).

Jeremy