[ZODB-Dev] Estrange behaviour with BTreeFolder2

[Marius Gedminas]

On Wed, Apr 01, 2009 at 11:46:01AM +0200, Santi Camps wrote:
> I'm having memory problems and, after some debugging with gc, I see
> some estrange behaviours with BTreeFolder2 (or with BTrees itself, I'm
> not sure).   I've a BTreeFolder2 with 40000 objects, more o less.
> Some of them are emails.   Just accessing the container, about 700
> mails are loaded in memory !!
> 
> from AccessControl.SecurityManagement import newSecurityManager
> from Testing.makerequest import makerequest
> user = app.beta.acl_users.getUser('manager').__of__(app.beta.acl_users)
> newSecurityManager({}, user)
> app = makerequest(app)
> km = app.beta
> km.portal_repository._tree
> import gc
> objects = gc.get_objects()
> objects2 = [obj for obj in objects if getattr(obj, '__class__', None)]

(I'm a bit surprised that you get objects without a __class__ attribute
-- could you elaborate about those?  Old-style class instances?
Extension types?)

> emails = [obj for obj in objects2 if 'mail' in obj.__class__.__name__.lower()]

How many different classes are there with 'mail' in the (lowercased)
class name?

Do they all inherit from Persistent?

> print len(emails)
> 697
> 
> Following gc references, I see this mails are referenced from a
> OOBucket, that is referenced by another OOBucket, ...

I assume those mails are values rather than keys.

> and some of them
> are referenced from persistent.PickleCache (instead I've configured
> cache-size = 0 in zope.conf).   Seems that some buckets and its
> referenced objects are readed during btree loading.   So, the garbage
> collector never clears this from memory, and after some hours of
> intensive work, the server RAM is finished
> 
> Is that normal ?  I'm making some mistake ?  Anybody knows any way to solve it ?

Your description would make perfect sense to me if your mail classes
weren't subclasses of Persistent (and the fix would be: make them
subclass Persistent).

Marius Gedminas
-- 
I used to (somewhat) sneer at people who describe themselves thus: "I program
HTML". Then I tried to make a web-site look as I wanted it to (ie. not
grotesquely ugly), wow - perhaps they were onto something. Some think there is
a pile of broken mess in our desktop, but at least you know where you are with
a GtkHBox.
                -- Michael Meeks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20090401/eb6cab38/attachment.bin