[ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

Hello,
> Just zodbbrowser with no prefix:
>
>   http://pypi.python.org/pypi/zodbbrowser
>   https://launchpad.net/zodbbrowser
>
> It's a web-app: it can connect to your ZEO server so you can inspect the
> DB while it's being used.
>   
We tried this, but we currently get an error related with the general 
security policy for zope.app. Maybe we need to install Zope?
This would be a very handy tool.
> I'd suggest dumping the last few transactions with one of the ZODB
> scripts (fsdump.py perhaps) and seeing what objects get modified.
>   
That's what we've being doing, and we got some clues. We've modified 
Jim's script in order to find out which OIDs are being rewritten, and 
how much space they are taking, and this is a fragment of it:

OID class_name total_size percent_size n_pickles min_size avg_size max_size
'\x00\x00\x00\x00%T\x89{' BTrees.OOBTree.OOBucket 17402831841 30% 8683 
1977885 2004241 2026518
'\x00\x00\x00\x00%T\x89|' BTrees.OOBTree.OOBucket 14204430890 24% 8683 
1616904 1635889 1651956
'\x00\x00\x00\x00\x04dUH' MaKaC.common.indexes.StatusIndex 11955954522 
20% 28513 418230 419315 420294
'\x00\x00\x00\x00%\xa0%\x7f' BTrees.OOBTree.OOBucket 3532998238 6% 11238 
307112 314379 320647
'\x00\x00\x00\x00%\xa0%\x80' BTrees.OOBTree.OOBucket 2193843302 3% 11238 
190816 195216 199007
'\x00\x00\x00\x00\x04\x8e\xb6\x04' BTrees.OOBTree.OOBucket 1728216003 3% 
1953 880615 884903 887285
[...]

As you can see, we have an OOBucket occupying more than 2MB (!) per 
write. That's almost 17GB only considering the last 1M transactions of 
the DB (we get ~3M transactions per week). We believe this bucket 
belongs to some OOBTree-based index that we are using, whose values are 
Python lists (maybe that was a bad choice to start with?). In any case, 
how do OOBuckets work? Is it a simple key space segmentation strategy, 
or are the values taken into account as well?
Our theory is that an OOBTree simply divides the N keys in K buckets, 
and doesn't care about the contents. So, since we are adding very large 
lists as values, the tree remains unbalanced, and since new contents 
will be added to this last bucket, each rewrite will imply the addition 
of ~2MB to the file storage.
Will the replacement of these lists with a persistent structure such as 
a PersistentList solve the issue?

Thanks for all the help everyone,

Regards,

Pedro