[ZODB-Dev] BTree pickle size

[Roché Compaan]

I've been doing some benchmarks on Plone and got some surprising stats
on the pickle size of btrees and their buckets that are persisted with
each transaction. Surprising in the sense that they are very big in
relation to the actual data indexed. I would appreciate it if somebody
can help me understand what is going on, or just take a look to see if
the sizes look normal.

In the benchmark I add and index 10000 ATDocuments. I commit after each
document to simulate a transaction per request environment. Each
document has a 100 byte long description and 100 bytes in it's body. The
total transaction size however is 40K in the beginning. The transaction
sizes grow linearly to about 350K when reaching 10000 documents.

What concerns me is that the footprint of indexed data in terms of
BTrees, Buckets and Sets are huge! The total amount of data committed
that related directly to ATDocument is around 30 Mbyte. The total for
BTrees, Buckets and IISets is more than 2 Gbyte. Even taking into
account that Plone has a lot of catalog indexes and metadata columns (I
think 71 in total), this seems very high. 

This is a summary of total data committed per class:

Classname,Object Count,Total Size (Kbytes)
BTrees._IIBTree.IISet,640686,1024506
BTrees._IOBTree.IOBucket,655025,1007623
BTrees._IIBTree.IIBucket,252121,163524
BTrees._OIBTree.OIBucket,132417,101472
BTrees._IOBTree.IOBTree,25645,71072
BTrees._OOBTree.OOBucket,115332,70789
BTrees._IIBTree.IIBTree,143942,53566
BTrees._OOBTree.OOBTree,15875,52354
BTrees._IIBTree.IITreeSet,49383,25975
BTrees._OIBTree.OIBTree,4613,23008
Products.ATContentTypes.content.document.ATDocument,10000,15077
Persistence.mapping.PersistentMapping,20000,8261
Products.Archetypes.BaseUnit.BaseUnit,30000,7504
BTrees.Length.Length,220107,6382
OFS.Folder.Folder,10000,537
Products.PlonePAS.tools.memberdata.MemberData,1,0

Here is a summary of transaction sizes for the first few transactions:

Txn id,Object count,Txn size (bytes)
#00099,179,42119
#00100,175,40021
#00101,167,41746
#00102,171,45480
#00103,171,48411
#00104,173,51524
#00105,171,54265
#00106,175,57744
#00107,175,60380
#00108,180,64854
#00109,172,61819
#00110,176,66281
#00111,173,66906
#00112,176,70307
#00113,174,71629
#00114,184,78853
#00115,181,79756
#00116,188,84928
...

An the last few transactions:

Txn id,Object count,Txn size (bytes)
#10081,234,343926
#10082,226,341061
#10083,245,394237
#10084,237,367932
#10085,228,338461
#10086,184,310049
#10087,189,314684
#10088,246,405305
#10089,215,334854
#10090,221,346977
#10091,195,318492
#10092,224,351770
#10093,221,345032
#10094,206,332271
#10095,241,541394
#10096,191,283578
#10097,236,323354
#10098,242,329099
#10099,226,339302

Transaction detail for txn #00099 (first document):

Txn id,Classname,Object count,Size (bytes)
#00099,BTrees._IIBTree.IIBTree,3,286
#00099,OFS.Folder.Folder,1,55
#00099,BTrees._IOBTree.IOBucket,9,4572
#00099,BTrees._OIBTree.OIBucket,5,2964
#00099,BTrees._IOBTree.IOBTree,39,17552
#00099,BTrees.Length.Length,27,768
#00099,Persistence.mapping.PersistentMapping,2,846
#00099,Products.ATContentTypes.content.document.ATDocument,1,1544
#00099,BTrees._OOBTree.OOBTree,20,3986
#00099,BTrees._IIBTree.IISet,3,184
#00099,BTrees._OIBTree.OIBTree,9,1404
#00099,Products.Archetypes.BaseUnit.BaseUnit,3,767
#00099,BTrees._OOBTree.OOBucket,2,3286
#00099,BTrees._IIBTree.IITreeSet,55,3905

Transaction detail for txn #10099 (last document):

Txn id,Classname,Object count,Size (bytes)
#10099,BTrees._IIBTree.IIBTree,8,2517
#10099,OFS.Folder.Folder,1,55
#10099,BTrees._IOBTree.IOBucket,57,81564
#10099,BTrees._OIBTree.OIBucket,13,9872
#10099,BTrees._IIBTree.IIBucket,29,20024
#10099,BTrees._IOBTree.IOBTree,1,85
#10099,Persistence.mapping.PersistentMapping,2,846
#10099,BTrees.Length.Length,22,655
#10099,Products.ATContentTypes.content.document.ATDocument,1,1544
#10099,BTrees._OOBTree.OOBTree,6,30455
#10099,BTrees._IIBTree.IISet,65,182708
#10099,Products.Archetypes.BaseUnit.BaseUnit,3,767
#10099,BTrees._OOBTree.OOBucket,16,8088
#10099,BTrees._IIBTree.IITreeSet,2,122


Does it look normal?


-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za