[ZODB-Dev] BTree pickle size

[Roché Compaan]

On Sun, 2008-08-24 at 08:55 +0200, Roché Compaan wrote:
> Thanks for the feedback. I'll re-run the tests without any text indexes,
> as well as run it with other implementations such as TextIndexNG3 and
> SimpleTextIndex and compare the results.
> 

Some more tests show that text indexes aren't the worst offenders. Date
and DateRangeIndexes use IISet in cases where IITreeSet seem more
appropriate. To me there isn't much more value to investigate other text
index implementations. I'd rather spend to time to compare the overall
results with other indexing implementations altogether, like solr or
indexing in a RDMBS.

Listed below are some stats (where I ran my original test in which I
create 10000 documents) that compare an unmodified setup, a catalog
without text indexes, a catalog without date indexes, a catalog without
metadata and no catalog at all.

Total size of default setup:              2569.97 MB
Total size excluding text indexes:        1963.89 MB
Total size excluding date range indexes:  2043.26 MB
Total size excluding catalog:              270.90 MB


Unmodified setup (default portal_catalog):
==========================================
Classname, Count, Size (kB), Size (mB), Avg (kB)
------------------------------------------------
Products.PlonePAS.tools.memberdata.MemberData, 1, 0, 0, 0
OFS.Folder.Folder, 10000, 537, 0.52, 0.05
BTrees.Length.Length, 220107, 6382, 6.23, 0.03
Products.Archetypes.BaseUnit.BaseUnit, 30000, 7504, 7.33, 0.25
Persistence.mapping.PersistentMapping, 20000, 8261, 8.07, 0.41
Products.ATContentTypes.content.document.ATDocument, 10000, 15077,
14.72, 1.51
BTrees._OIBTree.OIBTree, 4613, 23008, 22.47, 4.99
BTrees._IIBTree.IITreeSet, 49383, 25975, 25.37, 0.53
BTrees._OOBTree.OOBTree, 15875, 52354, 51.13, 3.3
BTrees._IIBTree.IIBTree, 143942, 53566, 52.31, 0.37
BTrees._OOBTree.OOBucket, 115332, 70789, 69.13, 0.61
BTrees._IOBTree.IOBTree, 25645, 71072, 69.41, 2.77
BTrees._OIBTree.OIBucket, 132417, 101472, 99.09, 0.77
BTrees._IIBTree.IIBucket, 252121, 163524, 159.69, 0.65
BTrees._IOBTree.IOBucket, 655025, 1007623, 984.01, 1.54
BTrees._IIBTree.IISet, 640686, 1024506, 1000.49, 1.6

Total size: 2569.97 MB


No text indexes:
================
Classname, Count, Size (kB), Size (mB), Avg (kB)
------------------------------------------------
OFS.Folder.Folder, 10000, 537, 0.52, 0.05
BTrees.Length.Length, 120077, 3397, 3.32, 0.03
Products.Archetypes.BaseUnit.BaseUnit, 30000, 7504, 7.33, 0.25
Persistence.mapping.PersistentMapping, 20000, 8242, 8.05, 0.41
Products.ATContentTypes.content.document.ATDocument, 10000, 15057, 14.7,
1.51
BTrees._OIBTree.OIBTree, 3467, 15740, 15.37, 4.54
BTrees._IIBTree.IITreeSet, 45734, 24355, 23.78, 0.53
BTrees._IOBTree.IOBTree, 20895, 37559, 36.68, 1.8
BTrees._OOBTree.OOBTree, 15878, 52460, 51.23, 3.3
BTrees._OOBTree.OOBucket, 115336, 70789, 69.13, 0.61
BTrees._OIBTree.OIBucket, 92499, 75572, 73.8, 0.82
BTrees._IOBTree.IOBucket, 287777, 670918, 655.19, 2.33
BTrees._IIBTree.IISet, 644224, 1028891, 1004.78, 1.6

Total size: 1963.89 MB


No date and date range indexes:
===============================
Classname, Count, Size (kB), Size (mB), Avg (kB)
------------------------------------------------
OFS.Folder.Folder, 10000, 537, 0.52, 0.05
BTrees.Length.Length, 220002, 6379, 6.23, 0.03
Products.Archetypes.BaseUnit.BaseUnit, 30000, 7504, 7.33, 0.25
Persistence.mapping.PersistentMapping, 20000, 8242, 8.05, 0.41
Products.ATContentTypes.content.document.ATDocument, 10000, 15057, 14.7,
1.51
BTrees._OIBTree.OIBTree, 2688, 16786, 16.39, 6.24
BTrees._IIBTree.IITreeSet, 35868, 20593, 20.11, 0.57
BTrees._OOBTree.OOBTree, 15898, 52997, 51.75, 3.33
BTrees._IIBTree.IIBTree, 143806, 53443, 52.19, 0.37
BTrees._IOBTree.IOBTree, 24944, 68928, 67.31, 2.76
BTrees._OOBTree.OOBucket, 115356, 70806, 69.15, 0.61
BTrees._OIBTree.OIBucket, 81022, 76848, 75.05, 0.95
BTrees._IIBTree.IIBucket, 251985, 163777, 159.94, 0.65
BTrees._IIBTree.IISet, 582704, 519508, 507.33, 0.89
BTrees._IOBTree.IOBucket, 644640, 1010896, 987.2, 1.57

Total size: 2043.26 MB


No metadata:
============
Classname, Count, Size (kB), Size (mB), Avg (kB)
------------------------------------------------
OFS.Folder.Folder, 10000, 537, 0.52, 0.05
BTrees.Length.Length, 220095, 6381, 6.23, 0.03
Products.Archetypes.BaseUnit.BaseUnit, 30000, 7504, 7.33, 0.25
Persistence.mapping.PersistentMapping, 20000, 8242, 8.05, 0.41
Products.ATContentTypes.content.document.ATDocument, 10000, 15057, 14.7,
1.51
BTrees._OIBTree.OIBTree, 4608, 22971, 22.43, 4.99
BTrees._IIBTree.IITreeSet, 47718, 25185, 24.59, 0.53
BTrees._OOBTree.OOBTree, 15882, 52566, 51.33, 3.31
BTrees._IIBTree.IIBTree, 143948, 53520, 52.27, 0.37
BTrees._IOBTree.IOBTree, 25604, 69710, 68.08, 2.72
BTrees._OOBTree.OOBucket, 115340, 70774, 69.12, 0.61
BTrees._OIBTree.OIBucket, 132412, 101655, 99.27, 0.77
BTrees._IIBTree.IIBucket, 252127, 164492, 160.64, 0.65
BTrees._IOBTree.IOBucket, 655008, 689299, 673.14, 1.05
BTrees._IIBTree.IISet, 642280, 1027442, 1003.36, 1.6

Total size: 2261.07 MB


No catalog:
===========
Classname, Count, Size (kB), Size (mB), Avg (kB)
------------------------------------------------
OFS.Folder.Folder, 9524, 511, 0.5, 0.05
BTrees._IIBTree.IITreeSet, 550, 632, 0.62, 1.15
BTrees.Length.Length, 66668, 1904, 1.86, 0.03
BTrees._OIBTree.OIBTree, 1114, 6211, 6.07, 5.58
Products.Archetypes.BaseUnit.BaseUnit, 28572, 7147, 6.98, 0.25
Persistence.mapping.PersistentMapping, 19048, 7722, 7.54, 0.41
BTrees._IIBTree.IISet, 19122, 9239, 9.02, 0.48
BTrees._IOBTree.IOBTree, 3312, 10453, 10.21, 3.16
Products.ATContentTypes.content.document.ATDocument, 9524, 14352, 14.02,
1.51
BTrees._OOBTree.OOBTree, 12010, 23170, 22.63, 1.93
BTrees._OOBTree.OOBucket, 49857, 30574, 29.86, 0.61
BTrees._OIBTree.OIBucket, 29354, 33597, 32.81, 1.14
BTrees._IOBTree.IOBucket, 88074, 131886, 128.79, 1.5

Total size: 270.9 MB

-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za