[ZODB-Dev] BTree pickle size

[Roché Compaan]

On Sun, 2008-08-24 at 08:13 +0200, Dieter Maurer wrote:
> Roché Compaan wrote at 2008-8-23 19:31 +0200:
> >On Sat, 2008-08-23 at 14:09 +0200, Dieter Maurer wrote:
> >> Roché Compaan wrote at 2008-8-22 14:49 +0200:
> >> >I've been doing some benchmarks on Plone and got some surprising stats
> >> >on the pickle size of btrees and their buckets that are persisted with
> >> >each transaction. Surprising in the sense that they are very big in
> >> >relation to the actual data indexed. I would appreciate it if somebody
> >> >can help me understand what is going on, or just take a look to see if
> >> >the sizes look normal.
> >> >
> >> >In the benchmark I add and index 10000 ATDocuments. I commit after each
> >> >document to simulate a transaction per request environment. Each
> >> >document has a 100 byte long description and 100 bytes in it's body. The
> >> >total transaction size however is 40K in the beginning. The transaction
> >> >sizes grow linearly to about 350K when reaching 10000 documents.
> >> 
> >> The "Bucket" nodes store usually between 22 ("OOBucket") and 90 ("IIBucket")
> >> objects in a single bucket.
> >> 
> >> With any change, the transaction will contain unmodified data
> >> for several dozens other objects.
> >
> >Are you saying *all* 22 OOBuckets and 90 IIBuckets will be persisted
> >again whether they are modified or not?
> 
> I did not speak of "22 OOBuckets" but of typically 22 entries in an
> "OOBucket" (similarly for "IIBucket").
> 
> And indeed, when a single entry in an "OOBucket" is changed, then all
> entries are rewritten even if the other entries did not change.
> 
> That is because the ZODB load/store granularity is the persistent object
> (without persistent subobjects). An "OOBucket" is a persistent object --
> it is loaded/stored always as a whole (all entries together).

Yes, that's how I understand it. I misunderstood your original
statement and thought that there is something else at play here.

-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za