[ZODB-Dev] Data.fs size grows non-stop

[Pedro Ferreira]

Hello,
>
> In the case of BTrees, yes.  I assume your OOBuckets are used within
> OOBTrees. (?)
>
>   
Yes. I was not sure whether OOBTrees behaved exactly like a BTree.
>> So, since we are adding very large
>> lists as values, the tree remains unbalanced,
>>     
>
> No, they trees tend to stay fairly well balenced wrt keys.
>   
Well, I mean, they are balanced, but the contents are not distributed in 
an homogeneous way... which is the expected behavior since they don't 
care about the values.
>   
>> and since new contents
>> will be added to this last bucket,
>>     
>
> Why would the contents only be added to one bucket?
>   
Because we're using this OOBTree as a date index and each time we add 
new contents they're added to the entry that concerns the current day. 
Currently, the bucket that contains the current day also contains some 
days that themselves contain huge lists of contents. This will probably 
change as the buckets split, but we want to correct this once and for all...
>
>> Will the replacement of these lists with a persistent structure such as
>> a PersistentList solve the issue?
>>     
>
> It might help, but if the lists are very large, you'll still have a problem
> because a persistent list is still stored in one database record.
>   
Well, at least we won't have to rewrite the whole bucket... but still, 
it would be much nicer to fragment the list
in smaller chunks. We could use an OOBTree instead... but something less 
complex would suffice... any suggestions?

We are really considering replacing all our indexes with something more 
efficient and ZODB-friendly... do you thing zope.catalog would be a good 
choice?

Thanks once again,

Pedro