[Zope-dev] Re: [ZODB-Dev] Re: Advice on ZODB with large datasets

[Laurence Rowe]

It's helpful to post your responses to the mailing list, that way when
someone else has a similar problem in the future they'll be able to
find the information.

Inheriting from Persistent is also necessary to control the
granularity of the database. Persistent objects are saved as separate
`records` by ZODB. Other objects do not have a _p_oid attribute and
have to be saved as part of their parent record.

Laurence

2008/6/19  <AFoglia at princeton.com>:
> Laurence Rowe wrote:
>>
>> AFoglia at princeton.com wrote:
>> Does your record class inherit from persistent.Persistent? 650k integers +
>> object pointers should only be of the order 10 Mb or so. It sounds to me
>> like the record data is being stored in the btrees bucket directly.
>
>        No, it does not.  It's just a simple dictionary for the time being.
>  I assumed the BTree bucket would itself know to load the values only when
> they are explicitly requested, and that the Persistence of the objects just
> merely meant that the database didn't keep track of changes of nonpersistent
> objects.  I will try copying my dictionaries to Persistent Mappings for now.
>
>> Something like this should lead to smaller bucket objects where the record
>> data is only loaded when you access the values of the btree:
>>
>>  >>> from BTrees.IOBTree import IOBTree
>>  >>> bt = IOBTree()
>>  >>> from persistent import Persistent
>>  >>> class Record(Persistent):
>> ...     def __init__(self, data):
>> ...         super(Record, self).__init__()
>> ...         self.data = data
>> ...
>>  >>> rec = Record("my really long string data")
>>  >>> bt[1] = rec
>
>