[ZODB-Dev] Optimizing RelStorage Packing for large DBs
Jens W. Klein
jens at bluedynamics.com
Mon Nov 18 18:10:34 CET 2013
On 2013-11-18 17:29, Shane Hathaway wrote:
> On 11/15/2013 06:01 PM, Jens W. Klein wrote:
>> The idea is simple:
>>
>> - iterate over all transactions starting with the lowest
>> transaction id (tid)
>> - for each transaction load the object states connected with tid
>> - for each state fetch its outgoing references and fill a table where
>> all incoming references of an object are stored as an array.
>> if an state has no references write it anyway to the table with empty
>> outgoing references
>
> I would describe the RelStorage packing algorithm with the same words,
> but since you reimplemented the algorithm from scratch, you found a more
> optimal implementation for your database. Good work!
>
Thanks for the feedback!
The only change I made is regarding to arrays - they are difficult to
handle in postgres (no fun at all). It is much easier (and faster!) to
add one row for each incoming reference (plus a reference to self).
Deleting is now easy by fetching all references with only the
self-reference and removing the objects, its left self-reference and
deleting incoming references of the objects referenced by the deleted
object.
My SQL was a bit rusty after zome years of ODBs. I'am sure a person with
deeper SQL knowledge may optimize the queries in some way.
regards Jens
--
Klein & Partner KG, member of BlueDynamics Alliance
More information about the ZODB-Dev
mailing list