[ZODB-Dev] Relstorage and over growing database.
Jens W. Klein
jens at bluedynamics.com
Wed Nov 13 10:24:37 CET 2013
On 2013-11-12 17:48, Martijn Pieters wrote:
>
> On Mon, Nov 11, 2013 at 9:24 PM, Daniel Widerin <daniel at widerin.net
> <mailto:daniel at widerin.net>> wrote:
>
> Anyone experienced similar problems packing large relstorage databases?
> The graph traversal takes a really long time. maybe we can improve that
> by storing additional information in the relational database?
>
>
> You should (at least initially) pack *without* GC (set pack-gc true to
> false); I packed a humongous RelStorage-backed database before, and
> packed to earlier dates in the past first to minimize the amount of data
> removed in a single transaction.
>
> Only when we were down to a reasonable size database did we enable
> garbage collection.
Thanks Martijn for the hint, but we are using a history free database,
so growing does only happen by deleted objects in our case.
When in history free mode, is it possible to detect deleted objects at
store-time? This way we could add the zoid at store time to a
objects_deleted table in order to clean them up later.
Another way to speed up graph traversal would be to store the
object-references in a field of object_state. At the moment we have to
read the pickle in order to get the referenced zoids. Storing additional
- redundant - information might be not perfect, but it would allow to
pack/gc the database without any knowledge about the state objects
structure, i.e. using a stored procedure.
I would like to know what the relstorage experts think about this ideas.
kind regards
Jens
--
Klein & Partner KG, member of BlueDynamics Alliance
More information about the ZODB-Dev
mailing list