[ZODB-Dev] Relstorage and over growing database.
Daniel Widerin
daniel at widerin.net
Mon Nov 11 22:24:55 CET 2013
Hi, just want to share our experience:
My ZODB contains 300mio objects on relstorage/pgsql. The amount of
objects is caused by btrees stored on plone dexterity contenttypes. It's
size is 160GB. At that size it's impossible to pack because the pre-pack
takes >100 days.
jensens and me are searching for different packing algorithms and
methods to achieve better packing performance. We're keeping you updated
here!
How i solved my problem for now:
I converted into FileStorage which took about 40 hours and Data.fs was
55GB in size. Now i tried to run zeopack on that database - which
succeeded and database was reduced to 7.8 GB - still containing 40mio
objects. After that i migrated back to relstorage because of better
performance and the result is a 11 GB db in pgsql.
Anyone experienced similar problems packing large relstorage databases?
The graph traversal takes a really long time. maybe we can improve that
by storing additional information in the relational database?
Any hints or comments are welcome.
Daniel
Am 07.02.13 23:10, schrieb Shane Hathaway:
> On 02/07/2013 01:54 PM, Jürgen Herrmann wrote:
>> Am 07.02.2013 21:18, schrieb Jürgen Herrmann:
>>> I know that's entirely not your fault but may be worth mentioning
>>> in the docs. Relstorage with MySQL works *very* well for DB sizes
>>> <5GB or so, above that - not so much :/
>>
>> Also for the docs: on disk Restorage/MySQL uses 4x the size of a
>> FileStorage with same contents. As packing tables are filled this
>> grows by another factor of ~2. If you don't pack very regularly
>> you might up ending in DBs that donb't permit packing anymore
>> because of the big size very quickly.
>
> I suspect there are ways to fix all of that in the MySQL configuration.
> Like any SQL database, MySQL needs tuning as it grows. Meanwhile,
> FileStorage doesn't really have any knobs, and it always stores in a
> fairly optimal way, so it's easier to use.
>
> FileStorage has a couple of issues that often drive people to
> RelStorage: (1) the on-disk format is unique to FileStorage, and there
> aren't many tools available for analyzing and fixing a broken Data.fs.
> (2) FileStorage only supports multiple clients through ZEO, which has
> relatively high latency. If these issues don't impact you, then
> FileStorage is clearly the better choice for you.
>
> Shane
>
> _______________________________________________
> For more information about ZODB, see http://zodb.org/
>
> ZODB-Dev mailing list - ZODB-Dev at zope.org
> https://mail.zope.org/mailman/listinfo/zodb-dev
More information about the ZODB-Dev
mailing list