Re: [Zope-dev] [ZODB-Dev] Splitting a large ZODB file storage - any light-weight approach?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote:
Hi there,
we have some huge ZODB file storage (60-80 GB each) and we want to split the storage into a number of smaller storage (easier to handle for backup and packing). The storages themselves only contain a few (Zope 2) folders - let's say folders A, B, C - within the root and we want to create seperate storages for A, B, C.
Is there some easier approach than using the Zope 2 CopyManager API for moving data around. The ZODB storage iterator API comes to my mind... is this a suitable approach or is there some other light-weight approach (compared to using CopyManager API)?
Why move any data between ZODBs at all? In this situation I would try the following: - create a new empty ZODB. This one holds the root. - copy the old Data.fs into three new storage server buildouts. Attach to each with a ZEO client and delete the folders you do not need in that particular storage in the ZMI or via zopectl debug. - pack each storage so the deleted data gets trashed and the files shrink. - mount the "folder" storages into the new empty root using mount points on the clients that will use the storages. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAks7CpEACgkQRAx5nvEhZLKjYgCdGI71f/yNJS3EIpZxkyFVBmSe qhMAoJ8RBK2eq27P67Iki4h4MYMZ1kZB =FC8f -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jens Vagelpohl wrote:
- copy the old Data.fs into three new storage server buildouts. Attach to each with a ZEO client and delete the folders you do not need in that particular storage in the ZMI or via zopectl debug.
Delete operations will trigger events modifying other global metadata related to the those objects. So this approach requires modification to the application in order to avoid such side effects. Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAks7DFEACgkQCJIWIbr9KYxIcACgonZfwXWmTqguW2tlcoEsLfNv 9KMAn0+h5EZ59ConlsKUNa1BWn2+bTt2 =bP9X -----END PGP SIGNATURE-----
On Wed, Dec 30, 2009 at 9:16 AM, Andreas Jung <lists@zopyx.com> wrote:
Delete operations will trigger events modifying other global metadata related to the those objects. So this approach requires modification to the application in order to avoid such side effects.
But if you do it from a debug prompt, you can easily prevent those events from being fired. For a normal ObjectManager you do: delattr(root, 'A') # this is what _delOb does root._objects = tuple([i for i in root._objects if i['id'] != 'A']) That's the actually important part of _delObject method, without firing any events or calling any manage_ methods. Hanno
participants (3)
-
Andreas Jung -
Hanno Schlichting -
Jens Vagelpohl