[Zope3-Users] Copy / Move ZODB Content From "Dev" Zope3Instance to "Production" Zope3 Instance?

[Marius Gedminas]

On Tue, Jan 22, 2008 at 12:54:52PM -0500, Jeremy Roberts wrote:
> It's true, I am looking for an easy way to move both content space objects 
> and software space objects between ZODB storages.
>
> I'd considered simply copying Data.fs files but I learned that there is a 
> small but possible risk of making a copy while a transaction is incomplete, 
> risking data integrity.

My understanding is that this is perfectly safe: ZODB will ignore the
incomplete transaction record at the end of the Data.fs, so you'll get a
snapshot of earlier state.  If you try to make any changes, ZODB will
overwrite that fragment with a new transaction record and truncate the
Data.fs.

(If my understanding is incorrect, I'd like someone to thwack me with a
cluebat.)

> I then stumbled upon the repozo.py script which can safely handle making 
> copies of Data.fs files while the database is in use. Unfortunately it 
> doesn't seemed to be included in my linux Zope 3.3.1 install.

It lives in ZODB/scripts in the source tree.  Or, alternatively, here:
svn://svn.zope.org/repos/main/ZODB/trunk/src/ZODB/scripts/repozo.py

> I am only just starting to play with eggs, but I did find that the 
> repozo.py script is packaged with a recent ZODB3-3.8.0b3 egg install.
>
> Even so, this approach (replicating entire Data.fs files) affords no 
> control over the objects that are transfered between storages - it's all or 
> nothing. This is not a deal breaker for many types of development 
> situations, but I anticipate finding myself in other types of situations 
> where it would be useful to be able to be able to transfer a few objects 
> here and a few objects there - even if it required some premeditation on my 
> part in terms of the z3 components available to me in the instances to make 
> it possible.

I miss Zope2-style .zexp files every now and then.

> My needs stem from my desire to provide the client with a dev instance, a 
> production instance, and a sane controlled process for moving not only file 
> system code (easy: darcs), but also content objects in ZODB or ZEO storage 
> between the instances.

fssync would've solved that problem (and more -- it would let you
keep your data objects in a revision control system).

You could probably cobble some solution from zc.copy.

> PS Is there a way to mount multiple Data.fs files in a Zope3 instance via 
> configuration? I've only seen it done with code in the debugger prompt.
>
> I have found description of how to acheive this with configuration in Z2 
> (mount-point directive in zope.conf's zodb element), but not Z3...
>
> If I had that capability, the only problem left for me would be the 
> reference vs deep copy problem which occurs when using copy operations 
> through the ZMI between storages. 

zope.location.pickling.locationCopy and zc.copy solve that problem.

(A hopefully unrelated accident: I used Zope 2.9 ZMI to *cut* and paste
an object across storages -- from a TemporaryStorage to a FileStorage.
The next day I discovered that my object kept some reference to
TemporaryStorage objects that got garbage-collected in the meantime.
Oops.)

> I had a friend recently point me towards 
> the python __deepcopy__ api which I haven't looked at yet, and zc.copy.

Do look at zc.copy (I just did).  It works by (smartly) making a pickle
that includes some objects and references other objects, and then
unpickling it in the next breath.

What you need to do to get Zope2-style object import/export is take that
logic, verify that all objects are included in the pickle and there are
no external references (other than a single __parent__ reference from
the original object you're exporting, which you should convert to None
before pickling) and then write that pickle to a file (or return it as
the response body to a GET request).  On the other system just unpickle
and stick it in a container somewhere.

But beware of the big can of worms you're opening:

 - pickles can execute arbitrary code, so be sure you only unpickle
   strings received from trusted sources
 - ZODB schema is implicit and not explicit, so make sure you're not
   letting your users break your system by importing objects with
   incompatible attributes (from earlier/later DB generations).
 - if you use catalogs or indexes, make sure you'll update them properly
   after importing a pickled bundle of objects
 - in general, be very wary of direct references between objects that
   may cross the export boundaries
 - beware of objects providing ILocation that aren't properly attached
   (i.e. have None as the __parent__) -- these will never be
   copied/exported

In general, I'd say it's impossible to provide reliable object copying
(or import/export) without knowing (or limiting) what your application
stores in the DB.

Marius Gedminas
-- 
No sane person should use frame buffers if they have the choice.
        -- Linus Torvalds on lkml
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zope3-users/attachments/20080123/b68f79f2/attachment.bin