[ZODB-Dev] merging databases containing multidatabase references

[David Glick]

On Apr 24, 2009, at 2:37 AM, Shane Hathaway wrote:
> Christian Theune wrote:
>> On Thu, 2009-04-23 at 15:02 -0700, David Glick wrote:
>>> Having now realized the folly of trying to split things into  
>>> multiple  databases, I would really like to merge these 3  
>>> databases back into  1.  Any ideas for how I could go about this?   
>>> Based on http://docs.zope.org/zope3/Code/ZODB/cross-database-references.txt/index.html 
>>>    and some experimentation, multi-database references are not   
>>> supported by the import/export code.  Do I have other options?
>> I think it should be possible to write a tool that merges one  
>> database
>> into another. This needs some processing on the database objects as
>> their OIDs (and thus the internal references) need to be reorganized.
>
> The import machinery in ZODB/ExportImport.py provides a good example  
> of how to rewrite OIDs.  However, I am concerned that understanding  
> that code requires a lot of experience with ZODB.
>
> David, if you want to tackle writing a tool that merges multi- 
> databases by rewriting OIDs, it probably won't be easy, but I think  
> you can count on help from this list.  I also think the tool would  
> be useful for others.


Taking the existing ExportImport.py code as a starting point, it looks  
like I would need to:
1. modify the persistent_load method to handle rewriting extended  
references of types 'n' and 'm'
2. provide some way to process several export files at once, and input  
a mapping of filename to database name (to handle when I am, say,   
importing 2 databases with references between each other)

Does this sound right?  I'm not in any rush but I may tackle this if I  
find time.  It looks like OFS has a custom importer/exporter for XML  
that I may be able to use as a model....hmm, though on closer  
inspection this only rewrites the XML format into the standard format  
and then uses the same _importDuringCommit implementation on the  
Connection; it looks like I probably can't provide an alternate  
_importDuringCommit without monkeypatching.

thanks all for the advice so far...

David Glick
Web Developer
ONE/Northwest

New tools and strategies for engaging people in protecting the  
environment

http://www.onenw.org
davidglick at onenw.org
work: (206) 286-1235 x32
mobile: (206) 679-3833

Subscribe to ONEList, our email newsletter!
Practical advice for effective online engagement
http://www.onenw.org/full_signup