[Zope-dev] Re: Help! Need to extract files out of Data.fs!

[Philipp von Weitershausen]

Kris Adcock wrote:
> Hiya,
> 
> My company has been using a Zope server to store some reference material, but
> it has grown beyond belief, and I need to extract all the collected files
> (about 1.3 terabytes!) from the Data.fs into distinct files onto a different
> server.
> 
> Unfortunately, I've run into some problems. Firstly, the FTP transfer process
> seems to be extremely slow, and if I try and transfer too large or deep a
> folder structure in one go then things grind to a halt.
> 
> To beat this I've been carefully recreating the folder structure manually,
> and copying things in easier-to-digest chunks. Unfortunately, because I was a
> bit rushed over the weekend I have accidentally deleted about 130,000 files!
> 
> I've tried going through the "Undo" procedure, but the idea of manually
> ticking 130,000 boxes (in pages of 20) and undoing them fills me with dread.

Hmm, this is something I don't understand. You deleted 130,000 files in
individual transactions? If you deleted the 130,000 files in one big
transaction, all you have to do is undo this one transaction. That is,
of course, provided that any subsequent transaction you made after
deleting the files won't cause inconsistencies and potentially prevent
the undo.

> Please please /please/ .... does anyone happen to have the file format for
> the Data.fs file so that I could write a C program to extract and recreate
> the files and folders to another server? Or has anyone written such a program
> already? If so, I'll happily have your children if you can help me out! As it
> stands, I'm going to have to break it to the librarian (the poor man who has
> been collating all the reference material for the last six months) that I've
> lost half of his work, and blaming it on the poor FTP client in Windows seems
> a bit lame!

No need for a C program. The ZODB is a Python library, you can use
Python to get at your data. The simplest way is to start the Zope debug
shell via bin/zopectl debug. You will then have access to the root
object in the interpreter prompt and can poke at your objects, including
their history. Writing a script and running it via bin/zopectl run
shouldn't be a big problem after that.

Philipp