[Zope] zope backups

[Paul Winkler]

On Tue, May 25, 2004 at 06:40:01AM +0100, Ben Last (Zope) wrote:
> > [mailto:zope-bounces+zope=benlast.com at zope.org]On Behalf Of Paul Winkler
> > Zope only appends to Data.fs.  (It has been mentioned that
> > it is *possible* to twiddle stuff in the middle of a filestorage,
> > but the core Zope code is not supposed to do this, and I've never
> > heard an example of any add-on product that does so either.)
>
> Out of curiosity, what then prevents Data.fs growing without bound?  Does it
> get truncated from the beginning?

No, never.  Instead, we "pack" it - a process which deletes
transactions older than a given date. I've never really looked into
how packing is implemented for the default storage, but
I know that it makes a backup copy (Data.fs.old), tells Zope to 
write new transactions since the pack started to Data.fs.tmp, and
finally when done with its work appends Data.fs.tmp to the new Data.fs.

If you never pack, your Data.fs will indeed only ever get larger...
OTOH, be sure you do your homework and know what version of Zope
you have and whether it has known pack bugs.  I know that 2.6.1-2.6.2 
had pack bugs and could lose data during a pack.
To the best of my knowledge, 2.6.4 and 2.7.0 do not have this bug.

> Even if it does only get appended to, I assume there's a race condition
> where cp may encounter Data.fs with half or a record written to it and thus
> copy an inconsistent file.

Not a problem. Due to the ZODB's transactional nature, all this means 
is that the last transaction in the backup is incomplete and invalid
and Zope will ignore it.

If we number transactions with consecutive integers, then a Data.fs
with complete transactions 0..N and incomplete transaction N+1
is functionally equivalent to a Data.fs that ends with complete
transaction N.

-- 

Paul Winkler
http://www.slinkp.com