[ZODB-Dev] [ zodb-Bugs-953518 ] repozo.py may include dat file in recovering ZODB file

[SourceForge.net]

Bugs item #953518, was opened at 2004-05-13 19:03
Message generated for change (Comment added) made by henderj
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=115628&aid=953518&group_id=15628

Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: James Henderson (henderj)
Assigned to: Nobody/Anonymous (nobody)
Summary: repozo.py may include dat file in recovering ZODB file

Initial Comment:
I have an identical repository on two different machines.  On one 
repozo.py -R works fine; on the other it includes a .dat file and 
produces a corrupt ZODB file:

[THIS IS CORRECT]
[zope at www2 zope]$ repozo.py -v -r backup -Ro recovered.fs
looking for files b/w last full backup and 2004-05-13-17-20-50...
files needed to recover state as of 2004-05-13-17-20-50:
        backup/2004-05-13-15-03-23.fs
        backup/2004-05-13-15-07-48.deltafs
        backup/2004-05-13-15-44-06.deltafs
        backup/2004-05-13-15-48-10.deltafs
Recovering file to recovered.fs
Recovered 561743152 bytes, md5: 
6cb9f177aab9cd09fcd7792076cdcc76

[THIS INCORRECTLY INCLUDE .dat FILE]
[zope at localhost zope]$ repozo.py -v -r backup -Ro recovered.fs
looking for files b/w last full backup and 2004-05-13-17-34-24...
files needed to recover state as of 2004-05-13-17-34-24:
        backup/2004-05-13-15-03-23.fs
        backup/2004-05-13-15-03-23.dat
        backup/2004-05-13-15-07-48.deltafs
        backup/2004-05-13-15-44-06.deltafs
        backup/2004-05-13-15-48-10.deltafs
Recovering file to recovered.fs
Recovered 561743535 bytes, md5: 
85508f41b0025454b92b5fb276ca43a4

It turns out that the find_files() function can return different lists on 
different machine.  This is because os.listdir() lists file in arbitrary 
order (as per the documentation) and then the relative order of the .fs 
file and its .dat file (which share the same basename) is preserved by 
the Python sort.  The workings of find_files() are illustrated 
interactively below:

[.fs LISTED BEFORE .dat]
[zope at www2 zope]$ python
Python 2.3.3 (#2, Mar 15 2004, 10:16:17)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, pprint
>>> def rootcmp(x, y):
...     return cmp(os.path.splitext(y)[0], os.path.splitext(x)[0])
...
>>> all = os.listdir('backup')
>>> pprint.pprint(all)
['2004-05-13-15-03-23.fs',
 '2004-05-13-15-03-23.dat',
 '2004-05-13-15-07-48.deltafs',
 '2004-05-13-15-44-06.deltafs',
 '2004-05-13-15-48-10.deltafs']
>>> all.sort(rootcmp)
>>> pprint.pprint(all)
['2004-05-13-15-48-10.deltafs',
 '2004-05-13-15-44-06.deltafs',
 '2004-05-13-15-07-48.deltafs',
 '2004-05-13-15-03-23.fs',
 '2004-05-13-15-03-23.dat']

[.dat LISTED BEFORE .fs]
[zope at localhost zope]$ python
Python 2.3.3 (#1, Feb 13 2004, 15:14:01)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, pprint
>>> def rootcmp(x, y):
...     return cmp(os.path.splitext(y)[0], os.path.splitext(x)[0])
...
>>> all = os.listdir('backup')
>>> pprint.pprint(all)
['2004-05-13-15-07-48.deltafs',
 '2004-05-13-15-03-23.dat',
 '2004-05-13-15-03-23.fs',
 '2004-05-13-15-44-06.deltafs',
 '2004-05-13-15-48-10.deltafs']
>>> all.sort(rootcmp)
>>> pprint.pprint(all)
['2004-05-13-15-48-10.deltafs',
 '2004-05-13-15-44-06.deltafs',
 '2004-05-13-15-07-48.deltafs',
 '2004-05-13-15-03-23.dat',
 '2004-05-13-15-03-23.fs']

I attach a patch to fix this.

----------------------------------------------------------------------

>Comment By: James Henderson (henderj)
Date: 2004-05-13 19:05

Message:
Logged In: YES 
user_id=687101

Oops. here's the patch.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=115628&aid=953518&group_id=15628