[ZODB-Dev] Is there an official patch for the problems with packing and data loss?

[Tim Peters]

[Joachim Werner]
> If a database has a bug that makes it lose data most people would call
> it "unusable" ...

For the most part, only the ones who have been burned by it.  There was
recently an amazing debate on comp.lang.python, concerning Python 2.3's
attempt to get away from the *known* to be highly buggy Berkeley 1.85
database code (shipped with Windows Python, and apparently still the default
on some Unixish distributions).  The short course is that some people have
never bumped into a 1.85 bug (that they know of), so fiercely resist any
change.  Other people have an endless string of 1.85 horror stories, of
course.

> Most people back up before packing, but you can't ask people to rely on
> manually recovering data from the backups if the pack went wrong (and
> that seems to be the only way of handling lost data WRT the 2.6.1 ZODB
> bugs). One of the problems is size. If you have large ZODBs (>1 Gig)
> even untaring takes a lot of time ...

I don't have easy answers here.  One hard truth is that bugs exist, and I'm
afraid that isn't going to change.  It would probably help if more people
took an interest in the implementations of these subsystems and did serious
code review and testing (Zope Corp's resources are limited too, of course,
and the "many eyeballs" part of the Open Source story doesn't seem to be
working in these components).

> The bugs might have been there before. But we definitely didn't have
> data loss before 2.5.x (with exactly the same setup and data) ...

If so, I bet a reason could be found if someone had sufficient time and
interest to dig into it to the bitter end.  For example, it could be that
bugfixes outside of FileStorage triggered patterns of database activity that
simply didn't happen before.  I don't know, but in the limited time I have
for it am more interested in repairing the bugs that were uncovered (I'm
glad they never burned you before, but not so glad they were nevertheless
there to burn others).