[ZODB-Dev] extreme clock skew

[Lalo Martins]

ok, now isn't this beautiful.

When upgrading hardware in one of our servers, our admins accidentally
set the clock to the right hour, minute, day and month, but 2015.

We took a few days to notice that our zope timestamps were not showing
up (or sorting) quite as expected. "Oops." They quickly fixed the clock.

Of course, the esteemed ZODB hackers know that this is when the problem
actually *started*.

A few days later, we realized that pack wasn't working anymore (when one
of the sites maxed out the server's hard disk.  Yes, there is some bad
code in there.  Never mind that part.)

For the first time, I introduced myself to the internals of FileStorage
and fspack, and after much careful testing, figured out how to make pack
work again.  Good.

However, I wasn't aware of the fact that timestamps and transaction IDs
are the same and one thing, and never go backwards.  So now (a few weeks
later), we suddenly realized we have our whole database with neatly
uniform timestamps, all nicely contained within the same one-second
interval nearly 10 years in the future.

Needless to say, anything that tries to use bobobase_modification_time
for anything - except maybe sorting - gets nonsense results.  Then
sorting doesn't work either, due to a funny interaction with caching
that I won't explain here to save space.

Finally, in the light of tid = timestamp, I'm not anymore sure my
packing fix even makes sense... it still *seems* to me like it should do
the right thing (only store transactions that are reachable), but I
can't be completely sure, and I can swear the Data.fs is much bigger
than it should be.

Soooo... is there any way to recover this data?  I don't mind losing all
past timestamps (maybe replacing them with current timestamps at
recovery time), as long as zodb goes back to normal from recovery on.

I do, actually, know how to write a script that does this;  what I'm
asking here is if there is already something out there, if anyone
encountered some similar situation before, and if anyone has some words
of wisdom to share in case I actually have to do it myself.

best,
                                               Lalo Martins
--
      So many of our dreams at first seem impossible,
       then they seem improbable, and then, when we
       summon the will, they soon become inevitable.
--
http://www.exoweb.net/                  mailto:lalo at exoweb.net
GNU: never give up freedom                 http://www.gnu.org/