[ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?

[Jim Fulton]

On Wed, Feb 3, 2010 at 3:39 AM, Andreas Jung <lists at zopyx.com> wrote:
> we had in the past several issues with hour-long outages of our
> ZEO servers. The reasons are pretty easy: with have several big
> storages (60-80GB) and in case of a cluster failures we were not able
> to shutdown the ZEO servers in a clean way.

Note that we had a similar problem until we realized that zdaemon
gets impatient and sends a KILL signal to a process shutting down after
10 seconds.  We have since set the zdaemon backoff-limit to 300 which
gives our servers plenty of time to shut down cleanly.

Also, note that yesterday, I checked in a change to file-storage index
saving logic and file format that makes index saving and loading *much*
faster.

> So at startup time the
> complete index files had to be recreated which takes pretty long
> for our storages.
>
> The question came up if it would be possible to write regular snapshots
> of the in-memory index back to disk (e.g. every 15 minutes) and then
> to modify
> the ZEO startup procedure to optionally start a ZEO server with such a
> snapshoted
> index file?

What is the difference between an index and a snapshot?


> This may cause some data loss (transactions between the last
> snapshot and the crash would be ignored - or perhaps the ZEO server
> could take
> the snapshot file and read all transactions happened after the
> snapshot until
> the crash in order to update the index with the completed transactions
> after
> the snapshot...

That's what happens when reading indexes now. Of course, an
index *is* just a snapshot.


> When I talk of ZEO here then this related almost everything to
> FileStorage
> in generell since the zodb index file is a FIleStorage feature.

Yup.

> Just some ideas...would that be doable?

Yes, FileStorage actually has some logic now to save
indexes periodically. Unfortunately, the algorithm is flawed and
almost never fires.

Another issue is that, as things are now, while an index is being
saved, no transactions can be committed.  This is less serious now
that saving indexes is much faster, however, saving a large
index may still take several seconds, and that might be a problem
for some applications.

I'd be happy to add an option to save indexes after every N records
written, where N would be given by the option.

Jim

-- 
Jim Fulton