[ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?

[Andreas Jung]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jim Fulton wrote:
> On Wed, Feb 3, 2010 at 3:39 AM, Andreas Jung <lists at zopyx.com>
> wrote:
>> we had in the past several issues with hour-long outages of our
>> ZEO servers. The reasons are pretty easy: with have several big
>> storages (60-80GB) and in case of a cluster failures we were not
>> able to shutdown the ZEO servers in a clean way.
>
> Note that we had a similar problem until we realized that zdaemon
> gets impatient and sends a KILL signal to a process shutting down
> after 10 seconds.  We have since set the zdaemon backoff-limit to
> 300 which gives our servers plenty of time to shut down cleanly.
oh - that's interesting
>
> Also, note that yesterday, I checked in a change to file-storage
> index saving logic and file format that makes index saving and
> loading *much* faster.
>
Would it be easy to backport this to ZODB (3.8) (we could eventually
handle this)?
>> So at startup time the complete index files had to be recreated
>> which takes pretty long for our storages.
>>
>> The question came up if it would be possible to write regular
>> snapshots of the in-memory index back to disk (e.g. every 15
>> minutes) and then to modify the ZEO startup procedure to
>> optionally start a ZEO server with such a snapshoted index file?
>
> What is the difference between an index and a snapshot?
>
>
In our understanding the in-memory index is written back to disk only
at ZEO shutdown time (proper closing of the Filestorage). A snapshot
would be written to disk regularly while the Filestorage is open.
>> This may cause some data loss (transactions between the last
>> snapshot and the crash would be ignored - or perhaps the ZEO
>> server could take the snapshot file and read all transactions
>> happened after the snapshot until the crash in order to update
>> the index with the completed transactions after the snapshot...
>
> That's what happens when reading indexes now. Of course, an index
> *is* just a snapshot.

Sure but ZEO/Filestorage does not written the in-memory index
as snaphot to disk - except at shutdown - or?
>> Just some ideas...would that be doable?
>
> Yes, FileStorage actually has some logic now to save indexes
> periodically. Unfortunately, the algorithm is flawed and almost
> never fires.
Aha - this explains the experience discussed above. But this information
about peridically savings is actually new to me (and perhaps many
other people).

>
> Another issue is that, as things are now, while an index is being
> saved, no transactions can be committed.  This is less serious now
> that saving indexes is much faster, however, saving a large index
> may still take several seconds, and that might be a problem for
> some applications.
>
> I'd be happy to add an option to save indexes after every N
> records written, where N would be given by the option.
So what is the difference between such a new option and fixing the
"flawed"
algorithm as mentioned above?

Andreas


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktqZV8ACgkQCJIWIbr9KYxRzQCff5/ZTVOXHx/msZS5VB+JT66L
6S8AnjLAYOlju2bIHFDnST4rOtLqd4eK
=ggTh
-----END PGP SIGNATURE-----

-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 316 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20100204/bee9c64e/attachment.vcf