[ZODB-Dev] [ZEO] Dealing with long startup times with out-dated index files/Index snapshots?

[Jim Fulton]

On Thu, Feb 4, 2010 at 1:12 AM, Andreas Jung <lists at zopyx.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jim Fulton wrote:
>> On Wed, Feb 3, 2010 at 3:39 AM, Andreas Jung <lists at zopyx.com>
>> wrote:
...
>> Also, note that yesterday, I checked in a change to file-storage
>> index saving logic and file format that makes index saving and
>> loading *much* faster.
>>
> Would it be easy to backport this to ZODB (3.8) (we could eventually
> handle this)?

It would be easy to backport it as a local patch.  I wouldn't want to release
it in 3.8, as it changes the index format.  I wouldn't want a situation where
3.9 releases couldn't read indexes created with 3.8.

>>> So at startup time the complete index files had to be recreated
>>> which takes pretty long for our storages.
>>>
>>> The question came up if it would be possible to write regular
>>> snapshots of the in-memory index back to disk (e.g. every 15
>>> minutes) and then to modify the ZEO startup procedure to
>>> optionally start a ZEO server with such a snapshoted index file?
>>
>> What is the difference between an index and a snapshot?
>>
>>
> In our understanding the in-memory index is written back to disk only
> at ZEO shutdown time (proper closing of the Filestorage). A snapshot
> would be written to disk regularly while the Filestorage is open.

The index is also written when the database is packed. The only limitation
to writing the index while the file storage is open is that we can't commit
transactions while writing the index.

...

>> Yes, FileStorage actually has some logic now to save indexes
>> periodically. Unfortunately, the algorithm is flawed and almost
>> never fires.
> Aha - this explains the experience discussed above. But this information
> about peridically savings is actually new to me (and perhaps many
> other people).

The information is moot, since the algorithm never fires. :)

>> Another issue is that, as things are now, while an index is being
>> saved, no transactions can be committed.  This is less serious now
>> that saving indexes is much faster, however, saving a large index
>> may still take several seconds, and that might be a problem for
>> some applications.
>>
>> I'd be happy to add an option to save indexes after every N
>> records written, where N would be given by the option.
> So what is the difference between such a new option and fixing the
> "flawed"
> algorithm as mentioned above?

There would be explicit control. You's be able to say how often (in
terms of records written) you'd save indexes.

Jim

-- 
Jim Fulton