[ZODB-Dev] HA Setup and zodb index file of FileStorage

[Christian Theune]

Hi,

Patrick Gerken wrote:
> Hello,
> 
> its funny, like Garth in may this year I am looking into making a HA
> system with ZEO for an ERP5 deployment. In my case I don't need to
> care for data replication, all is stored on a SAN considered HA by the
> customer already.
> 
> So my data.fs and index and all that stuff will already be available
> on my backup server.
> The idea is that the backup server will watch the real ZEO, and start
> all services if the server goes down. It seems quite safe for me to
> take over the files from the filesystem and restart everything (While
> writing I realise I should be really really sure the other zeo is
> down, but well, that is out of scope for this mail).
> 
> The thing which scared me to hell was the "rumour" that it can happen
> that the index can get corrupted and then everything has to be indexed
> again. With a large number of objects, this will be slow, either
> because of millions of seeks or many many blocks written to memory. So
> I wanted to look into the FileStorage implementation to see if this
> could be optimised. But I did not find a single place where killing
> the server would result in a corrupt index file (Simply assuming that
> we have journaling filesystems). Also if the index file is not up to
> date, restarting the zeo server would result in updating the zeo
> server starting from the latest transaction written to the index. That
> can take up to 5 minutes (assuming 10000 objects, the distance between
> writes, multiplied with 20 ms for processing and seeking, rounded up
> to the next prime.
> 
> Given that writing a file and renaming a file can be considered
> atomic, and that no solar winds or similar things can screw up my
> filesystem, how can I screw up my index file?

One of the last things I remember is that indexes can not be rebuilt
partially but are rebuilt completely. I had a request on my plate to
modify the index code to take old indexes into account as well. I'm not
sure if anybody else already did that (but that would be available only
on trunk anyway)

Christian

-- 
gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany
www.gocept.com - ct at gocept.com - phone +49 345 122 9889 7 -
fax +49 345 122 9889 1 - zope and plone consulting and development


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 254 bytes
Desc: OpenPGP digital signature
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20060922/2f30a7c1/signature.bin