[Zope] Will ZEO solve my problem?

[Kyler B. Laird]

On Tue, 03 Jul 2001 09:49:34 +0100 you wrote:

>>	tail -f -n +0 Data.fs | ssh proxyhost "cat >somepath/Data.fs"
>
>Thats a low-cost trick I hadnt thought of before, however there is
>still a small time in between:
>a) data being synced onto the main server, and
>   the transaction succeeding
>b) data being written to the backup server.
>
>Data can be lost if your main machine explodes at that point.

Certainly.  There's also time between when data is
written to a buffer and when it ends up on the
local disk.  I can live with it.  It's *much*
better than just hoping that nothing goes wrong
and making periodic backups.

>Another disadvantage to this solution; current versions of FileStore
>are not purely loggging; *undo* will write a single byte in the middle
>of the database file, which will corrupt your replica.

Ack!  I didn't realize this.  (Undo usually doesn't
work for me, but I would like to use it.)

>A recent change
>to FileStorage handles undo a different way, which  I guess that might
>fix this problem... Ill ask over on the zodb-dev list.

Thank you.  I am quite interested in this.  (It
reminds me of the undo in Berkeley Storage.)  

>>If you are running the ZSS on Linux, you can use a network block device
>>setup that establishes a raid1 over the netwrok, and have it do a split
>>read/write such that it only reads from the primary, but writes over the
>>raid.
>
>Have you tried this with FileStorage? Testing this configuration is
>approaching the top of my to-do list. I am a little worried that it
>may not be entirely stable in high-write usage, such as during a pack.
>(although at the moment that is just unsubstantiated FUD)

NBD could cause performance problems, but in
theory it should work.  (It's pretty cool.)  It
should have very little network overhead beyond
that of the "tail" method, but it would support
writing to random locations within the file.

It would not be a simple cross-platform solution.
In the most basic usage, it would also be
blocking; it would slow down the server.

>If the network block device doesnt work out, I am considering patching
>FileStorage to distribute writes across the network, and only allow
>the transaction commit to complete once data is consistent across
>multiple remote mirrors.

This would be great, and I do hope that you do
it.

I'm still interested in a simple method that
does not force the primary server to wait for
updates to be made.  If someone uploads a 1GB
PDF file to my primary server, I'd rather not
wait while every backup is made.  I wouldn't
mind it for my on-site hosts, but I'm
willing to sacrifice some transactions in a
(rare) failure than have them always delayed.

>Of course replicated-ZEO would solve this and
>many other problems, but I am hoping that by concentrating on just
>this one use case I can develop something useful, sooner.

I appreciate your efforts.

>http://www.zope.org/Wikis/DevSite/Projects/ZEOReplicatedStorage/SurviveTotalLossOfStorage
>
>I take it there would be willing beta testers for such a system?

Count on me.

--kyler