On Mon, 02 Jul 2001 13:58:23 +0100 you wrote:

Do you have a plan for replicating data onto the second ZEO server?

Yeah, but it's not pretty.

We are looking to implement a similar system and there seems to be no obviously good choice.

That's what I determined about a week ago. I was sad to see that work on Replicated Storage had been halted, but I'm not sure that it was the ideal solution for my needs. Then I had a little problem and became more intimate with the Data.fs file. Again, this is *not* pretty... Right now, our Zope server moves around (It even uses DHCP sometimes.) and our HTTPS server stays in one place. They're even in different buildings. I use the HTTPS server to convert from HTTPS to HTTP and to handle some other things like automatic selection of the source port (for WebDAV things). When my Zope host comes up, it sets up an SSH session to the HTTPS server. As part of the session, it makes tunnels for the ZopE ports. Now I also have it run a command. It's essentially this. tail -f -n +0 Data.fs | ssh proxyhost "cat >somepath/Data.fs" (The real server name, account and tunnel info are all in the SSH config. Instead of "Data.fs", for now I use a filename that has the current date in it. That gives me multiple versions.) This gives me an up-to-the-second copy of the database. If the building with the Zope server is suddenly destroyed, I should not lose anyuthing. My plan is to do something similar between two ZEO servers. One will run and stream its updates to the other. If the primary server fails, I will switch to the other one (by notifying the HTTPS proxy or by twiddling the SSH sessions). Without any modifications, the secondary server would have to start Zope and read through the Data.fs before becoming active. That's quite a delay for us. My plan is to modify the Zope startup so that it does something similar to a "tail -f" on Data.fs so that it can build its internal database as it goes along (mirroring the primary server). I would then send a signal to it to halt reading (when it reaches EOF) and start serving. I might be able to get away with *not* modifying the Zope startup by using a FIFO. I would either write directly to it from my SSH process or use tail to duplicate the streamed file. When the primary server dies, I'll kill whatever is writing to the FIFO (thus closing the write end) and Zope should discontinue reading and start serving. Of course, this is going to require some manual intervention before the primary server is made available again. For now, I am happy to have it killed and forgotten until someone puts everything right. If my primary and secondary ZEO servers are identical (in terms of performance), I'll probably just flip-flop them, and call the running server the primary. After I fix the dead server, I'll replace its Data.fs file with a stream of the running server's file. It will then be ready to go as the fail-over server. Note that although I am streaming the Data.fs file now, everything else I'm planning is untested. I'm hoping to get to it this week, but I've got some other things in the way. --kyler

On Mon, 2 Jul 2001 15:12:10 -0700 (PDT) you wrote:

...

That's what I determined about a week ago. I was sad to see that work on Replicated Storage had been halted, but I'm not sure that it was the ideal solution for my needs.

I don't think it's permenantly halted, the main project leader is on paternity leave.

I shouldn't have said "halted". It's waiting for an interrupt that could be awhile. http://www.zope.org//Wikis/DevSite/Projects/ZEOReplicatedStorage/CurrentStat... Work on ReplicatedStorage will be delayed until work on ZEO 1.0 and ZEO 2.0 is finished. ZEO 2.0 will be the basis for ReplicatedStorage. --kyler

On Tue, 03 Jul 2001 09:49:34 +0100 you wrote:

...

tail -f -n +0 Data.fs | ssh proxyhost "cat >somepath/Data.fs"

Thats a low-cost trick I hadnt thought of before, however there is still a small time in between: a) data being synced onto the main server, and the transaction succeeding b) data being written to the backup server.

Data can be lost if your main machine explodes at that point.

Certainly. There's also time between when data is written to a buffer and when it ends up on the local disk. I can live with it. It's *much* better than just hoping that nothing goes wrong and making periodic backups.

Another disadvantage to this solution; current versions of FileStore are not purely loggging; *undo* will write a single byte in the middle of the database file, which will corrupt your replica.

Ack! I didn't realize this. (Undo usually doesn't work for me, but I would like to use it.)

A recent change to FileStorage handles undo a different way, which I guess that might fix this problem... Ill ask over on the zodb-dev list.

Thank you. I am quite interested in this. (It reminds me of the undo in Berkeley Storage.)

...

If you are running the ZSS on Linux, you can use a network block device setup that establishes a raid1 over the netwrok, and have it do a split read/write such that it only reads from the primary, but writes over the raid.

Have you tried this with FileStorage? Testing this configuration is approaching the top of my to-do list. I am a little worried that it may not be entirely stable in high-write usage, such as during a pack. (although at the moment that is just unsubstantiated FUD)

NBD could cause performance problems, but in theory it should work. (It's pretty cool.) It should have very little network overhead beyond that of the "tail" method, but it would support writing to random locations within the file. It would not be a simple cross-platform solution. In the most basic usage, it would also be blocking; it would slow down the server.

If the network block device doesnt work out, I am considering patching FileStorage to distribute writes across the network, and only allow the transaction commit to complete once data is consistent across multiple remote mirrors.

This would be great, and I do hope that you do it. I'm still interested in a simple method that does not force the primary server to wait for updates to be made. If someone uploads a 1GB PDF file to my primary server, I'd rather not wait while every backup is made. I wouldn't mind it for my on-site hosts, but I'm willing to sacrifice some transactions in a (rare) failure than have them always delayed.

Of course replicated-ZEO would solve this and many other problems, but I am hoping that by concentrating on just this one use case I can develop something useful, sooner.

I appreciate your efforts.

http://www.zope.org/Wikis/DevSite/Projects/ZEOReplicatedStorage/SurviveTotal...

I take it there would be willing beta testers for such a system?

Count on me. --kyler

This seems more germane to zodb-dev. cc'ed there for further replies to go there. On 03 Jul 2001 07:21:09 -0500, Kyler B. Laird wrote:

...

...

If you are running the ZSS on Linux, you can use a network block device setup that establishes a raid1 over the netwrok, and have it do a split read/write such that it only reads from the primary, but writes over the raid.

Have you tried this with FileStorage? Testing this configuration is approaching the top of my to-do list. I am a little worried that it may not be entirely stable in high-write usage, such as during a pack. (although at the moment that is just unsubstantiated FUD)

Testing with Filestorage really is just icing. The process _does_ work, I know people using it for always-on hot-backup. it occurs at the OS level, and as far as FS is concerned, it just looks at a file on the fs. it has no clue that the fs is actually running on a RAID device, let alone across a network.

NBD could cause performance problems, but in theory it should work. (It's pretty cool.) It should have very little network overhead beyond that of the "tail" method, but it would support writing to random locations within the file.

It would not be a simple cross-platform solution.

Thus the 'if on Linux remark', but then again, remember this is only for the ZSS, and only until ZEO Replication is accomplished. The ZEO Clients can stil be on anything ZEO runs on.

In the most basic usage, it would also be blocking; it would slow down the server.

How do you figure this? From the server's standpoint, it is just writing to a file on a filesystem. Writes do go out on the network, but they occur simultaneously. Reads are limited to 'local', so there is no performance loss there. I don't know where you get this 'blocking' bit. Writes may suffer slight performance drop in writing speed, but will not increase the 'blockiness' of ZSS. Any replication method that occurs in real time, or even damned-near-real time will cause this to happen.

...

If the network block device doesnt work out, I am considering patching FileStorage to distribute writes across the network, and only allow the transaction commit to complete once data is consistent across multiple remote mirrors.

This would be great, and I do hope that you do it.

I'm still interested in a simple method that does not force the primary server to wait for updates to be made. If someone uploads a 1GB PDF file to my primary server, I'd rather not wait while every backup is made. I wouldn't mind it for my on-site hosts, but I'm willing to sacrifice some transactions in a (rare) failure than have them always delayed.

Hmmm, seems you may be misunderstanding ZEO and backing up/replicating of the ZODB. If someone uploads a 1gb PDF, you _will_ be waiting a while for that update to span networks in replicated ZSS, just as you will be waiting for that PDF to be uploaded in the first place. Your ZEO Clients will only be waiting on the server to receive the file. So, with a single ZSS, and 5 ZEO clients (let's pretend they are zope servers), the file is uploaded once. now, when a ZEO Client receives a request, it will indeed request the data from the server, and ship it to the client. There is possibly, depending on network bandwidth, a delay, but unles your the end-user browser is on a network connection to the zope server that is as fast or faster than the ZSS->Zope connection, they will not notice a difference. I don't think ZEO was meant for such large file transfers anyway, but I could be wrong. In any event, distributing large files across a network will always cost at a minimum, network delays, regardless of the method used. if there are network delays between primary and any secondaries, and the relay is not done 'offline' (that is to say, not in full sync with the primary), there will be on the primary. The reason is that if you are, in your example, uploading a 1Gb file, and ZEO is replicating that in sync, it has to monitor the progress, in the event that a) network data corruption occurs, or b) user cancels the upload (or it times out, or similar activities/events), at which point it needs to cancel the transaction across primaries. In NBD, you would only be waiting once, as the writes occur in unison. In replicated ZSS, I cannot say, as I don't know much of the details (does anyone ;") ? Bill

On Mon, 2 Jul 2001, Toby Dickenson wrote:

On Mon, 02 Jul 2001 07:14:43 -0500, "Kyler B. Laird" <laird@ecn.purdue.edu> wrote:

Sorry, Ive no suggestions for your problem, but...

...

We're planning to split things up into a cluster that consists of multiple HTTPS proxies, some ZEO clients, a ZEO server and a hot ZEO server spare.

Do you have a plan for replicating data onto the second ZEO server? We are looking to implement a similar system and there seems to be no obviously good choice.

have you looked at Coda or Intermezzo? Both distributed file systems, although I've used niether. -Michel