Fail-safe redundancy policies for Zope
What are the best, or most used fail-safe redundancy practices suitable to Zope? More in detail, I'm working on a Zope-based system that will store the data inside the ZODB, and one of the design requests is that given an unrecoverable hardware crash, the system could continue running using a mirror installation with the latest backup (not later than half an hour or even less.) APE looks like the way to go since it allows the use of standard backup software, but you may have tried other possible solutions for a problem like this. According to your experience how would you accomplish it? Thanks in advance, Ausum
On Wed, Oct 06, 2004 at 02:35:50PM -0500, Ausum Studio wrote:
What are the best, or most used fail-safe redundancy practices suitable to Zope? More in detail, I'm working on a Zope-based system that will store the data inside the ZODB, and one of the design requests is that given an unrecoverable hardware crash, the system could continue running using a mirror installation with the latest backup (not later than half an hour or even less.)
APE looks like the way to go since it allows the use of standard backup software, but you may have tried other possible solutions for a problem like this. According to your experience how would you accomplish it?
I haven't actually implemented either of these, but: You could spend some $$ and get ZRS from zope corp which IIRC might let you do automatic failover without loss of service. Not sure though. You could also use DirectoryStorage and frequently replicate to a backup storage server. The backup can't actually serve to ZEO clients as long as you're replicating to it, but you might be able to come up with some scripts that automatically start a live storage process on the backup if the primary fails. Reverting to the primary server might have to be a largely manual process. -- Paul Winkler http://www.slinkp.com
Paul Winkler wrote:
On Wed, Oct 06, 2004 at 02:35:50PM -0500, Ausum Studio wrote:
What are the best, or most used fail-safe redundancy practices suitable to Zope? More in detail, I'm working on a Zope-based system that will store the data inside the ZODB, and one of the design requests is that given an unrecoverable hardware crash, the system could continue running using a mirror installation with the latest backup (not later than half an hour or even less.)
APE looks like the way to go since it allows the use of standard backup software, but you may have tried other possible solutions for a problem like this. According to your experience how would you accomplish it?
I haven't actually implemented either of these, but:
You could spend some $$ and get ZRS from zope corp which IIRC might let you do automatic failover without loss of service. Not sure though.
For full QoS, ZRS out of the box would only support "warm" failover (it would require downing the secondary and bringing it back up as a primary). We have experimented with using 'heartbeat' to automate that, but haven't deployed it in production. If having the appservers degrade to read-only mode on primary failure is a sufficient QoS, ZRS does that automatically.
You could also use DirectoryStorage and frequently replicate to a backup storage server. The backup can't actually serve to ZEO clients as long as you're replicating to it, but you might be able to come up with some scripts that automatically start a live storage process on the backup if the primary fails. Reverting to the primary server might have to be a largely manual process.
Right. 'heartbeat' might help there, too. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
Paul, Tres, thanks for your responses. Has anyone in the community tried any other non-standard way to accomplish a live backup, like to rely on a synchronization proccess auto-triggered by parallel installations?. I'd like to go this way as the first thing to try , so now I'm looking for caveats from people who have alredy tried the Zsyncer way or the likes. :) Ausum ----- Original Message ----- From: "Tres Seaver" <tseaver@zope.com> To: <zope@zope.org> Sent: Wednesday, October 06, 2004 4:54 PM Subject: [Zope] Re: Fail-safe redundancy policies for Zope
Paul Winkler wrote:
On Wed, Oct 06, 2004 at 02:35:50PM -0500, Ausum Studio wrote:
What are the best, or most used fail-safe redundancy practices suitable to Zope? More in detail, I'm working on a Zope-based system that will store the data inside the ZODB, and one of the design requests is that given an unrecoverable hardware crash, the system could continue running using a mirror installation with the latest backup (not later than half an hour or even less.)
APE looks like the way to go since it allows the use of standard backup software, but you may have tried other possible solutions for a problem like this. According to your experience how would you accomplish it?
I haven't actually implemented either of these, but:
You could spend some $$ and get ZRS from zope corp which IIRC might let you do automatic failover without loss of service. Not sure though.
For full QoS, ZRS out of the box would only support "warm" failover (it would require downing the secondary and bringing it back up as a primary). We have experimented with using 'heartbeat' to automate that, but haven't deployed it in production.
If having the appservers degrade to read-only mode on primary failure is a sufficient QoS, ZRS does that automatically.
You could also use DirectoryStorage and frequently replicate to a backup storage server. The backup can't actually serve to ZEO clients as long as you're replicating to it, but you might be able to come up with some scripts that automatically start a live storage process on the backup if the primary fails. Reverting to the primary server might have to be a largely manual process.
Right. 'heartbeat' might help there, too.
Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Thu, Oct 07, 2004 at 08:16:33AM -0500, Ausum Studio wrote:
Paul, Tres, thanks for your responses.
Has anyone in the community tried any other non-standard way to accomplish a live backup, like to rely on a synchronization proccess auto-triggered by parallel installations?. I'd like to go this way as the first thing to try , so now I'm looking for caveats from people who have alredy tried the Zsyncer way or the likes. :)
Well, ZSyncer is an application-layer replication tool rather than a database-layer replication tool. That is, it takes live objects from one Zope and pushes them to another. This means that your two databases are not really identical, they contain the same object data only for the most recent revisions of the particular objects that you sync. Some limitations: *) It's slow for large data *) It eats a lot of RAM for large objects *) It's push-only - you cannot pull from the remote host to the local host. *) It doesn't preserve history *) it fails if you don't have exactly the same Products installed on both systems . *) it uses HTTP and basic auth, so it's up to you to ensure security by using SSL or a VPN tunnel or whatever. *) it frequently mis-reports whether objects are in sync or not (it uses bobobase_modification_time, an inherently flawed approach). But, it is often useful, and in fact I'm working on rolling out another release with a lot of fixes/improvements (and probably new bugs so there'll be a release candidate or two). -- Paul Winkler http://www.slinkp.com
Thanks Paul, for your precise answer. It contains exactly the sort of caveats I was looking for. Ausum ----- Original Message ----- From: "Paul Winkler" <pw_lists@slinkp.com> To: <zope@zope.org> Sent: Thursday, October 07, 2004 10:39 AM Subject: Re: [Zope] Re: Fail-safe redundancy policies for Zope
On Thu, Oct 07, 2004 at 08:16:33AM -0500, Ausum Studio wrote:
Paul, Tres, thanks for your responses.
Has anyone in the community tried any other non-standard way to accomplish a live backup, like to rely on a synchronization proccess auto-triggered by parallel installations?. I'd like to go this way as the first thing to try , so now I'm looking for caveats from people who have alredy tried the Zsyncer way or the likes. :)
Well, ZSyncer is an application-layer replication tool rather than a database-layer replication tool. That is, it takes live objects from one Zope and pushes them to another. This means that your two databases are not really identical, they contain the same object data only for the most recent revisions of the particular objects that you sync.
Some limitations:
*) It's slow for large data *) It eats a lot of RAM for large objects *) It's push-only - you cannot pull from the remote host to the local host. *) It doesn't preserve history *) it fails if you don't have exactly the same Products installed on both systems . *) it uses HTTP and basic auth, so it's up to you to ensure security by using SSL or a VPN tunnel or whatever. *) it frequently mis-reports whether objects are in sync or not (it uses bobobase_modification_time, an inherently flawed approach).
But, it is often useful, and in fact I'm working on rolling out another release with a lot of fixes/improvements (and probably new bugs so there'll be a release candidate or two).
--
Paul Winkler http://www.slinkp.com _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Wednesday 06 October 2004 21:05, Paul Winkler wrote:
You could also use DirectoryStorage and frequently replicate to a backup storage server. The backup can't actually serve to ZEO clients as long as you're replicating to it, but you might be able to come up with some scripts that automatically start a live storage process on the backup if the primary fails.
Reverting to the primary server might have to be a largely manual process.
Maybe, if your backup is read/write. The master maybe contains a few changes made just before the outage, and the backup contains the changes made during the outage. You need to manually merge those (or more likely, choose one set to throw away). If your secondary is read-only, then whatever heatbeat process switched in the backup server should be able to automatically switch it out too. -- Toby Dickenson
Ausum Studio wrote at 2004-10-6 14:35 -0500:
What are the best, or most used fail-safe redundancy practices suitable to Zope? More in detail, I'm working on a Zope-based system that will store the data inside the ZODB, and one of the design requests is that given an unrecoverable hardware crash, the system could continue running using a mirror installation with the latest backup (not later than half an hour or even less.)
We use an (HA) cluster for this. This is a quite expensive solution as all parts (computers, networks, disks) are redundant. -- Dieter
participants (5)
-
Ausum Studio -
Dieter Maurer -
Paul Winkler -
Toby Dickenson -
Tres Seaver