Hi, I'm looking at a HA setup for a project, and was wondering what the current best way forward would be. There seem to be a few potential options around for the storage end of the setup, and I'm wondering what's now considered the current "best practice" for this sort of setup. Specifically, I've come up with three options: 1. RelStorage Using this, I think I can then take care of replication/mirroring as I have access to a database that is already clustered in a HA environment. My questions are: + Are the connections opened only when zope is started? Say I unplugged a network cable and then plugged it back in again (breaking the database connection) - will it be re-opened? + How does RelStorage take care of the blob storage? + Are there any details of big sites out there that use RelStorage (particularly on Oracle)? 2. ZeoRAID I could use ZeoRAID to write to maintain several independent storages subject to stopping the ZeoRAID server being the single point of failure (I posted some questions about this separately). 3. ZRS If I'm right, ZRS still has a limitation in that there is a single server for writes. Also, budget may be an issue. Having read the factsheet, I'm a bit unsure as to how it is functionally different from ZeoRAID - can anyone explain? I also thought of using DirectoryStorage to take care of replication/mirroring at the hardware level, for example using a clustered SAN, but I think this would not work because the actual directory tree can only be writeable by one zope at a time. Am I right about this? Is there anything else worth considering? I'd be grateful if anyone operating a HA-setup for their zope site, could post your setup and experiences? To me, out of the above options, RelStorage looks to be the most promising, simply because it will involve the least setup and the database clustering and tools are pretty robust and well-understood by the sql experts here. Thanks in advance for your help, I will post a write-up once it's done, Miles
Is there anything else worth considering?
Dedicated HA-hardware. We are using a SUN Cluster for all of our backend service like ZEO, Oracle, Postgres and LDAP.
I'm interested in how that works specifically with zeo. how does it ensure the clustered Zeos don't try to write conflicting transactions at the same time? Or is it set up for another service to start on failover, rather than a load-balanced arrangement? Thanks Miles
Am Wed, 03 Jun 2009 00:04:12 +0100 schrieb Miles Waller:
Hi,
I'm looking at a HA setup for a project, and was wondering what the current best way forward would be. There seem to be a few potential options around for the storage end of the setup, and I'm wondering what's now considered the current "best practice" for this sort of setup. [...] Is there anything else worth considering?
the classics: DRBD and Heartbeat (for the zeo-server). regards Jensens -- Jens W. Klein - Klein & Partner KEG - BlueDynamics Alliance
Previously Miles Waller wrote:
Hi,
I'm looking at a HA setup for a project, and was wondering what the current best way forward would be. There seem to be a few potential options around for the storage end of the setup, and I'm wondering what's now considered the current "best practice" for this sort of setup. Specifically, I've come up with three options:
1. RelStorage Using this, I think I can then take care of replication/mirroring as I have access to a database that is already clustered in a HA environment. My questions are: + Are the connections opened only when zope is started? Say I unplugged a network cable and then plugged it back in again (breaking the database connection) - will it be re-opened? + How does RelStorage take care of the blob storage? + Are there any details of big sites out there that use RelStorage (particularly on Oracle)?
The initial RelStorage development was sponsored by Jarn and Elkjøp for exactly that purpose: Elkjøp (a chain of Scandinavian electronics stores) uses an Oracle cluster for storage of all their Zope data.
2. ZeoRAID I could use ZeoRAID to write to maintain several independent storages subject to stopping the ZeoRAID server being the single point of failure (I posted some questions about this separately).
If I remember correctly the ZeoRAID server is stateless, so you can use a standard redundant setup to remove the single point of failure.
3. ZRS If I'm right, ZRS still has a limitation in that there is a single server for writes. Also, budget may be an issue. Having read the factsheet, I'm a bit unsure as to how it is functionally different from ZeoRAID - can anyone explain?
From what I hear ZRS is also fairly expensive, especially compared to the other two options which are free.
Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Wichert Akkerman wrote:
Previously Miles Waller wrote:
1. RelStorage Using this, I think I can then take care of replication/mirroring as I have access to a database that is already clustered in a HA environment. My questions are: + Are the connections opened only when zope is started? Say I unplugged a network cable and then plugged it back in again (breaking the database connection) - will it be re-opened?
Connections will be re-opened should they be lost for any reason. I remember only one instance of non-recoverable connection loss in an entire year now. This was caused by maintenance tasks on the Oracle cluster. The changes in 1.1.3 might have fixed even this rare problem.
+ How does RelStorage take care of the blob storage?
RelStorage has no special blob storage support. You can wrap a file-system based blob storage around a RelStorage. We don't use blobs for this project at Jarn but store all information directly in Oracle. Depending on the kind of data you need to store this can be inefficient.
+ Are there any details of big sites out there that use RelStorage (particularly on Oracle)?
The initial RelStorage development was sponsored by Jarn and Elkjøp for exactly that purpose: Elkjøp (a chain of Scandinavian electronics stores) uses an Oracle cluster for storage of all their Zope data.
Exported to a regular Data.fs the data size is about 20gb. We have 20 Zope clients connected to the storage in the setup. The only problem we found with this setup is that the Oracle Cluster has quite a high latency, so reading and writing lots of objects to the database can be slow. We aren't using the "cache-servers" option yet, which should help with this specific problem. One additional advice: You want to increase your temporary table space to a much larger value. If you pack the database, the tempspace should be able to contain your entire data set. If you decide for RelStorage, Shane offers commercial support and consultancy for it, which might be worth considering depending on the criticality of the application. Hanno
On Wed, Jun 3, 2009 at 01:04, Miles Waller <miles@jamkit.com> wrote:
1. RelStorage Using this, I think I can then take care of replication/mirroring as I have access to a database that is already clustered in a HA environment. My questions are: + Are the connections opened only when zope is started? Say I unplugged a network cable and then plugged it back in again (breaking the database connection) - will it be re-opened? + How does RelStorage take care of the blob storage?
Hanno answered these 2 already.
+ Are there any details of big sites out there that use RelStorage (particularly on Oracle)?
The Elkjøp intranet cluster currently runs on 4 machines with 4 clients each, plus a maintenance client on a 5th machine. Varnish and fail-over rigged load balancers complete the cluster. Currently there are 100k+ content objects and 7k active users using this intranet (plus another 3k inactive). -- Martijn Pieters
On Jun 2, 2009, at 7:04 PM, Miles Waller wrote: ...
3. ZRS If I'm right, ZRS still has a limitation in that there is a single server for writes.
Yup. ...
Having read the factsheet, I'm a bit unsure as to how it is functionally different from ZeoRAID - can anyone explain?
I'm not very familiar with ZEORaid, but, I think ZEORaid replicates during transactions. That is, it writes to each replica at the same time it writes to the main storage. In ZRS, replication is asynchronous from writes. This means that transactions only have to wait for writes to be completed on the main storage. ZRS also support chaining replication, meaning a secondary can replicate from another secondary. For example, we typically have off- site secondaries that replicate from an on-site secondary. Note that ZRS has been used for years in large-scale critical environments. Also note that ZRS 2, which we've been using in production for more than 2 years doesn't use spread and is much easier to configure and more flexible than earlier versions. See http://www.zope.com/products/zope_replication_services.html for more information. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
Note that ZRS has been used for years in large-scale critical environments.
Also note that ZRS 2, which we've been using in production for more than 2 years doesn't use spread and is much easier to configure and more flexible than earlier versions.
See http://www.zope.com/products/zope_replication_services.html for more information.
It's a shame that the 2 or 3 times I've tried to organise purchase of ZRS for customers, the Zope Corp sales process hasn't succeeded in delivering anything :-( (I'm not 100% on the details, but it may just have been that the prices were extortionate...) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
It's a shame that the 2 or 3 times I've tried to organise purchase of ZRS for customers, the Zope Corp sales process hasn't succeeded in delivering anything :-(
(I'm not 100% on the details, but it may just have been that the prices were extortionate...)
I disagree. The kind of fault tolerance ZRS provides is hard to achieve without a lot of either time or money. I suspect ZRS is less expensive than a new replicated Oracle setup, or paying a developer to invent something new. (I only recommend Oracle + RelStorage to people who have already invested deeply in Oracle.) Shane
Shane Hathaway wrote:
Chris Withers wrote:
It's a shame that the 2 or 3 times I've tried to organise purchase of ZRS for customers, the Zope Corp sales process hasn't succeeded in delivering anything :-(
(I'm not 100% on the details, but it may just have been that the prices were extortionate...)
I disagree.
You disagree that I failed to purchase ZRS from Zope Corp both times I tried? ;-)
The kind of fault tolerance ZRS provides is hard to achieve without a lot of either time or money.
Indeed, but when it's easier to get two hearbeat-failover'd servers and some SAN to put the storage on, or go for clustered-MySQL-as-a-service and RelStorage or ZEORAID, it's a bit of a moot point...
I suspect ZRS is less expensive than a new replicated Oracle setup, or paying a developer to invent something new.
I'm not necessarily disagreeing with you, I wish getting hold of ZRS was easier (not necessarily cheaper!)... Chris
participants (9)
-
Andreas Jung -
Chris Withers -
Hanno Schlichting -
Jens W. Klein -
Jim Fulton -
Martijn Pieters -
Miles Waller -
Shane Hathaway -
Wichert Akkerman