RE: [Zope] Zope Myths?
I strongly considered this option at one time with a shared DAS/RAID between my primary and secondary ZSS. I never ended up doing this because of concerns. Out of curiosity, with shared storage over SCSI/SAN, do you have any thoughts on how to overcome the following issues? 1 - Data fencing (using SAN switches+integrated software, or STONITH devices, or something)?? 2 - Clean up a broken FileStorage? To what degree can this be successfully automated without manual intervention of a sysadmin? 3 - Deal with accidental damage to the storage done by the primary node, perhaps due to asynchronous writes to the file? It really seems like replication is really a less brittle option (though how much less brittle, I'm not sure). Though not committed at this point, for a big project I am working on, I am strongly considering use of DirectoryStorage and its snapshot capabilities for low-tech replication (via find+cpio+nfs) to try and minimize issues I might face similar to #2 and #3 if I had chosen FileStorage. Sean -----Original Message----- From: Bill Anderson [mailto:bill@libc.org] Sent: Wednesday, September 11, 2002 3:20 PM To: Paul Winkler Cc: zope@zope.org Subject: Re: [Zope] Zope Myths? On Wed, 2002-09-11 at 09:54, Paul Winkler wrote:
On Wed, Sep 11, 2002 at 03:01:14PM +0100, Paul Browning wrote:
What have we understated/exaggerated/misunderstood?
AFAIK, ZEO still does not allow you to distribute your ZODB. There is still a single, central ZEO server with a single ZODB.
Actually, using shared storage, such as one may have on a SAN, or using a shared SCSI bus (I *strongly* recommend the FC-SAN route over the shared SCSI ;), and a fail over server setup, you can indeed provide fail-over ZODB services. Yes, I know fibre channel SANs are beyond the means of most on the list, but it is certainly possible. In fact, I know of a way to provide multi-site fail over using ZODB and SAN technology on Linux. In fact, if one had one of the IBM zSeries boxes ... mmm toys .... I've yet to try it out (looking for a new job takes too much time in this market), but I believe one could use this method and provide multiple read-only ZEO servers for the same DB. -- Bill Anderson Linux in Boise Club http://www.libc.org Amateurs built the Ark, professionals built the Titanic. Amateurs build Linux, professionals build Windows(tm). _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
On Wed, Sep 11, 2002 at 03:46:43PM -0700, sean.upton@uniontrib.com wrote:
I strongly considered this option at one time with a shared DAS/RAID between my primary and secondary ZSS. I never ended up doing this because of concerns.
[...]
It really seems like replication is really a less brittle option (though how much less brittle, I'm not sure).
Hear, hear... Besides, you still have a single point of failure: your SAN system. OK, it's expensive and may thus fail less, but then you might buy an expensive system to begin with ;-) Another experience I had with such semi-high-end thingies is: if you haven't (some access to) experience with this very configuration, unexpected problems tend to crop up, thus eating up the advantage you thought to have. You are dealing with a more exotic system, with a smaller installed base, remember. You better know you understand very well how it works (or you have access to someone who does). Don't believe what marketing says -- those guys are in bed when one of your SCSI disks in the array fails and the RAID controller thinks it hasn't a hot spare and it'd better shut down the system (I had this one too -- luckyly I don't sell myself as hardware guy ;-)
Though not committed at this point, for a big project I am working on, I am strongly considering use of DirectoryStorage and its snapshot capabilities for low-tech replication (via find+cpio+nfs) to try and minimize issues I might face similar to #2 and #3 if I had chosen FileStorage.
I'd tend myself too to low-tech replication solutions -- in spite of the issues they present. Besides, if the systems are far apart, they still work if one of the buildings catches fire =:-0 Cheers -- tomas
On Thu, 2002-09-12 at 00:21, tomas@fabula.de wrote:
On Wed, Sep 11, 2002 at 03:46:43PM -0700, sean.upton@uniontrib.com wrote:
I strongly considered this option at one time with a shared DAS/RAID between my primary and secondary ZSS. I never ended up doing this because of concerns.
[...]
It really seems like replication is really a less brittle option (though how much less brittle, I'm not sure).
Hear, hear... Besides, you still have a single point of failure: your SAN system. OK, it's expensive and may thus fail less, but then you might buy an expensive system to begin with ;-)
Well, in that case, your network is a single point of failure, too. :^) Expensive, well that depend son what you are doing. For under 35000 you can have just shy of a 1TB of file space, with snapshot capability, multi-machine fail over, and a whole lot more. That cost includes two machines running Linux with fail over. It depends on your needs, and your uptime/availability requirements. For example, if you are running a site like cbsnewyork.com, 25-35 grand is not that much. If you are running a small site, then you don't need it. My point was that it (it being ZEO/ZODB/ZOPE) _can_ scale to that. I've done it.
Another experience I had with such semi-high-end thingies is: if you haven't (some access to) experience with this very configuration, unexpected problems tend to crop up, thus eating up the advantage you thought to have. You are
Well, speaking as a former tester of SAN technology, it would appear things have changed dramatically since your experiences. :) The configuration/setup is essentially the same as with SCSI, in fact, Fiber channel uses the SCSI subsystem in the OS. The underlying system is as robust as the SCSI system, since it is SCSI just over a different medium.
dealing with a more exotic system, with a smaller installed base, remember. You better know you understand very well how it works (or you have access to someone who does). Don't believe what marketing says -- those guys are in bed when one of your SCSI disks in the array fails and the RAID controller thinks it hasn't a hot spare and it'd better shut down the system (I had this one too -- luckyly I don't sell myself as hardware guy ;-)
I've never seen this with the Fiber Channel Arrays I dealt with. But then again, they had two or more controllers. :^)
Though not committed at this point, for a big project I am working on, I am strongly considering use of DirectoryStorage and its snapshot capabilities for low-tech replication (via find+cpio+nfs) to try and minimize issues I might face similar to #2 and #3 if I had chosen FileStorage.
I'd tend myself too to low-tech replication solutions -- in spite of the issues they present. Besides, if the systems are far apart, they still work if one of the buildings catches fire =:-0
Same thing with fibre channel SAN tech, the range is measured in miles. I know of several SANs that are spread over multiple states. You can literally have a fail over datacenter. -- Bill Anderson Linux in Boise Club http://www.libc.org Amateurs built the Ark, professionals built the Titanic. Amateurs build Linux, professionals build Windows(tm).
On Thu, Sep 12, 2002 at 11:12:27AM -0600, Bill Anderson wrote:
On Thu, 2002-09-12 at 00:21, tomas@fabula.de wrote:
On Wed, Sep 11, 2002 at 03:46:43PM -0700, sean.upton@uniontrib.com wrote:
[Big hardware vs. replication]
Well, in that case, your network is a single point of failure, too. :^)
Assuming just one network, assuming just one connectivity provider. Problem is that replication solutions depend heavyly on the type of application (slowly changing sets of files being the easiest and rapidly changing data sets with complex interdependencies (e.g. high-volume databases) the hardest.
Expensive, well that depend son what you are doing. For under 35000 you can have just shy of a 1TB of file space, with snapshot capability, multi-machine fail over, and a whole lot more. That cost includes two machines running Linux with fail over. It depends on your needs, and your uptime/availability requirements.
...or you can have ten cheapo off the shelf servers hosted at wildly different places... (OK, it's more like 0.5TB then ;)
For example, if you are running a site like cbsnewyork.com, 25-35 grand is not that much. If you are running a small site, then you don't need it. My point was that it (it being ZEO/ZODB/ZOPE) _can_ scale to that. I've done it.
[...]
Well, speaking as a former tester of SAN technology, it would appear things have changed dramatically since your experiences. :)
Yes, but this was mainly my point: if you have access to knowledge and experience with those things -- then you may go for it. If you don't... it's just a point against it.
The configuration/setup is essentially the same as with SCSI, in fact, Fiber channel uses the SCSI subsystem in the OS. The underlying system is as robust as the SCSI system, since it is SCSI just over a different medium.
[RAID system doing funny things]
I've never seen this with the Fiber Channel Arrays I dealt with. But then again, they had two or more controllers. :^)
Of course not -- but I take that you *know* what you are doing. This vendor didn't (at some point I realized that), but heck, it wasn't my job, I had enough on the plate myself. [...]
Same thing with fibre channel SAN tech, the range is measured in miles. I know of several SANs that are spread over multiple states. You can literally have a fail over datacenter.
Yes. It's a tradeoff. I just wanted to point out that experience with those things is one of the points to consider (besides application type, requirements and cost). It'd be interesting to know (I'm not a Zope guy) how well Zope as an application would play in each camp. Thanks -- tomas
On Wed, 2002-09-11 at 16:46, sean.upton@uniontrib.com wrote:
I strongly considered this option at one time with a shared DAS/RAID between my primary and secondary ZSS. I never ended up doing this because of concerns.
Out of curiosity, with shared storage over SCSI/SAN, do you have any thoughts on how to overcome the following issues?
1 - Data fencing (using SAN switches+integrated software, or STONITH devices, or something)??
I've found the best means for doing this in a failover situation is an FC switch with remote management through, say telnet. I've developed a python set of tools for controlling, for example, a Brocade FC switch through telnet. STONITH can be implemented through zoning control. For example, you set up two zones, one per server in this case, and when you need to "shoot" one, you disable the appropriate zone.
2 - Clean up a broken FileStorage? To what degree can this be successfully automated without manual intervention of a sysadmin?
In the configuration I describe, the same as a standard FileStorage. That is one of the gems of the system, all standard tools work like normal.
3 - Deal with accidental damage to the storage done by the primary node, perhaps due to asynchronous writes to the file?
I don't think I quite understand you here. The system I described provides fail over capabilities, so when the backup node comes online (it could be a hot standby) it opens data.fs just as it would normally. In the system I describe (and implemented), we are talking about a stock ZEO/ZODB/ZOPE setup, so I don't see this as a likelihood. Shared storage, but only one ZSS has write-capabilities.
It really seems like replication is really a less brittle option (though how much less brittle, I'm not sure).
It depends on what you are after. In my cases, there is a redundant ZSS, or multiple read-only ZSSes, all "sharing" the same data.fs. Sure, if your write-ZSS goes haywire and hoses the database, you are restoring from backup, or recovering some other way. Replication will suffer the same possibility, unless it provides some sort of check against a haywire write-process. Replicating a bad data.fs doesn't seem to be much help. :) Replication is one method, and has it's drawbacks (like synchronization). Shared storage with fail over is another. If you don't like SAN using FC, use a NAS box, with a physically separate network for mounting the NAS, and implement it that way. Granted, that has it's issues too, but then again so do all the options. :^) It all depends on your requirements and needs, as well as resources and experience. Myself, I have extensive experience in heavy-duty SAN workings, so I am most comfortable with a shared storage system. For "high-end" it is a very powerful option, and can be done at the midrange, etc.. This is where, as a consultant, the cost of ZEO/ZODB/ZOPE can become a powerful tool. If a client has a budget, of say 50 grand for hardware/software, expecting the software to be half that cost, for example, you can improve the hardware --improving reliability and scalability-- and possibly save money for them at the same time. Sometimes they are not too interested in saving money, but are interested in getting more out of it. Cheers, Bill -- Bill Anderson Linux in Boise Club http://www.libc.org Amateurs built the Ark, professionals built the Titanic. Amateurs build Linux, professionals build Windows(tm).
participants (3)
-
Bill Anderson -
sean.upton＠uniontrib.com -
tomas＠fabula.de