I saw we have it , in Zope Corp site, but there's no informatio about how to be certified and etc.
Hi Folks, This might be the world's dumbest idea... but I'm close enough to thinking that it's "OK" to start wasting time on it, and I want to get some feedback before I actually waste very much time. ;-) It goes like this: I need to be sure that my SQL data is replicated somewhere. Replication seems to have spotty support among the free database, and it's also fairly complicated to set up. Mere mortals would like something easy, and dumb, to help out in cases where these two obstacles are a significant hurdle. ZSQLMethods required you to choose a database adaptor, and database adaptors generally permit you to connect only to a single database. I'm thinking that I could cook up a simple "db-tee" adaptor that keeps track of two things: 1) a list of "real" database adaptors that it knows about and their current "status" (OK/broken) 2) a callable "policy" object (python script, external method, ZClass?) that can work with it's list of DAs to implement the replication by directing the iteration process. The basic plan is that a ZSQLMethod attaches itself to the tee, and calls the tee for database access. The tee then uses it's policy object to iterate through the list of database adaptors. For read operations (e.g., SELECT) the policy would use a "continue until successful" strategy, while write operations would use a "iterate over all DAs" strategy. Even the choice of which adaptors to iterate over could be variable (e.g., depend on the value of ZEO_CLIENT or somesuch). This way reading from the database(s) would be faster (since each ZEO_CLIENT could use a different DA as the first DA to check) but writes would be slower (since each DA would need to be called in turn to get replication). But, in my experience, writing is much less frequent than writing, so maybe that's OK? Also, the policy object could decide how to handle failures (on read: skip and set status to "broken", on write: raise an exception, or notify someone, or whatever you like). Anyway.. I'm just wondering if anyone has toyed with this sort of plan, or has come up with something better? thanks! -steve
Steve, Interesting idea. Several years ago (OK, a lot), prior to the commercial availability of reliable replication of non-mainframe databases I wrote something similar to this in C. The approach you recommend is very similar -- here are a few of the things that my version of the db-tee did: 1) Kept track of the network time or metric to reach each db. 2) Had a running reference count of active threads using the instance of the tee 3) Used a load-balancing technique taking the prior two items into consideration 4) Cached write operations (see below). One of the problems we ran into early on was the issue of cross-corrupted(?) caching. What I mean is that any caching you're doing at the unique db level is effectively unusable unless you build in the ability to blast out a "record invalidated" message to each cache to allow for the removal of any recently changed or deleted rows. An additional challenge that is faced is that of single machine failure. When a single machine fails, it does not receive the ongoing write operations from the tee. Therefore, when it is brought back up again, things are usually way out of sync. One option is allow an agent (in this case, perhaps the policy object) to cache all write operations and ensure they are executed at each adaptor instance. Anyway, I think the idea is timely and challenging. My hat's in the ring if you need help designing and implementing this. Kevin -----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Steve Spicklemire Sent: Thursday, May 30, 2002 5:17 PM To: zope@zope.org Cc: Steve Spicklemire Subject: [Zope] Poor man's (application level) database replication? Hi Folks, This might be the world's dumbest idea... but I'm close enough to thinking that it's "OK" to start wasting time on it, and I want to get some feedback before I actually waste very much time. ;-) It goes like this: I need to be sure that my SQL data is replicated somewhere. Replication seems to have spotty support among the free database, and it's also fairly complicated to set up. Mere mortals would like something easy, and dumb, to help out in cases where these two obstacles are a significant hurdle. ZSQLMethods required you to choose a database adaptor, and database adaptors generally permit you to connect only to a single database. I'm thinking that I could cook up a simple "db-tee" adaptor that keeps track of two things: 1) a list of "real" database adaptors that it knows about and their current "status" (OK/broken) 2) a callable "policy" object (python script, external method, ZClass?) that can work with it's list of DAs to implement the replication by directing the iteration process. The basic plan is that a ZSQLMethod attaches itself to the tee, and calls the tee for database access. The tee then uses it's policy object to iterate through the list of database adaptors. For read operations (e.g., SELECT) the policy would use a "continue until successful" strategy, while write operations would use a "iterate over all DAs" strategy. Even the choice of which adaptors to iterate over could be variable (e.g., depend on the value of ZEO_CLIENT or somesuch). This way reading from the database(s) would be faster (since each ZEO_CLIENT could use a different DA as the first DA to check) but writes would be slower (since each DA would need to be called in turn to get replication). But, in my experience, writing is much less frequent than writing, so maybe that's OK? Also, the policy object could decide how to handle failures (on read: skip and set status to "broken", on write: raise an exception, or notify someone, or whatever you like). Anyway.. I'm just wondering if anyone has toyed with this sort of plan, or has come up with something better? thanks! -steve _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Hi Kevin, Thanks for the feedback! Your system sounds a lot more sophisticated than what I was planning. My assumptions are basically: 1) Failure is very rare, but over long periods of time, it is likely there there will be some failure, 2) There are lots of folks out there who want to have a more robust setup, but can't afford to buy expensive solutions, or become experts in the more arcane issues of extreme reliability. The question is, when a failure does occur, is there any (low cost, easy, simple) way to mitigate its effect? I think that the idea of a 'policy' object could in principle handle "3", but I'm thinking that it probably mostly won't do any of these things. At least not right away. I'm thinking that 1) Existing DB adaptors already handle caching (I haven't looked into how they handle invalidating their internal cache.. but I suppose I should!). 2) The primary issue I'm worried about is not performance (though I don't want to incur a huge performance loss!) but increasing the likelihood that a database failure will not bring down the system. I think that load balancing would be nice, but something much cruder (i.e., each ZEO client uses a different DB server for reading) will also help performance, if not optimize it. (Note that if you load balance the ZEO clients, using TD's ICP code, or some other method, at least for database bound applications, it may well have a follow-on database load-balancing effect!) Anyway.. I'm encouraged enough to take a first stab at this. thanks! -steve On Friday, May 31, 2002, at 07:39 AM, Kevin Carlson wrote:
Steve,
Interesting idea. Several years ago (OK, a lot), prior to the commercial availability of reliable replication of non-mainframe databases I wrote something similar to this in C. The approach you recommend is very similar -- here are a few of the things that my version of the db-tee did:
1) Kept track of the network time or metric to reach each db. 2) Had a running reference count of active threads using the instance of the tee 3) Used a load-balancing technique taking the prior two items into consideration 4) Cached write operations (see below).
One of the problems we ran into early on was the issue of cross-corrupted(?) caching. What I mean is that any caching you're doing at the unique db level is effectively unusable unless you build in the ability to blast out a "record invalidated" message to each cache to allow for the removal of any recently changed or deleted rows.
An additional challenge that is faced is that of single machine failure. When a single machine fails, it does not receive the ongoing write operations from the tee. Therefore, when it is brought back up again, things are usually way out of sync. One option is allow an agent (in this case, perhaps the policy object) to cache all write operations and ensure they are executed at each adaptor instance.
Anyway, I think the idea is timely and challenging. My hat's in the ring if you need help designing and implementing this.
Kevin
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Steve Spicklemire Sent: Thursday, May 30, 2002 5:17 PM To: zope@zope.org Cc: Steve Spicklemire Subject: [Zope] Poor man's (application level) database replication?
Hi Folks,
This might be the world's dumbest idea... but I'm close enough to thinking that it's "OK" to start wasting time on it, and I want to get some feedback before I actually waste very much time. ;-) It goes like this: I need to be sure that my SQL data is replicated somewhere. Replication seems to have spotty support among the free database, and it's also fairly complicated to set up. Mere mortals would like something easy, and dumb, to help out in cases where these two obstacles are a significant hurdle. ZSQLMethods required you to choose a database adaptor, and database adaptors generally permit you to connect only to a single database. I'm thinking that I could cook up a simple "db-tee" adaptor that keeps track of two things:
1) a list of "real" database adaptors that it knows about and their current "status" (OK/broken) 2) a callable "policy" object (python script, external method, ZClass?) that can work with it's list of DAs to implement the replication by directing the iteration process.
The basic plan is that a ZSQLMethod attaches itself to the tee, and calls the tee for database access. The tee then uses it's policy object to iterate through the list of database adaptors. For read operations (e.g., SELECT) the policy would use a "continue until successful" strategy, while write operations would use a "iterate over all DAs" strategy. Even the choice of which adaptors to iterate over could be variable (e.g., depend on the value of ZEO_CLIENT or somesuch). This way reading from the database(s) would be faster (since each ZEO_CLIENT could use a different DA as the first DA to check) but writes would be slower (since each DA would need to be called in turn to get replication). But, in my experience, writing is much less frequent than writing, so maybe that's OK? Also, the policy object could decide how to handle failures (on read: skip and set status to "broken", on write: raise an exception, or notify someone, or whatever you like).
Anyway.. I'm just wondering if anyone has toyed with this sort of plan, or has come up with something better?
thanks! -steve
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
A certification program description was up on Zope.com for a while but it was taken down due to a lack of interest. Zope Corp no longer offers any sort of Zope certification program (no certifications were ever issued, FWIW). - C José Orlando Ribeiro wrote:
I saw we have it , in Zope Corp site, but there's no informatio about how to be certified and etc.
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
A certification program description was up on Zope.com for a while but it was taken down due to a lack of interest. Zope Corp no longer offers any sort of Zope certification program (no certifications were ever issued, FWIW).
I'm a certifiable zope developer. I've jst never been certified. :)
On Fri, 2002-05-31 at 11:28, Steve Drees wrote:
A certification program description was up on Zope.com for a while but it was taken down due to a lack of interest. Zope Corp no longer offers any sort of Zope certification program (no certifications were ever issued, FWIW).
I'm a certifiable zope developer. I've jst never been certified. :)
So that would make you certifiable then right? ;^) -Casey
participants (6)
-
Casey Duncan -
Chris McDonough -
José Orlando Ribeiro -
Kevin Carlson -
Steve Drees -
Steve Spicklemire