Steve, Interesting idea. Several years ago (OK, a lot), prior to the commercial availability of reliable replication of non-mainframe databases I wrote something similar to this in C. The approach you recommend is very similar -- here are a few of the things that my version of the db-tee did: 1) Kept track of the network time or metric to reach each db. 2) Had a running reference count of active threads using the instance of the tee 3) Used a load-balancing technique taking the prior two items into consideration 4) Cached write operations (see below). One of the problems we ran into early on was the issue of cross-corrupted(?) caching. What I mean is that any caching you're doing at the unique db level is effectively unusable unless you build in the ability to blast out a "record invalidated" message to each cache to allow for the removal of any recently changed or deleted rows. An additional challenge that is faced is that of single machine failure. When a single machine fails, it does not receive the ongoing write operations from the tee. Therefore, when it is brought back up again, things are usually way out of sync. One option is allow an agent (in this case, perhaps the policy object) to cache all write operations and ensure they are executed at each adaptor instance. Anyway, I think the idea is timely and challenging. My hat's in the ring if you need help designing and implementing this. Kevin -----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Steve Spicklemire Sent: Thursday, May 30, 2002 5:17 PM To: zope@zope.org Cc: Steve Spicklemire Subject: [Zope] Poor man's (application level) database replication? Hi Folks, This might be the world's dumbest idea... but I'm close enough to thinking that it's "OK" to start wasting time on it, and I want to get some feedback before I actually waste very much time. ;-) It goes like this: I need to be sure that my SQL data is replicated somewhere. Replication seems to have spotty support among the free database, and it's also fairly complicated to set up. Mere mortals would like something easy, and dumb, to help out in cases where these two obstacles are a significant hurdle. ZSQLMethods required you to choose a database adaptor, and database adaptors generally permit you to connect only to a single database. I'm thinking that I could cook up a simple "db-tee" adaptor that keeps track of two things: 1) a list of "real" database adaptors that it knows about and their current "status" (OK/broken) 2) a callable "policy" object (python script, external method, ZClass?) that can work with it's list of DAs to implement the replication by directing the iteration process. The basic plan is that a ZSQLMethod attaches itself to the tee, and calls the tee for database access. The tee then uses it's policy object to iterate through the list of database adaptors. For read operations (e.g., SELECT) the policy would use a "continue until successful" strategy, while write operations would use a "iterate over all DAs" strategy. Even the choice of which adaptors to iterate over could be variable (e.g., depend on the value of ZEO_CLIENT or somesuch). This way reading from the database(s) would be faster (since each ZEO_CLIENT could use a different DA as the first DA to check) but writes would be slower (since each DA would need to be called in turn to get replication). But, in my experience, writing is much less frequent than writing, so maybe that's OK? Also, the policy object could decide how to handle failures (on read: skip and set status to "broken", on write: raise an exception, or notify someone, or whatever you like). Anyway.. I'm just wondering if anyone has toyed with this sort of plan, or has come up with something better? thanks! -steve _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )