RE: [Zope] replication (was Zope: 5.4, jboss 0.3 million hits wit h google)
I suppose that one issue is that packing in DirectoryStorage uses snapshots so that using something like rsync or cpio+find+nfs for incremental backup/replication to a peer ZSS would need to use snapshots. You can't do this while packing though, because packing uses snapshots. So a replication tool that uses normal history isn't prohibited during packing. This is how I understand this, but Toby may have a different (and likely more correct - and certainly more informed) explanation. Sean -----Original Message----- From: Paul Winkler [mailto:pw_lists@slinkp.com] Sent: Wednesday, February 12, 2003 5:53 PM To: zope@zope.org Subject: Re: [Zope] replication (was Zope: 5.4, jboss 0.3 million hits with google) On Thu, Feb 13, 2003 at 09:01:37AM +0800, Bakhtiar A Hamid wrote:
On Thursday 13 February 2003 04:51, Paul Winkler wrote:
On Wed, Feb 12, 2003 at 11:26:05AM -0800, Jack Park wrote:
... Theoretically speaking, JBoss has tackled the scalability issues. I have yet to find that Zope has done the same.
A ZEO cluster running behind Squid... if that's not fault-tolerant or distributed enough, use DirectoryStorage and have automated replication of multiple ZEO clusters. Theoretically this could scale to really enormous amounts of data serving enormous numbers of users.
how can we replicate with DirStorage? would you mind doing a little brain
dump? with rsync?
I am not really the person to ask, as I haven't actually done it. But here's the official way to do it as of version 1.1: http://dirstorage.sourceforge.net/replica.html I would naively assume that you *should* be able to replicate by 1) putting the "master" into snapshot mode 2) running rsync 3) taking the "master" out of snapshot mode ... but there may be hidden issues with that; I would kind of assume so, since Toby Dickenson bothered to write the replication tool. Toby, are you reading this? Care to comment? -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's YET ANOTHER KATANA ROGERWILCO! (random hero from isometric.spaceninja.com) _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Thursday 13 February 2003 3:46 am, sean.upton@uniontrib.com wrote:
I suppose that one issue is that packing in DirectoryStorage uses snapshots so that using something like rsync or cpio+find+nfs for incremental backup/replication to a peer ZSS would need to use snapshots. You can't do this while packing though, because packing uses snapshots.
Correct. I am planning to attack this in three directions: 1. changing packing so that it doesnt keep the storage in snapshot mode during the whole packing process. 2. A good choice of linux kernel patches. Chris Mason's latest reiserfs patches help *alot*, but I am not yet willing to trust the 2.4.20 kernel in production. 3. Speeding up packing. Version 1.1 has an experimental option that is somewhat faster, but has a disk space overhead and leaks memory :-( I am waiting to see whether this has much benefit on a patched kernel before polishing this for production use.
So a replication tool that uses normal history isn't prohibited during packing.
The replication tool currently does need to force the storage into snapshot mode. Yes, this means you cant replicate and pack at the same time. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson
participants (2)
-
sean.upton＠uniontrib.com -
Toby Dickenson