----- Original Message ----- From: "Brian Brinegar" <brinegar@ecn.purdue.edu> To: <zope@zope.org> Sent: Wednesday, September 13, 2006 9:22 PM Subject: [Zope] ZEO with Multiple Storages
We currently have a ZEO environment with FileStorage (data.fs) over 100 gigs, which continues to grow. Currently we are approaching a major hardware and software upgrade. All new hardware and moving from Zope 2.6 to Zope 2.9.
Our server hosts sites for several different schools and departments within our college, what I have considered doing is trying to split up the data.fs into multiple smaller storages, possibly one per school. I suspect if this is possible it could potentially have several benefits. The first being that we would not have to deal with 100 gig files which are very difficult to move around backup/restore, etc. Secondly, we often have a very large number of users updating content at a given time and will run into slowdowns where a transaction is blocked while another commits. I hope that multiple storages could allow for a transaction to commit to one storage at the same time as another, though I am not sure. Along this line can a single transaction commit to multiple storages?
If you mean having zope commit a single transaction to multiple storages, then 'vanilla' zope won't do it (Zope Corp has a commercial product called ZRS which does this type of thing). Alternatively, if your server is disk-bound then you could look into RAID disk striping to try to improve performance.
First off, does this make any sense? Is it worth pursuing? Secondly, I assume that where an object is stored does not limit what other objects it can access in a transaction (e.g. object A could add a property to object B if they were in different storages) because all changes would occur in a client. (this ties into the single transaction committing to multiple storages)
If this does makes sense is there any documentation, recommendations, best practices for how to set something like this up?
Lastly, any thoughts on how we could split up our existing data.fs file? One thought I had was to attempt to import/export data.
If you have the time and hardware you could easily set up several zeo servers on different 'back-end' hardware servers (ie. put each zeo server on its on computer) to improve performance, increase reliability/robustness, etc. Your 'use case' in which you have different schools and departments would lend itself to this kind of distribution (put the biggest/heaviest users on their own servers, group together smaller/lighter users on shared servers). You just need to define a mount point (in zope.conf) for each different zeo server (and set up each zeo server of course). If you already have the data for the various schools & depts stored in separate folders, it would be easy to export each 'school' folder and then import it into the new configuration. hth Jonathan