[Zope] Hardware for Zope + ZEO

[Paul Winkler]

On Fri, Jan 31, 2003 at 04:00:39PM +0900, Wankyu Choi wrote:
> On Wed, Jan 29, 2003 at 03:32:20PM +0900, Wankyu Choi wrote:
> > I've been thinking:
> > 
> > 1. Take out as much binary data as possible via External File ( my 
> > version, not the ones found in Zope products page ).
> 
> >>i can understand doing this if you're running FileStorage and want to
> avoid a single bloated Data.fs file, but since you're leaning towards
> DirectoryStorage, I think there's no reason to use ExternalFile at all - it
> just complicates things.
> 
> Hm.. your reply pretty much convinces me^^

Well, i wouldn't go that far. :)
I should add the disclaimer that i have not actually used DirectoryStorage
in production yet. I made the above statement based on the DS docs. I have
only started evaluating options myself after our recent troubles with FS
corruption. All this is IM-very-humble O, and  YMMV and all that.
 
> Now I'll be trying DirectoryStorage in earnest.
> 
> > 2. Store textual data  ( raw data + its rendered version, etc ) in 
> > MySQL.
> 
> >> ditto
> 
> Why shouldn't I let DirectoryStorage also take care of this?

sorry, by "ditto" I meant to convey "same answer I gave to the last point".
If DS performs reliably as advertised, it would completely obviate the need 
for storing anything outside the ZODB, from a scalability & maintenance standpoint 
at least. 

**HOWEVER** you are concerned about performance, and DS is even slower than 
FileStorage for writes.  From the DS FAQ:

"""Intermittant writes are a factor of 1.5 slower. ...
Under high write pressure the journal queue becomes a bottleneck, and 
performance degrades to 3 times slower than FileStorage.
"""

The question then becomes, what is "high write pressure"?
And what does 3x slower than FS feel like to the user? 

> > 3. User data goes into MySQL.
> 
> >> ditto
> 
> Again why? In fact, I want to keep the existing user data where they are but
> am just curious what I would be missing if I didn't^^

again, by "ditto" I meant to convey "same answer I gave to the last point" :)

> Okay, but is there anything I should be aware of before deciding on the
> directory storage as my storage of choice? The data will be both read- and
> write-intensive.

given what the DS FAQ says about write performance, I'd look into 
setting up a test server and bombard it with automated writes to see 
if it will handle the load you anticipate.  But of course you were going
to do that anyway. ;)
  
> > If most Zope users ( including yourself ) feel Filestorage is not a 
> > solution for more than 20GB of data,
> 
> >>hell, I don't like it for 2 GB of data.
> >>(see recent thread "POSKeyError II: Dead by Dawn" ... I wasn't kidding
> when I titled it that!)
> 
> Yeah, that POSKeyError thing scares the hell out of me ;-)

I could almost feel the server screaming "JOIIINNNN USSSSSS" in an eery DSP-treated
voice.
 
-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's SNAZZY DEVIANT GIRL!
(random hero from isometric.spaceninja.com)