[Zope] Hardware for Zope + ZEO

[Toby Dickenson]

(cc the direstorage-users list too)

On Friday 31 January 2003 5:07 pm, Paul Winkler wrote:

> **HOWEVER** you are concerned about performance, and DS is even slower =
than
> FileStorage for writes.  From the DS FAQ:
>
> """Intermittant writes are a factor of 1.5 slower. ...
> Under high write pressure the journal queue becomes a bottleneck, and
> performance degrades to 3 times slower than FileStorage.
> """
>
> The question then becomes, what is "high write pressure"?

A benchmark that bombards the storage with nothing but writes sufficient =
to=20
saturate the disk interface. Most production loads dont look like that. T=
he=20
storage probably spends some of its time handling reads, and some (most?)=
=20
time idle.

DirectoryStorage is optimised for writes that come in bursts. It reduces =
the=20
latency of individual writes within the burst, under the assumption that =
it=20
can do the rest of the work asynchronously once the burst is over.=20

The 3x slowdown applies if the 'burst' goes on too long.

Yes, you can configure the size of a burst.

(for what its worth, I expect to be able to improve on that 1.5x with the=
=20
latest reiserfs kernel patches)

> And what does 3x slower than FS feel like to the user?

A typical human Zope user wont notice. Most of the time in a Zope request=
 is=20
spend in DTML processing, application logic, traversing, and security che=
cks.=20
In my experience only a small proportion of the time is spent in the stor=
age.=20
3 times small is still small.

Expect something like 3x for scripts that perform many writes.

The write response profile changes once the storage is pushed into this 3=
x=20
mode in its default configuration. Some writes will be much slower than=20
others, and this will be noticeable to a human. The cause and effect are=20
analogous to virtual memory thrashing. This can be tweaked, but I doubt=20
anyone will need to.

> given what the DS FAQ says about write performance, I'd look into
> setting up a test server and bombard it with automated writes to see
> if it will handle the load you anticipate.  But of course you were goin=
g
> to do that anyway. ;)

Indeed. Please share your results.

Note than for modern storages it is important to measure the performance =
under=20
a realistic load, rather than applying a huge load and seeing where it=20
saturates. The original Berkeley storage benchmarks were bogus (imo) for =
this=20
reason.

--=20
Toby Dickenson
http://www.geminidataloggers.com/people/tdickenson