[Zope] ZOPE, Medusa and the ZD WebBench 3.0 test suite...

[Jim Salmons]

Thanks for the thoughtful response, Chris, but something to add to the mix
is that this review marks the debut of ZD's WebBench 3.0. According to PC
Mag this version addresses the static, dynamic, and secure configs in a more
realistic manner than earlier versions of the test. (There may be info about
this test suite on the ZDNet site (there is a download dated 4-6-99 in the
software library), if not and you don't have access to the PCMag issue at
hand, drop me a note and I can scan and e-mail them to you, Chris, so you
can see if the updated benchmark addresses your concerns.)

So, assuming that they may have a more realistic "spread" of typical serving
demands, the question comes back to "Can a ZOPE/Medusa config be run against
WebBench 3.0?"

Despite whatever we and Twain may think about benchmarks in general, from a
marketing/sales standpoint saying "We don't believe in benchmarks and none
are appropriate for our technology," does not go over very well with
potential client/customers. (Nor do non-standard tests go over well since
they don't allow "comparative shopping.")

My experience is that it is much better to at least subject yourself to such
"standard benchmarks" and have a compelling "footnotes and explanations" to
address any mismatch between your technology and an industry benchmark.

A curious ZOPEer,
--Jim Salmons--
<salmons@sohodojo.com>


-----Original Message-----
From: Chris Petrilli [mailto:petrilli@digicool.com]
Sent: Monday, May 24, 1999 11:31 AM
To: 'Jim Salmons'; zope@zope.org
Subject: RE: [Zope] ZOPE, Medusa and the ZD WebBench 3.0 test suite...


Lies, damn lies and benchmarks.
     -Samuel Clements via me.

> My question, and the one I expect is on many of your minds,
> is... How does a
> ZOPE/Medusa server config stack up? My hope and expectation that the
> underlying Medusa architecture might really shine when compared to the
> pre-allocating multi-process approach of Apache, etc.

Well, this is for lack of a better statement, irrelevent to Zope.  They
are asking (last I looked) for a static-page system, with some tiny
CGIs, but this isn't really what Zope is about.  EVERY page is Zope is
dynamic, so if you compared a 100% CGI site with Zope, then it *might*
be a valid comparison, if you also pulled all the data out of a database
on the fly as well...

IMPORTANT:
The only benchmark that matters is the one that is your application, and
on that, Zope excells.

As for asyncronous v. threading/forking servers.  Zeus, which is the
fastest 'static' web server int he world, and what everyone uses for
benchmarking on UNIX is also an asyncronous server.  Different servers
have different behaviours.

One of the common problems with IIS v. Apache/etc is that Apache isn't
tuned to saturate a small number of clients (which is all they are
simulating) with very big pipes, but instead, a huge number of clients
with smaller bandwidth (what the Internet really looks like).  Also,
IIS4 runs a "cache" inside the server, which you could simulate iwth
Squid, or some such... One must be careful comparing the flavor of two
things sold as oranges when one is distinctly shaped like a pumpkin.

Chris
--
| Christopher Petrilli                        Digital Creations
| petrilli@digicool.com                 http://www.digicool.com