[Zope-CMF] Zope/CMF Performance Testing

[Kent Hoxsey]

The team I work with has been evaluating Zope as a content management
system to take over the content-publishing role of our current
Apache/cgi environment, and integrate seamlessly with the existing web
and application space. We are basing this implementation on the CMF,
and thought the community might be interested both in the path we have
taken for the evaluation, and in some of the results.

The intent of our project is to provide better control over the look &
feel of the entire intranet site and give our users the ability to
publish to the intranet, while at the same time reducing support
requirements on our team. We see promise for further efficiencies and
features in a number of Zope products, but have not yet advanced to
the stage of implementing any of them.

As a part of our evaluation, we ran a suite of performance tests to
get a feel for the response rates and tunability of the system. This
suite was developed by our Web Infrastructure team, drawing from their
experience running a very-high-hitrate ecommerce site. The test system
was installed according to the available best-practices published on
zope.org, and in accordance with our internal system standards.

The test system was configured to run a single ZEO client against a
Zope Storage Server, both running on the same machine. The system also
runs Apache and mod-proxy, in order to provide tunability for
cacheable objects. Our infrastructure team believes that this
configuration can be scaled by splitting the various processes (ZSS,
ZEO client, Apache/proxy server) onto their own servers, and by
distributing the load to additional ZEO clients and/or Apache servers
as needs arise due to demand or desire for geographic distribution.

The system under test is a Sun Ultra5, with 1G memory. The Zope/ZEO
setup was performed as detailed on zope.org, running the source
release of Zope 2.3.3, with CMF 1.1 and TransparentFolders.

The test suite was run against two different front-ends: directly
against Zope, as well as against Apache/ProxyPass. It exercised a
number of different configurations of the system, exploring the
performance ramifications of some of the different system settings
available for tuning. The performance tester itself is a tool
developed by our Infrastructure team that creates web clients
requesting URLs from a server, according to a specified load profile.
For the tests reported here, each client performed 10 loops of
requests for a sequence of 6 URLs, sleeping 5 seconds between each
request. Our team collected average response time and hitrates for
each iteration of the tests.

The results themselves are interesting. Our test system is relatively
underpowered in comparison with serving hardware available on the
market today, yet with caching enabled it seems capable of supporting
significant hitrates.

However, there appears to be a significant investment in tuning
knowledge required to attain these performance levels. During our
testing, we encountered a number of artifacts due to the caching
parameters. Different page components would be served from cache
inappropriately, leading to users receiving inaccurate, incorrect, or
out-of-date objects. We were able to resolve a couple issues we
investigated, but several more remain. It is clear that caching models
are not 'fire and forget'. An organization should not plan on the
higher hitrates available with a caching model without first investing
effort in testing and verifying the results of their tuning
parameters.

I would like to recognize the members of our team for their efforts:
Todd Hepler and Pete Bakas for all of the work on the systems and test
suite, and Carl Rendell for his work on the Zope/CMF site itself.

k1


The Test Results:
=================
6 configurations described below.

6 URLs, 10 loops, 5 sleep

test #  config  clients avg resp        hits/sec
------  ------  ------- --------        --------
1       1       10      02.55           01.85
2       1       20      07.60           01.93
3       1       50      27.96           01.61

4       2       10      02.71           01.88
5       2       20      07.92           01.87

6       3       10      00.07           03.55
7       3       20      00.02           07.02
8       3       100     00.06           26.32
9       3       200     00.14           35.71
10      3       300     00.22           40.72
11      3       400     00.30           44.04
12      3       500     00.45           46.09

13      4       10      00.10           03.59
14      4       20      00.07           07.10
15      4       200     05.58           18.13

16      5       10      00.14           03.53
17      5       20      00.11           06.70
18      5       200     08.89           13.95
19      5       300     12.21           15.71


configs
-------
1) straight against zope ZServer, no RCM
2) apache, zope (VHM, no RCM) and mod_proxy cache off
3) apache, zope (VHM, no RCM) and mod_proxy cache on
4) straight against zope ZServer, with RCM
5) apache, zope (VHM, RCM) and mod_proxy cache off

RCM is Ram Cache Manager, a Zope object.
VHM is Virtual Host Monster, a Zope object.