[Zope-dev] ZEO cache instrumentation -- any takers?

[Guido van Rossum]

A while ago I announced a new ZEO cache instrumentation feature, and
asked if anyone was interested in enabling this instrumentation in
their site.  I got exactly zero responses... :-(

I'd like to repeat the offer.  From the instrumentation data, it is
easy to determine the most effective cache size for your site (a tool
to do this is part of the release).  All you need to do is replace one
file, ClientCache.py.  Both ZEO 1 and ZEO 2 are supported.  The
instrumentation code does *not* use zLOG and causes very little
overhead.  It can create a large log file though (100 MB per day at
one Zope Corp customer).

Here's a copy of my original announement:

>Subject: ZEO cache analysis tools prerelease
>From: Guido van Rossum <guido@python.org>
>To: zope-announce@zope.org
>Date: Tue, 10 Sep 2002 17:02:41 -0400

* Do you want to know if your ZEO client cache is configured properly?

* Are you using ZEO but you didn't know it had a client cache?

* Do you want to contribute towards a better ZEO client cache design?

If you answered yes at least once, read on!

Jeremy is working on a new ZEO 2 release; the first beta was released
two weeks ago.  For that release I have written cache instrumentation
code that lets you review the effectiveness of the ZEO client cache.
There's also a version of the instrumented cache for ZEO 1.  I don't
plan to work on it more before the ZEO 2 release (unless I receive bug
reports).  All code is available *now* from the ZODB3 CVS tree:

  http://cvs.zope.org/ZODB3/.

In addition to adding instrumentation, some bugs affecting performance
but not correctness of the cache have been fixed in the instrumented
cache code.  The instrumentation is controlled by an environment
variable so you can leave the new cache code in place after turning
off the instrumentation.

I'm inviting everyone who is running ZEO seriously to install the
cache instrumentation code, run it for a while, and send me the
resulting trace file.  (The trace files contain no privacy-sensitive
data, so you don't have to worry about revealing anything about your
site.)  The trace files will be used as input to a simulation of a new
cache design.  The only way to validate a cache design is to run
experiments, and trace files are the most reliable way to run such
experiments.  The more trace files I receive the better.  The larger
they are the better (traces of several days of uninterrupted service
are ideal).  I've got the disk space to burn.

I should mention that I'm confident that the instrumentated cache
works correctly: we've been collecting traces without problems at a
Zope Corp customer site since last Wednesday.

To enable the cache instrumentation, you need to replace a single file
in the ZEO package used by the Zope application, set an environment
variable, and restart the Zope application.  The file you need to
replace is named ClientCache.py.  Here are the URLs:

  For ZEO 1:  http://cvs.zope.org/ZODB3/ZEO1/ClientCache.py?rev=HEAD
  For ZEO 2:  http://cvs.zope.org/ZODB3/ZEO/ClientCache.py?rev=HEAD

Make sure to get the file for the ZEO version you are using!

Details about enabling the tracing can be found here:

  http://cvs.zope.org/ZODB3/ZEO/README.txt?rev=HEAD

If you want to play with the statistics or simulation programs
yourself, get stats.py and simul.py from the ZEO directory:

  http://cvs.zope.org/ZODB3/ZEO/

These work with trace files produced by either version of the
instrumented cache code.

Once you have one or more trace files for me, the best way to get it
to me is probably to place a gzipped copy on an FTP or HTTP server
from which I can download it and mail me the URL.  As I said, these
trace files contain no sensitive data.  If you still prefer a
different approach, email me and we'll figure something out.

--Guido van Rossum (home page: http://www.python.org/~guido/)