[Zope] ZSERVER, THREADS, PERFORMANCE & FUTURE PERFORMANCE

[Paul Everitt]

Theodore Patrick wrote:
> In the ZOPE Documentation there are several tuning features for ZOPE ODB and
> ZSERVER. I was wondering about current Zope Performance and where it is
> heading. In certain areas we are looking at JAVA Servlets for performance

I'll be interested to hear the results of this.  The knock on servlets
is that performance isn't that great, but I haven't seen any specific
numbers.

> reasons, where ZOPE falls short of expectations. Here are some questions

I'd also like to hear some existing numbers (hit rates, response times,
kind of operations that are slow) that frame the discussion -- that is,
what are you seeing that makes you conclude Zope isn't performing up to
expectations?

> that we have come up with in regards to Zope Performance in an Enterprise
> Production Environment.
> 
> CURRENT ZOPE PERFORMANCE:
> 
> Q: Is there a performance improvement in raising the number of threads on
> the ZOPE ODB?

Chris answered this pretty well.  If you have operations that last
longer than a tenth of a second (they compute a lot of stuff, its a
management operation, its a request to a relational database), then yes.

> Q: Does increasing the # of threads cause any problems? With Threaded DA's?
> With the ODB?

Definately a Python threading voodoo area.  Increasing the number of
application threads can certainly increase memory required, as each
thread has its own "cache" of objects.

> Q: Is ZOPE performance with threads hardware restrictive? (ie. More threads
> on a strong box=good, Default on a Strong box = bottleneck, More on a weak
> box=worse performance)

Dunno.

> Q: How does ZEO improve ZOPE performance?

You get to add processors, boxes, and networks to your object system. 
On multiprocessor machines, you get around Python's non-wonderful
threading by spreading processes across processors rather than threads
across processors.  Finally, processes that are performing poorly (long
compute, hung, etc.) don't get requests sent to them.

> FUTURE ZOPE PERFORMANCE/DEVELOPMENT
> 
> Q: Is ZOPE limited/Dependent on the associated Python Performance? Can ZOPE
> run on separate instances of Python(Multiple JVM's in the JAVA World)?

Zope is written largely in Python, so obviously it is tied to the
performance of Python.  If there is evidence that Python itself, or Zope
itself, is the bottleneck, convert something to C.

> Q: Is there hidden ZEN out there on Performance Tuning Zope for a Heavy
> Production Environment(besides ZEO)? (HPE = up to 150 Requests/sec)
> 
>         How does one scale Zope to Java Sevlet performance levels (compiled
> in-memory performance)?
>                 Sevlets = 200+ Requests per second (once invoked and
> in-memory)

I have a hard time believing that servlets can retrieve data from
storage and render a complicated page 200 times per second.

You've obviously run the test, though.  Can you describe the test
scenario, then give the numbers for Zope performing the equivalent test?

Alternatively, about the fastest Zope test I can think of is to go to a
non-existant page in the top folder.  Try that and let us know what is
Zope's hit rate on the machine you are using.

>         Can you run Zope as a precompiled application(not-interpreted
> python)?
> 
> Q: Is there a roadmap in the Zope Development to get to these performance
> levels? What are the options?

If I had to make a guess, right now the priorities are: get the portal
toolkit story straight, usability, then scalability.

It has been a long time since (over a year) since we've studied how to
tune Zope's performance.  Especially since Z2 has landed, there should
be a number of places to look for incremental improvements.

You, however, are looking for something like a three-fold performance
improvement.  Tinkering won't get us there.  Instead, something like the
caching proposals discussed on the list several months back are probably
the most the radical proposal on the table.

You can certainly do your argument, and the whole issue of performance,
a great service by proposing a test suite, a hardware platform, and what
the numbers should look like.  Until then, it is hard to know what we
are discussing.

--Paul