[Zope] Zope Performance Methodology?

[Bill Anderson]

"Michael A. Alderete" wrote:
> 
> I have scanned through the ZAG, the various how-tos and tips on Zope.org,
> and searched the archives of this mailing list, but I haven't found
> anything which puts into writing a concrete methodology for optimizing
> performance of Zope.
> 
> If there is one and I'm just too stupid to find it, I'd love to get a
> pointer. If there isn't one I'll write it, if people will send me
> information and teach me the zen of optimal Zope performance.
> 
> Here are the kinds of things I would expect to see in this kind of document:
> 
> 1. Performance-oriented Platforms
> 
>  * Which platform (hardware / OS combination) is the best for Zope?:
>    - if money is no object

Alpha 21264, high MHz, lots of ram and CPUs, running on Fiberchannel
RAID with a quad port gigabit ethernet card, running Tru64 Unix (pardon
me whilst I wipe the drool from my keyboard ...).

>    - if money is the main object
 See below ;-)

>    - if it has to run on a cheap PC, 'cause that's what people have
>    - etc.

Linux on a celeron, or dual celeron.

> 
>  Questions this needs to answer:
> 
>   - Do multiple processors help?

Yes, if the OS can scale well with them. IE, CPU >=2 is a waste with
Windows9X.

>   - How much RAM?

As much as you can. Seriously. If you intend to run a really
high-traffic site, and or a big ZODB, you can never (again, with the
exception of win9x) have too much RAM.

>   - Disk subsystem recommendations / tradeoffs

RAID where possible. Personally, I favor RAID 0+1 (aka RAID 10, iirc) or
RAID 5.

RAID 1 is mirroring, providing redundancy, tradeoff: write speed is
slower (writing to two disks, not one), but under some conditions can be
nealy 2x as fast for reads (interleving of the reading)

RAID 0 is basically like srtiping, you get N drives, and they become one
big drive, with roughly N(Drive Speed) performance. Tradeoff: one drive
fails and you better have a backup.

IDE is cheap, but IMO, is a bad choice for a production server.

>   - Solaris vs. Linux vs. FreeBSD [vs. Windows NT?] ...

(I now have a 10 month old daughter sleeping on my shoulder, so please
forgive any typos  thatt slip through :-)

Here, money can be an issue. Obviously, your chepest bet would be a PC
ruuning one of the free unix-likes (personally, I'd avoid solaris
whenever possibly, YMMV).  From my experience, Linux (and likely *BSD)
will run circles around an NT machine of the same configuration for
servers.


>   - etc...
> 
> 2. Performance-oriented Configurations
> 
>  * What is the right way to set up Zope?:

Will vary by personal tastes and needs.

>    - which web server to use

From what I have heard, run at great spread from NES.

I don't -yet- have Zope+Zeus experience (nor have I heard any), so I
can't comment on it.

Straight ZServer, if all you need is Zope, go for it. I did this (and
for some still do) for my sites.

Apache: Well, the thing to remember about Apache was it was designed for
extensibility, not speed. Not to imply it is a slouch by any means. With
Apache, there are many things you can do outside of Zope.

Roxen: my current favourite. Roxen is very, very cool.

Another option I don't know whether or not has been tried, would be to
run a squid/apache proxy as a front-machine as an accellerator. Anyone
done this yet? I am considering trying a similiar setup with mirroring
Roxen servers.

>    - which CGI configuration to use

Depends on needs. Assuming here you are talking about using Zope
w/another server. I understand the FastCGI support is maturing nicely,
but have not yet tried it out (anyone done it with Roxen?). Standard
PCGI works well enough for me so far.

>    - which ZODB storage system to use

Depends on how deep you want to get into code :-)

>    - etc.
> 
>  Questions this needs to answer:
> 
>   - ZServer vs. Apache [vs. IIS?] ...

Needs dependent. IMO, for a site that will be pure Zope, any other
server (outside the accellerated option) is a waste.

>   - PCGI vs. FastCGI [vs. ??]

IMO, this boils down to stability, and traffic.

>   - ZODB in file system vs. database

Part of this is dependant on the filesystem. yes, I refer to the x86 2Gb
file limit in the standard Linux install. There are patches to dissolve
the issue, an done can use an Alpha machine, where the issue is
non-extant.

>   - which database?

Another personal and needs based choice. Many have (IMO, correctly)
indicated that MySQl would be inadequate for using as a ZODB
replacement, due to transactions being non-extant. As far as otherwise,
an RDMS really needs to be a choice brought about by a proper needs
analysis.

>   - etc.?
> 
> 3. Performance Tuning
> 
> I don't know where to even start asking questions here, but I read a
> cursory mention that zope.org was recently sped up considerably by playing
> with some caching parameters, or something like that. So detailed tuning
> suggestions are needed for:
> 
>   - operating system
>   - web server
>   - Zope
>   - ??

I haven't felt the need yet to do much tuning here, but I do have some
general suggestions.
IIRC, the caching tuning you mentioned on zope.org was an increase in
the amount of objects kept in cache. IMO, You need to correlate how much
RAM you can spare, and how much you want kept in Zope's cache. if you
have limited RAM, you probably don't want to have a high number. A
machine in swap is not a snappy machine.

IMo, most of the truly useful gains in a clean slate would be the
hardware and OS setup in general. More RAM, more MHz, and fast disks
will go a long way to making your serve rmore responsive. On that note,
there is a little tip for Linux users I ran into somewhere (where was
it?) about doing a chmod +t on an executable file would keep it in
memory. I have not tried this with Zope yet, but it does work nicely on
other executables (learned this from a Sybase HPUX tip IIRC)

Can't really say much about the rest, yet. I think it is to general to
really do justice with (an d this is getting long :-).

I will mention that I have or am running zope on the following
platforms:

o Alpha 4/233 48M Ram (RAM was a problem here)
o Intel p133, 128M RAM (multiple Zope instances ~4, no complaints)
o Dual Celeron 450s, 256MB RAM, RAID0 (~50MB/s) (FAST!)
o Dual PII 300s, 256MB RAM, HP AutoRAID cabinet (~80MB/s) (FAST!)
o HP C3000 Visualize, 1.5GB RAM (FAST!)
o HP N-Class 4+ processors, several Gig of RAM (Sit down, strap in, hold
on and shut up!)


I haven't run any standardized benchmarks against them yet, but I plan
on doing so for the Dual CPU machines, and the two HP babies. On the
latter, I am checking to see if my boss will let me publish my results
on them externally (wouldn't know why they wouldn't ... :).

Anybody have a preferred webserver benchmark that is appropriate for
Zope?

If desired, I could turn this into a howto ...

Bill

-- 
In flying I have learned that carelessness and overconfidence are 
usually far more dangerous than deliberately accepted risks. 
          -- Wilbur Wright in a letter to his father, September 1900