[Zope] ZSERVER, THREADS, PERFORMANCE & FUTURE PERFORMANCE

[Michel Pelletier]

Sam Gendler wrote:
> 
> > This time I changed the concurrency (-c , -t is timelimit) on zeus_benc side
> >  -c 1  - 41 TPS
> >  -c 2  and up ~ 35 TPS and max 4 threads showing as being used in top.
> >
> > the top lines of top looked like this on -t 30 -c 100
> 
> If even going to 2 threads on the client side causes a 20% degradation in
> performance, there has got to be something wrong with the threading model in the
> server.

I suspect this is due to the global python interpreter lock.  Two
threads content for the same locks throughout two simultaneous requests,
thus resulting in the 20% loss.

However, the goal of threading Zope was NOT to increase requests
per/sec, the goal was to allow simultaneous requests to happen, in Zope
1.x, requests were serialized and one requests wait for (possibly long
running) previous requests.  Threading allows many requests to happen
simultaneously, and means that individual threads can call 'blocking'
methods without worrying about locking up the whole site.  We knew it
wouldn't increase performance, because of the global interpreter lock. 
Further, the python lock prevents the threaded method from scaling to
anything more than a handful of threads.

This whole issue is sidesteped by ZEO.  ZEO is a client/server protocol
between Zope clients and a storage server.  The Zope clients are the
'frontend' web servers that connect to a common backend.  Thus, you can
have a whole farm of cheap, single processor Zope machines running the
ZEO client (which is a Zope process, ie a server to the browser client)
that connect to a common backend database which you can dump as much
money into as you want to make it an incredibly reliable storage system
(and, using something like Coda could allow you to have multiple backend
storage servers, but this have never been tested and might cause alot of
write-write contention).

Further, because ZEO clients are seperate *processes* and not threads,
multiple ZEO clients can run concurrently on multi-processor machines. 
ZEO is an incredibly flexible archetecture, and I think it will prove to
scale well beyond anything else out there right now.

> Make sure you send the largest possible packets.  If the client receives a 'small
> packet' (I can't remember the criteria, although 1/2 maximum segment size rings a
> bell), it will delay the ACK of that packet for 200ms while it waits for data to go
> back the other way.  Since this is a web client, there is no data to go back the
> other way, so there is a 200ms delay after every small packet from the server.  On
> most systems, you can set TCP_NODELAY to off, but linux doesn't support this, last
> time I checked.  Many servers started out sending each HTTP header as a different
> packet, or else they send the headers as one packet and the body as another.  In
> either case, you will usually trigger this behaviour.  Simply buffering the output
> is enough to prevent the problem.

It would be good to analyze ZServer (medusa) to see if this is done.
 
-Michel