[ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

[Jeremy Hylton]

On 5/29/05, Shane Hathaway <shane at hathawaymix.org> wrote:
> > Would a multi thread ZEO server improve anything here? Especially
> > with concurrent access?
> 
> It's possible.  Although ZEO talks over the network using async sockets,
> it reads files synchronously, so I suspect it will frequently sit around
> doing nothing for 10 ms, waiting for the disk to read data.  If your ZEO
> server has a load of 1.0 or more but low CPU usage, this is likely
> happening.  The easiest way to overcome this is to buy gigabytes of RAM
> for the ZEO server--ideally, enough gigabytes to hold your whole database.

A related problem is that the ZEO cache on the client is on disk, too.
 You may end up waiting for a disk seek to get it off disk on the
client.  If you've got it in memory on the server and if the ZEO
protocol were more efficient, that would be a drag.

> Also, the design of ZEO clients tends to serialize communication with
> the ZEO server, so the throughput between client and server is likely to
> be limited significantly by network latency.  "ping" is a good tool for
> measuring latency; 1 ms is good and .1 ms is excellent.  There are ways
> to tune the network.  You can also reduce the effects of network latency
> by creating and load balancing a lot of ZEO clients.

It's really too bad that ZEO only allows a single outstanding request.
 Restructuring the protocol to allow multiple simulatenous requests
was on the task list years ago, but the protocol implementation is so
complex I doubt it will get done :-(.  I can't help but think building
on top of an existing message/RPC layer would be profitable.  (What's
twisted's RPC layer?)  Or at least something less difficult to use
than asyncore.

Jeremy