[ZODB-Dev] write performance

[Jeremy Hylton]

I just took a quick look at the statistics and realized that we
probably need to see the profile data from the server side, too.  From
the client side we see that ZEO spends a lot of time (10.543 seconds
cumulative) waiting for synchronous ZEO calls to return.

It looks like most of that time is spent waiting for tpc_finish().  On
the server side of FileStorage, tpc_finish() copies data from a
transaction temporary file to the primary storage file and calls
sync().  It also sends invalidation messages to all the other clients.

I would expect the storage cost to be dominated by the cost of data
movement inside the storage server, but can't be sure without seeing
the data.

Do you have a lot of clients connected?  If there are many clients, it
costs more to send invalidations.

Have you tried running the ZEO server with -O?  That will eliminate a
bunch of log calls, which might shave a second from the 20 being
consumed during the commit.  5% isn't too bad for just -O.

It will also be interesting to fire off a helper thread that just ran
the asyncore mainloop.  I don't have any idea if it will make a
difference, but there are a lot of calls to poll() the way you're
currently running.  There might be fewer if you had a mainloop running
somewhere.

Jeremy