[ZODB-Dev] instrumenting a ZEO server?

[Alan Runyan]

On Wed, Aug 26, 2009 at 8:08 AM, Chris Withers<chris at simplistix.co.uk> wrote:
> Hi All,
>
> I'm wondering if there's any way to get a ZEO server to log about how
> it's performing?
> (eg: how many connections, how many clients are waiting, where they're
> waiting, etc)
>
> I'm trying to debug a problem where app servers are being dropped out of
> a load balancer pool for being unresponsive, and I think they're waiting
> on the ZEO server so I'm trying to get some stats on the ZEO server side
> as to what's going on...
>
> As far as the clients go, DeadlockDebugger is providing some insights:
>
>   File "/opt/Zope-2.9/lib/python/ZEO/ClientStorage.py", line 749, in loadEx
>     self._lock.acquire()    # for atomic processing of invalidations
>
> I assume this is the client waiting for the storage server's lock?
> Is this the ZEO server's single threadedness or is this a lock between
> threads on the app server?
>   File "/opt/Zope-2.9/lib/python/ZEO/ClientStorage.py", line 769, in loadEx
>     data, tid, ver = self._server.loadEx(oid, version)
>   File "/opt/Zope-2.9/lib/python/ZEO/ServerStub.py", line 192, in loadEx
>     return self.rpc.call("loadEx", oid, version)
>   File "/opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py", line 531, in call
>     r_flags, r_args = self.wait(msgid)
>   File "/opt/Zope-2.9/lib/python/ZEO/zrpc/connection.py", line 638, in wait
>     asyncore.poll(delay, self._singleton)
>   File "/usr/local/lib/python2.4/asyncore.py", line 122, in poll
>     r, w, e = select.select(r, w, e, timeout)
>
> What does this mean? I'm guessing it's the actual transfer of data from
> the storage server to the app server, right?

What version of ZODB are you using?  If you have zeo on in debugging
what you can see is if there are outstanding clients waiting to write
while a transaction is already writing. It says something like "# clients
blocked while transaction in progress" in zeo server log files.

How many connections:
iirc zrpc is connection oriented so you should have # of application clients
connected to ZEO server.  Or it may be # of clients * # of ZODB threads.

How many clients are waiting:
You should be able to see this in zeo log files

Where they're waiting:
deadlockdebugger or products.signalstack will show what the clients are doing.

may want to strace the client and server to see if there is long
pauses of inactivity.
maybe your running up against a real ZODB bug that was fixed in later releases?
look at ZODB 3.8 branch NEWS.txt

good luck
alan