[ZODB-Dev] Zope or Zeo hanging up sockets

[Chris McDonough]

> Perhaps I see if I can get it to do the same thing running in the
foreground
> with -D and put an import pdb and pdb.set_trace() and then sit and
wait to
> see if anything can get through, even when not busy.  Or am I
definately
> forcing Zope into single thread mode in that instance?   I use pdb
all the
> time when debuggin my products and methods and I think I remember
seeing
> other requests being honoured.  Anyway, not sure how non-linear that
would
> end up being.

Pretty nonlinear.  This is probably not a problem that can be debugged
by using pdb because it's only exhibited under load.

>  How long does it take if there is
> > no load whatsoever?
>
> about 1-2 seconds at the most when not loaded, and now that I look
at
> things, up to 20 seconds when very loaded, hmm, that could be a
clue, I
> should get more emperical evidence for that.

Yes... you can use the "big M log" to gather more evidence.  To turn
it on, use the following to invoke the Zope "start" (or z2.py) script:

./start -M detailed.log

A logfile named "detailed.log" will be created in the var directory.
You can get various kinds of reports on the data in written to this
log by using the utilities/requestprofiler.py script.  Reading the
requestprofiler.py source is also a good way to understand what the
raw -M log entries signify.

You can also cause additional system information like memory usage,
cpu usage, etc. to be written into this log every so often if you use
the "ForensicLogger" product
(http://www.zope.org/Members/mcdonc/Products/ForensicLogger/ForensicLo
gger-0.3-tar.gz/view); it only works on Linux currently, however.
It's wildly useful if you think that you're constrained by CPU
bandwidth or memory, because it allows you to associate requests in
the big M log directly with periods of high load or lack of resources.

>  How often does it get called?
>
> When those particular people are using their client application,
about every
> 40 seconds.

That's pretty often... you can probably make most of the pain go away
by just speeding it up.  I'm interested in the pathological behavior,
though, so if you're also interested, it would help to try to be able
to recreate the symptom on demand, even if the solution to your
problem this time is to just "not do that" ;-)

>  Can you speed it
> > up?
> >
>
> Yes I can speed it up, and I can also change the scripts on a dev
server so
> that I can pretend to be these clients.  BTW : I have a completely
identical
> setup on a dev server, so I really can hack into things without
ruining the
> live one, if you have things you want me to try that is.

It'd be great to be able to cause the "one request hangs all other
threads" behavior to be easily replicable.  I dont know how reasonable
that will be, though.  It might be a good idea to turn on Big M
logging on the production machine(s) because that's where the load is,
and try to capture the problem "in the wild".  Then whittle away at
the actual speed of the method until the problem diminishes and/or
goes away.  I can probably be of most help to you by helping to "read
the runes" of the Big M logs as you make changes.

Note that this is only tangentially related to ZODB proper, so we'll
probably want to move this conversation to the general Zope maillist
so as not to torture the folks here. ;-)

- C