[ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used

[Claudiu Saftoiu]

Hello all,

I recently made a lot of changes to my ZODB app, and now I'm experiencing
some puzzling behavior.

Normally I have ~15 clients making requests to the server every few
seconds. Some of these
requests commit new data to the database, while others just process
existing data. I also
have a 'heartbeat' client, which access the view called '/is_alive', which
is entirely described
by this function:

    def is_alive(request): return True

Once a day or so, I will get a report from the heartbeat client that the
server is down - no heartbeat
has succeeded in the past 10 minutes. Indeed, any URL I go to on the
server, will simply not
load, but take forever - even this '/is_alive' view.

I can still SSH into the server, however. Running 'top', I see that the
server is not taking any
CPU time, and is not taking any large amount of memory. The computer itself
runs
just fine. I can even access the database with the paster shell, make
requests, commit
things, etc., without any exceptional delays. If I CTRL+C the webserver and
restart it,
things immediately work just fine.

Looking at the immediately preceding stdout, I see no hints of errors of
any kind. However,
a printline is executed on the server every time two of those clients
completes a request,
and I notice that, leading up to the current point in the stdout (where the
webserver does not
respond anymore), there are a few "Pipe is broken" messages (which happen
when, e.g.,
I go to a view with a web-browser and close the tab before the page loads),
and I notice
that requests stopped completing from one of the two clients, then from the
other. It seems
a gradual slowdown of some kind, though I'm not entirely certain.

My question is: what could possibly be causing the server to 'lock up',
even on a
simple view like 'is_alive', without using any memory or CPU? Is there some
ZODB
resource that might be getting gradually exhausted because I'm not handling
it properly?

Thanks in advance,
- Claudiu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20120419/d4ae1a9d/attachment.html>