[ZODB-Dev] server stops handling requests - nowhere near 100% CPU or Memory used
Marius Gedminas
marius at gedmin.as
Fri Apr 20 14:58:01 UTC 2012
On Fri, Apr 20, 2012 at 10:06:40AM -0400, Claudiu Saftoiu wrote:
> >
> > Ah, so it seems that, as I leave the server running longer & longer,
> > more & more threads are taken up with a `.recv()` call. I think one of
> > my clients opens requests and does not read them/close them.
Does not write, rather.
> > Eventually
> > all the threads are blocking in that fashion.
> >
> > I will fix my clients. But, is there a server-side fix to this (again,
> > using Repoze.BFG)?
> >
>
> Additional info: in particular the blocked threads' stack dumps look like
> this:
> Thread 140605868680960:
> File "/usr/lib/python2.6/threading.py", line 504, in __bootstrap
> self.__bootstrap_inner()
> File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
> self.run()
> File "/usr/lib/python2.6/threading.py", line 484, in run
> self.__target(*self.__args, **self.__kwargs)
> File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 878, in worker_thread_callback
> runnable()
> File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 1052, in <lambda>
> lambda: self.process_request_in_thread(request, client_address))
> File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 1068, in process_request_in_thread
> self.finish_request(request, client_address)
> File "/usr/lib/python2.6/SocketServer.py", line 322, in finish_request
> self.RequestHandlerClass(request, client_address, self)
> File "/usr/lib/python2.6/SocketServer.py", line 617, in __init__
> self.handle()
> File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 442, in handle
> BaseHTTPRequestHandler.handle(self)
> File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
> self.handle_one_request()
> File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 437, in handle_one_request
> self.wsgi_execute()
> File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 287, in wsgi_execute
> self.wsgi_start_response)
> File
> "/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/connector.py",
> line 21, in __call__
> result = self.next_app(environ, start_response)
> File
> "/home/tsa/env/lib/python2.6/site-packages/repoze/zodbconn/cachecleanup.py",
> line 25, in __call__
> return self.next_app(environ, start_response)
> File
> "/home/tsa/env/lib/python2.6/site-packages/repoze/retry/__init__.py", line
> 65, in __call__
> chunk = original_wsgi_input.read(rest)
> File "/home/tsa/env/lib/python2.6/site-packages/paste/httpserver.py",
> line 474, in read
> data = self.file.read(length)
> File "/usr/lib/python2.6/socket.py", line 377, in read
> data = self._sock.recv(left)
>
> Was my assessment of the situation accurate?
It appears so. Here's the code of repoze.retry:
https://github.com/repoze/repoze.retry/blob/master/repoze/retry/__init__.py
Line numbers differ somewhat; the original_wsgi_input.read(rest) is on
line 71 on that github page right now. It does look like this is the
bit that reads the HTTP request from the client.
> > Something to time out the connection after 60 seconds or so if nothing has
> > happened?
You're using paste.httpserver. It has support for killing hung threads:
http://pythonpaste.org/modules/httpserver.html#paste.httpserver.server_runner
has a list of options.
You could also put nginx in front, to take care of ill-behaved HTTP
clients.
Marius Gedminas
--
Never be afraid to tell the world who you are.
-- Anonymous
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20120420/f0d5fe3a/attachment.sig>
More information about the ZODB-Dev
mailing list