I'm using Medusa by way of Zope's ZServer. I have ZServer sitting behind an Apache proxy and am bombing it from a threaded client script that reads a web server log file as input and throws requests at the Apache server. I can vary the number of threads to adjust the load. I normally run it with five simultaneous threads. By default, the system runs briefly then (as they say), "Whammo! Blammo!", I get the following error: Unhandled exception in thread: Traceback (innermost last): File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ response._finish() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send self.refill_buffer() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 184, in refill_buffer self.producer_fifo.pop() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 255, in pop del self.list[0] IndexError: list assignment index out of range If you look at the code in asynchat.fifo.pop, the error can only be explained by the fact that two threads are modifying the fifo simultaneously, since access to self.list[0] succeeded in the statement before the statement that throws the IndexError: def pop (self): if self.list: result = (1, self.list[0]) # non-null list here del self.list[0] # empty list here else: result = (0, None) return result I modified the fifo class to lock access to its list: class fifo: def __init__ (self, list=None): self.lock = thread.allocate_lock() self.lock.acquire() if not list: self.list = [] else: self.list = list self.lock.release() def __len__ (self): self.lock.acquire() l = len(self.list) self.lock.release() return l def first (self): self.lock.acquire() item = self.list[0] self.lock.release() return item def push (self, data): self.lock.acquire() self.list.append (data) self.lock.release() def pop (self): self.lock.acquire() if self.list: result = (1, self.list[0]) del self.list[0] else: result = (0, None) self.lock.release() return result but this didn't help: Unhandled exception in thread: Traceback (innermost last): File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in __init__ response._finish() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish self.stdout.close() File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close self._channel.push(CallbackProducer(self._channel.done)) File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push if send: self.initiate_send() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send self.refill_buffer() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 175, in refill_buffer p = self.producer_fifo.first() File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 254, in first item = self.list[0] IndexError: list index out of range This error also looks like a threading problem. When refill_buffer tests self.producer.fifo, it sees something, but by the time it calls fifo.first, the list has evaporated: while 1: if len(self.producer_fifo): # non-null list here p = self.producer_fifo.first() # null list here Any ideas? Is there a way to easily turn off threading? It seems that the data structures are not properly protected. Skip Montanaro | Mojam: "Uniting the World of Music" http://www.mojam.com/ skip@mojam.com | Musi-Cal: http://www.musi-cal.com/ 518-372-5583
Skip Montanaro wrote:
I'm using Medusa by way of Zope's ZServer. I have ZServer sitting behind an Apache proxy and am bombing it from a threaded client script that reads a web server log file as input and throws requests at the Apache server. I can vary the number of threads to adjust the load. I normally run it with five simultaneous threads.
By default, the system runs briefly then (as they say), "Whammo! Blammo!", I get the following error:
(snip)
Any ideas?
This was a bug in ZServer that was fixed in the most recent Zope release (2.0 alpha 1). Basically, the push method in ZHTTPServer.zhttp_channel has a second argument that must be passed a false value when called from a thread other than the main (medusa) thread. The ZServer code that runs in separate threads now supplies this argument.
Is there a way to easily turn off threading?
Not in ZServer. It would be a ZServer bug if a problem resulted from use of threads. This particular bug has been fixed. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
participants (2)
-
Jim Fulton -
Skip Montanaro