[Zope-dev] Strange exception in ZServer/medusa/asynchat.py

[Skip Montanaro]

I'm using Medusa by way of Zope's ZServer.  I have ZServer sitting behind an
Apache proxy and am bombing it from a threaded client script that reads a
web server log file as input and throws requests at the Apache server.  I
can vary the number of threads to adjust the load.  I normally run it with
five simultaneous threads.

By default, the system runs briefly then (as they say), "Whammo! Blammo!", I
get the following error:

    Unhandled exception in thread:
    Traceback (innermost last):
      File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in __init__
	response._finish()
      File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish
	self.stdout.close()
      File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close
	self._channel.push(CallbackProducer(self._channel.done))
      File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push
	if send: self.initiate_send()
      File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send
	self.refill_buffer()
      File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 184, in refill_buffer
	self.producer_fifo.pop()
      File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 255, in pop
	del self.list[0]
    IndexError: list assignment index out of range

If you look at the code in asynchat.fifo.pop, the error can only be
explained by the fact that two threads are modifying the fifo
simultaneously, since access to self.list[0] succeeded in the statement
before the statement that throws the IndexError:

	def pop (self):
		if self.list:
			result = (1, self.list[0])	# non-null list here
			del self.list[0]		# empty list here
		else:
			result = (0, None)
		return result

I modified the fifo class to lock access to its list:

    class fifo:
	    def __init__ (self, list=None):
		    self.lock = thread.allocate_lock()
		    self.lock.acquire()
		    if not list:
			    self.list = []
		    else:
			    self.list = list
		    self.lock.release()

	    def __len__ (self):
		    self.lock.acquire()
		    l = len(self.list)
		    self.lock.release()
		    return l

	    def first (self):
		    self.lock.acquire()
		    item = self.list[0]
		    self.lock.release()
		    return item

	    def push (self, data):
		    self.lock.acquire()
		    self.list.append (data)
		    self.lock.release()

	    def pop (self):
		    self.lock.acquire()
		    if self.list:
			    result = (1, self.list[0])
			    del self.list[0]
		    else:
			    result = (0, None)
		    self.lock.release()
		    return result

but this didn't help:

    Unhandled exception in thread:
    Traceback (innermost last):
      File "/home/dolphin/skip/src/Zope/ZServer/PubCore/ZServerPublisher.py", line 97, in __init__
	response._finish()
      File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 209, in _finish
	self.stdout.close()
      File "/home/dolphin/skip/src/Zope/ZServer/HTTPResponse.py", line 235, in close
	self._channel.push(CallbackProducer(self._channel.done))
      File "/home/dolphin/skip/src/Zope/ZServer/HTTPServer.py", line 307, in push
	if send: self.initiate_send()
      File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 200, in initiate_send
	self.refill_buffer()
      File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 175, in refill_buffer
	p = self.producer_fifo.first()
      File "/home/dolphin/skip/src/Zope/ZServer/medusa/asynchat.py", line 254, in first
	item = self.list[0]
    IndexError: list index out of range

This error also looks like a threading problem.  When refill_buffer tests
self.producer.fifo, it sees something, but by the time it calls fifo.first,
the list has evaporated:

	while 1:
		if len(self.producer_fifo):		# non-null list here
			p = self.producer_fifo.first()	# null list here

Any ideas?  Is there a way to easily turn off threading?  It seems that the
data structures are not properly protected.


Skip Montanaro	| Mojam: "Uniting the World of Music" http://www.mojam.com/
skip@mojam.com  | Musi-Cal: http://www.musi-cal.com/
518-372-5583