[Zope] Serious write-performance issue with multiple threads.

[Terry Hancock]

> From: Dieter Maurer <dieter@handshake.de>
> Arnar Lundesgaard writes:
>  > ...
>  > The problem has been scarily easy to reproduce, and manifests itself
>  > whenever multiple persons are working (writing articles etc). As long as
>  > only one person is working on the site, it has the expected performance.
>  > Not very fast, a few seconds per write operation, but acceptable.
>  > However as soon as more people log in and start publishing, we often
>  > experience hangs which may last for 10-15 minutes on write operations
>  > that usually takes only a few seconds to perform. In this situation Zope
>  > doesn't respond to any requests.
> 
> Maybe, there are "ConflictError"s. When they occur, Zope
> automatically restarts the requests. I have some doubts that
> Zope does it in a way that prevents a new "ConflictError" with high
> probability. On the other hand, it is not too likely, that
> the automatic repetition increases the time by almost two orders...

This interests me, I got "ConflictError" log messages when I was
trying to use "on-the-fly" generated images in Zope. When a
page would request the same image in several different sizes,
my server would start thrashing. I never waited 10-15 minutes
to find out, but that estimate doesn't surprise me (actually
I'm surprised to find that it would recover at all -- I wonder
if mine would've done so).  This happened often, though not
quite everytime (though it was almost certain to happen if
I used a new client to view the page -- the browser's cache
must've been saving me on the other times).

I then put in some code to cause threads to wait if the object
was being used (rather clumsily -- I didn't use the threading
module, since I hadn't learned about it yet, sigh -- I still
have to clear that mess).  This reduced the problem considerably,
though not quite completely (maybe doing the thread-locking
correctly would've solved it completely -- but actually I'm
going to try solving it with a solution based on Temp Folders,
since I just upgraded Zope, so I may not find out about the
original problem).

I'm describing this in the hopes that my experience may give
you some insight into what might cause this sort of problem,
though I'm not sure if it'll help.

Good luck,
Terry

-- 
------------------------------------------------------
Terry Hancock
hancock@anansispaceworks.com       
Anansi Spaceworks                 
http://www.anansispaceworks.com 
P.O. Box 60583                     
Pasadena, CA 91116-6583
------------------------------------------------------