[Zope] Problems with restarts, memory usage, DB connections, FastCGI ?? Help!

Charlie Reiman creiman@kefta.com
Thu, 24 Oct 2002 09:29:44 -0700


> So far I've been unable to track down WHY.

Uhh....

> One thing that also happened just once was that the entire server ended up
> running out of memory!

Sounds like you tracked it down just fine. You are clearly exhasting your
physical memory, the system is swapping like crazy, then it may eventually
slog through the piggie bits to resume normal operation.

You need more more memory and more threads. There are real limits as to what
any system can do reasonably. It sounds like just bumping up the number of
DB connections won't do anything for you. Arguably, the right thing for you
to do is _decrease_ the number of available connections until you get more
hardware (assuming the DB connections are used exclusively to upload files).
Your system clearly can't maintain the load you are asking it to handle. It
would be better to deny upload service to some folks than let the entire
system collapse.

Look into buying more memory and going with ZEO to split up the load. Read
the man page for ps (assuming you're on unix) and top so you can track the
amount of memory getting sucked down. Consider an emergency stop-gap
solution where only one user may upload a file at a time. Increasing the
number of theads (but not connections) may help.

Some background here on why this is so painful:

Zope starts with N theads (7, I think) for handling requests. A request is
handled by a single thread and that thread is locked up until the request
terminates. Python has no facility to preemptively stop thread and Zope
never times out requests. As a result, it only takes a few long requests to
completely lock up a zope server.

If you consider that you could have 4 users uploading large files over slow
links, you can see how you are in deep trouble even before considering the
memory issue. So start by upping the number of threads (-t 15 maybe?) in
your start script.

On top of that, you are uploading large files which as you already noted
zope doesn't handle really well. This topic comes up a lot here. Zope stores
objects in memory. These file objects are going to be huge and rapidly
consume all your physical memory. So in addition to starving your zope's
thread count, you are now swapping like crazy.

If I were you, I'd look into doing the file uploads out-of-band (ie, around
zope) or buying lots and lots of RAM.