[Zope] Problems with restarts, memory usage, DB connections, FastCGI ?? Help!

[Jean-Francois.Doyon@CCRS.NRCan.gc.ca]

Charlie,

Thanks for the insights, this should proove useful.

About the momory, to be honest, I'm not convinced that's it.  Yesterday for
instance, Zope showed that behavior TWICE in one day, the first time it
recovered itself, the second, I restarted.  But, throughtout all this, there
was PLENTY of swap available (My swap space is 2GB). I've NEVER seen the
swap usage go beyond ~150MB actually.

>From what you're saying, I think this issue of one thread per request might
be the problem. I'm going to start by tackling this. It's the easiest and
cheapest way for now.  The issue of long requests (People adding
data/content) mixed with the heavy load of the server (i.e. requests per
second), and it is forseeable that this would happen.

The issue of the large images is allways in my mind, but since that only
happens occasionally, and in batches (It was going on this week), I'm
willing to risk it for the increased convenience.

While on the topic, I'm just curious as to why the processes/threads are
structured as such:

 |-python2.1,11334) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w -
-p - -f - -D 
  |   `-python2.1,11337) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc
-w - -p - -f - -D 
  |       `-python2.1,11353) /usr/local/Zope-2.5.0/z2.py -F
/tmp/fcgi/zope.soc -w - -p - -f - -D 
  |           |-python2.1,11354) /usr/local/Zope-2.5.0/z2.py -F
/tmp/fcgi/zope.soc -w - -p - -f - -D 
  |           |-python2.1,11355) /usr/local/Zope-2.5.0/z2.py -F
/tmp/fcgi/zope.soc -w - -p - -f - -D 
  |           |-python2.1,11356) /usr/local/Zope-2.5.0/z2.py -F
/tmp/fcgi/zope.soc -w - -p - -f - -D 
  |           `-python2.1,11357) /usr/local/Zope-2.5.0/z2.py -F
/tmp/fcgi/zope.soc -w - -p - -f - -D 

One talks to one talks to one talks to four ... Also, if I'm seeing it in
PS, is it really a thread ? Or a process ? ps -m isn't implemented on unix,
therefore can't show threads, so what am I seeing ???

Thanks for the help!
J.F.

-----Original Message-----
From: Charlie Reiman [mailto:creiman@kefta.com]
Sent: Thursday, October 24, 2002 12:30 PM
To: Jean-Francois.Doyon@CCRS.NRCan.gc.ca; zope@zope.org
Subject: RE: [Zope] Problems with restarts, memory usage, DB
connections, FastCGI ?? Help!


> So far I've been unable to track down WHY.

Uhh....

> One thing that also happened just once was that the entire server ended up
> running out of memory!

Sounds like you tracked it down just fine. You are clearly exhasting your
physical memory, the system is swapping like crazy, then it may eventually
slog through the piggie bits to resume normal operation.

You need more more memory and more threads. There are real limits as to what
any system can do reasonably. It sounds like just bumping up the number of
DB connections won't do anything for you. Arguably, the right thing for you
to do is _decrease_ the number of available connections until you get more
hardware (assuming the DB connections are used exclusively to upload files).
Your system clearly can't maintain the load you are asking it to handle. It
would be better to deny upload service to some folks than let the entire
system collapse.

Look into buying more memory and going with ZEO to split up the load. Read
the man page for ps (assuming you're on unix) and top so you can track the
amount of memory getting sucked down. Consider an emergency stop-gap
solution where only one user may upload a file at a time. Increasing the
number of theads (but not connections) may help.

Some background here on why this is so painful:

Zope starts with N theads (7, I think) for handling requests. A request is
handled by a single thread and that thread is locked up until the request
terminates. Python has no facility to preemptively stop thread and Zope
never times out requests. As a result, it only takes a few long requests to
completely lock up a zope server.

If you consider that you could have 4 users uploading large files over slow
links, you can see how you are in deep trouble even before considering the
memory issue. So start by upping the number of threads (-t 15 maybe?) in
your start script.

On top of that, you are uploading large files which as you already noted
zope doesn't handle really well. This topic comes up a lot here. Zope stores
objects in memory. These file objects are going to be huge and rapidly
consume all your physical memory. So in addition to starving your zope's
thread count, you are now swapping like crazy.

If I were you, I'd look into doing the file uploads out-of-band (ie, around
zope) or buying lots and lots of RAM.