Charlie, Thanks for the insights, this should proove useful. About the momory, to be honest, I'm not convinced that's it. Yesterday for instance, Zope showed that behavior TWICE in one day, the first time it recovered itself, the second, I restarted. But, throughtout all this, there was PLENTY of swap available (My swap space is 2GB). I've NEVER seen the swap usage go beyond ~150MB actually.
From what you're saying, I think this issue of one thread per request might be the problem. I'm going to start by tackling this. It's the easiest and cheapest way for now. The issue of long requests (People adding data/content) mixed with the heavy load of the server (i.e. requests per second), and it is forseeable that this would happen.
The issue of the large images is allways in my mind, but since that only happens occasionally, and in batches (It was going on this week), I'm willing to risk it for the increased convenience. While on the topic, I'm just curious as to why the processes/threads are structured as such: |-python2.1,11334) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | `-python2.1,11337) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | `-python2.1,11353) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | |-python2.1,11354) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | |-python2.1,11355) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | |-python2.1,11356) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | `-python2.1,11357) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D One talks to one talks to one talks to four ... Also, if I'm seeing it in PS, is it really a thread ? Or a process ? ps -m isn't implemented on unix, therefore can't show threads, so what am I seeing ??? Thanks for the help! J.F. -----Original Message----- From: Charlie Reiman [mailto:creiman@kefta.com] Sent: Thursday, October 24, 2002 12:30 PM To: Jean-Francois.Doyon@CCRS.NRCan.gc.ca; zope@zope.org Subject: RE: [Zope] Problems with restarts, memory usage, DB connections, FastCGI ?? Help!
So far I've been unable to track down WHY.
Uhh....
One thing that also happened just once was that the entire server ended up running out of memory!
Sounds like you tracked it down just fine. You are clearly exhasting your physical memory, the system is swapping like crazy, then it may eventually slog through the piggie bits to resume normal operation. You need more more memory and more threads. There are real limits as to what any system can do reasonably. It sounds like just bumping up the number of DB connections won't do anything for you. Arguably, the right thing for you to do is _decrease_ the number of available connections until you get more hardware (assuming the DB connections are used exclusively to upload files). Your system clearly can't maintain the load you are asking it to handle. It would be better to deny upload service to some folks than let the entire system collapse. Look into buying more memory and going with ZEO to split up the load. Read the man page for ps (assuming you're on unix) and top so you can track the amount of memory getting sucked down. Consider an emergency stop-gap solution where only one user may upload a file at a time. Increasing the number of theads (but not connections) may help. Some background here on why this is so painful: Zope starts with N theads (7, I think) for handling requests. A request is handled by a single thread and that thread is locked up until the request terminates. Python has no facility to preemptively stop thread and Zope never times out requests. As a result, it only takes a few long requests to completely lock up a zope server. If you consider that you could have 4 users uploading large files over slow links, you can see how you are in deep trouble even before considering the memory issue. So start by upping the number of threads (-t 15 maybe?) in your start script. On top of that, you are uploading large files which as you already noted zope doesn't handle really well. This topic comes up a lot here. Zope stores objects in memory. These file objects are going to be huge and rapidly consume all your physical memory. So in addition to starving your zope's thread count, you are now swapping like crazy. If I were you, I'd look into doing the file uploads out-of-band (ie, around zope) or buying lots and lots of RAM.