[Zope] Problem with TemporaryFile?

[Marc Burgauer]

Zope 2.4, Solaris 8 on Intel (Compaq), Python 2.1

We are getting increasing problems with a specific installation of Zope on a
rather busy site. (Over 400,000 hits per day, over 90GB of monthly traffic
to give an indication.) There are about 50 people editing the site,
sometimes a lot of them at the same time, including uploading PDFs of
several Megabytes. Even ore people will be downloading those PDFs.

The error messages that Zope produces is usually:

Error Type: OSError
 
Error Value: [Errno 2] No such file or directory


Traceback (innermost last):
  File /sw1/amanda/lib/python/ZPublisher/Publish.py, line 223, in
publish_module
  File /sw1/amanda/lib/python/ZPublisher/Publish.py, line 187, in publish
  File /sw1/amanda/lib/python/Zope/__init__.py, line 226, in
zpublisher_exception_hook
    (Object: ApplicationDefaultPermissions)
  File /sw1/amanda/lib/python/ZPublisher/Publish.py, line 136, in publish
  File /sw1/amanda/lib/python/ZPublisher/HTTPRequest.py, line 405, in
processInputs
  File /sw1/packages/python/lib/python2.1/cgi.py, line 517, in __init__
  File /sw1/packages/python/lib/python2.1/cgi.py, line 601, in read_multi
  File /sw1/packages/python/lib/python2.1/cgi.py, line 519, in __init__
  File /sw1/packages/python/lib/python2.1/cgi.py, line 616, in read_single
  File /sw1/packages/python/lib/python2.1/cgi.py, line 636, in read_lines
  File /sw1/packages/python/lib/python2.1/cgi.py, line 723, in make_file
  File /sw1/amanda/lib/python/tempfile.py, line 154, in TemporaryFile
OSError: (see above)

At some time, the error reported is "too many files open".

We have customised the start script:

ulimit -n 1024
python /prod/zope/z2.py -X -a212.96.140.77 -w80

The "cure" for the problem at the moment requires to:

- Stop the Server
- Remove all Data.fs.* files (but not Data.fs itself, of course)
  [possibly removing the .tmp file would do?]
- Restarting the Server

Packing the DB does not help, neither does just restarting the server alone.
Unless we remove the extra files, the problem persists. The cure lasts a few
hours at the moment.

There are q number of caches installed. Could they be the source of the
problem?

I have problems finding out how many files are opened by this instance of
Zope. (There are other Zope instances running on the same machine.) In fact,
I have not find a way to see how many files are open really. (I found how to
do it on our testing server, which is BSD and on my Mac, but not for
Solaris, which is the customer's choice of OS.) Does anybody know?

Is this an issue of the server being stressed to its limits? The server
stats (using "top" for example) indicate high load, but not very high. And
if so, would using ZEO help? The server has 2 CPUs, but we deliberately did
not compile the python to utilise both CPUs. (Operational reasons.) Is there
somewhere a HOW-TO for using ZEO on 2-CPU box?

Any suggestions how to solve this problem is very much appreciated.

Cheers

Marc