Problem uploading large files
Hi. I thought I should send this to the Zope-dev-list, as there was little response to my mail on the main list. I have a peculiar problem: On our site, people should be able to upload big files (MP3s) that are stored outside the ZODB. This is done via an upload form and HTTP POST. We tested the upload forms, everything worked fine, until people started uploading via slow modems. At this point, I should explain our setup: We run ZServer at port 9090, and use Apache as a proxy to forward all requests to mp3.no:80 to login.mp3.no:9090. We also use SiteAccess in combination with this. So, the problem is: When uploading large files, ie. files that exceed 30 minutes or so of upload time, things stop working. It isn't file size, it is only dependant on the time used. We've tested with, and eliminated the following possible unknown variables: 1. Different modems (from 14.4Kbps to 56Kbps) 2. Different providers 3. Different browsers - Netscape, IE 4. Different ports (upload via both mp3.no:80 and login.mp3.no:9090 directly have been attempted) 5. Different locations (upload to the ZODB and the external python method have been attempted) 6. Raising the timeout values of the Apache server have been attempted So, I'm quite puzzled. Is there a timeout value in ZServer/Medusa that can be tweaked? Or is it possible that the error lies elsewhere? An example error message is: -- Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request POST /artists/boj/prox/add_mp3. Reason: Error reading from remote server -- The Apache logs show: [Mon Mar 6 14:07:39 2000] [error] [client 129.177.138.178] (54)Connection reset by peer: ap_bgets() - proxy receive - Error reading from remote server login.mp3.no (length -1) So, do you have any suggestions as to what I might try next? As mentioned, everything works fine if the connection is fast enough. Is there a timeout I can raise the value of? This is in a pretty critical phase of our development, we should have been "live" last Friday, so we're a bit pressed for time here. Any help would be greatly appreciated. And please CC me, as I'm not on the dev-list (I'm not worthy! :) Yours puzzlingly, Alexander Limi http://mp3.no
On Tue, 7 Mar 2000 01:46:14 +0100, Alexander Limi wrote:
On our site, people should be able to upload big files (MP3s) that are stored outside the ZODB. This is done via an upload form and HTTP POST.
So, the problem is: When uploading large files, ie. files that exceed 30 minutes or so of upload time, things stop working. It isn't file size, it is only dependant on the time used.
So, I'm quite puzzled. Is there a timeout value in ZServer/Medusa that can be tweaked?
Yes, there is a timeout value buried in ZServer. Go look in the file /usr/share/zope/ZServer/HTTPServer.py, class zhttp_channel. There is a class variable 'zombie_timeout', set to 100 minutes. Also look in /usr/share/zope/ZServer/medusa/http_server.py for 'zombie' code, *both* in the http_channel class and status_handler class. In the medua/http_server.py class, that timeout is *** 30 minutes ***. When a ZServer channel has been in existence for that long, it is automatically closed, in the method kill_zombies() of that class. This may be your problem. -Jeff Rush (Zope Consultant)
On Tue, 7 Mar 2000 01:46:14 +0100, Alexander Limi wrote:
So, I'm quite puzzled. Is there a timeout value in ZServer/Medusa that can be tweaked?
Yes, there is a timeout value buried in ZServer. Go look in the file /usr/share/zope/ZServer/HTTPServer.py, class zhttp_channel. In the medua/http_server.py class, that timeout is *** 30 minutes ***.
When a ZServer channel has been in existence for that long, it is automatically closed, in the method kill_zombies() of that class. This may be your problem.
Initial testing shows no difference, but I will do some more testing tomorrow. Meanwhile, would it be a sensible thing to test the setup with apache/pcgi instead of ZServer directly, or is that irrelevant? Thanks for the help so far. Alexander.
-----Original Message----- From: Jeff Rush [mailto:jrush@taupro.com]
Yes, there is a timeout value buried in ZServer. Go look in the file /usr/share/zope/ZServer/HTTPServer.py, class zhttp_channel. There is a class variable 'zombie_timeout', set to 100 minutes. Also look in /usr/share/zope/ZServer/medusa/http_server.py for 'zombie' code, *both* in the http_channel class and status_handler class. In the medua/http_server.py class, that timeout is *** 30 minutes ***.
Well, I tweaked the values to 120 minutes, but it still won't work. This time, it timed out after about 40 minutes. Apache showed the following: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request POST /artists/boj/dibugg/add_mp3. Reason: Error reading from remote server And Zope printed the following to the console: 2000-03-11T00:09:50 ERROR(200) ZServer uncaptured python exception, closing channel <zhttp_channel connected 212.71.72.22:20245 at 116c50 channel#: 1575 requests:1> (exceptions.AttributeError:data [/opt/zope-mp3/ZServer/medusa/asyncore.py|poll|59] [/opt/zope-mp3/ZServer/medusa/asyncore.py|handle_read_event|291] [/opt/zope-mp3/ZServer/medusa/asynchat.py|handle_read|103] [/opt/zope-mp3/ZServer/medusa/http_server.py|collect_incoming_data|439] [/opt/zope-mp3/ZServer/medusa/http_server.py|collect_incoming_data|143] [/opt/zope-mp3/ZServer/HTTPServer.py|collect_incoming_data|300]) And I logged everything with a HTTP proxy on my machine. Junkbuster put out this: junkbstr: accept connection ... scan: POST /artists/boj/zoff/add_mp3 HTTP/1.0 scan: Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */* scan: Referer: http://mp3.no/artists/boj/zoff/upload crunch! scan: Accept-Language: no scan: Content-Type: multipart/form-data; boundary=---------------------------7d01ae2c904ac scan: Accept-Encoding: gzip, deflate scan: User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) default scan: Host: mp3.no scan: Content-Length: 10689875 scan: Proxy-Connection: Keep-Alive crunch! scan: Pragma: no-cache scan: Cookie: dtpref_rows="20"; dtpref_cols="65" crunch! scan: Authorization: Basic Ym9qOmdhZ2E= junkbstr: GPC mp3.no/artists/boj/zoff/add_mp3 junkbstr: connect to: mp3.no ... OK OK junkbstr: accept connection ... scan: GET / HTTP/1.0 scan: Accept: */* scan: Accept-Language: no scan: Accept-Encoding: gzip, deflate scan: User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0) default scan: Host: mp3.no scan: Proxy-Connection: Keep-Alive crunch! scan: Cookie: dtpref_rows="20"; dtpref_cols="65" crunch! junkbstr: GPC mp3.no/ junkbstr: connect to: mp3.no ... OK scan: HTTP/1.0 502 Proxy Error scan: Date: Sat, 11 Mar 2000 13:36:49 GMT scan: Server: Apache/1.3.9 (Unix) PHP/3.0.13 mod_ssl/2.4.5 OpenSSL/0.9.4 scan: Connection: close crunch! scan: Content-Type: text/html scan: HTTP/1.0 502 Proxy Error scan: Date: Sat, 11 Mar 2000 13:37:40 GMT scan: Server: Apache/1.3.9 (Unix) PHP/3.0.13 mod_ssl/2.4.5 OpenSSL/0.9.4 scan: Connection: close crunch! scan: Content-Type: text/html --------------------------------- So, the questions are: 1. Will I solve this temorarily by switching to Apache/PCGI/FCGI? 2. What is wrong with medusa? Have anybody else experienced the same thing? 3. Does Junkbuster give you enough info? Or should I use some other logging product? (I can send the whole log if required. It's 13K.) Thanks in advance, Alexander.
participants (2)
-
Alexander Limi -
Jeff Rush