[BUG] Background processes interfere with Zope's HTTP responses
Today was again a hard Zope day. Did you ever hear, that usually (i.e. if you do not do special things) Zope renders a complete page before it starts sending the result back to the client. This implies that you should never see only half of a page. I heard it and I saw the code that does it. Therefore, I was convinced that this were really the case. Then our client phoned: He visits a page that allows him to generate a newsletter. When he presses the "generate newsletter button", a result page is build that tells him that the newsletter is being generated and send is a few minutes. He reported, that this response page is build only half, then stops, to be completed only half a minute later, at the same time when the newsletter arrives. The effect is reproducible. If newsletter generation takes longer, he must wait longer for the page to complete. He is convinced that we generate the newsletter synchronously and let him wait until the generation is complete. And he is angry. I know, that the newsletter is generated in a background process, started in an external method with: os.system("gen_newsletter &") I try to reproduce the behaviour in our test environment and fail. I do not have to wait, until the newsletter generation finished. Then, I use the official service URL and see, I observe the same behaviour. The difference: in the test environment, the browser connects directly to ZServer; with the official URL, it connects via a proxy. Puzzling! What happened? The background process inherits Zope's open file descriptors. Among them are all currently open HTTP request sockets. The process keeps these sockets open until it finishes. There are two HTTP modes: 1. single request mode a new TCP connection is created for each HTTP request, the request is completed, when the TCP connection is closed. 2. multi request mode several requests share a single TCP connection. The "Content-Length" HTTP header allows client and server to determine the request boundaries. That means: when a client uses the multi request mode, everything is fine. Clients, however, that use the single request mode wait until their connection is closed and can observe serious delays. These delays may be very difficult to explain, as the background process not only delays its own request but may delay arbitrary other requests that happen to be served at the same time. How to fix the problem: There should be a (file) control "CloseOnExec", that tells Unix to automatically close the file like object upon an exec. ZServer should probably use it on each of its sockets. I can not see a serious application that should have direct access to ZServer's HTTP request socket. I will soon file a bug report into the Collector. Dieter
Hi Deiter, Hmm.. I just tried this with my highly experimental product (LocalProc 0.0.1) and I didn't observe the same effect that you did behind my Proxy Server. (I normally run with Rewrite... but when I got your note I tried ProxyPass and it worked on the first try.) (Good howto Anser! http://www.zope.org/Members/anser/apache_zserver/) Anyway... I ran 'sleep 20' as my local process (I checked the 'background' button on my product) and the web page returned immediately, but the process ran on in the background as I wished. The main difference I see between what you are doing and what I do.. is that I redirect stdout and stderr so that my subprocess has no open files shared with the parent process.. anyway.. you might try this to see if it suits your needs. Lightly tested on FreeBSD only.. but any unix should work the same... http://www.zope.org/Members/sspickle/LocalProc -steve
"Dieter" == Dieter Maurer <dieter@handshake.de> writes:
Dieter> Today was again a hard Zope day. Dieter> Did you ever hear, that usually (i.e. if you do not do Dieter> special things) Zope renders a complete page before it Dieter> starts sending the result back to the client. This Dieter> implies that you should never see only half of a page. Dieter> I heard it and I saw the code that does it. Therefore, I Dieter> was convinced that this were really the case. Dieter> Then our client phoned: Dieter> He visits a page that allows him to generate a Dieter> newsletter. When he presses the "generate newsletter Dieter> button", a result page is build that tells him that the Dieter> newsletter is being generated and send is a few minutes. Dieter> He reported, that this response page is build only Dieter> half, then stops, to be completed only half a minute Dieter> later, at the same time when the newsletter arrives. The Dieter> effect is reproducible. If newsletter generation takes Dieter> longer, he must wait longer for the page to complete. Dieter> He is convinced that we generate the newsletter Dieter> synchronously and let him wait until the generation is Dieter> complete. And he is angry. Dieter> I know, that the newsletter is generated in a Dieter> background process, started in an external method with: Dieter> os.system("gen_newsletter &") Dieter> I try to reproduce the behaviour in our test Dieter> environment and fail. I do not have to wait, until the Dieter> newsletter generation finished. Then, I use the official Dieter> service URL and see, I observe the same behaviour. The Dieter> difference: in the test environment, the browser connects Dieter> directly to ZServer; with the official URL, it connects Dieter> via a proxy. Dieter> Puzzling! Dieter> What happened? Dieter> The background process inherits Zope's open file Dieter> descriptors. Among them are all currently open HTTP Dieter> request sockets. The process keeps these sockets open Dieter> until it finishes. Dieter> There are two HTTP modes: Dieter> 1. single request mode a new TCP connection is Dieter> created for each HTTP request, the request is completed, Dieter> when the TCP connection is closed. Dieter> 2. multi request mode several requests share a Dieter> single TCP connection. The "Content-Length" HTTP header Dieter> allows client and server to determine the request Dieter> boundaries. Dieter> That means: when a client uses the multi request Dieter> mode, everything is fine. Clients, however, that use the Dieter> single request mode wait until their connection is closed Dieter> and can observe serious delays. Dieter> These delays may be very difficult to explain, as the Dieter> background process not only delays its own request but may Dieter> delay arbitrary other requests that happen to be served at Dieter> the same time. Dieter> How to fix the problem: Dieter> There should be a (file) control "CloseOnExec", that Dieter> tells Unix to automatically close the file like object Dieter> upon an exec. ZServer should probably use it on each of Dieter> its sockets. I can not see a serious application that Dieter> should have direct access to ZServer's HTTP request Dieter> socket. Dieter> I will soon file a bug report into the Collector. Dieter> Dieter Dieter> _______________________________________________ Zope Dieter> maillist - Zope@zope.org Dieter> http://lists.zope.org/mailman/listinfo/zope ** No cross Dieter> posts or HTML encoding! ** (Related lists - Dieter> http://lists.zope.org/mailman/listinfo/zope-announce Dieter> http://lists.zope.org/mailman/listinfo/zope-dev )
Hi Steve, Steve Spicklemire writes:
Hmm.. I just tried this with my highly experimental product (LocalProc 0.0.1) and I didn't observe the same effect that you did behind my Proxy Server..... Whether you see this effect or not depends on the HTTP processing in the chain between your browser and the server.
When all chain members use multi-request mode, then the fact that the connection is kept open longer than necessary has no visible effect. However, applications that strictly follow old HTTP 1.0 versions will use single-request mode. Then you may observe the delay. An optimal application to view the effect is "ZPublisher.ZClient".
... The main difference I see between what you are doing and what I do.. is that I redirect stdout and stderr so that my subprocess has no open files shared with the parent process. That was also my first thought. But in fact, all currently open request sockets are shared with the process as well.
Dieter
participants (2)
-
Dieter Maurer -
Steve Spicklemire