[Zope-dev] [BUG] Background processes interfere with Zope's HTTP responses

[Dieter Maurer]

Today was again a hard Zope day.

Did you ever hear, that usually (i.e. if you do not do special
things) Zope renders a complete page before it starts
sending the result back to the client.
This implies that you should never see only half of a page.

I heard it and I saw the code that does it.
Therefore, I was convinced that this were really the case.

Then our client phoned:

      He visits a page that allows him to generate a
      newsletter. When he presses the "generate newsletter button",
      a result page is build that tells him that the newsletter
      is being generated and send is a few minutes.
    
      He reported, that this response page is build only
      half, then stops, to be completed only half a minute later,
      at the same time when the newsletter arrives.
      The effect is reproducible. If newsletter generation
      takes longer, he must wait longer for the page to complete.
    
      He is convinced that we generate the newsletter synchronously
      and let him wait until the generation is complete.
      And he is angry.
    
      I know, that the newsletter is generated in a background
      process, started in an external method with:
    
    	   os.system("gen_newsletter &")
    
      I try to reproduce the behaviour in our test environment
      and fail. I do not have to wait, until the newsletter
      generation finished. Then, I use the official service
      URL and see, I observe the same behaviour.
      The difference: in the test environment, the browser
      connects directly to ZServer; with the official
      URL, it connects via a proxy.
    
      Puzzling!


What happened?

     The background process inherits Zope's open file descriptors.
     Among them are all currently open HTTP request sockets.
     The process keeps these sockets open until it finishes.

     There are two HTTP modes:

       1. single request mode
          a new TCP connection is created for each HTTP request,
	  the request is completed, when the TCP connection
	  is closed.

       2. multi request mode
          several requests share a single TCP connection.
	  The "Content-Length" HTTP header allows
	  client and server to determine the request boundaries.

     That means: when a client uses the multi request mode,
     everything is fine. Clients, however, that use
     the single request mode wait until their connection
     is closed and can observe serious delays.

     These delays may be very difficult to explain, as
     the background process not only delays its own
     request but may delay arbitrary other requests
     that happen to be served at the same time.



How to fix the problem:

      There should be a (file) control "CloseOnExec", that tells
      Unix to automatically close the file like object upon
      an exec.
      ZServer should probably use it on each of its sockets.
      I can not see a serious application that should have
      direct access to ZServer's HTTP request socket.


I will soon file a bug report into the Collector.


Dieter