[Zope] To stream or not to stream

[Toby Dickenson]

On Mon, 7 Jan 2002 11:07:39 -0500, Casey Duncan <c.duncan@nlada.org>
wrote:

>Quick question for you HTTP gurus out there.
>
>I am writing a product that marshals Zope objects into an XML data =
stream. In=20
>implementing the publication of this data through Zope, I have two =
options:
>
> - Stream the data to the client as it is returned by the marshaler in=20
>chunks. I do not know the Content-Length in advance, so I cannot set =
that=20
>HTTP header.

Using RESPONSE.write, ZPublisher will use chunked encoding if it can,
or Connection:Close if not.

> - Get the entire XML document as a string, set the Content-Length and =
let=20
>ZPublisher take care of it.

If your method simply returns the XML document as a string then you
dont need to set Content-Length; ZPublisher will do it for you.
However, it will all stay in memory until the last byte is
transmitted. Also, the first byte will not be transmitted until the
last byte has been calculated.

>I have initially chosen the former as it seems to be the best use of =
memory=20
>resources for what could be an arbitrarily large data stream.

You will want to have a look at the HTTP implementation. Without a
manually-set Content-Length header, RESPONSE.write doesnt know that
the response is big and will keep every untransmitted chunk in memory.

I used to have some patches in the Collector that used to make this a
little better behaved, but I dont have time to rebuild those right
now. I suggest you have a look at the implementation and hack it a
little.



Toby Dickenson
tdickenson@geminidataloggers.com