Hi! Newer Zope versions have an interesting ability to serve large files efficiently given the file name ( http://plope.com/Members/chrism/ploneconf2004/2004pres.txt ). What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content... Does Zope has anything for this? Thank you for any ideas on this topic! Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by GNU/Linux RedHat 7.3
Roman Suzi schrieb:
Hi!
Newer Zope versions have an interesting ability to serve large files efficiently given the file name ( http://plope.com/Members/chrism/ploneconf2004/2004pres.txt ).
What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content...
Does Zope has anything for this?
Thank you for any ideas on this topic!
Have a look at: http://www.dataflake.org/software/filecachemanager it implements what you are asking for. HTH Tino
On Tue, 8 Nov 2005, Tino Wildenhain wrote:
Roman Suzi schrieb:
Hi!
Newer Zope versions have an interesting ability to serve large files efficiently given the file name ( http://plope.com/Members/chrism/ploneconf2004/2004pres.txt ).
What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content...
Does Zope has anything for this?
Thank you for any ideas on this topic!
Have a look at:
http://www.dataflake.org/software/filecachemanager
it implements what you are asking for.
Thank you for the link, however, it seems that it is not what I am looking for. I do not want the file contents be read into Python program as a whole but piped to the HTTP client. This piece of FileCacheManager code give me doubts that this is how it works: """ cache_entry = FileCacheEntry(self, ob, fname, self._tempfile_path) cache_entry.write(data) """ I want something which works similar to Unix pipe but for HTTP client: this_generates_output | process_output >> http_client I do not want to store it anywhere on the FS and/or ZODB! And I certainly do not want to read the resulting file as a Python string into RAM. So, I cant understand how RAMCache or FILECacheManager is going to help me. I don't know if writing to response one chunk at a time is proper solution? Will Zope store response body or sent it right away? I am not sure that it is the later...
HTH Tino
Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by GNU/Linux RedHat 7.3
----- Original Message ----- From: "Roman Suzi" <rnd@onego.ru> To: <zope@zope.org> Sent: Tuesday, November 08, 2005 8:47 AM Subject: Re: [Zope] Serve large files efficiently from a pipe
On Tue, 8 Nov 2005, Tino Wildenhain wrote:
Roman Suzi schrieb:
Hi!
Newer Zope versions have an interesting ability to serve large files efficiently given the file name ( http://plope.com/Members/chrism/ploneconf2004/2004pres.txt ).
What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content...
Does Zope has anything for this?
Thank you for any ideas on this topic!
Have a look at:
http://www.dataflake.org/software/filecachemanager
it implements what you are asking for.
Thank you for the link, however, it seems that it is not what I am looking for. I do not want the file contents be read into Python program as a whole but piped to the HTTP client. This piece of FileCacheManager code give me doubts that this is how it works:
""" cache_entry = FileCacheEntry(self, ob, fname, self._tempfile_path) cache_entry.write(data) """
I want something which works similar to Unix pipe but for HTTP client:
this_generates_output | process_output >> http_client
I do not want to store it anywhere on the FS and/or ZODB! And I certainly do not want to read the resulting file as a Python string into RAM. So, I cant understand how RAMCache or FILECacheManager is going to help me.
I don't know if writing to response one chunk at a time is proper solution? Will Zope store response body or sent it right away? I am not sure that it is the later...
It sounds like you are looking for a streaming type of solution. You can google zope and streaming to get more info. Jonathan
On 11/8/05, Roman Suzi <rnd@onego.ru> wrote:
I don't know if writing to response one chunk at a time is proper solution? Will Zope store response body or sent it right away? I am not sure that it is the later...
The classic way to do streaming data in Zope is like so: RESPONSE.setHeader('Content-length', N) for chunk in some_iterable_yielding_data: RESPONSE.write(chunk) ... where N is the size in bytes of the data you are going to write. However, you can also publish any method that returns an implementation of IStreamIterator as shown in http://svn.zope.org/Zope/trunk/lib/python/ZPublisher/Iterators.py?view=marku... The advantage is that this has less overhead, as once your method has returned, the Zope app and ZODB are no longer involved, and your iterator and ZPublisher do all the work. This can become quite significant when you are streaming large data, more than a few hundred kB. However, your iterator *must* be able to do its work without a ZODB connection. You haven't said where the data you want to stream is going to come from, so I can't guess whether the IStreamIterator technique will work for you... -- http://www.slinkp.com
On Tue, 8 Nov 2005, Paul Winkler wrote:
On 11/8/05, Roman Suzi <rnd@onego.ru> wrote:
I don't know if writing to response one chunk at a time is proper solution? Will Zope store response body or sent it right away? I am not sure that it is the later...
The classic way to do streaming data in Zope is like so:
RESPONSE.setHeader('Content-length', N) for chunk in some_iterable_yielding_data: RESPONSE.write(chunk)
... where N is the size in bytes of the data you are going to write.
OK... Then streaming from Unix pipe can't be done in Zope...
However, you can also publish any method that returns an implementation of IStreamIterator as shown in http://svn.zope.org/Zope/trunk/lib/python/ZPublisher/Iterators.py?view=marku...
The advantage is that this has less overhead, as once your method has returned, the Zope app and ZODB are no longer involved, and your iterator and ZPublisher do all the work. This can become quite significant when you are streaming large data, more than a few hundred kB. However, your iterator *must* be able to do its work without a ZODB connection.
You haven't said where the data you want to stream is going to come from,
Sorry if it was not clear: data comes from a pipe (that is, from another application, called by popen2...
so I can't guess whether the IStreamIterator technique will work for you...
- I do not have named file... only file handler. But maybe named pipe will do the trick... However, missing content-length (and unfortunately, resulting content-length is not known) is the show-stopper.
Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by GNU/Linux RedHat 7.3
Am Dienstag, den 08.11.2005, 22:34 +0200 schrieb Roman Suzi:
On Tue, 8 Nov 2005, Paul Winkler wrote:
On 11/8/05, Roman Suzi <rnd@onego.ru> wrote:
I don't know if writing to response one chunk at a time is proper solution? Will Zope store response body or sent it right away? I am not sure that it is the later...
The classic way to do streaming data in Zope is like so:
RESPONSE.setHeader('Content-length', N) for chunk in some_iterable_yielding_data: RESPONSE.write(chunk)
... where N is the size in bytes of the data you are going to write.
OK... Then streaming from Unix pipe can't be done in Zope...
Sure it can.
However, you can also publish any method that returns an implementation of IStreamIterator as shown in http://svn.zope.org/Zope/trunk/lib/python/ZPublisher/Iterators.py?view=marku...
The advantage is that this has less overhead, as once your method has returned, the Zope app and ZODB are no longer involved, and your iterator and ZPublisher do all the work. This can become quite significant when you are streaming large data, more than a few hundred kB. However, your iterator *must* be able to do its work without a ZODB connection.
You haven't said where the data you want to stream is going to come from,
Sorry if it was not clear: data comes from a pipe (that is, from another application, called by popen2...
so I can't guess whether the IStreamIterator technique will work for you...
- I do not have named file... only file handler. But maybe named pipe will do the trick... However, missing content-length (and unfortunately, resulting content-length is not known) is the show-stopper.
What makes you think a file is always a "named file"? After all, what you get from popen2 is just a file. The only hurdle you need to take is the __len__ method on this object. So you might need a little wrapper and/or investigate a bit what ZPublisher does with it and how to prevent it (since you dont have the size, do you?) Regards Tino
On Wed, 9 Nov 2005, Tino Wildenhain wrote:
Am Dienstag, den 08.11.2005, 22:34 +0200 schrieb Roman Suzi:
On Tue, 8 Nov 2005, Paul Winkler wrote:
- I do not have named file... only file handler. But maybe named pipe will do the trick... However, missing content-length (and unfortunately, resulting content-length is not known) is the show-stopper.
What makes you think a file is always a "named file"? After all, what you get from popen2 is just a file. The only hurdle you need to take is the __len__ method on this object. So you might need a little wrapper and/or investigate a bit what ZPublisher does with it and how to prevent it (since you dont have the size, do you?)
Ok... While it doesn't seem like "right" solution, anyway thanks! That was my last option: to look how ZPublisher really do it.
Regards Tino
Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by GNU/Linux RedHat 7.3
Roman Suzi wrote at 2005-11-8 13:05 +0200:
... What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content...
You must observe the restrictions required by HTTP! HTTP 1.1 requires that if a response contains a "Content-Length" header, then it *MUST* correctly specify the size of the response entity. If it does not contain a "Content-Length", then its end *MUST* be indicated by closing the connection. Zope is a bit buggy in this respect. It always adds a "Content-Length" header, if the application did not do so. Of course, this automatically added header has (almost surely) the wrong value for dynamically created content. Currently, Zope does not support the "no Content-Length; response end by connection close" possibility. -- Dieter
On Tue, 8 Nov 2005, Dieter Maurer wrote:
Roman Suzi wrote at 2005-11-8 13:05 +0200:
... What about serving large files given file(-like) handler? It could be very beneficial sometimes, especially with dynamically generated content...
You must observe the restrictions required by HTTP!
HTTP 1.1 requires that if a response contains a "Content-Length" header, then it *MUST* correctly specify the size of the response entity. If it does not contain a "Content-Length", then its end *MUST* be indicated by closing the connection.
Zope is a bit buggy in this respect. It always adds a "Content-Length" header, if the application did not do so. Of course, this automatically added header has (almost surely) the wrong value for dynamically created content.
Currently, Zope does not support the "no Content-Length; response end by connection close" possibility.
Thank you all for explanations! While I do not necessary agree that Zope is buggy in this respect, it could still be good for the purpose of serving large files to have some by side technology, which doesnt require ZODB access. Otherwise there is a need for Apache or whatever. Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by GNU/Linux RedHat 7.3
participants (5)
-
Dieter Maurer -
Jonathan -
Paul Winkler -
Roman Suzi -
Tino Wildenhain