-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I know serving large BLOBs is rather frowned up using Z2, but I have a requirement to use the zope security machinery prior to serving a blob circa 500Mb. I thought I'd just wrap it in an ExternalFile and it would simply stream to the client. In fact I was even anticipating taking advantage of chunked transfer encoding for HTTP1.1 clients. However, I'm getting out of memory exceptions thrown by the server. This would indicate that no intermediate buffer flushing is occuring at all! What am I missing? Alan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFBap9OCfroLk4EZpkRAlsPAJ9oztRXnAH8rGzeLY7+hlXKkXB4yQCfQ64I OXcuPhHJmn3V+5y/HDQEN2M= =7YAX -----END PGP SIGNATURE-----
Alan Milligan wrote:
Hi,
I know serving large BLOBs is rather frowned up using Z2, but I have a requirement to use the zope security machinery prior to serving a blob circa 500Mb.
I thought I'd just wrap it in an ExternalFile and it would simply stream to the client. In fact I was even anticipating taking advantage of chunked transfer encoding for HTTP1.1 clients.
However, I'm getting out of memory exceptions thrown by the server. This would indicate that no intermediate buffer flushing is occuring at all!
What am I missing?
I've looked at the code of the latest release of ExternalFile and I can't find the new streming interface being used at all? I'm also intrested in this, but I haven't had the time to as anyone about it, yet. Regards, Johan PS. I'm cc:ing zope@zope.org because I think this is more of an user issue. -- Johan Carlsson Tel: + 46 8 31 24 94 Colliberty Mob: + 46 70 558 25 24 Torsgatan 72 Email: johanc@easypublisher.com SE-113 37 STOCKHOLM
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Johan Carlsson wrote: | | I've looked at the code of the latest release of ExternalFile | and I can't find the new streming interface being used at all? | There's some strange stuff in the ExternalFile::FileUtils::copy_file() function whereby it writes 65565 byte chunks via the Response object, ZServer::ZServerHTTPResponse. This guy automatically sets streaming for writes with content size less than 128K. I would have expected to the ZServerHTTPResponse's stdout attribute to automatically flush on write(), but maybe not! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFBaqeiCfroLk4EZpkRAirFAJ9gMJLRhcJiJN+0vPbAVYb4iMpf4gCfZosl EXrzHTIHdsXa+C82pDND/W8= =043z -----END PGP SIGNATURE-----
Alan Milligan wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
I know serving large BLOBs is rather frowned up using Z2, but I have a requirement to use the zope security machinery prior to serving a blob circa 500Mb.
I thought I'd just wrap it in an ExternalFile and it would simply stream to the client. In fact I was even anticipating taking advantage of chunked transfer encoding for HTTP1.1 clients.
However, I'm getting out of memory exceptions thrown by the server. This would indicate that no intermediate buffer flushing is occuring at all!
What am I missing?
I don't know the ExternalFile product, but Chris McDonough's presentation to the Plone conference in Vienna should be relevant: http://plope.com/Members/chrism/2004_plone_pres Also, you might want to check out Railroad, which is a different take on the problem: http://zope.org/Members/infrae/news/railroad_products_0_2 Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
What exactly is the filestream_iterator that the FileCacheManager uses? As I understand it this is a new feature? Johan -- Johan Carlsson Tel: + 46 8 31 24 94 Colliberty Mob: + 46 70 558 25 24 Torsgatan 72 Email: johanc@easypublisher.com SE-113 37 STOCKHOLM
Johan Carlsson wrote:
What exactly is the filestream_iterator that the FileCacheManager uses?
As I understand it this is a new feature?
Of course the answer was in Chris' presentation you just sent :-) -- Johan Carlsson Tel: + 46 8 31 24 94 Colliberty Mob: + 46 70 558 25 24 Torsgatan 72 Email: johanc@easypublisher.com SE-113 37 STOCKHOLM
Johan Carlsson wrote:
What exactly is the filestream_iterator that the FileCacheManager uses?
As I understand it this is a new feature?
For Zope >= 2.7.2, ZPublisher defines an interface, ZPublisher.Iterators.IStreamIterator. An application can return an object implementing this interface to the publisher, instead of a string; if so, the publisher hands it directly to medusa, and frees up the application thread / database connection. medusa will use the iterator to send the response back to the user, without first converting it to a single big string. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
Tres Seaver wrote:
Johan Carlsson wrote:
What exactly is the filestream_iterator that the FileCacheManager uses?
As I understand it this is a new feature?
For Zope >= 2.7.2, ZPublisher defines an interface, ZPublisher.Iterators.IStreamIterator. An application can return an object implementing this interface to the publisher, instead of a string; if so, the publisher hands it directly to medusa, and frees up the application thread / database connection. medusa will use the iterator to send the response back to the user, without first converting it to a single big string.
That's actually brilliant. If I understand ExternalFile could benefit from using the IStreamIterator interface instead of RESPONSE.write? For instance, for someone that don't want to store blobs in the database temporarily :-) Johan -- Johan Carlsson Tel: + 46 8 31 24 94 Colliberty Mob: + 46 70 558 25 24 Torsgatan 72 Email: johanc@easypublisher.com SE-113 37 STOCKHOLM
Johan Carlsson wrote:
Tres Seaver wrote:
Johan Carlsson wrote:
What exactly is the filestream_iterator that the FileCacheManager uses?
As I understand it this is a new feature?
For Zope >= 2.7.2, ZPublisher defines an interface, ZPublisher.Iterators.IStreamIterator. An application can return an object implementing this interface to the publisher, instead of a string; if so, the publisher hands it directly to medusa, and frees up the application thread / database connection. medusa will use the iterator to send the response back to the user, without first converting it to a single big string.
That's actually brilliant.
If I understand ExternalFile could benefit from using the IStreamIterator interface instead of RESPONSE.write? For instance, for someone that don't want to store blobs in the database temporarily :-)
Yup, that is what it is for. Chris has released a FileCacheManager which does this for objects which *are* stored in the ZODB; I think OFS.Image.File does try to work with the FileCacheManager on that. FileCacheManager is available at: http://www.dataflake.org/software/filecachemanager/ with CVS at: http://cvs.zope.org/Products/FileCacheManager/ Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
On Mon, Oct 11, 2004 at 07:09:16PM +0200, Johan Carlsson wrote:
Tres Seaver wrote:
Johan Carlsson wrote:
What exactly is the filestream_iterator that the FileCacheManager uses?
As I understand it this is a new feature?
For Zope >= 2.7.2, ZPublisher defines an interface, ZPublisher.Iterators.IStreamIterator. An application can return an object implementing this interface to the publisher, instead of a string; if so, the publisher hands it directly to medusa, and frees up the application thread / database connection. medusa will use the iterator to send the response back to the user, without first converting it to a single big string.
That's actually brilliant.
If I understand ExternalFile could benefit from using the IStreamIterator interface instead of RESPONSE.write? For instance, for someone that don't want to store blobs in the database temporarily :-)
It's pretty easy to do. I have quick hacky patches for LocalFS-1.2-andreas, ExtFile (not ExternalFile), ExternalFolder, and CMFCore.FSFile, all here: http://www.slinkp.com/code/zopestuff/blobnotes Also some quick stupid benchmarks there. IIRC, Andreas incorporated such a patch into LocalFS-1.3-andreas or some such release. -- Paul Winkler http://www.slinkp.com
Alan Milligan wrote at 2004-10-12 00:57 +1000:
I know serving large BLOBs is rather frowned up using Z2, but I have a requirement to use the zope security machinery prior to serving a blob circa 500Mb.
Apache supports something like remote authentication (it allows you to delegate authentication to Zope). -- Dieter
participants (5)
-
Alan Milligan -
Dieter Maurer -
Johan Carlsson -
Paul Winkler -
Tres Seaver