[Zope-dev] Serving large files

[Chris McDonough]

Could you return an iterator to the server that knows enough to open  
its own database connection?  Provide it with a callback that opens  
the connection and finds and chunks the relevant object?

- C

On Sep 17, 2006, at 3:09 PM, Sidnei da Silva wrote:

> I remember having a conversation with Jim at some point where he
> proposed a strategy for requests that could potentially take a long
> time to finish. If I recall correctly, he proposed having a separate
> ZODB connection pool.
>
> One thing that is problematic today is serving large files
> from the ZODB (ignoring the upcoming blob support).
>
> a) You can't return an iterator that reads from the ZODB, because by
>    the time the iterator is consumed the connection has already been
>    closed. I believe the iterator is not consumed in the same thread,
>    which can cause yet more issues.
>
> b) If you dump the data to a temp file and return that as a file
>    iterator, it can potentially take twice the time.
>
> c) If you use RESPONSE.write() you can break other
>    applications. ExternalEditor comes to mind.
>
> d) If you just return the file as a string you can potentially run out
>    of memory if the file is too big.
>
> I would like to be able to return an iterator that can read from the
> ZODB. That would probably benefit the WSGI integration as well. Anyone
> has ideas about how to solve this?
>
> -- 
> Sidnei da Silva
> Enfold Systems                http://enfoldsystems.com
> Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists -
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope )
>