[Zope-dev] Serving large files

[Sidnei da Silva]

I remember having a conversation with Jim at some point where he
proposed a strategy for requests that could potentially take a long
time to finish. If I recall correctly, he proposed having a separate
ZODB connection pool.

One thing that is problematic today is serving large files
from the ZODB (ignoring the upcoming blob support).

a) You can't return an iterator that reads from the ZODB, because by
   the time the iterator is consumed the connection has already been
   closed. I believe the iterator is not consumed in the same thread,
   which can cause yet more issues.

b) If you dump the data to a temp file and return that as a file
   iterator, it can potentially take twice the time.

c) If you use RESPONSE.write() you can break other
   applications. ExternalEditor comes to mind.

d) If you just return the file as a string you can potentially run out
   of memory if the file is too big.

I would like to be able to return an iterator that can read from the
ZODB. That would probably benefit the WSGI integration as well. Anyone
has ideas about how to solve this?

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214