Hi, Sidnei da Silva wrote:

I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool.

One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support).

a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues.

b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time.

c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind.

d) If you just return the file as a string you can potentially run out of memory if the file is too big.

I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this?

I wonder whether this statement is true: Any connection handed out after the connection breaks the ACID compliance of the request. I think this is because: When finishing the actual work we have to either commit or abort the running transaction. In the time until we set up another transaction, there already might have been a change to the database by a third transaction, making the request neither isolated nor atomic. My statement is probably only true for all solutions not paying attention to the statement. ;) Christian -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - ct@gocept.com - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development

On Sun, 2006-17-09 at 16:09 -0300, Sidnei da Silva wrote:

b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time.

Potentially, yes... but from a practical standpoint I believe this will never happen. Writing to a tempfile over any sort of local storage (even a local network) will be *much* faster than downloading the file itself. If this is indeed a problem perhaps it's a deployment issue where you should ensure the temp dir is on a fast local volume to make sure this never happens. As it stands right now this seems to be the only sensible approach to returning a large file. - Rocky -- Rocky Burt ServerZen Software -- http://www.serverzen.com News About The Server (blog) -- http://www.serverzen.net

FileCacheManager does the copy-to-file-and-serve-as-iterator dance for you. http://www.dataflake.org/software/filecachemanager Stefan On 17. Sep 2006, at 21:09, Sidnei da Silva wrote:

One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support).

a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues.

b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time.

-- It doesn't necessarily do it in chronological order, though. --Douglas Adams

Could you return an iterator to the server that knows enough to open its own database connection? Provide it with a callback that opens the connection and finds and chunks the relevant object? - C On Sep 17, 2006, at 3:09 PM, Sidnei da Silva wrote:

I remember having a conversation with Jim at some point where he proposed a strategy for requests that could potentially take a long time to finish. If I recall correctly, he proposed having a separate ZODB connection pool.

One thing that is problematic today is serving large files from the ZODB (ignoring the upcoming blob support).

a) You can't return an iterator that reads from the ZODB, because by the time the iterator is consumed the connection has already been closed. I believe the iterator is not consumed in the same thread, which can cause yet more issues.

b) If you dump the data to a temp file and return that as a file iterator, it can potentially take twice the time.

c) If you use RESPONSE.write() you can break other applications. ExternalEditor comes to mind.

d) If you just return the file as a string you can potentially run out of memory if the file is too big.

I would like to be able to return an iterator that can read from the ZODB. That would probably benefit the WSGI integration as well. Anyone has ideas about how to solve this?

-- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )