[Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

Chris McDonough chrism at plope.com
Wed Mar 24 12:57:00 EST 2004


<moved to zope-dev only>

On Wed, 2004-03-24 at 09:28, Shane Hathaway wrote:
> This sounds useful for serving content from the filesystem.
> 
> However, I'm a little concerned about this because producers must not 
> read from the object database. 

I think the easiest thing to do is to turn this into a rule. ;-)  I'd be
comfortable saying "you can't access ZODB from a producer" for now and
then thinking about allowing people to do it later.

> To work around this, you can give to the producer an object that 
> contains the chunks already loaded from the database.  If you do that, 
> though, I think you've lost all the benefit of using a producer.

IMO code that needs to read from the database shouldn't return a
producer.  Instead, it should probably continue using the RESPONSE.write
streaming protocol in the worker thread when it needs to do
producer-like things.  Returning a producer to ZPublisher seems to only
really useful when the producer's "more" generator is guaranteed to be
exceedingly cheap because as you noted it is meant to operate in the
context of the main thread.

> Another workaround might be to open a special ZODB connection just for 
> the main thread.  The producer should load objects from that connection 
> instead of the application thread's connection.  Hey, I think that would 
> solve another problem at the same time: multiple requests for the same 
> large object would use the same ZODB cache.  Wow, do you see any holes 
> in that plan?  I'm thinking that would win both speed and memory.

The time spent waiting for the code that accessed the database would
block all other asyncore operations, though, right?  We'd need to test
it, but I suspect it might be a net lose for the "multiple requests for
the same object" case because the overhead of reading from the database
cache would be realized serially for each request.  And if the object
isn't in cache, it could potentially block for quite a long time.

That said, I dunno.  Do you think it might be a win?  I guess my worry
is that the the operation of producer should be  more or less
"guaranteed" to be cheap and it seems hard to make that promise about
ZODB access, especially as the data might be coming over the wire from
ZEO.

FWIW, Jim intimated a while back that he might be interested in
providing "blob" support directly within ZODB. I can imagine an
implementation of this where maybe you can mark an object as
"blobifiable" and when you do so, the ZODB caching code writes a copy of
that object into a named file on disk during normal operations
<hand-waving goes here ;->  Then we could use a producer to spool the
file data out without ever actually reading data out of a database from
a ZODB connection; we'd just ask the connection for the filename.

- C
 





More information about the Zope-Dev mailing list