[Zope-dev] Re: [Zope3-dev] proposal: serving static content faster

[Paul Winkler]

On Fri, Apr 09, 2004 at 12:25:50AM -0400, Chris McDonough wrote:
> and 1216K (I cannot explain the large difference in results at 38322k
> across products; it appears to be product-specific).

That's easy. LocalFS, CMFCore.FSFile, and ExtFile 1.1.3 all read the
entire file into memory before sending anything back to the client.
That's why ExtFile 1.4 is so much better - it uses RESPONSE.write()
with 64k chunks.

>  Also, for this
> file size, wrapping the data in a producer is actually just more
                                    ^^^^^^^^
iterators now, remember? :-)

> overhead; products would be better off to do a direct RESPONSE.write if
> the total file size was under 128k (as evidenced by the fact that the
> iterator-based products are slower at small file sizes).

Yeah, my patches were very minimal. They could easily be extended to
only use iterators for larger content-length.
I'll do that, add a working LocalFS and another thingie I heard about 
yesterday into the mix, and do another round of benchmarks maybe tomorrow.

> It's a bit of a mystery about why retrieving data over ZEO is an order
> of magnitude slower than retrieving it from a local file storage when
> the ZEO server is actually on the same physical machine.   I'm sure
> there are lots of optimization strategies for making this faster that
> involve a lot of heavy lifting and hard thinking in a dark room. ;-) 
> While that might be fun, in the meantime, for the hoi-polloi who just
> need to get stuff done, I believe that one simple answer to this is, as
> usual, to do very aggressive caching and avoid ZODB access altogether
> for large blobs.  The ZEO caching algorithm is largely agnostic about
> what kinds of objects it caches and doesn't allow for setting a
> cache-eviction policy other than whatever its implementors consider
> optimal for the most general case.   This would be fairly difficult to
> change in the best-case circumstance for someone with deep ZODB
> knowledge.  A more effective strategy for "normal" people might be to
> contribute to the "hardening" of something like FileCacheManager, which
> prevents ZODB access in the first place and allows for fine-grained
> policies on an application level by caching blobs on client filesystems.

Fine and dandy, but I'd still really love to see what can be done
about ZEO. Caches only prevent ZODB access for stuff that's in the cache...
Sooner or later, some user will get a cache miss and that poor soul will 
see speed go down by an order of magnitude or two.  Ouch.

-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's HIGHLY TOXIC PENGUIN THING!
(random hero from isometric.spaceninja.com)