[Zope-dev] Serving big files thru zope?

[Paul Winkler]

On Mon, Mar 01, 2004 at 01:03:24PM +0100, Dario Lopez-K?sten wrote:
> I am also wondering about the security aspects of bypassing Zope to 
> serve files - acess to some of the files we need to serve is restricted 
> to particular users with particular permissions, and I cannot clearly 
> see how this would be possible to accomplish without Zope's security 
> machinery.

Tricky.  The only solution I've heard of (but not tried)
is to use Squid configured for disk caching and forcing
revalidation against the proxied server (zope) before allowing
a download from the cache.
 
> So, I am looking for a solution that either utilises Zope+ZServer with 
> additional fixes or that replaces ZServer altoghether with something 
> better (Apache, Pound, Twisted?). All this assuming of course that 
> ZServer is not good at serving large files to many users.

I'm not sure about that anymore.
My latest experiments with large files 
suggest that Zope / ZServer might not be as
bad as I thought when compared to Apache *over a network*.
I had previously only tested on localhost and observed
that Apache is about 10x faster than ZServer for a 40 MB file.
I had (probably erroneously) thought that this explained
the poor download times from Zope that I had seen in the field.
But on a 100 Mbps local network, the download times were pretty
close - the network becomes the bottleneck and Apache was only
a little bit faster than Zope.

However, ZEO is a whole other story.
The time to load a 40 MB file from ZEO and serve it,
is about another 10x slower than plain Zope  without ZEO.
This is painfully bad and readily apparent to users.
But if your ZEO cache is large enough to hold all the data, 
subsequent download times are as fast as plain Zope. 
More investigation needed, I'm not sure about the accuracy
or relevance of these results.  But almost certainly,
if using large files with Zope with ZEO, you will want to configure 
a much larger ZEO cache size than the default (20 MB). 

Also, this experiment was only for a single download at
a time.  If you expect a lot of concurrent downloads of
large files, better do some testing to see if Zope's large
file download time deteriorates under load.
I am not especially optimistic ;-)

-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's THE PREPUBESCENT PROFESSOR!
(random hero from isometric.spaceninja.com)