[Zope-dev] Serving big files thru zope?

[Chris McDonough]

I think we should work on making Zope perform better when large files
are downloaded at the PyCon sprint in March, as you suggested Paul,
FWIW.

On Mon, 2004-03-01 at 13:41, Paul Winkler wrote:
> On Mon, Mar 01, 2004 at 01:03:24PM +0100, Dario Lopez-K?sten wrote:
> > I am also wondering about the security aspects of bypassing Zope to 
> > serve files - acess to some of the files we need to serve is restricted 
> > to particular users with particular permissions, and I cannot clearly 
> > see how this would be possible to accomplish without Zope's security 
> > machinery.
> 
> Tricky.  The only solution I've heard of (but not tried)
> is to use Squid configured for disk caching and forcing
> revalidation against the proxied server (zope) before allowing
> a download from the cache.
>  
> > So, I am looking for a solution that either utilises Zope+ZServer with 
> > additional fixes or that replaces ZServer altoghether with something 
> > better (Apache, Pound, Twisted?). All this assuming of course that 
> > ZServer is not good at serving large files to many users.
> 
> I'm not sure about that anymore.
> My latest experiments with large files 
> suggest that Zope / ZServer might not be as
> bad as I thought when compared to Apache *over a network*.
> I had previously only tested on localhost and observed
> that Apache is about 10x faster than ZServer for a 40 MB file.
> I had (probably erroneously) thought that this explained
> the poor download times from Zope that I had seen in the field.
> But on a 100 Mbps local network, the download times were pretty
> close - the network becomes the bottleneck and Apache was only
> a little bit faster than Zope.
> 
> However, ZEO is a whole other story.
> The time to load a 40 MB file from ZEO and serve it,
> is about another 10x slower than plain Zope  without ZEO.
> This is painfully bad and readily apparent to users.
> But if your ZEO cache is large enough to hold all the data, 
> subsequent download times are as fast as plain Zope. 
> More investigation needed, I'm not sure about the accuracy
> or relevance of these results.  But almost certainly,
> if using large files with Zope with ZEO, you will want to configure 
> a much larger ZEO cache size than the default (20 MB). 
> 
> Also, this experiment was only for a single download at
> a time.  If you expect a lot of concurrent downloads of
> large files, better do some testing to see if Zope's large
> file download time deteriorates under load.
> I am not especially optimistic ;-)