Hello, I need to serve big files from zope - PDF, movies, large PS-files, Students lab-results, etc. We recently moved from LocalFS to Dirstorage to manage ZEO-ability in a clustered enviromnet). Quite a while ago, I remember having read somewhere on the lists that there is a pacth/fix for Zope/ZServer that optimises the way zope serves large files. Is the current state of ZServer such that it is suboptimal for serving large files? if so, a) what is the problem with the current implementation? b) is there a fix available? if not b) then I am willing to start looking into it. I need this for my unis student web-site. Thanx, /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech.
Dario, The best solution for me always has been to serve such files statically from Apache, bypassing ZServer altogether. In fact I have added support for static URLs to my version of ExtFile <http://zope.org/Members/shh/ExtFile>. ZEO clusters will need a comon disk share (SAMBA, NFS) for this to work. HTH, Stefan --On Montag, 01. März 2004 08:24 +0100 Dario Lopez-Kästen <dario@ita.chalmers.se> wrote:
Hello,
I need to serve big files from zope - PDF, movies, large PS-files, Students lab-results, etc. We recently moved from LocalFS to Dirstorage to manage ZEO-ability in a clustered enviromnet).
Quite a while ago, I remember having read somewhere on the lists that there is a pacth/fix for Zope/ZServer that optimises the way zope serves large files.
Is the current state of ZServer such that it is suboptimal for serving large files? if so,
a) what is the problem with the current implementation? b) is there a fix available?
if not b) then I am willing to start looking into it. I need this for my unis student web-site.
Thanx,
/dario
-- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/
Stefan H. Holek wrote:
Dario,
The best solution for me always has been to serve such files statically from Apache, bypassing ZServer altogether. In fact I have added support for static URLs to my version of ExtFile <http://zope.org/Members/shh/ExtFile>.
ZEO clusters will need a comon disk share (SAMBA, NFS) for this to work.
Hi, thanks for the reply. I am a bit hesitant to revert to filesystem based storage at this point - we used to use LocalFS but switched from it in favor of DirectoryStorage. I am also wondering about the security aspects of bypassing Zope to serve files - acess to some of the files we need to serve is restricted to particular users with particular permissions, and I cannot clearly see how this would be possible to accomplish without Zope's security machinery. So, I am looking for a solution that either utilises Zope+ZServer with additional fixes or that replaces ZServer altoghether with something better (Apache, Pound, Twisted?). All this assuming of course that ZServer is not good at serving large files to many users. Thanks, /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech.
I am also wondering about the security aspects of bypassing Zope to serve files - acess to some of the files we need to serve is restricted to particular users with particular permissions, and I cannot clearly see how this would be possible to accomplish without Zope's security machinery.
Same situation here. Is there any way to provide authentication information for Apache from Zope (which works at least with cookie based auth)? Regards, Sandor
you could play with http://puggy.symonds.net/~srp/stuff/mod_auth_remote/ and have apache auth off zope. cheers, -kapil On Mon, 2004-03-01 at 11:06, Sandor Palfy wrote:
I am also wondering about the security aspects of bypassing Zope to serve files - acess to some of the files we need to serve is restricted to particular users with particular permissions, and I cannot clearly see how this would be possible to accomplish without Zope's security machinery.
Same situation here. Is there any way to provide authentication information for Apache from Zope (which works at least with cookie based auth)?
Regards, Sandor
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Mon, Mar 01, 2004 at 01:03:24PM +0100, Dario Lopez-K?sten wrote:
I am also wondering about the security aspects of bypassing Zope to serve files - acess to some of the files we need to serve is restricted to particular users with particular permissions, and I cannot clearly see how this would be possible to accomplish without Zope's security machinery.
Tricky. The only solution I've heard of (but not tried) is to use Squid configured for disk caching and forcing revalidation against the proxied server (zope) before allowing a download from the cache.
So, I am looking for a solution that either utilises Zope+ZServer with additional fixes or that replaces ZServer altoghether with something better (Apache, Pound, Twisted?). All this assuming of course that ZServer is not good at serving large files to many users.
I'm not sure about that anymore. My latest experiments with large files suggest that Zope / ZServer might not be as bad as I thought when compared to Apache *over a network*. I had previously only tested on localhost and observed that Apache is about 10x faster than ZServer for a 40 MB file. I had (probably erroneously) thought that this explained the poor download times from Zope that I had seen in the field. But on a 100 Mbps local network, the download times were pretty close - the network becomes the bottleneck and Apache was only a little bit faster than Zope. However, ZEO is a whole other story. The time to load a 40 MB file from ZEO and serve it, is about another 10x slower than plain Zope without ZEO. This is painfully bad and readily apparent to users. But if your ZEO cache is large enough to hold all the data, subsequent download times are as fast as plain Zope. More investigation needed, I'm not sure about the accuracy or relevance of these results. But almost certainly, if using large files with Zope with ZEO, you will want to configure a much larger ZEO cache size than the default (20 MB). Also, this experiment was only for a single download at a time. If you expect a lot of concurrent downloads of large files, better do some testing to see if Zope's large file download time deteriorates under load. I am not especially optimistic ;-) -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's THE PREPUBESCENT PROFESSOR! (random hero from isometric.spaceninja.com)
I think we should work on making Zope perform better when large files are downloaded at the PyCon sprint in March, as you suggested Paul, FWIW. On Mon, 2004-03-01 at 13:41, Paul Winkler wrote:
On Mon, Mar 01, 2004 at 01:03:24PM +0100, Dario Lopez-K?sten wrote:
I am also wondering about the security aspects of bypassing Zope to serve files - acess to some of the files we need to serve is restricted to particular users with particular permissions, and I cannot clearly see how this would be possible to accomplish without Zope's security machinery.
Tricky. The only solution I've heard of (but not tried) is to use Squid configured for disk caching and forcing revalidation against the proxied server (zope) before allowing a download from the cache.
So, I am looking for a solution that either utilises Zope+ZServer with additional fixes or that replaces ZServer altoghether with something better (Apache, Pound, Twisted?). All this assuming of course that ZServer is not good at serving large files to many users.
I'm not sure about that anymore. My latest experiments with large files suggest that Zope / ZServer might not be as bad as I thought when compared to Apache *over a network*. I had previously only tested on localhost and observed that Apache is about 10x faster than ZServer for a 40 MB file. I had (probably erroneously) thought that this explained the poor download times from Zope that I had seen in the field. But on a 100 Mbps local network, the download times were pretty close - the network becomes the bottleneck and Apache was only a little bit faster than Zope.
However, ZEO is a whole other story. The time to load a 40 MB file from ZEO and serve it, is about another 10x slower than plain Zope without ZEO. This is painfully bad and readily apparent to users. But if your ZEO cache is large enough to hold all the data, subsequent download times are as fast as plain Zope. More investigation needed, I'm not sure about the accuracy or relevance of these results. But almost certainly, if using large files with Zope with ZEO, you will want to configure a much larger ZEO cache size than the default (20 MB).
Also, this experiment was only for a single download at a time. If you expect a lot of concurrent downloads of large files, better do some testing to see if Zope's large file download time deteriorates under load. I am not especially optimistic ;-)
On Mon, 2004-03-01 at 13:41, Paul Winkler wrote:
However, ZEO is a whole other story. The time to load a 40 MB file from ZEO and serve it, is about another 10x slower than plain Zope without ZEO. This is painfully bad and readily apparent to users. But if your ZEO cache is large enough to hold all the data, subsequent download times are as fast as plain Zope. More investigation needed, I'm not sure about the accuracy or relevance of these results. But almost certainly, if using large files with Zope with ZEO, you will want to configure a much larger ZEO cache size than the default (20 MB).
It really would be good to get to the bottom of the bad ZEO performance for loading an object the first time. I won't have time to do it until after the ZODB 3.3 release, but I think it's a worthwhile effort to understand what ZEO is doing. You're right, too, that a large ZEO cache makes a big difference. If you've got the disk space, setting it several hundred MB shouldn't be a problem. Jeremy
Jeremy Hylton wrote at 2004-3-1 14:01 -0500:
... It really would be good to get to the bottom of the bad ZEO performance for loading an object the first time. I won't have time to do it until after the ZODB 3.3 release, but I think it's a worthwhile effort to understand what ZEO is doing.
I instrumented "ClientStorage" to get precise timings for ZEO operations. The obtained times are not bad: When the data file was not yet in the operating system's cache, loading an object took between 3 to 30 ms with very high variance. After the relevant parts of the data file have been cached by the OS, time was between 1 to 3 ms per object, almost independent from object size. I obsevered loading of about 1.600 objects between several hundred bytes and several kilobytes in size mostly index data for a ZCatalog "FieldIndex". I had disabled both the ZODB cache as well as the ZEO client cache for my measurement. ZEO client and server have been on the same machine (local communication) I interpreted these observations in the following way: * time is dominated by the disc access * time is largely independent of the object size (if the object is in the OS disc cache) * ZEO overhead is about 1 to 3 ms per object. -- Dieter
On Tue, Mar 02, 2004 at 07:06:31PM +0100, Dieter Maurer wrote: (snip)
I interpreted these observations in the following way:
* time is dominated by the disc access
* time is largely independent of the object size (if the object is in the OS disc cache)
* ZEO overhead is about 1 to 3 ms per object.
Hmm. I wondered if ZEO overhead per-object was significant, but this seems not too bad. If I read OFS/Image.py correctly, the pdata chunks are 65536 bytes. So a 40 MB file would split up into about 640 chunks, so ZEO overhead by your calculations should be roughly 1-2 seconds. This does not account for the order of magnitude difference I am seeing (initial load from filestorage: 0:03.87, initial load from zeo: 0:33.54). -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's PROTO FLASH TROMBONIST! (random hero from isometric.spaceninja.com)
Dario Lopez-Kästen wrote at 2004-3-1 08:24 +0100:
... Is the current state of ZServer such that it is suboptimal for serving large files? if so,
a) what is the problem with the current implementation?
In order to reduce memory consumption and decouple Zope from ZServer (and the response delivery), a large file is spooled via a temporary. This means, that the file content is read from ZODB, stored in a temporary file and then delivered from this temporary file. At least, when your client has a high bandwidth to your server, this serving through a temporary file is a waste. Serving static content via Python/Zope is about one order of magnitude slower than serving the same static content via Apache.
b) is there a fix available?
Probably, but it will be a lot of work: When your client (or its connection) is slow, you want to spool large files indirectly (to free the scares Zope worker thread soon). Only when you know your client is fast, would you want to serve directly. An adaptive scheme is possible but much more demanding than the simple schema (to use a temporary file for all sufficiently large content). You could code ZServer in "C" and speed it up thereby. Maybe "psyco" helps a bit? -- Dieter
On Monday, March 1, 2004, at 02:43 PM, Dieter Maurer wrote:
In order to reduce memory consumption and decouple Zope from ZServer (and the response delivery), a large file is spooled via a temporary.
This means, that the file content is read from ZODB, stored in a temporary file and then delivered from this temporary file.
At least, when your client has a high bandwidth to your server, this serving through a temporary file is a waste.
I found that putting my /var partition (the default place for zope's temp files) on a very fast SCSI disk with plenty of space (GB's) improved the situation tremendously....
participants (9)
-
Chris McDonough -
Dario Lopez-Kästen -
Dieter Maurer -
Jeremy Hylton -
Kapil Thangavelu -
Marc Lindahl -
Paul Winkler -
Sandor Palfy -
Stefan H. Holek