Download Large Files in Zope
Hi, I've just installed Zope 2.8 with Apache 2 and I made a download library (a CMS site is too much for me and I can't find a simple "download area" product, so I've done it!) Now, when I click on a large .exe file link (href) it seems it load all data before prompt for download and the zope service memory usage grow until 1/2 Gb. How can i manage downloads & uploads? Thank you in advance Illorca ------------------------------------------------------ Con Prometeo prestiti senza spese fino a 31.000 Euro! http://click.libero.it/webnation06marz07
----- Original Message ----- From: <jbusato@libero.it> To: "zope" <zope@zope.org> Sent: Tuesday, March 06, 2007 2:49 PM Subject: [Zope] Download Large Files in Zope
Hi, I've just installed Zope 2.8 with Apache 2 and I made a download library (a CMS site is too much for me and I can't find a simple "download area" product, so I've done it!) Now, when I click on a large .exe file link (href) it seems it load all data before prompt for download and the zope service memory usage grow until 1/2 Gb. How can i manage downloads & uploads?
This is one possible solution: http://www.infrae.com/download/tramline Jonathan
I've just installed Zope 2.8 with Apache 2 and I made a download library (a CMS site is too much for me and I can't find a simple "download area" product, so I've done it!) What about railorad: http://www.infrae.com/products/railroad
How can i manage downloads & uploads? In general it is bad idea to hold and serve large files directly from ZODB. This should be done by Apache etc.
-- Maciej Wisniowski
What about railorad: http://www.infrae.com/products/railroad I meant tramline: http://www.infrae.com/products/tramline
-- Maciej Wisniowski
Large files aren't store in ZODB but in a shared folder connected with LOCALFS. Maybe Zope load a file in zodb before begin the download? (Are there other storage product that work only in zope - without plone?) Tramline is an Apache add-on but my problem still remain without Apache so this product can help me after I resolved the Zope issue. I think that if you'll use tramline you'll have no 'Zope issue' anymore.
-- Maciej Wisniowski
Looking LocalFS specs I find the issue:
This version of LocalFS takes advantanges of Zope 2.7.1 <http://www.zope.org/Members/hewei/zoperpms/zope2.7> 's new /filestream_iterator/ feature and is believed to server large files better than before. However, my initial testing showed that *it still doesn't prevent the whole file being loaded into memory*. When I opened a URL pointing to a mpg file served by LocalFS, the file was loaded into memory and then immediately freeed twice.
I think that's the problem... load 300Mb into memory.... Maybe I look for other fs products. AFAIK it uses tempfile for serving such files. This allows Zope resources to be not blocked while returning big amounts of data. Although creation of tempfile or something like that (I'm not sure how this works in details) may still be slow.
In general serving static files is best done by servers like Apache and I think this shouldn't be done by application server like Zope (possibly until it has blobs support). I don't understand why you can't use tramline. It may be connected with zope easily. It gives you very fast filesystem storage. Even more, seems for me that it is designed to do exacly what you're asking for. You say that tramline may help you after resolving zope issues, but I don't get it... By using tramline you will have no files in Zope (ZODB or LocalFS) so you will have no problem to eliminate. But maybe I missed something in your's use case. -- Maciej Wisniowski
I want to use localfs instead of Apache becouse: 1) upload is done by web too
tramline supports uploads and downloads
2) upload/download operations cause db operation (files names and descriptions are stored in sql server) With tramline: "The only thing the appserver sees is a unique identifier of the uploaded file". I don't know whether it is a file name, but seems that while uploading/downloading you may do some writes into db.
Good luck! -- Maciej Wisniowski
Jary Busato schrieb:
Maciej Wisniowski ha scritto:
Large files aren't store in ZODB but in a shared folder connected with LOCALFS. Maybe Zope load a file in zodb before begin the download? (Are there other storage product that work only in zope - without plone?) Tramline is an Apache add-on but my problem still remain without Apache so this product can help me after I resolved the Zope issue.
I think that if you'll use tramline you'll have no 'Zope issue' anymore.
Looking LocalFS specs I find the issue:
This version of LocalFS takes advantanges of Zope 2.7.1 <http://www.zope.org/Members/hewei/zoperpms/zope2.7> 's new /filestream_iterator/ feature and is believed to server large files better than before. However, my initial testing showed that *it still doesn't prevent the whole file being loaded into memory*. When I opened a URL pointing to a mpg file served by LocalFS, the file was loaded into memory and then immediately freeed twice.
I think that's the problem... load 300Mb into memory.... Maybe I look for other fs products.
However download of files off ZODB isnt such a big problem (you should mount an extra storage for the files to avoid overly cache trashing) but the performance is more then acceptable. ZODB/Filestorage is currely only a problem if you regulary update those files (or objects anyway) because it would add a complete new version. Regards Tino Wildenhain
participants (5)
-
Jary Busato -
jbusato@libero.it -
Jonathan -
Maciej Wisniowski -
Tino Wildenhain