Storing many large objects in many folders
Hi, I need to store many large objects (30+MB video files) in a way that Zope can serve them; plus, these files should be distributed over a large folder tree /mytree/... in my Zope system. We've considered the following options: - use LocalFS; the problem is that I'd like to avoid having a LocalFS object in every folder of /mytree/..., and I neither want to put all the files into one filesystem directory; so I'd need some solution that automatically maintains the logical structure of /mytree/... in the filesystem directory. - use BLOBs on a RDBMS; unfortunately, I couldn't get the DBMS connection we are using (DB2 via SQLRelay) to work correctly with BLOBs; I've considered http://www.zope.org/Members/peterb/oracle_lobs, but without success. Are there any viable alternatives apart from these two? What do other people use when they have to maintain large numbers of large objects? Thanks in advance, Christoph -- -- Christoph Schmitz <cschm@REMOVEMEgmx.de>
Christoph Schmitz wrote:
Hi,
I need to store many large objects (30+MB video files) in a way that Zope can serve them; plus, these files should be distributed over a large folder tree /mytree/... in my Zope system.
I really wouldn't do this. Someone correct me if I'm wrong, but serving such big files blocks one of only 4 (per-default) of zopes worker threads. This means that 4 concurrent downloads make it impossible for someone else to get to your server, because there are no idle threads left to serve. Put them on the filesystem, serve them through an external server and just store the metadata (including the path to the file) in the ZODB. There is a product called mmmanager (or something) which does something similar IIRC like you seem to want. cheers, oliver
On Monday 09 December 2002 4:53 pm, Oliver Bleutgen wrote:
Christoph Schmitz wrote:
Hi,
I need to store many
how many is 'many'?
large objects (30+MB video files) in a way that Zope can serve them; plus, these files should be distributed over a large folder tree /mytree/... in my Zope system.
I really wouldn't do this.
Someone correct me if I'm wrong,
sure ;-)
but serving such big files blocks one of only 4 (per-default) of zopes worker threads. This means that 4 concurrent downloads make it impossible for someone else to get to your server,
No. Worker threads run as fast as they can, and are not constrained by client connection speed. Wirker threads transfer the response to the medusa thread, which spools the reponse back to the client. It can handle thousands of connections.
Toby Dickenson writes:
...
but serving such big files blocks one of only 4 (per-default) of zopes worker threads. This means that 4 concurrent downloads make it impossible for someone else to get to your server,
No. Worker threads run as fast as they can, and are not constrained by client connection speed. Wirker threads transfer the response to the medusa thread, which spools the reponse back to the client. It can handle thousands of connections. The worker thread needlessly copies the file and stores its content in a temporary file.
Something that can be avoided when the file is directly served from the file system. Dieter
On Mon, Dec 09, 2002 at 03:58:28PM +0100, Christoph Schmitz wrote:
Hi,
I need to store many large objects (30+MB video files) in a way that Zope can serve them; plus, these files should be distributed over a large folder tree /mytree/... in my Zope system.
We've considered the following options:
- use LocalFS; the problem is that I'd like to avoid having a LocalFS object in every folder of /mytree/..., and I neither want to put all the files into one filesystem directory;
Not necessary. If /mytree/ *is* a LocalFS, and it's pointed to e.g. $INSTANCE_HOME/mytree/ on the filesystem, then any subdirectories of $INSTANCE_HOME/mytree will map to folderish objects in /mytree. LocalFS does have some annoying limitations, though... e.g. objects in LocalFS cannot store properties. No title, nothing. The current UI makes you think that you can add properties, but you'll quickly discover that they are just discarded. -- Paul Winkler http://www.slinkp.com "Welcome to Muppet Labs, where the future is made - today!"
Christoph Schmitz writes:
I need to store many large objects (30+MB video files) in a way that Zope can serve them; plus, these files should be distributed over a large folder tree /mytree/... in my Zope system.
We've considered the following options:
- use LocalFS; the problem is that I'd like to avoid having a LocalFS object in every folder of /mytree/..., and I neither want to put all the files into one filesystem directory; so I'd need some solution that automatically maintains the logical structure of /mytree/... in the filesystem directory.
- use BLOBs on a RDBMS; unfortunately, I couldn't get the DBMS connection we are using (DB2 via SQLRelay) to work correctly with BLOBs; I've considered http://www.zope.org/Members/peterb/oracle_lobs, but without success. You may look for "ExtFile", maybe with a streaming "index_html" (i.e. you use "RESPONSE.write").
However, I would follow the advice of others and see, whether you cannot serve these objects directly from the file system. Dieter
participants (5)
-
Christoph Schmitz -
Dieter Maurer -
Oliver Bleutgen -
Paul Winkler -
Toby Dickenson