Re: [Zope] Should I use ZODB or not ?
Jerome Alet writes:
Our needs are to store hundreds, and possibly thousands of MPEG2 and MPEG4 (generally short) video movies, MP3 audio files, JPEG images, etc...
Altough most of the files will be in the 1 Mb -> 10 Mb range, some could be larger, e.g. some Gigabytes.
For each file we also want to keep some data: author, subject, encoding, description, keywords, domain, etc...
I see 5 solutions:
1 - store both files and their associated datas in ZODB 2 - store files in the filesystem and datas in ZODB 3 - store files in the filesystem and datas in an RDBMS (use Zope only for the presentation of files and datas) 4 - store files in ZODB and datas in an RDBMS. 5 - don't use Zope at all because it's not appropriate.
Do not store the files in Zope or serve them with Zope. A standard web server (e.g. Apache) is at least one order of magnitude faster than Zope for static content (such as multi media files). This is essential for you large file objects. If you have up to a few thousand objects and they do not change too often, you can put them (the meta data) into the ZODB. Otherwise, put them in a RDBMS. I would use the RDBMS whenever I have one or I have more than a few hundred objects or I would need access to the data from legacy processes, too. Dieter
Hello I am curious how do you all think LocalFS would perform for such a big multimedia application.? http://www.zope.org/Members/jfarr/Products/LocalFS Use Zope for managing meta-data, or mayeb even better Zope + a fast DB tool for the metadata part. Let Zope do its holy best for the 'virtual-object interfacing' jobs, let the others do their thing.. My understanding was that using the LocalFS is as fast as you can get really and that any file size limits are to do with the OS, disks RAM, caching etc, not Zope or anything else. Plus one is free to scale or mix and match zope with other tools and systems, as it develops without compromising all the work which has gone into the base data's filestructure. Might be much easier to develop too.. The data would not be locked into Zope or for that matter any RDBMS 'solution'. One coud performance tests, develop optimizations and determin along the way when and where to apply them. No headaches as Zope goes through upgrades either because one is not taxing it too much. Can anyone see what are the faults with this approach? I have been wondering about such what plan to use and if or with Zope for large animated map datasets project. This would use Flash SWF, and also SVG [XMLbased Scalable Vector Graphics] The SWF can be merged into Quicktime allowing for multiple tracks. This is a powerful combinatoin where one needs to package trainging, presentation. Video though big contains a huge amount of 'information' which is lost in all other infromation system: body language, human language, tone of voice, inflexion, accent, emphasis, mood, lighting, context, scale, enviromnental conditions. etc. - Jason ________________________________________________________________ Jason CUNLIFFE = NOMADICS.(Interactive Art and Technology).Design Director ----- Original Message ----- From: Dieter Maurer <dieter@handshake.de> To: Jerome Alet <alet@unice.fr> Cc: <zope@zope.org> Sent: Tuesday, September 19, 2000 3:44 PM Subject: Re: [Zope] Should I use ZODB or not ?
Jerome Alet writes:
Our needs are to store hundreds, and possibly thousands of MPEG2 and MPEG4 (generally short) video movies, MP3 audio files, JPEG images, etc...
Altough most of the files will be in the 1 Mb -> 10 Mb range, some could be larger, e.g. some Gigabytes.
For each file we also want to keep some data: author, subject, encoding, description, keywords, domain, etc...
I see 5 solutions:
1 - store both files and their associated datas in ZODB 2 - store files in the filesystem and datas in ZODB 3 - store files in the filesystem and datas in an RDBMS (use Zope only for the presentation of files and datas) 4 - store files in ZODB and datas in an RDBMS. 5 - don't use Zope at all because it's not appropriate.
Do not store the files in Zope or serve them with Zope. A standard web server (e.g. Apache) is at least one order of magnitude faster than Zope for static content (such as multi media files). This is essential for you large file objects.
If you have up to a few thousand objects and they do not change too often, you can put them (the meta data) into the ZODB. Otherwise, put them in a RDBMS. I would use the RDBMS whenever I have one or I have more than a few hundred objects or I would need access to the data from legacy processes, too.
Dieter
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
On Tue, 19 Sep 2000, Jason Cunliffe wrote:
I am curious how do you all think LocalFS would perform for such a big multimedia application.? http://www.zope.org/Members/jfarr/Products/LocalFS
Use Zope for managing meta-data, or mayeb even better Zope + a fast DB tool for the metadata part. Let Zope do its holy best for the 'virtual-object interfacing' jobs, let the others do their thing..
This may very well be the best solution, as far as I understand what LocalFS is (I've not used it for now, however), because this may allow us to set per-domain (Medicine, Maths, etc...) permissions under Zope, and do all the presentation in Zope, while not being attached to Zope if we want to easily dump our datas somewhere else. Thanks to all. Now all I have to do is just kill these PHP guys there ;-)) bye, Jerome Alet
Jason Cunliffe writes:
I am curious how do you all think LocalFS would perform for such a big multimedia application.? http://www.zope.org/Members/jfarr/Products/LocalFS In my view, LocalFS is an excellent product. I like it very much.
However, I would not upload and not serve objects with a size of several hundred megabytes through ZServer (and thereby through Zope). As I said in an earlier response, ZServer is at least on order of magnitude slower with static content than Apache. This matters, if you transfer big objects! Recently, there has been some simple benchmarks for Zope versus Apache performance in this list. Search the archive (there is a searchable one at NIP) to get a feeling about the bandwidth. You may well use LocalFS to view the metadata (id, size, modification time, ...) from Zope, but you should not go through ZServer for upload and HTTP requests to really large objects. Dieter
Dieter Thanks for your answer and advice..
I am curious how do you all think LocalFS would perform for such a big multimedia application.? http://www.zope.org/Members/jfarr/Products/LocalFS In my view, LocalFS is an excellent product. I like it very much.
However, I would not upload and not serve objects with a size of several hundred megabytes through ZServer (and thereby through Zope). As I said in an earlier response, ZServer is at least on order of magnitude slower with static content than Apache. This matters, if you transfer big objects!
OK, sorry to be so dense about this, but when you say "through ZServer" ..please clarify for me on thing: Does LocalFS actually go "through ZServer" ? I thought that all it does, is to make a virtual link, storing a pointer in the ZODB to the real physical file system. Upload thus via FTP would be as fast as could be, and download could be defined if you prefer FTP or HTTP. Does'nt the link in LocalFS work the same in both directions, merely storing the linkn in XODB adn pass the location to the protocol?
Recently, there has been some simple benchmarks for Zope versus Apache performance in this list. Search the archive (there is a searchable one at NIP) to get a feeling about the bandwidth.
You may well use LocalFS to view the metadata (id, size, modification time, ...) from Zope, but you should not go through ZServer for upload and HTTP requests to really large objects.
Again I dont understand. With LocalFS, isn't the "really large object" outside of Zope. Surely it does not care how big the file is? All it cares about is its location, and any extra metadata IF you choose to store it in Zope. yes / no? thanks - Jason
Jason Cunliffe writes:
OK, sorry to be so dense about this, but when you say "through ZServer" ...please clarify for me on thing:
Does LocalFS actually go "through ZServer" ? I thought that all it does, is to make a virtual link, storing a pointer in the ZODB to the real physical file system. Upload thus via FTP would be as fast as could be, and download could be defined if you prefer FTP or HTTP. Does'nt the link in LocalFS work the same in both directions, merely storing the linkn in XODB adn pass the location to the protocol? It does.
When I stress that you should not go through ZServer, it means do not upload/serve via Zope. Use FTP or another protocol appropriate for mass data transfer over unstable connections. Dieter
participants (3)
-
Dieter Maurer -
Jason Cunliffe -
Jerome Alet