Hi, we are currently planning to create a multimedia database and are investigating different solutions. I'm pushing toward a Zope based one, especially for the permissions needs we have, but I was wondering if ZODB can handle our needs or if we should store multimedia files directly on the file system. Our needs are to store hundreds, and possibly thousands of MPEG2 and MPEG4 (generally short) video movies, MP3 audio files, JPEG images, etc... Altough most of the files will be in the 1 Mb -> 10 Mb range, some could be larger, e.g. some Gigabytes. For each file we also want to keep some data: author, subject, encoding, description, keywords, domain, etc... I see 5 solutions: 1 - store both files and their associated datas in ZODB 2 - store files in the filesystem and datas in ZODB 3 - store files in the filesystem and datas in an RDBMS (use Zope only for the presentation of files and datas) 4 - store files in ZODB and datas in an RDBMS. 5 - don't use Zope at all because it's not appropriate. While I'm pretty sure I can design something more or less working like the 1 above in a few time with a few multimedia files and datas, please could you tell me what seems the best solution to you in the long run ? IMHO the 1 (if Zope scales well) and 2 are best, but we may as well use the 3. 4 doesn't seem natural to me, and I personnaly exclude the 5 ;-) Another question I wanted to ask: does Zope have a limit on the size of uploaded files, given the fact that we may want to upload files as large as some Gb and that we will use a linux filesystem not limited to 2 Gb per file. Please show me the light. thanks in advance. Jerome ALET - alet@unice.fr - http://cortex.unice.fr/~jerome Faculte de Medecine de Nice - http://noe.unice.fr - Tel: 04 93 37 76 30 28 Avenue de Valombrose - 06107 NICE Cedex 2 - FRANCE
From: "Jerome Alet" <alet@unice.fr> [snip]
Our needs are to store hundreds, and possibly thousands of MPEG2 and MPEG4 (generally short) video movies, MP3 audio files, JPEG images, etc...
[snip]
I see 5 solutions:
1 - store both files and their associated datas in ZODB 2 - store files in the filesystem and datas in ZODB 3 - store files in the filesystem and datas in an RDBMS (use Zope only for the presentation of files and datas) 4 - store files in ZODB and datas in an RDBMS. 5 - don't use Zope at all because it's not appropriate.
In general, large file uploads thru HTTP are clumsy. The browsers don't provide progress indicators. Broken connections occur due to timeout under network congestion but the browsers don't handle broken connections well. And (as far as I can tell) the entire request (including the entire file) is buffered in the OS swapspace. Some site I saw recently (was it xdrive? I'm not sure) uses client-side Java to manage the upload. I chose #2 for a similar application and ran into these additional issues: 1. A file upload done thru Zope into the filesystem passes thru a temporary file. This increases slightly the apparent upload time, while you copy the file to its final destination. And, you must also ensure that the partition where temporary files are placed has enough space for the maximum file size. 2. You must also decide how you are going to serve the file. Since it's in the file system, you could serve it directly thru Apache (perhaps thru a redirect initiated by Zope), thereby avoiding occupying a Zope thread for a long time for every downlaod. But if you take this approach, you must somehow make Apache apply the same access control to the file that you have in Zope. This can be done, but is an added complication. -- My .02 -- Loren
participants (2)
-
Jerome Alet -
Loren Stafford