Large file transfers
All, We desire to use our website (Zope) to gather large files from our users. These files can be rather large (10-70mb). From what I have read It would probably not be a good thing to import these files into the Zope database. So does anybody have any other ideas of how to move these files onto our server? Sherman
As long as you're aware of the issues involved, there are compelling reasons to store them in ZODB... some OS's limit data.fs to 2GB, but you can use the PartitionedFileStorage product to get around that. There are some great discussions going on in the CMF list on document/file history control and workflow, you might be interested.
From: "Sherman Lambert \(State Public Defenders Office\)"
We desire to use our website (Zope) to gather large files from our users. These files can be rather large (10-70mb). From what I have read It would probably not be a good thing to import these files into the Zope database. So does anybody have any other ideas of how to move these files onto our server?
It's easy to do with an external Python method. The main trick is to get the script to know where the files should be placed without hard-coding a specific directory. I like to use a location relative to the module that is doing the work. In python you can easily get the file location of a module. If you put all your files relative to that location you can avoid hardcoding an absolute path. Alternatively, you could use an enviromental variable, but that's one more thing to set up and keep synchronized. Next, how can you find these files from within Zope, in case you want to do something with them using Zope? For that you use the localFS product, and point it to the root directory where you are going to store the files. This combination is probably the simplest way to solve the task. It's true that you give up some authentification capability, but you can control that by the design of the page that the user uses to submit the files. It's probably possible to do the upload using localFS too, but I remember having some problems last year when I tried it - only with Netscape, not IE - the file was not getting MIME encoded/decoded properly. Sorry, I don't remember any details by now. May be fixed by now. Cheers, Tom P [Sherman Lambert]
We desire to use our website (Zope) to gather large files from our users. These files can be rather large (10-70mb). From what I have read It would probably not be a good thing to import these files into the Zope database. So does anybody have any other ideas of how to move these files onto our server?
We desire to use our website (Zope) to gather large files from our users. These files can be rather large (10-70mb). From what I have read It would probably not be a good thing to import these files into the Zope database. So does anybody have any other ideas of how to move these files onto our server?
Hi! Your question actually has got to parts: a) Does ZODB work with files that large? Answers: - Standard file storage: Yes, if you use it on a system where the 2GB limit does not apply, e.g. Linux with Kernel 2.4 on Intel ... - BerkeleyDB storage: Yes, certainly. The 2GB limit does apply, too, but not "that early", as the data is split into several files There are other possibilities, like uploading directly to the file system, as Thomas pointed out. All the code can be found in LocalFS, as it also has UPLOAD capabilities built in. If you prefer a file-by-file representation in Zope (LocalFS just "mounts" a filesystem tree), ExtFile/ExtImage might serve you better. b) Can I upload files of that size via the browser? Yes, but there might be problems. We tested uploads to the ZODB and got them working up to at least 100 MB per file. But that was over a 10MBit Ethernet line. Over the web, there might be timeout problems etc. Alternatives that are more reliable include FTP upload or maybe WebDAV (which seems to be quite slow, but very reliable to us on the Windows platform). Cheers Joachim.
Alternatives that are more reliable include FTP upload or maybe WebDAV (which seems to be quite slow, but very reliable to us on the Windows platform).
WebDAV goes over HTTP in the same way as a POST does, so I wouldn't expect it to be any more reliable than a normal web form upload. However, I'd love to be told I'm wrong :-) cheers, Chris
WebDAV goes over HTTP in the same way as a POST does, so I wouldn't expect it to be any more reliable than a normal web form upload. However, I'd love to be told I'm wrong :-)
I think it completely depends on the client implementation, as the underlying protocols are stateless. I just assumed that WebDAV clients would be more optimized for long transfer times than browsers, which normally specialize in GETTING data from the web, not posting (large) data. We haven't tested this in any way, though. And as I said, up to more than 100 MB we had no problems (apart from having to wait) with POST uploads (using Internet Explorer 5.x against a Linux Zope server). BTW: For uploads that large, a lot of RAM on the server might be needed. I am not sure whether Zope tries to receive and store the file in one bit, but if it does, you need enough memory to hold the whole thing if you don't want to put your server into "swap mode" ... Of course there are ways of doing the file handling on the server in a streaming way ... Joachim
BTW: For uploads that large, a lot of RAM on the server might be needed. I am not sure whether Zope tries to receive and store the file in one bit, but if it does, you need enough memory to hold the whole thing if you don't want to put your server into "swap mode" ...
I'm pretty sure this is unneccessary. IIRC, Zope uses sub-transactions to write the file to disk in small-ish bits if its large rather than hogging lots of RAM. cheers, Chris PS: Joachim: I get quite a lot of bounces from your Iuveno address, is everything okay there?
On Mon, 4 Jun 2001, Chris Withers wrote:
PS: Joachim: I get quite a lot of bounces from your Iuveno address, is everything okay there?
I often get timeouts trying to access demo webserver, so it is certainly not ok there. Oleg. ---- Oleg Broytmann http://www.zope.org/Members/phd/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
participants (6)
-
Chris Withers -
Joachim Werner -
marc lindahl -
Oleg Broytmann -
Sherman Lambert (State Public Defenders Office) -
Thomas B. Passin