Tracking upload of files to Zope
Hello, Sorry if I post to a wrong list - if that is the case, please tell me where to post instead. I have one huge problem with Zope: It's impossible to track how far an upload of a file to Zope is :-(( It seems like Zope "blocks" untill a file is uploaded to Zope making it impossible to tell how many bytes has been transfered or similar. Is it me that hasn't got a clue on how the HTTP protocol works or is it a "bad implementation" of file uploads in Zope ? I kinda hoped I could create some sort of "wait page" when users upload files to my website ... it's big files (video files) so it takes some time and therefor it would be really great to be able to display a wait page to people. -- Gitte Wange Technical Manager Email: gitte@mmmanager.org Web: http://www.mmmanager.org Tlf: +45 36 46 20 02 (On the statement print "42 monkeys"+"1 snake") BTW, both perl and Python get this wrong. Perl gives 43 and Python gives "42 monkeys1 snake", when the answer is clearly "41 monkeys and 1 fat snake". -- Jim Fulton, 10 Aug 1999
Gitte Wange wrote at 2003-9-29 13:11 +0200:
.. I have one huge problem with Zope: It's impossible to track how far an upload of a file to Zope is :-((
It seems like Zope "blocks" untill a file is uploaded to Zope making it impossible to tell how many bytes has been transfered or similar.
I have uploaded a 80 MB file into Zope and I can assure you that Zope did not block (although the server was heavily loaded and had more than 60 % IO-wait). In the first stage, ZServer will read your HTTP request (including the file to upload) and store the file content in a temporary file. After the complete request is read, ZServer hands the request to Zope with stores the file in ZODB. Neither of these steps block. Only the transaction commit blocks the storage (which may be ZEO Server) for the time of the transaction. Dieter
On Mon, 29 Sep 2003 20:03:46 +0200, Dieter Maurer wrote:
Gitte Wange wrote at 2003-9-29 13:11 +0200:
.. I have one huge problem with Zope: It's impossible to track how far an upload of a file to Zope is :-((
It seems like Zope "blocks" untill a file is uploaded to Zope making it impossible to tell how many bytes has been transfered or similar.
I have uploaded a 80 MB file into Zope and I can assure you that Zope did not block (although the server was heavily loaded and had more than 60 % IO-wait).
In the first stage, ZServer will read your HTTP request (including the file to upload) and store the file content in a temporary file. After the complete request is read, ZServer hands the request to Zope with stores the file in ZODB. Neither of these steps block.
Only the transaction commit blocks the storage (which may be ZEO Server) for the time of the transaction.
Okay - by "block" I didn't mean that Zope stops to receive connections or anything. I meant that the occuring request (the ones that post's a file to Zope) is "blocked" so you cannot do anything in that thread while the file is uploaded. Your explation on ZServer helped me here Dieter - thanks ;-) So what I want, is to (while the upload is progressing) read the size of the temporary file or at least see how many bytes is transfered. Looks like some patching in ZServer ? Or is it impossible to do some hacking here ? -- Gitte Wange Technical Manager Email: gitte@mmmanager.org Web: http://www.mmmanager.org Tlf: +45 36 46 20 02 Our goal is to be the "Linux of Content Management". This means Open Source, it means community... -- Paul Everitt: Zope-CMF Mailing List
Hi Gitte, Gitte Wange wrote:
So what I want, is to (while the upload is progressing) read the size of the temporary file or at least see how many bytes is transfered. Looks like some patching in ZServer ?
Or is it impossible to do some hacking here ?
The question is: to where do you want to read the progress? Most clients, however show you the amount of bytes already sent to the server, this should be informal enough. For every other approach, there is simply no channel in the HTTP-protocol to output the information to. So the question about how to get the information is completely secondary :-) IF you want to watch uploads for debugging purpose, and files are huge, you can try "lsof" in the temp directory. Regards Tino
On Tue, 30 Sep 2003 14:58:38 +0200, Tino Wildenhain wrote:
Hi Gitte,
Gitte Wange wrote:
So what I want, is to (while the upload is progressing) read the size of the temporary file or at least see how many bytes is transfered. Looks like some patching in ZServer ?
Or is it impossible to do some hacking here ?
The question is: to where do you want to read the progress?
I want to display to the users how far their upload is ... at this moment the browser looks like is has fallen a sleep. And the people uploading files are not used to do that (very novice users) so displaying and upload status to them would be nice.
Most clients, however show you the amount of bytes already sent to the server, this should be informal enough.
Browsers? No ... they just load the new page - that's what make the users think the website has fallen into sleep and they start clicking all sorts of things.
For every other approach, there is simply no channel in the HTTP-protocol to output the information to.
So the question about how to get the information is completely secondary :-)
Hmmm .. so Zope doesn't know how much data it has received ?
IF you want to watch uploads for debugging purpose, and files are huge, you can try "lsof" in the temp directory.
No - not debugging purpose but userfriendly (yikes!) purpose :-) -- Gitte Wange Technical Manager Email: gitte@mmmanager.org Web: http://www.mmmanager.org Tlf: +45 36 46 20 02 After 1.5 years of Python, I'm still discovering richness (and still unable to understand what the hell Jim Fulton is talking about). -- Gordon McMillan, 13 Mar 1998
The question is: to where do you want to read the progress?
I want to display to the users how far their upload is ... at this moment the browser looks like is has fallen a sleep. And the people uploading files are not used to do that (very novice users) so displaying and upload status to them would be nice.
Most clients, however show you the amount of bytes already sent to the server, this should be informal enough.
Browsers? No ... they just load the new page - that's what make the users think the website has fallen into sleep and they start clicking all sorts of things.
I suppose you could open a new frame or something that says "Please wait, upload is in progress" while the frame actually doing the uploading waits to return. Upon returning, it could have a bit of Javascript to blitz the "wait" frame. Ick! Did I just recommend Javascript? If you want a count, you'll have to do something like I previously posted, which will almost surely involve hacking ZServer to get the len() of the temporary file or some such and somehow streaming (constant refreshes?) the content back to a separate frame/window/client/whatever. But really, it's a client-side problem. --jcc -- "Code generators follow the 80/20 rule. They solve most of the problems, but not all of the problems. There are always features and edge cases that will need hand-coding. Even if code generation could build 100 percent of the application, there will still be an endless supply of boring meetings about feature design." (http://www.devx.com/java/editorial/15511)
I have one huge problem with Zope: It's impossible to track how far an upload of a file to Zope is :-((
It seems like Zope "blocks" untill a file is uploaded to Zope making it impossible to tell how many bytes has been transfered or similar.
Is it me that hasn't got a clue on how the HTTP protocol works or is it a "bad implementation" of file uploads in Zope ?
I kinda hoped I could create some sort of "wait page" when users upload files to my website ... it's big files (video files) so it takes some time and therefor it would be really great to be able to display a wait page to people.
HTTP doesn't preclude you from knowing how many bytes you've sent, but it does make it rather difficult for a server to do anything about it. It's really a client-side thing to provide a "status bar" or such on data transfer, because (1) it's stateful dynamic data and (2) the server's already occupied during that current transaction. The only scenario I can imagine goes like this: - user asks to upload, gets a unique key or id - opens two clients/browser windows, one to a status page, one to an upload page, both by key - user starts upload and then looks at the status page - upload page shows some sort of status that Zope has hooked into the file transfer upstream It's really quite complicated and cumbersome. (You may be able to do funny things with Javscript or write a special client to make this more transparent, but it's still a hack.) You're better off getting the browser/client to deal with it. Most FTP clients do this as a matter of course. --jcc -- "My point and period will be throughly wrought, Or well or ill, as this day's battle's fought."
I looked at that a while ago, using zope with the builtin Zserver, by the time you see the data in 'zope' it's already been 'eaten and regurgitated' by zserver... so you can't really tell the user anything at that point - it's already happened, basically, from his perspective. Because it's one transaction, you can't return stuff multiple times, even, within that connection. Basically, I think you're stuck with a client-side solution. On Monday, September 29, 2003, at 07:11 AM, Gitte Wange wrote:
Hello,
Sorry if I post to a wrong list - if that is the case, please tell me where to post instead.
I have one huge problem with Zope: It's impossible to track how far an upload of a file to Zope is :-((
It seems like Zope "blocks" untill a file is uploaded to Zope making it impossible to tell how many bytes has been transfered or similar.
Is it me that hasn't got a clue on how the HTTP protocol works or is it a "bad implementation" of file uploads in Zope ?
I kinda hoped I could create some sort of "wait page" when users upload files to my website ... it's big files (video files) so it takes some time and therefor it would be really great to be able to display a wait page to people.
-- Gitte Wange Technical Manager
Email: gitte@mmmanager.org Web: http://www.mmmanager.org Tlf: +45 36 46 20 02
(On the statement print "42 monkeys"+"1 snake") BTW, both perl and Python get this wrong. Perl gives 43 and Python gives "42 monkeys1 snake", when the answer is clearly "41 monkeys and 1 fat snake".
-- Jim Fulton, 10 Aug 1999
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
participants (6)
-
Dieter Maurer -
Gitte Wange -
J Cameron Cooper -
J. Cameron Cooper
-
Marc Lindahl -
Tino Wildenhain