AW: [Zope-dev] FileUpload and blob on Windows

[Leonardo Rochael]

Roger Ineichen wrote:
> 
> Why are you using a NamedTemporaryFile? If I'm right the 
> goal is to store the file stream from the upload directly
> in this file and copy this file over to the real directory 
> location.
> This means you can cut down the amount of read and write file data, right?
> 

Yes, and plone.app.blob was using a NamedTemporaryFile exactly because it
has a filesystem visible name that Blob.consumeFile() can os.rename() over
to the final location.


Roger Ineichen wrote:
> 
> Why not use a own file class like:
> 
> class TMPFile(file):
>     """Temorary file.
>     
>     This temporary file can remove a file in request.close() form the file
>     system.
>     """
> 
>     def release(self):
>         """Release the object in the requests (_held) list."""
>         if self.name is not None and os.path.exists(self.name):
>             os.unlink(self.name)
> 

I'm already using something like that, because NamedTemporaryFiles on
Windows disappear when they're closed, *even if* they've been renamed away
from the original file name, but then I hit the other snag in that open
files can't be renamed, and even win32file hard links to open files can't be
renamed while the original is open

And I also tried replacing the request file with a blob file opened for
reading, but the request outlives the transaction, so when the commit
happens, the blobfile is still open and BlobStorage complains. I even tried
surreptitiously opening the filesystem file from under blob and placing that
in the request, but at commit time, BlobStorage tries to rename the file to
it's final location, and Windows doesn't like it.

Now I'm wondering if I'll have to implement a transaction manager to close a
blob files before the transaction.


Roger Ineichen wrote:
> 
> You can move such a file with shutil.move(self.tmpPath, targetPath).
> 
> I implemented such a file upload accelerator. With this beast I was able
> to upload a ubuntu vmware image with > 950 MB in about 75 seconds to
> Zope3.
> And this with a memory usage below 50 MB. It really rocks.
> 
> Note; I implemented this on windows and it works well.
> 

I got it to work quite far in that direction, but the snag I hit is when I
try to open the blob on the same transaction, and *that* fails.

Interestingly, the filesystem file created by NamedTemporaryFile on Windows
*can* be renamed and even removed while still open. I suppose this is due to
the O_TEMPORARY flag, as mentioned in this code from NamedTemporaryFile:



>     # Setting O_TEMPORARY in the flags causes the OS to delete
>     # the file when it is closed.  This is only supported by Windows.
>     if _os.name == 'nt':
>         flags |= _os.O_TEMPORARY
> 

If only there was a flag so that Windows allowed renaming/removing an open
file, but didn't try to remove the file itself after it was closed...

Cheers, Leo
-- 
View this message in context: http://www.nabble.com/FileUpload-and-blob-on-Windows-tp15129190p15160439.html
Sent from the Zope - Dev mailing list archive at Nabble.com.