[Zope-CMF] best practices - dealing with large files and avoiding data corruption

[Raphael Ritz]

Sam Brauer wrote:

> I have two problems that may or may not be related, but I'll go ahead 
> and ask about both of them...
>
> I expect other folks with CMF sites have run into the problem of 
> dealing with users that need to upload large files (by "large" I mean 
> 5-100Mb) into a CMF site.  I'm wondering how other people have dealt 
> with this sort of situation. 

We wrote PloneExternalFile for us. It basically wraps ExternalFile for 
usage from within Plone.
http://www.neuroinf.de/LabTools/PloneExternalFile-0_1.tgz/view

One thing ExternalFile does to reduce the risk of the porblems you 
encouter is to split
the processing of the uploaded file in handsome blocks.

Raphael

>
>
> Yesterday one of our Zope sites "locked up" the server (had to be 
> rebooted).
> I think a contributing factor may have been that someone was uploading 
> some largish files into the site.  Unfortunately, there's no telling 
> error message in the log.
>
> I've seen the CMFExtFile product, but it doesn't look to be supported 
> anymore and the project page for it says that it doesn't work with 
> Zope 2.5 and higher.
>
> I'm also thinking about using DirectoryStorage instead of the normal 
> FileStorage.   Not sure if it deals better with large files or not, 
> but I'm hoping that perhaps it is more resistent to corruption than 
> the FileStorage.
>
> After only a couple of weeks with CMF sites in production, I've 
> already seen 2 corrupt data errors.  Not to imply that CMF was 
> responsible for the errors, it's just that the only Zope sites I have 
> are CMF sites.
>
> The corruption problems are really unsettling to me.
> At least I was able to recover (using the fsrecover.py from 
> Zope-2.7.0-a1), but shutting down zope and running that utility means 
> downtime for all sites served from that Zope instance.
>
> I love Zope, but need it to be reliable.  I'm running Zope 2.6.1 and 
> have seen that there have been several ZODB fixes since then, but 
> 2.6.1 is still the latest stable Zope release.
>
> Have any other folks had similar experiences?
> Any suggestions for avoiding data corruption?
> What about dealing with large file uploads?
>
> (Sorry if these are totally unrelated issues, but I'm curious about 
> both... so don't be shy about responding to just one of these issues.)
>
> Thanks for any help,
> Sam
>
>
> _______________________________________________
> Zope-CMF maillist  -  Zope-CMF@zope.org
> http://mail.zope.org/mailman/listinfo/zope-cmf
>
> See http://collector.zope.org/CMF for bug reports and feature requests
>