[Zope-CMF] best practices - dealing with large files and avoiding data corruption

[Sam Brauer]

Dieter Maurer wrote:
> Sam Brauer wrote at 2003-7-18 16:25 -0400:
>  > ...
>  > Yesterday one of our Zope sites "locked up" the server (had to be rebooted).
>  > I think a contributing factor may have been that someone was uploading 
>  > some largish files into the site.  Unfortunately, there's no telling 
>  > error message in the log.
> 
> Please see my recent post to Paul Winkler in "zodb-dev@zope.org"
> Subject "Metadata in zope / binary data on FS".
> 
>   Something strange happens with large file upload.
>   Almost surely, it is not Zope (in the narrow sense).
>   It may be ZServer...
> 
> I will probably investigate the problem in the next week.
> 

Sounds good!

>  > I've seen the CMFExtFile product, but it doesn't look to be supported 
>  > anymore and the project page for it says that it doesn't work with Zope 
>  > 2.5 and higher.
> 
> Search the mailing list archive.
> I saw recent posts of people using CMFExtFile with modern Zope 
> versions.

I actually wound up using the ExtFile code as inspiration to modify the 
custom CMF Type object I wrote that I am using as a base class for all 
of my content types.  Since I had already written a custom File class 
(which I use to store files (and images) as attributes of my content 
type classes) I simply changed it to write the data to the filesystem.

I buffer the data when both writing and reading, and it seems to work 
great.  RAM usage stays pretty constant and the ZODB only stores 
metadata about the files.


> 
> But, when I am right (ZServer is to blame), it will not help as uploads
> still go through ZServer.
> 
>  > ...
>  > After only a couple of weeks with CMF sites in production, I've already 
>  > seen 2 corrupt data errors.  Not to imply that CMF was responsible for 
>  > the errors, it's just that the only Zope sites I have are CMF sites.
> 
> Can you please be more specific.
> 
> What "corrupt data errors" have you seen?

I forget the exact error... if it happens again, I'll be sure to note it 
and any traceback info in the event log.
Seems like it contained the phrase "DataCorruption".
When I want in the ZMI and tried to look at the Undo tab, I got an 
IOError.
Running fsrecover.py cleared things up.

> 
> We know several "pack" problems with Zope 2.6.1 (see the collector).
> That's a main reason why there will soon be Zope 2.6.2.
> 

I look forward to its stable release.
Noticed that beta4 was just released.


> But: we use Zope+CMF for more than a 1 1/2 year and saw a single
> occurrence of this problem.
> 
>  > ...
>  > I love Zope, but need it to be reliable.
> 
> Please provide concise and complete problem reports.


Sorry for such a general statement.
Then again, I was just looking for general solutions or anecdotes.

I think my change to storing files on the filesystem will improve my 
stability.  I also found another potential problem of my own creation: I 
have an Ad class that maintains a BTrees of view and click data.  So 
everytime someone views a page with an Ad, a transaction is created in 
the ZODB.  I should change my Ad class to store the view and click data 
elsewhere (such as in a relation database).

I'm hoping that after I make these changes I will be in much better 
shape.  If after making them I still have problems that I can't figure 
out, I'll try to post more useful troubleshooting information.

Thanks for your comments and help Dieter!

- Sam


> 
> As soon as we see a serious problem pattern someone in the community
> starts analysing and chances are high that the next release
> will have fixed the problem.
>


> 
> Dieter


-- 
Sam Brauer