[Zope] ZoDB and BIG FILES

[Paul Winkler]

On Tue, Jan 13, 2004 at 12:01:09AM +0100, Sebastien Pastor wrote:
> Hi there,
> 
> 
> I am about to finalize a Zope 2 site. It will mainly play the role of 
> groupware app and storage server.
> My question is : i know i ll get quicly a big amount of PSD files 
> (photoshop) they are usually quite heavy. Do you rekon i should store 
> them on my server file system rather than the ZODB ? Do you see any 
> trouble putting them all into the DB in a longer term  ? (i know my 
> python is large-file compiled). 

two issues:

1) A single large FileStorage has been a bit cumbersome to maintain
(backups etc). recent developments (repozo.py) make it possible
to do an incremental backup.
There's also DirectoryStorage which on a suitable filesystem should 
be able to scale to a very large database.
You can also mix-and-match and have multiple databases using DBTab.

2) Performance of serving large files or images from Zope is pretty
bad, much slower than a static webserver like Apache. The worst
part is that a few simultaneous requests for large files will
essential DOS your entire zope site until they complete.

I have not identified the precise point of pain, but I know that
in my experience, 5 MB is OK and 50 MB is awful. I have worked
around this by using FSCacheManager (available from CVS at
collective.sf.net) which does "funky caching". It is easy to set 
up with Apache, but is only safe to use with anonymously-viewable content.  
I understand that Squid can be set up to do filesystem caching while 
forcing authentication on each request so it would be safe with 
secured content, so we might do that instead at a later date.

> Storing on both FS and ZODB is it an 
> easy thing to do ?

If you want the files on the filesystem, there are various solutions
depending on your needs: LocalFS, FilesystemDirectoryView, and APE. 
I have not evaluated these for performance with large files, except
LocalFS which I know is at least as slow as normal zope File objects.
I tend to assume they are all slow.

-- 

Paul Winkler
http://www.slinkp.com
Look! Up in the sky! It's TERRA-RANDY CHAIN MISTER!
(random hero from isometric.spaceninja.com)