This is a very necessary discussion, methinks... there's been bits of threads, but I don't think enough.
From: Bill Anderson <bill@libc.org>
o Some people are still (dare i say many?) running on a system that has a 2Gb limit on filesize.
OK, they can try this: http://www.zope.org/Members/hathawsh/PartitionedFileStorage
o A large ZODB _can_ cause problems when you need it back it up, or restore it
how, more than a large bunch of files in a directory tree? Either way, typical backup software splits for CD burning, multiple tapes, etc.
o Consider the problems involved with a large ZODB when starting up and shutting down I have an N-Class HPUX beastie that has a 362+MB ZODb, it refuses to pack, and takes a while to start up.
What do you mean, it refuses to pack? It gives an error or something?
Memory is not an issue, believe me, I have several gigs of that untouched yet. yet it still takes an unacceptable amountof time to strart up.
Several gigs of RAM? Cool! Maybe someone from DC can detail what goes on at startup that might take alot of time. I guess, this is important for a desktop machine, that you turn on every day -- for a server that's always on, no big deal, right?
o Consider the problem of someone uploading a diferent (perhaps improved mp3) file. Now, until you pack past this change, you have _two_ copies of the data. Actually, I believe that even if only some metadata is changed, you get another copy. This. Gets. Ugly. Fast.
Actually, this could be a good thing. What if the 'improved' version is broken somehow? Or the meta data was in error? 'Undo'! Also, importantly, version control! If you're talking about a site like MP3.COM where you have lots of people uploading their own stuff, then you might have a problem with packing, with mp3's, images, other stuff. What to do about it? A couple of ideas come to mind: pack often, like every day, or use the packless database and forgo the feature, or ...? I'm aiming at uploading of audio as something with a little more control (like a review process), so having 1000 versions of the same file unpacked isn't an issue, and the versioning is important.
o Consider the ZSP that would much rather have the data stored in the FS where they can keep better track of it (and use quotas perhaps)
If they're in linux, they could quota the data.fs, no difference there... I don't know that they should be monitoring your files in either case, should they?
o Consider the possibility of having the file sitting on the OS's file system, available for outbound-ftp through non-zope means, or local system use (for. those who would use it on their local machine as an MP3 organizer) or for an audio-streamer program, such as icecast or shoutcast, to stream them out
These are the important use cases I see for external storage. Though, as far as icecast goes, someone should beat that code into a Product, it would be great to have streaming built in to zope!