[Zope] Announce - Audio Product

[Bill Anderson]

On 02 May 2001 23:13:01 -0400, marc lindahl wrote:
> This is a very necessary discussion, methinks... there's been bits of
> threads, but I don't think enough.
> 
> > From: Bill Anderson <bill@libc.org>
> > 
> > o Some people are still (dare i say many?) running on a system that has
> > a 2Gb limit on filesize.
> 
> OK, they can try this:
> http://www.zope.org/Members/hathawsh/PartitionedFileStorage
> 
> > 
> > o A large ZODB _can_ cause problems when you need it back it up, or
> > restore it
> 
> how, more than a large bunch of files in a directory tree?  Either way,
> typical backup software splits for CD burning, multiple tapes, etc.


Which is worse: A corrupted Data.fs, or a corrupted .mp3 file? If you
get a corruption in the backup process (Yes, it ODES happen), or the
restore, which would you rather have?

> 
> > 
> > o Consider the problems involved with a large ZODB when starting up and
> > shutting down 
> > I have an N-Class HPUX beastie that has a 362+MB ZODb, it refuses to
> > pack, and takes a while to start up.
> 
> What do you mean, it refuses to pack?  It gives an error or something?


No error, and no pack. I haven't had  a lot of time to go into it ...

> > Memory is not an issue, believe me, I have several gigs of that
> > untouched yet. yet it still takes an unacceptable
> > amountof time to strart up.
> 
> Several gigs of RAM?  Cool!  Maybe someone from DC can detail what goes on
> at startup that might take alot of time.


Well, indexing the db is a starting point, I believe.

> 
> I guess, this is important for a desktop machine, that you turn on every day
> -- for a server that's always on, no big deal, right?

Wrong. Would you like to fsck that 20 GB filesystem each time you
started up? It is also common for the developer/admin to have a
duplicate install of their production site to test out new code on.
these servers tend to be restarted frequently. And yes, on a desktop
machine, it would indeed get nightmarish.

> 
> > o Consider the problem of someone uploading a diferent (perhaps improved
> > mp3)  file. Now, until you pack past this change,
> > you have _two_ copies of the data. Actually, I believe that even if
> > only some metadata is changed, you get another copy.
> > This.   Gets.   Ugly.   Fast.
> 
> Actually, this could be a good thing.  What if the 'improved' version is
> broken somehow?  Or the meta data was in error?  'Undo'!
> 
> Also, importantly, version control!

you can version control the metadata.

> 
> If you're talking about a site like MP3.COM where you have lots of people
> uploading their own stuff, then you might have a problem with packing, with
> mp3's, images, other stuff.  What to do about it?  A couple of ideas come to
> mind:  pack often, like every day, or use the packless database and forgo
> the feature, or ...?

pack every day, and you may as well forget about an undo from yesterday.


> I'm aiming at uploading of audio as something with a little more control
> (like a review process), so having 1000 versions of the same file unpacked
> isn't an issue, and the versioning is important.

Like I said, you can make it so the metadata is version-controllable.
1000 copies at 2MB=2GB ... for ONE lousy MP3? While _you_ may be fine
with that, most people ar enot. Still, it is your choice. Multiply that
by, say a hundred mp3s ...


 
> > 
> > o Consider the ZSP that would much rather have the data stored in the FS
> > where they can keep better track of it (and
> > use quotas perhaps)
> 
> If they're in linux, they could quota the data.fs, no difference there...  I


Except that you are then you are using quota on the _entire_ Data.fs.
With an ExtFile you can have a seperate mountpoint that has a separate
quota control. say I only want up to X Gb of mp3 files, but would also
like up toX Gb of other objects, such as ExtImage objects. by limiting
the data.fs, you can't do that very well. Additionally, maybe I don't
want 1.5-2.5 Gb of a single song stored. ;^)~

> don't know that they should be monitoring your files in either case, should
> they?

For filesize, to determine how much space is bein used, why not? What
they *should* be doing is not at issue her eanyway.

> > o Consider the possibility of having the file sitting on the OS's file
> > system, available for outbound-ftp through
> > non-zope means, or local system use (for. those who would use it on
> > their local machine as an MP3 organizer)
> > or for an audio-streamer program, such as icecast or shoutcast, to
> > stream them out
> 
> These are the important use cases I see for external storage.  Though, as
> far as icecast goes, someone should beat that code into a Product, it would
> be great to have streaming built in to zope!


If it could do it reliably and speedily, yes, it would be awesome.
Anyway, I was just kicking out some real-world reasons whyan ExtMP# type
file would be (more) useful. Feel free to disagree, just understand that
there are many of us out here here who have very valid reasons for not
storing file objects that average over 1MB in the ZODb, especially when
we are talking about potentially thousands, as you indicated. I've gone
through all these issues in developing a Zope Advertising Product.

Bill