[Zope] Announce - Audio Product

[marc lindahl]

> From: Bill Anderson <bill@libc.org>
> 
> Which is worse: A corrupted Data.fs, or a corrupted .mp3 file? If you
> get a corruption in the backup process (Yes, it ODES happen), or the
> restore, which would you rather have?

Well, if it was something like the MP3.COM site, there wouldn't be another
centralized source for replacements for the .mp3 files, so I don't see the
issue.  If the backup is corrupted, then you'd go to the next-oldest
backup...



>> What do you mean, it refuses to pack?  It gives an error or something?
> 
> 
> No error, and no pack. I haven't had  a lot of time to go into it ...


Anyone else out there have this problem???  Hope it isn't a serious bug...!




>> Also, importantly, version control!
> 
> you can version control the metadata.

That's another ball of worms - syncronizing with the external data (e.g.
what if the file moves/renames/changes?  Do you automatically scan?  Or have
a maintainance procedure of some kind to check?  And have error handling,
etc.?)  I haven't looked at the code of ExtFile or LocalFS, don't know how
they deal with that.


> pack every day, and you may as well forget about an undo from yesterday.

Right, which is bad for all the other stuff you might want to undo...

>> I'm aiming at uploading of audio as something with a little more control
>> (like a review process), so having 1000 versions of the same file unpacked
>> isn't an issue, and the versioning is important.
> 
> Like I said, you can make it so the metadata is version-controllable.
> 1000 copies at 2MB=2GB ... for ONE lousy MP3? While _you_ may be fine
> with that, most people ar enot. Still, it is your choice. Multiply that
> by, say a hundred mp3s ...


Well, I was overstating the case.  But I find it typical that, for example,
one might mistakenly put up the wrong mix of a song, and have to pull it
quickly... or that a mistake with some meta data, like publisher name or
song title, is made once or twice before it's gotten right.

I wonder if, to get super-tweeky, it would be possible for a 'partial pack',
where you could follow the 'thread' of a particular object or set of
objects, and just pack those?


> Except that you are then you are using quota on the _entire_ Data.fs.
> With an ExtFile you can have a seperate mountpoint that has a separate
> quota control. say I only want up to X Gb of mp3 files, but would also
> like up toX Gb of other objects, such as ExtImage objects. by limiting
> the data.fs, you can't do that very well. Additionally, maybe I don't
> want 1.5-2.5 Gb of a single song stored. ;^)~

A related thing I was thinking about was how to limit file size of
individual files, and individual user quotas.  My answer to that one was:
1. filesize would be limited by the 30 minute built-in timeout of zope...
you can do the math... as long as you have enough storage for one of those
(for each zope thread) you won't die...  then you could, after upload, check
the filesize of the file (either internal or external), and if it's too big,
delete it and warn the user.
2. again, you can have methods to scan the user's area and add up his file
storage, and perhaps disable uploading if it's over.

To me, it seems you could integrate these types of controls better with the
site functionality if everything is inside zope.




> Anyway, I was just kicking out some real-world reasons whyan ExtMP# type
> file would be (more) useful. Feel free to disagree, just understand that
> there are many of us out here here who have very valid reasons for not
> storing file objects that average over 1MB in the ZODb, especially when
> we are talking about potentially thousands, as you indicated. I've gone
> through all these issues in developing a Zope Advertising Product.

I'm looking at this issue keenly, and trying to dig down to the roots on
these issues... I don't think there are easy answers, but overall I think
the ZODB is underappreciated, and I'm trying to get a better grip on it's
limitations.  I'm still not clear on how the size of a python object is a
factor (as opposed to the number of objects in data.fs), though it's pretty
clear to me that anything that passes thru zope (e.g. via ExtFile) is at
some point inside zope and therefore the same, at that point, as something
stored in data.fs, as far as speed and performance goes.