Thanks for everyones advice, I will indeed have massive amount of data to store. I don't envision massive bandwidth. I also don't envision letting users to actually download audio files, rather to create playlist to listen to streaming. After reading the replies I see I have a lot of learning to do before I get to far afield. Thanks Aaron I am now reading about On Fri, 2004-05-21 at 19:24, Paul Winkler wrote:
On Fri, May 21, 2004 at 05:14:54PM +0200, Jerome R. Westrick wrote:
Basically. the way I see it is as follows:
In Zope I implent my Objects (if I'm using that model)
In Zope I implement my seperation of "Display code" and "application code".
In my last project I tried to keep the database mostly read only, and stored all data in postgress. I had applicational reasons for this but it also worked out rather well, having code in Zope and Data in PG.
In a preveous project, I stored PDF's in the Zope DB. This is definatetly a mistake, which causes problems with an Over fulled Zope DB holding a history of PDFs for nothing.
*shrug* there are a lot of options there. It is now possible to "mount" multiple databases in one Zope instance. So all the audio files could go in a dedicated ZODB - something capable of scaling to many GB of data, so I'd probably choose DirectoryStorage. Activity in this database wouldn't affect any other zope data. And you could pack it however often you like.
Re. some of the earlier comments:
On Fri, 2004-05-21 at 08:04, Aaron wrote:
On Fri, 2004-05-21 at 03:02, Jerome R. Westrick wrote:
Hi Aaron...
I wouldn't place the songs in Zope... Instead I would use LocalFS to get access to them...
At one time I would have agreed, to keep the ZODB from getting unwieldy; but nowadays, unless I had another reason that I wanted the files easily accessible as plain files on the filesystem, I would just use DirectoryStorage instead. It's robust, and given an appropriate filesystem it can handle really massive amounts of data, and it's easier to set up than fiddling around with LocalFS or ExternalFile or whatever.
Download speed of blobs is still an issue. Note that the commonly deployed versions of LocalFS and many other attempts at keeping Zope data on the filesystem actually perform MUCH worse than the normal ZODB-based File class.
I would handle the speed issue by throwing massive amounts of disk caching at the problem. If the content is anonymously downloadable I'd use Squid or apache + caching; if not, I'd use the FileCacheManager that we hacked up at Pycon. And if using ZEO, I'd use an enormous ZEO client cache.
For more background on blobs in zope, see: http://www.slinkp.com/code/zopestuff/blobnotes
Sure, you could end up eating several times the size of your data set for all these disk caches... but who cares? At roughly a dollar per gigabyte, buying more disk space is a much better investment than spending even an hour worrying about it.
Would I use the zope db to keep track of the audio or a relational db?
If it were me, regardless of where the data is stored, I would use zope's Catalog to provide searching and browsing capabilities based on metadata. But I guess it depends on what kind of queries you need to be able to do. The Catalog is pretty flexible, but probably not as flexible as a full-blown query language (SQL).
To get an idea of what zope's catalog can do, take a look at plone.org. The front page uses the catalog for all the cool stuff on the right:
* the Search feature * the News portlet to the right * the Upcoming Events portlet * the Calendar at lower right
My question Is generally more generic, what advantage does zope give me over another solution.
It's just a different way of working / thinking. Once you have some understanding of zope, you can work pretty darn fast. And it allows you to cleanly separate business logic from presentation code and from content (although many people do not do so).