[Zope] Zope Performance and Large Databases

[Michel Pelletier]

> -----Original Message-----
> From: davidbro@namshub.org [mailto:davidbro@namshub.org]
> Sent: Tuesday, September 07, 1999 3:24 PM
> To: zope@zope.org
> Subject: [Zope] Zope Performance and Large Databases
> 
> 
> How much stuff can Zope (or more properly ZODB3) handle in a folder
> before things get hairy and/or slow and buggy?

Folder are not meant to hold thousands and thousands of objects *in one
folder*.  For this, you should store your data by more efficient means,
like a BTree, multiple heirarchical(sp?) folders, or a relational
database.  The same things happens with, for example, linux directories.
They are not meant to hold on the order of a thousand files efficiently.

If you do pack a folder full of objects, things will not get 'hairy' or
buggy, just slow.  Note that *traversal* doesn't get noticably slower
(meaning, the 'fetching' of /X/Y/Z when 'Y' contains thousands of
objects) just the *listing* and managment of folders gets very slow.

> I want to use Zope as the center of my Home Automation intranet.

Cool.

> One of the applications of that HA intranet would be to control the
> MP3's playing throughout the house.  Now, it seems that the best place
> to store the information about all the MP3's (artist, album, genre,
> actual location, etc) would be in a relational database that Zope
> hits, but I also wondered if I could use ZClasses and ZCatalog to
> manage the information.

Sure.  Either solution would give reasonable performance over a small to
medium scale.  I would personally use ZClasses with ZCatalog.  We have a
customer who keeps almost 10000 objects in ZCatalog with no perceptable
delay.

Because ZCatalog uses BTrees, finding one object out of a million
typically requires less than 20 'comparison' operations.  I think that
'theoretically' ZCatalog can scale into the order of hundreds of
thousands of objects, but that is just my personal opinion (cuz I wrote
it) not any kind of official guarantee.
 
> But being that I have several gigs of MP3's (don't start about
> copyright stuff, these are ripped from my CD collection), we're
> talking about a large number of ZClass instances (one per song) stored
> in the ODB.

Be aware that this will make your Zope object database massive and
increse your memory requirements.  Perhaps it would be simpler to just
serve the actualy MP3s out of a static Apache and let Zope just handle
the 'meta' information?  If your running Linux, don't forget that ext2fs
has a 2GB file size limit.

> How robust is the ODB in these sorts of situations?  Has there been
> any testing of Zope with large (I had a boss that didn't think a
> database was interesting until it had at least a gig of information in
> it) amounts of data?  I know that current file system limits in Linux
> put an artificial cap on DB size, but you can still get really big
> before hitting the limitation.

One things I think you are bit confused about is the difference between
storage (ZODB) and indexing the storage (ZCatalog).  Their respective
features and performance issues should be looked at seperately.
 
ZODB is very robust and has no problem scaling into gigs of data.  of
course, many databases are also as robust as this.

-Michel