[Zope] ZODB and large data sets

[Howard Clinton Shaw III]

On Sat, 22 May 1999, Jimmie Houchin wrote:
> I am working through the design of my website as I study and learn Zope.
> 
> I've started to work on calculating storage requirements to determine
> server needs as I am also working on building my server.
> 
> My app can easily be divided into multiple datasets or databases. One of
> the datasets I am looking at has a potential of 4+ million objects with
> each object requiring 15-50kb minimum. This dataset can be subdivided.
> Initially I will not populate the database will all of the items in their
> full form, but will populate as requests for data come. However I need to
> develop as if and plan for complete population.
> 
> This makes for a very large database and one that spans more than one hard
> drive.
> 
> Can Zope create and use multiple ZODBs on multiple hard drives?
> If so, how will such a large dataset affect packing and the creation of the
> backup file?
> 
> Will I need to use multiple other database backends such as MySQL or
> possibly MetaKit?
> 
> Thanks for any help or thoughts.
> 
> Jimmie Houchin
> 

Currently, Zope 1 uses a single file to store its data. That means that it is subject to the ext2 fs limit of 2gb on 
32 bit systems. On an alpha, no such limit (ok, there is, but its on the order of 8 million terabytes), 
but in either case, one file. Zope 2 will use ZODB3, which should permit multiple files to be used for one Zope
site. Either way, MySQL or similar techniques can certainly be used to extend the holding capacity of either database.
Indeed, the ZODB3 will be able to store its data in an RDBM.

I'm sure the DC Gurus could tell you more, as they have worked with large datasets, but they are currently
otherwise occupied.

> 
> 
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://www.zope.org/mailman/listinfo/zope
> 
> (For developer-specific issues, use the companion list,
> zope-dev@zope.org - http://www.zope.org/mailman/listinfo/zope-dev )
--
Howard Clinton Shaw III - Grum
St. Thomas High School
#include "disclaimer.h"