[Zope] ZODB and large data sets

[Jim Fulton]

Jimmie Houchin wrote:
> 
> I am working through the design of my website as I study and learn Zope.
> 
> I've started to work on calculating storage requirements to determine
> server needs as I am also working on building my server.
> 
> My app can easily be divided into multiple datasets or databases. One of
> the datasets I am looking at has a potential of 4+ million objects with
> each object requiring 15-50kb minimum. This dataset can be subdivided.
> Initially I will not populate the database will all of the items in their
> full form, but will populate as requests for data come. However I need to
> develop as if and plan for complete population.
> 
> This makes for a very large database and one that spans more than one hard
> drive.
> 
> Can Zope create and use multiple ZODBs on multiple hard drives?

ZODB 3 (in Zope 2) will be able to do this eventually.  The file format
used by the ZODB 3 FileStorage will also support very large files on
systems with large file support (as described in
http://www.python.org/doc/current/lib/posix-large-files.html#l2h-1441).

So, perhaps your system will let you create large files split over multiple
drives, in which case, you'll be able to use large FileStorage-based databases.

> If so, how will such a large dataset affect packing and the creation of the
> backup file?

If you are using an OS with large file support, then this should not be a problem, 
as long as you have enough disk space.  If you use multiple databases (when
Zope supports them) you'll have to pack them individually.
 
> Will I need to use multiple other database backends such as MySQL or
> possibly MetaKit?

For what?

Jim

--
Jim Fulton           mailto:jim@digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org    

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission.  Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.