[Zope] ZODB and large data sets

[Jimmie Houchin]

At 10:48 AM 5/24/99 -0400, Jim Fulton wrote:
>
>Jimmie Houchin wrote:
>> 
>> I am working through the design of my website as I study and learn Zope.
>> 
>> I've started to work on calculating storage requirements to determine
>> server needs as I am also working on building my server.
>> 
>> My app can easily be divided into multiple datasets or databases. One of
>> the datasets I am looking at has a potential of 4+ million objects with
>> each object requiring 15-50kb minimum. This dataset can be subdivided.
>> Initially I will not populate the database will all of the items in their
>> full form, but will populate as requests for data come. However I need to
>> develop as if and plan for complete population.
>> 
>> This makes for a very large database and one that spans more than one hard
>> drive.
>> 
>> Can Zope create and use multiple ZODBs on multiple hard drives?
>
>ZODB 3 (in Zope 2) will be able to do this eventually.  

Will this be available in Zope 2.0 final, or in a subsequent 2.x release?

>The file format used by the ZODB 3 FileStorage will also support 
>very large files on systems with large file support (as described in
>http://www.python.org/doc/current/lib/posix-large-files.html#l2h-1441).
>
>So, perhaps your system will let you create large files split over multiple
>drives, in which case, you'll be able to use large FileStorage-based
databases.

Will this be available in Zope 2.0 final or even in the betas, or in a
subsequent 2.x release?

At the above URL it mentions these OSes/systems as being capable:
AIX, HPUX, Irix and Solaris

Do you know if RedHat Linux running on an AlphaServer such as DS10 would be
capable?

>> If so, how will such a large dataset affect packing and the creation of the
>> backup file?
>
>If you are using an OS with large file support, then this should not be a
problem, 
>as long as you have enough disk space.  If you use multiple databases (when
>Zope supports them) you'll have to pack them individually.
> 
>> Will I need to use multiple other database backends such as MySQL or
>> possibly MetaKit?
>
>For what?

Maybe I wasn't clear here. If Zope didn't do multiple ZODBs then I could
resort to using MySQL with multiple databases. Not my preference. It kind
of looks like I mean't multiple MySQLs or MetaKits.

I would like to stay within the Z System as much as possible.

My Zope Zen may have much growing to do, but I'm sold on Zope. :)

Y'all are doing great work.

Thanks.

Jimmie Houchin

>Jim
>
>--
>Jim Fulton           mailto:jim@digicool.com   Python Powered!        
>Technical Director   (888) 344-4332            http://www.python.org  
>Digital Creations    http://www.digicool.com   http://www.zope.org