[Zope] ZODB and large data sets

[Jim Fulton]

Jimmie Houchin wrote:
> 
> At 10:48 AM 5/24/99 -0400, Jim Fulton wrote:
> >
> >Jimmie Houchin wrote:
> >>
> >> I am working through the design of my website as I study and learn Zope.
> >>
> >> I've started to work on calculating storage requirements to determine
> >> server needs as I am also working on building my server.
> >>
> >> My app can easily be divided into multiple datasets or databases. One of
> >> the datasets I am looking at has a potential of 4+ million objects with
> >> each object requiring 15-50kb minimum. This dataset can be subdivided.
> >> Initially I will not populate the database will all of the items in their
> >> full form, but will populate as requests for data come. However I need to
> >> develop as if and plan for complete population.
> >>
> >> This makes for a very large database and one that spans more than one hard
> >> drive.
> >>
> >> Can Zope create and use multiple ZODBs on multiple hard drives?
> >
> >ZODB 3 (in Zope 2) will be able to do this eventually.
> 
> Will this be available in Zope 2.0 final, or in a subsequent 2.x release?

Probably in a subsequent release.

> >The file format used by the ZODB 3 FileStorage will also support
> >very large files on systems with large file support (as described in
> >http://www.python.org/doc/current/lib/posix-large-files.html#l2h-1441).
> >
> >So, perhaps your system will let you create large files split over multiple
> >drives, in which case, you'll be able to use large FileStorage-based
> databases.
> 
> Will this be available in Zope 2.0 final or even in the betas, or in a
> subsequent 2.x release?

It's available now. (Zope 2.0 alpha 1.)
 
> At the above URL it mentions these OSes/systems as being capable:
> AIX, HPUX, Irix and Solaris
> 
> Do you know if RedHat Linux running on an AlphaServer such as DS10 would be
> capable?

I have no idea.
 
> >> If so, how will such a large dataset affect packing and the creation of the
> >> backup file?
> >
> >If you are using an OS with large file support, then this should not be a
> problem,
> >as long as you have enough disk space.  If you use multiple databases (when
> >Zope supports them) you'll have to pack them individually.
> >
> >> Will I need to use multiple other database backends such as MySQL or
> >> possibly MetaKit?
> >
> >For what?
> 
> Maybe I wasn't clear here. If Zope didn't do multiple ZODBs then I could
> resort to using MySQL with multiple databases. Not my preference. It kind
> of looks like I mean't multiple MySQLs or MetaKits.

Hm.  You could also try to extend the ZODB 3 FileStorage to split a 
storage over multiple files.  This *might* not be that hard.
Maybe a MultiFileStorage. I think this would be pretty worthwhile
and would be willing to give alot of advice to make it happen.
 
> I would like to stay within the Z System as much as possible.

You should. :)
 
> My Zope Zen may have much growing to do, but I'm sold on Zope. :)

Yee ha! :)

Jim

--
Jim Fulton           mailto:jim@digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org    

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission.  Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.