[Zope-dev] Massive scalability

[Jimmie Houchin]

Michael Bernstein wrote:
> Jimmie Houchin wrote:
> >
> > I would like to echo Michael's sentiment and comments. I am at the
> > beginning stage of Zope development of a website which will have
> > millions of objects of a single object type and multiple of such.
> 
> are your objects intended to be indexed by ZCatalog as well,
> or are you planning some other method of finding your
> objects?

Yes as far as I know at this time. I am developing off of a cvs build. I
keep it updated regularly. This provides me with the latest in ZCatalog
and other development.

> > [snip stuff about mounted databases]
> >
> > I think there are many who would like to keep their development within
> > Zope using objects. Any information on best use or development
> > strategies for such setups would be greatly appreciated.
> 
> Thanks for adding your voice!

Your welcome. :)

> I also think that applications such as we're discussing
> should be possible to deploy using nothing but Zope's built
> in capabilities. I understand that Zope has certain
> limitations in situations that require many writes to the
> database, and I accept those limiattions as being a
> neccessary trade-off for a storage strategy that uses an
> appending file format. I am still trying to determine if
> Zope (especially using the two development options I
> outlined) has any built in limitations regarding the number
> (or number x size) of objects stored.

I agree and think yours is a good and valid question.

> I really appreciate Zope's features (especially with
> ZPatterns) that allow applications to be developed that are
> storage-agnostic, and I feel that this is especially useful
> for tying into existing legacy systems, but I don't want to
> develop a new application, storing new data, that is tied to
> a specific storage methodology such as an RDBMS.

Agreed. It also should be recognized that RDBMSes are not without issues
and problems of their own, especially when you are dealing with objects. 

> People who wish to customize an application to leverage
> their existing legacy data should be free to do so, but I've
> noticed that Zope products that have some external system as
> a pre-requisite (Worldpilot, ZCommerce, etc.) are deployed
> far less often than those which do not (Squishdot, Zwiki,
> etc.).
> 
> So, again: Has anyone run up against any performance or
> other limitations regarding large numbers (hundreds of
> thousands or more) of objects stored within the ZODB either
> in a BTree Folder or a Rack?
> 
> In other words, will the system slow down if you add enough
> objects?

If the general consensus came back and said that currently the ZODB is
inappropriate for such magnitude, the next place I would look is into
Forethought's OODBMS implementation 4ODS. It would require additional
work such as installing whatever backend you wish to use with it. It
currently works with I believe Oracle and PostgreSQL. Other backends are
or will be developed. But development is OO.

I don't know if anyone has used or considered using 4ODS with Zope.
Could provide interesting options. But I'm for all Zope first.

Jimmie Houchin

> Thanks,
> 
> Michael Bernstein.