[Zope-dev] Massive scalability

[Jimmie Houchin]

I would like to echo Michael's sentiment and comments. I am at the
beginning stage of Zope development of a website which will have
millions of objects of a single object type and multiple of such. I
currently plan on using mounted databases for the various object
repositories and am currently exploring the best means to implement
such. I know of the mountedfilestorage product by Anthony, but with the
massive refactoring happening for 2.3 I am watching the horizon for how
such will be implemented in the future.

I think there are many who would like to keep their development within
Zope using objects. Any information on best use or development
strategies for such setups would be greatly appreciated. 

Thanks,

Jimmie Houchin


Michael Bernstein wrote:
> 
> Andy McKay wrote:
> >
> > > I am currently planning two separate 'Archive' type
> > > projects/Products. In both cases, I need to make sure that
> > > my implementation will scale to hundreds of thousands or
> > > even millions of objects.
> >
> > I would recommend using an RDMBS behind Zope then. Its faster, simpler and I
> > have always had better results.
> 
> While that would work for the simple object case, I find the
> prospect of storing a bunch of BLOBs (for the image data of
> the Photos) in an RDBMS to be *most* un-appetizing. Storing
> them on the server's file-system seems in-elegant as well.
> 
> I'd like to build both of these applications as products
> that can be easily installed into a Zope server (as easily
> as Squishdot). Adding a dependency on an RDBMS or requiring
> additional setup on the server's FS seems a step in the
> wrong direction.
> 
> I'm working with objects here. I prefer to work with them
> compared to the approach of decomposing my objects into
> RDBMS records or files, and recomposing.
> 
> So the question remains: Will either approach (within the
> ZODB) allow me to scale the application to hundreds of
> thousands (or even millions) of objects indexed in a
> ZCatalog?
> 
> I should stress that I am far more concerned about the
> number of objects than I am about the number of 'hits'.
> 
> I know that the ZCatalog/ObjectManager approach used by
> Squishdot will scale to over 9,000 objects (the number of
> postings to date at technocrat.net), So I'm reasonably
> certain that my proposed ZCatalog/BTree Folder approach will
> be at least as scalable. I'm slightly less confident about
> the Specialist/Rack approach, because I don't know of any
> sites that have used them to store that many objects in the
> ZODB, but only slightly.
> 
> However, so far I have not heard of anyone storing and
> indexing as many homogenous objects as I am talking about.
> I'd really like to hear from anyone who has attempted to do
> this or something similar.
> 
> Does anyone know of any hidden 'gotchas' when dealing with
> this many objects, regardless of the hit-load on the system?
> 
> Thanks,
> 
> Michael Bernstein.