[Zope-dev] Massive scalability

[Steve Alexander]

Michael Bernstein wrote:

> I am currently planning two separate 'Archive' type
> projects/Products. In both cases, I need to make sure that
> my implementation will scale to hundreds of thousands or
> even millions of objects.
 >

> In one project the objects are very simple ZClasses with a
> few attributes, in the other project, the objects will be
> instances of the Photo Product, and considerably larger.

Do you mean "instances of the Photo Product" or "instances of class 
FooBar from the Photo Product" ?

> One implementation I'm considering is a simple Specialist
> with a Rack. Does anyone know if there are any inherent
> limitations on the number of objects that can be stored in a
> Rack? are there any performance limitations at the scale
> that I'm talking about?

Seeing as a Rack can provide data from absolutely anywhere, I can't see 
a problem with this.
If you're talking about the BTree implementation that Racks use when 
they store data in the ZODB, well, I've never stored more than a few 
thousand objects in one of those. There certainly aren't the same 
limitations that you get with the default ObjectManager, as that uses a 
python dict to hold its sub-objects.

The performance limitations will more likely be to do with searching and 
indexing the data, adding the data in bulk (if you need to do this), and 
retrieving the data if you have a vast number of clients wanting it all 
at once.

> The other implementation I'm considering is to create a
> ZClass that inherits from ZCatalog and Btree Folder.

I can't think why you'd want to do that. What role would instances of 
this class play in your application?

> Would
> this approach run into any scalability problems with the
> number and type of objects I'm talking about?

I think other aspects of your application will determine whether it will 
scale. Scalabillity is an emergent property of a system. You only get to 
know about it when you consider the system holisticly.

With Zope, where you store objects and how you plan to find objects, is 
more significant than what the objects you're storing are.

--
Steve Alexander
Software Engineer
Cat-Box limited
http://www.cat-box.net