[Zope-dev] Massive scalability

[Michael Bernstein]

Steve Alexander wrote:
> 
> Michael Bernstein wrote:
> 
> > I am currently planning two separate 'Archive' type
> > projects/Products. In both cases, I need to make sure that
> > my implementation will scale to hundreds of thousands or
> > even millions of objects.
>  >
> 
> > In one project the objects are very simple ZClasses with a
> > few attributes, in the other project, the objects will be
> > instances of the Photo Product, and considerably larger.
> 
> Do you mean "instances of the Photo Product" or "instances of class
> FooBar from the Photo Product" ?

Sorry. I meant instances of class Photo from the Photo
Product.

> > Does anyone know if there are any inherent
> > limitations on the number of objects that can be stored in a
> > Rack? are there any performance limitations at the scale
> > that I'm talking about?
> 
> If you're talking about the BTree implementation that Racks use when
> they store data in the ZODB, well, I've never stored more than a few
> thousand objects in one of those. There certainly aren't the same
> limitations that you get with the default ObjectManager, as that uses a
> python dict to hold its sub-objects.
> 
> The performance limitations will more likely be to do with searching and
> indexing the data, adding the data in bulk (if you need to do this), and
> retrieving the data if you have a vast number of clients wanting it all
> at once.

Yes, I was referring to the way a Rack stores data in the
ZODB.

Photo instances store several sizes of the same image as
attributes of the object, as well as various meta-data
fields. I anticipate indexing the meta-data in a ZCatalog.

Data will not be added in bulk, but several people may want
to retreive the data at the same time (if the site becomes
popular).

> > The other implementation I'm considering is to create a
> > ZClass that inherits from ZCatalog and Btree Folder.
> 
> I can't think why you'd want to do that. What role would instances of
> this class play in your application?

The same as the Rack. A single archive of indexed objects.
It seems that this would scale better than creating a ZClass
that inherits from ZCatalog and ObjectManager as described
here:

http://www.zope.org/Members/tseaver/inherit_ZCatalog

> > Would
> > this approach run into any scalability problems with the
> > number and type of objects I'm talking about?
> 
> I think other aspects of your application will determine whether it will
> scale. Scalabillity is an emergent property of a system. You only get to
> know about it when you consider the system holisticly.

The system is fairly simple: I want to store a large number
of objects in a single location, I want to index them in a
ZCatalog, I want to find objects by either searching for
them, or by browsing a ZTopics based heirarchy (that is
populated with ZCatalog searches as well).

The search time (whether a user or ZTopic initiates it)
should happen fairly fast, regardless of the number of
objects (potentially hundreds of thousands), and direct
object retreivals (or rendering) should also happen quickly,
without major penalties for the number of objects.

> With Zope, where you store objects and how you plan to find objects, is
> more significant than what the objects you're storing are.

I hope I've explained myself better this time.

Thanks for the help,

Michael Bernstein.