[Zope-dev] Massive scalability

Zope mailing lists bitz@bitdance.com
Tue, 16 Jan 2001 13:47:38 -0500 (EST)


On Mon, 15 Jan 2001, Michael Bernstein wrote:
> as Squishdot). Adding a dependency on an RDBMS or requiring
> additional setup on the server's FS seems a step in the
> wrong direction.
[...]
> So the question remains: Will either approach (within the
> ZODB) allow me to scale the application to hundreds of
> thousands (or even millions) of objects indexed in a
> ZCatalog?
[...]
> I know that the ZCatalog/ObjectManager approach used by
> Squishdot will scale to over 9,000 objects (the number of
> postings to date at technocrat.net), So I'm reasonably
> certain that my proposed ZCatalog/BTree Folder approach will
> be at least as scalable. I'm slightly less confident about
> the Specialist/Rack approach, because I don't know of any
> sites that have used them to store that many objects in the
> ZODB, but only slightly.

My understanding is that the point of ZPatterns is to hide
the data storage implementation from the application.  You refer
to this in another post.  The point being that you can *change your
mind* later, with minimal disruption to your application.  Not
only that, but people who *use* your product can make their
own decision about where to store the data.  So by using ZPatterns
you would have a product that could be installed without
any addons other than ZPatterns itself, and yet let the users
of your product use an RDBMs if that works better for them.

In addition, it seems to me that your comments about ZCatalog+BTree
apply equally well to ZPatterns, since you can use the Catalog
to index stuff stored in a rack through the use of appropriate
triggers, and it is my understanding that the default in-ZODB
rack storage uses BTree internally.

So overall I'd think going with ZPatterns would be the right
decision.  It gives you and others using your code the
maximum flexability.

Unfortunately I don't have much input on your question about
real-life scalability...the most I've done is stored 60000 small objects
in a hierarchy of zope folders, indexed by the catalog, with
no perceptable slowdown in search or retrieval speed.

--RDM