[Zope-dev] Massive scalability

[Michael Bernstein]

Andy McKay wrote:
> 
> > I am currently planning two separate 'Archive' type
> > projects/Products. In both cases, I need to make sure that
> > my implementation will scale to hundreds of thousands or
> > even millions of objects.
> 
> I would recommend using an RDMBS behind Zope then. Its faster, simpler and I
> have always had better results.

While that would work for the simple object case, I find the
prospect of storing a bunch of BLOBs (for the image data of
the Photos) in an RDBMS to be *most* un-appetizing. Storing
them on the server's file-system seems in-elegant as well.

I'd like to build both of these applications as products
that can be easily installed into a Zope server (as easily
as Squishdot). Adding a dependency on an RDBMS or requiring
additional setup on the server's FS seems a step in the
wrong direction.

I'm working with objects here. I prefer to work with them
compared to the approach of decomposing my objects into
RDBMS records or files, and recomposing.

So the question remains: Will either approach (within the
ZODB) allow me to scale the application to hundreds of
thousands (or even millions) of objects indexed in a
ZCatalog?

I should stress that I am far more concerned about the
number of objects than I am about the number of 'hits'.

I know that the ZCatalog/ObjectManager approach used by
Squishdot will scale to over 9,000 objects (the number of
postings to date at technocrat.net), So I'm reasonably
certain that my proposed ZCatalog/BTree Folder approach will
be at least as scalable. I'm slightly less confident about
the Specialist/Rack approach, because I don't know of any
sites that have used them to store that many objects in the
ZODB, but only slightly.

However, so far I have not heard of anyone storing and
indexing as many homogenous objects as I am talking about.
I'd really like to hear from anyone who has attempted to do
this or something similar.

Does anyone know of any hidden 'gotchas' when dealing with
this many objects, regardless of the hit-load on the system?

Thanks,

Michael Bernstein.