[Zope-dev] Massive scalability

[John Eikenberry]

Michael Bernstein wrote:

> So, again: Has anyone run up against any performance or
> other limitations regarding large numbers (hundreds of
> thousands or more) of objects stored within the ZODB either
> in a BTree Folder or a Rack?

I was looking into the same issues recently, but for a much smaller set of
data (50000ish). In my tests ZPatterns/binary-trees scaled well for storage
and retrieval. But ZCatalog did not. It was basically useless for partial
matching searches (taking many minutes for searches that retrieved more
than 100 matches). I was also concerned about the indexing overhead. It
doesn't scale well when changing/adding many things at a time (we might
have bulk adds/changes).

I ended up deciding to go with a RDBMS backend for data storage with a
ZPatterns interface. SkinScripts work so well for this that I'm actually
glad I switched. It simplified my design and implementation immensely. 

-- 

John Eikenberry
[jae@zhar.net - http://zhar.net]
______________________________________________________________
"A society that will trade a little liberty for a little order
 will deserve neither and lose both."
                                          --B. Franklin