[Zope-dev] 27 million objects.

[Erik Enge]

On Thu, 5 Apr 2001, Michael R. Bernstein wrote:

> I'm trying to find out of there is a point where you start getting
> non-linear performance penalties for additional objects (storing,
> retreiving, or indexing).

I don't know, but I feel that is the case.  Actually, I know it is the
case, but I don't know what is causing it.  I know what isn't helping
though; CatalogAwareness.  I added 2000 objects with XML-RPC.  No other
queries were done during that period.  For each object about 70 DTML
Method/Documents were added.  The first couple of hundres went with a pace
of 2-3 seconds per object.  After that it started to get real slow, and
when I reached about 500 I was down to 5 seconds per object.  I killed
that script, rewrote it to only add 20-25 DTML Methods/Documents and
removed the CatalogAwareness and whoosh!  Under 1 second for each object
and it stayed like that for the entire 2000 objects.

The server is a 1GHz thingy with 1GB RAM.  It wasn't working too hard - it
seemed.
 
> Meanwhile Erik, what approach *did* your programmer take?

Well, the obviously more correct one. :)  He just made the files (that I
were going to index in a Catalog) stay on the filesystem and wrote some
nice regexps to do the searching I though I needed the speed of the
Catalog to do (yeah, yeah, I'm a rookie).

Thanks Jim! :)  
 
> I'll look forward to it.

Ok, and you know what to do if you haven't heard from me and the year is
not 2001 any more ;)