[Zope-dev] ZCatalog scalability

[Michael R. Bernstein]

Erik Enge wrote:
> 
> [Michael R. Bernstein]
> 
> | Erik Enge wrote:
> | >
> | > I'll let you know how it goes.  (And please, do poke at me if it takes
> | > too long.)
> |
> | Ok, I'm poking :-).
> 
> Thanks.  Keep doing it till you get what you need, I truly don't
> mind.  :-)
> 
> | How did it go?
> 
> Thanks to the speed of delivery at Royal Mail in the UK, I haven't
> recieved the data yet (!).  They promised to have it too me by the
> coming friday (the 23rd, I believe).  I'll try to process all 27
> million records and give feeback to the community.
> 
> (I did a premature calculation of how long it would take to populate
> the Zope instance, and I'm guessing somewhere between 80 to 100 hours,
> meaning the feedback cannot come any sooner than Sunday, at best.)

What I'm looking for is any indication that object creation
time and/or indexing time goes up with the number of objects
already in the ZODB.

Will you be populating the ZODB in batches (say 100,000
objects or so)? If so, can you benchmark each batch, so we
can see if the batch proccessing time goes up as you
progress through the 270 batches?

Thanks,

Michael Bernstein.