[Zope-dev] 27 million objects.

[Chris McDonough]

Erik Enge wrote:
> 
> On Thu, 5 Apr 2001, Michael R. Bernstein wrote:
> 
> > I'm trying to find out of there is a point where you start getting
> > non-linear performance penalties for additional objects (storing,
> > retreiving, or indexing).
> 
> I've just finished adding a somewhat small number of objects: 5000.
> For every 1000th object, the Data.fs seemed to grow to about 900MB; that's
> when things started going slow, in a non-linear fashion (this is more a
> hunch than something I payed much attention to).
> 
> I paused the script (fancy Unix-command: "^Z") for every 1000th object,
> packed the database (which shrunk to 19.5MB!  Hmpf.) and restarted the
> script (again, fancy Unix-command: "fg").  Then I was back to the same
> speed as I initially had.

This level of growth doesn't seem like a sane level of growth... what
Zope version are you using?


> 
> Does ZODB have a problem with big Data.fs files?  Not that I
> know.  However, I do have a really fast SCSI-subsystem here so that
> shouldn't be a big problem either.
> 
> I did some copying around with a couple of gigs, and it seems that my
> hunch is right: ZODB does not have a problem with big Data.fs files, the
> hardware does.
> 
> This could be caused indirectly by ZODB if it does too many operations on
> the file, but I'm not too conserned about that.  Ie. a solution could be
> to have ZODB play around with the Data.fs at a less frequent pace, or do
> it in another fashion.  However, that's not really solving any problems,
> unless ZODB is a total maniac with the filesystem.
> 
> I'm converting to ReiserFS this afternoon, maybe that will improve things
> a bit.
> 
> Someone told me that ZEO and bulk-adding could be a thing to look at...

Isn't bulk-adding what you're doing now?