[Zope-dev] Specialist/Rack scalability

[Michael Bernstein]

Steve Alexander wrote:
> 
> Hi Michael,
> 
> Michael Bernstein wrote:
> 
> >
> > It seems clear that indexing and searching are more of a
> > botleneck than storage/retreival. Nevertheless, so far I
> > have not heard of anyone trying to store more than 60,000
> > objects in a rack. I need to know if there is any reason to
> > suspect that storage (in the ZODB) or retreival performance
> > would suffer if the number of objects was in the hundreds of
> > thousands or even millions.
> 
> [snip]
>
> So, storing things in a Rack happens in a number of stages:
> 
>    Your application interacts with the Rack
>    The Rack (perhaps) stores the object persistently in its BTree
>    The BTree is a collection of persistent ZODB objects
>    The ZODB objects are stored as Python Pickles in a FileStorage
> 
> We can consider what the effect of storing 60 000 objects is at each of
> these interfaces.

Are there any differences if you scale the number of objects
up to the hundreds of thousands or even into the millions?

> The Rack shouldn't have a problem with 60 000 objects.
> 
> I doubt a BTree would have a problem.
> 
> The ZODB might not like accessing many large objects during a single
> transaction, as all those objects need to be in memory at once.

Neither of my applications require batch adds to the DB,
however, one of them (the image archive) has objects
(Photos) with several images as attributes. This results in
a fairly large object. There is some question in my mind if
accessing any attribute (such as the thumbnail version)
causes all attributes to be loaded into memory. If so,
displaying a list of images with thumbnails may result in
many large objects being loaded into memory.

> A FileStorage should have no problem reading 60 000 stored objects.
> However, if these objects are changing much, your Data.fs will grow
> fast. In any case, you may find undo and history screens take a long
> time to appear.

No. Once added, I don't expect the data to change
frequently.

Thanks for the feedback.

Michael Bernstein.