[ZODB-Dev] [OT] NoSQL

[Roché Compaan]

On Sat, 2009-11-14 at 14:23 -0700, Shane Hathaway wrote:
> Roché Compaan wrote:
> > On Fri, 2009-11-13 at 13:33 -0700, Shane Hathaway wrote:
> >> Stephan Richter wrote:
> >>> http://svn.zope.org/z3c.sharding/trunk
> > 
> > Great stuff! This approaches scaling a large data set at application
> > level though. Don't you think a ZODB storage doing this for you would
> > solve the problem more generally?
> 
> I think proper construction of horizontally scalable databases must be 
> done partly at application level, since a lot of the issues to be solved 
> are specific to the application.

What are the issues you're talking about?

> > I think that the master index needs to be partitioned as well. In
> > benchmarks I performed early last year (http://bit.ly/pSVmd), a BTree
> > could only handle about 250 inserts / second when it approached 10
> > million objects, so I'm guessing it will be almost unusable at a 100
> > million.
> 
> Right.  The top level probably ought to be a dynamic hash, not a BTree. 
>   I intended z3c.sharding more as a proof of concept.

Sure, just thought I'd mention it. 

Also keep in mind that the dynamic hash needs to handle the introduction
of new partitions.


-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za