John Eikenberry wrote:
Michael Bernstein wrote:
So, again: Has anyone run up against any performance or other limitations regarding large numbers (hundreds of thousands or more) of objects stored within the ZODB either in a BTree Folder or a Rack?
I was looking into the same issues recently, but for a much smaller set of data (50000ish). In my tests ZPatterns/binary-trees scaled well for storage and retrieval. But ZCatalog did not. It was basically useless for partial matching searches (taking many minutes for searches that retrieved more than 100 matches)
Was this true even for cases where the batch size was smaller than 100? For example, if a search returns over 100 results but the batch size is only 20 (so that only 20 results at a time are displayed), do you still get the performance hit?
[snip] I ended up deciding to go with a RDBMS backend for data storage with a ZPatterns interface. SkinScripts work so well for this that I'm actually glad I switched. It simplified my design and implementation immensely.
So you're saying that you are doing all searching using SQL statements, and not just object retreival and storage, correct? How are you handling full text searches? Cheers, Michael Bernstein.