RDM wrote:
On Mon, 15 Jan 2001, Michael Bernstein wrote:
as Squishdot). Adding a dependency on an RDBMS or requiring additional setup on the server's FS seems a step in the wrong direction. [...] So the question remains: Will either approach (within the ZODB) allow me to scale the application to hundreds of thousands (or even millions) of objects indexed in a ZCatalog? [...] I know that the ZCatalog/ObjectManager approach used by Squishdot will scale to over 9,000 objects (the number of postings to date at technocrat.net), So I'm reasonably certain that my proposed ZCatalog/BTree Folder approach will be at least as scalable. I'm slightly less confident about the Specialist/Rack approach, because I don't know of any sites that have used them to store that many objects in the ZODB, but only slightly.
My understanding is that the point of ZPatterns is to hide the data storage implementation from the application.[snip] The point being that you can *change your mind* later, with minimal disruption to your application. Not only that, but people who *use* your product can make their own decision about where to store the data. So by using ZPatterns you [...] let the users of your product use an RDBMs if that works better for them.
Very good points, and ones that I will keep in mind. Thanks.
In addition, it seems to me that your comments about ZCatalog+BTree apply equally well to ZPatterns, since you can use the Catalog to index stuff stored in a rack through the use of appropriate triggers, and it is my understanding that the default in-ZODB rack storage uses BTree internally.
I do not know if BTree folders and Racks share the same B-Tree implementation, which is why I qualified my statement as 'slightly less confident'.
Unfortunately I don't have much input on your question about real-life scalability...the most I've done is stored 60000 small objects in a hierarchy of zope folders, indexed by the catalog, with no perceptable slowdown in search or retrieval speed.
Hmm. John Eikenberry mentioned a slowdown with about 50,000 objects on partial-match searches, but I don't know how simple/complex the objects were, or how many atributes were being indexed. How many indexes of various types was your ZCatalog maintaining on your objects? Thanks, Michael Bernstein.