On Mon, 2007-09-17 at 19:57 -0400, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Roché Compaan wrote:
Hi there
Are the current indexing solutions that we have in Zope 2 and 3 the best that we can have? Is there any way to improve the amount of concurrent indexing a field or text index can handle. The Zope 3 implementation doesn't look significantly different to the Zope 2 one in that it still uses a BTree for forward and reverse index.
I have a basic stress test where 10 concurrent threads index 10 random words in a FieldIndex and get too many conflict errors to even consider this a usable scalable indexing solution. I don't really want to index objects in another backend. I really would like to make the ZODB work for me here.
Are there solutions here? Can one employ some of the QueueCatalog conflict resolution strategies to make indexes more resilient? Or should one use some locking strategy instead?
Check out QueueCatalog, which batches up indexing changes for processing within a single thread.
svn://svn.zope.org/repos/main/Products.QueueCatalog
I use QueueCatalog often and I know how it works. But if an application requires immediate indexing then QueueCatalog is not a solution. Sorry if I was unclear but what I'm really asking is if it is possible to improve the conflict handling of the current indexes that we have in Zope. I am also asking if concurrent indexing in the ZODB is a realistic goal. The reason I mentioned QueueCatalog is not because of its batch indexing in one thread, but because it has a lot of conflict handling on the queue itself. Sessions might be another good example of a product that tries hard to handle conflicts. Do these products have strategies that can be made to work for indexes? -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za