[Zope-CMF] Bulk Indexing

[Shane Landrum]

On Mon, Jan 26, 2004 at 09:09:51AM +0000, Chris Withers wrote:
> Shane Landrum wrote:
> 
> >How many ZEO clients do you have running total when you do 
> >this? 
> 
> Only one that does and content writing ;-)
> 
> >Do you have trouble with database ConflictErrors?  
> 
> ...as a result, no, not really...

Ah, right. That's not possible in my environment.

> >It's quite possible that I'm dealing with a unique situation here.
> >We have 7 large Zope servers talking to a very large ZEO server,
> >and unusually for lots of Zope installs, we do a lot of database
> >writes, more or less all the time. We have an elaborate automatic
> >news feed system that's always dumping new content into the database,
> >an automatic workflow approval system for some of that new content, 
> >and a staff of 2-15 editorial people on the system at any one 
> >time, poking and prodding at content.  As a result, it's 
> >relatively likely that at least one object out of several hundred 
> >thousand is going to be being worked on when the reindexing is 
> >done; hence, ConflictErrors. 
> 
> Have you tried the QueuedCatalog?

No, and it doesn't come up on the zope.org products search. However,
a Google search and some poking around solved the problem for me.

For the benefit of the archives:
http://cvs.zope.org/Products/QueueCatalog/ is the CVS version
of QueuedCatalog. From the docstring:

--------------
 A QueueCatalog delegates most requests to a ZCatalog that is named
 as part of the QueueCatalog configuration.

 Requests to catalog or uncatalog objects are queued. They must be
 processed by a separate process (or thread). The queuing provides
 benefits:

 - Content-management operations, performed by humans, complete
 much faster, this making the content-management system more
 effiecient for it's users.

 - Catalog updates are batched, which makes indexing much more
 efficient.

 - Indexing is performed by a single thread, allowing more
 effecient catalog document generation and avoiding conflict
 errors from occuring during indexing.

 - When used with ZEO, indexing might e performed on the same
 machine as the storage server, making updates faster.
--------------

Looks potentially useful. Thanks Chris.

srl
-- 
Shane Landrum, Software Engineer    srl at boston.com
boston.com / NY Times Digital