[Zope-CMF] Bulk Indexing

[Chris Withers]

Shane Landrum wrote:

> How many ZEO clients do you have running total when you do 
> this? 

Only one that does and content writing ;-)

> Do you have trouble with database ConflictErrors?  

...as a result, no, not really...

> It's quite possible that I'm dealing with a unique situation here.
> We have 7 large Zope servers talking to a very large ZEO server,
> and unusually for lots of Zope installs, we do a lot of database
> writes, more or less all the time. We have an elaborate automatic
> news feed system that's always dumping new content into the database,
> an automatic workflow approval system for some of that new content, 
> and a staff of 2-15 editorial people on the system at any one 
> time, poking and prodding at content.  As a result, it's 
> relatively likely that at least one object out of several hundred 
> thousand is going to be being worked on when the reindexing is 
> done; hence, ConflictErrors. 

Have you tried the QueuedCatalog?

> We're mitigating this a bit by forcing the ZODB to commit a transaction
> after every 100 objects reindexed. It's not ideal--- I'm uncomfortable
> groping Zope's internals for optimization, because I don't know all
> the implications of forced transaction commits---- but it seems
> not to work too horribly.

Extra commits shouldn't be a problem, if it works for you, then cool :-)

cheers,

Chris