[Zope-CMF] Re: Bulk Indexing
Tres Seaver
tseaver at zope.com
Mon Jan 26 12:01:25 EST 2004
Shane Landrum wrote:
> On Mon, Jan 26, 2004 at 09:09:51AM +0000, Chris Withers wrote:
>
>>Shane Landrum wrote:
>>
>>
>>>How many ZEO clients do you have running total when you do
>>>this?
>>
>>Only one that does and content writing ;-)
>>
>>
>>>Do you have trouble with database ConflictErrors?
>>
>>...as a result, no, not really...
>
>
> Ah, right. That's not possible in my environment.
>
>
>>>It's quite possible that I'm dealing with a unique situation here.
>>>We have 7 large Zope servers talking to a very large ZEO server,
>>>and unusually for lots of Zope installs, we do a lot of database
>>>writes, more or less all the time. We have an elaborate automatic
>>>news feed system that's always dumping new content into the database,
>>>an automatic workflow approval system for some of that new content,
>>>and a staff of 2-15 editorial people on the system at any one
>>>time, poking and prodding at content. As a result, it's
>>>relatively likely that at least one object out of several hundred
>>>thousand is going to be being worked on when the reindexing is
>>>done; hence, ConflictErrors.
>>
>>Have you tried the QueuedCatalog?
>
>
> No, and it doesn't come up on the zope.org products search. However,
> a Google search and some poking around solved the problem for me.
>
> For the benefit of the archives:
> http://cvs.zope.org/Products/QueueCatalog/ is the CVS version
> of QueuedCatalog. From the docstring:
>
> --------------
> A QueueCatalog delegates most requests to a ZCatalog that is named
> as part of the QueueCatalog configuration.
>
> Requests to catalog or uncatalog objects are queued. They must be
> processed by a separate process (or thread). The queuing provides
> benefits:
>
> - Content-management operations, performed by humans, complete
> much faster, this making the content-management system more
> effiecient for it's users.
>
> - Catalog updates are batched, which makes indexing much more
> efficient.
>
> - Indexing is performed by a single thread, allowing more
> effecient catalog document generation and avoiding conflict
> errors from occuring during indexing.
>
> - When used with ZEO, indexing might e performed on the same
> machine as the storage server, making updates faster.
> --------------
>
> Looks potentially useful. Thanks Chris.
Shane,
Chat with Marc about this one: we already have QueueCatalog in the
buildout, but never wired it in, largely because it has expectations of
the "real" catalog which the asset repository doesn't fulfill.
Tres.
--
===============================================================
Tres Seaver tseaver at zope.com
Zope Corporation "Zope Dealers" http://www.zope.com
More information about the Zope-CMF
mailing list