"Pausing" ZCatalog and re-indexing later
Hi :-) The system I am working on is using a ZCatalog with some indexes (over 10) for faster searching. Unfortunatelly there are some use cases, where I have to add really many new objects to ZODB. It works, but very slowly. I suspect that this is because when I add an object to Folder (with _setObject) it gets indexed - couple thousands of objects multiplied by over 10 indexes = looooong. Am I guessing right or am I completely lost? ;-) If this is the probable reason - is there some way I could disable indexing temporarily and after adding all of the objects make it re-index the new ones? I think that even a full re-index would work faster than it works now... I would be very grateful for any suggestions. Best regards, Maciek
--On 11. August 2006 17:10:29 +0000 krokodylek@tenbit.pl wrote:
Hi :-)
The system I am working on is using a ZCatalog with some indexes (over 10) for faster searching.
Unfortunatelly there are some use cases, where I have to add really many new objects to ZODB. It works, but very slowly.
What kind of objects?
I suspect that this is because when I add an object to Folder (with _setObject) it gets indexed - couple thousands of objects multiplied by over 10 indexes = looooong.
Consider using CatalogQueue to defer indexing.
Am I guessing right or am I completely lost? ;-)
No idea, you might investigate your bottleneck(s) using ZopeProfiler. -aj
----- Original Message ----- From: <krokodylek@tenbit.pl> To: <zope@zope.org> Sent: Friday, August 11, 2006 1:10 PM Subject: [Zope] "Pausing" ZCatalog and re-indexing later
Hi :-)
The system I am working on is using a ZCatalog with some indexes (over 10) for faster searching.
Unfortunatelly there are some use cases, where I have to add really many new objects to ZODB. It works, but very slowly. I suspect that this is because when I add an object to Folder (with _setObject) it gets indexed - couple thousands of objects multiplied by over 10 indexes = looooong.
Am I guessing right or am I completely lost? ;-)
If this is the probable reason - is there some way I could disable indexing temporarily and after adding all of the objects make it re-index the new ones? I think that even a full re-index would work faster than it works now...
I would be very grateful for any suggestions.
Another approach is to build/update a new ZCatalog in another .fs file (you can do this offline, on another box, etc), and then just swap in the new .fs file. You can also create a mount point for a folder that contains nothing but the ZCatalog and then just swap in that (instead of swapping the entire Data.fs file). hth Jonathan
participants (3)
-
Andreas Jung -
Jonathan -
krokodylek＠tenbit.pl