ZCTextIndex question
I have a ZCatalog that has grown to about 6.3Gb in size and as a result no longer fits in memory - resulting in a severe degradation in performance (a search that used to take less than 2 cpu seconds, now takes 5-14 cpu seconds). Is there any way to 'extract' a single index from the ZCatalog (a ZCTextIndex) and hold it in memory to improve search retrieval performance ? Also, because we are using the OkapiIndex relevance ranking process, we will need to hold the Lexicon in memory. Has anyone tried/accomplished this? Hoping there is a solution... Jonathan
Jonathan Hobbs wrote:
I have a ZCatalog that has grown to about 6.3Gb in size and as a result no longer fits in memory - resulting in a severe degradation in performance (a search that used to take less than 2 cpu seconds, now takes 5-14 cpu seconds). Is there any way to 'extract' a single index from the ZCatalog (a ZCTextIndex) and hold it in memory to improve search retrieval performance ?
Also, because we are using the OkapiIndex relevance ranking process, we will need to hold the Lexicon in memory. Has anyone tried/accomplished this?
Hoping there is a solution...
Try ZODB mount points, moutn the index and the lexicon in a seperate storage... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
From: "Chris Withers" <chris@simplistix.co.uk>
Jonathan Hobbs wrote:
I have a ZCatalog that has grown to about 6.3Gb in size and as a result no longer fits in memory - resulting in a severe degradation in performance (a search that used to take less than 2 cpu seconds, now takes 5-14 cpu seconds). Is there any way to 'extract' a single index from the ZCatalog (a ZCTextIndex) and hold it in memory to improve search retrieval performance ?
Also, because we are using the OkapiIndex relevance ranking process, we will need to hold the Lexicon in memory. Has anyone tried/accomplished this?
Hoping there is a solution...
Try ZODB mount points, moutn the index and the lexicon in a seperate storage...
Thanks for the idea Chris! I can't see a way to take a single catalog and put the indexes into separate storages (DBTab wants a folder as a mount point, but ZCatalog index creation does not allow for this), however your suggestion has given me the idea of breaking the catalog into several pieces (one index + lexicon in each piece) and putting each 'sub-catalog' into its own storage on a temporary folder. I would then using a script/external method to combine the results from each catalog; sounds ugly but at this point I am willing to try anything! Thanks again, Jonathan
participants (2)
-
Chris Withers -
Jonathan Hobbs