[Zope] ZCTextIndex question

[Jonathan Hobbs]

From: "Chris Withers" <chris at simplistix.co.uk>


> Jonathan Hobbs wrote:
>
> > I have a ZCatalog that has grown to about 6.3Gb in size and as a result
no
> > longer fits in memory - resulting in a severe degradation in performance
(a
> > search that used to take less than 2 cpu seconds, now takes 5-14 cpu
> > seconds).  Is there any way to 'extract' a single index from the
ZCatalog (a
> > ZCTextIndex) and hold it in memory to improve search retrieval
performance ?
> >
> > Also, because we are using the OkapiIndex relevance ranking process, we
will
> > need to hold the Lexicon in memory.  Has anyone tried/accomplished this?
> >
> > Hoping there is a solution...
>
> Try ZODB mount points, moutn the index and the lexicon in a seperate
> storage...

Thanks for the idea Chris!

I can't see a way to take a single catalog and put the indexes into separate
storages (DBTab wants a folder as a mount point, but ZCatalog index creation
does not allow for this), however your suggestion has given me the idea of
breaking the catalog into several pieces (one index + lexicon in each piece)
and putting each 'sub-catalog' into its own storage on a temporary folder.
I would then using a script/external method to combine the results from each
catalog; sounds ugly but at this point I am willing to try anything!

Thanks again,

Jonathan