[Zope] ZCTextIndex Lexicon problems (resolved)

[Jonathan Hobbs]

----- Original Message -----
From: "Jonathan Hobbs" <toolkit at magma.ca>
> I have bulk uploaded about 100k records (zclass instances) into a
> BTreeFolder2 and am using the following code to create entries in a
zcatalog
> (I am not using catalog-aware instances):
>
> import os, sys, traceback
> import Zope
>
> app=Zope.app()
> from Testing import makerequest
> app = makerequest.makerequest(app)
>
> for item in app.sw.WebSitesFolder.WebSitesData.objectItems():
>         uid = item[0]
>         obj = item[1]
>         app.sw.WebSitesFolder.Catalog.catalog_object(obj, uid)
>         if cnt == 1000:
>                 get_transaction().commit()
>                 cnt=0
>
> get_transaction().commit()
>
>
> This works if I only have about 10k entries in the BTreeFolder2
> (WebSitesData), but when I have larger volumes of records the Lexicon
> doesn't work:  ie. 0 ( yes, 'zero') entries in the Lexicon, but no errors.
> The zcatalog contains the correct number of entries, but when you try to
> access a record in the zcatalog (via the ZMI) you get the following error:
>
> Traceback (innermost last):
>   Module ZPublisher.Publish, line 98, in publish
>   Module ZPublisher.mapply, line 88, in mapply
>   Module ZPublisher.Publish, line 39, in call_object
>   Module Shared.DC.Scripts.Bindings, line 252, in __call__
>   Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec
>   Module App.special_dtml, line 174, in _exec
>   Module DocumentTemplate.DT_In, line 628, in renderwob
>   Module DocumentTemplate.DT_Util, line 201, in eval
>    - __traceback_info__: rid
>   Module <string>, line 0, in ?
>   Module Products.ZCatalog.ZCatalog, line 572, in getIndexDataForRID
>   Module Products.ZCatalog.Catalog, line 441, in getIndexDataForRID
>   Module Products.ZCTextIndex.ZCTextIndex, line 193, in getEntryForObject
>   Module Products.ZCTextIndex.Lexicon, line 92, in get_word
> KeyError: 1965
>
> The  zclass instances in the BTreeFolder2 (WebSitesData) are ok (ie. you
can
> examine them via the ZMI and all the fields are displayed as expected).
>
> Has anyone encountered this situation???

The problem was not related to the volume of records, but was due to the
fact that prior to running the larger update I deleted the Lexicon (I wanted
a clean Lexicon for later testing) and then (as per the instructions on the
ZCTextIndex page within the Catalog) I cleared the ZCTextIndex, which is
supposed to allow the index to 'see' the new Lexicon.  This did not work
(zope 2.6.1), but when I deleted the entire Catalog and built a new one
(including lexicon and zctextindex) everything worked as expected.