[Zope] ZCTextIndex Lexicon problems

[Jonathan Hobbs]

I have bulk uploaded about 100k records (zclass instances) into a
BTreeFolder2 and am using the following code to create entries in a zcatalog
(I am not using catalog-aware instances):

import os, sys, traceback
import Zope

app=Zope.app()
from Testing import makerequest
app = makerequest.makerequest(app)

for item in app.sw.WebSitesFolder.WebSitesData.objectItems():
        uid = item[0]
        obj = item[1]
        app.sw.WebSitesFolder.Catalog.catalog_object(obj, uid)
        if cnt == 1000:
                get_transaction().commit()
                cnt=0

get_transaction().commit()


This works if I only have about 10k entries in the BTreeFolder2
(WebSitesData), but when I have larger volumes of records the Lexicon
doesn't work:  ie. 0 ( yes, 'zero') entries in the Lexicon, but no errors.
The zcatalog contains the correct number of entries, but when you try to
access a record in the zcatalog (via the ZMI) you get the following error:

Traceback (innermost last):
  Module ZPublisher.Publish, line 98, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 39, in call_object
  Module Shared.DC.Scripts.Bindings, line 252, in __call__
  Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec
  Module App.special_dtml, line 174, in _exec
  Module DocumentTemplate.DT_In, line 628, in renderwob
  Module DocumentTemplate.DT_Util, line 201, in eval
   - __traceback_info__: rid
  Module <string>, line 0, in ?
  Module Products.ZCatalog.ZCatalog, line 572, in getIndexDataForRID
  Module Products.ZCatalog.Catalog, line 441, in getIndexDataForRID
  Module Products.ZCTextIndex.ZCTextIndex, line 193, in getEntryForObject
  Module Products.ZCTextIndex.Lexicon, line 92, in get_word
KeyError: 1965

The  zclass instances in the BTreeFolder2 (WebSitesData) are ok (ie. you can
examine them via the ZMI and all the fields are displayed as expected).

Has anyone encountered this situation???

Thanks in advance,

Jonathan