----- Original Message ----- From: "Jonathan Hobbs" <toolkit@magma.ca>
I have bulk uploaded about 100k records (zclass instances) into a BTreeFolder2 and am using the following code to create entries in a zcatalog (I am not using catalog-aware instances):
import os, sys, traceback import Zope
app=Zope.app() from Testing import makerequest app = makerequest.makerequest(app)
for item in app.sw.WebSitesFolder.WebSitesData.objectItems(): uid = item[0] obj = item[1] app.sw.WebSitesFolder.Catalog.catalog_object(obj, uid) if cnt == 1000: get_transaction().commit() cnt=0
get_transaction().commit()
This works if I only have about 10k entries in the BTreeFolder2 (WebSitesData), but when I have larger volumes of records the Lexicon doesn't work: ie. 0 ( yes, 'zero') entries in the Lexicon, but no errors. The zcatalog contains the correct number of entries, but when you try to access a record in the zcatalog (via the ZMI) you get the following error:
Traceback (innermost last): Module ZPublisher.Publish, line 98, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 39, in call_object Module Shared.DC.Scripts.Bindings, line 252, in __call__ Module Shared.DC.Scripts.Bindings, line 283, in _bindAndExec Module App.special_dtml, line 174, in _exec Module DocumentTemplate.DT_In, line 628, in renderwob Module DocumentTemplate.DT_Util, line 201, in eval - __traceback_info__: rid Module <string>, line 0, in ? Module Products.ZCatalog.ZCatalog, line 572, in getIndexDataForRID Module Products.ZCatalog.Catalog, line 441, in getIndexDataForRID Module Products.ZCTextIndex.ZCTextIndex, line 193, in getEntryForObject Module Products.ZCTextIndex.Lexicon, line 92, in get_word KeyError: 1965
The zclass instances in the BTreeFolder2 (WebSitesData) are ok (ie. you can examine them via the ZMI and all the fields are displayed as expected).
Has anyone encountered this situation???
The problem was not related to the volume of records, but was due to the fact that prior to running the larger update I deleted the Lexicon (I wanted a clean Lexicon for later testing) and then (as per the instructions on the ZCTextIndex page within the Catalog) I cleared the ZCTextIndex, which is supposed to allow the index to 'see' the new Lexicon. This did not work (zope 2.6.1), but when I deleted the entire Catalog and built a new one (including lexicon and zctextindex) everything worked as expected.