* Jim Kutter <jim@ebizq.net> [2004-07-22 00:01]:
Hi
I'm trying to use ZCatalogs excellent indexing capability as a front end for data mining. We have 80k records which I'm reading using a ZSQL method (which also uses a custom base class for the records), for each record I then have to pull in data from three other tables, then I add that virtual object to the catalog. All of this is done via a python script.
Here's the kicker:
Zope takes forever to do this, and it tends to freeze the zope instance while it's doing it, in fact I have yet to successfully do this for a batch size larger than 100 records.
I am sure that the catalog is not the culprit here - I regularly index 300k objects in one go. Your instance freezing could be due to memory consumption, committing a transaction every n records will help keep that in check.
My suspicion is that zope is doing this all in a transaction, and not committing the change per object, but instead waiting for the whole 80k records to be worked on, then commit the whole bag. Needless to say this isn't optimal. Is there a way I can force the transaction to commit after each record is done?
get_transation().commit() but it won't work in a Python Script, use it in an external method. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za