It sounds like this system has a high write rate. These writes are causing the catalog to get updated often. This could explain both the poor performance and the errors. When a write occurs while simultaneously performing a query, it can cause a read conflict in the query because one of the objects in the index changed while the query was being processed. This causes the query to be retried. If another conflict is detected, the query will be retried again (up to 3 times). These retries slow things down a lot and if conflicts continue to happen, they will eventually cause the error you reported. You say you have 8000 pdfs in a single folder. What type of folder is this? How often are new files added, or old files deleted? If you are using a standard zope folder (not a BTreeFolder), then updates to it will be *very* slow with that many children. Slow transactions like that tend to cause conflicts elsewhere (or get retried themselves alot) which exacerbates the problem. The query you mention (against two field indexes) should be pretty cheap, however it's unclear how many times it gets executed in the loop. It would be cheaper to do this instead of querying in the loop. query = context.getDBSubjectForUser(userID=userID) subjects = [q.subject for q in query] docResults = context.portal_catalog( {'Subject' : subjects, 'Type' : 'Template'}) Also, what are you doing with docResults when you get it back? Are you calling getObject on the results returned? -Casey On Thu, 18 Mar 2004 13:15:45 -0500 Kevin Carlson <khcarlso@bellsouth.net> wrote:
I have a single folder in a zope instance that contains about 8000 pdf
files. Each of those files is indexed using TextIndexNG (1.09), along with having a Subject and Type index that are both Field indexes. I am using portal_catalog under CMF 1.2 , Zope 2.6.1, python 2.1.3.
Querying the catalog has begun to take much more time recently and I am wondering if there are any things that I can check into regarding what might be slowing things down. I have a simple query returned from a MySQL database that in turn drives a search of the catalog on Subject and Type. I am using the following syntax:
... query = context.getDBSubjectForUser(userID=userID) for q in query : docResults = context.portal_catalog({'Subject' : q.subject, 'Type' :
'Template'}) ...
This query is taking quite a while and occasionally errs out with a ZODB conflict error for certain subjects returned from the database(traceback below) :
----------------------------------------------
Site Error
An error was encountered while publishing this resource.
*ZODB.POSException.ConflictError*
Sorry, a site error occurred.
Traceback (innermost last):
* Module ZPublisher.Publish, line 150, in publish_module * Module ZPublisher.Publish, line 127, in publish * Module ZPublisher.Publish, line 127, in publish * Module ZPublisher.Publish, line 127, in publish * Module ZPublisher.Publish, line 122, in publish * Module Zope.App.startup, line 142, in zpublisher_exception_hook * Module ZPublisher.Publish, line 102, in publish * Module Zope.App.startup, line 200, in commit * Module ZODB.Transaction, line 235, in commit * Module ZODB.Transaction, line 349, in _commit_objects * Module ZODB.Connection, line 391, in commit __traceback_info__: (('Products.Transience.Transience', 'Increaser'), '\x00\x00\x00\x00\x00\x00\x00\x06', '') * Module Products.TemporaryFolder.TemporaryStorage, line 134, in store
ConflictError: database conflict error (oid 0000000000000006, serial was 0353d48778e55b88, now 0353d485d3dba833)
----------------------------------------------
Any ideas what could be causing this type of behavior?
Thanks,
Kevin
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )