[Zope] Re: Catalog query time

[Casey Duncan]

It sounds like this system has a high write rate. These writes are
causing the catalog to get updated often. This could explain both the
poor performance and the errors. When a write occurs while
simultaneously performing a query, it can cause a read conflict in the
query because one of the objects in the index changed while the query
was being processed.

This causes the query to be retried. If another conflict is detected,
the query will be retried again (up to 3 times). These retries slow
things down a lot and if conflicts continue to happen, they will
eventually cause the error you reported.

You say you have 8000 pdfs in a single folder. What type of folder is
this? How often are new files added, or old files deleted? If you are
using a standard zope folder (not a BTreeFolder), then updates to it
will be *very* slow with that many children. Slow transactions like that
tend to cause conflicts elsewhere (or get retried themselves alot) which
exacerbates the problem.

The query you mention (against two field indexes) should be pretty
cheap, however it's unclear how many times it gets executed in the loop.
It would be cheaper to do this instead of querying in the loop.

query = context.getDBSubjectForUser(userID=userID)
subjects = [q.subject for q in query]
docResults = context.portal_catalog(
    {'Subject' : subjects, 'Type' : 'Template'})

Also, what are you doing with docResults when you get it back? Are you
calling getObject on the results returned?

-Casey

On Thu, 18 Mar 2004 13:15:45 -0500
Kevin Carlson <khcarlso at bellsouth.net> wrote:

> I have a single folder in a zope instance that contains about 8000 pdf
> 
> files.  Each of those files is indexed using TextIndexNG (1.09), 
> along with having a Subject and Type index that are both Field
> indexes.  I am using portal_catalog under CMF 1.2 , Zope 2.6.1, python
> 2.1.3.
> 
> Querying the catalog has begun to take much more time recently and I
> am wondering if there are any things that I can check into regarding
> what might be slowing things down.  I have a simple query returned
> from a MySQL database that in turn drives a search of the catalog on
> Subject and Type.  I am using the following syntax:
> 
> ...
> query = context.getDBSubjectForUser(userID=userID)
> for q in query :
>   docResults = context.portal_catalog({'Subject' : q.subject, 'Type' :
>   
> 'Template'})
> ...
> 
> This query is taking quite a while and occasionally errs out with a
> ZODB conflict error for certain subjects returned from the
> database(traceback below) :
> 
> ----------------------------------------------
> 
> 
>     Site Error
> 
> An error was encountered while publishing this resource.
> 
> *ZODB.POSException.ConflictError*
> 
> Sorry, a site error occurred.
> 
> Traceback (innermost last):
> 
>     * Module ZPublisher.Publish, line 150, in publish_module
>     * Module ZPublisher.Publish, line 127, in publish
>     * Module ZPublisher.Publish, line 127, in publish
>     * Module ZPublisher.Publish, line 127, in publish
>     * Module ZPublisher.Publish, line 122, in publish
>     * Module Zope.App.startup, line 142, in zpublisher_exception_hook
>     * Module ZPublisher.Publish, line 102, in publish
>     * Module Zope.App.startup, line 200, in commit
>     * Module ZODB.Transaction, line 235, in commit
>     * Module ZODB.Transaction, line 349, in _commit_objects
>     * Module ZODB.Connection, line 391, in commit
>       __traceback_info__: (('Products.Transience.Transience',
>       'Increaser'), '\x00\x00\x00\x00\x00\x00\x00\x06', '')
>     * Module Products.TemporaryFolder.TemporaryStorage, line 134, in
>     store
> 
> ConflictError: database conflict error (oid 0000000000000006, serial
> was 0353d48778e55b88, now 0353d485d3dba833)
> 
> ----------------------------------------------
> 
> Any ideas what could be causing this type of behavior?
> 
> Thanks,
> 
> Kevin
> 
> 
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )
>