[Zope] Re: Catalog query time

[Chris McDonough]

The traceback indicates that the conflict is coming out of a Transience,
which would lead me to think that the catalog isn't at fault.  Sounds
like more of a sessioning write issue.

On Thu, 2004-03-18 at 14:03, Casey Duncan wrote:
> It sounds like this system has a high write rate. These writes are
> causing the catalog to get updated often. This could explain both the
> poor performance and the errors. When a write occurs while
> simultaneously performing a query, it can cause a read conflict in the
> query because one of the objects in the index changed while the query
> was being processed.
> 
> This causes the query to be retried. If another conflict is detected,
> the query will be retried again (up to 3 times). These retries slow
> things down a lot and if conflicts continue to happen, they will
> eventually cause the error you reported.
> 
> You say you have 8000 pdfs in a single folder. What type of folder is
> this? How often are new files added, or old files deleted? If you are
> using a standard zope folder (not a BTreeFolder), then updates to it
> will be *very* slow with that many children. Slow transactions like that
> tend to cause conflicts elsewhere (or get retried themselves alot) which
> exacerbates the problem.
> 
> The query you mention (against two field indexes) should be pretty
> cheap, however it's unclear how many times it gets executed in the loop.
> It would be cheaper to do this instead of querying in the loop.
> 
> query = context.getDBSubjectForUser(userID=userID)
> subjects = [q.subject for q in query]
> docResults = context.portal_catalog(
>     {'Subject' : subjects, 'Type' : 'Template'})
> 
> Also, what are you doing with docResults when you get it back? Are you
> calling getObject on the results returned?
> 
> -Casey
> 
> On Thu, 18 Mar 2004 13:15:45 -0500
> Kevin Carlson <khcarlso at bellsouth.net> wrote:
> 
> > I have a single folder in a zope instance that contains about 8000 pdf
> > 
> > files.  Each of those files is indexed using TextIndexNG (1.09), 
> > along with having a Subject and Type index that are both Field
> > indexes.  I am using portal_catalog under CMF 1.2 , Zope 2.6.1, python
> > 2.1.3.
> > 
> > Querying the catalog has begun to take much more time recently and I
> > am wondering if there are any things that I can check into regarding
> > what might be slowing things down.  I have a simple query returned
> > from a MySQL database that in turn drives a search of the catalog on
> > Subject and Type.  I am using the following syntax:
> > 
> > ...
> > query = context.getDBSubjectForUser(userID=userID)
> > for q in query :
> >   docResults = context.portal_catalog({'Subject' : q.subject, 'Type' :
> >   
> > 'Template'})
> > ...
> > 
> > This query is taking quite a while and occasionally errs out with a
> > ZODB conflict error for certain subjects returned from the
> > database(traceback below) :
> > 
> > ----------------------------------------------
> > 
> > 
> >     Site Error
> > 
> > An error was encountered while publishing this resource.
> > 
> > *ZODB.POSException.ConflictError*
> > 
> > Sorry, a site error occurred.
> > 
> > Traceback (innermost last):
> > 
> >     * Module ZPublisher.Publish, line 150, in publish_module
> >     * Module ZPublisher.Publish, line 127, in publish
> >     * Module ZPublisher.Publish, line 127, in publish
> >     * Module ZPublisher.Publish, line 127, in publish
> >     * Module ZPublisher.Publish, line 122, in publish
> >     * Module Zope.App.startup, line 142, in zpublisher_exception_hook
> >     * Module ZPublisher.Publish, line 102, in publish
> >     * Module Zope.App.startup, line 200, in commit
> >     * Module ZODB.Transaction, line 235, in commit
> >     * Module ZODB.Transaction, line 349, in _commit_objects
> >     * Module ZODB.Connection, line 391, in commit
> >       __traceback_info__: (('Products.Transience.Transience',
> >       'Increaser'), '\x00\x00\x00\x00\x00\x00\x00\x06', '')
> >     * Module Products.TemporaryFolder.TemporaryStorage, line 134, in
> >     store
> > 
> > ConflictError: database conflict error (oid 0000000000000006, serial
> > was 0353d48778e55b88, now 0353d485d3dba833)
> > 
> > ----------------------------------------------
> > 
> > Any ideas what could be causing this type of behavior?
> > 
> > Thanks,
> > 
> > Kevin
> > 
> > 
> > _______________________________________________
> > Zope maillist  -  Zope at zope.org
> > http://mail.zope.org/mailman/listinfo/zope
> > **   No cross posts or HTML encoding!  **
> > (Related lists - 
> >  http://mail.zope.org/mailman/listinfo/zope-announce
> >  http://mail.zope.org/mailman/listinfo/zope-dev )
> > 
> 
> 
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )