[Zope] Major problems with slow catalog querie

[Dieter Maurer]

=?iso-8859-1?Q?Lea_Smith?= writes:
 > The catalog search times mentioned below are the time
 > the query is with the catalog called from a python
 > script like...
 > 
 > context.do_nothing_script()
 > results = context.portal_catalog( search_dict )
 > context.do_nothing_script()
 > 
 > The 'do_nothing_script' does just that, but it shows
 > me exactly the time the catalog takes using the
 > CallProfiler Product.
How it does this? Looks quite strange for me.

 > ...
 > Passing the following to the portal_catalog causes the
 > catalog to take around 35-45 seconds on average (These
 > queries return 400 results from the 10,500 items
 > cataloged)...
 > 
 > {'modified': {'range': 'min', 'query':
 > DateTime('1995/01/01')}, 'Type': ['Classified'],
 > 'sort_on': 'modified', 'review_state': 'published',
 > 'sort_order': 'reverse'} 
 > 
 > Taking out the 'modified' key and passing the
 > following reduces the catalog search time to approx
 > 0.5 to 1 second on average...
 > 
 > {'Type': ['Classified'], 'sort_on': 'modified',
 > 'review_state': 'published', 'sort_order': 'reverse'} 
I expect, you do not have documents modified before 1995/01/01?

This would mean, the "modified" subquery will return all documents.
However, it will construct the resulting set piecemeal, with
each value for "modified" adding a single or a few documents only
(the chance that they are modified at the same second it not too large).
You must expect a quadratic runtime behaviour (in the number
of documents) in this case.

 > ...
 > As I wrote this I thought whether cataloging
 > the date as an interger would be any different.
 > Searching the modified date as an interger has reduced
 > the search time from average of 35-45 seconds down to
 > 1.5 - 5 seconds. Still not as fast as I would like,
 > but a whole lot better.
Surprising!

Did I not already suggest, you use Zope's profile support
("Contol_Panel --> Debug information --> Profile Information").
Do it, you will clearly see what causes the large time...


Dieter