Faster ZCatalog Searches (better? Optimized?)
Now that one of my ZUBB boards is over 5,000 messages I am noticing where some of the slowdowns are. In my tests I have found that it is the <dtml-in statement and cycle that is bogging things down. In taking a look at Zope.org search through their ZCatalog they use a different code: ZUBB: <dtml-in "searchResults(in_reply_to=[''])" size=batch_size reverse sort=id start=query_start> Zope.org: <!--#with "_(results = doCatalogQuery(textindex_operator='and',REQUEST=REQUEST))"--> Is one better than the other? Is there something else that could be slowing down these searches? Thanks, BZ
On Fri, 26 Apr 2002 14:26:39 -0400, BZ <bz@bwanazulia.com> wrote:
ZUBB: <dtml-in "searchResults(in_reply_to=[''])" size=batch_size reverse sort=id start=query_start>
You are are getting dtml-in to do the sorting, not the catalog. The difference may be very small, or very large Try this (untested) code as a perfomance test <dtml-in "searchResults(in_reply_to=[''],sort_on='id')" size=batch_size start=query_start> That should be faster, but reverses the sort order. In not sure if putting the 'reverse' dtml-in tag back in will incur a performance penalty. I have recently committed some significant micro-optimisations to the the catalog for sorted queries, which have improved the runtime for some of my queries by a factor of 8. If you want these, they are currently in the CVS trunk and will definitely go into 2.6. Toby Dickenson tdickenson@geminidataloggers.com
Hey Toby, I played around with this way too much last friday and the performance hit, from what I can tell, is having the dtml do the reverse. I looked at code from Zope.org and they do it this way: <dtml-with "_(results = searchResults(in_reply_to=[''], REQUEST=REQUEST))"> If I take out any of the sorting, I can get 10x the speed, but at that point, what is the point? BZ At 10:45 AM +0100 4/29/02, Toby Dickenson wrote:
On Fri, 26 Apr 2002 14:26:39 -0400, BZ <bz@bwanazulia.com> wrote:
ZUBB: <dtml-in "searchResults(in_reply_to=[''])" size=batch_size reverse sort=id start=query_start>
You are are getting dtml-in to do the sorting, not the catalog. The difference may be very small, or very large
Try this (untested) code as a perfomance test
<dtml-in "searchResults(in_reply_to=[''],sort_on='id')" size=batch_size start=query_start>
That should be faster, but reverses the sort order. In not sure if putting the 'reverse' dtml-in tag back in will incur a performance penalty.
I have recently committed some significant micro-optimisations to the the catalog for sorted queries, which have improved the runtime for some of my queries by a factor of 8. If you want these, they are currently in the CVS trunk and will definitely go into 2.6.
Toby Dickenson tdickenson@geminidataloggers.com
Doing *any* sorting at the DTML level from the Catalog will kill you simply because ZCatalog returns a lazy result set. Sorting it in DTML involves touching every element of the lazy result. This is very expensive both in time and memory. You should let the Catalog sort this for you using its indexes. Like so: <dtml-let results=searchResults(REQUEST, in_reply_to=('',), sort_on='id', sort_order='reverse'))> ... </dtml-let> hth, Casey On Monday 29 April 2002 07:09 am, BZ allegedly wrote:
Hey Toby,
I played around with this way too much last friday and the performance hit, from what I can tell, is having the dtml do the reverse. I looked at code from Zope.org and they do it this way:
<dtml-with "_(results = searchResults(in_reply_to=[''], REQUEST=REQUEST))">
If I take out any of the sorting, I can get 10x the speed, but at that point, what is the point?
BZ
At 10:45 AM +0100 4/29/02, Toby Dickenson wrote:
On Fri, 26 Apr 2002 14:26:39 -0400, BZ <bz@bwanazulia.com> wrote:
ZUBB: <dtml-in "searchResults(in_reply_to=[''])" size=batch_size reverse sort=id start=query_start>
You are are getting dtml-in to do the sorting, not the catalog. The difference may be very small, or very large
Try this (untested) code as a perfomance test
<dtml-in "searchResults(in_reply_to=[''],sort_on='id')" size=batch_size start=query_start>
That should be faster, but reverses the sort order. In not sure if putting the 'reverse' dtml-in tag back in will incur a performance penalty.
I have recently committed some significant micro-optimisations to the the catalog for sorted queries, which have improved the runtime for some of my queries by a factor of 8. If you want these, they are currently in the CVS trunk and will definitely go into 2.6.
Toby Dickenson tdickenson@geminidataloggers.com
Toby Dickenson writes:
On Fri, 26 Apr 2002 14:26:39 -0400, BZ <bz@bwanazulia.com> wrote:
ZUBB: <dtml-in "searchResults(in_reply_to=[''])" size=batch_size reverse sort=id start=query_start>
You are are getting dtml-in to do the sorting, not the catalog. The difference may be very small, or very large
Try this (untested) code as a perfomance test
<dtml-in "searchResults(in_reply_to=[''],sort_on='id')" size=batch_size start=query_start>
That should be faster, but reverses the sort order. In not sure if putting the 'reverse' dtml-in tag back in will incur a performance penalty. You can add "sort_order='reverse'" to the "searchResults".
Using the "reverse" attribute of "dtml-in" would probably make it slower because the lazyness of the search result is removed. Dieter
participants (4)
-
BZ -
Casey Duncan -
Dieter Maurer -
Toby Dickenson