[Zope-dev] ZCatalog caching with memcached

[Roché Compaan]

On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote:
> > Have you measures the time needs for some "standard" ZCatalog queries
> > used with a Plone site with the communication overhead with memcached?
> > Generally spoken: I think the ZCatalog is in general fast. Queries using a
> > fulltext index are known to be more expensive or if you have to deal with
> > large resultsets or complex queries.
> >
> 
> No I haven't. Roche Compaan has done extensive benchmarking using
> funkload testing plain catalog vs module level cache vs memcached, but
> the tests are more about page serving than catalog query time. I'll
> ask him to comment more on that.

I actually did some profiling as well and catalog searches were just too
damn slow. The average execution time for searchResults was 100
milliseconds and this is why I told Hedley we should do some caching at
query level in the first place. I experimented with this idea a couple
of years back but wasn't successful due to inexperience. I was trying to
cache brains which obviously leads to persistency bugs. This time around
it was obvious to me that we should cache the IISet result sets.

I suspect specific indexes are just performing suboptimally and needs to
be improved. ExtendPathIndex in Plone seems to be one of them.

The effect on performance is really awesome, now we just need to fine
tune the implementation.

-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za