[Zope-dev] Re: Caching ZCatalog results

[Dieter Maurer]

Roché Compaan wrote at 2007-2-25 11:48 +0200:
> ...
>> I see several problems:
>> 
>>   *  the RAMCacheManager does not provide an API to implement
>>      this policy
>> 
>>   *  a cache manager would need a special data structure
>>      to efficiently implement the policy (given a documentId,
>>      find all cached results containing the documentId).
>
>Can you elaborate. Would and IISet be efficient?

You need a mapping "documentId --> cached results"
and (maybe) the inverse map in order to update the first one
when cached results are invalidated for different reasons (e.g. timed out).

>>   *  Apparently, your cache key contains the indexes involved
>>      in producing the result.
>
>This is coincidental. I'm building a cache key from all arguments passed
>in as keyword arguments and on the REQUEST.
>
>>      The problem with this is that these indexes are known
>>      only after the query has been performed:
>> 
>>         The catalog API allows indexes to respond to subqueries,
>>         that do not contain their own name.
>> 
>>         I use this feature to allow a "Managable RangeIndex"
>> 	to transparently replace "effective, expires" queries.
>> 
>> 	But otherwise, the feature is probably not used
>> 	intensively.
>
>If these parameters are on the request or in keywords they will form
>part of the cache key.

The are on the request but do not identify the index directly.

Let's make an example:

  My "ValidityRange" "Managable RangeIndex" can respond to
  queries containing subqueries "effective <= some_time <= expires".

  The request contains "effective" and "expires" but does not
  indicate that in fact the "ValidityRange" index is relevant.

  When I correctly understood your policy, you wanted to
  invalidate any cached results depending on an index which
  is updated. Thus, how do you recognize (from the request alone)
  that you should invalidate all queries including "effective"
  and "expires" when the "ValidityRange" index is updsted?

> ...
>Let's assume we have a specialised cache manager and a cache that copes
>with the subtleties of sub queries, do think that the invaliding the
>cache according to the logic I suggested would work? Can you think of
>cases where it can lead to stale results that one should guard against.

Your original policy seemed quite conservative to me.

You weaken conservatism when you want to handle more special
cases for non invalidation (such as "do not invalidate UID query results,
when new entries are added").



-- 
Dieter