[Zope-dev] ZCatalog caching with memcached

[Andreas Jung]

On 26.10.2008 19:05 Uhr, Roché Compaan wrote:
> On Sun, 2008-10-26 at 18:50 +0100, Andreas Jung wrote:
>> On 26.10.2008 18:43 Uhr, Roché Compaan wrote:
>>> On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote:
>>> I suspect specific indexes are just performing suboptimally and needs to
>>> be improved. ExtendPathIndex in Plone seems to be one of them.
>> Path indexes and fulltext indexes have a much more complicated
>> implementation compared to field or keyword indexes.
>
> I know, and this alone makes a good argument for caching at catalog
> level. In our case we used membrane, which makes an excessive amount of
> catalog queries when looking up users so some level of caching was
> essential.
>

First caching is good thing :-)
But how about the following issue: CMF/Plone inject additional 
subqueries for expires/effective/typesAndRoles. At least the security 
related aubqueries make a cached catalog result very specific to a 
particular user. That seems to be very ok for a site with lots of 
anonymous users - it might be an issue with lots of authenticated users.
It might be necessary to add some kind of intelligence to decide what to 
cache and what not. I don't think it does not make sense to cache the 
result of a fulltext search. I am just thinking if it would make sense 
to cache on the index level instead of catalog level? So you could for 
example cache "expensive" index queries (path index) and combine them 
with uncached index which are supposed to be fast..however
such decisions require detailed mesurements on real systems.

One other thing concerning memcached: there is obviously a limit to 1MB
for data you can store as a value. We have not found an obvious way for 
increasing this limit other by patching the memcached sources. We came 
up with an implementation where data >1MB is split up into individual 
junks (we have a dedicated set_huge(), get_huge()) implementation.

Andreas

-- 
ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany
Web: www.zopyx.com - Email: info at zopyx.com - Phone +49 - 7071 - 793376
Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535
Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK
------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting

-------------- next part --------------
A non-text attachment was scrubbed...
Name: lists.vcf
Type: text/x-vcard
Size: 330 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20081026/023943c3/attachment.vcf