ZCatalog caching with memcached
Hi all The past few weeks I've been optimizing a busy Plone site and so collective.catalogcache was born. It uses memcached as a distributed ZCatalog cache. It is currently in production and seems to be holding just fine. The site went from being unusable to serving quite a bit of data. I'll modify it in future to not be too tied to just memcached, but that's still some way off. BTW, if you do not have memcached available the catalog functions as usual. It's only been tested for Zope 2.9.6 - 2.10.6. The repo is at http://dev.plone.org/collective/browser/collective.catalogcache. The URL may contain the word Plone but it's all Zope :) The readme contains more info and instructions. The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions. Hedley
On 24.10.2008 15:41 Uhr, Hedley Roos wrote:
Hi all
The past few weeks I've been optimizing a busy Plone site and so collective.catalogcache was born.
It uses memcached as a distributed ZCatalog cache. It is currently in production and seems to be holding just fine. The site went from being unusable to serving quite a bit of data.
What is your hit rate? -aj
The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most. If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that. It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly. Run a few tests and let me know please. Hedley
On 25.10.2008 8:48 Uhr, Hedley Roos wrote:
The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most.
If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that.
It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly.
Have you measures the time needs for some "standard" ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. Andreas
Have you measures the time needs for some "standard" ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries.
No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. As for "standard" queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now. Since memcached is distributed only a single Zope client needs to perform that query and the result is available to all other Zope clients. And the cache is "persistent" as long as memcached runs, so you can merrily restart Zope instances and have a warm cache. I didn't even realise this until Roche pointed it out to me. To answer the question: I believe catalogcache will win every time since the return time of a cached query is not dependent on the complexity of the query. We should get a few benchmarks running at query level. I'll have a bit of time next week. Hedley
Hedley Roos wrote:
As for "standard" queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now.
sortable_title is a field index and shouldn't be slower than any other index. This all sounds very cool, by the way. :) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote:
Have you measures the time needs for some "standard" ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries.
No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that.
I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets. I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. The effect on performance is really awesome, now we just need to fine tune the implementation. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
On 26.10.2008 18:43 Uhr, Roché Compaan wrote:
On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote:
I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them.
Path indexes and fulltext indexes have a much more complicated implementation compared to field or keyword indexes. Andreas
On Sun, 2008-10-26 at 18:50 +0100, Andreas Jung wrote:
On 26.10.2008 18:43 Uhr, Roché Compaan wrote:
On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote:
I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them.
Path indexes and fulltext indexes have a much more complicated implementation compared to field or keyword indexes.
I know, and this alone makes a good argument for caching at catalog level. In our case we used membrane, which makes an excessive amount of catalog queries when looking up users so some level of caching was essential. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
On 26.10.2008 19:05 Uhr, Roché Compaan wrote:
On Sun, 2008-10-26 at 18:50 +0100, Andreas Jung wrote:
On 26.10.2008 18:43 Uhr, Roché Compaan wrote:
On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them. Path indexes and fulltext indexes have a much more complicated implementation compared to field or keyword indexes.
I know, and this alone makes a good argument for caching at catalog level. In our case we used membrane, which makes an excessive amount of catalog queries when looking up users so some level of caching was essential.
First caching is good thing :-) But how about the following issue: CMF/Plone inject additional subqueries for expires/effective/typesAndRoles. At least the security related aubqueries make a cached catalog result very specific to a particular user. That seems to be very ok for a site with lots of anonymous users - it might be an issue with lots of authenticated users. It might be necessary to add some kind of intelligence to decide what to cache and what not. I don't think it does not make sense to cache the result of a fulltext search. I am just thinking if it would make sense to cache on the index level instead of catalog level? So you could for example cache "expensive" index queries (path index) and combine them with uncached index which are supposed to be fast..however such decisions require detailed mesurements on real systems. One other thing concerning memcached: there is obviously a limit to 1MB for data you can store as a value. We have not found an obvious way for increasing this limit other by patching the memcached sources. We came up with an implementation where data >1MB is split up into individual junks (we have a dedicated set_huge(), get_huge()) implementation. Andreas -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: info@zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting
Hi Roché, I can see your funkload profile? On Sun, Oct 26, 2008 at 3:43 PM, Roché Compaan <roche@upfrontsystems.co.za>wrote:
On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote:
Have you measures the time needs for some "standard" ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries.
No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that.
I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets.
I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them.
The effect on performance is really awesome, now we just need to fine tune the implementation.
-- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Fábio Rizzo Matos ThreePointsWeb fabiorizzo@threepointsweb.com http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto!
Hi Fabio The funkload tests were project specific. I plan to write up my findings and to do benchmarks on a standard Plone instance and blog about it. This will unfortunately have to wait since I'm on holiday this week :-) -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za On Sun, 2008-10-26 at 15:54 -0200, Fabio Rizzo Matos wrote:
Hi Roché,
I can see your funkload profile?
On Sun, Oct 26, 2008 at 3:43 PM, Roché Compaan <roche@upfrontsystems.co.za> wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: > > Have you measures the time needs for some "standard" ZCatalog queries > > used with a Plone site with the communication overhead with memcached? > > Generally spoken: I think the ZCatalog is in general fast. Queries using a > > fulltext index are known to be more expensive or if you have to deal with > > large resultsets or complex queries. > > > > No I haven't. Roche Compaan has done extensive benchmarking using > funkload testing plain catalog vs module level cache vs memcached, but > the tests are more about page serving than catalog query time. I'll > ask him to comment more on that.
I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets.
I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them.
The effect on performance is really awesome, now we just need to fine tune the implementation.
-- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Fábio Rizzo Matos ThreePointsWeb fabiorizzo@threepointsweb.com http://www.threepointsweb.com +55 61 3202-6480
Python, Zope e Plone com quem entende do assunto! _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Very Nice. Have a nice holiday :-) On Sun, Oct 26, 2008 at 3:58 PM, Roché Compaan <roche@upfrontsystems.co.za>wrote:
Hi Fabio
The funkload tests were project specific. I plan to write up my findings and to do benchmarks on a standard Plone instance and blog about it. This will unfortunately have to wait since I'm on holiday this week :-)
-- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
On Sun, 2008-10-26 at 15:54 -0200, Fabio Rizzo Matos wrote:
Hi Roché,
I can see your funkload profile?
On Sun, Oct 26, 2008 at 3:43 PM, Roché Compaan <roche@upfrontsystems.co.za> wrote: On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote: > > Have you measures the time needs for some "standard" ZCatalog queries > > used with a Plone site with the communication overhead with memcached? > > Generally spoken: I think the ZCatalog is in general fast. Queries using a > > fulltext index are known to be more expensive or if you have to deal with > > large resultsets or complex queries. > > > > No I haven't. Roche Compaan has done extensive benchmarking using > funkload testing plain catalog vs module level cache vs memcached, but > the tests are more about page serving than catalog query time. I'll > ask him to comment more on that.
I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets.
I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them.
The effect on performance is really awesome, now we just need to fine tune the implementation.
-- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Fábio Rizzo Matos ThreePointsWeb fabiorizzo@threepointsweb.com http://www.threepointsweb.com +55 61 3202-6480
Python, Zope e Plone com quem entende do assunto! _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Fábio Rizzo Matos ThreePointsWeb fabiorizzo@threepointsweb.com http://www.threepointsweb.com +55 61 3202-6480 Python, Zope e Plone com quem entende do assunto!
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Roché Compaan wrote:
On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote:
Have you measures the time needs for some "standard" ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries.
No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that.
I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets.
I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them.
The effect on performance is really awesome, now we just need to fine tune the implementation.
Before (or while) we work on caching, can we try to improve the underlying indexes, and the way that applications use them? I'm pretty sure that there is a lot of room for improvement: - Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around "just in case" is a sure lose a write time, and may even be expensive at query time (depending on the query). - Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be >> the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing. - I'm pretty sure that we haven't yet found the best data structure for "hierarchy indexes" (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies. - I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time. - Other special-purpose indexes (e.g., a "recent items" index) would be worth a look, especially for applications with large volumes of content. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJBLHb+gerLs4ltQ4RAp59AJwNlfjI0tBv4PdMiDdH4TLKSm5YfwCgu8xB F3u1G0onXKKZ4s7MbLj9B2w= =r0oE -----END PGP SIGNATURE-----
On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Roché Compaan wrote:
On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote:
Have you measures the time needs for some "standard" ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries.
No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that.
I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets.
I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them.
The effect on performance is really awesome, now we just need to fine tune the implementation.
Before (or while) we work on caching, can we try to improve the underlying indexes, and the way that applications use them? I'm pretty sure that there is a lot of room for improvement:
- Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around "just in case" is a sure lose a write time, and may even be expensive at query time (depending on the query).
- Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be >> the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing.
- I'm pretty sure that we haven't yet found the best data structure for "hierarchy indexes" (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies.
- I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time.
- Other special-purpose indexes (e.g., a "recent items" index) would be worth a look, especially for applications with large volumes of content.
I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it? -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 27, 2008, at 13:08 , Roché Compaan wrote:
On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote:
- Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around "just in case" is a sure lose a write time, and may even be expensive at query time (depending on the query).
- Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be >> the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing.
- I'm pretty sure that we haven't yet found the best data structure for "hierarchy indexes" (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies.
- I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time.
- Other special-purpose indexes (e.g., a "recent items" index) would be worth a look, especially for applications with large volumes of content.
I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it?
It's the typical trade-off. One course is expedient and fast for your use case now. The other requires more resources, but benefits everyone. Including those who don't want to depend on yet another package, like memcached, for performance. When it comes to integrating anything in Zope itself I'd choose the latter. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkkFssEACgkQRAx5nvEhZLITiQCgskifGYaixaj6lVLk85l6rz6E aQwAoI9PRcJHL8oZPatlHWADA0h6orCe =YLhP -----END PGP SIGNATURE-----
On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Oct 27, 2008, at 13:08 , Roché Compaan wrote:
On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote:
- Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around "just in case" is a sure lose a write time, and may even be expensive at query time (depending on the query).
- Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be >> the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing.
- I'm pretty sure that we haven't yet found the best data structure for "hierarchy indexes" (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies.
- I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time.
- Other special-purpose indexes (e.g., a "recent items" index) would be worth a look, especially for applications with large volumes of content.
I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it?
It's the typical trade-off. One course is expedient and fast for your use case now. The other requires more resources, but benefits everyone. Including those who don't want to depend on yet another package, like memcached, for performance.
I'm not tied to memcached. We started out using module level caches like zope.cache.ram but that has obvious problems when using ZEO.
When it comes to integrating anything in Zope itself I'd choose the latter.
Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one). -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 27, 2008, at 13:32 , Roché Compaan wrote:
On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote:
When it comes to integrating anything in Zope itself I'd choose the latter.
Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one).
Right, it's perfectly valid to create such a third party package. The discussion just highlights a greater issue. Personally, I don't think it's good practice to focus on the expediency of working around a problem as opposed to tackling the problem directly. The Zope world is littered with add-ons that act as band-aids on real or perceived shortcomings in Zope itself. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkkFtvQACgkQRAx5nvEhZLLOrwCaA+X3iGaTDmyt3vP4q93OoTfx CNsAoJXppoHwI17ISetv4iAwoJeb+Phd =auan -----END PGP SIGNATURE-----
On Mon, 2008-10-27 at 13:41 +0100, Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Oct 27, 2008, at 13:32 , Roché Compaan wrote:
On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote:
When it comes to integrating anything in Zope itself I'd choose the latter.
Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one).
Right, it's perfectly valid to create such a third party package. The discussion just highlights a greater issue. Personally, I don't think it's good practice to focus on the expediency of working around a problem as opposed to tackling the problem directly. The Zope world is littered with add-ons that act as band-aids on real or perceived shortcomings in Zope itself.
Improving the performance of indexes is really really hard. In this case I really don't think caching is a band-aid, it is a good solution. Even with optimised indexes, you will find that you need caching to get reasonable performance if you have a catalog with close to a million or more documents indexed. Given a large enough catalog, I would argue that caching is equally as necessary as having a large cache for a ZEO client. But caches expire and results get invalidated, and therefor we should continue to optimise indexes. With some help we should be able to contribute at this level too. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
On 27 Oct 2008, at 13:10, Roché Compaan wrote:
Improving the performance of indexes is really really hard. In this case I really don't think caching is a band-aid, it is a good solution. Even with optimised indexes, you will find that you need caching to get reasonable performance if you have a catalog with close to a million or more documents indexed. Given a large enough catalog, I would argue that caching is equally as necessary as having a large cache for a ZEO client.
But caches expire and results get invalidated, and therefor we should continue to optimise indexes. With some help we should be able to contribute at this level too.
-- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
Thanks to you and everyone else working on these sometimes thankless performance improvements. --r Russ Ferriday - Topia Systems - Open Source content management with Plone and Zope russf@topia.com - office: +44 2076 1777588 - mobile: +44 7789 338868 - skype: ferriday
Have you looked at whether you are overflowing the zeocache and having to fetch your catalog from disk too often? The timing mentioned in this thread seems about right for that to be the case. I have a client with about a 300k small documents, for whom I have separated the catalog from the rest of Data.fs. This means I can have different cache settings for small and large objects. Resulting in markedly faster catalog queries. --r. On 27 Oct 2008, at 12:32, Roché Compaan wrote:
On Mon, 2008-10-27 at 13:23 +0100, Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Oct 27, 2008, at 13:08 , Roché Compaan wrote:
On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote:
- Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around "just in case" is a sure lose a write time, and may even be expensive at query time (depending on the query).
- Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be >> the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing.
- I'm pretty sure that we haven't yet found the best data structure for "hierarchy indexes" (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies.
- I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time.
- Other special-purpose indexes (e.g., a "recent items" index) would be worth a look, especially for applications with large volumes of content.
I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it?
It's the typical trade-off. One course is expedient and fast for your use case now. The other requires more resources, but benefits everyone. Including those who don't want to depend on yet another package, like memcached, for performance.
I'm not tied to memcached. We started out using module level caches like zope.cache.ram but that has obvious problems when using ZEO.
When it comes to integrating anything in Zope itself I'd choose the latter.
Sure, we're not trying to get this into Zope, we're just sharing our experience and exploring the territory so that one can produce a third party package that really help people with the same use case (which I suspect is quite common one).
-- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Russ Ferriday - Topia Systems - Open Source content management with Plone and Zope russf@topia.com - office: +44 2076 1777588 - mobile: +44 7789 338868 - skype: ferriday
On Mon, 2008-10-27 at 12:42 +0000, Russ Ferriday wrote:
Have you looked at whether you are overflowing the zeocache and having to fetch your catalog from disk too often? The timing mentioned in this thread seems about right for that to be the case.
I have a client with about a 300k small documents, for whom I have separated the catalog from the rest of Data.fs. This means I can have different cache settings for small and large objects. Resulting in markedly faster catalog queries.
We mount all catalogs on separate filestorage instances and each have a cache of 200000 objects. Our queries are CPU bound, not memory or IO bound. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
Roché, On Mon, Oct 27, 2008 at 9:08 AM, Roché Compaan <roche@upfrontsystems.co.za> wrote:
I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it?
Improve catalog and application use of it really hard and memcached is here to scale much better. I know about lovely.memcached and I have some interest is zope and memcache integration, particulary transactional integration. I will sugest a package called zope.memcached (like zope.sqlalchemy does for SQLAlchemy integration). That way any application who need to talk memcached can do it with out loose atomicit. At Weimar, we have doing some memcached integration for a huge portal, all content there is grabed from SOA services, and parse XML is too slow. Memcached is in use for many high load sites to store database information because retrival time dont degrade and can scale so much better. Improve Indexes must be done too, but for large sites, no one beat memcached. ;-)
-- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- ===================================== Rudá Porto Filgueiras Weimar Consultoria http://python-blog.blogspot.com Hospedagem Plone, Django, Zope 3, Grok... http://www.pytown.com =====================================
On 27.10.2008 16:28 Uhr, Rudá Porto Filgueiras wrote:
I will sugest a package called zope.memcached (like zope.sqlalchemy does for SQLAlchemy integration). That way any application who need to talk memcached can do it with out loose atomicit.
I don't see a particular reason for creating a new package for here. Extend lovely.memcached and your done. There is not much need for scattering tiny functionalies into two modules here. The module world is already complicated enough. Andreas
On Mon, Oct 27, 2008 at 12:33 PM, Andreas Jung <lists@zopyx.com> wrote:
On 27.10.2008 16:28 Uhr, Rudá Porto Filgueiras wrote:
I will sugest a package called zope.memcached (like zope.sqlalchemy does for SQLAlchemy integration). That way any application who need to talk memcached can do it with out loose atomicit.
I don't see a particular reason for creating a new package for here. Extend lovely.memcached and your done. There is not much need for scattering tiny functionalies into two modules here. The module world is already complicated enough.
If lovely,memcached alredy is safe when some Exception is raised, discard my sugestion. It's also compatible with zope2?
Andreas
-- ===================================== Rudá Porto Filgueiras Weimar Consultoria http://python-blog.blogspot.com Hospedagem Plone, Django, Zope 3, Grok... http://www.pytown.com =====================================
On 27.10.2008 17:18 Uhr, Rudá Porto Filgueiras wrote:
On Mon, Oct 27, 2008 at 12:33 PM, Andreas Jung<lists@zopyx.com> wrote:
On 27.10.2008 16:28 Uhr, Rudá Porto Filgueiras wrote:
I will sugest a package called zope.memcached (like zope.sqlalchemy does for SQLAlchemy integration). That way any application who need to talk memcached can do it with out loose atomicit. I don't see a particular reason for creating a new package for here. Extend lovely.memcached and your done. There is not much need for scattering tiny functionalies into two modules here. The module world is already complicated enough.
If lovely,memcached alredy is safe when some Exception is raised, discard my sugestion. It's also compatible with zope2?
We are using it together with our cache tool I mentioned earlier with Zope 2.8.1. Andreas
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Roché Compaan wrote:
On Sun, 2008-10-26 at 14:07 -0400, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Roché Compaan wrote:
On Sat, 2008-10-25 at 09:20 +0200, Hedley Roos wrote:
Have you measures the time needs for some "standard" ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries.
No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. I actually did some profiling as well and catalog searches were just too damn slow. The average execution time for searchResults was 100 milliseconds and this is why I told Hedley we should do some caching at query level in the first place. I experimented with this idea a couple of years back but wasn't successful due to inexperience. I was trying to cache brains which obviously leads to persistency bugs. This time around it was obvious to me that we should cache the IISet result sets.
I suspect specific indexes are just performing suboptimally and needs to be improved. ExtendPathIndex in Plone seems to be one of them.
The effect on performance is really awesome, now we just need to fine tune the implementation. Before (or while) we work on caching, can we try to improve the underlying indexes, and the way that applications use them? I'm pretty sure that there is a lot of room for improvement:
- Plone uses too many indexes, and in particular, uses multiple text indexes. Having extra indexes around "just in case" is a sure lose a write time, and may even be expensive at query time (depending on the query).
- Particular indexes have performance characteristics based on their designed purpose: for instance, the stock FieldIndex implementation assumes that the number of documents indexed will be >> the number of discrete indexable values. Using such an index in an application domain with a very large set of indexable values probably loses, and in ways which don't show up in early / small-scale testing.
- I'm pretty sure that we haven't yet found the best data structure for "hierarchy indexes" (e.g., the Plone EPI index, or the stock Zope2 PathIndex, etc.). Something like a 'trie' might be optimal for pure prefix searching of hierarchies.
- I am confident that the TopicIndex is underutiliized: it does *all* the work for a given query at write time, and can thus be blindingly fast at query time.
- Other special-purpose indexes (e.g., a "recent items" index) would be worth a look, especially for applications with large volumes of content.
I agree that one should look at improving performance without caching as well. But this is a lot harder and takes significantly more development and debugging time than introducing some form caching. So I'm not convinced that it needs to happen in a certain order. If caching gives you lots of performance with little effort now, then why shouldn't you use it?
Because it introduces complexities and error cases, perhaps? I'm particularly worried that introducing caching within the framework will make it harder to deal properly with such cases at the application level: the awesomeness / usefulness of DWIM is a step function. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJBeyN+gerLs4ltQ4RAoxvAKCaa0x9Q6wCfolSR98INb813g7mMACdGWx+ LF/M2LQgsE3A5tovfa8ywL8= =PvxD -----END PGP SIGNATURE-----
Hedley Roos wrote:
Since memcached is distributed only a single Zope client needs to perform that query and the result is available to all other Zope clients.
This is where you'll get the big win: no need to load all the catalog-related objects into the zodb cache on all the clients which has the twin drawbacks of needing to be done and trashing your zodb cache...
And the cache is "persistent" as long as memcached runs, so you can merrily restart Zope instances and have a warm cache. I didn't even realise this until Roche pointed it out to me.
Coool :-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Despite protestations to the existence of a ZCatalog caching tool I've finally released it to http://pypi.python.org/pypi/collective.catalogcache :) A disclaimer is on order here: I agree that there are many other places where performance needs to be addressed and that catalogcache masks those areas; however, it works well and should solve performance problems on sites with expensive queries. Andreas Jung, Christian Theune and Roche Compaan gave me a few great tips - thanks for that. Roche has also done benchmarking with it. Andreas asked about the hit rate earlier. Memcached itself operates at a 96% overall hit rate. Individual Plone catalogs (portal_catalog etc) vary, but they usually run at above 90% hit rate with portal_catalog dropping to 80% in cases. This is of course dependent on the usage pattern of the site. So give it a go, or shout at me and tell me that it is a bad idea :) Hedley
Hi, On Fri, 2008-10-24 at 15:41 +0200, Hedley Roos wrote:
The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions.
I'd love if this wouldn't be a monkey patch. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
On 25.10.2008 14:37 Uhr, Christian Theune wrote:
Hi,
On Fri, 2008-10-24 at 15:41 +0200, Hedley Roos wrote:
The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions.
I'd love if this wouldn't be a monkey patch.
Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other.
The problem with memcached is that memcached isn't transactional. We managed to solve this problem by implementing a cache tool (for CMF) where where the set/get methods for the memcached participate in the Zope transaction using a DataManager. I can provide the code if someone should be interested. Andreas
On Sat, 2008-10-25 at 14:43 +0200, Andreas Jung wrote:
On 25.10.2008 14:37 Uhr, Christian Theune wrote:
Hi,
On Fri, 2008-10-24 at 15:41 +0200, Hedley Roos wrote:
The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions.
I'd love if this wouldn't be a monkey patch.
Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other.
The problem with memcached is that memcached isn't transactional. We managed to solve this problem by implementing a cache tool (for CMF) where where the set/get methods for the memcached participate in the Zope transaction using a DataManager. I can provide the code if someone should be interested.
In addition, you need to include a serial in your cache keys to avoid dirty reads. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Sat, 2008-10-25 at 14:55 +0200, Hedley Roos wrote:
In addition, you need to include a serial in your cache keys to avoid dirty reads.
The cache invalidation code actively removes items from the cache. Am I understanding you correctly?
I wasn't even talking about invalidation as your cache wouldn't see 'invalidations' anyways. It's memcached's task to forget stuff: it's a cache anyway. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
Previously Andreas Jung wrote:
On 25.10.2008 14:37 Uhr, Christian Theune wrote:
Hi,
On Fri, 2008-10-24 at 15:41 +0200, Hedley Roos wrote:
The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions.
I'd love if this wouldn't be a monkey patch.
Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other.
The problem with memcached is that memcached isn't transactional. We managed to solve this problem by implementing a cache tool (for CMF) where where the set/get methods for the memcached participate in the Zope transaction using a DataManager. I can provide the code if someone should be interested.
If I remember correctly memcached is transactional internally but the python bindings don't expose that. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
I'd love if this wouldn't be a monkey patch.
So would I, but I couldn't find another way in this case.
Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other.
I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Method search just inspects queries and stores results to memcache. Can you give me an example where the cache would deliver non-committed data?
On 25.10.2008 14:53 Uhr, Hedley Roos wrote:
I'd love if this wouldn't be a monkey patch.
So would I, but I couldn't find another way in this case.
Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other.
I patched 4 methods - clear, search, catalogObject, uncatalogObject.
Method clear is the simplest one - I simply flush the cache.
Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches.
Using a DataManager is likely the better and more safe choice. Andreas
On Sat, Oct 25, 2008 at 2:57 PM, Andreas Jung <lists@zopyx.com> wrote:
On 25.10.2008 14:53 Uhr, Hedley Roos wrote:
I'd love if this wouldn't be a monkey patch.
So would I, but I couldn't find another way in this case.
Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other.
I patched 4 methods - clear, search, catalogObject, uncatalogObject.
Method clear is the simplest one - I simply flush the cache.
Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches.
Using a DataManager is likely the better and more safe choice.
Andreas
Thanks Andreas. I'll have a look at your code when available. Christian, I do have a mistake in my reasoning. If an object is added to the catalog in a transaction and I cache that object as result of a query in that same transaction, and then the transaction fails I'll have a bad cache. H
On Sat, 2008-10-25 at 14:53 +0200, Hedley Roos wrote:
I'd love if this wouldn't be a monkey patch.
So would I, but I couldn't find another way in this case.
Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other.
I patched 4 methods - clear, search, catalogObject, uncatalogObject.
Method clear is the simplest one - I simply flush the cache.
This is probably harmless but will cause unnecessary cache flushes for other clients.
Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches.
Right. This is the same as clear.
Method search just inspects queries and stores results to memcache.
That's the issue. If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed. Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed.
Kind of like how I came to the same conclusion in parallel to you and stuffed up this thread :)
Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions.
So this is the area I have to focus on right now. H
Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions.
A possible solution is to keep a module level cache which can be committed to the memcache on transaction boundaries. That way I'll incur no performance penalty. H
On Sat, 2008-10-25 at 15:14 +0200, Hedley Roos wrote:
Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions.
A possible solution is to keep a module level cache which can be committed to the memcache on transaction boundaries. That way I'll incur no performance penalty.
Module-level should be 'transaction level'. You need to implement ZODB's snapshot-ability to provide similar MVCC semantics. The transaction boundary is gonna give you another hard time: two phase commit. Memcache doesn't support transactions at all, so the only way to come up with a minimized risk is to implement a "one phase" data manager which does its work during `vote` and is run as the last data manager in the vote phase. Note that this isn't a good idea in general but works reliable if your backend supports a "one phase commit" (which memcache doesn't seem to do). I'd also avoid putting a generally reusable "tool" package in the place that it uses the "one phase" data manager hack. Generally applications mixing ZODB with other database like MySQL are going to need that "slot". Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
participants (13)
-
Andreas Jung -
Chris Withers -
Christian Theune -
Fabio Rizzo Matos -
Hedley Roos -
Hedley Roos -
Jens Vagelpohl -
Martin Aspeli -
Roché Compaan -
Rudá Porto Filgueiras -
Russ Ferriday -
Tres Seaver -
Wichert Akkerman