AW: [Summary] Caching prob with AHCM and headers
Thanks Bjorn, now I remember the last-modified issue. As I remember it, someone said that the line is commented out for a reason not known... that's why I didn't change it :) /Carsten
-----Ursprüngliche Nachricht----- Von: Bjorn Stabell [mailto:bjorn@exoweb.net] Gesendet: Freitag, 12. September 2003 12:14 An: zope-dev@zope.org Cc: Janko Hauser; pixel@bitmechanic.com; Jens@zope.com; Germer, Carsten Betreff: [Summary] Caching prob with AHCM and headers
Just for the record to help others in the same situation...
PROBLEM Apache (1.3.28 and 2.0.47) will not cache content with just an Expires header, but it will correctly cache content with a Last-Modified header. This problem has been reported before in 1999 by James Cooper, but the bug request seems to have been forgotten/ left behind.
It makes sense for dynamic content to produce Expires headers, not Last-Modified headers, so this is a problem.
Looking at the code for Zope, it seems that the standard HTTP accelerator product used to produce a Last-Modified header, but no longer does so, thus making it useless with the current versions of Apache.
The problem has been re-reported to the Apache group with patches: http://nagoya.apache.org/bugzilla/show_bug.cgi?id=23129 (for Apache 1.3.28) http://nagoya.apache.org/bugzilla/show_bug.cgi?id=23130 (for Apache 2.0.47)
WORK-AROUNDS 1) Produce a Last-Modified header yourself by using 2) Patch Zope's standard HTTP accelerator product so that it produces a Last-Modified header (the line is commented out now)
SOLUTIONS 3) Patch Apache (see bug report links above for patches) 4) Wait until the problem is hopefully fixed in Apache
Bye, -- Bjorn Stabell bjorn@exoweb.net Tel +86 (10) 65918490
Germer, Carsten wrote:
Thanks Bjorn, now I remember the last-modified issue. As I remember it, someone said that the line is commented out for a reason not known... that's why I didn't change it :)
It has always been commented out in CVS. That may have been a mistake--it may have been a temporary debugging edit that I accidentally slipped in. I suggest we uncomment it and see what happens! ;-) Ideally, we should allow the user to choose policies for particular headers in the management interface. Shane
On Fri, 2003-09-12 at 09:51, Shane Hathaway wrote:
Germer, Carsten wrote:
Thanks Bjorn, now I remember the last-modified issue. As I remember it, someone said that the line is commented out for a reason not known... that's why I didn't change it :)
It has always been commented out in CVS. That may have been a mistake--it may have been a temporary debugging edit that I accidentally slipped in. I suggest we uncomment it and see what happens! ;-) Ideally, we should allow the user to choose policies for particular headers in the management interface.
The CMF's CachingPolicyManger, which is specialized for caching skin methods, already has such customizations: it allows spelling the 'Last-Modified' header using a TALES expression, and also has checkboxes for the 'no-cache', 'no-store', and 'must-revalidate' options of the 'Cache-Control' header. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
On Fri, Sep 12, 2003 at 10:13:44AM -0400, Tres Seaver wrote:
The CMF's CachingPolicyManger, which is specialized for caching skin methods, already has such customizations: it allows spelling the 'Last-Modified' header using a TALES expression, and also has checkboxes for the 'no-cache', 'no-store', and 'must-revalidate' options of the 'Cache-Control' header.
Based on that note I decided to try CachingPolicyManager for the first time, and I'm confused about how it's supposed to work. In a CMF/Plone portal instance (CMF 1.4) I created a CMF Caching Policy Manager object. If I create a policy instance with default Predicate and Mod. Time values and a Max age of 999, FSImage objects are then published with appropriate cache control headers so that they expire after 999 seconds. Cool. But I can't see how to create more restrictive Predicate. My initial attempt was to set Predicate like so: python: content.meta_type == 'Filesystem Image' But that predicate never seems to return true. Looking at _setCacheHeaders() in CMFCore/utils.py, 'content' is set to 'aq_parent(obj)'. Because of that, any tests against 'content' in the Predicate apply to the Filesystem Directory View that contains the particular FSImage objects, rather than to the FSImage object itself. What good is that? -- Fred Yankowski fred@ontosys.com tel: +1.630.879.1312 OntoSys, Inc PGP keyID: 7B449345 fax: +1.630.879.1370 www.ontosys.com 38W242 Deerpath Rd, Batavia, IL 60510-9461, USA
Fred Yankowski wrote:
But I can't see how to create more restrictive Predicate. My initial attempt was to set Predicate like so:
python: content.meta_type == 'Filesystem Image'
Do you have any content with this metatype? Try this script: for brain in context.portal_catalog(): obj = brain.getObject() print "metatype:",obj.meta_type print "portal_type:",obj.portal_type return printed cheers, Chris
On Mon, Sep 15, 2003 at 01:14:51PM +0100, Chris Withers wrote:
Fred Yankowski wrote:
But I can't see how to create more restrictive Predicate. My initial attempt was to set Predicate like so:
python: content.meta_type == 'Filesystem Image'
Do you have any content with this metatype?
There is nothing in the portal_catalog with that meta_type. But there are lots of objects with that type: most everything in portal_skins/plone_images/, for example. -- Fred Yankowski fred@ontosys.com tel: +1.630.879.1312 OntoSys, Inc PGP keyID: 7B449345 fax: +1.630.879.1370 www.ontosys.com 38W242 Deerpath Rd, Batavia, IL 60510-9461, USA
Fred Yankowski wrote:
python: content.meta_type == 'Filesystem Image'
Do you have any content with this metatype?
There is nothing in the portal_catalog with that meta_type. But there are lots of objects with that type: most everything in portal_skins/plone_images/, for example.
Oh, I getcha now... hmmm.. since these are served from disk anyway, what would be the benefit in caching them? (Dim memories suggest that OFS.Image.Image does some work to circumvent caching, that's probabyl why they'er not being cached...) cheers, Chris
On Wed, 2003-09-17 at 07:26, Chris Withers wrote:
Fred Yankowski wrote:
python: content.meta_type == 'Filesystem Image'
Do you have any content with this metatype?
There is nothing in the portal_catalog with that meta_type. But there are lots of objects with that type: most everything in portal_skins/plone_images/, for example.
Oh, I getcha now... hmmm.. since these are served from disk anyway, what would be the benefit in caching them?
(Dim memories suggest that OFS.Image.Image does some work to circumvent caching, that's probabyl why they'er not being cached...)
Two reasons: - Browser caches know about the HTTP headers, too. - Accelerators (Squid, Apache w/ mod_proxy and caching), cache content to reduce load on the appserver. Both kinds of caching are Good Things (TM) for sites where Zope is or can be the bottleneck, and where absolute freshenss is not a requirement. Even serving 304s is *much* slower in Zope than serving from the Squid cache, for instance. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
On Wed, Sep 17, 2003 at 10:03:15AM -0400, Tres Seaver wrote:
On Wed, 2003-09-17 at 07:26, Chris Withers wrote:
Fred Yankowski wrote:
python: content.meta_type == 'Filesystem Image'
Do you have any content with this metatype?
There is nothing in the portal_catalog with that meta_type. But there are lots of objects with that type: most everything in portal_skins/plone_images/, for example.
Oh, I getcha now... hmmm.. since these are served from disk anyway, what would be the benefit in caching them?
(Dim memories suggest that OFS.Image.Image does some work to circumvent caching, that's probabyl why they'er not being cached...)
Two reasons:
- Browser caches know about the HTTP headers, too.
- Accelerators (Squid, Apache w/ mod_proxy and caching), cache content to reduce load on the appserver.
Right. I assumed all of that was obvious when I jumped into this thread. The CMF Caching Policy Manager object does not itself cache any objects (unlike the RAM Cache Manager, which does) -- it merely applies cache control headers to set the freshness of some published objects so that downstream caches can do real caching and take some load off of Zope, and perhaps off the network. My original question concerned a detail of how the CMF Caching Policy Manager interprets the 'Predicate' of a caching policy: it seems to interpret any 'content' object in that predicate as a reference to the *parent* object of the object being considered for cache header management. That made no sense to me, and I pointed to the exact code involved, but I still don't know if that code is the result of Undocumented Wisdom or merely a Goof. Meanwhile, back on the ranch, I'm getting good results with this policy predicate: python: content.meta_type == 'Filesystem Directory View' With that, all objects in a Filesystem Directory View get cache control headers. The aforementioned Wisdom|Goof prevents me (AFAICT) from discriminating different meta_types of the individual objects within the various FSDVs, but so far one policy across all such is just fine. -- Fred Yankowski fred@ontosys.com tel: +1.630.879.1312 OntoSys, Inc PGP keyID: 7B449345 fax: +1.630.879.1370 www.ontosys.com 38W242 Deerpath Rd, Batavia, IL 60510-9461, USA
On Wed, 2003-09-17 at 17:38, Fred Yankowski wrote:
My original question concerned a detail of how the CMF Caching Policy Manager interprets the 'Predicate' of a caching policy: it seems to interpret any 'content' object in that predicate as a reference to the *parent* object of the object being considered for cache header management. That made no sense to me, and I pointed to the exact code involved, but I still don't know if that code is the result of Undocumented Wisdom or merely a Goof.
The CMF CPM was not initially designed to ease caching of skin images; it was originally intended to allow caching of the page templates which serve as views on the "real content". In fact, in CMF 1.3 only FSPageTemplates know to talk to the CPM at all; CMF 1.4 extended this to the other types, but without a lot of thought about how their needs were different. I have wanted to add a binding to the TALES namespace to allow access to the "skin object" for some time: the current values in the namespace are: 'content' -- the skin object's aq_parent 'view' -- the skin object's name (a really bad choice for the key) 'keywords' -- any keywords passed to the "method" 'request' -- the request object 'member' -- the authenticated member (None for anonymous) 'modules', 'nothing' -- typical TALES 'time' -- request time I think we should be adding in another binding, perhaps 'skinned' for the skin object, which would allow you to filter on its metatype in your predicate.
Meanwhile, back on the ranch, I'm getting good results with this policy predicate:
python: content.meta_type == 'Filesystem Directory View'
With that, all objects in a Filesystem Directory View get cache control headers. The aforementioned Wisdom|Goof prevents me (AFAICT) from discriminating different meta_types of the individual objects within the various FSDVs, but so far one policy across all such is just fine.
I typically want to have skin images, as well as javascript and CSS files, cached for much longer than rendered content views. YMMV. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
On Wed, Sep 17, 2003 at 06:25:47PM -0400, Tres Seaver wrote:
I have wanted to add a binding to the TALES namespace to allow access to the "skin object" for some time: the current values in the namespace are:
'content' -- the skin object's aq_parent
That definition matches the reality, but in CachingPolicyManager.py in a comment it has this: 'content' -- the content object itself which I took at face value for far too long.
I think we should be adding in another binding, perhaps 'skinned' for the skin object, which would allow you to filter on its metatype in your predicate.
I would like that.
I typically want to have skin images, as well as javascript and CSS files, cached for much longer than rendered content views. YMMV.
That makes sense. Is there a way to do that with the CMF 1.4 version of the "CPM"? One could, I suppose, test the names of the FSDV folders in CPM predicates, but that seems a royal PITA. That's why I'm making do with one-size-fits-all CPM policy for now. -- Fred Yankowski fred@ontosys.com tel: +1.630.879.1312 OntoSys, Inc PGP keyID: 7B449345 fax: +1.630.879.1370 www.ontosys.com 38W242 Deerpath Rd, Batavia, IL 60510-9461, USA
On Wed, 2003-09-17 at 19:10, Fred Yankowski wrote:
On Wed, Sep 17, 2003 at 06:25:47PM -0400, Tres Seaver wrote:
I have wanted to add a binding to the TALES namespace to allow access to the "skin object" for some time: the current values in the namespace are:
'content' -- the skin object's aq_parent
That definition matches the reality, but in CachingPolicyManager.py in a comment it has this:
'content' -- the content object itself
which I took at face value for far too long.
Skin objects aren't "content", in the CMF, they are "software", which is the source of the semantic mismatch. Skin images and files are somewhat anomalous in this regard: they are treated as "skin objects" because they need to be packaged and delivered with the other skin objects, but they don't "pretend to be methods on TV" as the other types do.
I think we should be adding in another binding, perhaps 'skinned' for the skin object, which would allow you to filter on its metatype in your predicate.
I would like that.
I typically want to have skin images, as well as javascript and CSS files, cached for much longer than rendered content views. YMMV.
That makes sense. Is there a way to do that with the CMF 1.4 version of the "CPM"? One could, I suppose, test the names of the FSDV folders in CPM predicates, but that seems a royal PITA. That's why I'm making do with one-size-fits-all CPM policy for now.
CMF 1.4 makes it easier to leave extensions on files, as well as images, which means that I have often used a predicate which matches on the suffix of the name, e.g: python: view.endswith( '.png' ) or view.endswith( '.css' ) Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
Fred Yankowski wrote:
That makes sense. Is there a way to do that with the CMF 1.4 version of the "CPM"? One could, I suppose, test the names of the FSDV folders in CPM predicates, but that seems a royal PITA. That's why I'm making do with one-size-fits-all CPM policy for now.
Here's how I handle caching, for the record. I use squid, which doesn't require last-modified headers. I guess that's not an option for you, or you'd probably already be using it ;-) But in any case, I don't see why Shane's suggestion of uncommenting the last-modified line shouldn't work. I use a standard HTTP Cache Manager for all our caching needs - I have yet to find the granuality allowed by CPM necessary. I add 2 or 3 cache managers to the site (e.g. slow_http and fast_http). I associate templates which need a fast turnaround (e.g. the home page) with fast_http and others (e.g. the site map) with slow_http. (BTW, in the CMF you can associate FS Objects with named cache managers by adding a file called foo.dtml.metadata with the following contents:: [default] cache=fast_http ) The only class of request for which I need to have a separate policy is requests from people who are actually editing a site. I have created a customised HTTP Cache Manager which doesn't cache requests for specified URLs. I then instruct site editors to edit the site using, e.g. 'edit.mysite.com' and to view it using 'www.mysite.com'. HTH seb -- [] j a m k i t web solutions for charities seb bacon T: 020 7549 0520 F: 020 7490 1152 M: 07968 301 336 W: www.jamkit.com
On Thu, Sep 18, 2003 at 11:09:57AM +0100, seb bacon wrote:
I use squid, which doesn't require last-modified headers. I guess that's not an option for you, or you'd probably already be using it ;-) But in any case, I don't see why Shane's suggestion of uncommenting the last-modified line shouldn't work.
I use Squid as an HTTP accelerator in front of ZServer. What do last-modified headers have to do with this discussion? Anyway, I have a strong preference for working with stock code, rather than patching Zope itself or any of the products.
I use a standard HTTP Cache Manager for all our caching needs - I have yet to find the granuality allowed by CPM necessary.
Aha, I didn't realize that an HTTP Cache Manger can now handle FSDV objects. That does seem to make the CMF CPM unnecessary for my purposes. (I'm using Zope 2.6.1, CMF 1.4, and Plone 1.1alpha2).
(BTW, in the CMF you can associate FS Objects with named cache managers by adding a file called foo.dtml.metadata with the following contents::
I'm able (I realize just now) to associate FS objects with an HTTP Cache Manager directly from the cache managers Associate tab in the ZMI, just like other cache-managable types.
The only class of request for which I need to have a separate policy is requests from people who are actually editing a site. I have created a customised HTTP Cache Manager which doesn't cache requests for specified URLs. I then instruct site editors to edit the site using, e.g. 'edit.mysite.com' and to view it using 'www.mysite.com'.
Interesting approach. An HTTP Cache Manager has properties allowing it to be associated with an external "downstream" cache such as Squid, so that when a cached object is updated in Zope the cache manager notifies the external cache to purge the corresponding URL. But my few experiments with that (some months back, on an earlier version of Zope) showed that Zope would send a NOTIFY message with a different URL than the one that squid had associated with particular object, apparently because of URL path transformations caused by virtual hosting and/or skin lookup; so the old version of the updated object would remain in the squid cache. I wonder if that works better now. I'll try a few tests... P.S. Even if the NOTIFY interaction above worked right, that still would not address the problem where a page template does not change, but the generated content on that page does. I can see where a site author/editor might want to get a fresher view of such pages than is necessary for a random visitor. -- Fred Yankowski fred@ontosys.com tel: +1.630.879.1312 OntoSys, Inc PGP keyID: 7B449345 fax: +1.630.879.1370 www.ontosys.com 38W242 Deerpath Rd, Batavia, IL 60510-9461, USA
Fred Yankowski wrote:
I use Squid as an HTTP accelerator in front of ZServer. What do last-modified headers have to do with this discussion?
I thought that was how this thread started. I must have mis-read something.
I use a standard HTTP Cache Manager for all our caching needs - I have yet to find the granuality allowed by CPM necessary.
Aha, I didn't realize that an HTTP Cache Manger can now handle FSDV objects. That does seem to make the CMF CPM unnecessary for my purposes. (I'm using Zope 2.6.1, CMF 1.4, and Plone 1.1alpha2).
FYI, it's nothing to do with the Cache Manager code, it's the FSObjects which have been changed to support *it*. seb
Tres Seaver wrote:
I have wanted to add a binding to the TALES namespace to allow access to the "skin object" for some time: the current values in the namespace are:
'content' -- the skin object's aq_parent
'view' -- the skin object's name (a really bad choice for the key)
'keywords' -- any keywords passed to the "method"
'request' -- the request object
'member' -- the authenticated member (None for anonymous)
'time' -- request time
Are these specific to the caching policy manager? Where and how are they implemented? cheers, Chris
On Thu, 2003-09-25 at 06:41, Chris Withers wrote:
Tres Seaver wrote:
I have wanted to add a binding to the TALES namespace to allow access to the "skin object" for some time: the current values in the namespace are:
'content' -- the skin object's aq_parent
'view' -- the skin object's name (a really bad choice for the key)
'keywords' -- any keywords passed to the "method"
'request' -- the request object
'member' -- the authenticated member (None for anonymous)
'time' -- request time
Are these specific to the caching policy manager?
Yes. Any object which uses TALES needs to set up the bindings for the top-level names which are available to TALES expressions, using the 'getContext' method of the TALES engine.
Where and how are they implemented?
Products.CMFCore.CachingPolicyManager.createCPContext. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
This thread and an application for a client has meant i've started to formulate a replacement for Caching Policy Manager. From the way I see it all the functionality in CPM (Vary etc) should be rolled into AHCM. CPM should then just be very simple set of policies that using a predicate point to a cache. Which could be an AHCM, or a Ram Cache or another cache. Admittedly Ram Cache is not something you'd normally want to do for CMF content, but there other caching options. Then we'll have one much more functional AHCM rather than two seperate cache managers trying to almost the same thing... and a much more flexible and pluggable cacheing strategy. -- Andy McKay http://www.agmweb.ca
Andy McKay wrote:
This thread and an application for a client has meant i've started to formulate a replacement for Caching Policy Manager. From the way I see it all the functionality in CPM (Vary etc) should be rolled into AHCM.
+1 What about the "don't cache the following hostnames" knob for the use case I outlined previously, which enables you to circumvent caching (useful for site editors, for example)? I think adding an extra virtual host is a very natural way to ensure fresh content as it is guaranteed to have a unique key in upstream caches. seb -- [] j a m k i t web solutions for charities seb bacon T: 020 7549 0520 F: 020 7490 1152 M: 07968 301 336 W: www.jamkit.com
On Thu, 2003-09-25 at 14:06, Andy McKay wrote:
This thread and an application for a client has meant i've started to formulate a replacement for Caching Policy Manager. From the way I see it all the functionality in CPM (Vary etc) should be rolled into AHCM.
CPM should then just be very simple set of policies that using a predicate point to a cache. Which could be an AHCM, or a Ram Cache or another cache. Admittedly Ram Cache is not something you'd normally want to do for CMF content, but there other caching options.
RAM caching is useful for "subtemplates." I am not convinced that we should think of "resource level" caching (AHCM or Andy's "bake-to-disk on the fly" solution) in the same framework as the tools like RCM, which are much better suited for caching intermediate results, but don't scale well for entire pages. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
RAM caching is useful for "subtemplates." I am not convinced that we should think of "resource level" caching (AHCM or Andy's "bake-to-disk on the fly" solution) in the same framework as the tools like RCM, which are much better suited for caching intermediate results, but don't scale well for entire pages.
RCM is an odd caching system that suits a particular need. Is that a no, this shouldn't be done or just a comment? -- Andy McKay http://www.agmweb.ca
participants (7)
-
Andy McKay -
Chris Withers -
Fred Yankowski -
Germer, Carsten -
seb bacon -
Shane Hathaway -
Tres Seaver