problems with file caching
I'm running into a few problems getting file caching to work correctly with the Local File System product. I am using the version 2.1.0 source distribution. The first problem has to do with the index_html() method of the File class in OFS/image.py. Line 188 says: if self._p_mtime > ms: RESPONSE.setStatus(304) return RESPONSE What this seems to say to me is, "If my modification time is later than the modification time of the cached file then send a 304 (Not Modified) response." If I'm not mistaken, this is exactly the opposite of what the HTTP specification says about the If-Modified-Since header. So, line 188 should say: if self._p_mtime <= ms: ... Secondly, once that modification is made I noticed that the DateTime class seems to be making slight rounding errors when parsing the http date string which is also throwing off the caching. To test this, I wrote a very simple http proxy so I can see what's being sent back and forth. I also added some debugging output to the image.py file. ms=DateTime(ms).timeTime() import time print 'ms:', time.ctime(ms), '(%.8f)' % ms print 'mtime:', time.ctime(self._p_mtime), '(%.8f)' % self._p_mtime print 'not modified:', self._p_mtime <= ms if self._p_mtime <= ms: Here's an example transaction that should send a 304 response. send: 'GET /local/gifs/python.gif HTTP/1.0\015\012' send: 'user-agent: Mozilla/4.6 [en] (WinNT; I)\015\012' send: 'accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*\015\012' send: 'authorization: Basic YWRtaW46bGV0bWVpbg==\015\012' send: 'accept-encoding: gzip\015\012' send: 'if-modified-since: Wed, 15 Sep 1999 21:11:30 GMT; length=649\015\012' send: 'cookie: tree-s="eJyLjlZ3hANPW/XYWAAtiQTP"\015\012' send: 'proxy-connection: Keep-Alive\015\012' send: 'referer: http://localhost:8080/local/gifs\015\012' send: 'host: localhost:8080\015\012' send: 'accept-charset: iso-8859-1,*,utf-8\015\012' send: 'accept-language: en\015\012' send: '\015\012' reply: 'HTTP/1.0 200 OK\015\012' reply: server: Zope/Zope 2.0.0b5 (binary release, python 1.5.2, win32-x86) ZServer/1.1b1 reply: content-length: 649 reply: content-type: image/gif reply: connection: close reply: last-modified: Wed, 15 Sep 1999 21:11:30 GMT reply: date: Thu, 09 Dec 1999 22:11:00 GMT Here's the debug output: ms: Wed Sep 15 14:11:29 1999 (937429889.99999952) mtime: Wed Sep 15 14:11:30 1999 (937429890.00000000) not modified: 0 As you can see, the request header says: if-modified-since: Wed, 15 Sep 1999 21:11:30 GMT; length=649 and the response header says: last-modified: Wed, 15 Sep 1999 21:11:30 GMT but the DateTime class converts this time to: Wed Sep 15 14:11:29 1999 which is off by one second, so it thinks that the file has in fact been modified. The workaround I came up with for this is to round off the time value generated by the DateTime class. ms=DateTime(ms).timeTime() ms = round(ms, 1) If I am correct, and this is actually a problem with the DateTime class, I would like to see it fixed, though. I'm not sure whether my third problem is a problem with Zope or my web browser (Netscape 4.6). What happens is that once the browser sends a request with an 'If-Modified-Since' header and the server responds with 200 (OK) instead of 304 (Not Modified), the browser never sends an 'If-Modified-Since' header again unless I restart the browser or clear the cache. In other words, if it tries to use the cached file and finds that it's out of date then it never tries to use the cached file again. I suspect that this is a problem with Netscape, though, and there's nothing I can really do about it. I am interested in any and all feedback regarding these problems. I would like to know if I have found some valid bugs here and should submit them to the collector. Thanks, --jfarr
Friday, December 10, 1999, 1:15:31 AM, Jonothan Farr wrote: JF> As you can see, the request header says: JF> if-modified-since: Wed, 15 Sep 1999 21:11:30 GMT; length=649 JF> and the response header says: JF> last-modified: Wed, 15 Sep 1999 21:11:30 GMT JF> but the DateTime class converts this time to: JF> Wed Sep 15 14:11:29 1999 JF> which is off by one second, so it thinks that the file has in fact been JF> modified. JF> The workaround I came up with for this is to round off the time value JF> generated by the DateTime class. JF> ms=DateTime(ms).timeTime() JF> ms = round(ms, 1) JF> If I am correct, and this is actually a problem with the DateTime class, I JF> would like to see it fixed, though. What platform are you running on? I saw the same rounding problem on SGI IRIX, but couldn't reproduce it on Linux. As I also saw similar differences in fp tests on PostgreSQL, I blame it on the OS and hardware. -- Best regards, Martijn Pieters mailto:mj@digicool.com
JF> If I am correct, and this is actually a problem with the DateTime class, I JF> would like to see it fixed, though.
What platform are you running on? I saw the same rounding problem on SGI IRIX, but couldn't reproduce it on Linux. As I also saw similar differences in fp tests on PostgreSQL, I blame it on the OS and hardware.
This is WinNT. Sorry, should have included that information. --jfarr
Jonothan Farr wrote:
The first problem has to do with the index_html() method of the File class in OFS/image.py.
Line 188 says:
if self._p_mtime > ms: RESPONSE.setStatus(304) return RESPONSE
What this seems to say to me is, "If my modification time is later than the modification time of the cached file then send a 304 (Not Modified) response." If I'm not mistaken, this is exactly the opposite of what the HTTP specification says about the If-Modified-Since header.
So, line 188 should say:
if self._p_mtime <= ms: ...
Here's an email I sent to Brian Lloyd, who said he'll fix this in 2.2. I hope you can make some sense out of it - the gist of it is that HTTP uses 1 second resolution, while DateTime has a resolution of more than 1 second: Got it! Or at least why <= wouldn't cache objects at all, and > had caching working. DateTime().timeTime() returns a float (resolution of more than 1 second). The If-Modified-Since sends an int (resolution of a second, as required by HTTP protocol). HTTP 1.1 recommends If-Modified-Since send the Last-Modified of the cached object. But the Last-Modified header is different than _p_mtime - _p_mtime is a float, most likely not a whole number, while Last-Modified only has a resolution of seconds - it's a int. So the 304 is never sent if you use 'self._p_mtime <= ms'. The fix is simple: in lib/python/OFS/Image.py, we do this: if int(self._p_mtime) <= ms: RESPONSE.setStatus(304) return RESPONSE instead of: if self._p_mtime > ms: RESPONSE.setStatus(304) return RESPONSE should solve the problem. Storing _p_mtime as a truncated integer would of course be more efficient. The cacheability engine at least likes it, while without the int() says that image won't get cached. With > caching validation also sort of worked, since self._p_mtime is usually bigger than int(self._p_mtime). Since int(self._p_mtime) is what cacheability, and presumabely proxies, send as If-Modified-Since (they take it from Last-Modified of the image, which is int(self_p_mtime). -- Itamar S.T. itamars@ibm.net
Got it! Or at least why <= wouldn't cache objects at all, and > had caching working. DateTime().timeTime() returns a float (resolution of more than 1 second). The If-Modified-Since sends an int (resolution of a second, as required by HTTP protocol). HTTP 1.1 recommends If-Modified-Since send the Last-Modified of the cached object. But the Last-Modified header is different than _p_mtime - _p_mtime is a float, most likely not a whole number, while Last-Modified only has a resolution of seconds - it's a int. So the 304 is never sent if you use 'self._p_mtime <= ms'. The fix is simple:
in lib/python/OFS/Image.py, we do this:
if int(self._p_mtime) <= ms: RESPONSE.setStatus(304) return RESPONSE
instead of:
if self._p_mtime > ms: RESPONSE.setStatus(304) return RESPONSE
As it turns out, this doesn't fix my problems. My self._p_mtime is always an integral number, but ms is often too small by several nanoseconds due to what looks like a floating point error. This solution always seems to work for me. if self._p_mtime <= round(ms, 1): --jfarr
Jonothan Farr wrote:
As it turns out, this doesn't fix my problems. My self._p_mtime is always an integral number, but ms is often too small by several nanoseconds due to what looks like a floating point error. This solution always seems to work for me.
if self._p_mtime <= round(ms, 1):
Windows has a time resolution of only 1 second, AFAIK, which explains the integer, although I'm not sure why it would be too small. Is there any way to make a portable fix for this bug? round(ms, 1) is wrong for Unix. -- Itamar S.T. itamars@ibm.net
participants (3)
-
Itamar Shtull-Trauring -
Jonothan Farr -
Martijn Pieters