Accept-Charset hearders causing 500 internal server error.[correct but not lenient]
Hi, We have problems with the Accept-Charset headers that are sent to the zope server by some mobile phones: fi: 'Accept-Charset':'ISO-8859-1, US-ASCII, UTF-8; Q=0.8' These cause a 500 internal server error. (Error Value: invalid literal for float(): =0.8 ) I checked with the HTTP1.1 RFC26160 section 14.2 and the header seems valid, but in fact isn't: there is a space between the semi-colon and the Q char (which shouldn't be there acoording to the spec). So, in principle, this is not a zope server bug, but an inflexibility. I think the server should be a bit more lenient. a simple .strip() in the right place should do. Could you guys fix this? TIA, Sloot. BTW, for completeness, I included a quick test program is included below (causes a 500 on my zope 2.6.0) : #--------------------- program illustrating the behavior ----------- import httplib, urllib def sendHTTP(): """ """ headers = {'Accept': 'application/vnd.wap.wmlc, application/vnd.wap.wmlscriptc, application/vnd.wap.wbxml, image/vnd.wap.wbmp, image/gif, application/*, text/html, application/xhtml+xml, application/vnd.wap.wml+xml, text/css', 'Accept-Charset':'ISO-8859-1, US-ASCII, UTF-8; Q=0.8'}#, ISO-10646-UCS-2; Q=0.6' conn = httplib.HTTPConnection("127.0.0.1:8080") conn.request("GET", "/mobile/wap/games/",headers=headers) response = conn.getresponse() print response.status, response.reason data = response.read() print data conn.close() sendHTTP()
Romain Slootmaekers wrote: Hi, below the original post: it is a bug in Products/Localizer/Accept.py here's the code to fix it: #--- code fix starting from line 102 --- # Get the quality try: if len(x) == 2: quality = x[1] # Get the quality quality =quality.strip() quality = quality[2:] # Get the number (remove "q=") quality = float(quality) # Change it to float else: quality = 1.0 except: quality=1.0 #--- end of code fix --- I added a strip() and a try/except for other things that can go wrong. This way Nokia Wap surfers can access your site while Localizer is installed. have fun, Sloot. BTW, exceptions that originate here are not caught in the error_log. this is probably unintended as well. ----------------- original post -----------------------------
Hi,
We have problems with the Accept-Charset headers that are sent to the zope server by some mobile phones: fi: 'Accept-Charset':'ISO-8859-1, US-ASCII, UTF-8; Q=0.8'
These cause a 500 internal server error. (Error Value: invalid literal for float(): =0.8 )
I checked with the HTTP1.1 RFC26160 section 14.2 and the header seems valid, but in fact isn't: there is a space between the semi-colon and the Q char (which shouldn't be there acoording to the spec).
So, in principle, this is not a zope server bug, but an inflexibility.
I think the server should be a bit more lenient. a simple .strip() in the right place should do.
Could you guys fix this?
TIA,
Sloot.
BTW, for completeness, I included a quick test program is included below (causes a 500 on my zope 2.6.0) :
#--------------------- program illustrating the behavior ----------- import httplib, urllib
def sendHTTP(): """
""" headers = {'Accept': 'application/vnd.wap.wmlc, application/vnd.wap.wmlscriptc, application/vnd.wap.wbxml, image/vnd.wap.wbmp, image/gif, application/*, text/html, application/xhtml+xml, application/vnd.wap.wml+xml, text/css', 'Accept-Charset':'ISO-8859-1, US-ASCII, UTF-8; Q=0.8'}#, ISO-10646-UCS-2; Q=0.6' conn = httplib.HTTPConnection("127.0.0.1:8080") conn.request("GET", "/mobile/wap/games/",headers=headers) response = conn.getresponse() print response.status, response.reason data = response.read() print data
conn.close()
sendHTTP()
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
On Mon, 2003-02-10 at 10:27, Romain Slootmaekers wrote:
Romain Slootmaekers wrote: Hi, below the original post:
it is a bug in Products/Localizer/Accept.py
here's the code to fix it:
#--- code fix starting from line 102 ---
# Get the quality try: if len(x) == 2: quality = x[1] # Get the quality quality =quality.strip() quality = quality[2:] # Get the number (remove "q=") quality = float(quality) # Change it to float else: quality = 1.0 except: quality=1.0 #--- end of code fix ---
I added a strip() and a try/except for other things that can go wrong.
Please, please, please, don't use generic "except:". They can cause ZODB corruption if they catch ConflictErrors. In the case of the code above, test for specific errors that "float(quality)" or "x[1]" can raise and catch those instead. -- Ideas don't stay in some minds very long because they don't like solitary confinement.
On Mon, Feb 10, 2003 at 06:11:15PM -0200, Leonardo Rochael Almeida wrote:
Please, please, please, don't use generic "except:". They can cause ZODB corruption if they catch ConflictErrors.
Can anyone else confirm or deny??? If true, it needs to be much more widely publicized! $ cd /usr/src/Zope-2.6.1-src/lib/python/Products/ $ find . -name "*py" -exec grep -H "except:" {} \; | wc -l 170 well, this is all stuff that comes with Zope, hopefully they have been vetted... but then there's all these 3rd-party products: $ cd $INSTANCE_HOME/Products $ find . -name "*py" -exec grep -H "except:" {} \; | wc -l 133 Great, so there's at least 133 things to examine to see if they could catch a ConflictError. And I only wrote about 15 of those. The rest could be very time-consuming to audit. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's THE GOBLET! (random hero from isometric.spaceninja.com)
Paul Winkler wrote:
On Mon, Feb 10, 2003 at 06:11:15PM -0200, Leonardo Rochael Almeida wrote:
Please, please, please, don't use generic "except:". They can cause ZODB corruption if they catch ConflictErrors.
Can anyone else confirm or deny??? If true, it needs to be much more widely publicized!
It's an open bug on Sourceforge: http://sourceforge.net/tracker/index.php?func=detail&aid=637279&group_id=156...
Great, so there's at least 133 things to examine to see if they could catch a ConflictError. And I only wrote about 15 of those. The rest could be very time-consuming to audit.
"tal:on-error" also catches all exceptions. It could be made to catch all exceptions except ConflictError, but I don't feel like that's the right solution. I think the right solution is to prevent applications from committing potentially conflicting data, even when ConflictErrors are ignored. This doesn't seem to be a showstopper for now, but as more applications use ZODB, it could become a bigger problem. Shane
On Mon, Feb 10, 2003 at 03:47:58PM -0500, Shane Hathaway wrote:
This doesn't seem to be a showstopper for now, but as more applications use ZODB, it could become a bigger problem.
Well, what kind of errors can this cause in the ZODB? If it's just a matter of writing a mixture of old and new data to the ZODB, that's annoying but I agree it's not a showstopper. If it does in fact lead to FileStorage damage, that's another story. I'm still feeling touchy about the latter since I recently found (and still have not explained) >10k "bad transaction length" errors, 700 "invalid transaction length" errors, and 200 POSKeyErrors in a single FileStorage. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's MALIFEROUS CONTORTIONIST! (random hero from isometric.spaceninja.com)
Paul Winkler wrote:
On Mon, Feb 10, 2003 at 03:47:58PM -0500, Shane Hathaway wrote:
This doesn't seem to be a showstopper for now, but as more applications use ZODB, it could become a bigger problem.
Well, what kind of errors can this cause in the ZODB? If it's just a matter of writing a mixture of old and new data to the ZODB, that's annoying but I agree it's not a showstopper. If it does in fact lead to FileStorage damage, that's another story. I'm still feeling touchy about the latter since I recently found (and still have not explained) >10k "bad transaction length" errors, 700 "invalid transaction length" errors, and 200 POSKeyErrors in a single FileStorage.
Badly handled conflict errors can result in POSKeyErrors (I think), but not "bad transaction length" errors. I think you might be dealing with a different problem. Shane
Could this be done by initializing a dictionary at startup keyed on thread-id that a ConflictError exception's __init__ could stick a marker into, then checking that dictionary at commit time and disallowing the commit if the marker still existed? On Mon, 2003-02-10 at 15:47, Shane Hathaway wrote:
"tal:on-error" also catches all exceptions. It could be made to catch all exceptions except ConflictError, but I don't feel like that's the right solution. I think the right solution is to prevent applications from committing potentially conflicting data, even when ConflictErrors are ignored.
This doesn't seem to be a showstopper for now, but as more applications use ZODB, it could become a bigger problem.
Chris McDonough wrote:
Could this be done by initializing a dictionary at startup keyed on thread-id that a ConflictError exception's __init__ could stick a marker into, then checking that dictionary at commit time and disallowing the commit if the marker still existed?
Yes, but Transaction objects are already keyed on the thread ID, so I think you just have to set a "doomed" flag on the transaction. Attempts to commit doomed transactions result in either ConflictError or "DoomedTransactionError". ;-) Aborting the transaction (or beginning a new transaction) resets the flag. Shane
On Mon, 2003-02-10 at 15:47, Shane Hathaway wrote:
"tal:on-error" also catches all exceptions. It could be made to catch all exceptions except ConflictError, but I don't feel like that's the right solution. I think the right solution is to prevent applications from committing potentially conflicting data, even when ConflictErrors are ignored.
This doesn't seem to be a showstopper for now, but as more applications use ZODB, it could become a bigger problem.
On Mon, 2003-02-10 at 20:46, Chris McDonough wrote:
Could this be done by initializing a dictionary at startup keyed on thread-id that a ConflictError exception's __init__ could stick a marker into, then checking that dictionary at commit time and disallowing the commit if the marker still existed?
+1 And logging a zLOG.ERROR (or maybe even zLOG.PANIC?) level message to that effect IMHO, something that blocked commits of ConflictError'ed transactions should be a requirement in Zope 2.7 I believe that unqualified "except:"'s are the most common cause of unexplained Zope ZODB corruption. -- Ideas don't stay in some minds very long because they don't like solitary confinement.
On Monday 10 February 2003 8:47 pm, Shane Hathaway wrote:
Great, so there's at least 133 things to examine to see if they could catch a ConflictError. And I only wrote about 15 of those. The rest could be very time-consuming to audit.
"tal:on-error" also catches all exceptions. It could be made to catch all exceptions except ConflictError, but I don't feel like that's the right solution.
A while ago I tracked down a bug in one of our products to a case where a mutator method failed half way. It raised an exception and left the object in an invalid state. This would be safe except for a dtml-try that swallowed the exception. Recently I have been experimenting with this attached module to ensure that an application-level object can veto a transaction. I guess it would be applicable to ConflicctErrors too.
Toby Dickenson wrote:
On Monday 10 February 2003 8:47 pm, Shane Hathaway wrote:
Great, so there's at least 133 things to examine to see if they could catch a ConflictError. And I only wrote about 15 of those. The rest could be very time-consuming to audit.
"tal:on-error" also catches all exceptions. It could be made to catch all exceptions except ConflictError, but I don't feel like that's the right solution.
A while ago I tracked down a bug in one of our products to a case where a mutator method failed half way. It raised an exception and left the object in an invalid state. This would be safe except for a dtml-try that swallowed the exception.
Recently I have been experimenting with this attached module to ensure that an application-level object can veto a transaction. I guess it would be applicable to ConflicctErrors too.
Ah-ha, that's a very elegant approach. By reusing the notional "transaction participant" interface, you didn't have to change the behavior of transactions in any way. Cool. We should definitely do it that way. I'm thinking the veto should be added by Connection.commit() and Connection.setstate() whenever a conflict is about to propagate. What do you think? Shane
On Tue, 2003-02-11 at 13:19, Shane Hathaway wrote:
Toby Dickenson wrote:
On Monday 10 February 2003 8:47 pm, Shane Hathaway wrote:
Great, so there's at least 133 things to examine to see if they could catch a ConflictError. And I only wrote about 15 of those. The rest could be very time-consuming to audit.
"tal:on-error" also catches all exceptions. It could be made to catch all exceptions except ConflictError, but I don't feel like that's the right solution.
A while ago I tracked down a bug in one of our products to a case where a mutator method failed half way. It raised an exception and left the object in an invalid state. This would be safe except for a dtml-try that swallowed the exception.
Recently I have been experimenting with this attached module to ensure that an application-level object can veto a transaction. I guess it would be applicable to ConflicctErrors too.
Ah-ha, that's a very elegant approach. By reusing the notional "transaction participant" interface, you didn't have to change the behavior of transactions in any way. Cool. We should definitely do it that way.
I'm thinking the veto should be added by Connection.commit() and Connection.setstate() whenever a conflict is about to propagate. What do you think?
Maybe I don't understand all the transaction code correctly, but shouldn't it be set in ConflictError.__init__()? I suggest that we also take the ConflictError.__init__() opportunity to store a representation of the current traceback. This representation would be dumped to zLOG in case the veto has to be enforced, so that we know when the ConflictError originally happened. This way we can take steps to remove the offending generic "except:". -- Ideas don't stay in some minds very long because they don't like solitary confinement.
On Tuesday 11 February 2003 11:04 pm, Leonardo Rochael Almeida wrote:
On Tue, 2003-02-11 at 13:19, Shane Hathaway wrote:
I'm thinking the veto should be added by Connection.commit() and Connection.setstate() whenever a conflict is about to propagate. What do you think?
Maybe I don't understand all the transaction code correctly, but shouldn't it be set in ConflictError.__init__()?
No. ConflictError exceptions can be created by the storage, but swallowed by conflict resolution code in the connection. Shane is correct; we assert a veto at the point where the conflict is terminal. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson
Leonardo Rochael Almeida wrote:
On Mon, 2003-02-10 at 10:27, Romain Slootmaekers wrote:
Romain Slootmaekers wrote: Hi, below the original post:
it is a bug in Products/Localizer/Accept.py
here's the code to fix it:
#--- code fix starting from line 102 ---
# Get the quality try: if len(x) == 2: quality = x[1] # Get the quality quality =quality.strip() quality = quality[2:] # Get the number (remove "q=") quality = float(quality) # Change it to float else: quality = 1.0 except: quality=1.0 #--- end of code fix ---
I added a strip() and a try/except for other things that can go wrong.
Please, please, please, don't use generic "except:". They can cause ZODB corruption if they catch ConflictErrors. In the case of the code above, test for specific errors that "float(quality)" or "x[1]" can raise and catch those instead.
In general, a generic except is evil. Here, I seriously don't see how the code inside the try/except can cause a conflict error or other serious damage. In general, general remarks are evil ;) Sloot.
The bug was fixed three weeks ago (branch-1-0), it is time for a release I guess. Well, only the "strip()", no "try/except" clause yet. By the way, Localizer bugs/patches should be sent to the Localizer mailing list or to the SourceForge tracker of the LLEU project. Romain Slootmaekers wrote:
Romain Slootmaekers wrote: Hi, below the original post:
it is a bug in Products/Localizer/Accept.py
here's the code to fix it:
#--- code fix starting from line 102 ---
# Get the quality try: if len(x) == 2: quality = x[1] # Get the quality quality =quality.strip() quality = quality[2:] # Get the number (remove "q=") quality = float(quality) # Change it to float else: quality = 1.0 except: quality=1.0 #--- end of code fix ---
I added a strip() and a try/except for other things that can go wrong.
This way Nokia Wap surfers can access your site while Localizer is installed.
have fun,
Sloot.
BTW, exceptions that originate here are not caught in the error_log. this is probably unintended as well.
----------------- original post -----------------------------
Hi,
We have problems with the Accept-Charset headers that are sent to the zope server by some mobile phones: fi: 'Accept-Charset':'ISO-8859-1, US-ASCII, UTF-8; Q=0.8'
These cause a 500 internal server error. (Error Value: invalid literal for float(): =0.8 )
I checked with the HTTP1.1 RFC26160 section 14.2 and the header seems valid, but in fact isn't: there is a space between the semi-colon and the Q char (which shouldn't be there acoording to the spec).
So, in principle, this is not a zope server bug, but an inflexibility.
I think the server should be a bit more lenient. a simple .strip() in the right place should do.
Could you guys fix this?
TIA,
Sloot.
BTW, for completeness, I included a quick test program is included below (causes a 500 on my zope 2.6.0) :
#--------------------- program illustrating the behavior ----------- import httplib, urllib
def sendHTTP(): """
""" headers = {'Accept': 'application/vnd.wap.wmlc, application/vnd.wap.wmlscriptc, application/vnd.wap.wbxml, image/vnd.wap.wbmp, image/gif, application/*, text/html, application/xhtml+xml, application/vnd.wap.wml+xml, text/css', 'Accept-Charset':'ISO-8859-1, US-ASCII, UTF-8; Q=0.8'}#, ISO-10646-UCS-2; Q=0.6' conn = httplib.HTTPConnection("127.0.0.1:8080") conn.request("GET", "/mobile/wap/games/",headers=headers) response = conn.getresponse() print response.status, response.reason data = response.read() print data
conn.close()
sendHTTP()
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- J. David Ibáñez, http://www.j-david.net Software Engineer / Ingénieur Logiciel / Ingeniero de Software
participants (7)
-
Chris McDonough -
Juan David Ibáñez Palomar -
Leonardo Rochael Almeida -
Paul Winkler -
Romain Slootmaekers -
Shane Hathaway -
Toby Dickenson