[Zope-CVS] CVS: Products/ZCTextIndex - QueryParser.py:1.1.2.16

[Tim Peters]

Update of /cvs-repository/Products/ZCTextIndex
In directory cvs.zope.org:/tmp/cvs-serv7705

Modified Files:
      Tag: TextIndexDS9-branch
	QueryParser.py 
Log Message:
The messier the regexp, the more verbose mode helps.


=== Products/ZCTextIndex/QueryParser.py 1.1.2.15 => 1.1.2.16 ===
 
 # Magical regex to tokenize.  A beauty, ain't it. :-)
-_tokenizer_regex = r'[()]|[^()\s"]*(?:"[^"]*"[^()\s"]*)+|[^()\s"]+'
+_tokenizer_regex = re.compile(r"""
+    # a paren
+    [()]
+    # or a fleeblegorg (something with double quotes, or lots of double quotes)
+    # XXX do we really want stuff like AB"C"D"EFG"H""I to be "a token"?
+|   [^()\s"]* (?: "[^"]*"
+                  [^()\s"]*
+              )+
+    # or a non-empty string without whitespace, parens or double quotes
+|   [^()\s"]+
+""", re.VERBOSE)
 
 class QueryParser: