[Zope3-checkins] SVN: Zope3/branches/jim-i18n-dev/src/zope/i18n/ Added an API for collating text and a fallback implementation.

Jim Fulton jim at zope.com
Fri Nov 18 14:31:11 EST 2005 

Log message for revision 40233:
  Added an API for collating text and a fallback implementation.
  (Apps that really care will probably use an ICU-based adapter that
  we will provide soonish.)
  

Changed:
  U   Zope3/branches/jim-i18n-dev/src/zope/i18n/interfaces/locales.py
  A   Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py
  A   Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt
  A   Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py

-=-
Modified: Zope3/branches/jim-i18n-dev/src/zope/i18n/interfaces/locales.py
===================================================================
--- Zope3/branches/jim-i18n-dev/src/zope/i18n/interfaces/locales.py	2005-11-18 18:29:30 UTC (rev 40232)
+++ Zope3/branches/jim-i18n-dev/src/zope/i18n/interfaces/locales.py	2005-11-18 19:31:10 UTC (rev 40233)
@@ -618,3 +618,22 @@
         If an key is not found or is None, the next higher up Locale
         object is consulted.
         """
+
+class ICollator(Interface):
+    """Provide support for collating text strings
+
+    This interface will typically be provided by adapting a locale.
+    """
+
+    def key(text):
+        """Return a collation key for the given text.
+        """
+
+    def cmp(text1, text2):
+        """Compare two text strings.
+
+        The return value is negative if text1 < text2, 0 is they are
+        equal, and positive if text1 > text2.
+        """
+    
+    

Added: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py
===================================================================
--- Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py	2005-11-18 18:29:30 UTC (rev 40232)
+++ Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py	2005-11-18 19:31:10 UTC (rev 40233)
@@ -0,0 +1,33 @@
+##############################################################################
+#
+# Copyright (c) 2004 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Fallback collator
+
+$Id$
+"""
+
+from unicodedata import normalize
+
+class FallbackCollator:
+
+    def __init__(self, locale):
+        pass
+
+    def key(self, s):
+        s = normalize('NFKC', s)
+        return s.lower(), s
+
+    def cmp(self, s1, s2):
+        return cmp(self.key(s1), self.key(s2))
+
+        


Property changes on: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py
___________________________________________________________________
Name: svn:keywords
   + Id
Name: svn:eol-style
   + native

Added: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt
===================================================================
--- Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt	2005-11-18 18:29:30 UTC (rev 40232)
+++ Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt	2005-11-18 19:31:10 UTC (rev 40233)
@@ -0,0 +1,63 @@
+Fallback Collator
+=================
+
+The zope.i18n.interfaces.locales.ICollator interface defines an API
+for collating text.  Why is this important?  Simply sorting unicode
+strings doesn't provide an ordering that users in a given locale will
+fine useful.  Various languages have text sorting conventions that
+don't agree with the ordering of unicode code points. (This is even
+true for English. :)
+
+Text collation is a fairly involved process.  Systems that need this,
+will likely use something like ICU
+(http://www-306.ibm.com/software/globalization/icu, 
+http://pyicu.osafoundation.org/).  We don't want to introduce a
+dependency on ICU and this time, so we are providing a fallback
+collator that:
+
+- Provides an implementation of the ICollator interface that can be
+  used for development, and
+
+- Provides a small amount of value, at least for English speakers. :)
+
+Application code should obtain a collator by adapting a locale to
+ICollator.  Here we just call the collator factory with None. The
+fallback collator doesn't actually use the locale, although
+application code should certainly *not* count on this.
+
+    >>> import zope.i18n.locales.fallbackcollator
+    >>> collator = zope.i18n.locales.fallbackcollator.FallbackCollator(None)
+
+Now, we can pass the collator's key method to sort functions to sort
+strings in a slightly friendly way:
+
+    >>> sorted([u'Sam', u'sally', u'Abe', u'alice', u'Terry', u'tim'],
+    ...        key=collator.key)
+    [u'Abe', u'alice', u'sally', u'Sam', u'Terry', u'tim']
+
+
+The collator has a very simple algorithm.  It normalizes strings and
+then returns a tuple with the result of lower-casing the normalized
+string and the normalized string.  We can see this by calling the key
+method, which converts unicode strings to collation keys:
+
+    >>> collator.key(u'Sam')
+    (u'sam', u'Sam')
+
+    >>> collator.key(u'\xc6\xf8a\u030a')
+    (u'\xe6\xf8\xe5', u'\xc6\xf8\xe5')
+
+There is also a cmp function for comparing strings:
+
+    >>> collator.cmp(u'Terry', u'sally')
+    1
+
+
+    >>> collator.cmp(u'sally', u'Terry')
+    -1
+
+    >>> collator.cmp(u'terry', u'Terry')
+    1
+
+    >>> collator.cmp(u'terry', u'terry')
+    0


Property changes on: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt
___________________________________________________________________
Name: svn:eol-style
   + native

Added: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py
===================================================================
--- Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py	2005-11-18 18:29:30 UTC (rev 40232)
+++ Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py	2005-11-18 19:31:10 UTC (rev 40233)
@@ -0,0 +1,25 @@
+##############################################################################
+#
+# Copyright (c) 2004 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+
+import unittest
+from zope.testing import doctest
+
+def test_suite():
+    return unittest.TestSuite((
+        doctest.DocFileSuite('../fallbackcollator.txt'),
+        ))
+
+if __name__ == '__main__':
+    unittest.main(defaultTest='test_suite')
+


Property changes on: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py
___________________________________________________________________
Name: svn:keywords
   + Id
Name: svn:eol-style
   + native

More information about the Zope3-Checkins mailing list