[Zope3-checkins] SVN: Zope3/branches/jim-i18n-dev/src/zope/i18n/
Added an API for collating text and a fallback implementation.
Jim Fulton
jim at zope.com
Fri Nov 18 14:31:11 EST 2005
Log message for revision 40233:
Added an API for collating text and a fallback implementation.
(Apps that really care will probably use an ICU-based adapter that
we will provide soonish.)
Changed:
U Zope3/branches/jim-i18n-dev/src/zope/i18n/interfaces/locales.py
A Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py
A Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt
A Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py
-=-
Modified: Zope3/branches/jim-i18n-dev/src/zope/i18n/interfaces/locales.py
===================================================================
--- Zope3/branches/jim-i18n-dev/src/zope/i18n/interfaces/locales.py 2005-11-18 18:29:30 UTC (rev 40232)
+++ Zope3/branches/jim-i18n-dev/src/zope/i18n/interfaces/locales.py 2005-11-18 19:31:10 UTC (rev 40233)
@@ -618,3 +618,22 @@
If an key is not found or is None, the next higher up Locale
object is consulted.
"""
+
+class ICollator(Interface):
+ """Provide support for collating text strings
+
+ This interface will typically be provided by adapting a locale.
+ """
+
+ def key(text):
+ """Return a collation key for the given text.
+ """
+
+ def cmp(text1, text2):
+ """Compare two text strings.
+
+ The return value is negative if text1 < text2, 0 is they are
+ equal, and positive if text1 > text2.
+ """
+
+
Added: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py
===================================================================
--- Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py 2005-11-18 18:29:30 UTC (rev 40232)
+++ Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py 2005-11-18 19:31:10 UTC (rev 40233)
@@ -0,0 +1,33 @@
+##############################################################################
+#
+# Copyright (c) 2004 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Fallback collator
+
+$Id$
+"""
+
+from unicodedata import normalize
+
+class FallbackCollator:
+
+ def __init__(self, locale):
+ pass
+
+ def key(self, s):
+ s = normalize('NFKC', s)
+ return s.lower(), s
+
+ def cmp(self, s1, s2):
+ return cmp(self.key(s1), self.key(s2))
+
+
Property changes on: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.py
___________________________________________________________________
Name: svn:keywords
+ Id
Name: svn:eol-style
+ native
Added: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt
===================================================================
--- Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt 2005-11-18 18:29:30 UTC (rev 40232)
+++ Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt 2005-11-18 19:31:10 UTC (rev 40233)
@@ -0,0 +1,63 @@
+Fallback Collator
+=================
+
+The zope.i18n.interfaces.locales.ICollator interface defines an API
+for collating text. Why is this important? Simply sorting unicode
+strings doesn't provide an ordering that users in a given locale will
+fine useful. Various languages have text sorting conventions that
+don't agree with the ordering of unicode code points. (This is even
+true for English. :)
+
+Text collation is a fairly involved process. Systems that need this,
+will likely use something like ICU
+(http://www-306.ibm.com/software/globalization/icu,
+http://pyicu.osafoundation.org/). We don't want to introduce a
+dependency on ICU and this time, so we are providing a fallback
+collator that:
+
+- Provides an implementation of the ICollator interface that can be
+ used for development, and
+
+- Provides a small amount of value, at least for English speakers. :)
+
+Application code should obtain a collator by adapting a locale to
+ICollator. Here we just call the collator factory with None. The
+fallback collator doesn't actually use the locale, although
+application code should certainly *not* count on this.
+
+ >>> import zope.i18n.locales.fallbackcollator
+ >>> collator = zope.i18n.locales.fallbackcollator.FallbackCollator(None)
+
+Now, we can pass the collator's key method to sort functions to sort
+strings in a slightly friendly way:
+
+ >>> sorted([u'Sam', u'sally', u'Abe', u'alice', u'Terry', u'tim'],
+ ... key=collator.key)
+ [u'Abe', u'alice', u'sally', u'Sam', u'Terry', u'tim']
+
+
+The collator has a very simple algorithm. It normalizes strings and
+then returns a tuple with the result of lower-casing the normalized
+string and the normalized string. We can see this by calling the key
+method, which converts unicode strings to collation keys:
+
+ >>> collator.key(u'Sam')
+ (u'sam', u'Sam')
+
+ >>> collator.key(u'\xc6\xf8a\u030a')
+ (u'\xe6\xf8\xe5', u'\xc6\xf8\xe5')
+
+There is also a cmp function for comparing strings:
+
+ >>> collator.cmp(u'Terry', u'sally')
+ 1
+
+
+ >>> collator.cmp(u'sally', u'Terry')
+ -1
+
+ >>> collator.cmp(u'terry', u'Terry')
+ 1
+
+ >>> collator.cmp(u'terry', u'terry')
+ 0
Property changes on: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/fallbackcollator.txt
___________________________________________________________________
Name: svn:eol-style
+ native
Added: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py
===================================================================
--- Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py 2005-11-18 18:29:30 UTC (rev 40232)
+++ Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py 2005-11-18 19:31:10 UTC (rev 40233)
@@ -0,0 +1,25 @@
+##############################################################################
+#
+# Copyright (c) 2004 Zope Corporation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.0 (ZPL). A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+
+import unittest
+from zope.testing import doctest
+
+def test_suite():
+ return unittest.TestSuite((
+ doctest.DocFileSuite('../fallbackcollator.txt'),
+ ))
+
+if __name__ == '__main__':
+ unittest.main(defaultTest='test_suite')
+
Property changes on: Zope3/branches/jim-i18n-dev/src/zope/i18n/locales/tests/test_fallbackcollator.py
___________________________________________________________________
Name: svn:keywords
+ Id
Name: svn:eol-style
+ native
More information about the Zope3-Checkins
mailing list