Additional locales for zope.i18n.locales.data?
I've run into a problem this morning with i18n in Zope3, and I'm not exactly sure what the appropriate approach is. We have a set of South African translations, paticularly Zulu (zu) and Sotho (st) that are getting picked up correctly. Upon some inspection it appears that the i18n negotiator is skipping over them, with an exception such as: The desired locale is not available. Path: /Users/nathan/p/cc.engine/branches/production/eggs/zope.i18n-3.4.0-py2.4.egg/zope/i18n/locales/data/st.xml Upon inspection I can confirm that the locales which are not working are those without .xml files in the data directory. So a few questions: * Is there an external source for those .xml files, or do we create them ourselves? * Translation seems to work OK (in the case of Zulu, not Sotho) from ZPT, but not from Python; does this sound sane, or perhaps I'm missing something? Thanks, Nathan
Nathan Yergler wrote:
I've run into a problem this morning with i18n in Zope3, and I'm not exactly sure what the appropriate approach is. We have a set of South African translations, paticularly Zulu (zu) and Sotho (st) that are getting picked up correctly. Upon some inspection it appears that the i18n negotiator is skipping over them, with an exception such as:
The desired locale is not available. Path: /Users/nathan/p/cc.engine/branches/production/eggs/zope.i18n-3.4.0-py2.4.egg/zope/i18n/locales/data/st.xml
Upon inspection I can confirm that the locales which are not working are those without .xml files in the data directory. So a few questions:
* Is there an external source for those .xml files, or do we create them ourselves?
The XML files are from the ICU repository (http://www.unicode.org/cldr/). AFAIK we're using quite an old version of the files. The problem is we can't simply upgrade to the newest release because the XML DTD changed and the code in zope.i18n.locales isn't very robust regarding DTD changes (which is quite hard anyway). I don't think anybody would object if the files and the code were updated *wink* ;).
* Translation seems to work OK (in the case of Zulu, not Sotho) from ZPT, but not from Python; does this sound sane, or perhaps I'm missing something?
How are you translating from Python? zope.i18n.translate(msg, context=request) should do it.
On 11/27/07, Philipp von Weitershausen <philipp@weitershausen.de> wrote:
Nathan Yergler wrote:
I've run into a problem this morning with i18n in Zope3, and I'm not exactly sure what the appropriate approach is. We have a set of South African translations, paticularly Zulu (zu) and Sotho (st) that are getting picked up correctly. Upon some inspection it appears that the i18n negotiator is skipping over them, with an exception such as:
The desired locale is not available. Path: /Users/nathan/p/cc.engine/branches/production/eggs/zope.i18n-3.4.0-py2.4.egg/zope/i18n/locales/data/st.xml
Upon inspection I can confirm that the locales which are not working are those without .xml files in the data directory. So a few questions:
* Is there an external source for those .xml files, or do we create them ourselves?
The XML files are from the ICU repository (http://www.unicode.org/cldr/). AFAIK we're using quite an old version of the files. The problem is we can't simply upgrade to the newest release because the XML DTD changed and the code in zope.i18n.locales isn't very robust regarding DTD changes (which is quite hard anyway). I don't think anybody would object if the files and the code were updated *wink* ;).
Where "wink" is defined as "so why don't you go ahead and do it", I expect :).
* Translation seems to work OK (in the case of Zulu, not Sotho) from ZPT, but not from Python; does this sound sane, or perhaps I'm missing something?
How are you translating from Python?
zope.i18n.translate(msg, context=request)
Ah; I wasn't passing in the context. Thanks.
should do it.
On 27 Nov 2007, at 17:14 , Nathan Yergler wrote:
On 11/27/07, Philipp von Weitershausen <philipp@weitershausen.de> wrote:
Nathan Yergler wrote:
I've run into a problem this morning with i18n in Zope3, and I'm not exactly sure what the appropriate approach is. We have a set of South African translations, paticularly Zulu (zu) and Sotho (st) that are getting picked up correctly. Upon some inspection it appears that the i18n negotiator is skipping over them, with an exception such as:
The desired locale is not available. Path: /Users/nathan/p/cc.engine/branches/production/eggs/ zope.i18n-3.4.0-py2.4.egg/zope/i18n/locales/data/st.xml
Upon inspection I can confirm that the locales which are not working are those without .xml files in the data directory. So a few questions:
* Is there an external source for those .xml files, or do we create them ourselves?
The XML files are from the ICU repository (http://www.unicode.org/cldr/). AFAIK we're using quite an old version of the files. The problem is we can't simply upgrade to the newest release because the XML DTD changed and the code in zope.i18n.locales isn't very robust regarding DTD changes (which is quite hard anyway). I don't think anybody would object if the files and the code were updated *wink* ;).
Where "wink" is defined as "so why don't you go ahead and do it", I expect :).
Exactly :).
* Translation seems to work OK (in the case of Zulu, not Sotho) from ZPT, but not from Python; does this sound sane, or perhaps I'm missing something?
How are you translating from Python?
zope.i18n.translate(msg, context=request)
Ah; I wasn't passing in the context. Thanks.
Right. zope.i18n.translate() needs a context that it can adapt to IUserPreferredLanguages so that it can deduct the target language (it can't guess it from thin air). This context is the request (sic!).
Nathan Yergler wrote:
On 11/27/07, Philipp von Weitershausen <philipp@weitershausen.de> wrote:
Nathan Yergler wrote:
I've run into a problem this morning with i18n in Zope3, and I'm not exactly sure what the appropriate approach is. We have a set of South African translations, paticularly Zulu (zu) and Sotho (st) that are getting picked up correctly. Upon some inspection it appears that the i18n negotiator is skipping over them, with an exception such as:
The desired locale is not available. Path: /Users/nathan/p/cc.engine/branches/production/eggs/zope.i18n-3.4.0-py2.4.egg/zope/i18n/locales/data/st.xml
Upon inspection I can confirm that the locales which are not working are those without .xml files in the data directory. So a few questions:
* Is there an external source for those .xml files, or do we create them ourselves? The XML files are from the ICU repository (http://www.unicode.org/cldr/). AFAIK we're using quite an old version of the files. The problem is we can't simply upgrade to the newest release because the XML DTD changed and the code in zope.i18n.locales isn't very robust regarding DTD changes (which is quite hard anyway). I don't think anybody would object if the files and the code were updated *wink* ;).
Where "wink" is defined as "so why don't you go ahead and do it", I expect :).
If you follow the wink, http://www.unicode.org/cldr/repository_access.html has the details to the files. Currently the latest is at http://unicode.org/Public/cldr/1.5.0/core.zip. Zope at this point still uses LDML 1.0 whereas the latest version is LDML 1.5. Upon casual inspection of the files it seems their basic structure is still the same, though more careful inspection is required. Hanno P.S. Updating those files is on my list for some month now, but I don't expect to have any time for that for real in the near future :(
[Re-including the list in the reply...] On 11/27/07, Hanno Schlichting <plone@hannosch.info> wrote:
Nathan Yergler wrote:
On 11/27/07, Philipp von Weitershausen <philipp@weitershausen.de> wrote:
Nathan Yergler wrote:
I've run into a problem this morning with i18n in Zope3, and I'm not exactly sure what the appropriate approach is. We have a set of South African translations, paticularly Zulu (zu) and Sotho (st) that are getting picked up correctly. Upon some inspection it appears that the i18n negotiator is skipping over them, with an exception such as:
The desired locale is not available. Path: /Users/nathan/p/cc.engine/branches/production/eggs/zope.i18n-3.4.0-py2.4.egg/zope/i18n/locales/data/st.xml
Upon inspection I can confirm that the locales which are not working are those without .xml files in the data directory. So a few questions:
* Is there an external source for those .xml files, or do we create them ourselves? The XML files are from the ICU repository (http://www.unicode.org/cldr/). AFAIK we're using quite an old version of the files. The problem is we can't simply upgrade to the newest release because the XML DTD changed and the code in zope.i18n.locales isn't very robust regarding DTD changes (which is quite hard anyway). I don't think anybody would object if the files and the code were updated *wink* ;).
Where "wink" is defined as "so why don't you go ahead and do it", I expect :).
If you follow the wink, http://www.unicode.org/cldr/repository_access.html has the details to the files. Currently the latest is at http://unicode.org/Public/cldr/1.5.0/core.zip.
Zope at this point still uses LDML 1.0 whereas the latest version is LDML 1.5.
Upon casual inspection of the files it seems their basic structure is still the same, though more careful inspection is required.
I've been avoiding even less interesting work this afternoon, taking a look at this. I started by dumping the new files into the data directory and just running the tests. As expected, things blew up in a really spectacular manner. After some wrangling I've discovered that in the newer versions of the CLDR dataset some of the information previously contained in the locale files (such as weekend start/end, etc), is now in located in a "supplemental" file. While this makes a certain amount of sense (it's tied to territories, not really languages), it does mean that the information needed for a Locale is no longer self-contained in a single XML file. So unfortunately it's going to require some more work to fix up the loader; I'll probably create a branch to work on this some...
Hanno
P.S. Updating those files is on my list for some month now, but I don't expect to have any time for that for real in the near future :(
On Nov 28, 2007 12:29 AM, Nathan Yergler <nathan@yergler.net> wrote:
If you follow the wink, http://www.unicode.org/cldr/repository_access.html has the details to the files. Currently the latest is at http://unicode.org/Public/cldr/1.5.0/core.zip.
Zope at this point still uses LDML 1.0 whereas the latest version is LDML 1.5.
Upon casual inspection of the files it seems their basic structure is still the same, though more careful inspection is required.
I've been avoiding even less interesting work this afternoon, taking a look at this. I started by dumping the new files into the data directory and just running the tests. As expected, things blew up in a really spectacular manner.
After some wrangling I've discovered that in the newer versions of the CLDR dataset some of the information previously contained in the locale files (such as weekend start/end, etc), is now in located in a "supplemental" file. While this makes a certain amount of sense (it's tied to territories, not really languages), it does mean that the information needed for a Locale is no longer self-contained in a single XML file.
So unfortunately it's going to require some more work to fix up the loader; I'll probably create a branch to work on this some...
Has anyone looked at Babel (http://babel.edgewall.org/)? It includes a python interface to CLDR, which if usable would let us off the hook of maintaining such an interface ourselves. -- Martijn Pieters
Martijn Pieters wrote:
On Nov 28, 2007 12:29 AM, Nathan Yergler <nathan@yergler.net> wrote:
If you follow the wink, http://www.unicode.org/cldr/repository_access.html has the details to the files. Currently the latest is at http://unicode.org/Public/cldr/1.5.0/core.zip.
Zope at this point still uses LDML 1.0 whereas the latest version is LDML 1.5.
Upon casual inspection of the files it seems their basic structure is still the same, though more careful inspection is required. I've been avoiding even less interesting work this afternoon, taking a look at this. I started by dumping the new files into the data directory and just running the tests. As expected, things blew up in a really spectacular manner.
After some wrangling I've discovered that in the newer versions of the CLDR dataset some of the information previously contained in the locale files (such as weekend start/end, etc), is now in located in a "supplemental" file. While this makes a certain amount of sense (it's tied to territories, not really languages), it does mean that the information needed for a Locale is no longer self-contained in a single XML file.
So unfortunately it's going to require some more work to fix up the loader; I'll probably create a branch to work on this some...
Has anyone looked at Babel (http://babel.edgewall.org/)? It includes a python interface to CLDR, which if usable would let us off the hook of maintaining such an interface ourselves.
Looking at the API docs, it seems that the ILocale implementations in zope.i18n.locale could simply make appropriate calls to functions in the Babel API. So +1 from me, but since it looks like Nathan will be doing the work, he should decide :). Babel seems to exist since mid-2007, by the way. I wonder, if we had made zope.i18n separately available sooner and actually told people about it, it possibly wouldn't have been necessary to duplicate the effort there. In fact, the Babel website lists zope.i18n as an alternative, but finds that it's tied too closely to Zope 3. * zope.i18n has a scope similar to Babel (covering both gettext and the CLDR), but is closely tied to Zope 3. * PyICU is a Python/SWIG wrapper for the ICU library, and provides access to locale data based on the CLDR (or the other way around). It would be interesting to get in touch with the original authors and ask them whether zope.i18n's 4 or so dependencies still make it too tied to Zope 3.
Philipp von Weitershausen wrote:
It would be interesting to get in touch with the original authors and ask them whether zope.i18n's 4 or so dependencies still make it too tied to Zope 3.
For me, it's be interesting to see what it'd take to drop zope.18n completely and just use Babel ;-) (ie: we as the zope community should be trying to "do less" and re-use good stuff from the python community rather than building our own which then ends up in a mess of unnecessary dependencies...) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Withers wrote:
Philipp von Weitershausen wrote:
It would be interesting to get in touch with the original authors and ask them whether zope.i18n's 4 or so dependencies still make it too tied to Zope 3.
For me, it's be interesting to see what it'd take to drop zope.18n completely and just use Babel ;-)
(ie: we as the zope community should be trying to "do less" and re-use good stuff from the python community rather than building our own which then ends up in a mess of unnecessary dependencies...)
Please separate out the dependency problem from NIH: in this case, zope.i18n predates the NIH package by *years* (the i18n support was the topic of the *first* extra-mural Zope3 sprint, in January of 2002). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHTeri+gerLs4ltQ4RApR8AKCYl0brIQszXFHVIMVltXIG+mRSJACgmVFt jZ3ll2SOgx5ElQQ3J3H/sFA= =iGLY -----END PGP SIGNATURE-----
Tres Seaver wrote:
Please separate out the dependency problem from NIH: in this case, zope.i18n predates the NIH package by *years* (the i18n support was the topic of the *first* extra-mural Zope3 sprint, in January of 2002).
Actually no, if I understand Philipp's comment correctly, the NIH problem *they* suffered from was because of the dependency problem that zope.i18n suffered from. Given they have a solution without the dependency problems, maybe we should use it? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On 11/28/07, Martijn Pieters <mj@zopatista.com> wrote:
On Nov 28, 2007 12:29 AM, Nathan Yergler <nathan@yergler.net> wrote:
If you follow the wink, http://www.unicode.org/cldr/repository_access.html has the details to the files. Currently the latest is at http://unicode.org/Public/cldr/1.5.0/core.zip.
Zope at this point still uses LDML 1.0 whereas the latest version is LDML 1.5.
Upon casual inspection of the files it seems their basic structure is still the same, though more careful inspection is required.
I've been avoiding even less interesting work this afternoon, taking a look at this. I started by dumping the new files into the data directory and just running the tests. As expected, things blew up in a really spectacular manner.
After some wrangling I've discovered that in the newer versions of the CLDR dataset some of the information previously contained in the locale files (such as weekend start/end, etc), is now in located in a "supplemental" file. While this makes a certain amount of sense (it's tied to territories, not really languages), it does mean that the information needed for a Locale is no longer self-contained in a single XML file.
So unfortunately it's going to require some more work to fix up the loader; I'll probably create a branch to work on this some...
Has anyone looked at Babel (http://babel.edgewall.org/)? It includes a python interface to CLDR, which if usable would let us off the hook of maintaining such an interface ourselves.
I've used Babel for some basic catalog manipulation, but didn't realize it did CLDR, too. I'll take a look at it.
-- Martijn Pieters
On Tuesday 27 November 2007, Nathan Yergler wrote:
So unfortunately it's going to require some more work to fix up the loader; I'll probably create a branch to work on this some...
Great! Let me know if you have questions. I have once updated the XML definitions and it was not too bad. It's pretty mechanical. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
participants (7)
-
Chris Withers -
Hanno Schlichting -
Martijn Pieters -
Nathan Yergler -
Philipp von Weitershausen -
Stephan Richter -
Tres Seaver