- Zope - Zope lists

Re: [Zope] How to convert Zope instance charset?
by Daniel Dekany 25 Apr '05

25 Apr '05

Monday, April 25, 2005, 5:34:04 AM, Andreas Jung wrote: > One last note from myside. I have experiences with unicode since over > 7 years while working with multilingual documents in the e-publishing > business. It is good practice to perform *any* unicode related work > *only* on unicode datatypes (Python unicode strings!!!) > and *not* some byte-encoded unicode strings as utf8 or whatever. These > encodings should only be used on the output level when presenting > unicode data to the user - either through-the-web, as export format > etc. This is a strong advice you should follow. As someone who works often with Java I absolutely agree with it. Just I don't know how to do it with Zope/Plone/other 3rd party products (not written by me), since they use not unicode strings. I don't know, maybe it can be specified for Python that it uses unicode for plain strings as well, but anybody has successfully done that with Zope? Like, what about the C parts of Zope then? > -aj -- Best regards, Daniel Dekany

2 1

Re: [Zope] How to convert Zope instance charset?
by Daniel Dekany 25 Apr '05

25 Apr '05

Sunday, April 24, 2005, 7:22:42 PM, Andreas Jung wrote: > > > --On Sonntag, 24. April 2005 18:34 Uhr +0200 Daniel Dekany > <ddekany(a)freemail.hu> wrote: > >> >> Maybe *you* don't get the point. Python has a "virtual machine level" >> setting that specifies the locale and encoding (the charset). You can >> set it for example like: locale.setlocale('hu_HU', 'ISO-8859-2'). And >> although there is no charset information attached to strings, >> locale.strcoll and such will assume that the string is in the encoding >> specified globally like above, right? All the strings (which is not an >> unicode string) is assumed to use that encoding. It seems to me that it >> works like that until I specify 'UTF-8' in the locale, in which case it >> goes mad. > > > I am very much aware of the issue (btw. it was me who integrated > sequence.sort()). You are aware of it, so why do you dispute if UTF-8 is supported (read: can be used in practice)? Just how on the earth should anybody solve this "on the application level"? Should everybody use only his own products (that convert everything to "unicode" string)? Also everybody should patch ZCatalog and similar core components which also sorts non-unicode strings? Simply said, people can't use Zope with UTF-8 in practice (while I guess they can with ISO-8859-x, right?), with my original words "Since Python/Zope/etc practically doesn't support utf-8". And I didn't meant to hurt or upbraid Zope fans with it at all, I just said it as a fact in the middle of a sentence, and then what happens... (Why Plone guys use UTF-8 as default, I don't know, maybe they didn't realized it doesn't work for people who really utilize utf-8. I'm for too long in the business to be surprised. :) I think I will ask them...) > And if you look carefully at the API of sequence.sort() > then you will see that there is already built-in support locale-aware > comparisons. Yeah, I tried to use that earlier, but as I said for many times here, it can't sort utf-8 encoded strings, despite that I have set the "global locale" to something.utf8, certainly because Python locale.strcoll can't. So at the end: You Can Not Use UTF-8 with Zope. Right? Soooo, then back to the original question: converting and utf-8 instance to ISO-8859-2 instance, any idea? (Or is it still the standpoint that I should not??? D-%) > -aj -- Best regards, Daniel Dekany

2 2

zope-2.8, i18n and global locales catalog
by Sascha Ottolski 24 Apr '05

24 Apr '05

Hi, does anybody know if and how I could build a global locales catalog in zope-2.8? I'm already through the i18n chapter in the zope-3 book, and think to know how to set one up on a per product level. I made a locales dir in my INSTANCE_HOME, but that doesn't seem to do the trick. However, I'm not sure if I might need some zcml-magic to use the i18n stuff globally and/or in a zope-2 product, and how I would have to do it. Having asked that, I hope I'm correct to assume that it is possible to use zope-3's i18n services in a zope-2.8 via Five... BTW, has the i18n service in zope3 something like I've seen in PlacelessTranslationServices, that would assist in collecting untranslated strings on the fly (PTS can append those strings to .missing files and/or present those strings via ZMI)? Thanks, Sascha

1 0

developer docs
by Milos Prudek 24 Apr '05

24 Apr '05

What are the best documents for creating Zope products? I know Zope Book very well and I write TTW scripts. I need to convert a bunch of ZClasses into Products. ZDG seems very outdated. Should I start with mxmEasyProduct and then read ZDG? Is there a book available in bookshops that is better than ZDG and contains all required information for building products? -- Milos Prudek

7 15

Re: [Zope] How to convert Zope instance charset?
by Daniel Dekany 24 Apr '05

24 Apr '05

Sunday, April 24, 2005, 6:05:42 PM, Andreas Jung wrote: > > > --On Sonntag, 24. April 2005 17:45 Uhr +0200 Daniel Dekany > <ddekany(a)freemail.hu> wrote: > >> Sunday, April 24, 2005, 4:22:10 PM, Andreas Jung wrote: >> First of all, in this thread I don't care whose mistake it is. My >> concern is if I can use Zope with UTF-8 (in fact, Plone) in reality or >> not. Assume that I'm using a few non-US-ASCII characters, and I want >> sometimes show things alphabetically sorted... >> > > You're not getting the point. As long as you handle with Python string > and not with unicode strings then there is no way in Zope deal correctly > with different kind of encodings...As I said...it is an application side > problem. Zope and Python provide you the tools to deal with UTF8 but > you need to solve such problems on in your application. That's my last > comment on this issue :-) Maybe *you* don't get the point. Python has a "virtual machine level" setting that specifies the locale and encoding (the charset). You can set it for example like: locale.setlocale('hu_HU', 'ISO-8859-2'). And although there is no charset information attached to strings, locale.strcoll and such will assume that the string is in the encoding specified globally like above, right? All the strings (which is not an unicode string) is assumed to use that encoding. It seems to me that it works like that until I specify 'UTF-8' in the locale, in which case it goes mad. And, to Max M., regarding patching sequence.sort: a) There is no guarantee that everything uses for sequence.sort. Some code may calls locale.strcoll directly and such, which can result in all sort of inconsistency. The fix could be done at the root of the problem, which is I belive strcoll. b) If the problem is in Zope (that I doubt) then it should be patched in Zope itself, not by everybody individually. That is, for Andreas Jung, if the locale.getlocale(locale.LC_COLLATE) indicates that the default charset is UTF-8, then it should be sorted like that. But again, I think it should be actually fixed in Python level (in locale.strcoll), and not on the Zope level. Anyway, I have already accepted earlier that while Zope will certainly work with ISO-8859-2 (that locale.strcoll handles correctly), it will not work with UTF-8. Hence, Zope doesn't work well with UTF-8, while it works with "older" charsets (it does without any extra effort, right?). So I just asked how to switch over ISO-8859-2, and then some start to tell that it works with UTF-8, and that it should be solved on application level(!!!)... > -aj -- Best regards, Daniel Dekany

2 1

Re: [Zope] How to convert Zope instance charset?
by Daniel Dekany 24 Apr '05

24 Apr '05

Sunday, April 24, 2005, 4:22:10 PM, Andreas Jung wrote: > > > --On Sonntag, 24. April 2005 16:03 Uhr +0200 Daniel Dekany > <ddekany(a)freemail.hu> wrote: > >> Sunday, April 24, 2005, 2:36:24 PM, Andreas Jung wrote: >> >>> --On Sonntag, 24. April 2005 14:18 Uhr +0200 Daniel Dekany >>> <ddekany(a)freemail.hu> wrote: >>> >>>> I have a Zope instance that uses utf-8 for everything. Since >>>> Python/Zope/etc practically doesn't support utf-8, >>> >>> Please explain in which sense Zope would not support utf-8. For your >>> information: >> >> It can't sort strings alphabetically *anywhere* (concretely: the >> accented letters will go to the end of the list -- I guess because 0x80 >> is mathematically greater than the code of the US-ASCII characters). > > This is neither a problem of Zope nor of Python! A Python string has no > notion an an encoding. The sort method can not smell the encoding. First of all, in this thread I don't care whose mistake it is. My concern is if I can use Zope with UTF-8 (in fact, Plone) in reality or not. Assume that I'm using a few non-US-ASCII characters, and I want sometimes show things alphabetically sorted... Then, of course if something wants to collate string for human reading, it will use locale.strcoll, which do consider charset and locale. That locale.strcoll is wrong with UTF-8, that's certainly the mistake of Python, right? > Instead use Python unicode strings and depend on the sorting order > defined by the Unicode standard. I take that advice, but unfortunately it's not about my Python code, but about other people's Python code. > This is an application-level problem but not a server-side problem. Zope itself gives a method for sorting strings: DocumentTemplate.sequence.sort. Many of the products relies on that for sorting. And that sorts UTF-8 incorrectly (I guess because locale.strcoll does it incorrectly). Also, ZCatalog sorts incorrectly (surely for the same reason), which is also the part of the standard Zope distribution. >>> Plone has UTF8 as default charset. >> >> Believe me, I really hope I'm wrong. So how could I achieve that strings >> are sorted correctly? If it works for someone, how? (I have locale >> hu_HU.UTF-8 in zope.conf, I have even printed >> locale.getlocale(locale.LC_COLLATE) from products and external methods, >> and it was hu_HU.UTF-8. Note that at least on Python level sorting with >> hu_HU.ISO-8859-2 works... so I hope it would work with Plone as well.) >> > > see above..Also the standard sort() methods of Python does not care about > your > locales (why should it)....strings are streams of bytes...nothing else... I know, and I have referred to locale.strcoll, which does care about encoding and locale. Seems many products use that (indirectly) when they want to sort something. > sort() accepts a user-defined comparison method of implement user-specific > sorting. Yes, but this doesn't help, unless I write an UTF-8 comparison method, and then find all sort() and locale.sort() calls in Zope, Plone, and in other products, and patch them all... > And there are also methods in Python "locale" module to perform > locale-dependent comparison. Which I can't get working with UTF-8, it puts non-US-ASCII letters at the end of the list. Somebody did? How? I'm all ears. I guess the Plone site should suddenly sort correctly then, at least on the places where the programmer of the Zope product was wise enough not to use raw sort(). > Once again: you must solve your problem on the application layer... (Anyway string collation is not an application level problem in principle. It is the same for a book store application and for a first person shooter, there is nothing application specific in it. If Python is not mature enough to take this task, that's a different question.) > Zope does not help you at this point because it can't. So however I formulate it, the end is that you *practically* can't use UTF-8 with Zope, unless you are using a language that doesn't use non-US-ASCII characters, in which case you don't utilize UTF-8. Hence, I said it is "not supported"... It doesn't mean that it is the mistake of Zope, it just means that you can't use it. So, back to the topic... Since UTF-8 is not working (it seems), how could I convert that already filled instance to use ISO-8859-2 instead of UTF-8? Some tool helps me in it done relatively easy? > -aj -- Best regards, Daniel Dekany

3 5

Re: [Zope] How to convert Zope instance charset?
by Daniel Dekany 24 Apr '05

24 Apr '05

Sunday, April 24, 2005, 2:36:24 PM, Andreas Jung wrote: > --On Sonntag, 24. April 2005 14:18 Uhr +0200 Daniel Dekany > <ddekany(a)freemail.hu> wrote: > >> I have a Zope instance that uses utf-8 for everything. Since >> Python/Zope/etc practically doesn't support utf-8, > > Please explain in which sense Zope would not support utf-8. For your > information: It can't sort strings alphabetically *anywhere* (concretely: the accented letters will go to the end of the list -- I guess because 0x80 is mathematically greater than the code of the US-ASCII characters). Kind of basic fundamental thing for a portal, or for text handling in general. I have asked here earlier how to solve it, but there was no answer that could be applied in practice (i.e. the answer was that I should write custom fixes for each individual products, and/or write a patch for Zope, and then maybe for Python... If so, this is equal of saying that UTF-8 is not supported yet). > Plone has UTF8 as default charset. Aha. Then this is why this Plone site I should fix/maintain used UTF-8 everywhere. I believed it was a bad decision of my predecessor. (But then this problem is even more mysterious for me: *if* it doesn't working (yet), then why did the Plone authors chose that?) > In general Zope does not care much about > encoded strings except for some conversions. Dealing with utf8 might be > tricky > in some cases but saying Zope does not support Utf-8 is wrong. Believe me, I really hope I'm wrong. So how could I achieve that strings are sorted correctly? If it works for someone, how? (I have locale hu_HU.UTF-8 in zope.conf, I have even printed locale.getlocale(locale.LC_COLLATE) from products and external methods, and it was hu_HU.UTF-8. Note that at least on Python level sorting with hu_HU.ISO-8859-2 works... so I hope it would work with Plone as well.) > -aj -- Best regards, Daniel Dekany

2 1

How to convert Zope instance charset?
by Daniel Dekany 24 Apr '05

24 Apr '05

I have a Zope instance that uses utf-8 for everything. Since Python/Zope/etc practically doesn't support utf-8, I would like to switch over to ISO-8859-2 (for everything). The problem is that this instance is a fat site that is on-line for several months, so I have to convert the strings stored inside the lot of already existing objects of the ZODB. Any idea how to do it, like an already existing tool for it? -- Best regards, Daniel Dekany

2 1

CMF 1.5.1 Released
by Jens Vagelpohl 24 Apr '05

24 Apr '05

Zope Corporation is pleased to announce the release of version 1.5.1 of the Zope Content Management Framework (CMF). This release marks the latest stable release in the CMF 1.5 series and should be suitable for production use. What is the CMF? The Zope Content Management Framework provides a set of services and content objects useful for building highly dynamic, content-oriented portal sites. As packaged, the CMF generates a site much like the Zope.org site. The CMF is intended to be easily customizable, in terms of both the types of content used and the policies and services it provides. Where do I get it? Download it from http://zope.org/Products/CMF/CMF-1.5.1 Points of interest include: - "Windows ZIP file", http://zope.org/Products/CMF/CMF-1.5.1/CMF-1.5.1.zip - "Unix tar/gzip archive", http://zope.org/Products/CMF/CMF-1.5.1/CMF-1.5.1.tar.gz - "Release notes", http://zope.org/Products/CMF/CMF-1.5.1/README.txt - "Change history", http://zope.org/Products/CMF/CMF-1.5.1/CHANGES.txt - "Installation instructions", http://zope.org/Products/CMF/CMF-1.5.1/INSTALL.txt Where do I go to learn more? The "CMF mailing list":mailto:zope-cmf@zope.org has many participants who are active in supporting the CMF. ...to report bugs? The "CMF Collector":http://zope.org/Collectors/CMF is ths place to report bugs (please search for existing reports of your issue first!) --------------- Jens Vagelpohl jens(a)zetwork.com Zetwork GmbH http://www.zetwork.com/

1 0

[ANN] Zope 2.8.0 beta 1 released
by Andreas Jung 24 Apr '05

24 Apr '05

Dear Zope Community, on behalf of Zope Corporation and all Zope 2 developers and contributors I am pleased to announce the release of Zope 2.8.0 b1. Zope 2.8.0 b1 can be downloaded from http://www.zope.org/Products/Zope/2.8.0b1 The release notes can be found at http://www.zope.org/Products/Zope/2.8.0b1/CHANGES.txt For information on using Python 2.4 with Zope 2.8: see doc/INSTALL.txt Every developer is strongly encouraged to test Zope 2.8 beta 1 *now* and not to wait until the final Zope 2.8 release...that's why we have beta releases :-) The quality and stability of the final release depends on *your* testing *now*. This release is a major step towards Zope 2.8.0 final (scheduled for the 3rd week of May 2005) and now includes for the first time the "Five" framework to make some Zope 3 technologies available within Zope 2. So you will be able to use Z3 technologies like interfaces, adapters, schemas and views within your Zope 2 installation and have hopefully the best from the Zope 2 and the Zope 3 world. Andreas Jung Zope 2 Release Manager

1 0