Zope supports the export of content either through Python pickles (.zexp) or as XML. Unfortunatly the XML export/import has several problems: - the generated XML is pretty much too low-level to use it e.g. for migration issues - it was broken multiple times in the past, it still has problems - it has no maintainer, nobody wants to touch it without gloves I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. We don't loose any functionality since .zexp is working fine. I don't know of any active projects/code that really uses XML export/import. Comments? Andreas
Hello, Em Sexta, 24 de Março de 2006 15:44, escreveu:
I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. We don't loose any functionality since .zexp is working fine. I don't know of any active projects/code that really uses XML export/import.
I've exported to XML and edited the generated file by hand as a quick way to successfully import a site to another Zope instance, when I was getting import errors when using the .zexp. I'm not sure I'd be able to do that with the .zexp file. But then again, it was about the only time I really found a use for the XML export/import. Still, it was nice to have to have the possibility at the time. Nuno
Andreas Jung wrote:
Zope supports the export of content either through Python pickles (.zexp) or as XML. Unfortunatly the XML export/import has several problems:
- the generated XML is pretty much too low-level to use it e.g. for migration issues
I have been working on a Zope 2 project lately and have found XML export to be extremely usable. Templates and other large strings are enclosed in CDATA sections, which make them surprisingly readable. We've had sucess writing XSLT templates to transform the pickle data into formats easily parsable for particular applications.
- it was broken multiple times in the past, it still has problems
How so?
- it has no maintainer, nobody wants to touch it without gloves
Is that any more true than for lots of other things?
I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. We don't loose any functionality since .zexp is working fine. I don't know of any active projects/code that really uses XML export/import.
Comments?
-1 As I said above, I've been finding it surprisingly useful lately. When I first wrote it, I viewed it as someowhat of an academic exercise, but I've come to realize that it is far more useful than I originally thought. This is more so now that XSLT is widely used and quite capable of transforming the pickles into useful forms. Note that, IMO, there isn't likely to be a more generally useful form because usefullness is generally determined by a particular application. XSLT + XML pickles together provide a general mechanism that can adapt to particular application needs. I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 24 Mar 2006, at 16:07, Jim Fulton wrote:
I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out.
We have used it to produce a large "backup" before, but consistently ran into unicode encode/decode errors. The only way I could get it to export was to do the nasty sitecustomize.py hack and setting the default encoding to UTF-8. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEJBmxRAx5nvEhZLIRAonMAJ4m7YeEH8jqBKJk+orpkNzx3u+N/QCfRww8 4dZO6mwmUj0DqsQ5MketkjU= =U5Nk -----END PGP SIGNATURE-----
Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 24 Mar 2006, at 16:07, Jim Fulton wrote:
I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out.
We have used it to produce a large "backup" before, but consistently ran into unicode encode/decode errors. The only way I could get it to export was to do the nasty sitecustomize.py hack and setting the default encoding to UTF-8.
Interesting. I imagine that this is fixible in a somewhat straightforward way, although it is probably tied up with the general encoding mess in Z2. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 24 Mar 2006, at 16:12, Jim Fulton wrote:
Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 24 Mar 2006, at 16:07, Jim Fulton wrote:
I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out. We have used it to produce a large "backup" before, but consistently ran into unicode encode/decode errors. The only way I could get it to export was to do the nasty sitecustomize.py hack and setting the default encoding to UTF-8.
Interesting. I imagine that this is fixible in a somewhat straightforward way, although it is probably tied up with the general encoding mess in Z2.
Yes, that's likely where the problem comes from. IMHO there is no reason to remove/deprecate it, but on the other hand it should not be advertised as a really user-friendly and easy way of exporting. It's still somewhat of a second class citizen when it comes to exporters. It might be in need of some warning stickers on the package ;) jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEJB3WRAx5nvEhZLIRAr3RAKCKomzIN1odFq/9fSfdWiet9m7r4ACguw/O EZRxyaVrtDb0NLCHO3RPp7U= =31YI -----END PGP SIGNATURE-----
Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 24 Mar 2006, at 16:12, Jim Fulton wrote:
Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 24 Mar 2006, at 16:07, Jim Fulton wrote:
I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out.
We have used it to produce a large "backup" before, but consistently ran into unicode encode/decode errors. The only way I could get it to export was to do the nasty sitecustomize.py hack and setting the default encoding to UTF-8.
Interesting. I imagine that this is fixible in a somewhat straightforward way, although it is probably tied up with the general encoding mess in Z2.
Yes, that's likely where the problem comes from. IMHO there is no reason to remove/deprecate it, but on the other hand it should not be advertised as a really user-friendly and easy way of exporting. It's still somewhat of a second class citizen when it comes to exporters. It might be in need of some warning stickers on the package ;)
Can you give an example of a first-class exporter? I'm not aware of a general solution. I guess there are solutions based on Archetypes. It should definately be advertized for what it is, but it is certainly much more than I realized in the past. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 24 Mar 2006, at 16:30, Jim Fulton wrote:
Interesting. I imagine that this is fixible in a somewhat straightforward way, although it is probably tied up with the general encoding mess in Z2. Yes, that's likely where the problem comes from. IMHO there is no reason to remove/deprecate it, but on the other hand it should not be advertised as a really user-friendly and easy way of exporting. It's still somewhat of a second class citizen when it comes to exporters. It might be in need of some warning stickers on the package ;)
Can you give an example of a first-class exporter? I'm not aware of a general solution. I guess there are solutions based on Archetypes. It should definately be advertized for what it is, but it is certainly much more than I realized in the past.
That was just a metaphor. For me, only the standard binary zexp exporter qualifies as "first class" exporter at the moment because it "just works". Apart from esoteric use cases like importing data into older ZODB implementations, which is highly questionable IMHO. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEJB+FRAx5nvEhZLIRAiUWAJ4807TgqX527sWgrZTTF7j0GoH2TQCfZDXj 0MrD1vpu79ERWnBCwu86NbY= =fpcK -----END PGP SIGNATURE-----
On Fri, 24 Mar 2006, Jim Fulton wrote:
Andreas Jung wrote:
I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12. We don't loose any functionality since .zexp is working fine. I don't know of any active projects/code that really uses XML export/import.
Comments?
I filed a bug report (2051: xml import export fail) which was triggered by the need to move material from a recent zope (2.9) to an older zope (2.7.4) where the zexp format was incompatible. Eventually, I was able to make the XML work.
--On 24. März 2006 11:07:06 -0500 Jim Fulton <jim@zope.com> wrote:
Andreas Jung wrote:
Zope supports the export of content either through Python pickles (.zexp) or as XML. Unfortunatly the XML export/import has several problems:
- the generated XML is pretty much too low-level to use it e.g. for migration issues
I have been working on a Zope 2 project lately and have found XML export to be extremely usable. Templates and other large strings are enclosed in CDATA sections, which make them surprisingly readable.
We've had sucess writing XSLT templates to transform the pickle data into formats easily parsable for particular applications.
That's a good argument :-)
- it was broken multiple times in the past, it still has problems
How so?
XML export did not work properly over some major Zope release (I think until Dieter fixed it). And there is at least one unicode related bug report pending (when I read such reports I am thinking about the following options: a) ignore the report and b) who the hell wrote the code and could fix it).
- it has no maintainer, nobody wants to touch it without gloves
Is that any more true than for lots of other things?
Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-)
As I said above, I've been finding it surprisingly useful lately.
When I first wrote it, I viewed it as someowhat of an academic exercise, but I've come to realize that it is far more useful than I originally thought.
I still have the impression that the implementation was made an academic excercie rather than something stable for production (.zexp works fine for production).
This is more so now that XSLT is widely used and quite capable of transforming the pickles into useful forms.
Note that, IMO, there isn't likely to be a more generally useful form because usefullness is generally determined by a particular application. XSLT + XML pickles together provide a general mechanism that can adapt to particular application needs.
In this case it has to be fixed if it should be used a general mechanism.
I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out.
Reasonable export/import should happen on a higher level. This low-level XML import/export is not really helpful when you want to migrate data between applications and frameworks. E.g. on the Plone/Archetypes level we have some mechanisms to export/import data defined through schemas (possibly we have something in Zope 3 (at least I once wrote a similar solution for Zope 3). The export/import mechanism is basically for moving data from one Zope instance to another instance. It should work in a reliable way and should not try to solve issues that should be solved on another level. To bring it to a point: further Zope 2 versions should get rid of buggy and esoteric features and apparently don't work for lots of ppl resulting in frustration and a bad image for Zope. Andreas ----------------------------------------------------------------------- - Andreas Jung ZOPYX Ltd. & Co KG - - E-mail: info@zopyx.com Web: www.zopyx.com, www.zopyx.de - -----------------------------------------------------------------------
Andreas Jung wrote:
...
Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-)
I don't see this as primarily a duplicate feature. I find the export aspect to be most compelling. Although, as others have pointed out, the openness of the XML format can be very useful for moving data between Zopes. Small data transformations (e.g. adjusting a class name) are possible and easy.
As I said above, I've been finding it surprisingly useful lately.
When I first wrote it, I viewed it as someowhat of an academic exercise, but I've come to realize that it is far more useful than I originally thought.
I still have the impression that the implementation was made an academic excercie rather than something stable for production (.zexp works fine for production).
It was an exercise. In some ways, it was ahead of it's time, as it is much more useful in the presense of XSLT. OTOH, there wasn't any intention to make it any lower quality than the binary format.
This is more so now that XSLT is widely used and quite capable of transforming the pickles into useful forms.
Note that, IMO, there isn't likely to be a more generally useful form because usefullness is generally determined by a particular application. XSLT + XML pickles together provide a general mechanism that can adapt to particular application needs.
In this case it has to be fixed if it should be used a general mechanism.
Sure
I think the XML export is a facility that is and should be advertized as a legitimate escape hatch for data kept in Zope. People really shouldn't feel afraid of putting data in Zope/ZODB as there really is a useful way to get it out.
Reasonable export/import should happen on a higher level. This low-level XML import/export is not really helpful when you want to migrate data between applications and frameworks.
I don't agree, if you are willing to use XSLT to extract the bits you want.
E.g. on the Plone/Archetypes level we have some mechanisms to export/import data defined through schemas
That's a good point. For Archetypes-based apps, this might be the way to go. I think that xml-pickle export is a reasonable approach for other apps.
(possibly we have something in Zope 3 (at least I once wrote a similar solution for Zope 3). The export/import mechanism is basically for moving data from one Zope instance to another instance. It should work in a reliable way and should not try to solve issues that should be solved on another level.
The unfinished fssync tool in Zope 3 is intended for this use case. It to uses an improved xmlpickle as a fallback. It would be nice to finish this, however, I can easily XSLT gun slingers prefering something like xml export. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
Andreas Jung wrote:
<snip>
E.g. on the Plone/Archetypes level
we have some mechanisms to export/import data defined through schemas
That's a good point. For Archetypes-based apps, this might be the way to go. I think that xml-pickle export is a reasonable approach for other apps.
The XML export format is conceptually like the "system view" XML mapping defined by JSR-170[1], as opposed to the "document view" mapping, which is more like the CMFSetup content export (and related Archetypes / Marshall ) formats.
(possibly we have something in Zope 3 (at least I once wrote a similar solution for Zope 3). The export/import mechanism is basically for moving data from one Zope instance to another instance. It should work in a reliable way and should not try to solve issues that should be solved on another level.
The unfinished fssync tool in Zope 3 is intended for this use case. It to uses an improved xmlpickle as a fallback. It would be nice to finish this, however, I can easily XSLT gun slingers prefering something like xml export.
[1] http://jcp.org/aboutJava/communityprocess/final/jsr170/index.html Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEJGMc+gerLs4ltQ4RArckAJ43rneghtA6TjyolVXJQq71BZ3uTwCfYXMk +rZYS14HXLkZZ/exnN+FCII= =ghB3 -----END PGP SIGNATURE-----
On Friday 24 March 2006 17:33, Andreas Jung wrote:
- it has no maintainer, nobody wants to touch it without gloves
Is that any more true than for lots of other things?
Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-)
We have been using XML export/import with ERP5[1] in many production systems for a long time. Only this feature allows us to understand changes among different versions in a human readable format, and even to edit the contents by ordinary text editors. Without this feature, it is nearly impossible to work on large systems in a distributed way. XML export/import is so valuable to make a bridge between TTW development and distributed development with the technology Business Template[2,3]. To make the feature more convenient, we have locally developed some monkey patches[4,5] to Zope so that: - the output is more stable (i.e. the order of tags and the reference numbers are more stable) to make "diff" usable for human - unicode objects work fine, assuming that data should be encoded in UTF-8 We would be happy to submit our patches, if you think they are generally useful. [1] http://www.erp5.org [2] http://wiki.erp5.org/HowToCreateBusinessTemplates [3] http://wiki.erp5.org/HowToCreateBusinessTemplates [4] http://cvs.erp5.org/ERP5Type/patches/XMLExportImport.py [5] http://cvs.erp5.org/ERP5Type/patches/ppml.py YO -- Yoshinori Okuji, Nexedi CTO Nexedi: Consulting and Development of Free / Open Source Software http://www.nexedi.com ERP5: Full Featured High End Open Source ERP http://www.erp5.com ERP5 Wiki: Developer Zone for ERP5 Community http://wiki.erp5.org
--On 24. März 2006 18:09:36 +0100 Yoshinori Okuji <yo@nexedi.com> wrote:
On Friday 24 March 2006 17:33, Andreas Jung wrote:
- it has no maintainer, nobody wants to touch it without gloves
Is that any more true than for lots of other things?
Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-)
We have been using XML export/import with ERP5[1] in many production systems for a long time. Only this feature allows us to understand changes among different versions in a human readable format, and even to edit the contents by ordinary text editors. Without this feature, it is nearly impossible to work on large systems in a distributed way. XML export/import is so valuable to make a bridge between TTW development and distributed development with the technology Business Template[2,3].
As I said, I am open to arguments and this convinces that the XML export/import is useful for some ppl and should stay. I am not dogmatic on such issues but such a discussion help to wake people but a bit....
To make the feature more convenient, we have locally developed some monkey patches[4,5] to Zope so that:
- the output is more stable (i.e. the order of tags and the reference numbers are more stable) to make "diff" usable for human
- unicode objects work fine, assuming that data should be encoded in UTF-8
We would be happy to submit our patches, if you think they are generally useful.
Of course we do appreciate any kind of patches as long they make sense and have reasonable test to document the intended behavior. In general patches are always welcome :-) Andreas
Yoshinori Okuji wrote:
On Friday 24 March 2006 17:33, Andreas Jung wrote:
- it has no maintainer, nobody wants to touch it without gloves
Is that any more true than for lots of other things?
Right, but for duplicate functionality that is not widely used and that I consider buggy it is legitimate to propose the deprecation. But as usually I am open to good arguments :-)
We have been using XML export/import with ERP5[1] in many production systems for a long time. Only this feature allows us to understand changes among different versions in a human readable format, and even to edit the contents by ordinary text editors. Without this feature, it is nearly impossible to work on large systems in a distributed way. XML export/import is so valuable to make a bridge between TTW development and distributed development with the technology Business Template[2,3].
To make the feature more convenient, we have locally developed some monkey patches[4,5] to Zope so that:
- the output is more stable (i.e. the order of tags and the reference numbers are more stable) to make "diff" usable for human
I did quite a bit of work along these lines for Zope 3. Perhaps we can find a way to share the Z2 and Z3 code.
- unicode objects work fine, assuming that data should be encoded in UTF-8
I think the Z3 xml pickle code handles Unicode too.
We would be happy to submit our patches, if you think they are generally useful.
Perhaps first, we should try the Zope 3 version of xml pickle. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On 3/24/06, Jim Fulton <jim@zope.com> wrote:
We've had sucess writing XSLT templates to transform the pickle data into formats easily parsable for particular applications.
As part of a recent task (likely the same one Jim's referring to here!), I transformed the XML export into another XML pickle, but for a simpler data structure that could be unpickled without dealing with the application classes. The CDATA marked sections that held the templates aren't useful for XML tools, but they do make it somewhat easier to read the pickles directly (valuable when trying to figure out what a transform is starting from!). -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Don't let schooling interfere with your education." -- Mark Twain
--On 24. März 2006 11:07:06 -0500 Jim Fulton <jim@zope.com> wrote:
- it was broken multiple times in the past, it still has problems
How so?
The import functionality seem to be broken since at least Zope 2.7. I created a DTML method exported it and _tried_ to reimport it. Zope 2.7 throws a BadPickleGet, 12 exception, Zope 2.8 throws BadPickleGet, 13 and Zope 2.9 raises the described UnicodeDecodeError. I don't expect that the import functionality works for even more complex objects. So I consider the whole functionality as totally broken. The generated XML might be useful to perform any processing outside Zope but using it for re-importing it into another Zope systems definitely does _not_ work. So if the functionality should remain in Zope then it should be fixed for Zope 2.10 lately. Andreas
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote:
--On 24. März 2006 11:07:06 -0500 Jim Fulton <jim@zope.com> wrote:
- it was broken multiple times in the past, it still has problems
How so?
The import functionality seem to be broken since at least Zope 2.7.
I created a DTML method exported it and _tried_ to reimport it.
Zope 2.7 throws a BadPickleGet, 12 exception,
I just exported the stock 'standard_error_message' as XML and re-imported it successfully in a brand new Zope-2_7-branch instance.
Zope 2.8 throws> BadPickleGet, 13
Likewise, my import succeeded in a Zope-2_8-instance, both with the XML file exported from 2.7 and with one created from 2.8.
Zope 2.9 raises the described UnicodeDecodeError.
My Zope-2_9-branch instance was successfully able to import the XML file from 2.7, but raises a UnicodeDecode error on the file it produces, as well as the one created from 2.8. The stack trace here makes it seem as though Python 2.4.2's expat parser is at fault: it is introducing non-ASCII characters into the stream where none exist in the source: (Pdb) u /home/tseaver/projects/Zope-CVS/Zope-2_9-branch/lib/python/OFS/XMLExportImport.py(114)importXML() - -> r=p.Parse(data) (Pdb) print [x for x in data if ord(x) > 127] [] (Pdb) d
/home/tseaver/projects/Zope-CVS/Python-2.4.3c1/Modules/pyexpat.c(639)EndElement() - -> ("(N)", string_intern(self, name))) (Pdb) d
/home/tseaver/projects/Zope-CVS/Zope-2_9-branch/lib/python/Shared/DC/xml/xyap.py(58)unknown_endtag() - -> top = end[tag](self, tag, top) (Pdb) print [type(x) for x in top] [<type 'unicode'>, <type 'dict'>, <type 'unicode'>, <type 'str'>] (Pdb) print [x for x in top[-1] if ord(x) > 127] ['\xcb'] (Pdb) print [chr(ord(x)) for x in top[-1][10:25]] ['U', '\x03', 'r', 'a', 'w', 'T', '\xcb', '\x04', '\x00', '\x00', '<', 'd', 't', 'm', 'l'] That output seems to me to indicate that the start of the CDATA section is the problem.
I don't expect that the import functionality works for even more complex objects. So I consider the whole functionality as totally broken. The generated XML might be useful to perform any processing outside Zope but using it for re-importing it into another Zope systems definitely does _not_ work. So if the functionality should remain in Zope then it should be fixed for Zope 2.10 lately.
I've just tried again, running with Python 2.4.3c1, because several of the changelog entries for the release suggest that unicode / encoding / expat stuff may have been fixed: the decode error is still present. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEJZsA+gerLs4ltQ4RArirAKC+a8k1GBTrujZ2uu5hygxnkSk9HACeLiAK n+8RTUlzAnsdBAkTDQ730mk= =6CcM -----END PGP SIGNATURE-----
On Saturday 25 March 2006 15:56, Andreas Jung wrote:
Zope 2.7 throws a BadPickleGet, 12 exception, Zope 2.8 throws BadPickleGet, 13 and Zope 2.9 raises the described UnicodeDecodeError. I don't expect that the import functionality works for even more complex objects. So I consider the whole functionality as totally broken. The generated XML might be useful to perform any processing outside Zope but using it for re-importing it into another Zope systems definitely does _not_ work. So if the functionality should remain in Zope then it should be fixed for Zope 2.10 lately.
Here is a quick patch for this problem (against 2.9.1). There were two different problems: - the id attributes were not generated, because the conditional was reverse. - unlike xmllib, expat always returns Unicode data, so simply concatenating binary values generates Unicode objects with non-ascii characters. For the latter problem, I'm not sure if my patch is enough. But this patch works with a simple dtml export/import. YO -- Yoshinori Okuji, Nexedi CTO Nexedi: Consulting and Development of Free / Open Source Software http://www.nexedi.com ERP5: Full Featured High End Open Source ERP http://www.erp5.com ERP5 Wiki: Developer Zone for ERP5 Community http://wiki.erp5.org
--On 25. März 2006 21:40:48 +0100 Yoshinori Okuji <yo@nexedi.com> wrote:
On Saturday 25 March 2006 15:56, Andreas Jung wrote:
Zope 2.7 throws a BadPickleGet, 12 exception, Zope 2.8 throws BadPickleGet, 13 and Zope 2.9 raises the described UnicodeDecodeError. I don't expect that the import functionality works for even more complex objects. So I consider the whole functionality as totally broken. The generated XML might be useful to perform any processing outside Zope but using it for re-importing it into another Zope systems definitely does _not_ work. So if the functionality should remain in Zope then it should be fixed for Zope 2.10 lately.
Here is a quick patch for this problem (against 2.9.1). There were two different problems:
- the id attributes were not generated, because the conditional was reverse.
- unlike xmllib, expat always returns Unicode data, so simply concatenating binary values generates Unicode objects with non-ascii characters.
Thanks for the patch (commited for Zope 2.8, 2.9. trunk). This solves at least the import problems on my side. Unfortunately there are no tests for export/import so fixing issues is like flying blindfolded. -aj
--On 26. März 2006 09:58:07 +0100 Andreas Jung <lists@zopyx.com> wrote:
Thanks for the patch (commited for Zope 2.8, 2.9. trunk). This solves at least the import problems on my side. Unfortunately there are no tests for export/import so fixing issues is like flying blindfolded.
Although the patch does not help to export a complete Plone site as XML. This fails with: Traceback (innermost last): * Module ZPublisher.Publish, line 113, in publish * Module ZPublisher.mapply, line 88, in mapply * Module ZPublisher.Publish, line 40, in call_object * Module OFS.ObjectManager, line 547, in manage_exportObject * Module OFS.XMLExportImport, line 58, in exportXML * Module OFS.XMLExportImport, line 33, in XMLrecord * Module Shared.DC.xml.ppml, line 263, in load * Module pickle, line 872, in load * Module Shared.DC.xml.ppml, line 418, in load_binput AttributeError: 'unicode' object has no attribute 'id' -aj
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Yoshinori Okuji wrote:
On Saturday 25 March 2006 15:56, Andreas Jung wrote:
Zope 2.7 throws a BadPickleGet, 12 exception, Zope 2.8 throws BadPickleGet, 13 and Zope 2.9 raises the described UnicodeDecodeError. I don't expect that the import functionality works for even more complex objects. So I consider the whole functionality as totally broken. The generated XML might be useful to perform any processing outside Zope but using it for re-importing it into another Zope systems definitely does _not_ work. So if the functionality should remain in Zope then it should be fixed for Zope 2.10 lately.
Here is a quick patch for this problem (against 2.9.1). There were two different problems:
- the id attributes were not generated, because the conditional was reverse.
- unlike xmllib, expat always returns Unicode data, so simply concatenating binary values generates Unicode objects with non-ascii characters.
For the latter problem, I'm not sure if my patch is enough. But this patch works with a simple dtml export/import.
I have extended your patch with a unit test (there weren't any for OFS.XMLExportImport), as well as modifying XMLExportImport to open files in binary mode. Before your patch, my tests fail, with the same Unicode error. Afterwards, they pass. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEJuQf+gerLs4ltQ4RAjaRAJ9tT6Xv0yIJpPx2SixFSKdghtVcrwCfU6Yk VrofPZTcbr1RzTu5knxSwJs= =mRAe -----END PGP SIGNATURE-----
Andreas Jung wrote at 2006-3-24 16:44 +0100:
Zope supports the export of content either through Python pickles (.zexp) or as XML. Unfortunatly the XML export/import has several problems:
- the generated XML is pretty much too low-level to use it e.g. for migration issues
- it was broken multiple times in the past, it still has problems
There was a single error. True, for a longer period -- until Tres provided the tiny fix....
... I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12.
What will remain of Zope in the longer run? -- Dieter
--On 24. März 2006 21:05:12 +0100 Dieter Maurer <dieter@handshake.de> wrote:
... I propose to deprecate XML export/import for Zope 2.10 and to remove it in Zope 2.12.
What will remain of Zope in the longer run?
The good things that work :-) In the past with every major release some problems with old and scary components came up. In this situation you have basically three options: - ignore such problems - fix the code - rewrite the code I have a bad feeling (as release manager) about ignoring problems and releasing stuff. Fixing and rewriting code often requires a deep understanding of the concepts, the implementation especially if there is no documentation and no tests. In many cases Jim is the original author or knows at least of the concepts since many of the original Zope developers are no longer with ZC or even part of the Zope community. So as a consequence in the long term we should get rid of stuff that is either broken or considered obsolete by the majority of the developers and for the majority of the developers. Figuring out what the majority is in a particular case is of course always a problem.... Returning with the discussion to ZClasses (also off-topic in this XML export/import thread): they blocked the 2.8.0 release for a long time. If there is an unimportant second-class or third-class component of Zope blocking a major release in the future than it should be subject to be deprecated (or even removed if necessary). Andreas ----------------------------------------------------------------------- - Andreas Jung ZOPYX Ltd. & Co KG - - E-mail: info@zopyx.com Web: www.zopyx.com, www.zopyx.de - -----------------------------------------------------------------------
participants (9)
-
Andreas Jung -
Dennis Allison -
Dieter Maurer -
Fred Drake -
Jens Vagelpohl -
Jim Fulton -
Nuno Maltez -
Tres Seaver -
Yoshinori Okuji