zope.index 3.5.2 broken
Hi, the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly: File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result Expected: True Got: False Thoughts? Andreas
On 02.08.09 20:48, Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result Expected: True Got: False
Obviously another Python 2.5/.6 incompatiblity. zope.index 3.5.1 works fine with Python 2.4-2.6 - obviously changes to okascore.c introduced with zope.index 3.5.2 are to blame?! Andreas
On 02.08.09 21:00, Andreas Jung wrote:
On 02.08.09 20:48, Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result Expected: True Got: False
Obviously another Python 2.5/.6 incompatiblity. zope.index 3.5.1 works fine with Python 2.4-2.6 - obviously changes to okascore.c introduced with zope.index 3.5.2 are to blame?!
The diff between 3.5.1 and 3.5.2 is pretty long and substantial. I doubt that such a major change us ok as a bugfix release. I should have become a new major release. I suggest that the current trunk should be released as 3.6.0 and the current 3.5.1 released should be copied to tags/3.5.3 and re-released. Andreas
On 8/2/09 3:07 PM, Andreas Jung wrote:
On 02.08.09 21:00, Andreas Jung wrote:
On 02.08.09 20:48, Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result Expected: True Got: False
Obviously another Python 2.5/.6 incompatiblity. zope.index 3.5.1 works fine with Python 2.4-2.6 - obviously changes to okascore.c introduced with zope.index 3.5.2 are to blame?!
The diff between 3.5.1 and 3.5.2 is pretty long and substantial. I doubt that such a major change us ok as a bugfix release. I should have become a new major release. I suggest that the current trunk should be released as 3.6.0 and the current 3.5.1 released should be copied to tags/3.5.3 and re-released.
I think your test failure is related to a 64-bit issue. Shane has some fixes on the trunk that may solve this. That said, why does Zope 2.1.2 use the zope.index textindex? The Zope 2 implementation is still very superior. Before we started fixing it, there were places that the zope.index text index didn't actually work at all Where it did work, it worked very, very slowly. We've been trying to improve it incrementally (mostly by backporting all the fixes that have accreted to the Zope 2 implementation). I'll try to pin this down and make a 3.6.0 release, but Zope 2 shouldn't be using this index. - C
On 8/2/09 4:41 PM, Chris McDonough wrote:
On 8/2/09 3:07 PM, Andreas Jung wrote:
On 02.08.09 21:00, Andreas Jung wrote:
On 02.08.09 20:48, Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result Expected: True Got: False
Obviously another Python 2.5/.6 incompatiblity. zope.index 3.5.1 works fine with Python 2.4-2.6 - obviously changes to okascore.c introduced with zope.index 3.5.2 are to blame?!
The diff between 3.5.1 and 3.5.2 is pretty long and substantial. I doubt that such a major change us ok as a bugfix release. I should have become a new major release. I suggest that the current trunk should be released as 3.6.0 and the current 3.5.1 released should be copied to tags/3.5.3 and re-released.
I think your test failure is related to a 64-bit issue. Shane has some fixes on the trunk that may solve this.
That said, why does Zope 2.1.2 use the zope.index textindex? The Zope 2 implementation is still very superior.
Before we started fixing it, there were places that the zope.index text index didn't actually work at all Where it did work, it worked very, very slowly. We've been trying to improve it incrementally (mostly by backporting all the fixes that have accreted to the Zope 2 implementation).
I'll try to pin this down and make a 3.6.0 release, but Zope 2 shouldn't be using this index.
The tests now pass on my 32-bit machine. Can someone with a 64-bit system try them? - C
On Sun, Aug 2, 2009 at 10:41 PM, Chris McDonough<chrism@plope.com> wrote:
That said, why does Zope 2.1.2 use the zope.index textindex? The Zope 2 implementation is still very superior.
The current policy for the test runs is to test both all direct dependencies and all transitive dependencies including all test dependencies on all levels. We have spent quite some time reducing the "real" dependencies already, but the test dependencies are still in a somewhat worse state. So in this case zope.app.zptpage has a code dependency on zope.index. And zope.app.zptpage is a test dependency of a number of packages, including zope.traversing, zope.app.exception and zope.app.publication. I think it might be time to remove the test extras again for "reusable zope.*" packages and declare them as real dependencies. This would allow us to actually see them more easily and get truly embarrassed - a good way of spurring motivation to fix the mess ;-) Hanno
On 8/2/09 5:23 PM, Hanno Schlichting wrote:
On Sun, Aug 2, 2009 at 10:41 PM, Chris McDonough<chrism@plope.com> wrote:
That said, why does Zope 2.1.2 use the zope.index textindex? The Zope 2 implementation is still very superior.
The current policy for the test runs is to test both all direct dependencies and all transitive dependencies including all test dependencies on all levels. We have spent quite some time reducing the "real" dependencies already, but the test dependencies are still in a somewhat worse state.
So in this case zope.app.zptpage has a code dependency on zope.index. And zope.app.zptpage is a test dependency of a number of packages, including zope.traversing, zope.app.exception and zope.app.publication.
OK. There's no way any release of Zope 2 should depend on zope.index, at least until its index implementations are as good as the ones in Zope 2 itself. I don't know if in the short term (for a release) that means that the Zope 2 test regime needs to change, or if the dependencies need to be fixed, or what. In the medium term, we should fix the dependencies obviously. Another immediate thing to do would just be to pin the zope.index version to 3.5.1, given that it doesn't matter one way or another whether these tests pass for purposes of a Zope 2 release.
I think it might be time to remove the test extras again for "reusable zope.*" packages and declare them as real dependencies. This would allow us to actually see them more easily and get truly embarrassed - a good way of spurring motivation to fix the mess ;-)
Oh good, I might win that beer from Martijn after all. ;-) - C
On Sun, Aug 2, 2009 at 11:35 PM, Chris McDonough<chrism@plope.com> wrote:
There's no way any release of Zope 2 should depend on zope.index, at least until its index implementations are as good as the ones in Zope 2 itself. I don't know if in the short term (for a release) that means that the Zope 2 test regime needs to change, or if the dependencies need to be fixed, or what. In the medium term, we should fix the dependencies obviously.
Note that we only depend on zope.index via a test dependency of one of our dependencies, so anyone downloading and installing the Zope2 distribution won't actually pull in zope.index. Even running the tests for Zope2 itself won't need it. Only if you want to run the tests for all dependencies of Zope2, like we do in our core development buildout, you actually get it. Hanno
Hey, Andreas Jung wrote: [snip]
The diff between 3.5.1 and 3.5.2 is pretty long and substantial. I doubt that such a major change us ok as a bugfix release. I should have become a new major release.
From what you say, agreed. Regards, Martijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result Expected: True Got: False
Thoughts?
Shoot that test: it sucks both as a test and as documentation. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKdgL4+gerLs4ltQ4RAntBAJ0eC3YcZxxDh8qw2hIRSFK2UA1Q1wCghbn+ +eMTXYnbaDt0cmHAbTyYgmY= =uVOK -----END PGP SIGNATURE-----
On 8/2/09 5:19 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result Expected: True Got: False
Thoughts?
Shoot that test: it sucks both as a test and as documentation.
Already shot on the trunk (by Shane). - C
On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result
I'm assuming items() returns a plain Python dictionary with string keys. Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed). Add a sorted() on both sides and the test should pass. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
Marius Gedminas wrote:
On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result
I'm assuming items() returns a plain Python dictionary with string keys.
Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed).
Add a sorted() on both sides and the test should pass.
Actually, those tests were plain insane and I've fixed them on the trunk. I intend to make a new zope.index release today. How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability. Furthermore, the C code was producing incorrect results due to an unsafe optimization that accidentally took 32 bits of a Python float and pretended it was an integer. The breakage only became obvious when we started taking 64 bits instead, leading to completely different results. Shane
On 8/3/09 1:07 PM, Shane Hathaway wrote:
Marius Gedminas wrote:
On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result
I'm assuming items() returns a plain Python dictionary with string keys.
Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed).
Add a sorted() on both sides and the test should pass.
Actually, those tests were plain insane and I've fixed them on the trunk. I intend to make a new zope.index release today.
How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability.
/me hangs head in shame. - C
On 03.08.09 20:15, Chris McDonough wrote:
On 8/3/09 1:07 PM, Shane Hathaway wrote:
Marius Gedminas wrote:
On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote:
Hi,
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
File "/home/ajung/.buildout/eggs/zope.index-3.5.2-py2.6-linux-x86_64.egg/zope/index/text/tests/../textindex.txt", line 143, in textindex.txt Failed example: [(k, "%.4f" % v) for (k, v) in index2.apply("Zorro").items()] == result
I'm assuming items() returns a plain Python dictionary with string keys.
Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed).
Add a sorted() on both sides and the test should pass.
Actually, those tests were plain insane and I've fixed them on the trunk. I intend to make a new zope.index release today.
How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability.
Is there a buildbot for the zope.* or ZTK packages testing them under Linux 32bit and 64 bit?
Andreas
* Andreas Jung <lists@zopyx.com> [2009-08-03 20:21]:
On 03.08.09 20:15, Chris McDonough wrote:
On 8/3/09 1:07 PM, Shane Hathaway wrote:
Marius Gedminas wrote:
On Sun, Aug 02, 2009 at 08:48:24PM +0200, Andreas Jung wrote:
the doctests for zope.index 3.5.2 - as used in Zope 2.12 - fail badly:
Python's string hashes return different valuesfor half of all the strings on 64-bit machines. This influences the ordering of dictionary keys and some other things too (such as the sequence of random numbers you get if you use a string as the seed).
Is there a buildbot for the zope.* or ZTK packages testing them under Linux 32bit and 64 bit?
AFAICS all three buildbots listed on http://docs.zope.org/zopetoolkit/process/tools.html do. Wolfgang
Chris McDonough wrote:
On 8/3/09 1:07 PM, Shane Hathaway wrote:
How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability.
/me hangs head in shame.
That was you? Well, it could have been worse. :-) Shane
On 8/3/09 2:34 PM, Shane Hathaway wrote:
Chris McDonough wrote:
On 8/3/09 1:07 PM, Shane Hathaway wrote:
How insane were these tests? Well, the author of the tests noticed that the C optimization produces different scores than the Python version, and compensated for that in a way that dramatically reduced readability.
/me hangs head in shame.
That was you? Well, it could have been worse. :-)
I didn't write the original tests but I did paper over the okascore result differences by making the tests compare true or false vs. the expected output. Ten lashes. - C
participants (8)
-
Andreas Jung -
Chris McDonough -
Hanno Schlichting -
Marius Gedminas -
Martijn Faassen -
Shane Hathaway -
Tres Seaver -
Wolfgang Schnerring