Summary of messages to the zope-tests list. Period Fri Jul 17 12:00:00 2009 UTC to Sat Jul 18 12:00:00 2009 UTC. There were 8 messages: 8 from Zope Tests. Test failures ------------- Subject: FAILED (failures=1) : Zope-2.12 Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:50:00 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012082.html Subject: FAILED (failures=1) : Zope-2.12-alltests Python-2.4.6 : Linux From: Zope Tests Date: Fri Jul 17 20:52:00 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012083.html Subject: FAILED (failures=7) : Zope-2.12-alltests Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:54:01 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012084.html Subject: FAILED (failures=1) : Zope-trunk Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:56:01 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012085.html Subject: FAILED (failures=7) : Zope-trunk-alltests Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:58:01 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012086.html Tests passed OK --------------- Subject: OK : Zope-2.10 Python-2.4.6 : Linux From: Zope Tests Date: Fri Jul 17 20:44:00 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012079.html Subject: OK : Zope-2.11 Python-2.4.6 : Linux From: Zope Tests Date: Fri Jul 17 20:46:00 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012080.html Subject: OK : Zope-2.12 Python-2.4.6 : Linux From: Zope Tests Date: Fri Jul 17 20:48:00 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012081.html
All failures appear to be due to the new box running Linux x86_64. Tests still pass fine on the Mac. Any experience with this? MemoryError in Acquisition? WTH? Stefan On 18.07.2009, at 14:00, Zope Tests Summarizer wrote:
Summary of messages to the zope-tests list. Period Fri Jul 17 12:00:00 2009 UTC to Sat Jul 18 12:00:00 2009 UTC. There were 8 messages: 8 from Zope Tests.
Test failures -------------
Subject: FAILED (failures=1) : Zope-2.12 Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:50:00 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012082.html
Subject: FAILED (failures=1) : Zope-2.12-alltests Python-2.4.6 : Linux From: Zope Tests Date: Fri Jul 17 20:52:00 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012083.html
Subject: FAILED (failures=7) : Zope-2.12-alltests Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:54:01 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012084.html
Subject: FAILED (failures=1) : Zope-trunk Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:56:01 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012085.html
Subject: FAILED (failures=7) : Zope-trunk-alltests Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:58:01 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012086.html
-- Stefan H. Holek stefan@epy.co.at
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Stefan H. Holek wrote:
All failures appear to be due to the new box running Linux x86_64. Tests still pass fine on the Mac.
Any experience with this? MemoryError in Acquisition? WTH?
I can reproduce the OFS.tests.testRange failures on my x86_64 box: $ uname -a Linux mred 2.6.27-14-generic #1 SMP Wed Apr 15 19:29:46 UTC \ 2009 x86_64 GNU/Linux $ cd /tmp $ svn co svn+ssh://svn.zope.org/repos/main/Zope/trunk Zope-trunk $ cd Zope-trunk $ /opt/Python-2.6.2/bin/python bootstrap.py ... $ bin/buildout ... $ bin/test --all ... Failure in test testMultipleRangesBigFileOutOfOrder (OFS.tests.testRanges.TestRequestRange) Traceback (most recent call last): File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 279, in run testMethod() File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py", line 332, in testMultipleRangesBigFileOutOfOrder (70000, 80001)]) File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py", line 209, in expectMultipleRanges str(len(body)), rsp.getHeader('content-length'))) File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in failIf if expr: raise self.failureException, msg AssertionError: Incorrect Content-Length is set! Expected 20425, got 20426. I don't grok the range support code at all: probably Martijn Pieters is the only person in the world who does. The tests are quite obscure to me, so I can't diagnose even whether the failure is a testing artifact or a real bug. I see another failure in a test Hanno recently added: Failure in test test_datetime (Products.PluginIndexes.DateRangeIndex.tests.test_DateRangeIndex.DRI_Tests) Traceback (most recent call last): File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 279, in run testMethod() File "/home/tseaver/projects/Zope-trunk/src/Products/PluginIndexes/DateRangeIndex/tests/test_DateRangeIndex.py", line 155, in test_datetime self.assertEqual(work.getEntryForObject(0), (20790915, 20790990)) File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 350, in failUnlessEqual (msg or '%r != %r' % (first, second)) AssertionError: (20791275, 20791350) != (20790915, 20790990) This one looks to be the result of using "naive" datetime objects while expecting them to be converted to hard-wired integers, so that tests which pass on Hanno's machine fail on mine, because of different timezones. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKZe9j+gerLs4ltQ4RAl4nAKDJQyWHdnlEEQv1zbQJeJiR7gGuTACfQxDi n5GO/tP+jpRhqUddn2CbaKg= =RdUR -----END PGP SIGNATURE-----
On Tue, Jul 21, 2009 at 18:40, Tres Seaver <tseaver@palladion.com> wrote:
File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py", line 332, in testMultipleRangesBigFileOutOfOrder (70000, 80001)]) File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py", line 209, in expectMultipleRanges str(len(body)), rsp.getHeader('content-length'))) File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in failIf if expr: raise self.failureException, msg AssertionError: Incorrect Content-Length is set! Expected 20425, got 20426.
I don't grok the range support code at all: probably Martijn Pieters is the only person in the world who does. The tests are quite obscure to me, so I can't diagnose even whether the failure is a testing artifact or a real bug.
Here's my response to Tres; for some reason my mail reader didn't include zope-dev. I can add now that I'll be at a sprint this weekend where I'll probably have time to take a look myself: ----8<------------------ Sorry you find them obscure; I followed unit test expectations as best as I could. The test that fails does the following: 1) Create a big file (uploadBigFile, enough random data to be more than one pdata chunk) 2) Request for multiple HTTP ranges (slices) of the file to be sent. These slices should be returned a) in the same order, b) of the correct length, and of course c) the correct slice of the big file created in 1. In this case line 330 specifies that bytes 10-15 (inclusive), the last 10000 bytes, and bytes 70000-80000 (inclusive) are are to be returned. The call to expectMultipleRanges includes the expected slices (python syntax, so end value is exclusive). 3) On line 209 of expectMultipleRanges it then tests if the resulting response has the correct content length header for the whole response set. The failure reported is that the Content-Length header reports 1 byte more than what was actually sent, so the body was 20425 bytes long, while the content-header claims it would be 20426 bytes. Because this is a multi-part range request, Image.py line 277 and onwards handle this request. The response ends up looking something like this: Content-length: size we pre-calculate Content-type: multipart/byteranges; boundary="boundarystring" More headers --boundarystring Content-type: image/whatever Content-range: bytes start-end/total-file-size .... data ... --boundarystring ... Another section ... etc. --boundarystring-- There are 2 ways the failure could be caused. Something could go wrong with the size calculation on lines 280-287; the total body length is the length of the range data plus the length of the headers per multi-part plus the length of the MIME boundaries between them. There is thus a length per part plus the length of the last boundary. Or something goes wrong when writing the range data in lines 309-358. Because there is only one byte difference between what was calculated on lines 280-287 and what is produced here this is probably a newline somewhere. I do find it puzzling this only fails on 64 bit platforms. It could be that the pdata chunk handling is borken and has a off-by-one error when running on 64-bit platforms. Hope this clarifies things a little. I have little time right now to chase this myself right now, sorry! ----8<------------------ -- Martijn Pieters
On 03.08.09 09:04, Martijn Pieters wrote:
On Tue, Jul 21, 2009 at 18:40, Tres Seaver <tseaver@palladion.com> wrote:
File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py", line 332, in testMultipleRangesBigFileOutOfOrder (70000, 80001)]) File "/home/tseaver/projects/Zope-trunk/src/OFS/tests/testRanges.py", line 209, in expectMultipleRanges str(len(body)), rsp.getHeader('content-length'))) File "/opt/Python-2.6.2/lib/python2.6/unittest.py", line 321, in failIf if expr: raise self.failureException, msg AssertionError: Incorrect Content-Length is set! Expected 20425, got 20426.
I don't grok the range support code at all: probably Martijn Pieters is the only person in the world who does. The tests are quite obscure to me, so I can't diagnose even whether the failure is a testing artifact or a real bug.
Here's my response to Tres; for some reason my mail reader didn't include zope-dev. I can add now that I'll be at a sprint this weekend where I'll probably have time to take a look myself:
Oki. Tnx, Andreas
On Mon, Aug 3, 2009 at 08:04, Martijn Pieters <mj@zopatista.com> wrote:
AssertionError: Incorrect Content-Length is set! Expected 20425, got 20426.
I don't grok the range support code at all: probably Martijn Pieters is the only person in the world who does. The tests are quite obscure to me, so I can't diagnose even whether the failure is a testing artifact or a real bug.
Here's my response to Tres; for some reason my mail reader didn't include zope-dev. I can add now that I'll be at a sprint this weekend where I'll probably have time to take a look myself:
I have found the problem, but do not understand where exactly the error occurs. The problem is caused by an acquisition wrapped object with a __getslice__ method get the wrong indices passed in when the end parameter is ommitted: data[start:] Image.Pdata classes have a __getslice__ method, and when data is acquisition wrapped, that method gets (start, -1) passed in on 64-bit platforms, while on my Macbook (start, sys.maxint) is passed in instead. Obviously this means that the slice returns not everything from 'start' to the end, but from 'start' until the one-but-last byte. However, as far as I can tell cAcquisition.c doesn't touch the start and end values. Anyone want to investigate why this goes wrong? Here is the simple test case: from Acquisition import Implicit class Root(Implicit): pass class Slicer(Implicit): def __getslice__(self, start, end): return [start, end] root = Root() slicer = Slicer().__of__(root) print slicer[1:] Run this as bin/zopepy acquisition.py and verify that [1, -1] is printed on 64-bit platforms as opposed to [1, 2147483647] on 32-bit platforms. -- Martijn Pieters
On Fri, Aug 7, 2009 at 14:55, Martijn Pieters <mj@zopatista.com> wrote:
Here is the simple test case:
from Acquisition import Implicit
class Root(Implicit): pass
class Slicer(Implicit): def __getslice__(self, start, end): return [start, end]
root = Root() slicer = Slicer().__of__(root) print slicer[1:]
Run this as bin/zopepy acquisition.py and verify that [1, -1] is printed on 64-bit platforms as opposed to [1, 2147483647] on 32-bit platforms.
Further datapoints, the following extra lines for the above script give more context to compare: import sys print slicer[sys.maxint:] print slicer[sys.maxint-1:] print Slicer()[1:] So we pass in sys.maxint and sys.maxint-1 as start values, and also run the Slicer without acquisition wrapping to demonstrate that the wrapper is the culprit, not python itself. Python passes in sys.maxint for the end value for the slice if you omit it, and these high values overflow somewhere in the wraper. -- Martijn Pieters
On Fri, Aug 7, 2009 at 14:55, Martijn Pieters <mj@zopatista.com> wrote:
The problem is caused by an acquisition wrapped object with a __getslice__ method get the wrong indices passed in when the end parameter is ommitted:
data[start:]
Image.Pdata classes have a __getslice__ method, and when data is acquisition wrapped, that method gets (start, -1) passed in on 64-bit platforms, while on my Macbook (start, sys.maxint) is passed in instead. Obviously this means that the slice returns not everything from 'start' to the end, but from 'start' until the one-but-last byte.
However, as far as I can tell cAcquisition.c doesn't touch the start and end values. Anyone want to investigate why this goes wrong?
The following checkin fixed this particular problem: http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102... The acquisition slice wrapper accepted Py_ssize_t arguments, but then passed these of as C ints (format string 'i'). Changing the format string to 'n' (Py_ssize_t) fixed *this particular case*. Likely more such fixes must be made. In any case, the HTTP range test no longer fails with this fix. -- Martijn Pieters
On Fri, Aug 7, 2009 at 3:26 PM, Martijn Pieters<mj@zopatista.com> wrote:
The following checkin fixed this particular problem:
http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102...
The acquisition slice wrapper accepted Py_ssize_t arguments, but then passed these of as C ints (format string 'i'). Changing the format string to 'n' (Py_ssize_t) fixed *this particular case*. Likely more such fixes must be made.
One thing to note here is that the above change is indeed only of of probably many that need to be made to support 64-bit platforms properly. The other thing to note is that the change now introduces a hard requirement on Python 2.5 or later. Neither Martijn nor me could figure out a way to make the function calls to Build have different format identifier arguments dependent on the Python version. On Python 2.4 there is only "i" for integer and 2.5 introduces "n" for Py_ssize_t. Since this is an argument to a function showing up in various combinations of "nn", "nO" and others, there seems to be no easy way to make this work. Someone with more knowledge about pre-processor tricks might come up with a solution to this, otherwise we will have to choose between dropping 64-bit support or dropping the unofficial Python 2.4 support. Hanno
On 08.08.09 14:19, Hanno Schlichting wrote:
On Fri, Aug 7, 2009 at 3:26 PM, Martijn Pieters<mj@zopatista.com> wrote:
The following checkin fixed this particular problem:
http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102...
The acquisition slice wrapper accepted Py_ssize_t arguments, but then passed these of as C ints (format string 'i'). Changing the format string to 'n' (Py_ssize_t) fixed *this particular case*. Likely more such fixes must be made.
One thing to note here is that the above change is indeed only of of probably many that need to be made to support 64-bit platforms properly. The other thing to note is that the change now introduces a hard requirement on Python 2.5 or later.
Neither Martijn nor me could figure out a way to make the function calls to Build have different format identifier arguments dependent on the Python version. On Python 2.4 there is only "i" for integer and 2.5 introduces "n" for Py_ssize_t. Since this is an argument to a function showing up in various combinations of "nn", "nO" and others, there seems to be no easy way to make this work.
Someone with more knowledge about pre-processor tricks might come up with a solution to this, otherwise we will have to choose between dropping 64-bit support or dropping the unofficial Python 2.4 support.
Dropping 64-bit support is no option, dropping Python 2.4 support is :) Adreas
On Sat, Aug 8, 2009 at 13:19, Hanno Schlichting <hanno@hannosch.eu> wrote:
One thing to note here is that the above change is indeed only of of probably many that need to be made to support 64-bit platforms properly. The other thing to note is that the change now introduces a hard requirement on Python 2.5 or later.
Neither Martijn nor me could figure out a way to make the function calls to Build have different format identifier arguments dependent on the Python version. On Python 2.4 there is only "i" for integer and 2.5 introduces "n" for Py_ssize_t. Since this is an argument to a function showing up in various combinations of "nn", "nO" and others, there seems to be no easy way to make this work.
Someone with more knowledge about pre-processor tricks might come up with a solution to this, otherwise we will have to choose between dropping 64-bit support or dropping the unofficial Python 2.4 support.
It turned out to be really simple as consecutive string literals are concatenated into one after macros are expanded. The following changeset should make Acquisition python 2.4 compatible again and also properly 64-bit capable: http://svn.zope.org/Acquisition/trunk/src/Acquisition/_Acquisition.c?rev=102... To summarize: when passing Py_BuildValue a Py_ssize_t value to build a python int, you need to use the 'n' format string instead of 'i'. But because 'n' and Py_ssize_t are both defined as of python 2.5, for C code to work in python 2.4 you need to redefine these two back to 'i' and int. -- Martijn Pieters
I can reproduce the failures in ExtensionClass and also get some weird errors while trying to buildout the Acquisition package: ajung@blackmoon:/develop/repository/svn.zope.org/Acquisition/trunk> bin/buildout Develop: '/data/develop/repository/svn.zope.org/Acquisition/trunk/.' Unused options for buildout: 'download-directory'. Installing interpreter. While: Installing interpreter. An internal error occured due to a bug in either zc.buildout or in a recipe being used: Traceback (most recent call last): File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/buildout.py", line 1519, in main File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/buildout.py", line 473, in install File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/buildout.py", line 1101, in _call File "/home/ajung/.buildout/eggs/zc.recipe.egg-1.2.2-py2.6.egg/zc/recipe/egg/egg.py", line 169, in install relative_paths=self._relative_paths, File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/easy_install.py", line 964, in scripts File "/tmp/tmpt2EP2J/zc.buildout-1.3.0-py2.6.egg/zc/buildout/easy_install.py", line 1111, in _pyscript IOError: [Errno 26] Text file busy: '/data/develop/repository/svn.zope.org/Acquisition/trunk/bin/python' Both issues are severe and are a blocker for the next release. Andreas On 18.07.09 14:08, Stefan H. Holek wrote:
All failures appear to be due to the new box running Linux x86_64. Tests still pass fine on the Mac.
Any experience with this? MemoryError in Acquisition? WTH?
Stefan
On 18.07.2009, at 14:00, Zope Tests Summarizer wrote:
Summary of messages to the zope-tests list. Period Fri Jul 17 12:00:00 2009 UTC to Sat Jul 18 12:00:00 2009 UTC. There were 8 messages: 8 from Zope Tests.
Test failures -------------
Subject: FAILED (failures=1) : Zope-2.12 Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:50:00 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012082.html
Subject: FAILED (failures=1) : Zope-2.12-alltests Python-2.4.6 : Linux From: Zope Tests Date: Fri Jul 17 20:52:00 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012083.html
Subject: FAILED (failures=7) : Zope-2.12-alltests Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:54:01 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012084.html
Subject: FAILED (failures=1) : Zope-trunk Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:56:01 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012085.html
Subject: FAILED (failures=7) : Zope-trunk-alltests Python-2.6.2 : Linux From: Zope Tests Date: Fri Jul 17 20:58:01 EDT 2009 URL: http://mail.zope.org/pipermail/zope-tests/2009-July/012086.html
-- Stefan H. Holek stefan@epy.co.at
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- ZOPYX Ltd. & Co KG \ ZOPYX & Friends Charlottenstr. 37/1 \ The experts for your Python, Zope and D-72070 Tübingen \ Plone projects www.zopyx.com, info@zopyx.com \ www.zopyx.de/friends, friends@zopyx.de ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting
On Sat, Jul 18, 2009 at 2:08 PM, Stefan H. Holek<stefan@epy.co.at> wrote:
All failures appear to be due to the new box running Linux x86_64. Tests still pass fine on the Mac.
Any experience with this? MemoryError in Acquisition? WTH?
I don't have any experience with either 64-bit nor can I write any C. And it seems obvious that these kind of crashes are caused by C extension code. I kind of suspect that we are seeing the results of http://www.python.org/dev/peps/pep-0353 though.
From what I understand we need to change code in C extension modules following the instructions in that PEP, to be able to run code reliably on 64-bit platforms. To my knowledge nobody changed any C code in Zope yet to do so.
The other interesting bit to this is, that it seems doable but hard to do these changes while preserving compatibility with Python 2.4. These changes along with the definition of "Py_ssize_t" were introduced into Python 2.5. At this point I think we need to declare 64-bit platforms as unsupported. Hopefully people with the right kind of knowledge would like to step up and start working on these issues. Hanno
Hanno Schlichting wrote:
I kind of suspect that we are seeing the results of http://www.python.org/dev/peps/pep-0353 though.
That is very likely. BTW, that PEP links to a handy tool that reveals most 64 bit portability issues in Python-oriented C code.
From what I understand we need to change code in C extension modules following the instructions in that PEP, to be able to run code reliably on 64-bit platforms. To my knowledge nobody changed any C code in Zope yet to do so.
We fixed the C portability problems in Zope 3 a while back, but it seems we didn't fix the extensions that are specific to Zope 2.
The other interesting bit to this is, that it seems doable but hard to do these changes while preserving compatibility with Python 2.4. These changes along with the definition of "Py_ssize_t" were introduced into Python 2.5.
Preprocessor macros solve that fairly cleanly, actually: #ifndef Py_ssize_t #define Py_ssize_t int #endif
At this point I think we need to declare 64-bit platforms as unsupported. Hopefully people with the right kind of knowledge would like to step up and start working on these issues.
I just checked in some code that solves the portability problem in zope.index, although I haven't created a release. It looks like the Acquisition module needs some 64 bit TLC. Shane
On Sat, Jul 25, 2009 at 8:01 PM, Shane Hathaway<shane@hathawaymix.org> wrote:
Hanno Schlichting wrote:
I kind of suspect that we are seeing the results of http://www.python.org/dev/peps/pep-0353 though.
That is very likely. BTW, that PEP links to a handy tool that reveals most 64 bit portability issues in Python-oriented C code.
Right, I've given that a shot and it reports plenty of problems for Acquisition.c for example. In addition to the general compiler warnings it spits out, which don't tell me much.
From what I understand we need to change code in C extension modules following the instructions in that PEP, to be able to run code reliably on 64-bit platforms. To my knowledge nobody changed any C code in Zope yet to do so.
We fixed the C portability problems in Zope 3 a while back, but it seems we didn't fix the extensions that are specific to Zope 2.
How thorough has that been? I can see the Py_ssize_t in some of the c files in zope.* packages, but looking for example at: http://svn.zope.org/zope.proxy/trunk/src/zope/proxy/_zope_proxy_proxy.c?rev=... I see a stuff like this: PyObject * WrapperType_Lookup(PyTypeObject *type, PyObject *name) { int i, n; [...] n = PyTuple_GET_SIZE(mro) - 1; [...] } where PyTuple_GET_SIZE is defined to return a Py_ssize_t and not an int. Either I misunderstand something here or I was just "lucky" to find something like this as my first test.
The other interesting bit to this is, that it seems doable but hard to do these changes while preserving compatibility with Python 2.4. These changes along with the definition of "Py_ssize_t" were introduced into Python 2.5.
Preprocessor macros solve that fairly cleanly, actually:
#ifndef Py_ssize_t #define Py_ssize_t int #endif
Ah ok. I was more concerned about the function pointer changes. intargfunc to ssizeargfunc and so on. I wasn't sure if these are as easy to redefine as simple data types.
At this point I think we need to declare 64-bit platforms as unsupported. Hopefully people with the right kind of knowledge would like to step up and start working on these issues.
I just checked in some code that solves the portability problem in zope.index, although I haven't created a release. It looks like the Acquisition module needs some 64 bit TLC.
Cool! I kinda suspect that specific code is also somewhere in Zope2 in the PluginIndexes where it has been ported from recently. Hanno
On Sun, Jul 26, 2009 at 02:55:58AM +0200, Hanno Schlichting wrote:
On Sat, Jul 25, 2009 at 8:01 PM, Shane Hathaway<shane@hathawaymix.org> wrote:
Hanno Schlichting wrote:
I kind of suspect that we are seeing the results of http://www.python.org/dev/peps/pep-0353 though.
That is very likely. BTW, that PEP links to a handy tool that reveals most 64 bit portability issues in Python-oriented C code.
Right, I've given that a shot and it reports plenty of problems for Acquisition.c for example. In addition to the general compiler warnings it spits out, which don't tell me much.
From what I understand we need to change code in C extension modules following the instructions in that PEP, to be able to run code reliably on 64-bit platforms. To my knowledge nobody changed any C code in Zope yet to do so.
We fixed the C portability problems in Zope 3 a while back, but it seems we didn't fix the extensions that are specific to Zope 2.
How thorough has that been?
I'm not familiar with the source code of the extension modules, and I haven't reviewed them for possible 64-bit issues. But we've been running a Zope 3 app in production on 64-bit Linux for almost two years now. No 64-bit-related problems bit us so far. There's a buildbot for Zope 3.4 packages that runs their tests on 32 and 64 bit platforms on Python 2.4 and 2.5: http://zope3.pov.lt/buildbot/ Currently ZODB tests are disabled on it because they used to have irritating spurious failures that seem to be related to the speed of the machine rather than its platform. I believe there's a similar buildbot for Zope 3.5 KGS out there somewhere.
I can see the Py_ssize_t in some of the c files in zope.* packages, but looking for example at:
http://svn.zope.org/zope.proxy/trunk/src/zope/proxy/_zope_proxy_proxy.c?rev=...
I see a stuff like this:
PyObject * WrapperType_Lookup(PyTypeObject *type, PyObject *name) { int i, n;
[...]
n = PyTuple_GET_SIZE(mro) - 1;
[...] }
where PyTuple_GET_SIZE is defined to return a Py_ssize_t and not an int.
It shouldn't be a problem until you start seeing tuples with 2 billion items in them, no?
Either I misunderstand something here or I was just "lucky" to find something like this as my first test.
Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
On 25.07.09 13:57, Hanno Schlichting wrote:
At this point I think we need to declare 64-bit platforms as unsupported. Hopefully people with the right kind of knowledge would like to step up and start working on these issues.
This is really not an option. Zope 2.12 must run on 32 and 64 bit systems without issues. Since this is a major blocker I will defer the next release (scheduled for this week) until we have fixed the issue. Andreas
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote:
On 25.07.09 13:57, Hanno Schlichting wrote:
At this point I think we need to declare 64-bit platforms as unsupported. Hopefully people with the right kind of knowledge would like to step up and start working on these issues. This is really not an option. Zope 2.12 must run on 32 and 64 bit systems without issues. Since this is a major blocker I will defer the next release (scheduled for this week) until we have fixed the issue.
I think I have fixed the ExtensionClass and Acquisition problems for 64-bit machines: I released new eggs to PyPI for them: - - http://pypi.python.org/pypi/ExtensionClass/2.11.3 - - http://pypi.python.org/pypi/Acquisition/2.12.2 Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKddsu+gerLs4ltQ4RArPiAJ0USyr/xHsBthWCdG4GAfpan1tirQCgh711 eu6hqMsep/VtglUXSshIgoY= =6amd -----END PGP SIGNATURE-----
On Sun, Aug 2, 2009 at 8:30 PM, Tres Seaver<tseaver@palladion.com> wrote:
I think I have fixed the ExtensionClass and Acquisition problems for 64-bit machines: I released new eggs to PyPI for them:
Awesome! You wouldn't be interested to look at the C code in Persistence and Zope2 itself? ;-) Hanno
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hanno Schlichting wrote:
On Sun, Aug 2, 2009 at 8:30 PM, Tres Seaver<tseaver@palladion.com> wrote:
I think I have fixed the ExtensionClass and Acquisition problems for 64-bit machines: I released new eggs to PyPI for them:
Awesome!
You wouldn't be interested to look at the C code in Persistence and Zope2 itself? ;-)
I have updated the cAccessControl and Record extensions for conformance with the PEP. I don't think the Persistence code needs anything: AFAICT, it is already good to go. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKdgWp+gerLs4ltQ4RApPJAJ4neMSFrdUaLomc5fnImtkuDQVkwgCgsZNL 1iBevGRfBSM5m1OFJILnqp0= =oXK6 -----END PGP SIGNATURE-----
On 02.08.09 20:30, Tres Seaver wrote:
Andreas Jung wrote:
On 25.07.09 13:57, Hanno Schlichting wrote:
At this point I think we need to declare 64-bit platforms as unsupported. Hopefully people with the right kind of knowledge would like to step up and start working on these issues. This is really not an option. Zope 2.12 must run on 32 and 64 bit systems without issues. Since this is a major blocker I will defer the next release (scheduled for this week) until we have fixed the issue.
I think I have fixed the ExtensionClass and Acquisition problems for 64-bit machines: I released new eggs to PyPI for them:
Thanks Tres - the issue seems to be resolved. Right now there is only one failure with the http ranges tests (Martijn, can you check this?) and a complete failure of the zope.index tests (see other mail). Andreas
participants (8)
-
Andreas Jung -
Hanno Schlichting -
Marius Gedminas -
Martijn Pieters -
Shane Hathaway -
Stefan H. Holek -
Tres Seaver -
Zope Tests Summarizer