[ZODB-Dev] BTree data loss bug under Python 3.x

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 05/25/2013 11:49 AM, Tres Seaver wrote:
> On 05/25/2013 02:50 AM, Marius Gedminas wrote:
>> On Fri, May 24, 2013 at 09:31:55PM -0400, Tres Seaver wrote:
>>> On 05/24/2013 01:28 PM, Marius Gedminas wrote:
>>>> I was trying to debug a failing ZEO test case 
>>>> (testConcurrentUpdates), and found out it had nothing to do with
>>>>  ZEO.  There's a bug either in BTrees or in ZODB itself that 
>>>> causes data corruption under Python 3.x.
>>>> 
>>>> Here's a test case: 
>>>> https://gist.github.com/mgedmin/5644876#file-zodbfail_simple-py
>>> 
>>> Thanks for the detective work!  I can confirm the bug, and that
>>> it is in the BTrees C extensions (the tests pass after 'rm 
>>> .tox/py32/lib/python3.2/site-packages/BTrees/*.so').
> 
>> It's the COMPARE macro in _compat.h.  This patch fixes the data 
>> corruption:
> 
>> diff --git a/BTrees/_compat.h b/BTrees/_compat.h index 
>> e004d54..4765df1 100644 --- a/BTrees/_compat.h +++
>> b/BTrees/_compat.h @@ -28,8 +28,8 @@ #define TEXT_FORMAT
>> PyUnicode_Format
> 
>> #define COMPARE(lhs, rhs) \ -    PyObject_RichCompareBool((lhs), 
>> (rhs), Py_LT) > 0 ? -1 : \ -    (PyObject_RichCompareBool((lhs), 
>> (rhs), Py_EQ) > 0 ? 0 : 1) +    (PyObject_RichCompareBool((lhs), 
>> (rhs), Py_LT) > 0 ? -1 : \ +     (PyObject_RichCompareBool((lhs), 
>> (rhs), Py_EQ) > 0 ? 0 : 1))
> 
> 
>> #else
> 
> 
>> I'm now trying to write a failing unit test.  It's a bit difficult
>> to get access to C macros from Python tests ;-)
> 
>> COMPARE is used in two places:
> 
>> BTrees/objectkeymacros.h-#define TEST_KEY_SET_OR(V, KEY, TARGET) \ 
>> BTrees/objectkeymacros.h:if ( ( (V) = COMPARE((KEY),(TARGET)) ), 
>> PyErr_Occurred() )
> 
>> and
> 
>> BTrees/objectvaluemacros.h:#define TEST_VALUE(VALUE, TARGET) 
>> COMPARE((VALUE),(TARGET))
> 
>> We can ignore TEST_KEY_SET_OR, as it works fine there.
> 
>> TEST_VALUE is used to implement BTree.byValue and bucket merging. 
>> The bug I was chasing down was caused by bucket mismerges, I think.
> 
>> It should be easier to write a test for byValue.  In fact there is 
>> one, in test__base.py, but it's not run against real C-implemented 
>> BTrees.  I copied it over into test_OOBTree.py and discovered that
>> it fails unexpectedly:

After adding your new test, and wrapping the 'COMPARE()' macro inside
extra parens within 'TEST_VALUE' (in 'objectvaluemacros.h'), the new
'byValue' test passes on Py3k.  I am trying to reproduce the ZEO test
failure now, and then against the BTrees trunk, before making a BTrees
release.


Tres.
- -- 
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/

iEYEARECAAYFAlGg9rIACgkQ+gerLs4ltQ6M5ACeIyIgCC9/jWwew1qSFyJNg+eb
EDIAnRORMJ2XN7xm8kjd+lZciEaJKkA0
=iXx3
-----END PGP SIGNATURE-----