[Zope-Coders] Unittests for 2.7 failing in a unpredictable way

[Tim Peters]

[Andreas Jung]
> I have been running the 2.7 unittests today multiple times. Sometimes they
> pass, sometimes they fail when running unchanged inside the same sandbox
> (Python 2.3.3, Linux):

...
> ERROR: checkConcurrentLargeUpdates (ZEO.tests.testConnection.FileStorageConnectionTests)
...
> "/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/Connection.py", line 348, in commit
>    raise ReadConflictError(object=object)
> ReadConflictError: database read conflict error (oid 0x0f, class
> BTrees.OOBTree.OOBucket)

> FAIL: checkConcurrentUpdates2StoragesMT
(ZEO.tests.testConnection.FileStorageConnectionTests)
...
> AssertionError: expected keys != actual keys
> key 263 expected but not in tree

> FAIL: checkSecondBeginFails (ZEO.tests.testZEO.MappingStorageTests)
...
> "/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZEO/tests/ThreadTests.py",
> line 39, in join
>    assert not self.isAlive()
> AssertionError

> FAIL: check7ZODBThreads (ZODB.tests.testFileStorage.FileStorageTests)
...
> "/develop/sandboxes/Zope-2_7-branch/Zope/lib/python/ZODB/tests/MTStorage.py",
> line 70, in runtest
>    self.test.fail()
>  File "/opt/python-2.3.4/lib/python2.3/unittest.py", line 270, in fail
>    raise self.failureException, msg
> AssertionError

> Any ideas what might go wrong?

Multi-thread and multi-process tests (ZEO tests in general spawn
multiple processes) are difficult to make work all the time, and
several ZEO tests in particular have always failed if you run them
"often enough", especially when run on a not-otherwise-quiet machine. 
For example, since there's no direct way for one process to peer
inside another, the tests too often rely on calling time.sleep(N) in
process A, desperately hoping N is long enough for process B to have
accomplished something necessary for the test to succeed.

The bottom lines:

- It's very time-intensive to truly fix one of these, usually requiring a much
 more difficult approach than the probabilistic "sleep and hope";
- I'm the only one who ever tries to fix one of these anymore; and
- Fixing one of these is rarely a top priority anymore.

So that's not good, but I rarely have time to make it better.  These
tests are in much better shape than they were in a year ago, and a
year ago were in much better shape than two years ago, and they're in
better shape today in ZODB 3.3 than in 3.2.  But they're still not in
good shape, and there's no relief on the immediate horizon.