Difficulties with MVCC implementation was: [ZODB-Dev] Potential BTrees splitting bug

[Dieter Maurer]

Thank you again.

Your message is again very valuable.

Tim Peters wrote at 2003-9-25 14:23 -0400:
 > [Dieter Maurer]
 > > I found the likely cause for the test failure: the limited
 > > time resolution.
 > >
 > >   The idea behind my MVCC implementation is to deliver the state
 > >   as it has been at transaction start.
 > >   Due to the limited time resolution, the transaction start
 > >   cannot be determined precisely but only within the resolution limit.
 > 
 > I'm assuming this became "a problem" for you on one particular system
 > because that system was exceptionally fast.

That is what I thought.
Indeed the systems on which the test fails are our fastest servers,
considerably faster than the systems where the test passed.

 > If so, you could probably
 > provoke problems much more easily on a Windows box, where time.time()
 > updates only 18.2 times per second (this is in some sense the opposite of
 > what happens on Linux:  on Windows time.time() has only ~= 0.055 second
 > resolution while on Linux time.time() typically has microsecond resolution;
 > but time.clock() on Windows typically has sub-microsecond resolution while
 > time.clock on Linux typically has 0.01 second resolution).

The resolution is much better than I thought (order of ms instead of us).
This reduces the likelyhood that I already found the true reason
for the failure.

To verify my assumption, I had added the "sleep= 0.001" parameter
to the "StressTest" call which I already saw in the
"ConcurrentUpdate1Storage". With this, the test passed.
However, I may have simply reduced the conflict probability enough...

We will see: I will remove the delay parameter again and implement
the "ReadConflictError" on state within resolution of the transaction
start time. If my assumption is right, the test will still pass...

 > > As a consequence, we cannot trust a state defined near the transaction
 > > start. I will have to raise a "ReadConflictError" in this case.
 > 
 > Brrr.  That is disappointing.

Indeed, but conceptionally, I need to synchronize my transaction
start time with the commit times such that I can decide for earch
commit time whether it is before or after the start time.
If I cannot do that, I must raise a "ReadConflictError".

We know that there is a race condition in ZEO's invalidation protocol
(documented by Andrew). Invalidation messages flush objects from
cache only in "_flush_invalidations" called in "_setDB" (open),
at the end of a transaction and in "sync". An object surviving such
an invalidation wave remains in the ZODB cache until the transaction
end, even when an invalidation message arrives for it in the meantime.

 > ...
 > We (Zope Corp) can't add MVCC to ZODB 3.1 because that's in maintenance mode
 > now, and new features don't get added after feature freeze.  The same
 > applies to ZODB 3.2 (not yet released, but entered feature freeze when the
 > first beta got released).  The protocol in 3.2 already changed, though, and
 > in 3.2 invalidation messages include the transaction id.  When Zope releases
 > an MVCC implementation, it will have to be in 3.3 (maybe not under that
 > name -- I don't know yet know what it will be called).

Do not worry. I do not request Zope Corp to solve our problems,
even so I much appreaciate your helpful recommendatins.


Dieter