[ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope 2.XBIGSession problems - blocker - our site dies - need help ofexperienceZope developer, please)

[Tim Peters]

[Tim Peters]
>> That code is scary to me -- it talks about threads in the comments,
>> but there's nary a mutex in sight.  If, for example, two threads
>> happen to run _gc() simulateously, what's to stop one from deleting
>> the same keys the other is crawling over?  Since it must be rare for
>> two threads to get into _gc at the same time (but I don't see
>> anything-- like a mutex --that guarantees to stop that), this theory
>> may also fit the "rare and hard-to-reproduce failure" symptom.  It
>> may in fact be safe against thread races, but if it is I don't see
>> how.

[Chris McDonough]
> I must misunderstand something.

Possibly, but maybe not here.  I looked at the code with half an eyeball
during a jam (you know what that's like ...), and *any* talk of threads
without a mutex scares me.

> Given that the code you're talking about should raise a
> ReadConflictError in _gc (I am not using a ZODB with MVCC)

Out of context, it wasn't evident that the code was used in conjunction with
ZODB at all -- BTree objects can be used fine without any persistence in
play, in which case sharing a BTree across threads without locking can be
deadly.  For example, this program *appears* to do something very similar,
and should die quickly with a KeyError in thread 1 or thread 2 (it will vary
across runs):

"""
import sys
import threading
import time

from BTrees.IIBTree import IIBTree

more = True
thetree = IIBTree()

class Worker(threading.Thread):
    def run(self):
        global more
        try:
            while more:
                if not thetree:
                    time.sleep(.1)
                    continue
                print self, 'deleting', len(thetree)
                for k in list(thetree.keys()):
                    del thetree[k]
        except KeyError:
            print "oops! KeyError in", self
            more = False

Worker().start()
Worker().start()

while more:
    for i in xrange(1000):
        thetree[i] = i
"""

In fact, you can add all this hair to it near the top, and it will still die
with a KeyError quickly:

"""
import ZODB
from ZODB import FileStorage
st = FileStorage.FileStorage('Data.fs')
db = ZODB.DB(st)
conn = db.open()
root = conn.root()

more = True
thetree = IIBTree()
root['t'] = thetree
get_transaction().commit()
"""

> when two threads try to simultaneously access that data structure
> where one attempts to obtain the results of "keys()" and the
> other is attempting to delete a key, why would I need to protect that
> access with a mutex?

In the program above, the threads access a single BTree object in memory, so
any change made by one thread is instantly visible to all others.  I don't
understand the intended context your code runs in well enough to guess
whether that's what's happening to you.  I can say threads *appear* to be
accessing your OOBTree self._data via a shared-across-threads self object,
but there's really no way for me to guess whether that's true.  If somehow
the distinct threads all obtain their own working copy of "self" via loading
it from a database, then they would get their own distinct (in memory)
working copy of self._data too.  The program above shows that this doesn't
*necessarily* happen by magic, though, even if the BTree is stored in a
database -- it depends on the code *using* the code we're staring at.
Printing

    "%s %s" % (thread.get_ident(), id(self._data))

across threads would answer that question for sure: if distinct threads
print the same number for id(self._data), then a single BTree object is
getting shared across threads, and the code can't work as intended without
locking.

> There is no code in Zope AFAIK that employs a mutex for simultaneous
> access to persistent data; if this is true,

Ya, but Zope scares me too <0.9 wink>.

> I need to admit to not knowing the rules about knowing when a mutex is
> required, and it will throw my world into a brief tailspin. ;-)

You understand Zope better than I do, so you're much closer to the solution:
just explain all the relevant hidden details of Zope, and I'll give you a
hundred ways that will work <wink>.