[ZODB-Dev] RE: [Zope] Re: Squid headaches

[Dieter Maurer]

Tim Peters wrote at 2003-5-31 14:58 -0400:
 > [Dieter Maurer]
 > > ...
 > > In the second packing phase (the so called copying phase),
 > > the packing thread helds the storage lock most of the time.
 > > This can make accessing the ZODB impossible by normal operations,
 > > especially because the "lock_release", "lock_acquire" probably
 > > does not succeed in giving the lock to a waiting thread.
 > 
 > That decision is made by the platform thread facilities:  the strongest that
 > can be said in general is that lock_release *offers* to yield to another
 > thread.  The conditions under which the platform accepts the offer vary a
 > lot across platforms.

In other (i.e. non-Zope) code, I saw code like:

	 lock.release()
	 sleep(0) # indicate, threads should switch, if necessary
	 lock.acquire()

Of course, "sleep(0)" may not force a thread switch on all platforms.


My lock time optimization holds the lock only while it decides
whether it has finished. It looks like:


                # Read the transaction record
                seek(pos)
                h=read(TRANS_HDR_LEN)
                if len(h) < TRANS_HDR_LEN:
                    # DM: lock and try again
                    _commit_lock_acquire(); _lock_acquire(); locked= 1
                    h+= read(TRANS_HDR_LEN - len(h))
                    if len(h) < TRANS_HDR_LEN: break
		....
                # DM: ensure the complete transaction can be read
                if not (packing or locked or _largeEnough(file,tend + 8)):
                    # it was not - lock
                    # when we get the lock, the transaction will be written completely
                    # this would not work with a concurrent pack;
                    # but hopefully, this cannot happen
                    _commit_lock_acquire(); _lock_acquire(); locked= 1
                    assert _largeEnough(file,tend+8)
	       # DM: we can release the locks now, as the currently
	       #     copied transaction is complete
	       if locked: locked= 0; _lock_release(); _commit_lock_release()

"_largeEnough(file,size)" checks that *file* has at least length *size*.


Because the locks are held only during a very short time, locking
out of other requests should be drastically reduced.



Dieter