[ZODB-Dev] Race condition in RelStorage history-free pack?

[Ian McCracken]

We have an area of our product that can, depending on certain conditions, produce lots of rapid transactions all at once. We’re getting many reports of POSKeyErrors at sites where the transaction volume is higher than others. They appear to coincide (in many cases, at least) with zodbpack runs.

After some investigation, it appears that it’s a timing issue. If an object is unreferenced during the pre_pack() stage, it will be marked for GC. If it then becomes referenced before the actual DELETE FROM is executed, it will be deleted nonetheless and POSKeyErrors will result.

Now, I don’t know how the object is unreferenced and referenced in separate transactions; as far as I can tell, there are no two-transaction moves or anything like that happening. Nonetheless, it’s entirely possible we can solve this by tracking down the code that sets up the race condition.

But is there any lower-level way around the race condition? A good amount of our application is pluggable by the end user, and I’d like to avoid vague technical warnings about transaction boundaries if possible :) If we could verify that no POSKeyErrors will result from the DELETE before running it, that would be much simpler.

I should also mention that running zodbpack with days=1 isn’t an across-the-board solution for us. We have some customers for whom rapid growth of the database is an issue, and they pack more frequently.

—Ian



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20140718/25bdd9b1/attachment.html>