[ZODB-Dev] Automating retry management

[Laurence Rowe]

On 11 May 2010 15:08, Jim Fulton <jim at zope.com> wrote:
> On Tue, May 11, 2010 at 8:38 AM, Benji York <benji at zope.com> wrote:
>> On Tue, May 11, 2010 at 7:34 AM, Jim Fulton <jim at zope.com> wrote:
>>> [...] The best I've been
>>> able to come up with is something like:
>>>
>>>    t = ZODB.transaction(3)
>>>    while t.trying:
>>>        with t:
>>>            ... transaction body ...
>>
>> I think you could get this to work:
>>
>> for transaction in ZODB.retries(3):
>>    with transaction:
>>        ... transaction body ...
>>
>> ZODB.retries would return an iterator that would raise StopIteration on
>> the next go-round if the previously yielded context manager exited
>> without a ConflictError.
>
> This is an improvement. It's still unsatisfying, but I don't think I'm going to
> get satisfaction. :)
>
> BTW, if I do something like this, I think I'll add a retry exception to
> the transaction package and have ZODB.POSException.ConflictError
> extend it so I can add the retry automation to the transaction package.

The repoze.retry package lets you configure a list of exceptions.
http://pypi.python.org/pypi/repoze.retry
http://svn.repoze.org/repoze.retry/trunk/repoze/retry/__init__.py

 Though it seems inspecting the error text is required for most sql
database errors to know if they are retryable, as ZPsycoPGDA does:

 188                 except (psycopg2.ProgrammingError,
psycopg2.IntegrityError), e:
 189                     if e.args[0].find("concurrent update") > -1:
 190                         raise ConflictError

(https://dndg.it/cgi-bin/gitweb.cgi?p=public/psycopg2.git;a=blob;f=ZPsycopgDA/db.py)

For PostgreSQL it should be sufficient to catch these errors and raise
Retry during tpc_vote.

For databases which do not provide MVCC in the same way as PostgreSQL,
concurrency errors could be manifested at any point in the
transaction. Even Oracle can raise an error during a long running
transaction when insufficient rollback space is available, resulting
in what is essentially a read conflict error. Such errors could not be
caught by a data manager and reraised as a Retry exception.

I think it might be useful to add an optional method to data managers
that is queried by the retry automation machinery to see if an
exception should potentially be retried. Perhaps this would best be
accomplished in two steps:

1. Add an optional property to data managers called ``retryable``.
This is a list of potentially retryable exceptions. When a data
manager is added to the transaction, the transaction's list of
retryable exceptions is extended by the joining data managers list of
retryable exceptions.

t = transaction.begin()
try:
    application()
except t.retryable, e:
    t.retry(e):

2. t.retry(e) is then checks with each registered data manager if that
particular exceptions is retryable, and if so raises Retry.

def retry(self, e):
    for datamanager in self._resources:
        try:
            retry = datamanager.retry
        except AttributeError:
            continue
        if isinstance(e, datamanager.retryable):
            datamanager.retry(e) # dm may raise Retry here

Laurence