[Zope] How to handle ZODB conflict errors

[Sean Hastings]

The slowdowns result when various transactions are trying to modify the same
data. When a conflict happens, one of the transactions is rolled back and
attempted again from the beginning. If the transaction takes a long time to
execute, and this happens many times, the delays can be considerable. Only
after some number of attempts, with continuing conflicts occurring, will the
transaction give up and allow the Conflict Error to actually be raised back
to the level of producing an error page, but each one will be logged in your
event.log file.

I wrote the following to explain conflicts to my own satisfaction:
-------------------------------------------------------------------

There are two types of conflict errors:

Read Conflict:
--------------
Transaction 1 reads object A.
Transaction 2 modifies object B.
Transaction 1 attempts to read object B, but notices that object B has been
modified since Transaction 1 began to read data. Since the data it read from
A, and the data that it now finds in B are not necessarily consistent, a
Read Conflict Error is produced.
Transaction 1 rolls back and restarts.


Write Conflict:
---------------
Transaction 1 reads object A.
Transaction 2 modifies object A.
Transaction 1 attempts to modify object A, but notices that object A has
been modified since it was read. Since overwriting the changes made by
Transaction 2 would cause "lost update problems" and possible data
inconsistency, a Write Conflict Error is produced.
If conflict is not handled (see below), Transaction 1 rolls back and
restarts.


Solutions:
---------


Read Conflicts - If you do not care about consistency during your read, you
can execute the statement "get_transaction().commit()" after each new object
accessed. This breaks your transactions up into smaller transactions that
are MUCH less likely to cause conflicts. I understand that Zope 2.8 (alpha
any day now) will eliminate Read Conflicts by including the ability to read
old constant states. In the example above, Transaction 1 would then just
read the old state of B before Transaction 2 had modified it.


Write Conflicts - You can define a method on any ZODB object called
"_p_resolveConflict". When a write conflict occurs, the first thing that
happens is that Zope tries to call this method to handle the problem,
passing it all the possible states of the object: OLD, FOUND, and TRIED. OLD
is the state that the transaction first read. FOUND is the new modified
state that some other transaction saved the object as. TRIED is the state
that the transaction wanted to save the object as when it realized that
there was a conflict. This method is intended to return a merged version of
the two objects. If you do not care about lost updates, this method could be
as simple as:

        def _p_resolveConflict(self,old,found,tried):
                return tried

-------------------------------------------------------------------

Here is an example of some code that I think works to handle write conflicts
in an order independent list. The object in question is a ZODB wrapper for a
list. The [] is stored in the "data" element referred to in saved.data,
old.data, etc:

	def _p_resolveConflict(self,old,saved,new):
		"""
		Merges two possible list states
		Accounts only for additions and deletions, not changes of order
		Prevents duplicate add or remove of same Item during conflict
		"""

		#get changes made in saved state
		saved_added = []
		saved_removed = []
		for item in saved.data:
			for i in range(saved.count(item) - old.count(item)):
				saved_added.append(item)
		for item in old.data:
			for i in range(old.count(item) - saved.count(item)):
				saved_removed.append(item)

		#get changes made in new state
		new_added = []
		new_removed = []
		for item in new.data:
			for i in range(new.count(item) - old.count(item)):
				saved_added.append(item)
		for item in old.data:
			for i in range(old.count(item) - new.count(item)):
				new_removed.append(item)

		#get duplicate changes
		both_added=[]
		for item in new_added:
			if saved_added.count(item):
				both_added.append(item)
				new_added.remove(item)
				saved_added.remove(item)
		both_removed=[]
		for item in new_removed:
			if saved_removed.count(item):
				both_removed.append(item)
				new_removed.remove(item)
				saved_removed.remove(item)

		#apply changes
		old.extend(saved_added)
		old.extend(new_added)
		old.extend(both_added)
		for item in saved_removed:
			old.remove(item)
		for item in new_removed:
			old.remove(item)
		for item in both_removed:
			old.remove(item)

		return old