[ZODB-Dev] Re: [Zope-dev] Re: [Bug] ZODB invalidation processing

[Joachim Schmitz]

some more findings:

1. The conflict error really happens on the Portalcatalog
2. It is a BTreesConflictError: BTrees conflict error at -1/47/47: 
Conflicting inserts which disguised as ConflictError, through the 
various try: excepts:

3. It happens on DateIndexes like "created" and "modified".

4. I put for a test:
         try:
             resolved = resolve(old, committed, newstate)
         except:
             return "invalidated"

into tryToResolveConflict

and into the store-method of FileStorage:

if serial != cached_tid:
      rdata = self.tryToResolveConflict(oid, cached_tid,serial, data)
      if rdata is None:
          raise POSException.ConflictError(oid=oid, serials=(cached_tid, 
serial), data=data)
          elif rdata == "invalidated":  #new for test
               old = False              #
               pass                     #

With these hacks I get rid of the conflictErrors. But when I delete the 
created Items I get:
2007-06-01 13:20:33 ERROR Zope.UnIndex DateIndex: unindex_object tried 
to retrieve set 1075379718 from index created but couldn't.  This should 
not happen.
2007-06-01 13:20:33 ERROR Zope.UnIndex DateIndex: unindex_object tried 
to retrieve set 1075379838 from index Date but couldn't.  This should 
not happen.
2007-06-01 13:20:33 ERROR Zope.UnIndex DateIndex: unindex_object tried 
to retrieve set 1075379719 from index modified but couldn't.  This 
should not happen.

So obviously I just fix the symptom with this hack.
I think the real cause:

lies in these lines:

             if old:
                 cached_tid = self._get_cached_tid(oid)
                 if cached_tid is None:
                     h = self._read_data_header(old, oid)
                     if h.version:
                         if h.version != version:
                             raise VersionLockError(oid, h.version)
                         pnv = h.pnv
                     cached_tid = h.tid

                 if serial != cached_tid:
                     rdata = self.tryToResolveConflict(oid, 
cached_tid,serial, data)

Which is also supported by the fact, that on our live system
which is running on 4 zeo-clients we get this traceback:


2007-06-01T12:38:59 ERROR Zope.SiteErrorLog 
http://uniben.waeup.org/campus/students/F541700/study_course/create_level
Traceback (innermost last):
   Module Zope2.App.startup, line 173, in zpublisher_exception_hook
   Module ZPublisher.Publish, line 121, in publish
   Module Zope2.App.startup, line 240, in commit
   Module transaction._manager, line 96, in commit
   Module Products.CPSCompat.PatchZODBTransaction, line 175, in commit
   Module transaction._transaction, line 436, in _commitResources
   Module ZODB.Connection, line 665, in tpc_vote
   Module ZEO.ClientStorage, line 893, in tpc_vote
   Module ZEO.ClientStorage, line 877, in _check_serials
ConflictError: database conflict error (oid 0x8fe833, class 
BTrees._IOBTree.IOBucket, serial this txn started with 
0x036e0418ddcccf55 2007-06-01 11:36:51.984384, serial currently 
committed 0x036e041a3a33e311 2007-06-01 11:38:13.641254)


Any suggestion for a temporary fix would be very welcome, since we get 
about 6000 conflict errors per day now about 15 % unresolved, and they 
are really killing our portal.

-- 
Gruß Joachim