[Zope] help debugging a "can't pickle" error deep within a catalog reindex

Tue Jun 17 18:47:53 EDT 2008

hi,

i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with over 
29,000 indexed objects.  it churns for a good long time, and eventually fails 
with a long set of tracebacks, of which i've included a sample at the end of 
this message.

i think i understand the gist of the issue... it's trying to write an object 
(probably a CatalogBrain) to the database, but this object's __dict__ contains 
a value that is of type instancemethod, which isn't allowed for persistent 
objects.

the problem is that i can't figure out which specific objects are causing the 
problem.  i've used pdb.post_mortem to get a debug prompt way down in the 
traceback, but the code goes in and out of C modules, so i'm missing a lot of 
what's happening.  and when i interactively peek at the objects that are being 
indexed when the error happens, there doesn't seem to be anything wrong, and i 
can index the objects w/ no problem.  i've even tried dropping the 
subtransaction threshold down to 1, so it will try to commit a savepoint after 
every object, but none of the objects being indexed seemed to have any problems.

i CAN verify that the instancemethod that is causing the problem renders like 
this:

<bound method SessionDataManager.getSessionData of <SessionDataManager at 
/session_data_manager>>

even that hasn't proven enough for me to concretely identify the source of the 
problem, though.

i've been working on this for a full day already, and am not sure how to 
proceed.  does anyone have any debugging tips that might help me figure out 
what, exactly, is causing the reindex attempts to blow up?

thanks!

-r

Traceback (most recent call last):
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py", 
line 296, in refreshCatalog
     self.catalog_object(obj, p, pghandler=pghandler)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py", 
line 367, in catalog_object
     self._increment_counter()
   File 
"/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py", 
line 395, in _increment_counter
     self._counter.change(1)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/BTrees/Length.py", 
line 55, in change
     self.value += delta
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py", 
line 890, in register
     self._register(obj)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py", 
line 900, in _register
     self.transaction_manager.get().join(self)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py", 
line 273, in join
     self._prior_operation_failed() # doesn't return
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py", 
line 267, in _prior_operation_failed
     raise TransactionFailedError("An operation previously failed, "
TransactionFailedError: An operation previously failed, with traceback:

   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZServer/PubCore/ZServerPublisher.py", 
line 25, in __init__
     response=b)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py", 
line 401, in publish_module
     environ, debug, request, response)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py", 
line 202, in publish_module_standard
     response = publish(request, module_name, after_list, debug=debug)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py", 
line 119, in publish
     request, bind=1)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/mapply.py", 
line 88, in mapply
     if debug is not None: return debug(object,args,context)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py", 
line 42, in call_object
     result=apply(object,args) # Type s<cr> to step into published object.
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py", 
line 260, in manage_catalogReindex
     self.refreshCatalog(clear=1, pghandler=handler)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py", 
line 296, in refreshCatalog
     self.catalog_object(obj, p, pghandler=pghandler)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py", 
line 385, in catalog_object
     update_metadata, pghandler=pghandler)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch.py", 
line 96, in catalog_object
     uid, idxs, update_metadata, pghandler)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch_utils.py", 
line 6, in call
     return getattr(self, PATTERN % __name__)(*args, **kw)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py", 
line 560, in catalog_object
     transaction.savepoint(optimistic=True)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_manager.py", 
line 110, in savepoint
     return self.get().savepoint(optimistic)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py", 
line 312, in savepoint
     self._saveAndRaiseCommitishError() # reraises!
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py", 
line 422, in _saveAndRaiseCommitishError
     t, v, tb = self._saveAndGetCommitishError()
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py", 
line 309, in savepoint
     savepoint = Savepoint(self, optimistic, *self._resources)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py", 
line 737, in __init__
     savepoint = savepoint()
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py", 
line 1046, in savepoint
     self._commit(None)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py", 
line 555, in _commit
     self._store_objects(ObjectWriter(obj), transaction)
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py", 
line 582, in _store_objects
     p = writer.serialize(obj)  # This calls __getstate__ of obj
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/serialize.py", 
line 407, in serialize
     return self._dump(meta, obj.__getstate__())
   File 
"/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/serialize.py", 
line 416, in _dump
     self._p.dump(state)
   File "copy_reg.py", line 69, in _reduce_ex
     raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle instancemethod objects