[Zope] Re: help debugging a "can't pickle" error deep within a
catalog reindex
Ross Patterson
me at rpatterson.net
Tue Jun 17 21:42:25 EDT 2008
Rob Miller <ra at burningman.com> writes:
> hi,
>
> i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with
> over 29,000 indexed objects. it churns for a good long time, and
> eventually fails with a long set of tracebacks, of which i've included
> a sample at the end of this message.
>
> i think i understand the gist of the issue... it's trying to write an
> object (probably a CatalogBrain) to the database, but this object's
> __dict__ contains a value that is of type instancemethod, which isn't
> allowed for persistent objects.
>
> the problem is that i can't figure out which specific objects are
> causing the problem. i've used pdb.post_mortem to get a debug prompt
> way down in the traceback, but the code goes in and out of C modules,
> so i'm missing a lot of what's happening. and when i interactively
> peek at the objects that are being indexed when the error happens,
> there doesn't seem to be anything wrong, and i can index the objects
> w/ no problem. i've even tried dropping the subtransaction threshold
> down to 1, so it will try to commit a savepoint after every object,
> but none of the objects being indexed seemed to have any problems.
>
> i CAN verify that the instancemethod that is causing the problem
> renders like this:
>
> <bound method SessionDataManager.getSessionData of <SessionDataManager
> at /session_data_manager>>
>
> even that hasn't proven enough for me to concretely identify the
> source of the problem, though.
>
> i've been working on this for a full day already, and am not sure how
> to proceed. does anyone have any debugging tips that might help me
> figure out what, exactly, is causing the reindex attempts to blow up?
>
> thanks!
Can you get a pdb.post_mortem prompt at the actual ZODB/serialize.py:416
error or only at transaction/_transaction.py:267 where some sort of
previous error is handled?
If the former, I often find it informative to step up to
ZODB/serialize.py:407 where obj.__getstate__() is called and find out
what obj is. Is it the same object every time? If not, is it of the
same type every time?
A next step can also be to put a pdb.set_trace() at
transaction/_transaction.py:340 in the register() method such that it's
is only called when the offending object is added to the transaction.
Hope some of that helps,
Ross
> Traceback (most recent call last):
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py",
> line 296, in refreshCatalog
> self.catalog_object(obj, p, pghandler=pghandler)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py",
> line 367, in catalog_object
> self._increment_counter()
> File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py",
> line 395, in _increment_counter
> self._counter.change(1)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/BTrees/Length.py",
> line 55, in change
> self.value += delta
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 890, in register
> self._register(obj)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 900, in _register
> self.transaction_manager.get().join(self)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 273, in join
> self._prior_operation_failed() # doesn't return
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 267, in _prior_operation_failed
> raise TransactionFailedError("An operation previously failed, "
> TransactionFailedError: An operation previously failed, with traceback:
>
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZServer/PubCore/ZServerPublisher.py",
> line 25, in __init__
> response=b)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py",
> line 401, in publish_module
> environ, debug, request, response)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py",
> line 202, in publish_module_standard
> response = publish(request, module_name, after_list, debug=debug)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py",
> line 119, in publish
> request, bind=1)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/mapply.py",
> line 88, in mapply
> if debug is not None: return debug(object,args,context)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZPublisher/Publish.py",
> line 42, in call_object
> result=apply(object,args) # Type s<cr> to step into published object.
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py",
> line 260, in manage_catalogReindex
> self.refreshCatalog(clear=1, pghandler=handler)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py",
> line 296, in refreshCatalog
> self.catalog_object(obj, p, pghandler=pghandler)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CMFPlone/CatalogTool.py",
> line 385, in catalog_object
> update_metadata, pghandler=pghandler)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch.py",
> line 96, in catalog_object
> uid, idxs, update_metadata, pghandler)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/zope/Products/CacheSetup/patch_utils.py",
> line 6, in call
> return getattr(self, PATTERN % __name__)(*args, **kw)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/Products/ZCatalog/ZCatalog.py",
> line 560, in catalog_object
> transaction.savepoint(optimistic=True)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_manager.py",
> line 110, in savepoint
> return self.get().savepoint(optimistic)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 312, in savepoint
> self._saveAndRaiseCommitishError() # reraises!
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 422, in _saveAndRaiseCommitishError
> t, v, tb = self._saveAndGetCommitishError()
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 309, in savepoint
> savepoint = Savepoint(self, optimistic, *self._resources)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/transaction/_transaction.py",
> line 737, in __init__
> savepoint = savepoint()
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 1046, in savepoint
> self._commit(None)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 555, in _commit
> self._store_objects(ObjectWriter(obj), transaction)
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/Connection.py",
> line 582, in _store_objects
> p = writer.serialize(obj) # This calls __getstate__ of obj
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/serialize.py",
> line 407, in serialize
> return self._dump(meta, obj.__getstate__())
> File
> "/home/rob/topp/14000/builds/20080611/opencore/lib/zope/lib/python/ZODB/serialize.py",
> line 416, in _dump
> self._p.dump(state)
> File "copy_reg.py", line 69, in _reduce_ex
> raise TypeError, "can't pickle %s objects" % base.__name__
> TypeError: can't pickle instancemethod objects
>
> _______________________________________________
> Zope maillist - Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announcehttp://mail.zope.org/mailman/listinfo/zope-dev )
More information about the Zope
mailing list