[Zope] Re: help debugging a "can't pickle" error deep within a catalog reindex

[Ross Patterson]

Rob Miller <ra at burningman.com> writes:

> Ross Patterson wrote:
>> Rob Miller <ra at burningman.com> writes:
>>
>>> hi,
>>>
>>> i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with
>>> over 29,000 indexed objects.  it churns for a good long time, and
>>> eventually fails with a long set of tracebacks, of which i've included
>>> a sample at the end of this message.
>>>
>>> i think i understand the gist of the issue... it's trying to write an
>>> object (probably a CatalogBrain) to the database, but this object's
>>> __dict__ contains a value that is of type instancemethod, which isn't
>>> allowed for persistent objects.
>>>
>>> the problem is that i can't figure out which specific objects are
>>> causing the problem.  i've used pdb.post_mortem to get a debug prompt
>>> way down in the traceback, but the code goes in and out of C modules,
>>> so i'm missing a lot of what's happening.  and when i interactively
>>> peek at the objects that are being indexed when the error happens,
>>> there doesn't seem to be anything wrong, and i can index the objects
>>> w/ no problem.  i've even tried dropping the subtransaction threshold
>>> down to 1, so it will try to commit a savepoint after every object,
>>> but none of the objects being indexed seemed to have any problems.
>>>
>>> i CAN verify that the instancemethod that is causing the problem
>>> renders like this:
>>>
>>> <bound method SessionDataManager.getSessionData of <SessionDataManager
>>> at /session_data_manager>>
>>>
>>> even that hasn't proven enough for me to concretely identify the
>>> source of the problem, though.
>>>
>>> i've been working on this for a full day already, and am not sure how
>>> to proceed.  does anyone have any debugging tips that might help me
>>> figure out what, exactly, is causing the reindex attempts to blow up?
>>>
>>> thanks!
>>
>> Can you get a pdb.post_mortem prompt at the actual ZODB/serialize.py:416
>> error or only at transaction/_transaction.py:267 where some sort of
>> previous error is handled?
>
> yes, i put the post_mortem around the "transaction.savepoint" call at
> line 559 of Products/ZCatalog/ZCatalog.py.  this is the place where
> the error is happening, but it's where the subtransaction threshold
> has been reached, so IIUC the error could be caused by any of the
> objects in the subtransaction.

Then it seems like you're not getting at the actual pickle error and
thus aren't looking at the object whose __getstate__ returns the
offending instance method.  If you have enough memory, try running the
refreshCatalog with subtransactions disabled to get the *actual* pickle
error.

>> If the former, I often find it informative to step up to
>> ZODB/serialize.py:407 where obj.__getstate__() is called and find out
>> what obj is.  Is it the same object every time?  If not, is it of the
>> same type every time?
>
> right.  as long as the subtxn threshold number is the same, then the
> object is the same.  but if i change the subtxn threshold, then the
> object will be different.  as i said, i even tried reducing the subtxn
> threshold to '1', so, assuming my understanding is correct, this would
> be triggered after every object.  it took many hours to get to my
> debug prompt; when i did, the obj was FileAttachment object that
> didn't seem to have any problems.  i was able to interactively index
> the object from the pdb prompt, and even commit the transaction, w/ no
> problems.

The actual error of your TB is the error that savepoint raises when a
previous error has occurred.  So changing the subtransaction threshold
just changes which object is being indexed when the savepoint *happens*
to be run and notices that an error has been raised for a previous
object in the savepoint.  At least, that's my read of it.

>> A next step can also be to put a pdb.set_trace() at
>> transaction/_transaction.py:340 in the register() method such that
>> it's is only called when the offending object is added to the
>> transaction.
>
> i'll give this a shot, thx.
>
>> Hope some of that helps,
>
> even just having someone to discuss it with is helpful, for my sanity
> at least...  ;-)

Well, then I'll have to enjoy your sanity vicariously.  :)

Ross