[Zope] Re: help debugging a "can't pickle" error deep within a catalog reindex

[Rob Miller]

Ross Patterson wrote:
> Rob Miller <ra at burningman.com> writes:
> 
>> hi,
>>
>> i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with
>> over 29,000 indexed objects.  it churns for a good long time, and
>> eventually fails with a long set of tracebacks, of which i've included
>> a sample at the end of this message.
>>
>> i think i understand the gist of the issue... it's trying to write an
>> object (probably a CatalogBrain) to the database, but this object's
>> __dict__ contains a value that is of type instancemethod, which isn't
>> allowed for persistent objects.
>>
>> the problem is that i can't figure out which specific objects are
>> causing the problem.  i've used pdb.post_mortem to get a debug prompt
>> way down in the traceback, but the code goes in and out of C modules,
>> so i'm missing a lot of what's happening.  and when i interactively
>> peek at the objects that are being indexed when the error happens,
>> there doesn't seem to be anything wrong, and i can index the objects
>> w/ no problem.  i've even tried dropping the subtransaction threshold
>> down to 1, so it will try to commit a savepoint after every object,
>> but none of the objects being indexed seemed to have any problems.
>>
>> i CAN verify that the instancemethod that is causing the problem
>> renders like this:
>>
>> <bound method SessionDataManager.getSessionData of <SessionDataManager
>> at /session_data_manager>>
>>
>> even that hasn't proven enough for me to concretely identify the
>> source of the problem, though.
>>
>> i've been working on this for a full day already, and am not sure how
>> to proceed.  does anyone have any debugging tips that might help me
>> figure out what, exactly, is causing the reindex attempts to blow up?
>>
>> thanks!
> 
> Can you get a pdb.post_mortem prompt at the actual ZODB/serialize.py:416
> error or only at transaction/_transaction.py:267 where some sort of
> previous error is handled?

yes, i put the post_mortem around the "transaction.savepoint" call at line 559 
of Products/ZCatalog/ZCatalog.py.  this is the place where the error is 
happening, but it's where the subtransaction threshold has been reached, so 
IIUC the error could be caused by any of the objects in the subtransaction.

> If the former, I often find it informative to step up to
> ZODB/serialize.py:407 where obj.__getstate__() is called and find out
> what obj is.  Is it the same object every time?  If not, is it of the
> same type every time?

right.  as long as the subtxn threshold number is the same, then the object is 
the same.  but if i change the subtxn threshold, then the object will be 
different.  as i said, i even tried reducing the subtxn threshold to '1', so, 
assuming my understanding is correct, this would be triggered after every 
object.  it took many hours to get to my debug prompt; when i did, the obj was 
FileAttachment object that didn't seem to have any problems.  i was able to 
interactively index the object from the pdb prompt, and even commit the 
transaction, w/ no problems.

> A next step can also be to put a pdb.set_trace() at
> transaction/_transaction.py:340 in the register() method such that it's
> is only called when the offending object is added to the transaction.

i'll give this a shot, thx.

> Hope some of that helps,

even just having someone to discuss it with is helpful, for my sanity at 
least...  ;-)

thx.

-r