Rob Miller <ra@burningman.com> writes:
Ross Patterson wrote:
Rob Miller <ra@burningman.com> writes:
hi,
i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with over 29,000 indexed objects. it churns for a good long time, and eventually fails with a long set of tracebacks, of which i've included a sample at the end of this message.
i think i understand the gist of the issue... it's trying to write an object (probably a CatalogBrain) to the database, but this object's __dict__ contains a value that is of type instancemethod, which isn't allowed for persistent objects.
the problem is that i can't figure out which specific objects are causing the problem. i've used pdb.post_mortem to get a debug prompt way down in the traceback, but the code goes in and out of C modules, so i'm missing a lot of what's happening. and when i interactively peek at the objects that are being indexed when the error happens, there doesn't seem to be anything wrong, and i can index the objects w/ no problem. i've even tried dropping the subtransaction threshold down to 1, so it will try to commit a savepoint after every object, but none of the objects being indexed seemed to have any problems.
i CAN verify that the instancemethod that is causing the problem renders like this:
<bound method SessionDataManager.getSessionData of <SessionDataManager at /session_data_manager>>
even that hasn't proven enough for me to concretely identify the source of the problem, though.
i've been working on this for a full day already, and am not sure how to proceed. does anyone have any debugging tips that might help me figure out what, exactly, is causing the reindex attempts to blow up?
thanks!
Can you get a pdb.post_mortem prompt at the actual ZODB/serialize.py:416 error or only at transaction/_transaction.py:267 where some sort of previous error is handled?
yes, i put the post_mortem around the "transaction.savepoint" call at line 559 of Products/ZCatalog/ZCatalog.py. this is the place where the error is happening, but it's where the subtransaction threshold has been reached, so IIUC the error could be caused by any of the objects in the subtransaction.
Then it seems like you're not getting at the actual pickle error and thus aren't looking at the object whose __getstate__ returns the offending instance method. If you have enough memory, try running the refreshCatalog with subtransactions disabled to get the *actual* pickle error.
If the former, I often find it informative to step up to ZODB/serialize.py:407 where obj.__getstate__() is called and find out what obj is. Is it the same object every time? If not, is it of the same type every time?
right. as long as the subtxn threshold number is the same, then the object is the same. but if i change the subtxn threshold, then the object will be different. as i said, i even tried reducing the subtxn threshold to '1', so, assuming my understanding is correct, this would be triggered after every object. it took many hours to get to my debug prompt; when i did, the obj was FileAttachment object that didn't seem to have any problems. i was able to interactively index the object from the pdb prompt, and even commit the transaction, w/ no problems.
The actual error of your TB is the error that savepoint raises when a previous error has occurred. So changing the subtransaction threshold just changes which object is being indexed when the savepoint *happens* to be run and notices that an error has been raised for a previous object in the savepoint. At least, that's my read of it.
A next step can also be to put a pdb.set_trace() at transaction/_transaction.py:340 in the register() method such that it's is only called when the offending object is added to the transaction.
i'll give this a shot, thx.
Hope some of that helps,
even just having someone to discuss it with is helpful, for my sanity at least... ;-)
Well, then I'll have to enjoy your sanity vicariously. :) Ross