[Zope] Re: help debugging a "can't pickle" error deep within a catalog reindex

[Rob Miller]

Rob Miller wrote:
> hi,
> 
> i'm trying to perform a ZCatalog.refreshCatalog() on a catalog with over 
> 29,000 indexed objects.  it churns for a good long time, and eventually 
> fails with a long set of tracebacks, of which i've included a sample at 
> the end of this message.
> 
> i think i understand the gist of the issue... it's trying to write an 
> object (probably a CatalogBrain) to the database, but this object's 
> __dict__ contains a value that is of type instancemethod, which isn't 
> allowed for persistent objects.
> 
> the problem is that i can't figure out which specific objects are 
> causing the problem.  i've used pdb.post_mortem to get a debug prompt 
> way down in the traceback, but the code goes in and out of C modules, so 
> i'm missing a lot of what's happening.  and when i interactively peek at 
> the objects that are being indexed when the error happens, there doesn't 
> seem to be anything wrong, and i can index the objects w/ no problem.  
> i've even tried dropping the subtransaction threshold down to 1, so it 
> will try to commit a savepoint after every object, but none of the 
> objects being indexed seemed to have any problems.

okay, after leaving this for a few days, i came back to it and managed to work 
it out.  for posterity's sake, what ended up working was that i added an 
explicit transaction.commit() after every object reindex, wrapped in a try: 
except, like so:

--- src/Zope/lib/python/Products/ZCatalog/ZCatalog.py   2007-10-29 
06:09:30.000000000 -0700
+++ lib/zope/lib/python/Products/ZCatalog/ZCatalog.py   2008-06-24 
10:47:49.000000000 -0700
@@ -294,6 +294,11 @@
              if obj is not None:
                  try:
                      self.catalog_object(obj, p, pghandler=pghandler)
+                    try:
+                        transaction.commit()
+                    except:
+                        import sys, pdb
+                        pdb.post_mortem(sys.exc_info()[2])
                  except ConflictError:
                      raise
                  except:

this made the reindex take a ridiculously long time, and it leaves the catalog 
in an inconsistent state (i.e. it should only be done on a copy of the 
database, one that you can throw away), but it did in the end reveal to me 
which record was causing the problem.

the problematic record ended up being a ghosted catalog entry that happened to 
share the same name as a view.  when the catalog tried to look up the object 
by its path (via unrestrictedTraverse), the view object was returned.

thanks for the help!

-r