[ZODB-Dev] Unpickler.noload, zc.zodbdgc and multi-database reference bug (IndexError)?

[jason.madden at nextthought.com]

On Jan 30, 2014, at 11:12, jason.madden at nextthought.com wrote:

> So it seems that the behaviour of `noload` might have changed between 2.6.x and 2.7.x?

Apologies for replying to myself, but I think I found the root cause.

After some further investigation, I found issue 1101399 (http://bugs.python.org/issue1101399), complaining that noload is broken for subclasses of dict. The fix for this issue was applied to the cPython trunk in October of 2009 without any corresponding tests (http://hg.python.org/releasing/2.7.6/rev/d0f005e6fadd). In releases with this fix (if I'm reading the code correctly), the pickle opcodes append and appends become no-ops, clearing the pickle stack when they are encountered. 

A multi-database reference is pickled as a list, so it uses the appends opcode:

   23: ]                    EMPTY_LIST
   24: q                    BINPUT     3
   26: (                    MARK
   27: U                        SHORT_BINSTRING 'm'
   30: (                        MARK
   31: U                            SHORT_BINSTRING 'Users_1_Prod'
   45: q                            BINPUT     4
   47: U                            SHORT_BINSTRING '\x00\x00\x00\x00\x00\x00\x00\x01'
   57: q                            BINPUT     5
   59: c                            GLOBAL     'zope.site.folder Folder'
   84: q                            BINPUT     6
   86: t                            TUPLE      (MARK at 30)
   87: q                        BINPUT     7
   89: e                        APPENDS    (MARK at 26)
   90: Q                    BINPERSID

This fix means that multi-database references are always going to be returned as an empty list under noload (again, if I'm reading the code correctly). This means that multi-references and noload don't work under Python 2.7.x or 3.x with zodbpickle and so consequently neither does an unmodified zc.zodbdgc. 

I don't know what the best way forward is. Our solution to use `load` instead seems to work for us, but may not work for everyone. Maybe zodbpickle could revert the fix in its branch and zc.zodbgc could depend on that? I'm happy to help test other ideas.

Thanks again,

Jason