[ZODB-Dev] Loading objects directly by OID

[Greg Ward]

--82I3+IH0IqGh5yIs
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

On 05 November 2001, I said:
> I'm going to try to strip the database in question down more -- I
> already got it from 80 MB down to 2 MB, but there are still a couple
> thousand objects in it.  Then maybe I can throw it at one of you guys
> and you can tell me if our storage is (gulp) corrupt, or if I'm just
> making incorrect assumptions about the nature of FileStorage.

OK, I've stripped this problem down to the bare minimum.  The database
is 411 bytes with 2 (or is that 3?) objects, and 40 lines of code are
ample to demonstrate.  I'll attach weird_db.tar.gz which contains the
three files necessary to reproduce the problem:
  mxdb.fs       2 (3?)-object database with weird object reference
  make_mods.py  code to create the modules and classes needed to
                load the objects in mxdb.fs
  opendb        script to open mxdb.fs and demonstrate the weird
                reference

On the surface, mxdb.fs appears to contain 2 objects:
>>> pprint(storage._index)
{'\x00\x00\x00\x00\x00\x00\x00\x00': 225L,
 '\x00\x00\x00\x00\x00\x07\xf8\x9a': 27L}

The first is obviously the root and therefore uninteresting.  The
interesting object is:

>>> connection['\x00\x00\x00\x00\x00\x07\xf8\x9a']
<WaferDescription instance at 816a2e0>

...an instance of mems.run.wafer.WaferDescription, which is defined in
make_mods.py:

  class WaferDescription (Persistent):
      pass

  sys.modules["mems.run.wafer"].WaferDescription = WaferDescription

(Yes, I did try it with a regular filesystem module/package hierarchy,
and got exactly the same result.  make_mods.py just does everything in
one place so it's easier to understand.)

Things get interesting when we look at the pickle of this object:

>>> storage.load('\x00\x00\x00\x00\x00\x07\xf8\x9a', None)
('((U\x0emems.run.waferq\x01U\x10WaferDescriptionq\x02tq\x03Nt.}q\x04U\x04dimsq\x05(U\x08\x00\x00\x00\x00\x00\x07\xf5\xc6q\x06(U\x16mems.lib.physical_unitq\x07U\x13DimensionCollectionq\x08ttQs.', '\x03@~mo\x94\xb7"')

If I understand pickle-speak correctly, this:
  (U\x08

means "push a mark on the stack and prepare to read an 8-byte string".
The 8-byte string is
  \x00\x00\x00\x00\x00\x07\xf5\xc6

I'm a little fuzzy on the details, but I get the impression that the
next bit of code:
  q\x06(U\x16mems.lib.physical_unitq\x07U\x13DimensionCollection

means "that 8-byte string was the OID of an instance of the
DimensionCollection class, which you'll find in the
mems.lib.physical_unit module".  (No, I do *not* understand cPickle.c
that well, but this seems to be the pattern for object refs in ZODB
pickles.  I would love to have this hunch confirmed!)

So, if I'm reading the pickle for object
'\x00\x00\x00\x00\x00\x07\xf8\x9a' correctly, it means that object has a
reference to object '\x00\x00\x00\x00\x00\x07\xf5\xc6'.

But that second OID does not appear in the storage's index.

This is what it all boils down to: object A appears to reference object
B, but B's OID is not in the storage.  Is this a bug in our code?  A bug
in ZODB?  A corrupt storage?  A possible occurence of the corner case
Barry explained yesterday?  (A refs B, removes ref to B, commits,
storage packed, B gone from storage but still in memory.)

Any ideas?

        Greg
-- 
Greg Ward - software developer                gward@mems-exchange.org
MEMS Exchange                            http://www.mems-exchange.org

--82I3+IH0IqGh5yIs
Content-Type: application/octet-stream
Content-Disposition: attachment; filename="weird_db.tar.gz"
Content-Transfer-Encoding: base64

H4sIAG4k6DsAA+1WX2sbRxBXGid2t1BaCiWP2zPCp8Q530m2RQKGYit5KW1aggjUNsrpbmVt
c7d7t7tGEsV97gfIU54C+RB962Mg+Q79GIU8dvb+6J+lNC1V0of9IXG7M7+ZnVlp5iYehl2n
JyurhOu57v7ubsV13XrD9eAJgl1X7zPsNxoVt9lo1t26C0/ge97eXgW7K42qwLlUvsC4cjbw
RfgWXkzi1d7Sh8H9h3Xv6te/xPzpb1Yh+j2Z0q+/eTanLrE2Xl3YdvtTfT+OOGfOwO8RkV5p
f/ZIL1pEBoIminKWfqTSq98p5yJda6+FNJbpNbu9kZ/y58v0ut3+MnMS0a6T9EeSBn7UOWdU
pevtL1o0JkyClyMeRSTI/G0o9YN0yqghzJ9f3Hj+R7F/lcyGO6cu8Xq8+hWy+OR7IiSVirCA
QAqfj7fqWz9JKDubzgF1As6UTxmke+0ivd5GWeqdEFJO18epvXmWbly+n48X3A/S+YwTevUP
fsV/D54QFnZXe4Yu9yYU9JL69xr7zbL+oe49Xf/AN/X/PrD51c5gMNhJRqrP2U6XsmKJb1OE
aJxwofCPD1qHqCd4nK2c+zQiDxUX/hnBBWNKhDZz5r0HzlFEoXLmqDPCKa+tw5IBhyEyJEEP
nNpW7D8hnZiH0klGVg2hTVn4O5h1ZdtWxKFj9LlU1jb27jTrtRqakKdCBKf5Ww/8hV3QtQ7t
glhDUNMs7y+gAJYuELuGBOcKBBOtoyWgQE/ISIJGbx29BlkiKFPYymx49ycwkHct1OMCAwFT
ph/yLsKAgorhc0uLM6HOHlvV0u3xVlVunVq4im1gbGsanBuSHlZi1OkRFfSxzWlYy12CMF9o
wPEzYR8D77Q4JCCJwt+Q0T0huJiYlDFNGVXFKXi5WXJv6ljAUe4nkuQdjKsiSwCstnVQkAAs
Qb51MnTdBd/mybC3dzIM9rfQOMssyeJ2J73W0e+SQvp4Tux0kg4YPZ738aHL7n+D6fJa1Rl/
M/95da+c/7zmLjR+kNQ9z/T/94Gi58qR3MaMDCYNuezGkxEIei8+EsRXBKs+wfCnOY+IxD4L
cRD5UsKaERKSEHdHGUMqmGwSEt4O+YDh0Fd+15fwImDgqOjADkJwtFP4Orb0LVu6XUAshdTO
hbXLRD1OLSRniiUG+fy11KxQLzKGuXSxmVYsMZgdZJeaz9GgOWY3iudHRGxPfo6i2yfAK+kL
huSlFm+9HOfSwQeXYnm3hJ1FMR0sitT0ZAMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDg/8CfwGh
tL49ACgAAA==

--82I3+IH0IqGh5yIs--