[ZODB-Dev] Disappearing volatile attributes in ZODB 3.6

[Jim Fulton]

Terry Jones wrote:
> In trying to debug some code last night and today, I ran across what looks
> to be a problem with ZODB 3.6. I'm not 100% sure, and I'm also not running
> the SVN code (see below). I'm on Mac OS X 10.4.8 with Python 2.4.3 from the
> Darwin Ports collection.
> 
> Briefly, I have a class that puts a volatile _v_xxx attribute on instances
> via a method called from its __init__ method. I make 4 instances of this
> class, and immediately assert(hasattr(instance, '_v_xxx')) on return. All
> these assertions succeed.
> 
> A little later in my program though, another set of asserts shows that two
> of the four volatile attributes are gone.
> 
> So of course I went into the debugger to try to see where/when the
> attributes were disappearing. But in one long trip through the code, the
> attribute I was looking for was still present.
> 
> Then I discovered that if I simply moved a copy of the assert statements up
> two lines in my code, the attributes do not disappear!
> 
> E.g., with this:
> 
>     # assert(hasattr(instance, '_v_xxx'))
>     func1(x)
>     func2(x)
>     assert(hasattr(instance, '_v_xxx'))
> 
> the assertion fails. But with this
> 
>     assert(hasattr(instance, '_v_xxx'))
>     func1(x)
>     func2(x)
>     assert(hasattr(instance, '_v_xxx'))
> 
> everything runs just fine.

Do the functions commit transactions or create savepoints?

> 
> I.e., the presence or absence of a single assert(hasattr(...)) call causes
> a volatile attribute to remain or to disappear. That seemed pretty odd. 

If the instances are saved in the database, it's not odd.  Once an instance
has been saved, it's state, including volatile attributes can be freed.
The more an object is accessed, including by an assert, the less likely that
it's state will be freed.

 > I
> hesitate to send mail here, having learned the hard way that to resist the
> idea that "the system" is probably at fault. But I've spent probably 5
> hours trying to find and analyze this - and I'm wondering if it may be a
> known issue.
> 
> I should add that my class is a subclass of Persistent, but that I remove
> the ZODB file storage file before each test run, so the new instances are
> not being brought out of a persisted state.

Are they saved in the database in the course of the test?

> Seeing as I'm running the stock 3.6 ZODB, I went looking for the latest
> ZODB, and grabbed svn://svn.zope.org/repos/main/ZODB/trunk. The svn log
> includes:
> 
>   r66125 | tseaver | 2006-03-22 16:43:22 +0100 (Wed, 22 Mar 2006) | 3 lines
>   PersistentMapping was inadvertently pickling volatile attributes
>   (http://www.zope.org/Collectors/Zope/2052).
> 
> Which sounded promising, especially seeing as my class does contain a
> PersistentMapping, and this fix is not in the 3.6 tarball. So I fixed that
> by hand in my local 3.6 version and made sure a new created mapping.pyc was
> created in the right place, etc. But after making this change, the problem
> continues.
> 
> So I went to install the latest ZODB. I found 3.7.0b3 at
> http://cheeseshop.python.org/pypi/ZODB3/3.7.0b3 and I've made 3 attempts to
> install this and none of them worked:
> 
>   - using python setup.py install, which gets me raise TypeError, "dist
>     must be a Distribution instance", and there's not much help for this to
>     be found via google.

Sound like a bug.  I'll look at that when I have a chance.

>   - using easy_install, but I couldn't figure out what argument to give it
>     (I tried all of ZODB, ZODB3.7, ZODB3-3.7.0b3, ZODB3-3.7.0b3.xml)

ZODB3.

>   - using zc.buildout, which I installed via easy_install but when I tried
>     reading how to use it at http://www.python.org/pypi/zc.buildout
>     I ended up feeling a little overwhelmed.

I'm sorry to hear that.  I suggest looking first at
http://www.python.org/pypi/zc.buildout#buildout-examples

...

> So I have 2 questions:
> 
>   - Does the error I'm seeing ring any bells with ZODB developers?

Not for me, but you haven't given much information.

Do you understand that volatile attributes should only be used
for caching and that they can go away at pretty much any time?
(In practice, they generally only go away at transaction boundaries
when ZODB does garbage collection.  Sometimes applications
intentionally trigger garbage collection or ghostify objects
to reduce memory.)

>   - Can someone tell me what I'm doing wrong in trying to install the
>     latest ZODB?

See above.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org