[ZODB-Dev] BTrees and Mutables, was Re: [IndexedCatalog] bug in default indexing

[Shane Hathaway]

Christian Reis wrote:
> On Tue, Feb 11, 2003 at 03:24:05PM -0500, Shane Hathaway wrote:
> 
>>Casey Duncan wrote:
>>
>>>An alternative which is perhaps more palatable is to just set the whole 
>>>mutable object into the btree again. Like:
>>>
>>>l = tree['bar']
>>>l.append(4)
>>>tree['bar'] = l # Trigger list persistance
>>
>>In fact, if you put simple lists in BTrees, you have to use this 
>>pattern.  Setting tree._p_changed will not get the right object stored. 
> 
> 
> Whoa. You mean to say that:
> 
>     l = tree['bar']
>     l.append(4)
>     tree['bar'] = l
> 
> is *not* the same as:
> 
>     tree['bar'].append(4)
> 
> right? And the first form *does* work as expected, or are there
> side-effects?

That's correct, although the purist way is like this:

     l = tree['bar']
     tree['bar'] = l
     l.append(4)

That way the notification happens before the actual change.  If an 
exception happens and the transaction is aborted, the changed list will 
be reverted.

> This *really* should be documented on the BTrees page - I've never heard
> it mentioned and it is biting us badly inside IC (since we can't use
> Persistent* for performance reasons). Can anybody offer a bit of
> explanation as to why so I can put up on the page a reasonable
> explanation beyond "don do dat"?

This isn't specific to BTrees.  It's a ZODB wart, and it's explained in 
the Zope Developer's Guide: if you store mutable, non-persistent objects 
in ZODB, you have to do special things to get ZODB to see changes to 
them.  I can understand how you might miss it, since it's such a big 
surprise.

Under the hood, a BTree consists of many ZODB records.  If you set 
tree._p_changed = 1, you're just storing a small object that refers to 
the real data.  But if you set a key in a BTree, even if you set it to 
the same value it had before, the BTree will store the correct object.

I've tried to come up with ways to change Python to fix this nasty 
issue.  Python provides no object mutation notification mechanism.  The 
solution, whatever it might be, must have zero impact on non-ZODB software.

Shane