I have a Persistent derived class where I want to upgrade from using a PersistentList to a BTrees.IOBTree.IOBTree. _articleList is the old Persistent list _oidsToArticles is the new IOBTree. def __setstate__(self, state): articleList = state.get('_articleList') if articleList: oidsToArticles = self.makeBTree() for each in articleList: oidsToArticles[each.oid()] = each del state['_articleList'] state['_oidsToArticles'] = oidsToArticles Persistent.__setstate__(self, state) This seems to work fine, and I am adding more objects to _oidsToArticles. The problem is that when I commit the transaction, nothing is saved. I have tested the same code without __setstate__ and then it saves just fine. What hoops does one have to jump through to upgrade the state of a ZODB object?
[Syver Enstad]
I have a Persistent derived class where I want to upgrade from using a PersistentList to a BTrees.IOBTree.IOBTree.
_articleList is the old Persistent list _oidsToArticles is the new IOBTree.
def __setstate__(self, state): articleList = state.get('_articleList') if articleList: oidsToArticles = self.makeBTree() for each in articleList: oidsToArticles[each.oid()] = each del state['_articleList'] state['_oidsToArticles'] = oidsToArticles Persistent.__setstate__(self, state)
This seems to work fine, and I am adding more objects to _oidsToArticles. The problem is that when I commit the transaction, nothing is saved.
Quick guess (untested, untried, just from eyeballing this snippet quickly): nothing is marking self itself as being changed. The only things getting changed are the throw-away in-memory 'state' dict, the throw-away in-memory self.__dict__, and a throw-away in-memory BTree. The primary purpose of p.__setstate__ is to activate a ghost, after which point p is in the up-to-date state (i.e., not changed -- calling Persistent.__setstate__(self, ...) isn't going to mark self as changed, and nothing in the code above modifies self itself). If self doesn't look changed, its state in the DB won't change during commit(), and the DB will still reference your old PersistentList. Perhaps shuffling the code around would work, a la: Persistent.__setstate__(self, state) articleList = state.get('_articleList') if articleList is not None: # make the BTree in local oidsToArticles as above, then del self._articlelist self._oidsToArticles = oidsToArticles The thinking there is that then you've truly modified self, after self has been activated.
[Syver Enstad, wants to switch an attribute of a Persitent subclass From PersistentList to an IOBTree] [Tim, guessing]
Quick guess (untested, untried, ... ... Perhaps shuffling the code around would work, a la:
Persistent.__setstate__(self, state) articleList = state.get('_articleList') if articleList is not None: # make the BTree in local oidsToArticles as above, then del self._articlelist self._oidsToArticles = oidsToArticles
The thinking there is that then you've truly modified self, after self has been activated.
Nope, sorry, not a chance. You have truly modified self then, but __setstate__ is called by magic as part of activation (unghostifying), and the activation machinery resets self._p_changed after it calls __setstate__. So same result as your code: the in-memory version of self has the BTree, but commit() doesn't change anything about self in the database (again because self._p_changed is false -- and will be no matter what you try to do in __setstate__()). This appears to be one of those "severely underdocumented" minefields. The best older thread I found is here: http://mail.zope.org/pipermail/zodb-dev/2002-March/002442.html but it doesn't actually spell out something "that works" in the way you want. One possibility is to use any of these __setstate__ implementations temporarily, in a one-shot database conversion script: load every object of your subclass's type, and for each one "obj", after loading it do obj._p_changed = True get_transaction().commit() Note again that setting _p_changed *inside* __setstate__ won't do any good. Note too that Jim Fulton has a recent proposal for Zope3 in this area: http://tinyurl.com/ypkhk [Zope URLs are generally so long my mailer refuses to keep them on one line]
FWIW, __setstate__ for Zope objects has typically been used as a "one-way" kind of backwards compatibility mechanism (where the object's database state remains unmodified, but the in-memory state is changed) not only for the reasons that Tim just explained, but also because changing the persistent state of an object as a side effect of unghosting would can be a bad idea anyway. ZODB writes can be very expensive and having them occur as side effects of loading state from the database could be problematic in some circumstances. This is commonly referred to as the "write on read" pattern and is largely discouraged. Tim pointed at a proposal by Jim which aims to allow for orderly upgrade of database state based on developer-defined schema versions where the upgrade is done at known times (like, when someone presses a button, or at Zope startup time) which seems saner. There is an (experimental) Zope 2 product that implements the pattern in Jim's proposal (using mostly the same code that Zope 3 uses for its experimental implementation of the same) available at http://cvs.zope.org/Products/zzz_generations/ On Fri, 2004-05-28 at 14:51, Tim Peters wrote:
[Syver Enstad, wants to switch an attribute of a Persitent subclass From PersistentList to an IOBTree]
[Tim, guessing]
Quick guess (untested, untried, ... ... Perhaps shuffling the code around would work, a la:
Persistent.__setstate__(self, state) articleList = state.get('_articleList') if articleList is not None: # make the BTree in local oidsToArticles as above, then del self._articlelist self._oidsToArticles = oidsToArticles
The thinking there is that then you've truly modified self, after self has been activated.
Nope, sorry, not a chance. You have truly modified self then, but __setstate__ is called by magic as part of activation (unghostifying), and the activation machinery resets self._p_changed after it calls __setstate__. So same result as your code: the in-memory version of self has the BTree, but commit() doesn't change anything about self in the database (again because self._p_changed is false -- and will be no matter what you try to do in __setstate__()).
This appears to be one of those "severely underdocumented" minefields. The best older thread I found is here:
http://mail.zope.org/pipermail/zodb-dev/2002-March/002442.html
but it doesn't actually spell out something "that works" in the way you want.
One possibility is to use any of these __setstate__ implementations temporarily, in a one-shot database conversion script: load every object of your subclass's type, and for each one "obj", after loading it do
obj._p_changed = True get_transaction().commit()
Note again that setting _p_changed *inside* __setstate__ won't do any good.
Note too that Jim Fulton has a recent proposal for Zope3 in this area:
[Zope URLs are generally so long my mailer refuses to keep them on one line]
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Tim Peters wrote at 2004-5-28 14:51 -0400:
... This appears to be one of those "severely underdocumented" minefields. The best older thread I found is here:
http://mail.zope.org/pipermail/zodb-dev/2002-March/002442.html
but it doesn't actually spell out something "that works" in the way you want.
I think, this is a ZODB buglet: It should set "_p_changed = 0" before it calls "__setstate__" and not afterwards... -- Dieter
[Dieter Maurer]
I think, this is a ZODB buglet:
It should set "_p_changed = 0" before it calls "__setstate__" and not afterwards...
I don't know; Jim (or Jeremy) may know the reasoning here, but I don't. Activation currently sets the state to changed *before* calling __setstate__ too. A comment says this is to prevent recursively calling _PyPersist_Load, and that makes sense to me, else unbounded recursion could occur. Given this abuse of _p_changed, and that the purpose of activating a ghost is to transition it to the up-to-date state, making mutations to the persistent state inside __setstate__ seems hard to accommodate. On ZODB head, Jeremy also documented that __setstate__ can't affect the persistent state (in persistent/interfaces.py's IPersistent). Whether this is deliberate design, or a consequence of the current implementation, I can't say.
"Tim Peters" <tim@zope.com> writes:
[Syver Enstad, wants to switch an attribute of a Persitent subclass From PersistentList to an IOBTree]
[Tim, guessing]
Quick guess (untested, untried, ... ... Perhaps shuffling the code around would work, a la:
Persistent.__setstate__(self, state) articleList = state.get('_articleList') if articleList is not None: # make the BTree in local oidsToArticles as above, then del self._articlelist self._oidsToArticles = oidsToArticles
The thinking there is that then you've truly modified self, after self has been activated.
Nope, sorry, not a chance. You have truly modified self then, but __setstate__ is called by magic as part of activation (unghostifying), and the activation machinery resets self._p_changed after it calls __setstate__. So same result as your code: the in-memory version of self has the BTree, but commit() doesn't change anything about self in the database (again because self._p_changed is false -- and will be no matter what you try to do in __setstate__()).
I was aware that __setstate__ doesn't allow me to commit my changes after only loading the object into memory (__setstate__ is called). I may have explained myself unclearly (not a native english speaker/writer), the problem is that it won't save my changes when I add more items to the _oidsToArticles BTree long after __setstate__ time. Example code: # articleDb is an ArticleDb and has the __setstate__ method articleDb = connection.root()['articledb'] for each in articleDb.articles() # loop through all article print each computer = Computer(1030) articleDb.addArticle(computer) connection.commit() # nothing is saved Everything works smoothly until connection.commit(). The __setstate__ method correctly converts to using the BTree and the addArticle method adds a new computer to ArticleDb. If I instead create a new ArticleDb instance with the BTree instead of using __setstate__ the new computer is saved as it should. In none of the cases is articleDb._p_changed == True, but when the ArticleDb is created "cleanly" instead of upgraded, commit works anyway. Since this works if the object that holds _oidsToArticles is created from scratch, it seems that ZODB isn't properly aware of the _oidsToArticles attribute since it was created in __setstate__ instead of in __init__. I would really like to know exactly what happens here as the reason I am utilizing an object database is to be able to refactor the database with a minimum of fuss, and I would expect __setstate__ to allow me to make additional attributes in my instances.
[Syver Enstad]
I was aware that __setstate__ doesn't allow me to commit my changes after only loading the object into memory (__setstate__ is called). I may have explained myself unclearly (not a native english speaker/writer),
I don't think that matters much: English instead of code is always ambiguous, even for native writers.
the problem is that it won't save my changes when I add more items to the _oidsToArticles BTree long after __setstate__ time.
Right. I tried to explain that in my first reply. It's expected. You have to get _p_changed set on your *object*. It doesn't matter how much you mutate new objects attached *to* the object you loaded, you have to get _p_changed set to 1 on the original object.
Example code:
# articleDb is an ArticleDb and has the __setstate__ method articleDb = connection.root()['articledb']
At this point articleDb._p_changed is not 1, and nothing you do later changes this fact.
for each in articleDb.articles() # loop through all article print each
It's obvious that this doesn't set articleDb._p_changed, right?
computer = Computer(1030) articleDb.addArticle(computer)
That adds something to the BTree that is bound to an attribute of articleDb, but still doesn't set _p_changed on articleDb (mutating the BTree does not mutate the object containing the BTree). So far as the *database* is concerned, there's still no path from the root object to the BTree your __setstate__() created, so the BTree never gets committed. If you do articleDb._p_changed = 1 at this point too, then your changes will get stored.
connection.commit() # nothing is saved
Right. As far as the persistence machinery is concerned, articleDb itself never changed, so there was no reason to store it. Since it didn't get stored, the BTree you attached to it in __setstate__ doesn't get stored either.
Everything works smoothly until connection.commit(). The __setstate__ method correctly converts to using the BTree and the addArticle method adds a new computer to ArticleDb.
The problem is that, while adding the BTree to articleDb did change (did set _p_changed to 1) articleDb, because that happened as a side effect of loading the object, the unghostification machinery clears _p_changed after __setstate__ returns. Then nothing you did after that set articleDb._p_changed to a true value.
If I instead create a new ArticleDb instance with the BTree instead of using __setstate__ the new computer is saved as it should.
Yes. If you had called __setstate__ *directly*, it would also work (contradicting the current minimal docs, but so it goes). The problem is that unghostifying "by magic" clears _p_changed.
In none of the cases is articleDb._p_changed == True,
When you create an object directly, it's not associated with a Connection (a "jar"), and some of the persistence machinery is sidestepped then. New objects only get into the database if you attach them to an object that's already in the database, and the latter object is marked changed. Think about that <wink>, and the purpose of the root() object will become clearer.
but when the ArticleDb is created "cleanly" instead of upgraded, commit works anyway.
Sorry, I'm not clear on what that means. In any case, it doesn't really matter how an object gets created, what matters is whether the persistence machinery "sees" a path to it from a changed object already in the database.
Since this works if the object that holds _oidsToArticles is created from scratch, it seems that ZODB isn't properly aware of the _oidsToArticles attribute since it was created in __setstate__ instead of in __init__.
Both ways set _p_changed *at the time* oidsToArticles is bound to the new BTree. The difference is that uhghostification deliberately clears _p_changed after calling __setstate__.
I would really like to know exactly what happens here as the reason I am utilizing an object database is to be able to refactor the database with a minimum of fuss, and I would expect __setstate__ to allow me to make additional attributes in my instances.
Calling __setstate__ as a side effect of a persistent load does not, by itself, support making changes that persist. It apparently wasn't designed to, either (see the other msgs in this thread, particularly from ChrisM).
"Tim Peters" <tim@zope.com> writes:
That adds something to the BTree that is bound to an attribute of articleDb, but still doesn't set _p_changed on articleDb (mutating the BTree does not mutate the object containing the BTree). So far as the *database* is concerned, there's still no path from the root object to the BTree your __setstate__() created, so the BTree never gets committed.
So the BTree is not "registered" with the database until _p_changed and a commit? After that first commit, the BTree is "registered" as reachable and then when just the BTree is _p_changed I get my changes saved. Right?
Calling __setstate__ as a side effect of a persistent load does not, by itself, support making changes that persist. It apparently wasn't designed to, either (see the other msgs in this thread, particularly from ChrisM).
I can understand the reasoning behind it, in the way that you don't want nothing to change on disk when you logically haven't called a mutating method on your object, just loaded it. But I do think it is badly broken to not save the changes when I have made additional changes to the state of the object after load time. Why not just take the performance hit and be done with it? What do other object databases do with schema migration issues, f.ex. GemStone? Now well, I believe I understand why this happens and that is a very good thing! Thanks to all who have helped me in gaining a better understanding of ZODB.
Syver Enstad wrote at 2004-5-28 12:41 +0200:
I have a Persistent derived class where I want to upgrade from using a PersistentList to a BTrees.IOBTree.IOBTree.
_articleList is the old Persistent list _oidsToArticles is the new IOBTree.
def __setstate__(self, state): articleList = state.get('_articleList') if articleList: oidsToArticles = self.makeBTree() for each in articleList: oidsToArticles[each.oid()] = each del state['_articleList'] state['_oidsToArticles'] = oidsToArticles Persistent.__setstate__(self, state)
This seems to work fine, and I am adding more objects to _oidsToArticles. The problem is that when I commit the transaction, nothing is saved. I have tested the same code without __setstate__ and then it saves just fine. What hoops does one have to jump through to upgrade the state of a ZODB object?
This does not work as changes made in "__setstate__" are not explicitly persisted. Whether or not they become persistent depends on whether the object is changed again outside "__setstate__". Use an explicit conversion script instead... -- Dieter
participants (5)
-
Chris McDonough -
Dieter Maurer -
Syver Enstad -
Tim Peters -
Tim Peters