Has anyone run into this one? ------ 2003-12-10T14:07:53 ERROR(200) Zope Couldn't install Formulator Traceback (most recent call last): File "/var/src/zope/head/lib/python/Zope/Startup/run.py", line 27, in ? run() File "/var/src/zope/head/lib/python/Zope/Startup/run.py", line 24, in run start_zope(opts.configroot) File "/var/src/zope/head/lib/python/Zope/Startup/__init__.py", line 50, in start_zope starter.startZope() File "/var/src/zope/head/lib/python/Zope/Startup/__init__.py", line 218, in startZope Zope.startup() File "/var/src/zope/head/lib/python/Zope/__init__.py", line 46, in startup _startup() File "/var/src/zope/head/lib/python/Zope/App/startup.py", line 95, in startup OFS.Application.initialize(application) File "/var/src/zope/head/lib/python/OFS/Application.py", line 420, in initialize install_products(app) File "/var/src/zope/head/lib/python/OFS/Application.py", line 579, in install_products folder_permissions, raise_exc=debug_mode) File "/var/src/zope/head/lib/python/OFS/Application.py", line 694, in install_product get_transaction().commit() File "/var/src/zope/head/lib/python/ZODB/Transaction.py", line 254, in commit self._cleanup(objects, ncommitted, jars, subjars) File "/var/src/zope/head/lib/python/ZODB/Transaction.py", line 233, in commit ncommitted += self._commit_objects(objects) File "/var/src/zope/head/lib/python/ZODB/Transaction.py", line 352, in _commit_objects jar.commit(o, self) File "/var/src/zope/head/lib/python/ZODB/Connection.py", line 279, in commit p = w.serialize(obj) File "/var/src/zope/head/lib/python/ZODB/serialize.py", line 114, in serialize return self._dump(getClassMetadata(obj), obj.__getstate__()) File "/var/src/zope/head/lib/python/ZODB/serialize.py", line 123, in _dump self._p.dump(state) ZODB.POSException.InvalidObjectReference: Attempt to store an object from a foreign database connection -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher Don't hit the keys so hard, it hurts.
On Wed, 2003-12-10 at 11:14, Sidnei da Silva wrote:
Has anyone run into this one? [ traceback snipped ] ZODB.POSException.InvalidObjectReference: Attempt to store an object from a foreign database connection
Do you actually have more than one database connection? In other words, is the error totally nonsensical or could the bug be that something is getting stored in a foreign database connection? Jeremy
The default Zope setup has two databases: the main database and the session database. So it's not completely nonsensical for any default-configured Zope to raise the error (although I have no idea how this error is occurring). It might be worthwhile to specify a main database explicitly in zope.conf. This will prevent the default database setup from being invoked and then you will be sure that you only have one database connection. On Wed, 2003-12-10 at 11:19, Jeremy Hylton wrote:
On Wed, 2003-12-10 at 11:14, Sidnei da Silva wrote:
Has anyone run into this one? [ traceback snipped ] ZODB.POSException.InvalidObjectReference: Attempt to store an object from a foreign database connection
Do you actually have more than one database connection? In other words, is the error totally nonsensical or could the bug be that something is getting stored in a foreign database connection?
Jeremy
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Wed, Dec 10, 2003 at 11:32:27AM -0500, Chris McDonough wrote: | The default Zope setup has two databases: the main database and the | session database. So it's not completely nonsensical for any | default-configured Zope to raise the error (although I have no idea how | this error is occurring). It might be worthwhile to specify a main | database explicitly in zope.conf. This will prevent the default | database setup from being invoked and then you will be sure that you | only have one database connection. If I don't specify the temp_folder mount database, I get this error: 2003-12-10T14:37:46 ERROR(200) ZODB Failed to mount database. DBTab.Exceptions.DBTabConfigurationError (No database configured for mount point at /temp_folder) Traceback (most recent call last): File "/var/src/zope/head/lib/python/Products/ZODBMountPoint/Mount.py", line 86, in _getOrOpenObject conn = self._getMountedConnection(anyjar) File "/var/src/zope/head/lib/python/Products/ZODBMountPoint/Mount.py", line 63, in _getMountedConnection db_name = self._getDBName() File "/var/src/zope/head/lib/python/Products/ZODBMountPoint/MountedObject.py", line 154, in _getDBName return getConfiguration().getDatabaseFactory(self._path).getName() File "/var/src/zope/head/lib/python/DBTab/DBTab.py", line 107, in getDatabaseFactory self._mountPathError(mount_path) File "/var/src/zope/head/lib/python/DBTab/DBTab.py", line 71, in _mountPathError raise DBTabConfigurationError( DBTabConfigurationError: No database configured for mount point at /temp_folder Traceback (most recent call last): File "/var/src/zope/head/lib/python/Zope/Startup/run.py", line 27, in ? run() File "/var/src/zope/head/lib/python/Zope/Startup/run.py", line 24, in run start_zope(opts.configroot) File "/var/src/zope/head/lib/python/Zope/Startup/__init__.py", line 50, in start_zope starter.startZope() File "/var/src/zope/head/lib/python/Zope/Startup/__init__.py", line 218, in startZope Zope.startup() File "/var/src/zope/head/lib/python/Zope/__init__.py", line 46, in startup _startup() File "/var/src/zope/head/lib/python/Zope/App/startup.py", line 95, in startup OFS.Application.initialize(application) File "/var/src/zope/head/lib/python/OFS/Application.py", line 348, in initialize tf._setObject('session_data', toc) AttributeError: _setObject -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher "Nuclear war can ruin your whole compile." -- Karl Lehenbauer
Right. Ugh. I need to fix this. To work around if for the meantime, you could comment out Application.py's line 348 and other related lines around it. On Wed, 2003-12-10 at 11:45, Sidnei da Silva wrote:
On Wed, Dec 10, 2003 at 11:32:27AM -0500, Chris McDonough wrote: | The default Zope setup has two databases: the main database and the | session database. So it's not completely nonsensical for any | default-configured Zope to raise the error (although I have no idea how | this error is occurring). It might be worthwhile to specify a main | database explicitly in zope.conf. This will prevent the default | database setup from being invoked and then you will be sure that you | only have one database connection.
If I don't specify the temp_folder mount database, I get this error:
2003-12-10T14:37:46 ERROR(200) ZODB Failed to mount database. DBTab.Exceptions.DBTabConfigurationError (No database configured for mount point at /temp_folder) Traceback (most recent call last): File "/var/src/zope/head/lib/python/Products/ZODBMountPoint/Mount.py", line 86, in _getOrOpenObject conn = self._getMountedConnection(anyjar) File "/var/src/zope/head/lib/python/Products/ZODBMountPoint/Mount.py", line 63, in _getMountedConnection db_name = self._getDBName() File "/var/src/zope/head/lib/python/Products/ZODBMountPoint/MountedObject.py", line 154, in _getDBName return getConfiguration().getDatabaseFactory(self._path).getName() File "/var/src/zope/head/lib/python/DBTab/DBTab.py", line 107, in getDatabaseFactory self._mountPathError(mount_path) File "/var/src/zope/head/lib/python/DBTab/DBTab.py", line 71, in _mountPathError raise DBTabConfigurationError( DBTabConfigurationError: No database configured for mount point at /temp_folder Traceback (most recent call last): File "/var/src/zope/head/lib/python/Zope/Startup/run.py", line 27, in ? run() File "/var/src/zope/head/lib/python/Zope/Startup/run.py", line 24, in run start_zope(opts.configroot) File "/var/src/zope/head/lib/python/Zope/Startup/__init__.py", line 50, in start_zope starter.startZope() File "/var/src/zope/head/lib/python/Zope/Startup/__init__.py", line 218, in startZope Zope.startup() File "/var/src/zope/head/lib/python/Zope/__init__.py", line 46, in startup _startup() File "/var/src/zope/head/lib/python/Zope/App/startup.py", line 95, in startup OFS.Application.initialize(application) File "/var/src/zope/head/lib/python/OFS/Application.py", line 348, in initialize tf._setObject('session_data', toc) AttributeError: _setObject
On Wed, Dec 10, 2003 at 12:19:47PM -0500, Chris McDonough wrote: | Right. Ugh. I need to fix this. To work around if for the meantime, | you could comment out Application.py's line 348 and other related lines | around it. Ok, commented out 292-360, and now I'm back to the previous traceback. So, it seems to be happening with only one database open. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher Linux - Where do you want to fly today? -- Unknown source
And you see only the "main" database in Control_Panel->Databases? On Wed, 2003-12-10 at 12:35, Sidnei da Silva wrote:
On Wed, Dec 10, 2003 at 12:19:47PM -0500, Chris McDonough wrote: | Right. Ugh. I need to fix this. To work around if for the meantime, | you could comment out Application.py's line 348 and other related lines | around it.
Ok, commented out 292-360, and now I'm back to the previous traceback. So, it seems to be happening with only one database open.
On Wed, Dec 10, 2003 at 12:36:53PM -0500, Chris McDonough wrote: | And you see only the "main" database in Control_Panel->Databases? No, I can't get this far. The error happens on startup. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher %DCL-MEM-BAD, bad memory VMS-F-PDGERS, pudding between the ears
Right. ;-) At this point I should just try it myself but if you remove the Formulator product, does Zope start? On Wed, 2003-12-10 at 12:52, Sidnei da Silva wrote:
On Wed, Dec 10, 2003 at 12:36:53PM -0500, Chris McDonough wrote: | And you see only the "main" database in Control_Panel->Databases?
No, I can't get this far. The error happens on startup.
On Wed, Dec 10, 2003 at 12:50:29PM -0500, Chris McDonough wrote: | Right. ;-) At this point I should just try it myself but if you remove | the Formulator product, does Zope start? If I remove Plone and Formulator it starts yes. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher Equal bytes for women.
On Wed, 2003-12-10 at 13:07, Sidnei da Silva wrote:
On Wed, Dec 10, 2003 at 12:50:29PM -0500, Chris McDonough wrote: | Right. ;-) At this point I should just try it myself but if you remove | the Formulator product, does Zope start?
If I remove Plone and Formulator it starts yes.
Probably a good next step is to set a breakpoint in ZODB/coptimizations.c where this exception is raised and find out what object is causing the problem, which jar it's already in, and why it's getting added to a different jar. Or you could add some print statements. Jeremy
| Probably a good next step is to set a breakpoint in | ZODB/coptimizations.c where this exception is raised and find out what | object is causing the problem, which jar it's already in, and why it's | getting added to a different jar. Or you could add some print | statements. I figured out how to set a breakpoint in gdb, but I don't know how to get at the object and inspect it. Can I get a quick hint? I know there has to be some pointer cast probably, just need to know which one. :) -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher fortune: cannot execute. Out of cookies.
On Wed, 2003-12-10 at 14:01, Sidnei da Silva wrote:
| Probably a good next step is to set a breakpoint in | ZODB/coptimizations.c where this exception is raised and find out what | object is causing the problem, which jar it's already in, and why it's | getting added to a different jar. Or you could add some print | statements.
I figured out how to set a breakpoint in gdb, but I don't know how to get at the object and inspect it. Can I get a quick hint? I know there has to be some pointer cast probably, just need to know which one. :)
I think you're looking for the helpers defined in Misc/gdbinit in the Python distribution. If you add them to your .gdbinit, you'll be golden. I've included that file's contents below so you don't have to go hunting. # -*- ksh -*- # # If you use the GNU debugger gdb to debug the Python C runtime, you # might find some of the following commands useful. Copy this to your # ~/.gdbinit file and it'll get loaded into gdb automatically when you # start it up. Then, at the gdb prompt you can do things like: # # (gdb) pyo apyobjectptr # <module 'foobar' (built-in)> # refcounts: 1 # address : 84a7a2c # $1 = void # (gdb) # Prints a representation of the object to stderr, along with the # number of reference counts it current has and the hex address the # object is allocated at. The argument must be a PyObject* define pyo print _PyObject_Dump($arg0) end # Prints a representation of the object to stderr, along with the # number of reference counts it current has and the hex address the # object is allocated at. The argument must be a PyGC_Head* define pyg print _PyGC_Dump($arg0) end # If you are in an eval_frame() function, calling pyframe with no # arguments will print the filename, function name, and line number. # It assumes that f is the name of the current frame. define pyframe x/s ((PyStringObject*)f->f_code->co_filename)->ob_sval x/s ((PyStringObject*)f->f_code->co_name)->ob_sval p f->f_lineno end
| I think you're looking for the helpers defined in Misc/gdbinit in the | Python distribution. If you add them to your .gdbinit, you'll be | golden. | | I've included that file's contents below so you don't have to go | hunting. Neat! There we have 'object', 'self' and 'self->jar', as of line 183 of coptimizations.c: object : <class 'Products.Formulator.StandardFields.LabelField'> type : ExtensionClass.ExtensionClass refcount: 7 address : 0x8503bec object : <persistent_id object at 0x415b87b8> type : persistent_id refcount: 1 address : 0x415b87b8 object : <Connection at 414b80ac> type : Connection refcount: 238 address : 0x414b80ac 'jar' gives: object : <attribute '_p_jar' of 'persistent.Persistent' objects> type : getset_descriptor refcount: 2 address : 0x40621e6c Does that help? -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher [We] use bad software and bad machines for the wrong things. -- R.W. Hamming
On Wed, 2003-12-10 at 14:18, Sidnei da Silva wrote:
'jar' gives: object : <attribute '_p_jar' of 'persistent.Persistent' objects> type : getset_descriptor refcount: 2 address : 0x40621e6c
Does that help?
Yes. The code is getting an unexpected raw descriptor. Someone should be calling __get__() on this descriptor to get the value. I'll have to look into it to see how it's going wrong. Jeremy
On Wed, Dec 10, 2003 at 02:17:23PM -0500, Jeremy Hylton wrote: | On Wed, 2003-12-10 at 14:18, Sidnei da Silva wrote: | > 'jar' gives: | > object : <attribute '_p_jar' of 'persistent.Persistent' objects> | > type : getset_descriptor | > refcount: 2 | > address : 0x40621e6c | > | > Does that help? | | Yes. The code is getting an unexpected raw descriptor. Someone should | be calling __get__() on this descriptor to get the value. I'll have to | look into it to see how it's going wrong. If I can do anything to help, just ask. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher The program isn't debugged until the last user is dead.
Actually, there are two equally valuable clues from the debugger session: object : <class 'Products.Formulator.StandardFields.LabelField'> type : ExtensionClass.ExtensionClass refcount: 7 address : 0x8503bec If I understand correctly: LabelField is a class that inherits from Persistence and some other object has a reference to it that isn't via __class__.
persistent.Persistent._p_jar <attribute '_p_jar' of 'persistent.Persistent' objects>
If you ask a class for an attribute, it seems you can get a descriptor back. I vaguely recall Guido saying that this was the intended behavior last time I asked about it. 'jar' gives: object : <attribute '_p_jar' of 'persistent.Persistent' objects> type : getset_descriptor refcount: 2 address : 0x40621e6c Thus, asking for the class's _p_jar gives the descriptor back. In ZODB4 I hacked around this by requiring that the value of _p_jar be None or a string. Would there be any objection to having the same requirement in ZODB3? Jeremy
On Wed, Dec 10, 2003 at 03:26:03PM -0500, Jeremy Hylton wrote: | Actually, there are two equally valuable clues from the debugger | session: | | object : <class 'Products.Formulator.StandardFields.LabelField'> | type : ExtensionClass.ExtensionClass | refcount: 7 | address : 0x8503bec | | If I understand correctly: LabelField is a class that inherits from | Persistence and some other object has a reference to it that isn't via | __class__. Makes sense to me. How can I find the referencing object? | >>> persistent.Persistent._p_jar | <attribute '_p_jar' of 'persistent.Persistent' objects> | | If you ask a class for an attribute, it seems you can get a descriptor | back. I vaguely recall Guido saying that this was the intended behavior | last time I asked about it. Yes, I've learnt this the hard way, when Jim tried to explain me and I kept banging my head against the wall :) | 'jar' gives: | object : <attribute '_p_jar' of 'persistent.Persistent' objects> | type : getset_descriptor | refcount: 2 | address : 0x40621e6c | | Thus, asking for the class's _p_jar gives the descriptor back. In ZODB4 | I hacked around this by requiring that the value of _p_jar be None or a | string. Would there be any objection to having the same requirement in | ZODB3? I have no clue about the implications of this, but it seems like a reasonable constraint to me. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher MS-DOS, you can't live with it, you can live without it. -- from Lars Wirzenius' .sig
On Wed, 2003-12-10 at 15:36, Sidnei da Silva wrote:
| 'jar' gives: | object : <attribute '_p_jar' of 'persistent.Persistent' objects> | type : getset_descriptor | refcount: 2 | address : 0x40621e6c | | Thus, asking for the class's _p_jar gives the descriptor back. In ZODB4 | I hacked around this by requiring that the value of _p_jar be None or a | string. Would there be any objection to having the same requirement in | ZODB3?
I have no clue about the implications of this, but it seems like a reasonable constraint to me.
Actually, that's just a brain cramp on my part. We can check that an oid is a string or None, but not a jar. I'll have to think about an efficient check. Jeremy
I think there's a fix for this in CVS. Can you try it again? Jeremy
On Thu, Dec 11, 2003 at 10:07:51AM -0500, Jeremy Hylton wrote: | I think there's a fix for this in CVS. Can you try it again? Just tried. Seems to have fixed it. Now, for the next step *wink*. If I try starting Zope with an old existing database I get the traceback below. Correct me if I'm wrong, but I think the error with 'UserDict' has nothing to do with the 'Could not import class 'BTree'' or am I wrong? BTW, I've hit a similar problem (AttributeError on 'data') recently on Zope 3, and I thought it could be my fault. ,----[ Python Info ] | sidnei@moria:~/src/instance/blog$ python | Python 2.3.3c1 (#2, Dec 6 2003, 16:44:56) | [GCC 3.3.3 20031203 (prerelease) (Debian)] on linux2 `---- ,----[ Traceback ] | | 2003-12-11T13:14:51 PROBLEM(100) ZODB Could not import class 'BTree' | from module 'BTree' | Traceback (most recent call last): | File "/var/src/zope/head/lib/python/Zope/Startup/run.py", line 27, | in ? | run() | File "/var/src/zope/head/lib/python/Zope/Startup/run.py", line 24, | in run | start_zope(opts.configroot) | File "/var/src/zope/head/lib/python/Zope/Startup/__init__.py", line | 50, in start_zope | starter.startZope() | File "/var/src/zope/head/lib/python/Zope/Startup/__init__.py", line | 218, in startZope | Zope.startup() | File "/var/src/zope/head/lib/python/Zope/__init__.py", line 46, in | startup | _startup() | File "/var/src/zope/head/lib/python/Zope/App/startup.py", line 90, | in startup | Globals.VersionNameName) | File "/var/src/zope/head/lib/python/ZODB/ZApplication.py", line 32, | in __init__ | if not root.has_key(name): | File "/usr/lib/python2.3/UserDict.py", line 41, in has_key | def has_key(self, key): return self.data.has_key(key) | AttributeError: data `---- -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher Type louder, please.
On Thu, 2003-12-11 at 10:24, Sidnei da Silva wrote:
On Thu, Dec 11, 2003 at 10:07:51AM -0500, Jeremy Hylton wrote: | I think there's a fix for this in CVS. Can you try it again?
Just tried. Seems to have fixed it. Now, for the next step *wink*.
If I try starting Zope with an old existing database I get the traceback below. Correct me if I'm wrong, but I think the error with 'UserDict' has nothing to do with the 'Could not import class 'BTree'' or am I wrong?
I think the attribute error on data is just a symptom of the BTree failure. The object has no data attribute, because ZODB failed to unpickle the object that would be assigned to data. I don't know what happened with backwards compatibility for the old BTree classes. When Jim and I discussed it a while back, I thought we concluded that we would provide a script that would convert all the old BTree instances to new BTrees instance. But I don't know if that decision stuck or if the script was ever written. Jeremy
[Jeremy]
I think the attribute error on data is just a symptom of the BTree failure. The object has no data attribute, because ZODB failed to unpickle the object that would be assigned to data.
I don't know what happened with backwards compatibility for the old BTree classes. When Jim and I discussed it a while back, I thought we concluded that we would provide a script that would convert all the old BTree instances to new BTrees instance. But I don't know if that decision stuck or if the script was ever written.
Jim asked about it on zope-dev, and got no response; see: "Can we forsake database backward compatability on the head for a while?" http://mail.zope.org/pipermail/zope-dev/2003-November/021055.html ... > This would mean that old databases would not be useable with the > CVS head in the near term. Would this cause anyone any problems? I'm happy no one responded to this. Silence is assent, :)
| Jim asked about it on zope-dev, and got no response; see: | | "Can we forsake database backward compatability on the head for | a while?" | | http://mail.zope.org/pipermail/zope-dev/2003-November/021055.html | | ... | | > This would mean that old databases would not be useable with the | > CVS head in the near term. Would this cause anyone any problems? | | I'm happy no one responded to this. Silence is assent, :) :P Is there any special trick to such script? Provided some pointers of what the script should do, I wouldn't mind writing and testing it. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher baz bat bamus batis bant. -- James Troup
Jim asked about it on zope-dev, and got no response; see:
"Can we forsake database backward compatability on the head for a while?"
http://mail.zope.org/pipermail/zope-dev/2003-November/021055.html
[Sidnei da Silva]
Is there any special trick to such script? Provided some pointers of what the script should do, I wouldn't mind writing and testing it.
I'm not sure Jim had a script in mind. For example, he may have wanted to convert "old" data structures to "new" ones magically when they're unpickled (and then when they're written again, the new data structure would get written out, so that the next load wouldn't suffer conversion expense again). I don't know, but Jim's Somehow, Zope 2.8 will convert these automatically reads as if he doesn't want to force people to "run a script" to do explicit database conversion. I have to expect that writing a script would be the most straightforward approach at this time, though. While I've maintained ZODB's "new" BTrees for the last couple years, I've never looked at "the old" BTree code, so I'm not even sure what the possible complications are. I *suspect* a script to convert old BTree instances to new ones would be straightforward, but don't know enough about the old ones to be sure, and am booked solid on different projects now. Anyone? Since Sidnei's efforts to use the head are exposing problems, it would be great if we could keep him obsessed <wink> with it a bit longer.
On Thu, 2003-12-11 at 15:28, Tim Peters wrote:
Anyone? Since Sidnei's efforts to use the head are exposing problems, it would be great if we could keep him obsessed <wink> with it a bit longer.
Just point Sidnei torwards (sp?) some pretty API docs about old and new BTrees and he'll be happy as an otter. He'll come up with a conversion script in no time :-) []'s -- Ideas don't stay in some minds very long because they don't like solitary confinement.
On Thu, 2003-12-11 at 13:16, Leonardo Rochael Almeida wrote:
On Thu, 2003-12-11 at 15:28, Tim Peters wrote:
Anyone? Since Sidnei's efforts to use the head are exposing problems, it would be great if we could keep him obsessed <wink> with it a bit longer.
Just point Sidnei torwards (sp?) some pretty API docs about old and new BTrees and he'll be happy as an otter. He'll come up with a conversion script in no time :-)
I don't know if there is any documentation for BTree, so the code will need to be your guide. I checked the code and each of the BTree types does support items(). So walk over the database, find each top-level BTree object, extract its items, put them in a new BTrees-style BTree, then save that in place of the old object. Jeremy
[Jeremy Hylton]
I don't know if there is any documentation for BTree,
I don't know of any dcos for (the old) BTree either.
so the code will need to be your guide. I checked the code and each of the BTree types does support items(). So walk over the database, find each top-level BTree object, extract its items, put them in a new BTrees-style BTree, then save that in place of the old object.
Then it may not be necessary to extract the items -- a constructor for a BTrees-style BTree object: /** ** Accepts a sequence of 2-tuples, or any object with an items() ** method that returns an iterable object producing 2-tuples. */ IOW, the constructor should invoke .items() for you -- just pass it the old-style BTree. Of course this has to be done under a version of ZODB that still *has* old-style BTree support! I don't know whether the old-style BTree comes in different type-dependent flavors, either. If not, an old BTree should probably become a new OOBTree, and an old IntSet a new IISet.
| Then it may not be necessary to extract the items -- a constructor for a | BTrees-style BTree object: | | /** | ** Accepts a sequence of 2-tuples, or any object with an items() | ** method that returns an iterable object producing 2-tuples. | */ Neat! I'm glad I didn't started on it as Leonardo suggested ;) | IOW, the constructor should invoke .items() for you -- just pass it the | old-style BTree. Of course this has to be done under a version of ZODB that | still *has* old-style BTree support! Do'h, that's a bit difficult since Jim removed them for good :( Now, back to the automagically-converting idea, would it be possible to provide an alias for the old BTree package that would feed the old pickle into a new BTrees constructor on unpickling *wink*? -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher fortune: cannot execute. Out of cookies.
On Thu, 2003-12-11 at 15:15, Sidnei da Silva wrote:
Do'h, that's a bit difficult since Jim removed them for good :(
Hmmm... I suppose you could write the script in Zope 2.7 and tell people they have to use that version of Zope to convert the database before upgrading.
Now, back to the automagically-converting idea, would it be possible to provide an alias for the old BTree package that would feed the old pickle into a new BTrees constructor on unpickling *wink*?
I just remembered why the problem is harder. Any object that references as BTree has the name of the BTree class embedded in its pickle. The conversion script needs to frob each of those database records to refer to the new class. I'm not sure how to do that. Unpickle and repickle each one? A "simple" binary rewrite of the pickle to past in the new name. Just not sure. Jeremy
Sidnei da Silva wrote:
| Jim asked about it on zope-dev, and got no response; see: | | "Can we forsake database backward compatability on the head for | a while?" | | http://mail.zope.org/pipermail/zope-dev/2003-November/021055.html | | ... | | > This would mean that old databases would not be useable with the | > CVS head in the near term. Would this cause anyone any problems? | | I'm happy no one responded to this. Silence is assent, :)
:P
Is there any special trick to such script? Provided some pointers of what the script should do, I wouldn't mind writing and testing it.
There are lots of special tricks to such a script. Otherwise, I would have written it already. :) OK, here's how I think it needs to happen, with some, hopefuly helpful notes. 1. The script needs to be run under Zope 2.6 or 2.7. There should be a product that people can install that installs some method to be run. People should be able to run this method in a running zope. They should not need to shut down their site to do the conversion. 2. The method should walk the object tree, converting: - old-style BTrees to modern BTrees - old-style buckets to modern buckets - intSets to IITreeSets. The individual conversions should be pretty simple. At worst involving a for loop, but, in many cases, the constructors for the new types will be able to accept the old types. 3. The hard part is walking the object tree. You will need a function that, given an object, will return the oids of the objects it references. Perhaps Jeremy can help you with that. 4. You should do each conversion in a separate transaction. That is, when you encounter an object that needs to be converted, you should convert it and commit. 5. You will want to keep a dictionary of the oids visited, just in case there are any cyles. Also, if you encounter an oid that was converted, you will need to subtitute the new value. This might be really hard. :( I think this probably won't happen, but perhaps you should at least check for it. 6. After the conversion has been run, the database should be usable with the head. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
| >Is there any special trick to such script? Provided some pointers of | >what the script should do, I wouldn't mind writing and testing it. | | There are lots of special tricks to such a script. Otherwise, I would | have written it already. :) Obviously. /me slaps his forehead | OK, here's how I think it needs to happen, with some, hopefuly helpful | notes. | | 1. The script needs to be run under Zope 2.6 or 2.7. There should | be a product that people can install that installs some method to | be run. People should be able to run this method in a running zope. | They should not need to shut down their site to do the conversion. | | 2. The method should walk the object tree, converting: | | - old-style BTrees to modern BTrees | - old-style buckets to modern buckets | - intSets to IITreeSets. Just to check: +----------+-----------+ | Old | New | +----------+-----------+ | BTree | OOBTree | +----------+-----------+ | IIBTree | IIBtree | +----------+-----------+ | OIBTree | OIBTree | +----------+-----------+ | IOBTree | IOBTree | +----------+-----------+ | intSet | IITreeSet | +----------+-----------+ | bucket? | bucket? | +----------+-----------+ | The individual conversions should be pretty simple. At worst involving | a for loop, but, in many cases, the constructors for the new types will | be able to accept the old types. | | 3. The hard part is walking the object tree. You will need a function that, | given an object, will return the oids of the objects it references. | Perhaps Jeremy can help you with that. I hope so *wink*. | 4. You should do each conversion in a separate transaction. That is, when | you | encounter an object that needs to be converted, you should convert it | and commit. | | 5. You will want to keep a dictionary of the oids visited, just in case | there are | any cyles. Also, if you encounter an oid that was converted, you will | need to | subtitute the new value. This might be really hard. :( I think this | probably | won't happen, but perhaps you should at least check for it. | | 6. After the conversion has been run, the database should be usable with | the head. Kewl. I'm eager to get my hands on it. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher Somebody's terminal is dropping bits. I found a pile of them over in the corner.
Sidnei da Silva wrote:
| >Is there any special trick to such script? Provided some pointers of | >what the script should do, I wouldn't mind writing and testing it. | | There are lots of special tricks to such a script. Otherwise, I would | have written it already. :)
Obviously.
/me slaps his forehead
| OK, here's how I think it needs to happen, with some, hopefuly helpful | notes. | | 1. The script needs to be run under Zope 2.6 or 2.7. There should | be a product that people can install that installs some method to | be run. People should be able to run this method in a running zope. | They should not need to shut down their site to do the conversion. | | 2. The method should walk the object tree, converting: | | - old-style BTrees to modern BTrees | - old-style buckets to modern buckets | - intSets to IITreeSets.
Just to check:
+----------+-----------+ | Old | New | +----------+-----------+ | BTree | OOBTree | +----------+-----------+ | IIBTree | IIBtree | +----------+-----------+ | OIBTree | OIBTree | +----------+-----------+ | IOBTree | IOBTree | +----------+-----------+ | intSet | IITreeSet | +----------+-----------+ | bucket? | bucket? | +----------+-----------+
No, this isn't quite right. I don't remember the details off hand. Think of it in terms of modules: objects from BTree -> objects from BTrees.OOBTree objects from IOBTree -> objects from BTrees.IOBTree objects from OIBTree -> objects from BTrees.OIBTree objects from IIBTree -> objects from BTrees.IIBTree objects from intSet -> objects from BTrees.IIBTree
| The individual conversions should be pretty simple. At worst involving | a for loop, but, in many cases, the constructors for the new types will | be able to accept the old types. | | 3. The hard part is walking the object tree. You will need a function that, | given an object, will return the oids of the objects it references. | Perhaps Jeremy can help you with that.
I hope so *wink*.
Me too. He's pretty busy. If he can't, I'll try to help. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
| Think of it in terms of modules: | | objects from BTree -> objects from BTrees.OOBTree | objects from IOBTree -> objects from BTrees.IOBTree | objects from OIBTree -> objects from BTrees.OIBTree | objects from IIBTree -> objects from BTrees.IIBTree | objects from intSet -> objects from BTrees.IIBTree Ok. +--------------------+------------------------+ |Old |New | +--------------------+------------------------+ |BTree.BTree |BTrees.OOBTree.OOBtree | +--------------------+------------------------+ |BTree.Bucket |BTrees.OOBTree.OOBucket | +--------------------+------------------------+ |IOBTree.BTree |BTrees.IOBTree.IOBTree | +--------------------+------------------------+ |IOBTree.Bucket |BTrees.IOBTree.IOBucket | +--------------------+------------------------+ |OIBTree.BTree |BTrees.OIBTree.OIBTree | +--------------------+------------------------+ |OIBTree.Bucket |BTrees.OIBTree.OIBucket | +--------------------+------------------------+ |IIBTree.IIBTree |BTrees.IIBTree.IIBTree | +--------------------+------------------------+ |IIBTree.Bucket |BTrees.IIBTree.IIBUcket | +--------------------+------------------------+ |intSet |BTrees.IIBTree.IITreeSet| +--------------------+------------------------+ | >| 3. The hard part is walking the object tree. You will need a function | >that, | >| given an object, will return the oids of the objects it references. | >| Perhaps Jeremy can help you with that. | > | >I hope so *wink*. | | Me too. He's pretty busy. If he can't, I'll try to help. I started with a slight variation of Shane's change_modules.py [1]. Had to make some changes because stuff seem to have changed in the Connection class. For example, invalidate() now expects a dict instead of a oid. My plan is to change around line 130, where it creates a new pickle with new_class_spec to wake the old object, create a new one and pickle the new object. Sounds reasonable? [1] http://cvs.sourceforge.net/viewcvs.py/zodbex/zodbex/ChangeModules/change_mod... -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher A programming language is low level when its programs require attention to the irrelevant.
/me grumbles Is it right that in general the only old-style BTree in a ZODB is 'ZGlobals' at the root? It seems ZCatalog have a manage_convertBTrees method since 2.3.1, and at least on my instance, the indexes BTrees are already new. I'm going to change the code to use BTrees.convert, now that I've found it *wink*. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher Is your job running? You'd better go catch it!
On Fri, 2003-12-12 at 10:09, Sidnei da Silva wrote:
/me grumbles
Is it right that in general the only old-style BTree in a ZODB is 'ZGlobals' at the root? It seems ZCatalog have a manage_convertBTrees method since 2.3.1, and at least on my instance, the indexes BTrees are already new.
I think that is true. On the other hand, it was a bit of a surprise to find a BTree in ZGlobals. Maybe they're hiding somewhere else, too.
I'm going to change the code to use BTrees.convert, now that I've found it *wink*.
Didn't know that was there either. Jeremy
| I think that is true. On the other hand, it was a bit of a surprise to | find a BTree in ZGlobals. Maybe they're hiding somewhere else, too. At least, the script didn't found another one. It did found 7800 objects total, of which one was a BTree (ZGlobals). Now, in 2.7 I get this output (starting zope via 'zopectl debug'):
app <Application instance at 414c8c80> app._p_jar <Connection at 414ac74c> app._p_jar.root() {'Application': <Application instance at 414c8c80>, 'ZGlobals': <OOBTree object at 0x414d1c60>, 'RefreshData': {'auto': {'CCHS': 1}, 'products': {'Archetypes': {'dependent_products': ('CCHS',)}}}}
Which looks sane to me. But if I copy the same Data.fs to another instance with Zope HEAD, the "AttributeError: data" is there again. It *seems* to me that the root PersistentMapping is not being unpickled correctly.
/var/src/zope/head/lib/python/ZODB/ZApplication.py(32)__init__() -> conn=db.open() (Pdb) n /var/src/zope/head/lib/python/ZODB/ZApplication.py(33)__init__() -> root=conn.root() (Pdb) n /var/src/zope/head/lib/python/ZODB/ZApplication.py(34)__init__() -> if not root.has_key(name): (Pdb) print root *** AttributeError: data (Pdb) type(root) <class 'Persistence.PersistentMapping.PersistentMapping'> (Pdb) print conn.root() *** AttributeError: data
-- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher One person's error is another person's data.
I think this problem is actually easier to solve in the common case than requring that people run a conversion script. The ZGlobals data structure is a cache, and can be blown away and recreated indiscriminately. It's recreated whenever a Product is added or removed, and Zope already does this as a matter of course if anything about the data structure is goofy. I think the problem Sidnei has run into isn't so much that the BTree module can't be imported, it's instead that the Persistence.PersistentMapping module redefines PersistentMapping in order to do something I don't quite understand, and the redefinition is what appears to be causing the AttributeError, 'data' to appear. If it's not used, everything works fine. Here's the module: import Persistence import persistent from persistent.mapping import PersistentMapping if Persistence.Persistent is not persistent.Persistent: class PersistentMapping(Persistence.Persistent, PersistentMapping): """Legacy persistent mapping class This class mixes in ExtensionClass Base if it is present. Unless you actually want ExtensionClass semantics, use persistent.mapping.PersistentMapping instead. """ If you comment out the "if" clause and everything underneath of it, make a dummy BTree.py module with the following and put in in the software home: from Globals import Persistent class BTree(Persistent): pass ... and restart Zope with an "old" filestorage file, all is well: [chrism@beach zopemafia.org]$ bin/zopectl-trunk debug Starting debugger (the name "app" is bound to the top-level Zope object)
app.objectIds() ['acl_users', 'Control_Panel', 'temp_folder', 'browser_id_manager', 'session_data_manager', 'error_log', 'standard_error_message', 'standard_html_footer', 'standard_html_header', 'standard_template.pt', 'vhm', 'tacoma.bridge.jpg', 'tacoma.bridge1.jpg', 'index_html_orig', 'index_html', 'misc', 'innomatch', 'newplone', 'New Folder', 'scratch'] root = app._p_jar.root() root {'Application': <OFS.Application.Application object at 0x41419cac>, 'ZGlobals': <BTrees._OOBTree.OOBTree object at 0x4185826c>} type(root) <class 'persistent.mapping.PersistentMapping'>
I'm not sure what the redefinition is supposed to achieve? FWIW, I've sort of got plope.com (which implies Plone) working against the trunk, although some other unrelated problems still appear to exist. On Fri, 2003-12-12 at 10:41, Sidnei da Silva wrote:
| I think that is true. On the other hand, it was a bit of a surprise to | find a BTree in ZGlobals. Maybe they're hiding somewhere else, too.
At least, the script didn't found another one. It did found 7800 objects total, of which one was a BTree (ZGlobals).
Now, in 2.7 I get this output (starting zope via 'zopectl debug'):
app <Application instance at 414c8c80> app._p_jar <Connection at 414ac74c> app._p_jar.root() {'Application': <Application instance at 414c8c80>, 'ZGlobals': <OOBTree object at 0x414d1c60>, 'RefreshData': {'auto': {'CCHS': 1}, 'products': {'Archetypes': {'dependent_products': ('CCHS',)}}}}
Which looks sane to me. But if I copy the same Data.fs to another instance with Zope HEAD, the "AttributeError: data" is there again. It *seems* to me that the root PersistentMapping is not being unpickled correctly.
/var/src/zope/head/lib/python/ZODB/ZApplication.py(32)__init__() -> conn=db.open() (Pdb) n /var/src/zope/head/lib/python/ZODB/ZApplication.py(33)__init__() -> root=conn.root() (Pdb) n /var/src/zope/head/lib/python/ZODB/ZApplication.py(34)__init__() -> if not root.has_key(name): (Pdb) print root *** AttributeError: data (Pdb) type(root) <class 'Persistence.PersistentMapping.PersistentMapping'> (Pdb) print conn.root() *** AttributeError: data
Chris McDonough wrote:
I think this problem is actually easier to solve in the common case than requring that people run a conversion script. The ZGlobals data structure is a cache, and can be blown away and recreated indiscriminately. It's recreated whenever a Product is added or removed, and Zope already does this as a matter of course if anything about the data structure is goofy.
Hm, OK. Then the question is how to decide that it's goofy. I'll note that I certainly did not intend it to be a cache when I added it a long time ago. I'm a bit uncomfortable assuming that it's always used that way, but you know more than I about the current usage.
I think the problem Sidnei has run into isn't so much that the BTree module can't be imported, it's instead that the Persistence.PersistentMapping module redefines PersistentMapping in order to do something I don't quite understand, and the redefinition is what appears to be causing the AttributeError, 'data' to appear.
Hm, I'll add a comment explaining what's going on.
If it's not used, everything works fine. Here's the module:
import Persistence import persistent from persistent.mapping import PersistentMapping
if Persistence.Persistent is not persistent.Persistent: class PersistentMapping(Persistence.Persistent, PersistentMapping): """Legacy persistent mapping class
This class mixes in ExtensionClass Base if it is present.
Unless you actually want ExtensionClass semantics, use persistent.mapping.PersistentMapping instead. """
If you comment out the "if" clause and everything underneath of it, make a dummy BTree.py module with the following and put in in the software home:
from Globals import Persistent
class BTree(Persistent): pass
... and restart Zope with an "old" filestorage file, all is well:
There needs to be a cleaner solution than this. A Dummy BTree module is not acceptable. ...
I'm not sure what the redefinition is supposed to achieve?
persistent.Persistent is not an extension class. It's a pure new-style persistent base class. Persistent.Persistent is an extension class if ExtensionClass is installed. If ExtensionClass is not installed, then a warning is issued and it is set tp persistent.Persistent. This is to ease the transition for people who use ZODB outside of Zope. The check above is to cause an extension-class-based PersistentMapping is ExtensionClass is installed. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Sun, 2003-12-14 at 22:08, Jim Fulton wrote:
Chris McDonough wrote:
I think this problem is actually easier to solve in the common case than requring that people run a conversion script. The ZGlobals data structure is a cache, and can be blown away and recreated indiscriminately. It's recreated whenever a Product is added or removed, and Zope already does this as a matter of course if anything about the data structure is goofy.
Hm, OK. Then the question is how to decide that it's goofy.
The current way it's done is in OFS.Application.Application via: def checkGlobalRegistry(self): """Check the global (zclass) registry for problems, which can be caused by things like disk-based products being deleted. Return true if a problem is found""" try: keys=list(self._p_jar.root()['ZGlobals'].keys()) except: LOG('Zope', ERROR, 'A problem was found when checking the global product '\ 'registry. This is probably due to a Product being '\ 'uninstalled or renamed. The traceback follows.', error=sys.exc_info()) return 1 return 0 If the check fails, the registry is blown away and rebuilt in the same class' fixupZClassDependencies method which does a bunch of sniffing around in Products for zclass ids. I admit I have no idea whether this check is valid and haven't had to care in a long time. It's been there since before 2.3, however.
I'll note that I certainly did not intend it to be a cache when I added it a long time ago. I'm a bit uncomfortable assuming that it's always used that way, but you know more than I about the current usage.
Probably not, but I do know that there haven't been many complaints about the rebuilding code.
If you comment out the "if" clause and everything underneath of it, make a dummy BTree.py module with the following and put in in the software home:
from Globals import Persistent
class BTree(Persistent): pass
... and restart Zope with an "old" filestorage file, all is well:
There needs to be a cleaner solution than this. A Dummy BTree module is not acceptable.
Here's an idea: create a simple script that manufactured a module which set up the Zope configuration, "filled in" for BTree in sys.modules, and got a hold of Zope.app(), causing ZGlobals to be replaced by a new-style BTree due to the above magic. People upgrading to 2.8 with databases created by Zope versions older than 2.3 would still need to run convertBTrees inside a different Zope version, but people upgrading databases created in versions equalling or later than 2.3 wouldn't because we stopped using the old BTree module for anything except ZGlobals after 2.3 AFAICT.
I'm not sure what the redefinition is supposed to achieve?
persistent.Persistent is not an extension class. It's a pure new-style persistent base class.
Out of curiosity, could you explain why it matters in this context whether it's an extensionclass or not? Is it because there might be a set of people using PersistentMapping objects for whom its important that they be able to use extenionclass semantics against them? I'd just never thought of using __of__ or inheritedAttribute or any of the other EC-specific stuff on a PersistentMapping.
Persistent.Persistent is an extension class if ExtensionClass is installed. If ExtensionClass is not installed, then a warning is issued and it is set tp persistent.Persistent. This is to ease the transition for people who use ZODB outside of Zope.
I see.. thanks. Well, even in the case that a (dummy or real) BTree module doesn't exist, if Zope is told to use the persistent.mapping.PersistentMapping class instead of the one that inherits from Persistence.Persistent, it does not fail in the way Sidnei reported. Additionally, if you switch the redefined PersistentMapping's base class order, it also begins to work. The fact it can't import BTree.BTree causes it to complain, but the ZGlobals rebuilder does begin work under both circumstances and Zope starts. ------ 2003-12-14T22:45:49 ERROR(200) ZODB Couldn't load state for 000000000000000a Traceback (most recent call last): File "/home/chrism/software/Trunk/lib/python/ZODB/Connection.py", line 428, in setstate self._reader.setGhostState(obj, p) File "/home/chrism/software/Trunk/lib/python/ZODB/serialize.py", line 203, in setGhostState obj.__setstate__(state) TypeError: function takes exactly 2 arguments (0 given) ------ 2003-12-14T22:33:35 ERROR(200) Zope A problem was found when checking the global product registry. This is probably due to a Product being uninstalled or renamed. The traceback follows. Traceback (most recent call last): File "/home/chrism/software/Trunk/lib/python/OFS/Application.py", line 233, in checkGlobalRegistry keys=list(self._p_jar.root()['ZGlobals'].keys()) File "/home/chrism/software/Trunk/lib/python/ZODB/Connection.py", line 428, in setstate self._reader.setGhostState(obj, p) File "/home/chrism/software/Trunk/lib/python/ZODB/serialize.py", line 203, in setGhostState obj.__setstate__(state) TypeError: function takes exactly 2 arguments (0 given) ------ 2003-12-14T22:33:35 INFO(0) Zope Beginning attempt to rebuild the global ZClass registry. ------ 2003-12-14T22:33:36 INFO(0) Zope The global ZClass registry has successfully been rebuilt. ------ Given this, would it make sense that the base class order of the redefined PersistentMapping class just needs to be turned around? - C
On Sun, 2003-12-14 at 23:02, Chris McDonough wrote:
Here's an idea: create a simple script that manufactured a module which set up the Zope configuration, "filled in" for BTree in sys.modules, and got a hold of Zope.app(), causing ZGlobals to be replaced by a new-style BTree due to the above magic. People upgrading to 2.8 with databases created by Zope versions older than 2.3 would still need to run convertBTrees inside a different Zope version, but people upgrading databases created in versions equalling or later than 2.3 wouldn't because we stopped using the old BTree module for anything except ZGlobals after 2.3 AFAICT.
Actually, given my findings later in this email, this isn't even necessary. If it's sensible to turn the inheritance ordering around for the EC-based persistent mapping, it appears that no conversion script needs be run at all. We might catch the error and report it more nicely in checkGlobalRegistry, but that's all that needs be done by programmers and users need do nothing except copy their database over. - C
Chris McDonough wrote:
On Sun, 2003-12-14 at 23:02, Chris McDonough wrote:
Here's an idea: create a simple script that manufactured a module which set up the Zope configuration, "filled in" for BTree in sys.modules, and got a hold of Zope.app(), causing ZGlobals to be replaced by a new-style BTree due to the above magic. People upgrading to 2.8 with databases created by Zope versions older than 2.3 would still need to run convertBTrees inside a different Zope version, but people upgrading databases created in versions equalling or later than 2.3 wouldn't because we stopped using the old BTree module for anything except ZGlobals after 2.3 AFAICT.
Actually, given my findings later in this email, this isn't even necessary.
Right. Whew. I checked in the necessary fix for PersistentMapping. There was a change in the data structure and I had to add some logic to convert some old state. I also had to check in a %$#@ travesty because Data.fs.in still has references to BoboPOS. With these changes, I can create a database with 2.6 and open it with the head. Thanks Chris for helping us figure out that this was easier than we thought. :) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Mon, 2003-12-15 at 02:39, Jim Fulton wrote:
I checked in the necessary fix for PersistentMapping. There was a change in the data structure and I had to add some logic to convert some old state.
I also had to check in a %$#@ travesty because Data.fs.in still has references to BoboPOS.
Oh god. Well, the good news is that we don't use Data.fs.in in 2.7+ anymore, so this shouldn't be a problem after about 3 years. ;-)
With these changes, I can create a database with 2.6 and open it with the head.
Thanks Chris for helping us figure out that this was easier than we thought. :)
Just lazy. ;-) - C
On Mon, 2003-12-15 at 02:39, Jim Fulton wrote:
I checked in the necessary fix for PersistentMapping. There was a change in the data structure and I had to add some logic to convert some old state.
Oh, geez, I remember this setstate/getstate business now. FWIW, I think Jeremy added it at my insistence after he changed PersistentMapping to inherit from UserDict in order to maintain backwards compatibility for versions that didn't. The next panic will start when UserDict becomes a new-style class. ;-) - C
Chris McDonough wrote: ...
Out of curiosity, could you explain why it matters in this context whether it's an extensionclass or not? Is it because there might be a set of people using PersistentMapping objects for whom its important that they be able to use extenionclass semantics against them? I'd just never thought of using __of__ or inheritedAttribute or any of the other EC-specific stuff on a PersistentMapping.
<shrug> Who knows. I'd rather be safe. It's not that hard. In any case, we would have needed the fix to handle old pickles correctly. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Mon, 2003-12-15 at 02:47, Jim Fulton wrote:
Chris McDonough wrote: ...
Out of curiosity, could you explain why it matters in this context whether it's an extensionclass or not? Is it because there might be a set of people using PersistentMapping objects for whom its important that they be able to use extenionclass semantics against them? I'd just never thought of using __of__ or inheritedAttribute or any of the other EC-specific stuff on a PersistentMapping.
<shrug> Who knows. I'd rather be safe. It's not that hard. In any case, we would have needed the fix to handle old pickles correctly.
Gotcha. Can you answer a question about the features we're allowed now? If we don't use extensionclass, is it possible to persist instances of new-style classes in ZODB now (ie. is persistent.Persistent really a new-style type/class)? - C
On Mon, 2003-12-15 at 08:15, Chris McDonough wrote:
Can you answer a question about the features we're allowed now? If we don't use extensionclass, is it possible to persist instances of new-style classes in ZODB now (ie. is persistent.Persistent really a new-style type/class)?
Yes. (i.e. yes.) Jeremy
Woot! ;-) Descriptors, yum... On Mon, 2003-12-15 at 10:02, Jeremy Hylton wrote:
On Mon, 2003-12-15 at 08:15, Chris McDonough wrote:
Can you answer a question about the features we're allowed now? If we don't use extensionclass, is it possible to persist instances of new-style classes in ZODB now (ie. is persistent.Persistent really a new-style type/class)?
Yes. (i.e. yes.)
Jeremy
Chris McDonough wrote:
Woot! ;-) Descriptors, yum...
But you can use descriptors in extension classes now too. Extension classes are now just new-style classes with some extra features and a different mro algorithm. Jim
On Mon, 2003-12-15 at 10:02, Jeremy Hylton wrote:
On Mon, 2003-12-15 at 08:15, Chris McDonough wrote:
Can you answer a question about the features we're allowed now? If we don't use extensionclass, is it possible to persist instances of new-style classes in ZODB now (ie. is persistent.Persistent really a new-style type/class)?
Yes. (i.e. yes.)
Jeremy
-- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Mon, 2003-12-15 at 05:21, Jim Fulton wrote:
Chris McDonough wrote:
Woot! ;-) Descriptors, yum...
But you can use descriptors in extension classes now too. Extension classes are now just new-style classes with some extra features and a different mro algorithm.
Whoa. Cool. That really is great. You guys are so clever. ;-) - C
On Mon, Dec 15, 2003 at 05:21:55AM -0500, Jim Fulton wrote:
Chris McDonough wrote:
Woot! ;-) Descriptors, yum...
But you can use descriptors in extension classes now too.
I wondered what the heck you guys were on about... google turned up this: http://users.rcn.com/python/download/Descriptor.htm very very cool stuff. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's ZORRO SLUDGE! (random hero from isometric.spaceninja.com)
On Mon, Dec 15, 2003 at 02:47:01AM -0500, Jim Fulton wrote:
<shrug> Who knows. I'd rather be safe. It's not that hard. In any case, we would have needed the fix to handle old pickles correctly.
Asking a possibly redundant question... not being much of a ZODB hacker, I didn't follow this thread in detail... Q: Will Zope >= 2.8 be able to import .zexp's exported from zope <= 2.6.2? From Jim's comment above I would guess "yes", but I need to be sure. Some of us out here in the wild use import / export machinery pretty heavily (see e.g. ZSyncer), and we typically like to burn in a new zope version on a dev system for some time before upgrading the production systems... but during that time we need to keep syncing stuff from dev to production. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's FOOD-COVERED STRAIGHT FROM UNCLE! (random hero from isometric.spaceninja.com)
Jeremy Hylton wrote at 2003-12-12 10:23 -0500:
Is it right that in general the only old-style BTree in a ZODB is 'ZGlobals' at the root? It seems ZCatalog have a manage_convertBTrees method since 2.3.1, and at least on my instance, the indexes BTrees are already new.
I think that is true. On the other hand, it was a bit of a surprise to find a BTree in ZGlobals. Maybe they're hiding somewhere else, too.
Shane's "BTreeFolder" uses them, too. -- Dieter
| Shane's "BTreeFolder" uses them, too. The old one. BTreeFolder2 uses the new BTrees. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher You have mail.
Sidnei da Silva wrote:
/me grumbles
Why grumble, the problem may be easier than we thought!
Is it right that in general the only old-style BTree in a ZODB is 'ZGlobals' at the root?
I dunno. Maybe. :)
It seems ZCatalog have a manage_convertBTrees method since 2.3.1, and at least on my instance, the indexes BTrees are already new.
Huh, that may may things *much easier*.
I'm going to change the code to use BTrees.convert, now that I've found it *wink*.
Cool. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Thu, Dec 11, 2003 at 11:03:35PM -0500, Jim Fulton wrote:
3. The hard part is walking the object tree. You will need a function that, given an object, will return the oids of the objects it references. Perhaps Jeremy can help you with that.
Perhaps some of the zodb utilities contain code that might be useful as a starting point? fsrefs.py, checkbtrees.py...
5. You will want to keep a dictionary of the oids visited, just in case there are any cyles.
I believe the version of checkbtrees.py in cvs now does that. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's PENGUIN FINALE! (random hero from isometric.spaceninja.com)
I announced that, for now, The Zope head won't work with old databases. We are going to have to come up with some code to deal with the old btrees and intsets. The good news is that, databases created with Zope 2.7 final should work. :) Jim Jeremy Hylton wrote:
On Thu, 2003-12-11 at 10:24, Sidnei da Silva wrote:
On Thu, Dec 11, 2003 at 10:07:51AM -0500, Jeremy Hylton wrote: | I think there's a fix for this in CVS. Can you try it again?
Just tried. Seems to have fixed it. Now, for the next step *wink*.
If I try starting Zope with an old existing database I get the traceback below. Correct me if I'm wrong, but I think the error with 'UserDict' has nothing to do with the 'Could not import class 'BTree'' or am I wrong?
I think the attribute error on data is just a symptom of the BTree failure. The object has no data attribute, because ZODB failed to unpickle the object that would be assigned to data.
I don't know what happened with backwards compatibility for the old BTree classes. When Jim and I discussed it a while back, I thought we concluded that we would provide a script that would convert all the old BTree instances to new BTrees instance. But I don't know if that decision stuck or if the script was ever written.
Jeremy
-- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
| Do you actually have more than one database connection? In other words, | is the error totally nonsensical or could the bug be that something is | getting stored in a foreign database connection? I'm just starting Zope by './bin/runzope' from a newly-created instance. No ZEO, no nothing. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher I came, I saw, I deleted all your files.
On Wed, Dec 10, 2003 at 02:14:22PM -0200, Sidnei da Silva wrote: | Has anyone run into this one? <snip> For the records, here's 'classmeta' and 'state' one line before the exception: (('Products.Formulator.FieldHelpTopic', 'FieldHelpTopic'), None) {'field_class': <class 'Products.Formulator.StandardFields.LabelField'>, 'id': 'LabelField', 'title': 'Formulator Field - LabelField'} -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher Basic is a high level languish. APL is a high level anguish.
participants (8)
-
Chris McDonough -
Dieter Maurer -
Jeremy Hylton -
Jim Fulton -
Leonardo Rochael Almeida -
Paul Winkler -
Sidnei da Silva -
Tim Peters