RFC: A proposal for ZODB schema migration in Zope 3
I've posted a proposal: http://dev.zope.org/Zope3/DatabaseGenerations for a framework for managing the migration of application/database schemas in ZODB databases in an orderly manner. The proposal is to apply this new framework in Zope 3, however, if it works well, it would be applicable to Zope 2 and other ZODB applications. Comments are welcome. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
I am keen on such functionality. I will be working on something related to this in the near future to support a customer. I would be interested in implementing something like this for Zope 2 as a result. I had planned on implementing it as a completely external kind of thing, but maybe some support from Zope itself would be useful. There is another practical problem that is logically related to this one which has the potential to be solved by the same machinery. I'm not sure we want to conflate the problems, but I mention it here just because it's possible that we do. The "upgrade problem" isn't always limited to the updating the schema of database objects. Sometimes an upgrade requires a mass update of "already-schema-current" objects. For example, during an upgrade, you might want to reset local role values across a number of objects in Zope 2. Or you might want to change a data value across a number of objects. Or whatever. This is a common problem in production, and usually it's solved by writing one-off scripts that connect to the database and do recursion and a commit. There's nothing in your proposal that implies that the proposed machinery couldn't be used to perform these kinds of mass-update duties except the names "schema" (and maybe "generation"). So immediately the only thing I suggest (if we want to conflate the two problems) is to change the name "schema" to "state" and the name "generation" to "version". If we don't want to conflate the two problems, do you have a suggestion as to how to more uniformly deal with the data-value-update problem? (BTW, it's very hard to follow threads following a message that is cross-posted to a number of lists. I've made this mistake a number of times in the past, and it's almost never worth doing as invariably I miss something or need to reread every followup on every list). On Sat, 2004-04-17 at 08:14, Jim Fulton wrote:
I've posted a proposal:
http://dev.zope.org/Zope3/DatabaseGenerations
for a framework for managing the migration of application/database schemas in ZODB databases in an orderly manner. The proposal is to apply this new framework in Zope 3, however, if it works well, it would be applicable to Zope 2 and other ZODB applications.
Comments are welcome.
Jim
Chris McDonough wrote:
I am keen on such functionality. I will be working on something related to this in the near future to support a customer. I would be interested in implementing something like this for Zope 2 as a result. I had planned on implementing it as a completely external kind of thing, but maybe some support from Zope itself would be useful.
I think that the implemntation for Zope 3 will be pretty directly usable in Zope 2. I'm mainly limiting the scope to Zope 3 now, because the technique is untried. (One area that I've punted on is mutiple databases. I think the mechanism will need som expansion to handle multiple databases. In general, though mounting is a god initial step, I think we need a better system for dealing with multiple databases, but I just don't have the badwidth for that now.)
There is another practical problem that is logically related to this one which has the potential to be solved by the same machinery. I'm not sure we want to conflate the problems, but I mention it here just because it's possible that we do.
The "upgrade problem" isn't always limited to the updating the schema of database objects. Sometimes an upgrade requires a mass update of "already-schema-current" objects. For example, during an upgrade, you might want to reset local role values across a number of objects in Zope 2. Or you might want to change a data value across a number of objects. Or whatever. This is a common problem in production, and usually it's solved by writing one-off scripts that connect to the database and do recursion and a commit.
To me, the term "schema" is (intentionally) vague enough to include this sort of thing.
There's nothing in your proposal that implies that the proposed machinery couldn't be used to perform these kinds of mass-update duties except the names "schema" (and maybe "generation"). So immediately the only thing I suggest (if we want to conflate the two problems) is to change the name "schema" to "state" and the name "generation" to "version".
I don't think this is necessary. I think that "schema" is general anough and I like the term "generation" because it implies a step-by-step evolution, which is key. Central to the proposal is that we can describe the evolution of the data as a step=by-step progression and that we can update a database through the orderly application of independent changes. Feel free to ad a note to the proposal that the scope includes the sort of scenario you described. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Sat, 17 Apr 2004 08:14:35 -0400 Jim Fulton <jim@zope.com> wrote:
I've posted a proposal:
http://dev.zope.org/Zope3/DatabaseGenerations
for a framework for managing the migration of application/database schemas in ZODB databases in an orderly manner. The proposal is to apply this new framework in Zope 3, however, if it works well, it would be applicable to Zope 2 and other ZODB applications.
Comments are welcome.
+1 It might be nice if some of this mechanism was distributed in ZODB itself, or at least the ZODB docs pointed to the implementation as an example of how it could be done. This is definitely a problem not limited to Zope. -Casey
participants (3)
-
Casey Duncan -
Chris McDonough -
Jim Fulton