New-style ExtensionClasses (Zope 2.8) -- MRO issue
I've been working on a "new-style" ExtensionClass (nsEC) project to reimplement ExtensionClass as a new-style meta class. This will allow ExtensionClasses (i.e. most Zope 2 classes) to be able to use features of new-style Python classes, including: - New protocols, - Descriptors, - Garbage Collection, ... I think I've got this largely working, but have encountered a significant issue that I'd like some input on. New-style classes and classic classes use different algorithms for looking up inherited attributes (e.g. methods). Classic classes use a left-to-right depth-first algorithm. Python 2.3 uses a "C3 Method-Resolution Order" algorithm. I won't try to explain how this algorithm works. If you are curious, see: http://www.python.org/2.3/mro.html This new algorithm has a couple of noteworthy properties: - It yields different lookup results than the left-to-right depth-first algorithm used for classic classes. - It is sometimes impossible to compute a "method resolution order" for a given class and base classes. >>> class A(object): ... pass >>> class B(A): ... pass >>> class C(A, B): ... pass Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: Cannot create a consistent method resolution order (MRO) for bases B, A The original ExtensionClass used the left-to-right depth-first algorithm used for classic classes. Currently, nsEC inherits the new method lookup algorithm from type (the standard Python 2.3 metaclass used for new-style classes). This has run into significant difficulties with the Zope 2 classes, which have a highly complex inheritence graph. An initial analysis showed that for more than half of the classes in Zope 2, it was impossible to compute a method-resolution order using the new algorithm. After about a day of analysis (and development of a tool to aid in the analysis) I've been able to adjust the Zope classes so that they will work with the new algorithm. I also analyzed the CMF head. To use the new algorithm with the CMF, it was necessary to make some changes to the CMF and to Zope. In one case, it was necessary to copy some methods around. An interesting result of the analysis is that it identified several methods that were being inherited incorrectly. The new inheritence algorithm yeilded better results for these cases, once the classes were modified enough to use the algorithm at all. It is very likely that, if we use the new algorithm for nsEC, many products will not work anymore (meaning with Zope 2.8). Modifying the products to work will probably be a non-trivial undertaking. We have three alternatives: 1. Use the old method-lookup algorithm for nsEC. Pro: backward compatible Con: may produce odd results if a new-style class with a complex arrangement of base classes is mixed with an ExtensionClass. 2. Use the new-style method-lookup algithm Pro: Most consistent with new-style classes. Cons: Likely that many products will be broken and require change to work with nsEC/Zope 2.8. Analysis of the changes needed to fix method-resolution-order problems is extremely complex. 3. Use a hybrid schema. I'll call this the "encapsulated base" scheme. Consider: >>> class C(X, Y, Z): ... def foo(self): ... ... ... When we look up an attribute, we do the following: - Look in C's dictionary first. All three algorithms agree on this. :) - Look up the attribute in X. We don't care how we het the attribute from X. If X is a new-style-class, we use the new algithm. If X is a classic class, we use left-to-right depth-first. If X is an nsEC, use the "encapsulated base" algithm. If we don't find the attribute in X, look in Y and then in Z, using the same approach. This algithm will produce backward compatible results, providing the equivalent of left-to-right depth-first for nsECs and classic classes. (Note for the more advanced reader: we'll actually do something less abstract. We'll use a simple algorthm to merge the __mro__ of the base classes, computing an __mro__ for classes classes using the left-to-right depth-first algorithm. We'll basically lay the mros end-to-end left-to-right and remove repeats, keeping the first occurence of each class.) I find this approach appealing for reasons of encapsulation. When reasoning about C, I can reason from known behavior of X, Y, and Z, without considering their respective inherietnce hierarchies, which I consider to be an implementation detail. Pros: backward compatible Less likely than option 1 to produce odd results if a new-style class with a complex arrangement of base classes is mixed with an ExtensionClass. Con: may produce odd results if a new-style class with a complex arrangement of base classes is mixed with an ExtensionClass. Thoughts? I am worried enough about breaking products that I'm inclined to go with option 3. Does anybody think we ought to use the new algorithm (option 2)? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Fri, Oct 31, 2003 at 12:14:27PM -0500, Jim Fulton wrote: | Thoughts? | | I am worried enough about breaking products that I'm inclined to go | with option 3. | | Does anybody think we ought to use the new algorithm (option 2)? I'm for option 2. Given that a huge amount of software that is based on CMF is in the Collective these days, we could probably convert all of them in a bug day, and in the same sweep, remove the ones that are not being used anymore. The same is true for Plone. -- Sidnei da Silva <sidnei@awkly.org> dreamcatching :: making your dreams come true http://awkly.org My apologies if I sound angry. I feel like I'm talking to a void. -- Avery Pennarun
Sidnei da Silva wrote:
On Fri, Oct 31, 2003 at 12:14:27PM -0500, Jim Fulton wrote: | Thoughts? | | I am worried enough about breaking products that I'm inclined to go | with option 3. | | Does anybody think we ought to use the new algorithm (option 2)?
I'm for option 2. Given that a huge amount of software that is based on CMF is in the Collective these days, we could probably convert all of them in a bug day, and in the same sweep, remove the ones that are not being used anymore. The same is true for Plone.
Before you commit to this, perhaps you should try figuring out how to get some products working on the mro-advanture branch. (See my other note.) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
I am worried enough about breaking products that I'm inclined to go with option 3.
Does anybody think we ought to use the new algorithm (option 2)?
I think we should use the new algorithm. However, I don't have any products that I would be responsible for updating and maintaining. I'd be happy to help people work out problems, though. If we were starting from scratch, the C3 algorithm would be the obvious choice. The Dylan linearization paper makes a pretty good argument for it: The first reason for the change is that a monotonic linearization seems a better match for users' intuitions about how inheritance and linearizations work. With a monotonic linearization, it is easier to understand the behavior of classes when multiple inheritance is used, largely because behavior of instances of a class can be explained in terms of the direct superclasses. The depth-first resolution order can produce un-intuitive results because it doesn't honor local precedence order. It also leads to sloppy code where the same base class is mixed into the inheritance hierarchy at multiple levels. If we stick with a non-standard MRO, then Zope programmers will have to learn special rules for dealing with multiple inheritance with Zope code that don't apply to Python programming in general. I don't see much value in creating such a special case here. We'd also need to implement our own MRO, maintain it, make sure it's correct, etc. That's a long-term maintenance burden, as opposed to a short-term problem of updating existing code. We can write tools to help people update their code if it ends up being a real problem, and they get clearer code as an added bonus. Jeremy
Jeremy Hylton wrote:
I am worried enough about breaking products that I'm inclined to go with option 3.
Does anybody think we ought to use the new algorithm (option 2)?
I think we should use the new algorithm. However, I don't have any products that I would be responsible for updating and maintaining. I'd be happy to help people work out problems, though.
Ooooohkay. If anyone has a problem, I'll send them to you and Sidnei. :) Note that, in many cases, the people having problems will *not* be the product authors. They'll be people who just want to upgrade thier Zope installationa that use third-party products. Oh, and as you help people fix these products, you'll want to redistribute the fixed products so as to avoid having to fix them over and over. ...
If we stick with a non-standard MRO, then Zope programmers will have to learn special rules for dealing with multiple inheritance with Zope code that don't apply to Python programming in general.
Well, except that they are the same rules used for classic classes, which probably still predominate in the Python world.
I don't see much value in creating such a special case here.
We'd also need to implement our own MRO, maintain it, make sure it's correct, etc. That's a long-term maintenance burden,
That's not a big deal. Options 1 and 3 both have simple algorithms that are easy to implement and won't really need much maintanance. Fortunately, type actually provides a hook for overriding the mro computation that makes this easy.
as opposed to a short-term problem of updating existing code.
It won't be that short term. It will come up each time someone wants to use a product that hasn't been converted before.
We can write tools to help people update their code if it ends up being a real problem,
I think you are underestimating the complexity of the analysis.
and they get clearer code as an added bonus.
I don't think it makes the code any clearer. Zope 2 has an extremely complex inheritence graph. Changing the mro algorithm won't change that. Zope 3, of course, makes much less use of inheritence and has clearer code. It also uses new-style classes and, thus, uses the new algorithm. I personally don't like the new algorithm, but I don't really care in the long run. One should avoid inheritence complex enough to show a difference. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
| Ooooohkay. If anyone has a problem, I'll send them to you and Sidnei. :) /me takes his iBook and a plane to The Amazon | Note that, in many cases, the people having problems will *not* be the | product authors. They'll be people who just want to upgrade thier Zope | installationa that use third-party products. Most of the time, this is the same people who can't update their products, so if they *do* want a new zope, they will need to get a new product as well. I think its safe to make that assumption. | Well, except that they are the same rules used for classic classes, which | probably still predominate in the Python world. That may be worth considering. | > as opposed to a | >short-term problem of updating existing code. | | It won't be that short term. It will come up each time someone wants to | use a product that hasn't been converted before. Considering that 2.8 may be at least 6-8 months in the future (is that a nice assumption?) I would consider that a just-long-enough term. | I don't think it makes the code any clearer. Zope 2 has an extremely | complex inheritence graph. Changing the mro algorithm won't change that. | | Zope 3, of course, makes much less use of inheritence and has clearer | code. It also uses new-style classes and, thus, uses the new algorithm. Now *that* is a good point for going with the new algorithm. We don't want to have even another round later on to make it compatible with Zope 3. If we could do it at the same time it would be a bonus. Though, from what I understand, your custom algorithm would provide that. | I personally don't like the new algorithm, but I don't really care | in the long run. One should avoid inheritence complex enough to show | a difference. I hearthly agree here. -- Sidnei da Silva <sidnei@awkly.org> dreamcatching :: making your dreams come true http://awkly.org The trouble with computers is that they do what you tell them, not what you want. -- D. Cohen
On Fri, 2003-10-31 at 13:52, Sidnei da Silva wrote:
| I personally don't like the new algorithm, but I don't really care | in the long run. One should avoid inheritence complex enough to show | a difference.
I hearthly agree here.
Home domestic of you <wink>. I do like the new algorithm, but we can all agree to avoid complex uses of inheritance. Jeremy
Sidnei da Silva wrote:
| Ooooohkay. If anyone has a problem, I'll send them to you and Sidnei. :)
/me takes his iBook and a plane to The Amazon
| Note that, in many cases, the people having problems will *not* be the | product authors. They'll be people who just want to upgrade thier Zope | installationa that use third-party products.
Most of the time, this is the same people who can't update their products, so if they *do* want a new zope, they will need to get a new product as well. I think its safe to make that assumption.
I don't follow this. What do you mean be "Can't update their products"? Are you saying that to upgrade to a new Zope, they should actually have to replace the products they are using with different ones? ..
| > as opposed to a | >short-term problem of updating existing code. | | It won't be that short term. It will come up each time someone wants to | use a product that hasn't been converted before.
Considering that 2.8 may be at least 6-8 months in the future (is that a nice assumption?)
I hope that 2.8 will come out much sooner than that. I hope to see 2.9 in that timeframe. Remember that 2.8 will have just nsEC, ZODB 3.3, and any other stuff that's piled up in the interim. :) ...
| I don't think it makes the code any clearer. Zope 2 has an extremely | complex inheritence graph. Changing the mro algorithm won't change that. | | Zope 3, of course, makes much less use of inheritence and has clearer | code. It also uses new-style classes and, thus, uses the new algorithm.
Now *that* is a good point for going with the new algorithm. We don't want to have even another round later on to make it compatible with Zope 3. If we could do it at the same time it would be a bonus. Though, from what I understand, your custom algorithm would provide that.
Huh? That doesn't make sense. As long as people use EC, they will use whatever algorthm we pick now. If people use ExtensionClasses in Zope 3, then those classes will use the algorithm we pick now. People would switch to the new algorithm when they convert (or mor likely rewrite) their products to be ExtensionClass-less and Zope3-ish. If they make their products Zope 3-ish, the lookup algorithm won't matter much. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Fri, Oct 31, 2003 at 02:46:51PM -0500, Jim Fulton wrote: | >Most of the time, this is the same people who can't update their | >products, so if they *do* want a new zope, they will need to get a new | >product as well. I think its safe to make that assumption. | | I don't follow this. What do you mean be "Can't update their products"? | Are you saying that to upgrade to a new Zope, they should actually have to | replace the products they are using with different ones? I meant, people that 'are afraid' of updating their existing products to new versions, or just can't because they would break compatibility. | >Considering that 2.8 may be at least 6-8 months in the future (is that | >a nice assumption?) | | I hope that 2.8 will come out much sooner than that. I hope to see 2.9 | in that timeframe. Remember that 2.8 will have just nsEC, ZODB 3.3, and | any other stuff that's piled up in the interim. :) Yay! Thats very nice to hear. | >Now *that* is a good point for going with the new algorithm. We don't | >want to have even another round later on to make it compatible with | >Zope 3. If we could do it at the same time it would be a | >bonus. Though, from what I understand, your custom algorithm would | >provide that. | | Huh? That doesn't make sense. As long as people use EC, they will use | whatever algorthm we pick now. If people use ExtensionClasses in Zope 3, | then those classes will use the algorithm we pick now. People would | switch to the new algorithm when they convert (or mor likely rewrite) | their products to be ExtensionClass-less and Zope3-ish. If they | make their products Zope 3-ish, the lookup algorithm won't matter much. Ok, I got it wrong then. Sorry. -- Sidnei da Silva <sidnei@awkly.org> dreamcatching :: making your dreams come true http://awkly.org The value of a program is proportional to the weight of its output.
Jim Fulton wrote: ...
An initial analysis showed that for more than half of the classes in Zope 2, it was impossible to compute a method-resolution order using the new algorithm. After about a day of analysis (and development of a tool to aid in the analysis) I've been able to adjust the Zope classes so that they will work with the new algorithm.
I also analyzed the CMF head. To use the new algorithm with the CMF, it was necessary to make some changes to the CMF and to Zope. In one case, it was necessary to copy some methods around.
I've checked the results of my work into the mro-advanture-branch (waa, I wish I cud spell) branch. You might find it entertaining to check out Zope on this branch: cvs co -rmro-advanture-branch Zope (and build it) (Note, Python2.3 is required.) Install your favorite products (CMF makes a nice example) and then use the functions in the included mrohell module to find out if you have a problem: cd Zope bin/zopectl debug
import mrohell base = merohell.step1() mrohell.step2(base, mroonly=True)
and see if you get any mro problems. If you do, see if you can figure out how to fix them. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Fri, Oct 31, 2003 at 02:39:28PM -0500, Jim Fulton wrote: | I've checked the results of my work into the mro-advanture-branch (waa, | I wish I cud spell) branch. | | You might find it entertaining to check out Zope on this branch: | | cvs co -rmro-advanture-branch Zope | | (and build it) (Note, Python2.3 is required.) | | Install your favorite products (CMF makes a nice example) | and then use the functions in the included mrohell module to | find out if you have a problem: | | cd Zope | bin/zopectl debug | >>> import mrohell | >>> base = merohell.step1() | >>> mrohell.step2(base, mroonly=True) | | and see if you get any mro problems. If you do, see if you can | figure out how to fix them. Amazing! A perfect task for a saturday evening ;) -- Sidnei da Silva <sidnei@awkly.org> dreamcatching :: making your dreams come true http://awkly.org This is an unauthorized cybernetic announcement.
Sidnei da Silva wrote:
On Fri, Oct 31, 2003 at 02:39:28PM -0500, Jim Fulton wrote: | I've checked the results of my work into the mro-advanture-branch (waa, | I wish I cud spell) branch. | | You might find it entertaining to check out Zope on this branch: | | cvs co -rmro-advanture-branch Zope | | (and build it) (Note, Python2.3 is required.) | | Install your favorite products (CMF makes a nice example) | and then use the functions in the included mrohell module to | find out if you have a problem: | | cd Zope | bin/zopectl debug | >>> import mrohell | >>> base = merohell.step1() | >>> mrohell.step2(base, mroonly=True) | | and see if you get any mro problems. If you do, see if you can | figure out how to fix them.
Amazing! A perfect task for a saturday evening ;)
Actually, I was thinking it would be well suited to Halloween. Trick or treat! Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Fri, Oct 31, 2003 at 03:08:34PM -0500, Jim Fulton wrote: | Sidnei da Silva wrote: | >On Fri, Oct 31, 2003 at 02:39:28PM -0500, Jim Fulton wrote: | >| I've checked the results of my work into the mro-advanture-branch (waa, | >| I wish I cud spell) branch. | >| | >| You might find it entertaining to check out Zope on this branch: | >| | >| cvs co -rmro-advanture-branch Zope | >| | >| (and build it) (Note, Python2.3 is required.) | >| | >| Install your favorite products (CMF makes a nice example) | >| and then use the functions in the included mrohell module to | >| find out if you have a problem: | >| | >| cd Zope | >| bin/zopectl debug | >| >>> import mrohell | >| >>> base = merohell.step1() | >| >>> mrohell.step2(base, mroonly=True) ERmm.... am I missing something? sidnei@moria:~/src/zope/2_7-mro$ find . -name "mrohell" sidnei@moria:~/src/zope/2_7-mro$ cat CVS/Tag Tmro-advanture-branch [sidnei@moria:~/src/instance/mro]$ ./bin/zopectl debug Starting debugger (the name "app" is bound to the top-level Zope object)
import mrohell Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named mrohell
-- Sidnei da Silva <sidnei@awkly.org> dreamcatching :: making your dreams come true http://awkly.org That does not compute.
Sorry, it was just on the wrong place. sidnei@moria:~/src/zope/2_7-mro$ find . -name "*mroh*" ./mrohell.py -- Sidnei da Silva <sidnei@awkly.org> dreamcatching :: making your dreams come true http://awkly.org The steady state of disks is full. -- Ken Thompson
Not bad. On a 'stock' plone installation, i've got these results: sidnei@moria:~/src/instance/mro$ ./bin/zopectl debug Starting debugger (the name "app" is bound to the top-level Zope object)
import mrohell base = mrohell.step1() mrohell.step2(base, mroonly=True) Couldn't get mro for Products.GroupUserFolder.GroupUserFolder.GroupUserFolder Couldn't get mro for Products.CMFFormController.Script.FSPythonScript Couldn't get mro for Products.CMFFormController.FSControllerValidator.FSControllerValidator Couldn't get mro for Products.CMFFormController.FSControllerPythonScript.FSControllerPythonScript Couldn't get mro for Products.CMFPlone.PropertiesTool.PropertiesTool 488 5 210
I'm going to fix those (after my english class) and then try something harder ;) -- Sidnei da Silva <sidnei@awkly.org> dreamcatching :: making your dreams come true http://awkly.org A modem is a baudy house.
On Sat, Nov 01, 2003 at 09:10:11AM -0200, Sidnei da Silva wrote: | Couldn't get mro for | Products.GroupUserFolder.GroupUserFolder.GroupUserFolder | Couldn't get mro for Products.CMFFormController.Script.FSPythonScript | Couldn't get mro for | Products.CMFFormController.FSControllerValidator.FSControllerValidator | Couldn't get mro for | Products.CMFFormController.FSControllerPythonScript.FSControllerPythonScript | Couldn't get mro for Products.CMFPlone.PropertiesTool.PropertiesTool | 488 5 210 The ones in CMFPlone and CMFFormController seemed easy to fix, I haven't dived much into GroupUserFolder, as it's hieararchy seems very complex to me. I checked in the changes on separate branches of each product. In Plone I just changed Folder for ObjectManager, and in CMFFormController I separated the class into a mixin class with no bases, that is then mixed into FSPythonScript. The mrohell script is very nice, though I haven't really understood the output. I fixed those by trial and error :( []'s -- Sidnei da Silva <sidnei@awkly.org> dreamcatching :: making your dreams come true http://awkly.org BYTE editors are people who separate the wheat from the chaff, and then carefully print the chaff.
Sidnei da Silva wrote:
I'm going to fix those (after my english class) and then try something harder ;)
Here is more stuff:
import mrohell base = mrohell.step1() mrohell.step2(base, mroonly=True) Couldn't get mro for Products.Archetypes.BaseBTreeFolder.BaseBTreeFolder Couldn't get mro for Products.Archetypes.BaseFolder.BaseFolder Couldn't get mro for Products.Archetypes.OrderedBaseFolder.OrderedBaseFolder Couldn't get mro for Products.GroupUserFolder.GroupUserFolder.GroupUserFolder Couldn't get mro for Products.CMFPlone.PloneFolder.BasePloneFolder Couldn't get mro for Products.CMFPlone.LargePloneFolder.LargePloneFolder Couldn't get mro for Products.Archetypes.BaseFolder.BaseFolderMixin Couldn't get mro for Products.CMFDefault.SkinnedFolder.SkinnedFolder Couldn't get mro for Products.CMFPlone.PloneFolder.PloneFolder Couldn't get mro for Products.Archetypes.ArchetypeTool.ArchetypeTool Couldn't get mro for Products.CMFFormController.Script.FSPythonScript Couldn't get mro for Products.CMFFormController.FSControllerValidator.FSControllerValidator Couldn't get mro for Products.CMFCore.FSPythonScript.FSPythonScript Couldn't get mro for Products.CMFFormController.FSControllerPythonScript.FSControllerPythonScript Couldn't get mro for Products.CMFCore.FSPageTemplate.FSPageTemplate Couldn't get mro for Products.CMFFormController.FSControllerPageTemplate.FSControllerPageTemplate Couldn't get mro for Products.CMFPlone.PropertiesTool.PropertiesTool Couldn't get mro for Products.CMFCore.FSZSQLMethod.FSZSQLMethod Couldn't get mro for Products.CMFPlone.FactoryTool.TempFolder Couldn't get mro for Products.Archetypes.OrderedBaseFolder.OrderedFolder Couldn't get mro for Products.Archetypes.examples.SimpleFolder.SimpleFolder 526 21 210
It seems that there are only several classes that cause problems: Products.CMFDefault.SkinnedFolder.SkinnedFolder Products.Archetypes.BaseFolder.BaseFolder Products.CMFCore.FS* Products.Archetypes.BaseFolder.BaseFolderMixin AFAIK most of the other classes with mro problems are just subclasses of these classes above. Christian
I think it was decided for the non-mro approach, though no one stated it clearly. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher I must have slipped a disk -- my pack hurts!
On Mon, 2003-11-24 at 13:22, Sidnei da Silva wrote:
I think it was decided for the non-mro approach, though no one stated it clearly.
If it was, they sure didn't. A summary of the opinions that lead to the decision would be nice. Even if it's decided, it wouldn't hurt to fix the inheritance problems in CMF. One of the key properties of C3 is local order is honored. If a class A has two base classes B and C, then a method defined in B and C (perhaps via their base classes) will be found in B first. This matches the way I think about inheritance, but not the way the old algorithm works. You may be able to change the inheritance hierarchy so that C3 and classic MROs both have the same results. Then programmers will be able to understand the code :-). Jeremy
On Mon, 24 Nov 2003 14:06:07 -0500 Jeremy Hylton <jeremy@zope.com> wrote:
[snip] Then programmers will be able to understand the code :-).
Very interesting conclusion... I might argue that it would be a premature optimization however ;^) -Casey
Jeremy Hylton wrote at 2003-11-24 14:06 -0500:
... One of the key properties of C3 is local order is honored. If a class A has two base classes B and C, then a method defined in B and C (perhaps via their base classes) will be found in B first.
This matches the way I think about inheritance, but not the way the old algorithm works.
Are you sure? The old algorithm is described in the Python Language Reference: Class attribute references are translated to lookups in this dictionary, e.g., "C.x" is translated to "C.__dict__["x"]". When the attribute name is not found there, the attribute search continues in the base classes. The search is depth-first, left-to-right in the order of occurrence in the base class list. Thus, if class "B" comes before class "C" in the list of base classes, "B"s attributes are found in preference of "A"s attributes. -- Dieter
[Jeremy Hylton]
... One of the key properties of C3 is local order is honored. If a class A has two base classes B and C, then a method defined in B (perhaps via their base classes) will be found in B first.
This matches the way I think about inheritance, but not the way the old algorithm works.
[Dieter Maurer]
Are you sure?
The old algorithm is described in the Python Language Reference:
Class attribute references are translated to lookups in this dictionary, e.g., "C.x" is translated to "C.__dict__["x"]". When the attribute name is not found there, the attribute search continues in the base classes. The search is depth-first, left-to-right in the order of occurrence in the base class list.
Thus, if class "B" comes before class "C" in the list of base classes, "B"s attributes are found in preference of "A"s attributes.
There are three MRO algorithms in the mix, and it's not always clear which are being discussed: 1. "Classic", depth-first left-to-right, used for old-style classes. 2. The new-style class MRO used in Python 2.2. 3. The distinct new-style class MRO, called C3, used in Python 2.3. There are two properties of MROs people (well, the people who argued about it, not necessarily people on this mailing list) agreed were desirable: 1. Local ordering is respected (what Jeremy talked about). 2. Monotonicity is preserved. #2 is subtler, and so violations also cause subtler bugs. From: http://www.python.org/2.3/mro.html A MRO is monotonic when the following is true: if C1 precedes C2 in the linearization of C, then C1 precedes C2 in the linearization of any subclass of C. Otherwise, the innocuous operation of deriving a new class could change the resolution order of methods, potentially introducing very subtle bugs. WRT these criteria: + Classic MRO respects local ordering, but routinely violates monotonicity in inheritance graphs with diamonds. This is a bigger deal for new-style classes than old because diamonds are very common (every new-style class ultimately derives from class "object"). + The 2.2 new-style MRO blew it on both, although it's apparently hard to concoct an example where monotonicity fails (the link above gives an example requiring 9 classes). In all the simpler examples, it preserved monotonicity where classic MRO violated it, which is what made it attractive. + C3 respects both.
Sidnei da Silva wrote:
I think it was decided for the non-mro approach, though no one stated it clearly.
We are using a mro approach. :) We aren't using the C3 algorithm for new-style extension classes. The MRO computation for new-style extension classes is designed to maximize backward compatability. The algorithm is described in detail in the docstring for the test_mro test in lib/python/ExtensionClass/tests.py. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Mon, 2003-11-24 at 12:03, Christian Heimes wrote:
Sidnei da Silva wrote:
I'm going to fix those (after my english class) and then try something harder ;)
Here is more stuff:
import mrohell base = mrohell.step1() mrohell.step2(base, mroonly=True) Couldn't get mro for Products.Archetypes.BaseBTreeFolder.BaseBTreeFolder Couldn't get mro for Products.Archetypes.BaseFolder.BaseFolder Couldn't get mro for Products.Archetypes.OrderedBaseFolder.OrderedBaseFolder Couldn't get mro for Products.GroupUserFolder.GroupUserFolder.GroupUserFolder Couldn't get mro for Products.CMFPlone.PloneFolder.BasePloneFolder Couldn't get mro for Products.CMFPlone.LargePloneFolder.LargePloneFolder Couldn't get mro for Products.Archetypes.BaseFolder.BaseFolderMixin Couldn't get mro for Products.CMFDefault.SkinnedFolder.SkinnedFolder Couldn't get mro for Products.CMFPlone.PloneFolder.PloneFolder Couldn't get mro for Products.Archetypes.ArchetypeTool.ArchetypeTool Couldn't get mro for Products.CMFFormController.Script.FSPythonScript Couldn't get mro for Products.CMFFormController.FSControllerValidator.FSControllerValidator Couldn't get mro for Products.CMFCore.FSPythonScript.FSPythonScript Couldn't get mro for Products.CMFFormController.FSControllerPythonScript.FSControllerPythonScript Couldn't get mro for Products.CMFCore.FSPageTemplate.FSPageTemplate Couldn't get mro for Products.CMFFormController.FSControllerPageTemplate.FSControllerPageTemplate Couldn't get mro for Products.CMFPlone.PropertiesTool.PropertiesTool Couldn't get mro for Products.CMFCore.FSZSQLMethod.FSZSQLMethod Couldn't get mro for Products.CMFPlone.FactoryTool.TempFolder Couldn't get mro for Products.Archetypes.OrderedBaseFolder.OrderedFolder Couldn't get mro for Products.Archetypes.examples.SimpleFolder.SimpleFolder 526 21 210
It seems that there are only several classes that cause problems: Products.CMFDefault.SkinnedFolder.SkinnedFolder Products.Archetypes.BaseFolder.BaseFolder Products.CMFCore.FS* Products.Archetypes.BaseFolder.BaseFolderMixin
AFAIK most of the other classes with mro problems are just subclasses of these classes above.
Jim made checkins to deal with the CMF-related MRO problems, on his 'mro-advanture-branch':: === CMF/CMFDefault/SkinnedFolder.py 1.15 => 1.15.6.1 === --- CMF/CMFDefault/SkinnedFolder.py:1.15 Sat Jun 28 12:31:21 2003 +++ CMF/CMFDefault/SkinnedFolder.py Fri Oct 31 14:24:19 2003 @@ -61,8 +61,7 @@ , ) - -class SkinnedFolder(CMFCatalogAware, PortalFolder): +class SkinnedFolder(PortalFolder): """ """ meta_type = 'Skinned Folder' @@ -70,6 +69,10 @@ security = ClassSecurityInfo() manage_options = PortalFolder.manage_options + + indexObject = CMFCatalogAware.indexObject + unindexObject = CMFCatalogAware.unindexObject + reindexObject = CMFCatalogAware.reindexObject def __call__(self): ''' === CMF/CMFCore/FSObject.py 1.15 => 1.15.2.1 === --- CMF/CMFCore/FSObject.py:1.15 Fri Sep 12 20:46:40 2003 +++ CMF/CMFCore/FSObject.py Fri Oct 31 14:24:18 2003 @@ -18,7 +18,7 @@ from string import split from os import path, stat -import Acquisition, Globals +import Acquisition, Globals, ExtensionClass from AccessControl import ClassSecurityInfo from OFS.SimpleItem import Item from DateTime import DateTime @@ -31,7 +31,7 @@ from OFS.Cache import Cacheable -class FSObject(Acquisition.Implicit, Item, Cacheable): +class FSObject(Item, Cacheable, Acquisition.Implicit, ExtensionClass.Base): """FSObject is a base class for all filesystem based look-alikes. Subclasses of this class mimic ZODB based objects like Image and I imagine making similar small changes to Archetypes won't be hard. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
Note that this is moot, as I've decided to use a backward-compatible mro. That is, new-style extension classes will use an mro that is backward compatible with old-style extension classes. They do *not* use the same mro algorithm as new-style classes. Jim Tres Seaver wrote:
On Mon, 2003-11-24 at 12:03, Christian Heimes wrote:
Sidnei da Silva wrote:
I'm going to fix those (after my english class) and then try something harder ;)
Here is more stuff:
import mrohell base = mrohell.step1() mrohell.step2(base, mroonly=True) Couldn't get mro for Products.Archetypes.BaseBTreeFolder.BaseBTreeFolder Couldn't get mro for Products.Archetypes.BaseFolder.BaseFolder Couldn't get mro for Products.Archetypes.OrderedBaseFolder.OrderedBaseFolder Couldn't get mro for Products.GroupUserFolder.GroupUserFolder.GroupUserFolder Couldn't get mro for Products.CMFPlone.PloneFolder.BasePloneFolder Couldn't get mro for Products.CMFPlone.LargePloneFolder.LargePloneFolder Couldn't get mro for Products.Archetypes.BaseFolder.BaseFolderMixin Couldn't get mro for Products.CMFDefault.SkinnedFolder.SkinnedFolder Couldn't get mro for Products.CMFPlone.PloneFolder.PloneFolder Couldn't get mro for Products.Archetypes.ArchetypeTool.ArchetypeTool Couldn't get mro for Products.CMFFormController.Script.FSPythonScript Couldn't get mro for Products.CMFFormController.FSControllerValidator.FSControllerValidator Couldn't get mro for Products.CMFCore.FSPythonScript.FSPythonScript Couldn't get mro for Products.CMFFormController.FSControllerPythonScript.FSControllerPythonScript Couldn't get mro for Products.CMFCore.FSPageTemplate.FSPageTemplate Couldn't get mro for Products.CMFFormController.FSControllerPageTemplate.FSControllerPageTemplate Couldn't get mro for Products.CMFPlone.PropertiesTool.PropertiesTool Couldn't get mro for Products.CMFCore.FSZSQLMethod.FSZSQLMethod Couldn't get mro for Products.CMFPlone.FactoryTool.TempFolder Couldn't get mro for Products.Archetypes.OrderedBaseFolder.OrderedFolder Couldn't get mro for Products.Archetypes.examples.SimpleFolder.SimpleFolder 526 21 210
It seems that there are only several classes that cause problems: Products.CMFDefault.SkinnedFolder.SkinnedFolder Products.Archetypes.BaseFolder.BaseFolder Products.CMFCore.FS* Products.Archetypes.BaseFolder.BaseFolderMixin
AFAIK most of the other classes with mro problems are just subclasses of these classes above.
Jim made checkins to deal with the CMF-related MRO problems, on his 'mro-advanture-branch'::
=== CMF/CMFDefault/SkinnedFolder.py 1.15 => 1.15.6.1 === --- CMF/CMFDefault/SkinnedFolder.py:1.15 Sat Jun 28 12:31:21 2003 +++ CMF/CMFDefault/SkinnedFolder.py Fri Oct 31 14:24:19 2003 @@ -61,8 +61,7 @@ , )
- -class SkinnedFolder(CMFCatalogAware, PortalFolder): +class SkinnedFolder(PortalFolder): """ """ meta_type = 'Skinned Folder' @@ -70,6 +69,10 @@ security = ClassSecurityInfo()
manage_options = PortalFolder.manage_options + + indexObject = CMFCatalogAware.indexObject + unindexObject = CMFCatalogAware.unindexObject + reindexObject = CMFCatalogAware.reindexObject
def __call__(self): '''
=== CMF/CMFCore/FSObject.py 1.15 => 1.15.2.1 === --- CMF/CMFCore/FSObject.py:1.15 Fri Sep 12 20:46:40 2003 +++ CMF/CMFCore/FSObject.py Fri Oct 31 14:24:18 2003 @@ -18,7 +18,7 @@ from string import split from os import path, stat
-import Acquisition, Globals +import Acquisition, Globals, ExtensionClass from AccessControl import ClassSecurityInfo from OFS.SimpleItem import Item from DateTime import DateTime @@ -31,7 +31,7 @@
from OFS.Cache import Cacheable
-class FSObject(Acquisition.Implicit, Item, Cacheable): +class FSObject(Item, Cacheable, Acquisition.Implicit, ExtensionClass.Base): """FSObject is a base class for all filesystem based look-alikes.
Subclasses of this class mimic ZODB based objects like Image and
I imagine making similar small changes to Archetypes won't be hard.
Tres.
-- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote at 2003-10-31 12:14 -0500:
... Thoughts?
I am worried enough about breaking products that I'm inclined to go with option 3.
Does anybody think we ought to use the new algorithm (option 2)?
Please do not use the new algorithm (option 2). I like (multiple) inheritance and often used it to the extreme. I fear lots of my products will break when there are class definitions which fails due to MRO problems. -- Dieter
Jim Fulton wrote: <snip>
3. Use a hybrid schema. I'll call this the "encapsulated base" scheme.
<snip> Is it possible for the hybrid schema to generate a 'deprecation' warning for each instance of a class that doesn't match the requirements of the C3 resolution order when Zope is first started, and thus increase the chances that product authors will modify their products (or be coerced to do so :)), without actually breaking them? --Richard
I am also very, very worried about breaking b/w compatibility in Zope 2. I am responsible for about 15 sites, with say 10 distinct products each. Are you saying I have to evaluate/upgrade 150 products because I want to go to Zope 2.8(9)? No customer is going to pay for that effort (and I will likely not find the time to do it anyway) so I will be locked in 2.7-land forever. Bah! So yes, option 3 it must be. Let's save the fundamental changes for Zope 3. IMHO, Stefan On Freitag, Okt 31, 2003, at 18:14 Europe/Vienna, Jim Fulton wrote:
Thoughts?
I am worried enough about breaking products that I'm inclined to go with option 3.
Does anybody think we ought to use the new algorithm (option 2)?
Jim
-- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/
On Sat, Nov 01, 2003 at 12:18:47PM +0100, Stefan H. Holek wrote: | I am also very, very worried about breaking b/w compatibility in Zope 2. | | I am responsible for about 15 sites, with say 10 distinct products | each. Are you saying I have to evaluate/upgrade 150 products because I | want to go to Zope 2.8(9)? No customer is going to pay for that effort | (and I will likely not find the time to do it anyway) so I will be | locked in 2.7-land forever. Bah! | | So yes, option 3 it must be. Let's save the fundamental changes for | Zope 3. As far as I understand, even if we dont go with option 2 there will be *some* effort on migrating from 2.7 to 2.8. Adding the update of a few products may not be as bad as you painted here. I've analised the changes needed to make the current Plone from CVS to work with option 2 and they are not that intrusive. Its basically change the order of one or two base classes and nothing more than that. Now I have a interesting question to ask: *If* we go for option 3, and we fix some products to work with option 2, would that cause any incompatibility? -- Sidnei da Silva <sidnei@awkly.org> dreamcatching :: making your dreams come true http://awkly.org The first version always gets thrown away.
participants (11)
-
Casey Duncan -
Christian Heimes -
Dieter Maurer -
Jeremy Hylton -
Jim Fulton -
Richard Waid -
Sidnei da Silva -
Sidnei da Silva -
Stefan H. Holek -
Tim Peters -
Tres Seaver