New-style ExtensionClass, ZODB 3.3, and Zope 2.8 status
I spent what turned into a couple of weeks on new-style ExtensionClass (NSEC). This is a reimplementation of ExtendionClass as a new-style type. Actually, ExtensionClass.ExtensionClass is a subclass of type and ExtensionClass.Base is a subclass of object whose type is ExtensionClass.ExtensionClass. :) NSEC combines the features of ExtensionClass: - __of__ - __class_init__ - inheritedAttribute - traditional method-lookup order (for better or worse) with the features of new-style classes, including: - descriptors, - slots, - garbage collection NSEC is mostly backward compatible with old ExtensionClass. The major backward incompatability is that method attributes are handled differently. Unlike old EC, NSEC doesn't provide it's own method type. It uses standard Python methods. Old EC methods got their attributes from theor instances by combining their names with the attribute names. For example, if you ask for the '__roles__' attribute on a method named 'foo', the method will try to get the value my looking up the 'foo__roles__' attribute on the bound instance. Standard Python methods don't have this behavior. They get their attributes from their underlying functions. This has major implications for Zope 2's security machinery. More on this later. A few months ago, Jeremy Hylton and Tim Peters created a version of ZODB 3 that didn't use ExtensionClass. This was refered to as "ZODB 3.3" and was implemented on the zodb33-devel-branch (pronounced "devil branch" ;). I've extended this branch to all of Zope and integrated NSEC into the branch. Despite it's name, this branch should be thought of as the Zope 2.8 development branch. Work is underway to add multi-version concurrency control (no read conflicts :) to ZODB 3. This work is underway and will land in ZODB 3.3. During the time working on this, a plan for Zope 2.8 emerged that I think is pretty reasonable. Zope 2.8 will include: - NSEC - ZODB 3.3 and whetever other changes happen to be made while the work on NSEC and ZODB 3.3 is progressing. Currently, on the zodb33-devel-branch, Zope 2 runs and the great majority of it's tests pass with NSEC and ZODB 3.3. It you'd like to see this in action, feel free to check Zope out on the branch and play with it. I'm anxious to get this work merged into the Zope 2 CVS head, but have run into a problem that has stopped me for now. After 2 weeks, I really needed to get back to work on Zope 3. Gory details The issue preventing me from finishing this work is that a significant (though not massive) refactoring of the Zope 2 security machinery is needed to accomidate NSEC. This happens to be a refactoring that is loooong overdue. (It should have been done before Principia was created.) Long ago (even before Principia), our security model consisted of users, roles and objects. Objects required roles. Users had roles. To provide greater control, we added another layer of indirection: permissions. Now, object's require permissions to be accessed. Roles are granted permissions. Users are assigned roles. When we introduced permissions, we didn't want to refactor existing code, so we created permission-roles. Permission roles are complex computed attributes. When you access a __roles__ attribute on an object, a complex computation, making heavy-use of acquisition, is performed by to find all of the roles that have the required permission for the object. What can I say, this is a travesty. :) When we ask for the __roles__ of a method of an old-style ExtensionClass instance, the method turns around and accesses an attribute on the instance, triggering the roles computation. NSEC methods are just standard Python methods. We can store security assertions on them, but we can't use computed attributes. The whole computed roles hack won't work. I could make it work my making the NSEC implementation much more complex, but this would be wrong. The hack should have been done away with ages ago. We need to refactor the way security assertions (permission settings) are stored and accessed. We need to store required permissions (__permissions__) on objects. When we need to figure out roles, we need to compute them at a higher level. (For example, we could compute the needed roles in the zope security policy It's likely that we can speed security checks when doing this.) I estimate that the necessary refactoring would take me 3-5 days. The vast majority of the required time will be spent writing tests. I really need to focus on Zope 3 for a while, so I may not be able to get back to this soon. I think that this is an area where some volunteers could make a big difference. I'd be happy to work with some folks on this. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
<snip> | Gory details <snip> | | I estimate that the necessary refactoring would take me 3-5 | days. The vast majority of the required time will be spent writing | tests. I really need to focus on Zope 3 for a while, so I may not | be able to get back to this soon. I think that this is an area | where some volunteers could make a big difference. I'd be happy to | work with some folks on this. You didn't mention if this would have any impact on existing products. In case that was the case, would porting the zcml machinery and the security policy from zope3 completely out of question? This is something I would really love to see in zope2. -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher Avoid the Gates of Hell. Use Linux -- unknown source
Sidnei da Silva wrote:
<snip> | Gory details <snip> | | I estimate that the necessary refactoring would take me 3-5 | days. The vast majority of the required time will be spent writing | tests. I really need to focus on Zope 3 for a while, so I may not | be able to get back to this soon. I think that this is an area | where some volunteers could make a big difference. I'd be happy to | work with some folks on this.
You didn't mention if this would have any impact on existing products.
The expectation is that existing Zope 2 products will work with 2.8. The security APIs will be unchanged.
In case that was the case, would porting the zcml machinery and the security policy from zope3 completely out of question?
We want Zope 2.8 to happen as soon as possible, so we can start working on Zope 2.9. Zope 2.9 will be the release that incorprates major parts of Zope 3. We think that NSEC and ZODB 3.3 provide more than enough of a change for one release. :)
This is something I would really love to see in zope2.
Me too. We need to get 2.7 and 2.8 out the door as soon as we can so we can get working on 2.9. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Thu, Nov 13, 2003 at 04:35:52PM -0500, Jim Fulton wrote: | The expectation is that existing Zope 2 products will work with 2.8. | The security APIs will be unchanged. Great. | > In case that was the case, would porting the zcml machinery | >and the security policy from zope3 completely out of question? | | We want Zope 2.8 to happen as soon as possible, so we can start working | on Zope 2.9. Zope 2.9 will be the release that incorprates major parts | of Zope 3. | | We think that NSEC and ZODB 3.3 provide more than enough of a change for | one release. :) Agreed. It didn't hurt for asking though :) | > This is something I would really love to see in zope2. | | Me too. We need to get 2.7 and 2.8 out the door as soon as we | can so we can get working on 2.9. Yay! I can hardly wait for that one! -- Sidnei da Silva <sidnei@awkly.org> http://awkly.org - dreamcatching :: making your dreams come true http://plone.org/about/team#dreamcatcher From: Ean Schuessler <ean@novare.net> The unrecognized minister of propaganda, E -- Debian, joking
Jim Fulton wrote at 2003-11-13 15:22 -0500:
... We need to refactor the way security assertions (permission settings) are stored and accessed. We need to store required permissions (__permissions__) on objects. When we need to figure out roles, we need to compute them at a higher level. (For example, we could compute the needed roles in the zope security policy It's likely that we can speed security checks when doing this.)
I estimate that the necessary refactoring would take me 3-5 days. The vast majority of the required time will be spent writing tests. I really need to focus on Zope 3 for a while, so I may not be able to get back to this soon. I think that this is an area where some volunteers could make a big difference. I'd be happy to work with some folks on this.
Folklore says that Zope cannot protect attributes of simple types (because they do not provide the method magic that will be lost for NSEC). However, whenever I looked at ZopeSecurityPolicy (and I did often), I could not believe that this is true. I always thought, it would be easy to provide security declarations for simple type attributes, too. Of course, Zope cannot check a bare value of simple type, but usually it has "container" and/or "parent" and then checking would be easy by looking at related ("__roles__") attributes of the container/parent. I will see this weekend whether I have been true. If so, the same mechanism could (in principle) be used for methods. Bound methods can even be checked without "container/parent" as they allow access to the bound instance. -- Dieter
Dieter Maurer wrote at 2003-11-14 20:43 +0100:
Jim Fulton wrote at 2003-11-13 15:22 -0500:
... new security policy for NSEC ...
Folklore says that Zope cannot protect attributes of simple types (because they do not provide the method magic that will be lost for NSEC). ... Of course, Zope cannot check a bare value of simple type, but usually it has "container" and/or "parent" and then checking would be easy by looking at related ("__roles__") attributes of the container/parent.
I will see this weekend whether I have been true. If so, the same mechanism could (in principle) be used for methods.
Patch attached. -- Dieter
Dieter Maurer wrote:
Jim Fulton wrote at 2003-11-13 15:22 -0500:
... We need to refactor the way security assertions (permission settings) are stored and accessed. We need to store required permissions (__permissions__) on objects. When we need to figure out roles, we need to compute them at a higher level. (For example, we could compute the needed roles in the zope security policy It's likely that we can speed security checks when doing this.)
I estimate that the necessary refactoring would take me 3-5 days. The vast majority of the required time will be spent writing tests. I really need to focus on Zope 3 for a while, so I may not be able to get back to this soon. I think that this is an area where some volunteers could make a big difference. I'd be happy to work with some folks on this.
Folklore says that Zope cannot protect attributes of simple types (because they do not provide the method magic that will be lost for NSEC).
From your patch, I think you are talking about protection of attributes of non-simple types whose values are not simple types. That is, protecting the title attribute of a folder even though the value of the attribute is a string and thus, can't be assigned a __roles__ attribute.
However, whenever I looked at ZopeSecurityPolicy (and I did often), I could not believe that this is true. I always thought, it would be easy to provide security declarations for simple type attributes, too.
Of course, Zope cannot check a bare value of simple type, but usually it has "container" and/or "parent" and then checking would be easy by looking at related ("__roles__") attributes of the container/parent.
I will see this weekend whether I have been true. If so, the same mechanism could (in principle) be used for methods.
Bound methods can even be checked without "container/parent" as they allow access to the bound instance.
And a day later, Dieter Maurer wrote:
Patch attached.
So, when we are accessing a named attribute and the value doesn't have security data, look for security data on the object we got it from. I like the general idea. Hm, I suppose that this would let us keep the __roles__ travesty for now, which would allow us to proceed with 2.8 without doing the refactoring now. Very nice. (well, sort of ;) I'll see if this idea lets me get the "devil branch" merged to the trunk next week. (I need to make the next Z3 milestone release first.) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Dieter Maurer wrote
... protecting simple type attributes by roles ...
Patch attached.
I have a small optimization: if ( # start with inexpensive checks roles is not _noroles or name is None or value is None or container is None # now the expensive ones or hasattr(value,'__roles__') or not hasattr(aq_base(container), name + '__roles__') ): If we replace "hasattr(value,'__roles__')" above by "hasattr(aq_base(value),'__roles__')", then the (discarded) roles computation becomes a bit cheaper. -- Dieter
Dieter Maurer wrote:
Dieter Maurer wrote
... protecting simple type attributes by roles ...
Patch attached.
I have a small optimization:
if ( # start with inexpensive checks roles is not _noroles or name is None or value is None or container is None # now the expensive ones or hasattr(value,'__roles__') or not hasattr(aq_base(container), name + '__roles__') ):
If we replace "hasattr(value,'__roles__')" above by "hasattr(aq_base(value),'__roles__')", then the (discarded) roles computation becomes a bit cheaper.
I'm going with a bit simpler approach See my upcoming checkin. Basically, I've created a getRoles method to be used instead of "getattr(ob, '__roles__', _noroles)". One issue is that the security polict is not the only place where this special computation needs to be done. Also note that I had to get rid of the validateValue call. It's important that we always pass the name and container to code that needs to get roles. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote at 2003-11-22 12:14 -0500:
... Also note that I had to get rid of the validateValue call. It's important that we always pass the name and container to code that needs to get roles.
We need "name" and "container" only for objects that do not have their own "__roles__" attribute. This essientially means for the newly implemented methods. But, for bound methods, we can determine name and container from the objects. I would therefore keep "validateValue" and try to determine container and name. -- Dieter
Dieter Maurer wrote:
Jim Fulton wrote at 2003-11-22 12:14 -0500:
... Also note that I had to get rid of the validateValue call. It's important that we always pass the name and container to code that needs to get roles.
We need "name" and "container" only for objects that do not have their own "__roles__" attribute. This essientially means for the newly implemented methods. But, for bound methods, we can determine name and container from the objects.
We can determine the container, but we can't reliably the name:
class Foo: ... def eeek(self): pass ... oook = eeek
Foo().oook.__name__ 'eeek'
I would therefore keep "validateValue" and try to determine container and name.
No can do. I'll note that validateValue is hardly every used, at least in the Zope CVS. We really *do* need to refactor this stuff. What we're doing now is a stop-gap measure. Before releasing 2.8, I want to try to design what we want in 2.9 and have 2.8 include warnings and other guidence that will help people get ready for 2.9. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote: ...
Currently, on the zodb33-devel-branch, Zope 2 runs and the great majority of it's tests pass with NSEC and ZODB 3.3. It you'd like to see this in action, feel free to check Zope out on the branch and play with it.
Jeremy pointed out that this wasn't true. I had forgotten to check a bunch of things in. (I thought I had. :) Fixed now. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
participants (3)
-
Dieter Maurer -
Jim Fulton -
Sidnei da Silva