Re: [Zope-dev] SVN: Zope/branches/gsoc-python-2.5/utilities/mkzopeinstance.py sha -> hashlib
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote:
Log message for revision 92197: sha -> hashlib
That needs to be a conditional import, with a fallback to the old sha module: hashlib is not present in python 2.4. E.g.: try: import hashlib except ImportError: # Python < 2.5 import sha _sha = sha.new else: _sha = hashlib.sha1 and then use '_sha' in the code. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI9NDv+gerLs4ltQ4RAuAuAJ9KX9WaftLbd1rQwCJ+JVNnA5QAeQCfTNjw cq+cees5SixBxFY5J2SS3FI= =PmQk -----END PGP SIGNATURE-----
On 14.10.2008 19:03 Uhr, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Andreas Jung wrote:
Log message for revision 92197: sha -> hashlib
That needs to be a conditional import, with a fallback to the old sha module: hashlib is not present in python 2.4. E.g.:
try: import hashlib except ImportError: # Python< 2.5 import sha _sha = sha.new else: _sha = hashlib.sha1
and then use '_sha' in the code.
Depends on which Python versions we want/must support for the future. One option would to jump on Python 2.6 for the next releases (skipping Python 2.5 and dropping Python 2.4 support). Andreas
Andreas Jung wrote:
Depends on which Python versions we want/must support for the future. One option would to jump on Python 2.6 for the next releases (skipping Python 2.5 and dropping Python 2.4 support).
Just as a voice from the Plone crowd. With our current planned roadmap we won't move to Zope 2.12 / trunk until sometime next year at the earliest. A hard requirement of Python 2.6 would be fine with us. Hanno
On 14.10.2008 19:12 Uhr, Hanno Schlichting wrote:
Andreas Jung wrote:
Depends on which Python versions we want/must support for the future. One option would to jump on Python 2.6 for the next releases (skipping Python 2.5 and dropping Python 2.4 support).
Just as a voice from the Plone crowd. With our current planned roadmap we won't move to Zope 2.12 / trunk until sometime next year at the earliest.
A hard requirement of Python 2.6 would be fine with us.
Thanks for starting the discussion. Going for Python 2.6 also requires that we get the ZCA running on top of Python 2.6 until some time next year. Andreas
On Tue, Oct 14, 2008 at 2:16 PM, Andreas Jung <lists@zopyx.com> wrote:
Thanks for starting the discussion. Going for Python 2.6 also requires that we get the ZCA running on top of Python 2.6 until some time next year.
FWIW, that's what I've been working on. There's enough of the the ZCA working on Python 2.6 to run Zope 2. Would anything else be needed? I would say we are pretty much done as far as Zope 2 is concerned. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
On 14.10.2008 19:56 Uhr, Sidnei da Silva wrote:
On Tue, Oct 14, 2008 at 2:16 PM, Andreas Jung<lists@zopyx.com> wrote:
Thanks for starting the discussion. Going for Python 2.6 also requires that we get the ZCA running on top of Python 2.6 until some time next year.
FWIW, that's what I've been working on. There's enough of the the ZCA working on Python 2.6 to run Zope 2. Would anything else be needed?
not sure
I would say we are pretty much done as far as Zope 2 is concerned.
Cool. Thanks for your efforts and the guys working on this GSOC project. Andreas -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: info@zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting
Sidnei da Silva wrote:
On Tue, Oct 14, 2008 at 2:16 PM, Andreas Jung <lists@zopyx.com> wrote:
Thanks for starting the discussion. Going for Python 2.6 also requires that we get the ZCA running on top of Python 2.6 until some time next year.
FWIW, that's what I've been working on. There's enough of the the ZCA working on Python 2.6 to run Zope 2. Would anything else be needed? I would say we are pretty much done as far as Zope 2 is concerned.
Concerning the component architecture or the entire port? If the latter, what about the security review for untrusted code? Regards, Martijn
On 15.10.2008 14:25 Uhr, Martijn Faassen wrote:
Sidnei da Silva wrote:
On Tue, Oct 14, 2008 at 2:16 PM, Andreas Jung<lists@zopyx.com> wrote:
Thanks for starting the discussion. Going for Python 2.6 also requires that we get the ZCA running on top of Python 2.6 until some time next year. FWIW, that's what I've been working on. There's enough of the the ZCA working on Python 2.6 to run Zope 2. Would anything else be needed? I would say we are pretty much done as far as Zope 2 is concerned.
Concerning the component architecture or the entire port? If the latter, what about the security review for untrusted code?
You mean the review of RestrictedPython? Andreas
Hi there, On Wed, Oct 15, 2008 at 2:30 PM, Andreas Jung <lists@zopyx.com> wrote:
If the latter, what about the security review for untrusted code?
You mean the review of RestrictedPython?
Yes. If RestrictedPython is to be reviewed for changes, it *might* be easier to do this for 2.4 -> 2.5 instead of the big leap of two python versions. This depends a bit on how such a review is structured. Regards, Martijn
Martijn Faassen wrote:
On Wed, Oct 15, 2008 at 2:30 PM, Andreas Jung <lists@zopyx.com> wrote:
You mean the review of RestrictedPython?
Yes.
If RestrictedPython is to be reviewed for changes, it *might* be easier to do this for 2.4 -> 2.5 instead of the big leap of two python versions. This depends a bit on how such a review is structured.
It should be easier to never officially support Python 2.5, though. We do want to end up at supporting Python 2.6 in the end. I doubt that the review work of Python 2.4 -> 2.5 plus 2.5 -> 2.6 is smaller than the Python 2.4 -> 2.6 work. If it is easier doing this internally without having to support the intermediate result should still be less work. Hanno
On 15.10.2008 14:44 Uhr, Martijn Faassen wrote:
Hi there,
On Wed, Oct 15, 2008 at 2:30 PM, Andreas Jung<lists@zopyx.com> wrote:
If the latter, what about the security review for untrusted code? You mean the review of RestrictedPython?
Yes.
If RestrictedPython is to be reviewed for changes, it *might* be easier to do this for 2.4 -> 2.5 instead of the big leap of two python versions. This depends a bit on how such a review is structured.
If Python 2.6 is the latest official Python version of the 2.X line that there is a chance that this version will be supported by the Python community in the long term. So supporting Python 2.4 or Python 2.5 does not make much sense to me. RestrictedPython: I talked with Alan Runyan about this topic at the Plone conference....if we are serious then RP has to be reviewed. But who can review it and how long would it take. There are possibly only a limited number of people with the experience and skills for performing such a review. The last review took several person days (as far as I can remember). So the review process is possibly tied to some financial resource in order to sponsor the review. Andreas
Andreas Jung wrote:
RestrictedPython: I talked with Alan Runyan about this topic at the Plone conference....if we are serious then RP has to be reviewed. But who can review it and how long would it take. There are possibly only a limited number of people with the experience and skills for performing such a review. The last review took several person days (as far as I can remember). So the review process is possibly tied to some financial resource in order to sponsor the review.
The problem is two-fold: 1. Identify the people who are able (and could be convinced) to do the review 2. Find the money to sponsor this work Is the Zope Foundation in any position to lead this process? Who else would be in a position to do so? My impression is that the second problem requires only organization. There are enough companies around in the market that still use Zope2 as their strategic platform. I'd expect some of them to be willing to pay for this investment. In a joined effort this should not be a huge amount for any individual company. Hanno
Folks, I don't want to rain on your parade, but I already did a first pass at reviewing the changes in Python 2.5 and Python 2.6. There are no significant changes that I could spot so far. Apparently the major changes are: - Ternary 'if' - Context Managers ('with' keyword) Both of those are just syntactic sugar for already existing idioms and they are pretty much covered by already existing transformation of the ast tree. Also, the only new bytecode seems to be 'WITH_CLEANUP' which is part of the handling of the 'with' keyword. In general, RestrictedPython seems to be well-structured and the tests are easy to extend and understand for someone that played with ast (like the folks playing with z3c.pt). -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
On Wednesday 15 October 2008, Sidnei da Silva wrote:
I don't want to rain on your parade, but I already did a first pass at reviewing the changes in Python 2.5 and Python 2.6. There are no significant changes that I could spot so far. Apparently the major changes are:
I also did a review for Python 2.5 a while ago and all I could find were: * new missing builtins (zope.security) * A bug in the proxy code concerning correct slicing (zope.proxy, I think) I fixed both of these in the respective package. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Stephan Richter wrote:
On Wednesday 15 October 2008, Sidnei da Silva wrote:
I don't want to rain on your parade, but I already did a first pass at reviewing the changes in Python 2.5 and Python 2.6. There are no significant changes that I could spot so far. Apparently the major changes are:
I also did a review for Python 2.5 a while ago...
So does this mean RestrictedPython just had a bad emotional status in the community, but it is actually well proven and reviewed now? I always was under the impression that Jim feared the code and the required security audit was perceived as a major painful undertaking. Hanno
On Wed, Oct 15, 2008 at 11:28 AM, Hanno Schlichting <hannosch@hannosch.eu> wrote:
Stephan Richter wrote:
On Wednesday 15 October 2008, Sidnei da Silva wrote:
I don't want to rain on your parade, but I already did a first pass at reviewing the changes in Python 2.5 and Python 2.6. There are no significant changes that I could spot so far. Apparently the major changes are:
I also did a review for Python 2.5 a while ago...
So does this mean RestrictedPython just had a bad emotional status in the community, but it is actually well proven and reviewed now?
I always was under the impression that Jim feared the code and the required security audit was perceived as a major painful undertaking.
That was my perception too. But after looking at the code it is really not bad at all. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
Sidnei da Silva wrote:
I always was under the impression that Jim feared the code and the required security audit was perceived as a major painful undertaking.
That was my perception too. But after looking at the code it is really not bad at all.
The PyPy guys, who seem to be the authority on this kind of stuff, always shake their heads in woe at the mention of RestrictedPython. I've tried to understand what they perceive the "real problems" with it to be but haven't managed to extract anything meaningful :-( Perhaps one of you guys might have more luck engaging with them? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Oct 21, 2008, at 8:29 AM, Chris Withers wrote:
Sidnei da Silva wrote:
I always was under the impression that Jim feared the code and the required security audit was perceived as a major painful undertaking.
That was my perception too. But after looking at the code it is really not bad at all.
The PyPy guys, who seem to be the authority on this kind of stuff, always shake their heads in woe at the mention of RestrictedPython.
As well they should. Relying on it is a lot of work and brittle.
I've tried to understand what they perceive the "real problems" with it to be but haven't managed to extract anything meaningful :-(
The problem is that it it starts with an environment in which things are allowed by default, and takes things away. This means that if anything is forgotten, then you end up with holes. A better approach is to start with something that lets you do nothing and add capabilities that are known to be safe. This is why some folks get all excited about capability-based security. The Zope 3 security proxy approach is much better than relying solely on restricted Python, because, as with a capability-based approach, it prevents what isn't explicitly allowed. It does this (for the most part) without having to do code manipulation. It still uses restricted Python do deal in a narrow way with basic objects, like strings and numbers, that are unproxied. It's use of restricted Python is so narrow that it is far less problematic. It would be really great if Zope 2 would switch to security proxies, although the transition is likely to be painful. I'm not sure that the PyPy guys are really authorities on the sorts of problems we're trying to address, although there is some overlap. If I remember correctly, they are just focussed on protecting the system from untrusted scripts. Our problem is harder because we also want to protect objects available to the scripts. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
The problem is that it it starts with an environment in which things are allowed by default, and takes things away. This means that if anything is forgotten, then you end up with holes.
Isn't there a way we could change the AST manipulation such that we start with nothing and only allow opcodes as and when they're added to the RestrictedPython implementation?
The Zope 3 security proxy approach is much better
I agree, but it doesn't solve all the problems. My understanding of the original set of requirements which we're trying to solve here was basically that of Python Scripts: to allow python code to be written through the web. This means: - restricting access to atributes of objects (security proxies give us this, right?) - restricting access to features of the language such as imports such that unsafe things such as stripping security proxies can't be done. (security proxies *don't* give us this, right?) ...and some nice to haves: - restricting memory used by executing the code - restricting cpu used by executing the code I know RestrictedPython doesn't support these last two very well, but there are hints that it would have liked to if it could.
to do code manipulation. It still uses restricted Python do deal in a narrow way with basic objects, like strings and numbers, that are unproxied. It's use of restricted Python is so narrow that it is far less problematic. It would be really great if Zope 2 would switch to security proxies, although the transition is likely to be painful.
How would you see the problem of the second requirement above being met?
I'm not sure that the PyPy guys are really authorities on the sorts of problems we're trying to address, although there is some overlap.
Who is then? ;-)
If I remember correctly, they are just focussed on protecting the system from untrusted scripts. Our problem is harder because we also want to protect objects available to the scripts.
Indeed :-( Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Oct 24, 2008, at 10:01 AM, Chris Withers wrote:
Jim Fulton wrote:
The problem is that it it starts with an environment in which things are allowed by default, and takes things away. This means that if anything is forgotten, then you end up with holes.
Isn't there a way we could change the AST manipulation such that we start with nothing and only allow opcodes as and when they're added to the RestrictedPython implementation?
No. we're starting with an existing program written in a Python script or expression. We then have to sanitize it.
The Zope 3 security proxy approach is much better
I agree, but it doesn't solve all the problems. My understanding of the original set of requirements which we're trying to solve here was basically that of Python Scripts: to allow python code to be written through the web. This means:
- restricting access to atributes of objects (security proxies give us this, right?)
Yes
- restricting access to features of the language such as imports such that unsafe things such as stripping security proxies can't be done. (security proxies *don't* give us this, right?)
Yes, they do, as long as all builtins, including __import__, are wrapped and as long as you only include safe builtins.
...and some nice to haves:
- restricting memory used by executing the code - restricting cpu used by executing the code
These require deep changes to the interpreter.
I know RestrictedPython doesn't support these last two very well, but there are hints that it would have liked to if it could.
It can't support these at all.
I'm not sure that the PyPy guys are really authorities on the sorts of problems we're trying to address, although there is some overlap.
Who is then? ;-)
In the Python community, I;d say we are. Beyond that, I don't know. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
On Oct 24, 2008, at 10:01 AM, Chris Withers wrote:
Jim Fulton wrote:
The problem is that it it starts with an environment in which things are allowed by default, and takes things away. This means that if anything is forgotten, then you end up with holes.
Isn't there a way we could change the AST manipulation such that we start with nothing and only allow opcodes as and when they're added to the RestrictedPython implementation?
No. we're starting with an existing program written in a Python script or expression. We then have to sanitize it.
Could that sanitization could be a filter that lets nothing through, and then builds from there?
- restricting access to features of the language such as imports such that unsafe things such as stripping security proxies can't be done. (security proxies *don't* give us this, right?)
Yes, they do, as long as all builtins, including __import__, are wrapped and as long as you only include safe builtins.
The PyPy guys seem pretty sure you can always break out of this. How would you set up an environment which would meet your criterea and couldn't be broken out of?
...and some nice to haves:
- restricting memory used by executing the code - restricting cpu used by executing the code
These require deep changes to the interpreter.
...which the PyPy guys seem to have achieved, if only we could leverage it.
I'm not sure that the PyPy guys are really authorities on the sorts of problems we're trying to address, although there is some overlap.
Who is then? ;-)
In the Python community, I;d say we are. Beyond that, I don't know.
Who is "we"? It certainly doesn't include me :-S Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Jim Fulton wrote:
On Oct 24, 2008, at 10:01 AM, Chris Withers wrote:
Jim Fulton wrote:
The problem is that it it starts with an environment in which things are allowed by default, and takes things away. This means that if anything is forgotten, then you end up with holes. Isn't there a way we could change the AST manipulation such that we start with nothing and only allow opcodes as and when they're added to the RestrictedPython implementation? No. we're starting with an existing program written in a Python script or expression. We then have to sanitize it.
Could that sanitization could be a filter that lets nothing through, and then builds from there?
Yes, and if such a change leads to faster adoption of new Python releases by Zope, then it seems like a worthwhile effort. Instead of a tree mutator, RestrictedPython would use a tree copier with a filter. New Python features would initially not be supported at all, but that's better than accidentally, insecurely, supporting new features. Shane
Shane Hathaway wrote:
Yes, and if such a change leads to faster adoption of new Python releases by Zope, then it seems like a worthwhile effort. Instead of a tree mutator, RestrictedPython would use a tree copier with a filter. New Python features would initially not be supported at all, but that's better than accidentally, insecurely, supporting new features.
This sounds like a great idea apart from the caveats I've already mentioned about breaking out of an environment comprised of only safe builtins, which no-one replied to yet. How would we get going on this? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Hanno Schlichting wrote:
Stephan Richter wrote:
On Wednesday 15 October 2008, Sidnei da Silva wrote:
I don't want to rain on your parade, but I already did a first pass at reviewing the changes in Python 2.5 and Python 2.6. There are no significant changes that I could spot so far. Apparently the major changes are: I also did a review for Python 2.5 a while ago...
So does this mean RestrictedPython just had a bad emotional status in the community, but it is actually well proven and reviewed now?
It has been reviewed by Jim for Python 2.4. When he did this, he wrote notes.txt which gives you a quick overview over the internals. The GSoC-sponsored efforts to port Zope 3 and Zope 2 to Python 2.5 included a review of RestrictedPython as well. As far as I can tell, the only changes to RestrictedPython were made by Sidnei a couple of days ago when he fixed it up for the new keywords in Python 2.6 and added some tests for features new in Python 2.5 and 2.6.
I always was under the impression that Jim feared the code and the required security audit was perceived as a major painful undertaking.
It's certainly something that should be undertaken carefully. New syntactical features (such as the =+ operator in the past, or the 'with' statement or inplace 'if' now) have to be analyzed with respect to their bytecode. Bytecode changes have to be tracked as well. It looks like Sidnei is on top of that already, though, which is certainly great to hear! Go Sidnei! :)
Hi there, On Wed, Oct 15, 2008 at 3:35 PM, Andreas Jung <lists@zopyx.com> wrote: [snip]
If Python 2.6 is the latest official Python version of the 2.X line that there is a chance that this version will be supported by the Python community in the long term. So supporting Python 2.4 or Python 2.5 does not make much sense to me.
RestrictedPython: I talked with Alan Runyan about this topic at the Plone conference....if we are serious then RP has to be reviewed. But who can review it and how long would it take. There are possibly only a limited number of people with the experience and skills for performing such a review. The last review took several person days (as far as I can remember). So the review process is possibly tied to some financial resource in order to sponsor the review.
Yes, I agree. How do we go about finding sponsors and people willing to do the review? Regards, Martijn
participants (10)
-
Andreas Jung -
Chris Withers -
Hanno Schlichting -
Jim Fulton -
Martijn Faassen -
Philipp von Weitershausen -
Shane Hathaway -
Sidnei da Silva -
Stephan Richter -
Tres Seaver