2.9.4? reStructuredText support?
I think we should do a 2.9.4 release to incorporate the recent hot fix. This is easy for me to say, since I won't be doing it. :) Because this recent fix actually fixed the same problem that the previous hot fix was supposed to fix, I think someone needs to work up some decent tests. This is not a trivial task, bit it is necessary. If no one is willing to do this, I think we need to drop the TTW reStructuredText support from Zope 2, as it is too great a risk. BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
+1 on dropping it completely, but then I hate all types of structured text so I doubt I'm in the majority... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Jim Fulton wrote:
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
+1 on dropping it completely, but then I hate all types of structured text so I doubt I'm in the majority...
zwiki for example just recently switched to rst as default page type. how about making usage of directives configuable ? Michael -- http://zope.org/Members/d2m http://planetzope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
I think we should do a 2.9.4 release to incorporate the recent hot fix. This is easy for me to say, since I won't be doing it. :)
Because this recent fix actually fixed the same problem that the previous hot fix was supposed to fix, I think someone needs to work up some decent tests. This is not a trivial task, bit it is necessary. If no one is willing to do this, I think we need to drop the TTW reStructuredText support from Zope 2, as it is too great a risk.
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
I actually checked in a fix last night which disabled the 'include' directive completely, and blew out the 'raw' directive for the 'file' and 'url' options. The fix went into the 2.7, 2.8, 2.9, and 2.10 branches and the head. We neeed to do a Zope3 3.2.2 release to go along with 2.9.4 (and maybe a ZODB 3.4.x release as well). Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFErn3I+gerLs4ltQ4RAubeAJ9LmNhC1ucX9jgCMH2YNXy9uMQe6gCfQsDS 98DkWrI4a32Cqhc7LMu94yY= =frHO -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Philipp von Weitershausen wrote:
Tres Seaver wrote:
We neeed to do a Zope3 3.2.2 release to go along with 2.9.4
Two reasons: - Better release management for Zope2: we should be shipping with appropriately tagged releases of our dependencies (all the svn:externals, actually), including particularly ZODB, Five, and Zope3. We could get by with only tagging an "internal releasw" of Zope3 (which is actually where ZODB has been for a while), but: - The 3.2 branch is about to be retired as the actively-maintained version: we should do a release anyway to "tie it off," consolidating the not-trivial set of bugfixes made since March. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEr5ho+gerLs4ltQ4RAvfgAJ9/2HIdTTX7JMCoC5SZ6TpK+CuIKACeN0TT Qzd9Kz0d6pTvRfzCwMlnaK0= =zvuW -----END PGP SIGNATURE-----
--On 8. Juli 2006 07:35:04 -0400 Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Philipp von Weitershausen wrote:
Tres Seaver wrote:
We neeed to do a Zope3 3.2.2 release to go along with 2.9.4
Two reasons:
- Better release management for Zope2: we should be shipping with appropriately tagged releases of our dependencies (all the svn:externals, actually), including particularly ZODB, Five, and Zope3. We could get by with only tagging an "internal releasw" of Zope3 (which is actually where ZODB has been for a while), but:
If Zope 3.2.2 has no issues with Docutils then there is little need to tie a 2.9.4 release to a new Zope 3.2.3 release. A Zope 3.2.3 would be nice in general if there are enough fixes in Zope 3.2 (but not because of the Docutils issue).
- The 3.2 branch is about to be retired as the actively-maintained version: we should do a release anyway to "tie it off," consolidating the not-trivial set of bugfixes made since March.
I agree. -aj
Tres' patch (removing 'include' and 'raw' altogether) looks fairly low on violence to me. No reason to drop reST from Zope, IMO. Stefan On 7. Jul 2006, at 17:03, Jim Fulton wrote:
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
-- Anything that, in happening, causes something else to happen, causes something else to happen. --Douglas Adams
I'm not sure what effect "disabling TTW RST" would have on Zwiki. Definitely lots of Zwiki users using and enjoying RST. I think losing include is fine, raw will be missed, but oh well. Thanks for working on this, -Simon
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Simon Michael wrote:
I'm not sure what effect "disabling TTW RST" would have on Zwiki. Definitely lots of Zwiki users using and enjoying RST. I think losing include is fine, raw will be missed, but oh well.
The hotfix disables raw: the fixed docutils in the checkin actually just disables 'file' and 'url' optoins for raw: http://svn.zope.org/docutils/tags/0.4.0-zope/ Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFErrnh+gerLs4ltQ4RAjibAJ0RLwVEkuqZBrFhDM68atof7Rh3ngCePgVB qXcGTg9YZTaLc2EGEvcijRU= =LEBG -----END PGP SIGNATURE-----
On Jul 7, 2006, at 12:17 PM, Stefan H. Holek wrote:
Tres' patch (removing 'include' and 'raw' altogether) looks fairly low on violence to me. No reason to drop reST from Zope, IMO.
Well, I wouldn't want to apply the patch for Z3, as we use reST on the file system and include and raw have legitimate uses. In fact, I think include and maybe even include of "system" files could have use in some TTW applications. In fact, Tres' patch would make it hard to a well-written 3rd-party Zope 2 app to use raw in legitimate way. Don't get me wrong. I like Tres' patch. It was absolutely the best patch for the situation at hand. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 7. Juli 2006 11:03:06 -0400 Jim Fulton <jim@zope.com> wrote:
I think we should do a 2.9.4 release to incorporate the recent hot fix. This is easy for me to say, since I won't be doing it. :)
Because this recent fix actually fixed the same problem that the previous hot fix was supposed to fix, I think someone needs to work up some decent tests. This is not a trivial task, bit it is necessary. If no one is willing to do this, I think we need to drop the TTW reStructuredText support from Zope 2, as it is too great a risk.
Dropping TTW reST is absolutely not an option. I breaks backward compatibility.
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
Tres' patch is looking in fine to me. I don't see a need right now for dropping reST with having file inclusing *removed*. -aj
On Jul 8, 2006, at 1:11 AM, Andreas Jung wrote:
--On 7. Juli 2006 11:03:06 -0400 Jim Fulton <jim@zope.com> wrote:
I think we should do a 2.9.4 release to incorporate the recent hot fix. This is easy for me to say, since I won't be doing it. :)
Because this recent fix actually fixed the same problem that the previous hot fix was supposed to fix, I think someone needs to work up some decent tests. This is not a trivial task, bit it is necessary. If no one is willing to do this, I think we need to drop the TTW reStructuredText support from Zope 2, as it is too great a risk.
Dropping TTW reST is absolutely not an option. I breaks backward compatibility.
Sorry, security trumps backward compatibility.
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
Tres' patch is looking in fine to me. I don't see a need right now for dropping reST with having file inclusing *removed*.
Has anyone written tests for Tres' patch? Apparently no one wrote adequate tests for the last hot fix, which helped put us in this situation. I'm not opposed to keeping TTW reST if *someone takes responsibility* for it. I don't see this happening. If someone cares enough about TTW reST to stand behind it and properly address the security risks by writing tests, then great. Otherwise it has to go. I also think Tres' patch was the right emergency measure, but I'm not so sure it is the right long-term fix. It reflects a sorry, but perhaps sadly accurate, view of the community's commitment to quality. :( Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 8. Juli 2006 07:45:01 -0400 Jim Fulton <jim@zope.com> wrote:
On Jul 8, 2006, at 1:11 AM, Andreas Jung wrote:
--On 7. Juli 2006 11:03:06 -0400 Jim Fulton <jim@zope.com> wrote:
I think we should do a 2.9.4 release to incorporate the recent hot fix. This is easy for me to say, since I won't be doing it. :)
Because this recent fix actually fixed the same problem that the previous hot fix was supposed to fix, I think someone needs to work up some decent tests. This is not a trivial task, bit it is necessary. If no one is willing to do this, I think we need to drop the TTW reStructuredText support from Zope 2, as it is too great a risk.
Dropping TTW reST is absolutely not an option. I breaks backward compatibility.
Sorry, security trumps backward compatibility.
Only if there is no other option. Tres' patch seems to resolve this issue and with further testing there is no need to remove the functionality.
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
Tres' patch is looking in fine to me. I don't see a need right now for dropping reST with having file inclusing *removed*.
Has anyone written tests for Tres' patch? Apparently no one wrote adequate tests for the last hot fix, which helped put us in this situation.
I'm not opposed to keeping TTW reST if *someone takes responsibility* for it. I don't see this happening. If someone cares enough about TTW reST to stand behind it and properly address the security risks by writing tests, then great.
There is currently litte need to break this over the knee. We have a hotfix, we have a stripped down version of Docutils. We have some time until the next releases. Perhaps nobody had time so far (at least me) for writing further tests..that does not mean that nobody takes responsibility. If we would rip of everything from Zope 2 where nobody takes over responsibility....what would be left? In addition I don't see a big problem for Zope-only(!) apps. Using reST in Zope requires access to the ZMI which is in general available only to trusted users. Removing TTW-editing of reST in Zope does *not* solve any problem e.g. for Plone where reST can be edited through the Plone UI by usually untrusted users. It is *our* task to make reST (basically Docutils) secure enough. It's safe enough for Zope-only apps but I agree that the Docutils code and the "hotfix" requires some more testing and review.
Otherwise it has to go.
No :-)
It reflects a sorry, but perhaps sadly accurate, view of the community's commitment to quality. :(
Sorry, I've no idea what you mean with this remark. Andreas
On Jul 8, 2006, at 8:12 AM, Andreas Jung wrote:
--On 8. Juli 2006 07:45:01 -0400 Jim Fulton <jim@zope.com> wrote:
On Jul 8, 2006, at 1:11 AM, Andreas Jung wrote:
--On 7. Juli 2006 11:03:06 -0400 Jim Fulton <jim@zope.com> wrote:
I think we should do a 2.9.4 release to incorporate the recent hot fix. This is easy for me to say, since I won't be doing it. :)
Because this recent fix actually fixed the same problem that the previous hot fix was supposed to fix, I think someone needs to work up some decent tests. This is not a trivial task, bit it is necessary. If no one is willing to do this, I think we need to drop the TTW reStructuredText support from Zope 2, as it is too great a risk.
Dropping TTW reST is absolutely not an option. I breaks backward compatibility.
Sorry, security trumps backward compatibility.
Only if there is no other option. Tres' patch seems to resolve this issue and with further testing there is no need to remove the functionality.
"Seems" isn't good enough. It's not even close. The hot fix last fall "seemed" to fix the problem. :( Heck (I wanted to use another 4-letter-word, because I'm getting kinda angry), even the current patch hasn't been adequately tested. Michael suggested that the patch needed to be tested against all recent Zope versions. Has this been done? I don't think so. Do we even have *tests* that it works? I doubt it. I don't fault Tres for this. We needed to get the hotfix out in a hurry. Do I think Tres should have to write tests for this? After he plugged a hole in something he didn't want included in the first place? Heck no.
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
Tres' patch is looking in fine to me. I don't see a need right now for dropping reST with having file inclusing *removed*.
Has anyone written tests for Tres' patch? Apparently no one wrote adequate tests for the last hot fix, which helped put us in this situation.
I'm not opposed to keeping TTW reST if *someone takes responsibility* for it. I don't see this happening. If someone cares enough about TTW reST to stand behind it and properly address the security risks by writing tests, then great.
There is currently litte need to break this over the knee. We have a hotfix, we have a stripped down version of Docutils. We have some time until the next releases. Perhaps nobody had time so far (at least me) for writing further tests..that does not mean that nobody takes responsibility. If we would rip of everything from Zope 2 where nobody takes over responsibility....what would be left?
In addition I don't see a big problem for Zope-only(!) apps. Using reST in Zope requires access to the ZMI which is in general available only to trusted users. Removing TTW-editing of reST in Zope does *not* solve any problem e.g. for Plone where reST can be edited through the Plone UI by usually untrusted users. It is *our* task to make reST (basically Docutils) secure enough. It's safe enough for Zope-only apps but I agree that the Docutils code and the "hotfix" requires some more testing and review.
Otherwise it has to go.
No :-)
Wrong. Sorry, I'll invoke Pope if I have to. I'm not talking about 2.9 and earlier. but if no one takes responsibility for this feature, wi'll rip it out of 2.10.
It reflects a sorry, but perhaps sadly accurate, view of the community's commitment to quality. :(
Sorry, I've no idea what you mean with this remark.
Tres came up with this sledge hammer because he has no confidence in people's willingness to test and implement this feature properly. Sadly, he has good evidence for this point of view. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Jul 8, 2006, at 8:12 AM, Andreas Jung wrote:
--On 8. Juli 2006 07:45:01 -0400 Jim Fulton <jim@zope.com> wrote:
Only if there is no other option. Tres' patch seems to resolve this issue and with further testing there is no need to remove the functionality.
"Seems" isn't good enough. It's not even close. The hot fix last fall "seemed" to fix the problem. :(
That's is still not an argument. I'll agree with you when we are all convinced that we are all unable to fix this issue if a reasonable effort or when come to conclusion that Docutils is a problem by itself...sorry, but we are not at that point so far.
Otherwise it has to go.
No :-)
Wrong. Sorry, I'll invoke Pope if I have to.
Sorry Jim, that's weak. See above. I'll accept the decision of the Pope as long as it is comprehensible...so far it is not.
Tres came up with this sledge hammer because he has no confidence in people's willingness to test and implement this feature properly.
I am fine with the sledge-hammer. I've never claimed that we need to support file insertion and raw support in any way. We don't need, we can kick it. But removing or disabling a feature because we are possibly incompetent would be just ridiculous. Andreas
On Jul 8, 2006, at 9:17 AM, Andreas Jung wrote:
On Jul 8, 2006, at 8:12 AM, Andreas Jung wrote:
--On 8. Juli 2006 07:45:01 -0400 Jim Fulton <jim@zope.com> wrote:
Only if there is no other option. Tres' patch seems to resolve this issue and with further testing there is no need to remove the functionality.
"Seems" isn't good enough. It's not even close. The hot fix last fall "seemed" to fix the problem. :(
That's is still not an argument. I'll agree with you when we are all convinced that we are all unable to fix this issue if a reasonable effort or when come to conclusion that Docutils is a problem by itself...sorry, but we are not at that point so far.
Otherwise it has to go.
No :-)
Wrong. Sorry, I'll invoke Pope if I have to.
Sorry Jim, that's weak. See above. I'll accept the decision of the Pope as long as it is comprehensible...so far it is not.
Maybe you aren't listening.
Tres came up with this sledge hammer because he has no confidence in people's willingness to test and implement this feature properly.
I am fine with the sledge-hammer. I've never claimed that we need to support file insertion and raw support in any way. We don't need, we can kick it. But removing or disabling a feature because we are possibly incompetent would be just ridiculous.
I can live with the sledge hammer for Zope 2. All I ask for is tests. If there are tests for each way of invoking reST through the web that verifies that file-inclusion isn't enabled, then it's alright with me if the sledge hammer is used to make the tests pass. I won't tolerate an untested feature with so much security risk. I'll also note that the sledgehammer might not itself be safe in the presense of the various reload products for Zope 3. Would Tres' patch be defeated by reloading docutils.parsers.rst.directives.misc? Is there a chance that a reload product could reload this module and undo the fix? I dunno. It is worrisome. You seem to be the only one championing TTW reST? Are you unwilling to write the tests necessary to keep it? If so, it's hard to have any sympathy for your desire to keep it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 8. Juli 2006 09:53:47 -0400 Jim Fulton <jim@zope.com> wrote:
Maybe you aren't listening.
I am listening very well.
Tres came up with this sledge hammer because he has no confidence in people's willingness to test and implement this feature properly.
I am fine with the sledge-hammer. I've never claimed that we need to support file insertion and raw support in any way. We don't need, we can kick it. But removing or disabling a feature because we are possibly incompetent would be just ridiculous.
I can live with the sledge hammer for Zope 2. All I ask for is tests.
If there are tests for each way of invoking reST through the web that verifies that file-inclusion isn't enabled, then it's alright with me if the sledge hammer is used to make the tests pass. I won't tolerate an untested feature with so much security risk.
Yes, someone has to write the tests at some time, soon. As I pointed out the risk is minimal for Zope-apps because you need to have access to the ZMI.. so what are security concerns in this case? And file inclusion won't work if the related code is stripped off...so what are your security concerns in this case?
I'll also note that the sledgehammer might not itself be safe in the presence of the various reload products for Zope 3. Would Tres' patch be defeated by reloading docutils.parsers.rst.directives.misc? Is there a chance that a reload product could reload this module and undo the fix? I dunno. It is worrisome.
You seem to be the only one championing TTW reST?
I am only champion against crude removal of features and against and a shortsighted preception.
Are you unwilling to write the tests necessary to keep it?
This is really not the point. As release manager I am allowed to speak up. But that does not imply I have to fix all and everything. Andreas
On Jul 8, 2006, at 10:09 AM, Andreas Jung wrote:
--On 8. Juli 2006 09:53:47 -0400 Jim Fulton <jim@zope.com> wrote:
...
Tres came up with this sledge hammer because he has no confidence in people's willingness to test and implement this feature properly.
I am fine with the sledge-hammer. I've never claimed that we need to support file insertion and raw support in any way. We don't need, we can kick it. But removing or disabling a feature because we are possibly incompetent would be just ridiculous.
I can live with the sledge hammer for Zope 2. All I ask for is tests.
If there are tests for each way of invoking reST through the web that verifies that file-inclusion isn't enabled, then it's alright with me if the sledge hammer is used to make the tests pass. I won't tolerate an untested feature with so much security risk.
Yes, someone has to write the tests at some time, soon.
Right. Before 2.10.
As I pointed out the risk is minimal for Zope-apps because you need to have access to the ZMI..
No, it's not. Getting at arbitrary files is not acceptable from the ZMI.
so what are security concerns in this case? And file inclusion won't work if the related code is stripped off...so what are your security concerns in this case?
I am concerned by the lack of tests. Whoever created the last hot fix was sure the problem was fixed. They were wrong and we're paying the price.
I'll also note that the sledgehammer might not itself be safe in the presence of the various reload products for Zope 3. Would Tres' patch be defeated by reloading docutils.parsers.rst.directives.misc? Is there a chance that a reload product could reload this module and undo the fix? I dunno. It is worrisome.
You seem to be the only one championing TTW reST?
I am only champion against crude removal of features and against and a shortsighted preception.
That doesn't deserve an answer.
Are you unwilling to write the tests necessary to keep it?
This is really not the point. As release manager I am allowed to speak up. But that does not imply I have to fix all and everything.
Yes, it really is the point. We've had a serious security failure due to a lack of adequate testing. This is not acceptable. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 8. Juli 2006 10:16:30 -0400 Jim Fulton <jim@zope.com> wrote:
Yes, someone has to write the tests at some time, soon.
Right. Before 2.10.
...so we have some time...
As I pointed out the risk is minimal for Zope-apps because you need to have access to the ZMI..
No, it's not. Getting at arbitrary files is not acceptable from the ZMI.
...which won't be possible with *removed* file inclusion code...
so what are security concerns in this case? And file inclusion won't work if the related code is stripped off...so what are your security concerns in this case?
I am concerned by the lack of tests. Whoever created the last hot fix was sure the problem was fixed. They were wrong and we're paying the price.
This can happen all the time. A problem in the release process does not justify the removal of a feature until we tried our best to solve the problem. Use the sledge hammer as a last resort.
You seem to be the only one championing TTW reST?
I am only champion against crude removal of features and against and a shortsighted preception.
That doesn't deserve an answer.
Sorry for being harsh but the lack of tests after two days is really not appropriate approach.
Are you unwilling to write the tests necessary to keep it?
This is really not the point. As release manager I am allowed to speak up. But that does not imply I have to fix all and everything.
Yes, it really is the point.
No, it is not. I haven't worked on the hotfix...so why would it be up to me write tests? I don't want blame Tres...he was doing his best in the situation...but this is totally unrelated that I would be unwilling to write tests in this case. I would have helped but it was late evening and at some point you need some sleep... Andreas
We've had a serious security failure due to a lack of adequate testing. This is not acceptable.
On Jul 8, 2006, at 10:41 AM, Andreas Jung wrote:
--On 8. Juli 2006 10:16:30 -0400 Jim Fulton <jim@zope.com> wrote:
Yes, someone has to write the tests at some time, soon.
Right. Before 2.10.
...so we have some time...
Sadly, but that's a different problem.
As I pointed out the risk is minimal for Zope-apps because you need to have access to the ZMI..
No, it's not. Getting at arbitrary files is not acceptable from the ZMI.
...which won't be possible with *removed* file inclusion code...
Good, right some tests and prove it.
so what are security concerns in this case? And file inclusion won't work if the related code is stripped off...so what are your security concerns in this case?
I am concerned by the lack of tests. Whoever created the last hot fix was sure the problem was fixed. They were wrong and we're paying the price.
This can happen all the time. A problem in the release process does not justify the removal of a feature until we tried our best to solve the problem. Use the sledge hammer as a last resort.
The problem in the release process was an inattention to basic process. This is unacceptable in a security-related issue.
You seem to be the only one championing TTW reST?
I am only champion against crude removal of features and against and a shortsighted preception.
That doesn't deserve an answer.
Sorry for being harsh but the lack of tests after two days is really not appropriate approach.
Who said anything about 2 days. I said we need tests and we need someone to be responsible for this feature or we'll have to drop the feature. I didn't say we had to drop it right this second.
Are you unwilling to write the tests necessary to keep it?
This is really not the point. As release manager I am allowed to speak up. But that does not imply I have to fix all and everything.
Yes, it really is the point.
No, it is not. I haven't worked on the hotfix...so why would it be up to me write tests?
It's not. The person who *did* write the hot-fix didn't want the feature in the first place. Tres stepped up and helped us in an emergency. I imagine that he isn't signing up to maintaint the feature.
I don't want blame Tres...he was doing his best in the situation...but this is totally unrelated that I would be unwilling to write tests in this case.
That's fine.
I would have helped but it was late evening and at some point you need some sleep...
That's fine too. I know it was late and you tried to help. You were there and helping and I appreciate it. I really do. A lot. So, we're past the emergency -- we hope. The problem is that we have a feature with an implementation that is a security risk. We have a feature that doesn't seem to have a champion -- because no one is willing to come forward and maintain it properly. In that case, the feature is orphaned and we have to get rid of it. It is too risky to keep it under the circumstances. I'm perfectly willing to keep it if someone takes responsibility. That hasn't happened yet. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 8. Juli 2006 14:42:31 -0400 Jim Fulton <jim@zope.com> wrote:
This can happen all the time. A problem in the release process does not justify the removal of a feature until we tried our best to solve the problem. Use the sledge hammer as a last resort.
The problem in the release process was an inattention to basic process. This is unacceptable in a security-related issue.
This can happen all the time, it should not happen..but it happened (likely because the private emails around this issue caused a lot of trouble and noise).
You seem to be the only one championing TTW reST?
I am only champion against crude removal of features and against and a shortsighted preception.
That doesn't deserve an answer.
Sorry for being harsh but the lack of tests after two days is really not appropriate approach.
Who said anything about 2 days. I said we need tests and we need someone to be responsible for this feature or we'll have to drop the feature. I didn't say we had to drop it right this second.
It sounded to me that way..
Are you unwilling to write the tests necessary to keep it?
This is really not the point. As release manager I am allowed to speak up. But that does not imply I have to fix all and everything.
Yes, it really is the point.
No, it is not. I haven't worked on the hotfix...so why would it be up to me write tests?
It's not. The person who *did* write the hot-fix didn't want the feature in the first place. Tres stepped up and helped us in an emergency. I imagine that he isn't signing up to maintaint the feature.
When you talk of "the feature"...you mean file inclusion? This feature was not supposed to be there. It was never a goal of reST to provide this feature. So Tres' solution (removing the code) is perfectly fine. There are a lot of modules where we don't want to take over the maintainer. The important thing is that we have clever ppl who understand the code and can deal with such problems in such a case.
The problem is that we have a feature with an implementation that is a security risk. We have a feature that doesn't seem to have a champion -- because no one is willing to come forward and maintain it properly. In that case, the feature is orphaned and we have to get rid of it. It is too risky to keep it under the circumstances.
I'm perfectly willing to keep it if someone takes responsibility. That hasn't happened yet.
See above...it's not a question of general responsibility...it's a question of taking over the responsibility for a particular problem in particular situation...of course maintainers for modules are highly welcome...things are as they are in the Zope 2 world... Andreas
On Jul 8, 2006, at 3:06 PM, Andreas Jung wrote:
No, it is not. I haven't worked on the hotfix...so why would it be up to me write tests?
It's not. The person who *did* write the hot-fix didn't want the feature in the first place. Tres stepped up and helped us in an emergency. I imagine that he isn't signing up to maintaint the feature.
When you talk of "the feature"...you mean file inclusion? This feature was not supposed to be there. It was never a goal of reST to provide this feature. So Tres' solution (removing the code) is perfectly fine.
No, the feature I'm talking about is TTW reST. Because reST has a feature that has to be turned off to be secure when processing text from untrusted users, it requires special care.
There are a lot of modules where we don't want to take over the maintainer. The important thing is that we have clever ppl who understand the code and can deal with such problems in such a case.
We need a better chain of responsibility than that, especially when there is a known security thread.
See above...it's not a question of general responsibility...it's a question of taking over the responsibility for a particular problem in particular situation...of course maintainers for modules are highly welcome...things are as they are in the Zope 2 world...
I don't agree. Our current approach isn't working. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 9. Juli 2006 08:51:12 -0400 Jim Fulton <jim@zope.com> wrote:
We need a better chain of responsibility than that, especially when there is a known security thread.
See above...it's not a question of general responsibility...it's a question of taking over the responsibility for a particular problem in particular situation...of course maintainers for modules are highly welcome...things are as they are in the Zope 2 world...
I don't agree. Our current approach isn't working.
I think we are all open for new ideas. But I doubt that we will find a better approach that would work. But as always I can be convinced of the opposite. -aj
Just to make the matters clear, when you say 'the last hotfix' Jim, do you mean the Hotfix-20060705? I ask because I'm about to roll a hotfix installer for Plone and if there's an issue with that one I can hold back the installer. -- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
On 7/8/06, Sidnei da Silva <sidnei@enfoldsystems.com> wrote:
Just to make the matters clear, when you say 'the last hotfix' Jim, do you mean the Hotfix-20060705?
I ask because I'm about to roll a hotfix installer for Plone and if there's an issue with that one I can hold back the installer.
It looks to me like the only issue with it is the lack of tests. The inadequate hotfix appears to be one from last fall which attempted to address the same issue. Alec
On Jul 8, 2006, at 12:05 PM, Alec Mitchell wrote:
On 7/8/06, Sidnei da Silva <sidnei@enfoldsystems.com> wrote:
Just to make the matters clear, when you say 'the last hotfix' Jim, do you mean the Hotfix-20060705?
I ask because I'm about to roll a hotfix installer for Plone and if there's an issue with that one I can hold back the installer.
It looks to me like the only issue with it is the lack of tests. The inadequate hotfix appears to be one from last fall which attempted to address the same issue.
Right Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Jul 8, 2006, at 10:53 AM, Sidnei da Silva wrote:
Just to make the matters clear, when you say 'the last hotfix' Jim, do you mean the Hotfix-20060705?
No, I was referring to the one before that. The November '0f hot fix purported to solve the same problem. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
...
You seem to be the only one championing TTW reST?
I am only champion against crude removal of features and against and a shortsighted preception.
I'm for keeping it (or something like it) too.
That doesn't deserve an answer.
Are you unwilling to write the tests necessary to keep it?
This is really not the point. As release manager I am allowed to speak up. But that does not imply I have to fix all and everything.
Yes, it really is the point. We've had a serious security failure due to a lack of adequate testing. This is not acceptable.
You mean auditing. Testing would not help imho. Testing only checks if expected behavior still works. And nobody expects the spanish inquisiton *wink* ;) Regards Tino
On Jul 8, 2006, at 11:32 AM, Tino Wildenhain wrote:
...
You seem to be the only one championing TTW reST?
I am only champion against crude removal of features and against and a shortsighted preception.
I'm for keeping it (or something like it) too.
Are you volunteering to do a decent job of maintaining it?
That doesn't deserve an answer.
Are you unwilling to write the tests necessary to keep it?
This is really not the point. As release manager I am allowed to speak up. But that does not imply I have to fix all and everything.
Yes, it really is the point. We've had a serious security failure due to a lack of adequate testing. This is not acceptable.
You mean auditing. Testing would not help imho. Testing only checks if expected behavior still works. And nobody expects the spanish inquisiton *wink* ;)
You can test that trying to do fil-inclusion fails. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 8. Juli 2006 14:37:06 -0400 Jim Fulton <jim@zope.com> wrote:
On Jul 8, 2006, at 11:32 AM, Tino Wildenhain wrote:
...
You seem to be the only one championing TTW reST?
I am only champion against crude removal of features and against and a shortsighted preception.
I'm for keeping it (or something like it) too.
Are you volunteering to do a decent job of maintaining it?
During the keep-or-don't-keep-zclasses discusssionyou said something like "not every package needs a maintainer in order keep it in the zope core". I think this applies here as well. Andreas
On Jul 8, 2006, at 2:47 PM, Andreas Jung wrote:
--On 8. Juli 2006 14:37:06 -0400 Jim Fulton <jim@zope.com> wrote:
On Jul 8, 2006, at 11:32 AM, Tino Wildenhain wrote:
...
You seem to be the only one championing TTW reST?
I am only champion against crude removal of features and against and a shortsighted preception.
I'm for keeping it (or something like it) too.
Are you volunteering to do a decent job of maintaining it?
During the keep-or-don't-keep-zclasses discusssionyou said something like "not every package needs a maintainer in order keep it in the zope core". I think this applies here as well.
1. ZClasses are not a security threat. reST is. That's a huge difference. 2. This event illustrates that I was wrong. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 8. Juli 2006 15:05:21 -0400 Jim Fulton <jim@zope.com> wrote:
I think this applies here as well.
1. ZClasses are not a security threat. reST is. That's a huge difference.
Being a security thread or not ...how will you prove that a module X is a thread or not? Without source code review every module has the potential to be a thread. I would never claim that the modules I've written or maintain in some way are totally safe...
2. This event illustrates that I was wrong.
Possibly, but a lot of modules were written by ppl that are no longer active in the community and a lot of these modules are a real cruft that nobody want to touch (and that little ppl understand). For the time being we have to live with this situation in the Zope 2 world. The only way out is to replace more and more code with Zope 3 modules which is actually happening. So what does it mean to be a maintainer of a package? A maintainer has to keep the code in shape and should of course care about security issues. But a maintainer might have a different view on security than you...so how to get out of this dilemma? Code audits? They would help but you know how much time they take (impractical for most code if you ask me). The current "unofficial" code auditing by watching the checkin lists seems to work to a certain degree (perhaps not directly related to security issues but to wrong code in general). Getting maintainers for Zope core packages is even more harder than some yrs ago when the Zope community wasn't split up as it is today (CPS, Zope3,Zope2, Plone, CMF). The common view on the Zope 2 core seems to be "it works, it's a cruft, don't touch it"..and ppl prefer to put their hands on other stuff outside the Zope 2 core. I am realistic enough to see that this won't change in the near future. Andreas
On Jul 8, 2006, at 3:27 PM, Andreas Jung wrote:
--On 8. Juli 2006 15:05:21 -0400 Jim Fulton <jim@zope.com> wrote:
I think this applies here as well.
1. ZClasses are not a security threat. reST is. That's a huge difference.
Being a security thread or not ...how will you prove that a module X is a thread or not? Without source code review every module has the potential to be a thread. I would never claim that the modules I've written or maintain in some way are totally safe...
One difference is that between our code and 3rd-party code. I wrote the ZClasses code and paid a lot of attention to security. Whoever integrated reST didn't even read the documentation, much less the code.
2. This event illustrates that I was wrong.
Possibly, but a lot of modules were written by ppl that are no longer active in the community and a lot of these modules are a real cruft that nobody want to touch (and that little ppl understand). For the time being we have to live with this situation in the Zope 2 world. The only way out is to replace more and more code with Zope 3 modules which is actually happening.
So what does it mean to be a maintainer of a package?
This is something that the Zope Foundation needs to work out. I'd like to start a discussion of this when Martijn gets back from vacation. Or perhaps we should put off the discussion till September when most people are back from vacation.
A maintainer has to keep the code in shape and should of course care about security issues. But a maintainer might have a different view on security than you...so how to get out of this dilemma? Code audits? They would help but you know how much time they take (impractical for most code if you ask me). The current "unofficial" code auditing by watching the checkin lists seems to work to a certain degree (perhaps not directly related to security issues but to wrong code in general). Getting maintainers for Zope core packages is even more harder than some yrs ago when the Zope community wasn't split up as it is today (CPS, Zope3,Zope2, Plone, CMF). The common view on the Zope 2 core seems to be "it works, it's a cruft, don't touch it"..and ppl prefer to put their hands on other stuff outside the Zope 2 core. I am realistic enough to see that this won't change in the near future.
My view is that both Zope 2 and Zope 3 are too big. IMO, they need to be split into smaller projects packaged more or less separately. reST and ZClasses should be add-ons, not a part of the code. It should be possible for each project/package to tell if the project is active. Then it's up to users to decide whether to take the risk of using an unsupported package. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
...
You mean auditing. Testing would not help imho. Testing only checks if expected behavior still works. And nobody expects the spanish inquisiton *wink* ;)
You can test that trying to do fil-inclusion fails.
For example if I'd were the one who would have written the naive test - I would not have known a file inclusion feature even exists or is supposed to be exposed to reST. So my test would not have tested it. So we had perfectly tests for all the reST things we want and expect but the hole would exist anyway. To cut a long story short, I guess the current fix can work or there can be other holes (which we constantly would not be aware no matter how many tests tell us the file inclusion does not work anymore). So whats the solution? Audit of the docutils package? Putting it into restricted environment like the other template engines? Inclusion of own docutils like, but audited code? Regards Tino
On Jul 8, 2006, at 5:38 PM, Tino Wildenhain wrote:
Jim Fulton wrote:
...
You mean auditing. Testing would not help imho. Testing only checks if expected behavior still works. And nobody expects the spanish inquisiton *wink* ;)
You can test that trying to do fil-inclusion fails.
For example if I'd were the one who would have written the naive test - I would not have known a file inclusion feature even exists or is supposed to be exposed to reST. So my test would not have tested it. So we had perfectly tests for all the reST things we want and expect but the hole would exist anyway.
I agree that testing is not enough if you don't know what to test for. It's sad that whoever enabled this didn't bother to read the docutils documentation which documents the feature and even provides warning about it's security issues: http://docutils.sourceforge.net/docs/ref/rst/ directives.html#including-an-external-document-fragment Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
On Jul 8, 2006, at 10:09 AM, Andreas Jung wrote:
--On 8. Juli 2006 09:53:47 -0400 Jim Fulton <jim@zope.com> wrote:
...
Tres came up with this sledge hammer because he has no confidence in people's willingness to test and implement this feature properly.
I am fine with the sledge-hammer. I've never claimed that we need to support file insertion and raw support in any way. We don't need, we can kick it. But removing or disabling a feature because we are possibly incompetent would be just ridiculous.
I can live with the sledge hammer for Zope 2. All I ask for is tests.
If there are tests for each way of invoking reST through the web that verifies that file-inclusion isn't enabled, then it's alright with me if the sledge hammer is used to make the tests pass. I won't tolerate an untested feature with so much security risk.
Yes, someone has to write the tests at some time, soon.
Right. Before 2.10.
As I pointed out the risk is minimal for Zope-apps because you need to have access to the ZMI..
No, it's not. Getting at arbitrary files is not acceptable from the ZMI.
Agreed. Much of Zope's security machinery would be irrelevant if we didn't care about untrusted users entering more-or-less executable content TTW.
so what are security concerns in this case? And file inclusion won't work if the related code is stripped off...so what are your security concerns in this case?
I am concerned by the lack of tests. Whoever created the last hot fix was sure the problem was fixed. They were wrong and we're paying the price.
I'll note that tests wouldn't have helped here in the absence of a more careful security review of docutils: none of us was aware of the 'raw' directive as an attack vector for file inclusion until you mentioned it the other day. We *did* disable the vector we knew about (the 'include' directive, when processed from a ZMI-based ReST Document). I think we can be off the hook for the Plone version, as I think they don't call the same function to render the text; the DTML-based version, OTOH, was our fault (I didn't know 'fmt="restructured-text"' existed until this week). Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEsAot+gerLs4ltQ4RAuiGAKCfqNcNx2g9Ffw1879ornZVWLmpHACfUZXv 6c3PGtRAwtXdY7xFgmGE76U= =7tjp -----END PGP SIGNATURE-----
On Jul 8, 2006, at 3:40 PM, Tres Seaver wrote: ...
I'll note that tests wouldn't have helped here in the absence of a more careful security review of docutils: none of us was aware of the 'raw' directive as an attack vector for file inclusion until you mentioned it the other day.
Except that, as you discovered, it was *not* an attack vector. setting file_insertion_enabled to False disables file insertion via the raw directive too. The real problem was that you could still use the include directive to include files via DTML and Plone. We didn't have a test to demonstrate that you couldn't use file insertion from DTML. And, obviously, the author of the Plone feature didn't have tests either. I agree that tests are not enough. The person who brought this issue up at EuroPython had a good point that whenever we use 3rd-party code, we need to consider it's security implications. We didn't even read the documentation for reST when we incorporated this feature. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
On Jul 8, 2006, at 3:40 PM, Tres Seaver wrote: ...
I'll note that tests wouldn't have helped here in the absence of a more careful security review of docutils: none of us was aware of the 'raw' directive as an attack vector for file inclusion until you mentioned it the other day.
Except that, as you discovered, it was *not* an attack vector. setting file_insertion_enabled to False disables file insertion via the raw directive too. The real problem was that you could still use the include directive to include files via DTML and Plone. We didn't have a test to demonstrate that you couldn't use file insertion from DTML. And, obviously, the author of the Plone feature didn't have tests either.
I agree that tests are not enough. The person who brought this issue up at EuroPython had a good point that whenever we use 3rd-party code, we need to consider it's security implications. We didn't even read the documentation for reST when we incorporated this feature.
I think we picked up the feature (file inclusion) unnoticed in an upgrade (but could be wrong). Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEsQf/+gerLs4ltQ4RAnXuAJ0QCeVnsG2XDzUFnYP9ffxr4Ab1ZwCgtvJ+ H4/5PeonI01DXMoy9+DskK0= =m94+ -----END PGP SIGNATURE-----
On Jul 9, 2006, at 9:43 AM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jim Fulton wrote:
On Jul 8, 2006, at 3:40 PM, Tres Seaver wrote: ...
I'll note that tests wouldn't have helped here in the absence of a more careful security review of docutils: none of us was aware of the 'raw' directive as an attack vector for file inclusion until you mentioned it the other day.
Except that, as you discovered, it was *not* an attack vector. setting file_insertion_enabled to False disables file insertion via the raw directive too. The real problem was that you could still use the include directive to include files via DTML and Plone. We didn't have a test to demonstrate that you couldn't use file insertion from DTML. And, obviously, the author of the Plone feature didn't have tests either.
I agree that tests are not enough. The person who brought this issue up at EuroPython had a good point that whenever we use 3rd-party code, we need to consider it's security implications. We didn't even read the documentation for reST when we incorporated this feature.
I think we picked up the feature (file inclusion) unnoticed in an upgrade (but could be wrong).
I dunno. If this is so, why didn't the person who incorporated the new version check for new features that might be harmful? That doesn't change the fact that when we found out about the threat last fall, we didn't check all of the places in Zope where we were using reST. You might say that this was because the person who did the hot fix didn't know about all of the places we were using reST. But that just illustrates that our current approach of "everyone is responsible for everything" or, cynically, "no one is responsible for anything" isn't working. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 9. Juli 2006 10:10:53 -0400 Jim Fulton <jim@zope.com> wrote:
That doesn't change the fact that when we found out about the threat last fall, we didn't check all of the places in Zope where we were using reST. You might say that this was because the person who did the hot fix didn't know about all of the places we were using reST.
As far as I can remember at least Tres and I were involved in this issue. I think Tres was working on the hotfix and I was working on the releases...something like that. So we both were possibly blind...
But that just illustrates that our current approach of "everyone is responsible for everything" or, cynically, "no one is responsible for anything" isn't working.
Isn't that the approach how Zope is working since years? It is a working process - not a perfect process. Look how often major vendors like Microsoft, Oracle or Apple deliver patches for their patches...we're neither better nor worse. That's not a excuse for mistakes (which *will* happen as long as humans are involved) but better look how far we got with Zope so far given the fact that a big part of the Zope core is just a cruft. Responsibility for a particular code part requires a solid understanding of the code. There are a bunch of modules where I assume that only a small number of people understands the code (who understand ZClasses except you and Dieter?). Responsibility for a particular code part requires dedication. You may find a maintainer for module X or Y but I doubt that some will show dedication e.g. to ZClasses....which is a perfect example...Some month ago we had again the discussion about ZClasses and their future and one person spoke up to do something (take over the code or reimplement them).....lots of noise...nothing else... in my experience most contributors are of course dedicated in the first place to their own code but very little to some cruft code that dates back to the DC and early ZC times. So my conclusion: dedication and taking over responsibility won't solve the general problem especially when it comes to security. As a maintainer you're usually blind or have a narrowed perception on things (which might depend on the personal skills and experiences)...not everyone of the contributors is a mastermind as you...that's just the situation..so only outstanding persons can help in such a situation (e.g. through regular reviews). -aj
On Jul 9, 2006, at 10:47 AM, Andreas Jung wrote: ...
But that just illustrates that our current approach of "everyone is responsible for everything" or, cynically, "no one is responsible for anything" isn't working.
Isn't that the approach how Zope is working since years?
Yes, but Zope is much bigger than it was years ago. We are maintaining 2 separate Zopes and both continue to grow.
It is a working process
I beg to differ. 1. This security problem is a case in point. 2. Our inability to get releases out on time is another.
Responsibility for a particular code part requires a solid understanding of the code.
Right. That's why responsibility for everything doesn't work.
There are a bunch of modules where I assume that only a small number of people understands the code (who understand ZClasses except you and Dieter?).
I don't know. I am certainly willing to work on them if necessary as I did for 2.8.
Responsibility for a particular code part requires dedication.
Yes it does. I think users of a feature deserve to know that someone has that dedication.
You may find a maintainer for module X or Y but I doubt that some will show dedication e.g. to ZClasses....which is a perfect example...Some month ago we had again the discussion about ZClasses and their future and one person spoke up to do something (take over the code or reimplement them).....lots of noise...nothing else... in my experience most contributors are of course dedicated in the first place to their own code but very little to some cruft code that dates back to the DC and early ZC times.
Sure. I think we should take a hard look at some of these things in light of these discussions.
So my conclusion: dedication and taking over responsibility won't solve the general problem especially when it comes to security. As a maintainer you're usually blind or have a narrowed perception on things (which might depend on the personal skills and experiences)...not everyone of the contributors is a mastermind as you...that's just the situation..so only outstanding persons can help in such a situation (e.g. through regular reviews).
I don't understand this argument. On the one hand you seem to be saying that people aren't that dedicated so we should not use a process that requires dedication. I can't except that. Maybe we need to do much less, so that we have the necessary commitment to the things we do do. Of course that won't guarantee that there won't be problems, but, hopefully, it will: - Increase the likelihood that basic processes will be followed, such as actually reading the documentation (much less auditing the code) of 3rd-party libraries and including tests of critical features (or missfeatures). - Make sure we have someone (preferably more than one person) we can call on when things go wrong. - provide a basis for evaluation of whether or not to use a feature. If ZClasses were an add-on product with documentation of how lightly maintained it was, it would allow people to better judge the risk of using them. On the other hand, you say that many people who are contributing are not as smart as me. That's true, just as many contributors are smarter than me. It doesn't really matter. We have some capacity for management of software and we've exceeded it. We have to figure out what we *can* maintain with the budget of brain cells we have and be brutal about only maintaining what we can. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
On Jul 8, 2006, at 9:17 AM, Andreas Jung wrote:
On Jul 8, 2006, at 8:12 AM, Andreas Jung wrote:
--On 8. Juli 2006 07:45:01 -0400 Jim Fulton <jim@zope.com> wrote:
Only if there is no other option. Tres' patch seems to resolve this issue and with further testing there is no need to remove the functionality.
"Seems" isn't good enough. It's not even close. The hot fix last fall "seemed" to fix the problem. :(
That's is still not an argument. I'll agree with you when we are all convinced that we are all unable to fix this issue if a reasonable effort or when come to conclusion that Docutils is a problem by itself...sorry, but we are not at that point so far.
Otherwise it has to go.
No :-)
Wrong. Sorry, I'll invoke Pope if I have to.
Sorry Jim, that's weak. See above. I'll accept the decision of the Pope as long as it is comprehensible...so far it is not.
Maybe you aren't listening.
Tres came up with this sledge hammer because he has no confidence in people's willingness to test and implement this feature properly.
I am fine with the sledge-hammer. I've never claimed that we need to support file insertion and raw support in any way. We don't need, we can kick it. But removing or disabling a feature because we are possibly incompetent would be just ridiculous.
I can live with the sledge hammer for Zope 2. All I ask for is tests.
If there are tests for each way of invoking reST through the web that verifies that file-inclusion isn't enabled, then it's alright with me if the sledge hammer is used to make the tests pass. I won't tolerate an untested feature with so much security risk.
I'll also note that the sledgehammer might not itself be safe in the presense of the various reload products for Zope 3. Would Tres' patch be defeated by reloading docutils.parsers.rst.directives.misc? Is there a chance that a reload product could reload this module and undo the fix? I dunno. It is worrisome.
The monkeypatch in the hotfix *might* be defeated that way, sure. The updated version of docutils I checked in will *not*, because it disables file inclusion inside the source of the dangerous handlers. Another possible fix would be to patch docutils to make the configuration directive for file inclusion disabled by default; that would allow a trusted module to enable them for a given parse, without exposing the feature for untrusted code.
You seem to be the only one championing TTW reST? Are you unwilling to write the tests necessary to keep it? If so, it's hard to have any sympathy for your desire to keep it.
There are way too many uses of TTW documents out there "live" to just rip it out, I think. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEsAjS+gerLs4ltQ4RAvoaAJ0Tsv3mfKB9vnJ0ugH4lQtrqBxFnQCfWMpQ qrxYmHZNAItTXxJoUx1Kwfc= =DRLx -----END PGP SIGNATURE-----
On Jul 8, 2006, at 3:34 PM, Tres Seaver wrote: ...
The monkeypatch in the hotfix *might* be defeated that way, sure. The updated version of docutils I checked in will *not*, because it disables file inclusion inside the source of the dangerous handlers.
Another possible fix would be to patch docutils to make the configuration directive for file inclusion disabled by default; that would allow a trusted module to enable them for a given parse, without exposing the feature for untrusted code.
I like this. I would feel better, if we choose to maintain a hacked docutils, to rename it so that it remains possible for an add-on to use a non- hacked version. Also, if we maintain a hacked version, of course, we are taking extra responsibility on ourselves.
You seem to be the only one championing TTW reST? Are you unwilling to write the tests necessary to keep it? If so, it's hard to have any sympathy for your desire to keep it.
There are way too many uses of TTW documents out there "live" to just rip it out, I think.
Unless we have much better maintenance of this feature than we've had in the past, then we'll have no choice. Hopefully this will change. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Tres Seaver wrote:
Another possible fix would be to patch docutils to make the configuration directive for file inclusion disabled by default; that would allow a trusted module to enable them for a given parse, without exposing the feature for untrusted code.
Which should be how upstream docutils should be coded in the first place. That file inclusion is allowed by default is beyond me, when the experience of many other systems like TeX or PostScript show that it's a huge security hole. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
Andreas Jung wrote at 2006-7-8 14:12 +0200:
... removing TTW reST ...
[Andreas]
In addition I don't see a big problem for Zope-only(!) apps.
Of course, you must also consider applications built on top of Zope -- such as "ZWiki" and "Plone". They, too, need to be protected. [Jim] ... retain only when someone takes responsibility ...
Otherwise it has to go.
[Andreas]
No :-)
This, time I am on your side, Andreas :-) I agree with you that a feature ("file/url" inclusion code) physically removed from the shipped code can be considered no longer causing security risks -- even without extensive tests. I also agree with you, that this is preferable to dropping "reST" altogether (despite the fact, that I personnally do not use it). Of course, an artistically set Python path could cause Python's docutils to be used -- but hey, that's then a locally caused problem. -- Dieter
On Jul 8, 2006, at 3:51 PM, dieter@handshake.de wrote: ...
This, time I am on your side, Andreas :-)
I agree with you that a feature ("file/url" inclusion code) physically removed from the shipped code can be considered no longer causing security risks -- even without extensive tests.
Your recent expression of distain for testing causes me to be unsurprised by this position. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote at 2006-7-9 09:10 -0400:
... On Jul 8, 2006, at 3:51 PM, dieter@handshake.de wrote:
... I agree with you that a feature ("file/url" inclusion code) physically removed from the shipped code can be considered no longer causing security risks -- even without extensive tests.
Your recent expression of distain for testing causes me to be unsurprised by this position.
I do not feel distain for testing altogether -- just for testing posing a burden without significantly increasing quality... -- Dieter
--On 8. Juli 2006 07:45:01 -0400 Jim Fulton <jim@zope.com> wrote:
On Jul 8, 2006, at 1:11 AM, Andreas Jung wrote:
--On 7. Juli 2006 11:03:06 -0400 Jim Fulton <jim@zope.com> wrote:
I think we should do a 2.9.4 release to incorporate the recent hot fix. This is easy for me to say, since I won't be doing it. :)
Because this recent fix actually fixed the same problem that the previous hot fix was supposed to fix, I think someone needs to work up some decent tests. This is not a trivial task, bit it is necessary. If no one is willing to do this, I think we need to drop the TTW reStructuredText support from Zope 2, as it is too great a risk.
Dropping TTW reST is absolutely not an option. I breaks backward compatibility.
Sorry, security trumps backward compatibility.
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
Tres' patch is looking in fine to me. I don't see a need right now for dropping reST with having file inclusing *removed*.
Has anyone written tests for Tres' patch? Apparently no one wrote adequate tests for the last hot fix, which helped put us in this situation.
I've written some tests (checked in on the trunk). They test the 'raw' and 'include' directives @Tres: what is the reason to keep the 'raw' code in docutils? I am in favor to remove it and replace it with a NotImplementedError exception (same as for the the 'include' code). The related tests (for reStructredText and ZReST are commented for now) do except a NotImplementedError for a 'raw' directive. Andreas
According to Andreas Jung:
Tres' patch is looking in fine to me. I don't see a need right now for dropping reST with having file inclusing *removed*.
Has anyone written tests for Tres' patch? Apparently no one wrote adequate tests for the last hot fix, which helped put us in this situation.
I've written some tests (checked in on the trunk). They test the 'raw' and 'include' directives
Thank you, Andreas! We make extensive use of reST (via ZWiki) and it would be very hard for us to do without reST.
@Tres: what is the reason to keep the 'raw' code in docutils? I am in favor to remove it and replace it with a NotImplementedError exception (same as for the the 'include' code). The related tests (for reStructredText and ZReST are commented for now) do except a NotImplementedError for a 'raw' directive.
In ZWiki reST pages you can use the 'raw' directive to call e.g. python scripts (useful for custom index generation, ...). If it goes away due to security reasons, so be it. But if there is a way to keep the 'raw' functionality and remove only 'file' and 'include', we are certanly in favour of that... \wlang{} -- Willi.Langenberger@wu-wien.ac.at Fax: +43/1/31336/9207 Zentrum fuer Informatikdienste, Wirtschaftsuniversitaet Wien, Austria
--On 9. Juli 2006 12:29:24 +0200 Willi Langenberger <wlang@wu-wien.ac.at> wrote:
@Tres: what is the reason to keep the 'raw' code in docutils? I am in favor to remove it and replace it with a NotImplementedError exception (same as for the the 'include' code). The related tests (for reStructredText and ZReST are commented for now) do except a NotImplementedError for a 'raw' directive.
In ZWiki reST pages you can use the 'raw' directive to call e.g. python scripts (useful for custom index generation, ...). If it goes away due to security reasons, so be it. But if there is a way to keep the 'raw' functionality and remove only 'file' and 'include', we are certanly in favour of that...
You mean 'file' and 'url'...they are now disabled. -aj
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote:
--On 8. Juli 2006 07:45:01 -0400 Jim Fulton <jim@zope.com> wrote:
On Jul 8, 2006, at 1:11 AM, Andreas Jung wrote:
--On 7. Juli 2006 11:03:06 -0400 Jim Fulton <jim@zope.com> wrote:
I think we should do a 2.9.4 release to incorporate the recent hot fix. This is easy for me to say, since I won't be doing it. :)
Because this recent fix actually fixed the same problem that the previous hot fix was supposed to fix, I think someone needs to work up some decent tests. This is not a trivial task, bit it is necessary. If no one is willing to do this, I think we need to drop the TTW reStructuredText support from Zope 2, as it is too great a risk.
Dropping TTW reST is absolutely not an option. I breaks backward compatibility.
Sorry, security trumps backward compatibility.
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it.
Tres' patch is looking in fine to me. I don't see a need right now for dropping reST with having file inclusing *removed*.
Has anyone written tests for Tres' patch? Apparently no one wrote adequate tests for the last hot fix, which helped put us in this situation.
I've written some tests (checked in on the trunk). They test the 'raw' and 'include' directives
Great! Maybe we can add a similar set for the 'fmt="restructured-text"' in DTML.
@Tres: what is the reason to keep the 'raw' code in docutils? I am in favor to remove it and replace it with a NotImplementedError exception (same as for the the 'include' code). The related tests (for reStructredText and ZReST are commented for now) do except a NotImplementedError for a 'raw' directive.
'raw' can be used to disable ReST processing for a block, even if the ':file:' or ':url:' options aren't set.; I was trying to leave the possibility of that use case. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEsVdq+gerLs4ltQ4RAr5lAKCD7gYS/162mz8YPfUm7NshA2lJmwCgtXOe nVX21AXxsYmMfNEgmtAaPmk= =tWJZ -----END PGP SIGNATURE-----
--On 9. Juli 2006 15:22:18 -0400 Tres Seaver <tseaver@palladion.com> wrote:
I've written some tests (checked in on the trunk). They test the 'raw' and 'include' directives
Great! Maybe we can add a similar set for the 'fmt="restructured-text"' in DTML.
Jup, but I won't the able to this over the next days (being on the road). Porting the tests would be trivial.
@Tres: what is the reason to keep the 'raw' code in docutils? I am in favor to remove it and replace it with a NotImplementedError exception (same as for the the 'include' code). The related tests (for reStructredText and ZReST are commented for now) do except a NotImplementedError for a 'raw' directive.
'raw' can be used to disable ReST processing for a block, even if the ':file:' or ':url:' options aren't set.; I was trying to leave the possibility of that use case.
I figured that out meanwhile. There are now tests for 'url' and 'file'. I also noticed tonight that there is another directive 'csv-table' that also provides an 'url' flag:-) We might check that as well however I could not see a risk right now. Andreas
Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Andreas Jung wrote:
--On 8. Juli 2006 07:45:01 -0400 Jim Fulton <jim@zope.com> wrote:
On Jul 8, 2006, at 1:11 AM, Andreas Jung wrote:
--On 7. Juli 2006 11:03:06 -0400 Jim Fulton <jim@zope.com> wrote:
I think we should do a 2.9.4 release to incorporate the recent hot fix. This is easy for me to say, since I won't be doing it. :)
Because this recent fix actually fixed the same problem that the previous hot fix was supposed to fix, I think someone needs to work up some decent tests. This is not a trivial task, bit it is necessary. If no one is willing to do this, I think we need to drop the TTW reStructuredText support from Zope 2, as it is too great a risk.
Dropping TTW reST is absolutely not an option. I breaks backward compatibility.
Sorry, security trumps backward compatibility.
BTW, I suspect that a less violent patch could be created, if anyone wants to champion TTW reStructuedText support in Zope 2. Personally, I'm for dropping it. Tres' patch is looking in fine to me. I don't see a need right now for dropping reST with having file inclusing *removed*. Has anyone written tests for Tres' patch? Apparently no one wrote adequate tests for the last hot fix, which helped put us in this situation. I've written some tests (checked in on the trunk). They test the 'raw' and 'include' directives
Great! Maybe we can add a similar set for the 'fmt="restructured-text"' in DTML.
@Tres: what is the reason to keep the 'raw' code in docutils? I am in favor to remove it and replace it with a NotImplementedError exception (same as for the the 'include' code). The related tests (for reStructredText and ZReST are commented for now) do except a NotImplementedError for a 'raw' directive.
'raw' can be used to disable ReST processing for a block, even if the ':file:' or ':url:' options aren't set.; I was trying to leave the possibility of that use case.
Isn't that leaving the door open for XSS (cross-site scripting) attacks? Allowing arbitrary raw HTML to be input allows javascript in the pages, and therefore cookie stealing. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
participants (15)
-
Alec Mitchell -
Andreas Jung -
Chris Withers -
Dieter Maurer -
dieter＠handshake.de
-
Florent Guillaume -
Jim Fulton -
Michael Haubenwallner -
Philipp von Weitershausen -
Sidnei da Silva -
Simon Michael -
Stefan H. Holek -
Tino Wildenhain -
Tres Seaver -
Willi Langenberger