[Zope-dev] DTML and REQUEST data changes about to be checked in

[Martijn Pieters]

Hi folks,

I am about to land some big changes in the way DTML deals with data taken
from the REQUEST object when accessed implicitly, in both the Zope Trunk
and the Zope 2.5 branch. These changes could potentially break existing
Zope sites.

Without these changes, Zope is somewhat vulnerable to cross-scripting
attacks, where a well-crafted URL can cause a Zope server to serve out
arbitrary HTML. Because DTML does not automatically html quote any data,
and can implicitly get information out of the REQUEST even when it was not
the intention of the template author, it is easy to cause REQUEST data to
be rendered as HTML on a page.

My changes cause the REQUEST to keep track of suspected strings, where
suspect is defined as any string with a '<'. These are marked as tainted.
Any normal, explicit access to the REQUEST will still give you normal
values. However, as soon as a DTML template requests a variable from the
general namespace, and this variable is then satisfied from the REQUEST,
the value of this variable could potentially be a TaintedString object
instead of the original string. When rendering such a value, DTML will
automatically HTML quote it if not already done so explicitly. All DTML
string operations dealing with TaintedString objects are careful to retain
the TaintedString status.

I also fixed all exceptions raised in Zope that I could find, where
untrusted REQUEST data was used in the exception message; these exceptions
now html quote the data. I also made sure that the REQUEST calculated
variables URLx and BASEx and such were not shadowed by untrusted form
variables of the same name.

These changes can break existing sites in the following ways:

- If you relied on getting HTML-like data from the REQUEST in DTML and
  want to render this as HTML, and you got this data implicitly, this data
  will now be HTML quoted. Note that you were vulnerable to a
  cross-scripting attack here already.

  You can retrieve your information from the REQUEST directly (with
  <dtml-with REQUEST> for example), at your own risk. ;)

- HTML quoting will also take place in templates that do not otherwise
  generate HTML to be sent back to the browser, such as email forms and Z
  SQL Methods. For Z SQL Methods, <dtml-sqlvar> does not quote
  TaintedStrings and is otherwise ignorant of them. For emails, use
  explicit access to REQUEST instead.

- If you relied on being able to override URLx or BASEx variables through
  a form variable, this no longer works. Use explicit access to
  REQUEST.form instead.

- Using the string method .join (''join(items)) cannot handle
  TaintedString objects. You can use _.string.join instead.

- Passing a TaintedString value from a DTML template to other objects such
  as Python code, External Methods, Python Scripts, etc, may cause them to
  break because they did not anticipate a TaintedString object.

What doesn't break (among others):

- Accessing REQUEST data from Python code, Python scripts, or ZPT. Only
  DocumentTemplate.DT_String derivatives (DTML Document, DTML Method, etc)
  and DTMLFile objects are affected.

- If you already HTML quoted, nothing gets double quoted.

- Using the _.string module in DTML retains taints.

- Zope 2.6 unicode marshalling (var:ustring:utf8) works with
  TaintedStrings as well.

TaintedString objects try to mimic strings as best as they can, but until
we move to python 2.2 definitely and we can inherit from str directly,
certain python code will not accept TaintedString objects as substitutes.
I found that the normal string module, and the string ''.join module don't
accept TaintedString objects for example.

Also, using the string interpolation operator % will cause TaintedString
objects to be unwrapped. When TaintedString becomes a subclass of str,
more operations will unwrap them, such as unicode() and ''.join; or just
about any operation that manipulates strings through other ways than
string methods.

Because of size of the change and the impact on existing DTML code, well
release betas of Zope 2.5 and 2.6 soon to facilitate wider testing. For
those following CVS, please test the changes rigorously and let me know what
you find.

-- 
Martijn Pieters
| Software Engineer  mailto:mj@zope.com
| Zope Corporation   http://www.zope.com/
| Creators of Zope   http://www.zope.org/
---------------------------------------------