[Zope-dev] ngDTML (aka "PSP" or ZScript?) proposal

[Phillip J. Eby]

I have a proposal for a template language which (hopefully) won't be too
difficult to create, building as it does on existing works.  My proposal is
specifically addressed towards making a scripting language that is amenable
to learning, doing, and performance, at the expense of being itself valid
HTML or XML, and sometimes at the expense of brevity of expression.

The syntax I propose we steal from ASP+Python, with one minor alteration.
I already have code (written almost 3 years ago and not interfaced to Zope)
which parses this syntax, including the necessary machinations to compile
out "end if", "end for", etc. and reformat the Python as indented, using
the end statements as block delimiters.  Such delimiters are necessary when
embedding Python in arbitrary text, both for clarity and because indenting
is a royal pain in embedded script.  Anyway, a basic page might have
something like:


<% from _ import id, somesqlquery %>

Information for <%= _(id) %>

<table>
<% for record in _(somesqlquery): %>
<tr>
 <td><%= _(record.name1) %></td>
 <td><%= _(record.name2, fmt="%05.2d") %></td>
</tr>
<% end for %>
</table>


Let's go through this and look at the interesting bits.  First, we use the
name '_' to represent the current Zope namespace.  Importing variables from
it, or using it as a dictionary always returns those objects
*uninterpreted*.  However, calling _ with an unnamed first parameter will
attempt to execute the object (if it is callable), and then take any
keyword parameters as "var"-style formatting arguments.

Let's get more specific about the execution part.  The execution algorithm is:

Push locals() onto the namespace stack, then:

1) If the object has isDocTemp==1, return the result of calling the object
with (None,namespace)

2) Otherwise, use ZPublisher's "mapply" function so that the namespace
stack can be passed through to the called object's parameters.  Thus, if
you call a Python function, SQL method, etc., its arguments can
automatically be filled in for you as a timesaver.

Note that one is under no obligation to use _() to render an object, and
that if you want to prevent calling an object you wish to render in its
"raw" form, you can simply use str() or repr() or `` on it first.  (e.g.
_(str(something),fmt="*%s*"))

More interesting bits.  The Python namespace takes precedence unless you
specifically import or otherwise retrieve an object from _.  _ itself,
REQUEST and RESPONSE, however, are already in the namespace when your
method starts, in addition to any parameters defined by the method (ala SQL
methods.)

There are some special aspects of how import should work.  Importing a
module should attempt to find an object of that name in the _ namespace,
and check the current user's permissions for that object (taking proxy
roles into account) before allowing the import.  This would allow the use
of "PythonModule" objects which you add to a Zope hierarchy and set
permissions on.  They would import modules only from the "safe" extension
and product directories.  Of course, any modules generally considered safe
could be imported without a lookup/permissons check.

Now on to the compilation and execution model.  An ngDTML script is
rewritten as straight Python code, replacing all <%= expr %> blocks with
_.write(expr) and all non-Python text as _.write('text') statements.  The
resulting code is compiled, and the important aspects of both DTML and Evan
Simpson's bytecode transforms and alteration of __builtins__ are performed,
to render the resulting code safe for execution.  The code block is set up
as a function which does a "return _" if execution falls off the end of the
script.  (Thus, a method can return something instead of writing output,
ala the DTML "return" tag.)  The _ object would be able to render itself as
a string by returning the accumulated writes, and thus when an ngDTML
object is called by ZPublisher, its output will be properly sent to the
RESPONSE.

Whew.  It's a long and tedious list of things that have to be implemented.
However, it is clearly able to be implemented, and all the pieces already
exist, although some bits may have to be re-implemented in order to be
integrated.  To sum up, the advantages of this language are:

1. It's Python, so the basic syntax is clean and simple.  Plenty of books
are out there on Python, and when it comes right down to it, we've been
continuously extending DTML trying to make it more like Python, so why not
just go there?

2. It's based on a well-established embedded scripting paradigm which is
used in numerous other application environments including the giants ASP
and PHP.  It is easier to teach to someone coming over from one of those
environments, and many editing tools can already cope with <% %> syntax.

3. All the bits of DTML which are really handy, such as auto-calling,
namespace passing, formatting utilities, etc., are all tidily packaged into
the "_" swiss-army knife, and there is no confusing Perl-ish automagic
anywhere.  (Note, by the way, that using _.string, _.None, etc. is not
valid in ngDTML; these things should be imported with import.  _ is for
generally useful ngDTML utilities only.)

What about disadvantages?  Well, there's no "in" tag and all its goodies,
although these could be conceivably implemented using an object that
wrapped a sequence for you, and maybe even fitted in as another capability
of the '_' object.  I haven't given this one a great deal of thought yet.
Also, doing things like the tree, sendmail, and sqlgroup tags are tricky
unless you can ask the _ object to push and pop its write() buffer; then
you can do something like:

<% try:
      _.push()
%>From: <%= sender %>
To: <%= string.join(recipients,',') %>
Subject: test

Hi!
<% somehost.sendmail(recipients,str(_))
finally: _.pop()
%>

Which, I have to admit is a lot messier than
<dtml-sendmail></dtml-sendmail>.  On the other hand, the above can also be
written:

<% somehost.sendmail(recipients,_(message)) %>

Since one can define functions within the body of an ngDTML block, and pass
them to utility routines, dtml-tree and other things that want blocks to
render could be passed functions.  This leaves text-transforming tags like
sqlgroup, which are harder to manage in this form.

Anyway, other than these points, I believe I have described here a language
which is capable of doing anything the DTML core can do, while being much
clearer and more consistent, and at the same time practical to implement.
Comments, anyone?