On Wed, 10 Nov 1999, Christopher Petrilli wrote:

...

to methods (and the whole quoted and unquoted thing kicks in, and suddenly they have to understand about the _ namespace, and the magic variables to pass a DTML method to make it work).

well, this is beyond the above outlined problem domain. So we have a seperate problem, not a DTML problem, but a lack of a solution problem.

Let me add here that in many cases passing arguments to DTML Methods can be avoided if acquisition and factoring are properly employed. Seems to me that newcomers to Zope thing in terms of defining functions and calling them with parameters and thus hitting the namespace issues. Maybe better explanations (with clear examples) are needed to expose acquisition even more. My thoughts ... Pavlos

Evan Gibson wrote:

On Wed, Nov 10, 1999 at 07:43:44PM -0500, Christopher Petrilli wrote:

...

On 11/10/99 7:11 PM, Stuart 'Zen' Bishop at zen@cs.rmit.edu.au wrote:

...

I'm with you 100%. It makes me feel like I'm programming a reverse polish calculator sometimes :-)

It is sub-optimal, but it is not a programming language, so please don't try and hold it to those standards. DTML is a reporting language, and trying to make it do things it isn't designed to is simply going to further complicate the matter. Things like 'dtml-let' are dangerous in this reguard... they make you think you actually have a language worth doing anything beyond reporting in.

We've been talking about this around work and have pretty much agreed that DTML lets you do too much... It _lets_ you use it as a programming language, when what we should really be doing is restricting dtml to reporting stuff, make Python methods an intrinsic part of Zope and _require_ people to use python to do the really nasty stuff.

Ah, I said almost exactly the same in another post I just sent. :) Using Python or something else non-DTML should be obvious and natural and easy, whenever you reach the limits of DTML. And it should be easy to know that you have reached those limits.

DTML is _perfect_ as a reporting language and should always provide the front end to objects, but the objects themselves should be either database queries or python code.

Agreed.

Initially we did things in DTML because it was easier than writing external methods, but if you can write python methods inside Zope and cut out a lot of the dangerous namespace games in DTML (like _, dtml-let, REQUEST.set) then you _force_ the divorce between functionality and display, get DTML looking elegant again [snip]

Yes.. Though it probably takes a bit of thought figuring out how to make this natural and easy. Hm. Throwing out the _ namespace for instance might induce people to do something else that's horrible instead, if care isn't taken. [snip discussion on how to adapt DTML, and other discussions about ZSQL methods] Regards, Martijn

At 01:13 PM 11/11/99 +1100, Evan Gibson wrote:

I _shouldn't_ be doing things like this in DTML. It shouldn't even let me. The whole _.getattr(blah) sections are far beyond the reality of a reporting language, and they don't go properly with an Object Oriented paradigm either. I should instead have an object made and be going:

<dtml-with "currentuser.get_skin()">

But the existing framework doesn't encourage this. If you banned _ access in anything but python methods and made it so python methods can only be used in Products, then you'd have the proper divide.

I've got to say I disagree on a fundamental level with the whole, "DTML shouldn't let you do stuff" idea. Having to have two different languages to program one application is really pretty ridiculous to me. If it made sense, then why *aren't* we all "doing the right thing" and making objects and using DTML only to format HTML? Well, for one thing, it breaks the flow of development. It's like saying you should use Python only for scripting and anything that's "real programming" you should immediately drop down to C or C++ and implement something you can call from Python. Doesn't that seem rather silly? But there's a second, and far more significant reason why trying to make DTML "just a reporting language" doesn't work in reality. It's this: *generating HTML is a programming task*. The reason you need programming capability in DTML is because you need to program the bloody HTML, URL's, query strings, and all the other overhead of making an interactive application. If you start pushing the hard bits into Python, then sure, you don't "pollute your presentation with programming", but then, you start polluting your programming with presentation, which is equally bad, if not worse. Should people use good style? Sure. Should you make products that implement all the data/business rules stuff and provide a clean interface? Absolutely! But doing this won't keep the need for programming out of DTML, because there will always be the need to perform transformations and compositions of arbitrary kinds on those data objects, precisely *because* they encapsulate data and ignore presentation! That's the dirty little secret of why none of us can kick the programming-in-DTML habit.

So it should be the Python of reporting languages. :)

Ah!

...

Things like 'dtml-let' are dangerous in this reguard... they make ou think you actually have a language worth doing anything beyond reporting in.

Yes, but the problem is people *will* currently apparently use DTML beyond its design specifications. It's a difficult situation; if you don't give people dtml-let, they'll use some other feature (dtml-with and namespace and whatnot) to do the same. This is even *more* complicated.

DTML is currently moving in several incompatible directions at once. Under pressure of its users DTML moves to a more full-fledged programming language. At the same time, features are added that are rather ad-hoc to make specific reporting tasks more easy. The interaction between these two pressures causes DTML to become unnecessarily difficult for both programming *and* reporting tasks. Note that many programming and reporting tasks *are* very easy in DTML; it's just that too many aren't.

I'm not sure what one could do to let DTML become a conceptually pure reporting language. Redesign it in some unspecified fashion. Offer an alternative that is as easy to use as DTML; people currently use DTML as it's quick and easy. PythonMethods are perhaps the solution there.

We need to ponder more about what makes anything a conceptually pure reporting language. This topic must've been studied before, perhaps there's material on the net...

[sequence-item vs sequence_item]

...

...

How about <dtml-python>

Well, if it were my decision, I'd say "over my dead body," but it's not... HOWEVER, I'll fight this personally because I think this makes it more PHPish, and looses the point of the isolation of presentation and logic. DTML is for presentation, not logic. It's simply been moved too much into the logic side because of a lack of something suitable for that application.

Agreed.

...

We are working on this, including talking with Evan Simpson abou bringing PythonMethods in line with what we need to include it out of the box. I would vigorously recommend working on THAT ,rather than trying to make DTML something it isn't designed to be.

Agreed. We should move DTML *more* into the direction of its original target.

...

Always measure something by the scope of its problem domain, not by what people try and use it for. The latter indicates a separate problem enirely.

Agreed.

...

...

Is existing DTML good enough for 'simple' scripting? I think the current confusion threshold occurs when people start needing to pass parameters to methods (and the whole quoted and unquoted thing kicks in, and suddenly they have to understand about the _ namespace, and the magic variables to pass a DTML method to make it work).

well, this is beyond the above outlined problem domain. So we have a seperate problem, not a DTML problem, but a lack of a solution problem.

Yes, but. I still have (admittedly completely vague) ideas about how DTML could be a more pure reporting language. A DTML where ugly constructs simply aren't *possible* (utopic, but at least we could move it in that direction). Instead, in this ideal DTML it'll be *natural* and *easy* to use a PythonMethod (or whatever), whenever you notice (and this should be easy) that you're going beyond DTML's abilities.

...

I sound a bit harsh, but I'm trying to clarify the problem domain that DTML solves (just as we're trying to be more explicit about the domain ZClasses are designed to solve). It's dangerous to take it outside its problem domain, as it collapses quickly into a morass of tangled code.

Yes, but this is already happening. And worse, this causes DTML to miss its original reporting target as well, in some cases; witness my dtml-with discussion that prompted me to start the rant. That wasn't a deep programming problem, it was a simple reporting issue, and the DTML was just too confusing.

Yes. I recall it took me _days_ to discover how to make an object with an id provided from a request, then change to the object and change things in it. THis is a simple operation, and it is ubiquitous, but it does involve such esoteric constructs as <dtml-var "_.getitem(REQUEST['id']).manage_changeProperties(REQUEST)"> or worse: <dtml-var "_.getitem(_['sequence- item']).manage_changeProperties(title=REQUEST['title']"> Or something alike, because I always forget what it is. Reading all the thread and the different remarks, I'm tempted to make Paul's compilation but I won't, because I do not have time. There is a few important conclusions to be made, however: 1. DTML may be meant to be a reporting language, but it's able to do many more things and used for many more things 2. There are Python alternatives, but these are not obvious, nor is it clear when to use what 3. Thus, there is no integration between DTML and Python in the framework. By this, I mean that it should be easy to do simple things in DTML. Then (as it goes), find out you have to do more complicated things, and then be able to switch to a Python alternative (be it Python Methods, ZClasses, External Methods, although these are presented in the documentation a sort of 'hack glue' ). I'm sure this can be done by now, or be accomplished with little effort. IMHO, the advantage would be that it would bring the modularity Zope excels in in its design to programming Zope. While it will not be possible to enforce this by technological means (as many have remarked), there must be a solution in design practices. After all, there in a product so well designed as Zope, there are already al lot of assumptions of what is the best place/way of doing. The most easy solution to this would be something like a 'best practices in Zope programming' document. In it, it we could point out a number of 'rules of thumb' to Zope users/programmers/designers of 'what to use when and where in programming Zope, and how to integrate your efforts to a coherent whole. Then people will still get tangled up in their dtml, but at least we can point them out that 'the preferred way of doing this is Zope is not in DTML, but in Python, along such and such lines' (I'll be happy to contribute, but I'm afraid this is too much/complicated for me alone) What do you all think. Is this a direction for (at least a partial) solution to a very complicated problem? Rik

From: Andrew M. Kuchling <akuchlin@mems-exchange.org>

This leads on to a security-related, I think. DC staff have said in the past that Python code isn't stored in the database for security reasons. PythonMethods change this; someone who gets your manager password can create a Python method that makes a setuid copy of your shell or whatever. (Disclaimer: I haven't actually gotten around to trying out PythonMethods yet.)

The latest release of the PythonMethod Product has two kinds of PythonMethod: regular and XXX. A switch in the source code allows you to turn "XXXPythonMethod"s on and off, and they are off by default. XXXPMs are wild, unsafe, and unrestricted, with (lack of) security equivalent to External Methods. The only limit on them (under *nix, anyway) is that they're likely to be running as user "nobody". Regular PMs, on the other hand, are meant to be completely safe, and subject to all the same restrictions as DTML, although this goal has not yet been acheived. Any holes in regular PM security, however, only give improper access to Zope objects, *not* the system at large.

Question: Is there a way we can reconcile these two conflicting drives? If some solution can be found, then maybe PythonMethods could be added to the products that come with basic Zope. Suggestions?

The DC folks have started talking with me about this. I'm optimistic.

1) Perhaps PythonMethods could be enabled or disabled when you install Zope; if people are going to be using Zope over insecure links, and shouldn't be using PythonMethods, don't install them..

Substitute "XXXPythonMethods" for "PythonMethods", and I agree. Already done, too :-)

2) Perhaps they could use the rexec module or Zope's existing sandbox to run their code (but would the sandbox limit their usefulness? -- maybe not, if you take the attitude that serious Python code should still be in a product or ExternalMethod).

XXXPythonMethods should never be used on a system with untrusted clients, or which is accessed via an unsecure link. People will anyway, of course, but they've been warned. Regular PMs with proxy roles and External Methods should be the norm. Cheers, Evan @ 4-am

On 11/11/99 5:35 PM, Stuart 'Zen' Bishop at zen@cs.rmit.edu.au wrote:

Magic sequence- variables need to have aliases of sequence_ (everyone rabidly agrees on this) Strangely enough no one has owned up to actually implementing the '-' variables, most likely as they are afraid of being lynched.

I looked at this today, it's not going into 2.1 most likely, it's a LOT of work to make sure it works, *and* doesn't have any negative performance impact. Since it will by its nature, I'll need to balance that with a performance enhancement somewhere :-)

Program code should not be embedded in the Reporting language.

Amen.

DTML sucks when used beyond its intended scope as a Reporting language. The ability to program in DTML should be discouraged or possibly depricated.

Discouraged, but that's all we can really do in reality.

DTML is constantly being used beyond its intended scope, as there is no way to program Zope without resorting to External methods or Python Products with their various caveats. In particular, there is no way of running program code in a sandbox without using DTML which means all Zope programmers need to be given effective full control over the Zope installation.

This is why we're a bit skeptical of new tags which encourage this...

PythonMethods is available now and could fill the void if it is integrated with the Zope distribution. Work will need to be done proving that Python Methods opens no security concerns not already valid with DTML.

This is being done, I think... it *will* introduce new security concerns, but we hope to quantify and mitigate them wherever possible. More power always comes with more danger. Chris -- | Christopher Petrilli Python Powered Digital Creations, Inc. | petrilli@digicool.com http://www.digicool.com

On Fri, 12 Nov 1999, Stuart 'Zen' Bishop wrote:

Is this any good as a temporary fix?

Grr... just found an obvious flaw. Lets try take 2 at embarassing myself. *** DT_InSV.py.org Fri Nov 12 15:26:50 1999 --- DT_InSV.py Fri Nov 12 16:29:56 1999 *************** *** 88,94 **** $Id: DT_InSV.py,v 1.16 1999/11/02 16:51:32 brian Exp $''' __version__='$Revision: 1.16 $'[11:-2] ! from string import lower, rfind, split, join from math import sqrt TupleType=type(()) try: --- 88,95 ---- $Id: DT_InSV.py,v 1.16 1999/11/02 16:51:32 brian Exp $''' __version__='$Revision: 1.16 $'[11:-2] ! from UserDict import UserDict ! from string import lower, rfind, split, join, replace from math import sqrt TupleType=type(()) try: *************** *** 96,101 **** --- 97,127 ---- mv=Missing.Value except: mv=None + def _transkey(key): + if key[:13] == 'sequence-var-': + # Only translate the first 2 '-' characters in a sequence-var- + return replace(key,'-','_',2) + else: + return replace(key,'-','_') + + class sequence_kwdict(UserDict): + '''This class is used to implement the dictionary in sequence_variables + It duplicates keys containing '-' characters to a corresponding entry + with a key mapping all the first two '-' characters to '_' characters. + This will greatly improve DTML readability.''' + + def __init__(self,data): + self.data = data + for i in data.items(): + data[_transkey(i[0])] = i[1] + + def __setitem__(self,key,value): + self.data[key] = value + self.data[_transkey(key)] = value + + def __delitem__(self,key,value): + del self.data[key] + del self.data[_transkey(key)] class sequence_variables: *************** *** 105,116 **** self.query_string=query_string self.start_name_re=start_name_re ! self.data=data={ 'previous-sequence': 0, 'next-sequence': 0, 'sequence-start': 1, 'sequence-end': 0, ! } def number(self,index): return index+1 --- 131,142 ---- self.query_string=query_string self.start_name_re=start_name_re ! self.data=data=sequence_kwdict({ 'previous-sequence': 0, 'next-sequence': 0, 'sequence-start': 1, 'sequence-end': 0, ! }) def number(self,index): return index+1 ___ // Zen (alias Stuart Bishop) Work: zen@cs.rmit.edu.au // E N Senior Systems Alchemist Play: zen@shangri-la.dropbear.id.au //__ Computer Science, RMIT WWW: http://www.cs.rmit.edu.au/~zen

Stuart 'Zen' Bishop wrote:

Is this using r_exec and Bastion from Python? Or do you have to roll your own?

Since PythonMethods live and work entirely within Zope, they aren't allowed to import anything, and get the same restricted set of builtins as DTML, so r_exec and Bastion are irrelevant. All that remains is to prevent damage or improper access to objects passed into a PM. That's where zbytecodehacks come in. The compiled bytecode is scanned for forbidden operations and munged to call secured versions of other operations.

My summary of the thread so far would be along the lines of:

[snip]

Until program code can be implemented using the web interface and run securly in a sandbox, DTML will continue to be abused.

Even this won't be really satisfactory. What everyone (myself included) seems to want is the ability to use DTML or similar tags in their simplest form -- no expressions, more declarative than procedural -- and have complex logic live separately, written in nice normal Python. Unfortunately, even a moderately complex case will make you either calculate *everything* up front, and pass it to the DTML for rendering, or else fragment the logic into a swarm of little mini-methods. I do all of my web work with my wife, who's a designer. She uses Dreamweaver and other nice Whizzywig tools to produce pretty, friendly pages, and then I install the engine. I've been able to come up with code and procedures which make the DW-Zope partnership tolerable, but it still isn't nice *at all*. As long as we're careful, DW doesn't mess up DTML tags or get confused, but we're still using a visual tool to edit what is essentially a *program* rather than a *template*. My ideal, as near as I can express it, is something like this: <Z name="standard_html_header"> <html><head><title>The Title</title></head> <body> <p>This whole section is a placeholder for visual editing. It gets replaced with standard_html_header.</p> </Z> <table> <tr Z-in="item_list"> <td>Item #&Z-number;</td> <td>Qty: &Z-qty;</td> <td>Price: &Z-price;</td> <td>Total: &Z-total;</td> </tr> </table> <form Z=relative_form method=POST> <input type=text name=bob Z=bob> <input type=checkbox name=uncle_p Z=uncle_p> <input type=submit name=submit value="Submit! Surrender!"> </form> <Z name="standard_html_footer"> </body></html></Z> Give or take the legality of the syntax and the reaction of editors other than Dreamweaver to the "bonus" attributes and entities, what we have here is a perfectly well-formed, intelligible HTML page. The source isn't too hard to read, and a browser (well, NS4.61 for Linux, anyway) renders it nicely, acting as a sort of mock-up of the dynamic page-to-be. Now imagine that there's a block of code joined to this page, something like this: Z.item_list = SQL_get_items() for item in Z.item_list: item.set('total', item.qty * item.price) Z.relative_form.set('action', '/here/there/everywhere') Z.bob.set('value', 'Bob') if uncle_default: Z.uncle_q.set('CHECKED') if not choose_name: Z.bob = 'Bob' # Replace the entire <input> tag with this fixed string. Here, 'Z' is a magic object like '_' in DTML. It holds the values which will replace the &Z-entities; and allows attributes of Z-named tags to be set or cleared. Tags with Z-in attributes are repeated using the named value, much like <dtml-in>. Well, I'm done rambling now. This is all ill-defined (how are code and page joined?), hard to code, and quite possibly a bad idea, but I had to share it. Thanks, Evan @ 4-am

Just thought I'd throw another permutation into the mix. I always thought it would be nice to have a python subroutines page associated with a DTML page. It would be limited in capability like PythonMethods and would be scoped to the page. Since this would greatly simplify the addition of small Python subroutines, it would ease the demand on DTML to perform so much logic processing and allow it to focus on presentation. Comments? David Jacobs

Stuart 'Zen' Bishop quoted:

On Fri, 12 Nov 1999, David Jacobs wrote:

...

I always thought it would be nice to have a python subroutines page associated with a DTML page. It would be limited in capability like PythonMethods and would be scoped to the page. Since this would greatly simplify the addition of small Python subroutines, it would ease the demand on DTML to perform so much logic processing and allow it to focus on presentation.

I did something similar for the American Greetings site; you had a .pd page containing the HTML and a .py file containing Python code. The .py file would be executed to set up a bunch of variables, and then their values would be substituted into the .pd. For example, to show the current time, you would do: page.py: from time import * now = asctime(localtime(time() )) page.pd: The time is [now]. The templating system grew in an ugly way; first it did only variable substitution, then I needed conditionals and hacked them in, then I need to loop over lists and hacked that in, etc. -- very bad code, in the end. Another problem is that I still sometimes embedded HTML in my Python code; one .pd page couldn't easily call another in the way that DTML methods can. If doing it today, I'd use DTML, because DTML's better structure, and the ability to easily call DTML submethods, would make this much neater. -- A.M. Kuchling http://starship.python.net/crew/amk/ Thanks. The sooner I get discouraged and quit, the more time I'll save overall. -- Frank Sergeant, 28 Mar 1999