Movies, audiences, wasted effort, was Re: [Zope3-dev] The vision thing

[Jeff Shell]

On 3/6/06, Paul Winkler <pw_lists at slinkp.com> wrote:
> On Sun, Mar 05, 2006 at 04:04:48PM -0500, Benji York wrote:
> > Geoff Davis wrote:
> > >* What can we learn from Rails / Django / TurboGears?
> >
> > Fun presentation along those lines:
> >       http://oodt.jpl.nasa.gov/better-web-app.mov
> >
> > One of the best put together movies I've seen.
>
> Really interesting stuff.
> I was struck by the way he presents zope/plone.  He gives it a winning
> "fun factor" score for "hello world"... by doing through-the-web
> development.  So he says "Zope [2] rocks" precisely because of features
> that many of us have been largely ignoring and advising against for a
> couple years now.  The kind of thing that the built-in zope 2 tutorial
> shows you how to do.
>
> Why is Zope 2 TTW fun?
>
>  - No restarts
>
>  - No configuration
>
> That's certainly food for thought.

It definitely is. Why is Zope 2 TTW difficult? Ignoring the file
system aspects, one of the big stickers for us at Bottlerocket has
been:

If using Zope 2 TTW to get something done quickly, we need to use
something else for storage. Designing persistent objects AND
'templates and scripts' in the same database in the Zope 2 style never
quite meshed in my brain. The CMF did a good job of separating them
out, but then brought a larger overhead in terms of concepts to figure
out and grasp, and it didn't help the developer in a hurry. A finished
application like Plone does... But then you're working in Plone's UI
and Plone's universe and if that works for you, great. If not, there's
a lot of work.

Starting new projects would be *anguishing* here at Bottlerocket, as
we'd often lose two or three days just thinking of "what is the best
way, what is the fastest way, let's try that real quick, this thing is
too slow, this is too cumbersome, this doesn't look like what we
promised them", and so on.

> A similar app could've been written pretty quickly in Zope 3 by writing
> a schema and using browser:addform, browser:editform, and
> browser:schemadisplay.  It would be interesting to see how that would go
> in the movie.  I suspect that the movie author (named Sean Kelly i
> think?) would've complained about the xml "sit-ups" and the numerous
> server restarts.

Those are bad options anyways. They do not have growth potential
either, as you then have to make the conceptual leap from "something
magically generated by this XML declaration" into "how do I customize
what happens on edit?"

zope.formlib's forms do better. Although I (personally) find IAdding a
scary thing to work with and roll some of my own add-support
utilities.

I've been thinking about how to start up an application with very
little. I think that you could actually do quite a bit with Zope 3 and
introspection. If you had a folder that implemented 'IPersonContainer'
and it had a containment restraint on 'IPassenger, ICrewMember,
IPassengerGroup', you could (theoretically) have something (a ZCML
directive, a Python function, a base class) that could dynamically
generate a contents table, an add list, edit forms, all from that one
'IPersonContainer' declaration.

>From what I understand of Rails, you can have dynamic scaffolding
pretty easily, and watch your application change as you change the
underlying model (the RDBMS in most cases). But one of the real
strengths, I think, is the generator tool.

The generator tool is pluggable - other generators can be written and
installed. That alone is cool. As you install other plug-ins, they can
add generators to help you start using them. Or other generators can
be made that might stamp out an example web log application.

But the really cool thing about it is that it creates really easy to
use and understand start points. I can see the 'create' and 'edit'
actions it gives me.

  def update
    @article = Article.find(params[:id])
    if @article.update_attributes(params[:article])
      flash[:notice] = 'Article was successfully updated.'
      redirect_to :action => 'show', :id => @article
    else
      render :action => 'edit'
    end
  end

I added a field (through a plug-in) that was too complex to be saved
by the basic 'update_attributes' action, so I had to extract it myself
and call a different method. I don't know if this is the best way to
do it, but this is what I got working this weekend:

  def update
    @article = Article.find(params[:id])
    tags = params[:article][:tag_list]
    @article.tag_with(tags) if tags

    other = params[:article].reject{|k,v| k == 'tag_list'}
    if @article.update_attributes(other)
      flash[:notice] = 'Article was successfully updated.'
      redirect_to :action => 'show', :id => @article
    else
      render :action => 'edit'
    end
  end

This is a really nice balance. Instead of just having something like
<browser:editform> taking care of everything for me, or even have code
generated that subclasses from formlib.EditForm but doesn't actually
show the 'handleEdit' action taking care of things, I have something
right here that I can play with and edit.

>  (In the middle of the movie he gives a link to his
> article at http://www.developer.com/xml/article.php/3583081 which
> includes the line "The fad of applications using XML for their
> configuration files is dismaying, to say the least...")
>
> This is the kind of guy for whom TTW development really is compelling.
> Watching the "hello world" section reminded me of when I used to really
> enjoy doing that stuff.
>
> But there are some important things left out of that story.
> Why did I stop enjoying TTW work? Like a lot of Zope 2 developers:
>
> * I needed my work to live in CVS.  ZODB history and undo isn't good enough
>   as version control.

Yep. I prefer to use the ZODB these days as a content database. I also
like that local utilities can store extra application information in
the ZODB (catalog indexes, user accounts) in a section that's away
from the content area.

At Bottlerocket, convincing my co-worker to use the ZODB for any kind
of application data in Zope 2 was nearly impossible, no matter what
the development model was. It just *felt* more out of control. Too
many things potentially being inherited or acquired that might add or
interfere. Catalog was very hard to work with, especially without
events, and we'd often mess up and forget something critical. But I
hated using an RDBMS for a lot of our applications and having to go
through that extra layer.

None of this was the fault of the ZODB itself. It was the fault of
Zope 2 / TTW being so enmeshed and confusing.

In Zope 3, it's so easy to develop for the ZODB and feel in control. I
look at everyone else with their quaint little O-R mappers and example
applications with two tables with 3-4 columns each and wonder "you
want me to install MySQL for that?" Not to mention wiring up todo into
a page notification message is cool: "Deleted Article (Undo Delete?)"

> * I needed a sane way to deploy software from dev to staging and from
>   staging to production. ZSyncer is fine for what it does, but it too is
>   a poor substitute for version control, shell scripts, make, et al.

Zope 3 could still stand to have import/export of raw ZODB points...
But that's another issue.

> * The unix shell still beats the pants off the ZMI as a complete
>   working environment (even with ExternalEditor, the find tab, etc.)
>
> * I got tired of bouncing between restricted and unrestricted code.
>   I want to live in unrestricted code as much as possible.

Yep. I think one of the reasons my code in Zope 3 feels less
claustrophobic is that it's easier to refactor, often to helper
functions, utility modules, and so on. It's easier to move code
around. It's easy, most of all, to just import something and use it. I
still don't understand how the module security stuff works in Zope 2,
although I've managed to make some useful utility modules from time to
time.

> Ultimately the zope 2 restart time started to become less of a problem
> than dealing with all those problems when working TTW.
>
> For CMF development, I settled on a pretty nice compromise: templates
> and scripts in filesystem directory views, with the scripts doing
> only view-related glue.  This got me files on the filesystem and in CVS,
> and no restarts when tweaking UI. The scripts are easily testable using
> CMFTestCase. Pretty nice way to work. I still have to deal with
> some restricted code, but I'm mostly resigned to that.
>
> In Zope 3, we take restarts and filesystem-only development for granted,
> because it's intended specifically for the audience that I'm a member
> of...  developers who have those concerns.
>
> I'm hoping to see a similarly interactive, yet long-term-sane,
> working style evolve for in zope 3.  Maybe we'll get there
> with Persisent Modules and fssync.

I really like the concept of through the web tweaking and
manipulation. But I'm sick of templates and scripts. Between viewlets
and formlib, I've been able to start getting out of the big template /
big request/response loop cycle of thinking - aided by my own dynamic
HTML generation library based very loosely on Nevow's Stan.

As I've poked around at Seaside (a Smalltalk web tool), I've come away
with all kinds of dreams. One of the cool things with Seaside is that
in develop mode, you can turn on 'halos' in the web browser. This puts
boxes around all of the *objects* in a page - because the page is
actually constructed of objects and not of strings inserted into a
bigger string. You can inspect the shopping cart in the corner. You
can inspect the item listing. You can tweak the CSS for any of those
objects right there. Or you can go directly to the Smalltalk code
behind the object in a simple web version of the Smalltalk system
browser, with full access to the full environment. It's all live. And
it is all...totally...cool.  But this is a style that Smalltalk's
nature supports. It's not so natural for Python. Still, it's something
that I think about. (Plus I love its Smalltalk based way of HTML
generation. I hate Page Templates for most things now, and pretty much
have my newer applications boiled down to two or three with content
providers/viewlets responsible for their own HTML, with CSS taking
care of most of the major look and feel).

I'd like to give certain customers the ability to see their
e-commmerce stores from the front page, turn on something like halos,
and be able to go in and directly edit how an individual item gets
displayed. No full templates - just turn the promotional item around
and see a simple template language (perhaps similar to Django's) where
displayed values are obvious. Or to turn it around and change some
other values, right there. But ultimately, the big thing is to treat
and think of many of the little things on a page as the objects that
they are - viewlets, most likely, wrapped around a PurchasableItem,
and to add in a layer that allows direct manipulation. It's not quite
like Portlets or any design like that. It's more for direct
customization of applications that aren't a bloody content management
entry tool.

The 'customize' feature of CMF Skins is actually kindof close to what
I'm thinking of. The core system look and feel would all come from
disk based products, still, with the ability to overridden by
something stored in the ZODB or some other tool. But instead of
monkeying with scripts and full templates, it would be with much
smaller pieces that make up a page.

Recent experiences trying to tweak a few small layout oriented
features in a very complex and nasty shopping cart template through
the web in Zope 2 - even with a lot of logic 'coded behind' the
template - have brought this idea closer to mind.

> If there's a moral to this story, it's this:
> Scaffolding that gets you up and running with a minimum of
> fuss is a great thing.  Rapid interactive and iterative development
> is also a great thing.  But if you can't easily transition from there to
> more complex apps that are still maintainable, it sucks. It's irritating
> to have to throw away some of your knowledge and completely replace it
> with new ways of thinking; it's better if the new knowledge strictly
> supplements the old.  It's worse than irritating to have to throw away
> your work and rebuild it from scratch; it's better if your new work can
> cleanly leverage the old.

This is an area where Rails is particularly strong. I'm normally not a
fan of code generation. But their tool generates just-enough. It's
code you can actually understand and start building from, and a quick
run to the api docs they have online is usually all that's needed to
start understanding the code you're looking at. The code their tool
generates runs basically what you see if you have it dynamically
providing 'scaffolding', so the conceptual difference between the
automatically generated and what it gives you out of the box is pretty
small.

I think that zope.formlib (and maybe zc.table, from what I've looked
at) are great tools for things like this. formlib is pure Python. It's
got big documentation. It's got a nice API that you can follow. There
are base classes and common functions that do a LOT of the 'heavy
lifting', but they're understandable... Mostly. Much more so than
trying to follow a ZCML directive.

This is all still for a developer focused crowd, which is still my
main interest. I'll provide the cool things for my customers. The more
cool and usable things go into the core that help me more than they
coddle me, the better application I can create. I can shape 'formlib'
into just about anything... well, anything supporting forms :). I like
that. I love it. I can't stop praising it enough.

> Put another way, if we consider Jim's first two audiences, how do we
> teach a single person to move from "i don't want to have to care" to
> zope zen master / SVN contributor with minimal wasted effort along the
> way?
>
> Today I don't know if there's a clear coherent story to be told there,
> even for zope 2. If there was... wow, that would be a great.
>
> Sorry if I haven't really said anything new.
>
> --
>
> Paul Winkler
> http://www.slinkp.com
> _______________________________________________
> Zope3-dev mailing list
> Zope3-dev at zope.org
> Unsub: http://mail.zope.org/mailman/options/zope3-dev/eucci.group%40gmail.com
>
>


--
Jeff Shell