[Grok-dev] grok futures: the return of the ZMI

[Martijn Faassen]

Hi there,

I'm going to be posting a series of longer mails discussing various
possible future developments for Grok on this list.

I'm going to try something new there: after I post them on the list and
we've had some discussion (hopefully!), I'm going to post them on my
blog as well, to get more public exposure of what ideas we're having and
what we're working on. There may be some editing that results from the
discussion. Of course if there seems to be no interest in the plans, or
a lot of controversy, or if it's pretty hopeless to get volunteers on
it, I shouldn't post them, perhaps. :)

Let's start off with a topic that has been on my mind for a while. Back
in may 2007, I had a discussion with Jim Fulton in real life in Potsdam.
I argued that while the Zope ZMI was successful in its time, it might
make less sense these days to bring it back. The points I made for this
argument were:

* people are comfortable enough developing on the filesystem

* patterns of web developments are much better established these days,
so a very easy UI may be overkill.

Jim argued strenuously that he disagreed and that there was a place for
the ZMI. Things got heated then, and didn't get very far. It did make me
think about this topic some more. Some more recent discussions on the
Zope mailing list, and some playing with five.customerize, have given me
some ideas on where we could be going. Here are my thoughts on how we
could bring back the ZMI in Grok, and in particular its development
features.

The return of the ZMI
---------------------

The Zope Management Interface (ZMI) is one of Zope 2's greatest
features. In Zope's early days, this was Zope's main killer app. It
allows even inexperienced developers to quickly create web sites and
applications using a simple folderish interface, using nothing but their
web browser. Developers can choose from a palette of reusable
components to expand the capabilities of their application: templates,
scripts, user management, relational database access, forms and much 
more. Each component offers a web user interface; people can learn about 
new features simply by exploring the user interface. Development 
proceeds quickly, without the need for server restarts. The barrier to 
entry is really low.

As the Zope community learned, ZMI-based development also has severe
drawbacks. Code is stored in the object database, instead of on the
filesystem. This makes it hard to access using common development tools
such as text editors, search tools (grep) and version control systems.
These tools all expect the filesystem.

For small applications maintained by one or two people this isn't too
much of a problem. For larger applications that need to be maintained
by multiple people over a longer time it starts to be a major
pain. Various workarounds have been created to allow the use of
filesystem development tools with ZMI-based applications, but
fundamentally if you want to use filesystem development tools, you
should be storing your code on the filesystem.

The patterns of filesystem based development are quite different than
ZMI-based development, so a transition between the two isn't easy: it's
hard to convert code, and switching from one to the other means there
are a lot of new things to learn.

So later on in the history of Zope people more and more started to use
the filesystem for development. Zope 3 development and Grok development
is completely on the filesystem. So is most of Zope 2-based development
by now.

At the same time, using a web user interface for quick experiments,
prototypes and simple adjustments is very attractive, even to 
experienced developers. Since components can offer user interfaces, it 
also can be a great help to beginners.

I think we need to start to explore bringing back the best of the ZMI
into Grok, while avoiding the drawbacks. In fact, long ago we started
down this path with the Zope 3 project already. Its initial approach was
still much focused on enabling ZMI-based development. We had ideas
about allowing a simpler form of development through the web, while 
offering advanced tools to synchronize between the filesystem and the 
web so as not to get locked in.

This early development is why Zope 3 today ships with a ZMI built-in. A
ZMI that doesn't allow much in the way of useful programming. A ZMI that
doesn't quite know what it wants to be. Its most useful purpose is as an
introspection and minimal tweaking tool for existing Zope 3
applications. It's not ideally suitable for that purpose either though.

The Zope 3 approach was very ambitious, and the developers who were
developing it didn't really need it - they were comfortable enough
with development on the filesystem and weren't really looking for
easy-to-use quick-prototyping tools. The Zope 3 ZMI fell by the wayside.

The Grok project has already worked on bringing back the introspection
part of the ZMI. Grok's UI allows you to introspect Grok applications,
see what classes implement which content objects, and what interfaces
these classes provide. It's already very useful. I hope we can expand on
it quite a bit still over the course of this year. But even though this
is a great feature, it's not what I want to discuss here.

The Grok project as of yet hasn't worked on bringing back development
aspects of the ZMI. If we are careful not to get too ambitious and go
in small steps, I believe we can bring back some of these aspects
without too much work. We should be careful to scratch our own itches: 
we need to make something that makes the life of experienced developers 
easier, while at the same time appeal to beginners.

The aspect I'd like to focus in a first step is simple customization 
tasks, such as small tweaks to template or some CSS.

The underlying Zope 3 component architecture is actually powerful
enough to allow all sorts of advanced application customization
already. The technology in fact already allows page template
customization.  Interestingly enough, the system that makes most use
of this technology is Zope 2. Plone 3 uses it, for
instance. The ``five.customerize`` Zope 2 extension allows people to
customize page templates of existing applications using a web
interface.

I'd like the Grok project to investigate whether we cannot bring 
something like this technology into Grok as well. This should allow page 
template customization without restarts with Grok. That sounds cool!

I think the Zope community has the experience to avoid the drawbacks of 
through the web development. We need to think carefully about the user 
interface. We also need to provide for the transition from 
through-the-web customizations to the filesystem: we need to make it 
very easy for developers to transform through-the-web customizations 
into a grok-based Python package that contains the same customizations.

Once we have worked out a reasonable story for customizations, we can 
turn to the next bit: through the web model, schema and form 
development. Allowing through the web development of both views and 
models should bring us to something like the feature-set of ZClasses 
without the reek of evil that they gained in Zope-land.

This starts to look like a CRUD development tool, as we discussed 
before. I like the word 'bread' for a project like this: "browse, read, 
edit, add, delete". ME GROK LIKE BREAD, BREAD YUMMY

Let's think about the possibilities and experiment for a bit. I believe 
Grok is sitting on a goldmine of features and potential that we've 
barely started to tap into.

I'm not sure what shape all of this will take. The filesystem is
perfectly suitable for many developers, and I certainly don't propose
we go away from this approach. But I do believe we can reach a whole
new group of people if we go beyond this, and it's a group that Zope
has traditionally catered to. To try this, and at the same time make 
experienced developers happier, seems to be within Grok's mandate.

Please discuss!

Regards,

Martijn