[Grok-dev] Re: An untapped area..?

[Martijn Faassen]

Jeff Shell wrote:
> On Jan 23, 2008 4:09 AM, Martijn Faassen <faassen at startifact.com> wrote:
[snip]
> There are a lot of resources and resource-library setups. There are a
> few overrides (the IWriteFile adapters for zope.app.file objects are
> incredibly stupid and broken). There are a couple of major and
> different Skin layers. It's a big system.

There's nothing in there besides possibly the overrides that wouldn't 
inherently work in Grok. We do probably need to expose more powerful 
resource handling functionality. For the time being, people can fall 
back on the existing Zope 3 options for such and use them with Grok, of 
course.

> Big systems go through a lot of refactorings. This one certainly has.
> It didn't take me long to run into a surprise when I tried to do a
> simple "Extract Class" kind of refactoring in a small Grok application
> I was trying. As someone who's had to explain why ``<dtml-var
> full_name>`` works fine while ``<dtml-var name>`` doesn't, I'm
> sympathetic to the anguish that forgotten conventions can cause. And
> again, I think this mostly happens when things get to be large scale,
> both in code size and in team size. "But I just moved this class to a
> different module and now I'm getting this funny error..." "Oh, for
> that kind of class you have to do grok.baseclass()" "Oh, Grok was
> picking up a context automatically before, but now it isn't. you have
> to explicitly set it."

Sure, true, you have to know the rules. How does this differ from having 
to memorize the myriad ZCML directives and attributes, though? I mean, 
your claim is that Grok has *more* problems than with Zope 3.

Besides, if in Zope 3 you move a class, you'll have a much larger chance 
your code will break, as you'll have to adjust the ZCML that's 
registering it.

> I've maintained enough of these big systems through the years to see
> conventions go forgotten, and have seen new batches of developers
> struggle with the surprises.

Yes, but compare and contrast with Zope 3 there. Are you really claiming 
that a new developer is going to struggle *less* with learning the 
effect of various ZCML directives? And won't forget about them? I 
certainly have struggled memorizing them, still need to look them up 
continuously, and I've been using them for years (even reimplementing 
them for Five!).

> So when I have such a big core framework, used by multiple customer
> sites/applications, that I hope to have last for years and years, I
> personally like to turn all the magic off.

Defend the use of the word 'magic' when you apply it to Grok. How is 
Grok's configuration mechanism more magic than ZCML-driven 
configuration? You need to memorize rules for both.

> That's not a knock against the design or aims of Grok. I just would
> not be comfortable using that in this situation. I have too many bad
> memories from Zope 2 (I have a lot of great memories too, but I
> remember some bare knuckle fights as the system and I'd clash when
> it's DWIM did not do what I meant).

I think that this is part of the problem. You're reacting to perceived 
flaws in Zope 2 and applying them to Grok. You cited a very obscure DTML 
rule (dtml-var full_name>. Do you think that:

a) this is comparable to Grok's rules for configuration, which typically 
are generally applicable and are about component configuration?

b) the need to memorize particular details cannot be compared with the 
complexity of learning *any* API, such as Zope 3'? One needs to know how 
to use IAbsoluteURL in Zope 3, for instance.

You're comparing a system built with the knowledge and sensibilities and 
web development experience of 1997 with a system built for a different 
purpose in 2006, with all the experience we have now, test-driven 
development, and so on. Do you really think the problems of Zope 2 apply 
to something like Grok?

> In fact, I'd like to our end-point applications using Grok at some
> point, as that's the area where I hope new hires will do most of the
> work and I'd like for the system to be a lot easier for them to learn,
> run, and maintain.

So why is a system that's easier to learn, run and maintain harder to 
maintain and refactor? :)

>>> The area that I'm unsure of is what happens when someone grows beyond
>>> what Grok makes easy. Because very few things stay small and simple
>>> for long.
>> Examples? Grok can do views, adapters, multiadapters, local and global
>> utilities, layers, skins, and just about anything Zope 3 can. Soon
>> enough, viewlets. The only thing that Grok doesn't do out of the box is
>> content-level security. Someone could add it though - I don't see it as
>> a major project.
> 
> I worry more about not what Grok can do, but where it will suddenly
> surprise you. There just comes a time when conventions feel like
> they're getting in the way more than they are helping. And the only
> thing that I find more frustrating is a system that makes me have to
> do a lot of explicit work to get something done is the system that
> makes me have to do even more explicit work to start turning its magic
> features off.

I think the word 'magic' may lead to some of the wrong assumptions. I'd 
use the word 'magic' for a set of inconsistently grown rules that are 
hard to memorize and predict, and are not generally applicable and tend 
to interfere with trying to do things that look obvious. I'd claim Grok 
has *rules*, not magic (exceptions admitted, see the -Base discussion 
below):

* registrations of components happen based on subclass.

* registration information is in directives on the classes.

* directives can be left out, in which case they will default, according 
to a set of rules.

* if you don't want the rules, add explicit directives to the class.

Do you really think that the exercise in adding explicit directives is 
more work than adding ZCML statements? I'd argue that in some ways the 
many attributes on ZCML directives are more home-grown and less 
consistent then Grok's directives.

> Part of what I'm trying to point out is that the narrative should talk
> about what's going on, and why, and how to control it. A lot of the
> magic 'DWIM' systems never do this, partially because they have no
> control story to tell. They have the air of "You follow the
> convention, and if that doesn't work out for you, too bad, you're
> writing your code the wrong way anyways and we don't want you as a
> member."

Yes, we indeed need to write down the rules. I generally describe some 
of the basic rules used in my Grok presentation, but I should capture 
them in a document. There you're entirely right, of course! I don't 
argue that. I argue against the implication that Grok is DWIM and magic. 
Grok is an alternative, more concise configuration system for Zope 3 
applications.

[snip]
>> Grok *does* allow you to set up views and adapters explicitly. It's just
>> that the explicitness is in Python code; if you don't use defaults for
>> your directives you're completely explicit. Not that I think there's
>> anything very magic about the defaults it uses.
> 
> It was counter-intuitive for me to try to make my own base View class.
> In normal Python intuition, I should be able to make a module
> 'base.py', and do a `class View(grok.View):` and that's it. Grok blew
> up. I eventually found that I needed to add `grok.baseclass()` to the
> class statement. Which wasn't a big deal. But that's not the kind of
> defaults that I expect.

Yes, I can see how this rule is surprising. It's an implication of 
registering classes by default, so that one is relatively hard to make 
disappear. It's rather fundamental to the system.

> Likewise, I do hope that the name of 'ViewBase' turning into one of
> those base classes is going away. "I have this ArmyBase view, why
> isn't it picking up my context?" "Oh, because it's name ends in Base.
> Change it to ArmyBaseView or something".

That's a good example and you just convinced me that it should go away. 
You indeed can argue it's magic that will interfere. I will add an issue 
about getting rid of this behavior.

> Those are the kinds of defaults that worry me. Just like how <dtml-var
> foo> was really a shortcut for <dtml-var name="foo"> which worked out
> awesome - until you tried to display a variable named 'name'.
> 
> A big way to help this situation is to have good reference
> documentation though, and Grok does appear to have a good start there.
> Make sure the docs have some good examples. This is, again, an area
> where many other systems fall short.

Yes, I agree. I think good reference documentation is important. 
Conceptual integrity is also important. The -Base bit in class name is 
an example of where conceptual integrity breaks down - we base behavior 
elsewhere on directives strictly. *naming* in many cases is auto-derived 
from class names, which I think is not harmful as easily and 
consistently overridden using grok.name().

Another idea I had is to expand the grok introspector so it can actually 
show the implicit directives. So if you go to a view in the introspector 
(currently also not possible yet I believe), and the view has an 
implicit grok.context(), you'll see it explicitly, but color coded (or 
something like that) to see that this is actually the result of rules 
application.

Please do continue giving specific instances of problems! Even though I 
am defensive, I believe this discussion is valuable in getting some 
issues fixed. I think the introspector feature I described above would 
for instance be tremendously useful in teaching people the rules in 
operation. I think the next grok sprint would be a good time to make a 
large push forward on the introspector.

Regards,

Martijn