[Grok-dev] a nice research project

[Chris McDonough]

Martijn Faassen wrote:
> Hey guys,
> 
> This is going to be long as it contains thoughts I've been having for a 
> while that I probably should have been sharing more explicitly.
> 
> My first response to any comparison where Grok comes out badly is to 
> find excuses for Grok. The main excuse that will always work is that 
> this is just *a* benchmark and if you just adjust what you look at then 
> Grok will do better. And of course if someone advocating another 
> framework does the benchmark, it's likely that they'll pick a benchmark 
> which makes their framework comes out best. I'd do the same. :)
> 
> (...man had I trouble finding one when cElementTree came out while lxml 
> was just newborn!)
> 
> It also turned out that when Chris adjusted the benchmark suddenly a lot 
> of the weird ZPT and XML-related stuff went away and Grok is doing a bit 
> better. That's good - I'm still confused about where that came from 
> initially, but at least the page template engine wasn't operating during 
> almost all of the test (except possibly the first request which is now, 
> I take it, unmeasured).

Yes, apologies for that.  I don't know what's getting measured there where
minidom and friends come in to play.  It's probably worth looking into, though
(you'd use repoze.profile with discard_first_request=false).

> My second response to any comparison where Grok comes out badly is to 
> look at Grok itself and search for ways to make Grok better. We're 
> certainly not at the end of improvement here. Let's acknowledge Grok can 
> be improved here. Let's learn.

Yay!  I think improving stuff along this particular vector will be *great* for Zope.

> I got a head start in this, as I've been looking at the repoze.bfg 
> project for a while (from a distance). So Chris's blog wasn't 
> particularly news to me though it does put things starkly (he got to 
> pick the benchmark). There are, in my mind, two main things Grok can be 
> envious about compared to repoze.bfg:
> 
> * much simpler dependency structure (therefore bfg is easier to 
> understand and less intimidating to install)
> 
> * a much simpler and more comprehensible publication process.
> 
> The dependency structure bit has been long known as an issue in the Zope 
> framework, since it got split up into little pieces. Seeing repoze.bfg 
> only highlighted this. So we had a sprint recently to help clean up 
> dependency structure, which I also see as a prerequisite for improving 
> the libraries themselves. We've made good progress on it, though tons of 
> work remains to be done; feel free to pitch in.
> 
> Improving the publication process in the Zope framework and Grok is a 
> next step. Grok (and the Zope framework) are indeed doing a lot to get a 
> page published. In part this is because the Zope framework tries to be 
> flexible and pluggable in the publication process in ways that other 
> frameworks care less about. Some of this flexibility is useful, and much 
> of it is quite possibly less useful. To make things worse, the Zope 
> framework publication procedure *is* scattered around the codebase, 
> which makes it rather difficult to understand and modify.

One thing became clear to me when I was trying to understand the Z3 publisher
(before I decided to do BFG).  The concept of plugins-to-a-plugin is very
difficult to understand.  For example, PAU in Z3 tries to make it possible to
replace the simpler authentication service.  So it's a plugin to Z3.  But then
there are plugins to this plugin (in the form of challengers, authenticators,
and so on).  And they tend to all get configured in one big ZCML pile; it's not
immediately clear what is the plugger and what is the pluggee, and what slot
fills which hole.

It would be bad enough if it was a *static* situation, but people come in behind
the initial implementation, and *add more code* to the thing to make it more
extensible, often completely misunderstanding the purpose of the original code.
 So it's not impossible that there gets to be plugins-to-plugins-to-plugins, and
it just gets very difficult to understand.

I have a feeling the some of the same issue exists in the Zope3 publisher.
IIRC, it provides a way to override all traversal (by replacing the
publication).   But *also* makes it possible to override parts of that default
traverser (e.g. the publication does a component lookup to find a named adapter
for the object and request during publishTraverse).

I don't really think anyone understands that the fact that you can replace the
publication and make it all behave differently (and make it much faster).  But
IMO, only the former (overriding all traversal) should be possible, and the
default publication should be dead simple.  If you want a different traversal
scheme, you can just write a different publication.  Even if you have to cut and
paste the default one over to your code and change it slightly, at least it's
only *you* that's paying the runtime cost.  This is the strategy we've tried to
use in bfg, and it seems to be paying off.

> I'm not the only one interested in this stuff:
> 
> http://shane.willowrise.com/archives/repozublisher/
> 
> Grok is not a new web framework, like repoze.bfg. It builds on the Zope 
> framework, and tries to stay compatible with it. We believe the Zope 
> framework has a lot to offer. It's mature and the codebase of individual 
> components is typically high quality, and there are a lot of extensions.
> 
> This year, it's time we (Grok developers) go back to the underlying Zope 
> framework to improve its comprehensibility. We're a major consumer of 
> the Zope framework after all, so we have that responsibility (and 
> *need*). That means a better dependency structure, less code shipped 
> that isn't actually used much and that we can do better at anyway (the 
> ZMI), and improvements to the publisher system, among other things.
> 
> I understand all emotions in this discussion. Our task is to take the 
> emotions and channel them to productive activities. We learn and 
> improve. Let's go for it.

Thanks for the constructiveness!

- C