[Grok-dev] Re: Grok component reuse in the Zope 3 world (was Re: Re: ANN: megrok.quarry)

[Martijn Faassen]

Hey,

Gary Poster wrote:
[reusing grok-based code in Zope 3 projects]

> This reply crystalized a concern I have been feeling.
> 
> The grok approach is throwing out zcml, at the expense of...zcml.

Indeed. Not at the expense of configuration, just ZCML. :)

> There actually was a reason why we wanted something like zcml--something 
> *not* in the Python file.  The reason was that we wanted to make it 
> possible to reuse Python without being forced to reuse configuration.  
> We wanted components.

It is in fact pretty infrequent, at least in my experience, to see 
packages reused without their ZCML being reused. Packages are typically 
reused together with their registrations. I'm sure it happens on 
occasion that a package's configure.zcml is replaced with other ZCML 
external to it, but in these relatively rare cases I see other patterns 
to fulfill the underlying use cases with Grok.

> Whatever its failures--perhaps in spelling?--zcml appears to have 
> accomplished this.

In my opinion, it's mostly not ZCML that accomplished this but:

* the component architecture

* explicit interfaces

* focus on test-driven development

> We, the Zope 3 community, are growing little 
> microcosms of code reuse.  ZC, Lovely, z3c (Roger and Stephan, 
> largely?), Gocept, Canonical/SchoolTool, Martijn in the "hurry" 
> namespace, and others have grown their own worlds of software that are 
> being mixed and matched and used as components by a variety of projects.

The ability to do this is probably the strongest point of Zope 3.

> I argue that a significant contributor to this flexible ecosystem is the 
> fact that the vast majority of this code uses ZCML--stuff outside of the 
> Python files--for configuration and glue.

I argue ZCML is a small factor, below.

> I haven't looked at megrok.quarry, but I expect it is interesting, like 
> other projects in the grok world.  It also sounds like it is, in part, 
> general--like it has code that other projects might want.  Why is it 
> grok code?  Perhaps megrok.quarry is a bad example, but I read about 
> catalog conveniences, view conveniences, and other interesting things on 
> this list.

Some of these conveniences are tied up strongly with grok, others could 
in theory be reused externally to Grok. Splitting up Grok will help in 
making the latter more easy. Anyway, there are indeed real cases of 
packages that use Grok and can be generically reusable in Zope 3 
packages. I have one or two but I haven't thrown them in the public pool 
yet as Grok isn't ready. This demonstrates that your concern is real - 
but I at least am very much aware of it. :)

> Grok's approach combats reuse outside of your community when it is used 
> for anything other than an application.   Grok feels like it is on the 
> way to building another code vacuum--like Plone is, or at least used to 
> be: code goes in from other projects, but little comes out.

I don't believe Grok's approach is combating code reuse, but we do have 
a little bit of work to be done to avoid some issues that came up.

As I described, we're actively looking into not just making Zope 3 
compatible with Grok but Grok-based code compatible with Zope 3. 
Initially grok-based code was entirely compatible with Zope 3, but as we 
added things to Grok like, especially, disabling security, this has 
changed. This is a problem that needs fixing, and one we didn't 
originally anticipate. The approach I sketched out is a response to 
exactly this problem as it came up when I wanted to port a package - 
Grok code is not reusable in Zope 3 system. It's *almost* the case, but 
currently not entirely. This will change.

> What Martijn describes in this reply sounds like it might be in the 
> right direction.  But IMO and AFAICT it is still insufficient.

Why do you consider it insufficient? In a package that uses grok, I can 
declare an adapter. Same with ZCML. What's the difference?

Is it because a grok-based component wouldn't include enough security 
declarations? (because in grok you don't need to). That's the only thing 
I can think of. It's a hard one to solve, as writing code with a lot of 
permissions sprinkled around is a major pain Grok tries to get rid of.

> I have two ideas on how to combat this.
> 
> First, be vigilant about recognizing components that could be shared.  
> Grok is about applications, not components.  Make separate packages for 
> the bits that we all (the Zope 3 community outside of Grok) can use.  
> Make Grok packages that import from the general ones and add the 
> Python-based glue in the grok-specific packages.  Plone appears to be 
> doing this now.  Looks good to me.  (I hope they do more!)

It's you who says Grok is about applications and not components, not me. 
:) Grok should also be about components, because otherwise there is no 
easy way to spin off components from applications. This is a very 
important mechanism of component generation. In my opinion it would 
defeat the purpose of Grok if people would have to go back to ZCML as 
soon as they want to spin off a component. Spinning off a component 
should be moving code into another package, and the addition of ZCML 
shouldn't be required.

Grok is about making the component architecture easier to use. I don't 
want to write code, applications *or* components, where I need to go to 
separate ZCML files for registrations. I considered it a pain after 
having tried it with a large application, before working with Grok. Now 
that I've used Grok, I *really* don't want to go back anymore. :) I also 
want other people to use Grok-based components, which is why we got a 
plan. :)

> Second, when the first approach is inappropriate for whatever reason, 
> consider putting the grok glue code in a separate module.  That way 
> someone can import your generally interesting code without depending on 
> grok, or the grok approach.  Perhaps someone else will even add a zcml 
> file, and folks can choose to go the grok road or the zcml road for your 
> package.

All grok packages already have a configure.zcml file. It just contains a 
single line, telling it to grok the package using the only ZCML 
directive defined by grok, namely 'grok' :). The result of including 
that configure.zcml is the same - component registrations. This is why 
we are already close to reusability of grok-based code in Zope 3. Two 
reasons why more needs to be done:

* pulling in grok right now pulls in more dependencies than you want if 
just do simple component registration

* most importantly, grok messes with Zope 3 security. This shouldn't be 
the case in plain Zope 3, so depending on the grokking infrastructure 
shouldn't mean that this gets pulled in.

> In sum, I laud the Grok community's energy and desire to make cool Zope 
> 3 technologies easier to begin with and use.  But I encourage you to 
> direct some of your energy to remembering that one of the cool Zope 3 
> ideas is component reuse.  Don't code yourselves away from the rest of 
> us: we want you around. :-)

I find it somewhat ironic that you say this in response to a message 
where I talk about how we are going to tackle exactly this problem. :) 
Not in the way you envision, perhaps - ZCML won't come back in play, but 
I'm pretty certain the sketched out approach will solve this problem.

> If these ideas are already prevalent in the Grok philosophy, and I'm 
> spouting poorly informed tripe, I hope you at least recognize the 
> goodwill in which this email was written.

I don't think it's poorly informed, though of course you miss a few 
things as an outside observer. Other points, are, of course, of course, 
debatable. :) It's a very valid concern.

I personally am indeed very much aware we need the compatibility both 
ways - Grok is compatible with Zope 3, and Zope 3 compatible with Grok. 
Only that way can we have cross-fertilization. It is also very important 
as part of the secret Grok outreach program to Zope 3 developers - we 
need to be able to say that your worries are simply incorrect and get 
you to use grok-based code. :)

How to accomplish this, I believe, is to use the component architecture 
APIs, not ZCML. Component registrations are component registrations in 
the end, no matter how they are accomplished.

Regards,

Martijn