[Grok-dev] Re: Grok component reuse in the Zope 3 world (was Re: Re: ANN: megrok.quarry)

[Gary Poster]

On May 4, 2007, at 9:30 PM, Martijn Faassen wrote:

> Hey,
>
> Gary Poster wrote:
> [reusing grok-based code in Zope 3 projects]
>
>> This reply crystalized a concern I have been feeling.
>> The grok approach is throwing out zcml, at the expense of...zcml.
>
> Indeed. Not at the expense of configuration, just ZCML. :)
>
>> There actually was a reason why we wanted something like zcml-- 
>> something *not* in the Python file.  The reason was that we wanted  
>> to make it possible to reuse Python without being forced to reuse  
>> configuration.  We wanted components.
>
> It is in fact pretty infrequent, at least in my experience, to see  
> packages reused without their ZCML being reused. Packages are  
> typically reused together with their registrations. I'm sure it  
> happens on occasion that a package's configure.zcml is replaced  
> with other ZCML external to it, but in these relatively rare cases  
> I see other patterns to fulfill the underlying use cases with Grok.

I have a decent amount of experience both with packages whose zcml is  
fairly replaceable, and with situations that switch security policies  
significantly.  With the security information outside of the Python,  
for instance, this isn't that big of a deal.  As you say in another  
email, yes you then have to redo/maintain the zcml, but the burden in  
comparison to the burden of maintaining code is relatively small.   
Moreover, as you encounter situations like this, you divide up zcml  
into smaller files so that it is easier to say "I want just the  
adapter registrations" or "I just want the subscribers" or "I just  
want these features installed, but not these other ones".

(I certainly don't do a great job with splittling up the zcml  
initially, to be clear.  I do it on an as-needed basis, regarding  
real-world needs as the only reasonable drivers for this kind of  
refactoring.  Even then, of course, there's a decent chance that one  
person's division is no good for another.  It's worked out ok so far.)

>> Whatever its failures--perhaps in spelling?--zcml appears to have  
>> accomplished this.
>
> In my opinion, it's mostly not ZCML that accomplished this but:
>
> * the component architecture
>
> * explicit interfaces
>
> * focus on test-driven development

Sure those are very important.  Configuration separate from the  
Python is integral too, IMO.  Ranking the relatively importances  
seems a relatively useless argument.

>> We, the Zope 3 community, are growing little microcosms of code  
>> reuse.  ZC, Lovely, z3c (Roger and Stephan, largely?), Gocept,  
>> Canonical/SchoolTool, Martijn in the "hurry" namespace, and others  
>> have grown their own worlds of software that are being mixed and  
>> matched and used as components by a variety of projects.
>
> The ability to do this is probably the strongest point of Zope 3.
>
>> I argue that a significant contributor to this flexible ecosystem  
>> is the fact that the vast majority of this code uses ZCML--stuff  
>> outside of the Python files--for configuration and glue.
>
> I argue ZCML is a small factor, below.
>
>> I haven't looked at megrok.quarry, but I expect it is interesting,  
>> like other projects in the grok world.  It also sounds like it is,  
>> in part, general--like it has code that other projects might  
>> want.  Why is it grok code?  Perhaps megrok.quarry is a bad  
>> example, but I read about catalog conveniences, view conveniences,  
>> and other interesting things on this list.
>
> Some of these conveniences are tied up strongly with grok, others  
> could in theory be reused externally to Grok. Splitting up Grok  
> will help in making the latter more easy. Anyway, there are indeed  
> real cases of packages that use Grok and can be generically  
> reusable in Zope 3 packages. I have one or two but I haven't thrown  
> them in the public pool yet as Grok isn't ready. This demonstrates  
> that your concern is real - but I at least am very much aware of  
> it. :)

I actually figured as much--and with the other initial grok folks.   
But I hadn't seen much discussion on this list about it, and had seen  
a number of things (alluded to in my paragraph above) that made me  
think it was worth bringing up.

>> Grok's approach combats reuse outside of your community when it is  
>> used for anything other than an application.   Grok feels like it  
>> is on the way to building another code vacuum--like Plone is, or  
>> at least used to be: code goes in from other projects, but little  
>> comes out.
>
> I don't believe Grok's approach is combating code reuse,

Certainly not for other Grok applications--I don't mean to imply  
anything like that.

> but we do have a little bit of work to be done to avoid some issues  
> that came up.
>
> As I described, we're actively looking into not just making Zope 3  
> compatible with Grok but Grok-based code compatible with Zope 3.  
> Initially grok-based code was entirely compatible with Zope 3, but  
> as we added things to Grok like, especially, disabling security,  
> this has changed. This is a problem that needs fixing, and one we  
> didn't originally anticipate. The approach I sketched out is a  
> response to exactly this problem as it came up when I wanted to  
> port a package - Grok code is not reusable in Zope 3 system. It's  
> *almost* the case, but currently not entirely. This will change.
>
>> What Martijn describes in this reply sounds like it might be in  
>> the right direction.  But IMO and AFAICT it is still insufficient.
>
> Why do you consider it insufficient? In a package that uses grok, I  
> can declare an adapter. Same with ZCML. What's the difference?

If you write good Python code I want to reuse, and I don't want to  
use grok (or have my package "grokked"), I'd like to be able to use  
it without depending on grok.  That's the heart of it.  "Martian"  
appears to be about being able to disseminate the grok approach along  
with your grok packages.  The issue I'm trying to raise is the  
opposite: what if I don't want approach but I do want to share your  
code.

For instance, what if you had written hurry.query for Grok.  It's a  
nice API.  What if I wanted to use it, and I didn't want to use  
Grok.  All I'm asking is that grok encourage their community  
developers who are making packages that are generically useful, like  
hurry.query, to do it in a way that the larger Zope 3 community can  
use them without having to use grok.  There are several ways to  
accomplish this, I'm sure.  I suggested a couple.

> Is it because a grok-based component wouldn't include enough  
> security declarations? (because in grok you don't need to). That's  
> the only thing I can think of. It's a hard one to solve, as writing  
> code with a lot of permissions sprinkled around is a major pain  
> Grok tries to get rid of.

Certainly that's an issue.  Especially given that security changes  
can be very important to us, as I alluded above.  If grok security  
assumptions pervade a bit of code, it's almost certain that I can't  
use it.

>> I have two ideas on how to combat this.
>> First, be vigilant about recognizing components that could be  
>> shared.  Grok is about applications, not components.  Make  
>> separate packages for the bits that we all (the Zope 3 community  
>> outside of Grok) can use.  Make Grok packages that import from the  
>> general ones and add the Python-based glue in the grok-specific  
>> packages.  Plone appears to be doing this now.  Looks good to me.   
>> (I hope they do more!)
>
> It's you who says Grok is about applications and not components,  
> not me. :)

Hm.  OK.

> Grok should also be about components, because otherwise there is no  
> easy way to spin off components from applications. This is a very  
> important mechanism of component generation. In my opinion it would  
> defeat the purpose of Grok if people would have to go back to ZCML  
> as soon as they want to spin off a component. Spinning off a  
> component should be moving code into another package, and the  
> addition of ZCML shouldn't be required.

I don't want grok developers to *have* to make some ZCML; but I don't  
want grok developers to make nice components that exclude those of us  
who continue to choose to use zcml.

> Grok is about making the component architecture easier to use. I  
> don't want to write code, applications *or* components, where I  
> need to go to separate ZCML files for registrations. I considered  
> it a pain after having tried it with a large application, before  
> working with Grok. Now that I've used Grok, I *really* don't want  
> to go back anymore. :) I also want other people to use Grok-based  
> components, which is why we got a plan. :)

I know that's how you feel, I know you speak from experience, and I  
respect your opinion.  All I'm saying is, to the grok community,  
please don't force other Zope 3 micro-communities to agree with you  
in order to you use your *potentially sharable* cool code.

>> Second, when the first approach is inappropriate for whatever  
>> reason, consider putting the grok glue code in a separate module.   
>> That way someone can import your generally interesting code  
>> without depending on grok, or the grok approach.  Perhaps someone  
>> else will even add a zcml file, and folks can choose to go the  
>> grok road or the zcml road for your package.
>
> All grok packages already have a configure.zcml file. It just  
> contains a single line, telling it to grok the package using the  
> only ZCML directive defined by grok, namely 'grok' :). The result  
> of including that configure.zcml is the same - component  
> registrations. This is why we are already close to reusability of  
> grok-based code in Zope 3. Two reasons why more needs to be done:

I was trying to be constructive when I suggested that you divide up  
your Python grok hints.  That would let you do the kind of division I  
said I (we) do with multiple zcml files.  Otherwise, all you are  
teling me I have is a big switch: grok, or not grok.  Even there, I  
wonder about the true level of support for the "not grok" switch, but  
I'd have to re-read your docs again to see what has changed since the  
last time I looked.  Certainly the template/view automatic assembly  
story wasn't friendly to non-grokkers.

My contention is that the grok approach makes some sense for  
applications, but is not friendly for sharable packages.

> * pulling in grok right now pulls in more dependencies than you  
> want if just do simple component registration

right.

> * most importantly, grok messes with Zope 3 security. This  
> shouldn't be the case in plain Zope 3, so depending on the grokking  
> infrastructure shouldn't mean that this gets pulled in.

+1

>> In sum, I laud the Grok community's energy and desire to make cool  
>> Zope 3 technologies easier to begin with and use.  But I encourage  
>> you to direct some of your energy to remembering that one of the  
>> cool Zope 3 ideas is component reuse.  Don't code yourselves away  
>> from the rest of us: we want you around. :-)
>
> I find it somewhat ironic that you say this in response to a  
> message where I talk about how we are going to tackle exactly this  
> problem. :) Not in the way you envision, perhaps - ZCML won't come  
> back in play, but I'm pretty certain the sketched out approach will  
> solve this problem.

So, I believe you are saying "I'm already solving this problem by  
forcing everyone who wants to use our code to not use other  
approaches (like ZCML)".  Perhaps I'm overstating it for drama, but  
do you see the kernel of truth?

I'm afraid I don't see the irony in my response.

>> If these ideas are already prevalent in the Grok philosophy, and  
>> I'm spouting poorly informed tripe, I hope you at least recognize  
>> the goodwill in which this email was written.
>
> I don't think it's poorly informed, though of course you miss a few  
> things as an outside observer. Other points, are, of course, of  
> course, debatable. :)

Of course, and thank you for your debate.

> It's a very valid concern.
>
> I personally am indeed very much aware we need the compatibility  
> both ways - Grok is compatible with Zope 3, and Zope 3 compatible  
> with Grok. Only that way can we have cross-fertilization. It is  
> also very important as part of the secret Grok outreach program to  
> Zope 3 developers - we need to be able to say that your worries are  
> simply incorrect and get you to use grok-based code. :)

:-)

> How to accomplish this, I believe, is to use the component  
> architecture APIs, not ZCML. Component registrations are component  
> registrations in the end, no matter how they are accomplished.

I'm afraid we disagree there.  But I can see the attraction of the  
direction you are pursuing for building apps, even if we disagree on  
its applicability to more generally reusable packages.

I very much appreciate your reply.  I don't need to debate this  
further.  My only horse in the race is that I don't want the grok  
community's Python and ZODB-level code to exclude folks like me who  
don't use your application assembly story, for whatever reason.  I  
stated my case, and it sounds like you agreed with at least some of  
it before I even wrote.

Gary