[Zope-dev] Dependencies for ZCML

[Chris McDonough]

Martijn Faassen wrote:
>>  "mandatory configuration is a contradiction
>> in terms."  I therefore don't believe that tests which try to load ZCML
>> are useful, at least for "library" pacakges (as opposed to "applications").
> 
> Yeah, I know you're a purist on this topic.
> 
> Here are some observations on configuration, mandatory or not, and ZCML.
> 
> Sometimes you can only test a library when a particular utility or 
> adapter or event handler is registered. The library uses this utility or 
> adapter in its own logic and while the utility or adapter is intended to 
> be replaceable (to make the library pluggable), it is mandatory to 
> actually register a component that fulfills the interface requirements 
> in order to use the library at all.

Can you provide an example?  In my own libraries, very rarely does this happen;
I tend to only depend on components like this "at the edges"; eg. in I only try
to allow pluggability via the CA within an application or a framework, very
rarely in a straight library.

But if somehow it does happen, and I need to test the library, and the library
relies on some utility external to itself being registered, I'll register a
"dummy" utility for purposes of testing.  I'll never use a "real" implementation
of the component during *unit* testing (although during integration testing I'll
use the real one instead of the dummy one of course).

> As a convenience to the users of this library, that library's 
> configure.zcml will provide default implementations (which may be the 
> right ones in most cases). This is a useful pattern.

Minor nit: the configure.zcml won't *provide* any default implementation, but
might point at one.  This implementation may or may not live in the package that
the configure.zcml lives in.

> I'll note another pattern. A library could define an interface, and a 
> few event handlers that react on this interface. Now if you use that 
> interface on your own objects, your object will trigger those event 
> handlers. Your interest as a library user is not directly in those event 
> handlers and you may be unaware of their existence, but they do maintain 
> something that is important to you as a user (an index, for example). So 
> this is an example of registrations that should be loaded in order to 
> use the library, and there's not even much interest here in overriding 
> those registrations - mandatory configuration.

If some library comes along that thinks it knows enough to register mandatory
arbitrary CA configuration, I'll classify it along with the Python stdlib
"logging" module (which registers "atexit" handlers) or asyncore dispatchers
(whose constructors register the dispatcher with a socket map); these libraries
tend to be wrong about the configuration they register under some circumstance
and it probably would have been better for their authors to do less, pushing the
configuration more into some glue package or the application that uses them
rather than down into the library itself.  That would offer maximum configurability.

In particular, I'm pretty sure that to be maximally useful outside a Zope
context (which may or may not be desireable for the maintainer), library
packages should probably be broken up into two pieces: a piece that is a
straight Python library that contains no ZCML, then some "glue" package that
provides both an API to use the library in a Zope context along with (possibly)
some ZCML that is meant to be loaded by an application.  An example of this
pattern today is in Chameleon: chameleon.zpt and chameleon.core have no
dependency on ZCML; however, the z3c.pt package contains ZCML and an API that
allows Chameleon to be used in a Zope context.  It depends on chameleon.zpt, but
chameleon.zpt can be used in, say, Turbogears without the TG folks needing to
understand anything about ZCML, which feels quite right.

> Now when testing these libraries you could do three things:
> 
> * not use ZCML at all and recreate the effect of these registrations in 
> Python code.

This is exactly what I do always (at least when unit testing; integration
testing is a different story).

> 
> * use the ZCML in the package's configure.zcml. (perhaps through 
> ftesting.zcml)
> 
> * construct ZCML in the tests itself and load it.
> 
> In fact there's a fourth way you could go and use martian-style 
> patterns, where you can manually 'grok' a component in tests that 
> inherits a particular base class.
> 
> We could argue that all ZCML in use in tests should be rewritten to 
> manual registrations from Python code. Is this indeed a useful exercise? 
> Doesn't that in some cases make tests harder to understand, as 
> lower-level APIs are in use that are not as recognizable as the 
> equivalent ZCML directives? (say, registering an event) Don't we place a 
> burden on the test writers to learn these APIs while they could use the 
> ones abstracted away behind ZCML instead?

Folks who write *unit* tests (as opposed to integration tests) need to supply
mock implementations of registered components anyway.. otherwise the test isn't
a unit test, it's an integration test.  Personally, I don't really write
automated integration tests other than via something like twill or selenium
running against an actual application, because it's not all that interesting to
me to do integration testing of only two components in isolation that might
theoretically be used together; I'd rather just be done with it and test the
entire application end-to-end.

> I will also repeat my observation that if a package has ZCML in it that 
> never gets loaded by the tests of that package, that means that there 
> are no automatic tests for this ZCML. There is something in this package 
> that is not tested and can only be tested indirectly. Isn't that 
> something we try to avoid?

The ZCML may be there only as a convenience.  If the package has no integration
tests, it might never get tested *within the package's tests*.  That'd sometimes
be OK, because it'd get tested as part of some integration (e.g. integration
tests for an application or a framework that uses the package).  And maybe it
wouldn't get tested at all; but that'd often be a small price to pay if you
could use it without 5 otherwise useless dependencies outside of a Zopey context.

> You're saying that we should be an include of the zope.component 
> 'meta.zcml' in *all* ZCML files that register an adapter? This is 
> certainly not happening always now. That's like import before usage in 
> Python modules, right?

No.  He's saying that an application that makes use of some ZCML by inclusion
from some package may need to load some metaconfiguration before the included
file can be successfully used.  Much like including some snippet of
configuration in an Apache config file; e.g. maybe you can't use the
"RewriteRule" directive unless mod_rewrite is loaded.  People understand that
when they use a configuration snippet, they may need to make changes to some
other part of the configuration to use the snippet.

> I'll note that martian-based code actually does use that pattern - any 
> package based on grokcore.* for instance automatically follows this 
> pattern.
> 
> I'll also note that with martian-based configuration the situation is 
> clear for setup.py: the dependencies needed for configuration are always 
> normal dependencies of a package.
> 
> With ZCML-based configuration the situation is far less clear.

I don't know much about grokcore stuff.  I think there's some fundamental
disagreement about the role of configuration;  I think Grok tends to treat
configuration more like software, while non-Grok stuff tends to treat
configuration like ... well... configuration.

> So what does all of this mean for Dan's question? I don't know yet.
> 
> I think we should observe some packages. We strive for library-like 
> packages. More library-like packages should likely not have to do a lot 
> of work in their configure.zcml, but the amount of work is not always 
> zero.

I'll throw something out there: if a package *requires* configuration via ZCML
to operate properly, I don't think it's really just a library (in the common
Python sense).  Instead, I think its either a framework itself or some plugin to
a framework.  OTOH, if it just offers up some default configuration that needn't
be used to use the software, it might be a library.

 Dan, do you have any examples of packages where you are wondering
> about what to do? Let's examine then and reason about how they could be 
> organized.

- C