[Zope3-dev] Re: Autoconfiguring a site.zcml

[David Pratt]

Hi Tres and Martijn. I apologize for the length of this post in advance. 
I have been spending a bit of time on app buildouts over the past days. 
What I have learned about some of this is that there are a couple of 
ways to configure apps with pluses and minuses.

There are also major differences in time running buildouts depending on 
how you construct an app (since buildout checks against eggs in the main 
setup, dev eggs etc). To get a good sense of the acrobatics it is good 
to increase the verbosity of the output generated in buildout to -vv 
(you can add more v's to see more).

In looking at the way I am developing, my goals are package reuse and 
thin glue in an app package (that is also the app egg) to bind packages 
to make an application consisting mostly of installation, security, 
testing and perhaps skinning. That said, I do not really want or care 
about organizing packages neatly under a single src folder to pull them 
together into a larger egg. To keep it simple, I just want to list the 
eggs I use in app part of my buildout. In the end it all about making 
sure that the code is able to connect.

You are penalized for this approach in app buildouts since more small 
independent eggs (holding some generic functionality) with their own 
dependencies equals more checks. Being explicit like this increases the 
amount of time for the buildout to run, particularly if there are plenty 
of eggs with many dependencies. Think in numbers like 150 - 200+ eggs 
with zope and zope app - no exaggeration. Ok - now think again - that 
could be your time and not machine time working out that all 
dependencies are met. You start saying I like what computers can do and 
I can be patient while the checks are made.

An approach that that leads to shorter app buildouts is to group 
packages under the main app setup.py. This is currently what fits with 
the development approach spelled out in the app recipe docs but it has 
downsides also. This is nice for speed in running a buildout but don't 
like the fact that the main setup has got to capture dependencies for 
the collection as a whole because you just a bigger egg with multiple 
packages where you still need to track down dependencies. True, some of 
these may be packages from external eggs you need to list, but probably 
more are packages that you have assembled within your app (that may be 
desirable to re-use) but where you have not been being explicit about 
the dependencies (since these are determined in an egg not a package).

Personally, I don't want to take the time to try to pull the 
dependencies together in an app that is a collection of packages like 
this - since it is a chore not worthy of a human. A machine can do this 
better than I can (and it is welcome to the task!). Also, apps are prone 
to change - each change equals more time re-examining this over this 
again and again amidst a constantly changing landscape of package 
revisions - yuk. When you add that as a zope3/python developer you are 
mostly working at a package level, you realize that spelling 
dependencies with more granularity in an egg is much more attractive and 
just easier overall than figuring this out for an app with a cluster of 
packages afterwards (on top of how this all goes together with other 
external/namespace packages like zope, zope.app, z3c, zc etc, etc).

I feel the same way about the site.zcml. In the initial docs on the app 
recipe, a smaller site.zcml is recommended since you can tuck most of 
the meat in your app configuration. In experimenting, the bottom line is 
it doesn't really matter where it is - because is really the same size 
regardless of where the pieces reside.  So I have been considering - 
what am I really trying to do here - and how much grief do I want to 
encumber putting it all together. The semantics of having a large 
site.zcml that takes care of app configuration is not inconsistent with 
the fact that I am using the buildout to configure and build my app. So 
in the end I would be happy to consider a larger site.zcml that does the 
work of organizing includes to all packages in the app. And at the 
package level being left with just making the includes within it (such 
as including a browser or an ftest sub package) to complete the 
configuration tree.

That said, I don't relish to time it will take to looking at ordering 
all of the includes in a site.zcml. It is tedious work and not fun at 
all. Here, this is still about dependency first. We need to make sure 
that as configuration is read, the app already knows enough for the next 
bit of configuration so it does not choke when run. Put it in the wrong 
order and your app will not understand what an adapter is, or a provider 
or one of your custom components. Unlike the comfort we had in 
automatically having all of the zope or zope app packages when we use 
zope we are now in the drivers seat to determine what we need by 
spelling it out. This provides power and flexibility at the expense of 
understanding what you need which can take time.

Perhaps the biggest hurdle is that when we are speaking of dependencies, 
we are talking apples and oranges to some extent - because eggs spell 
dependencies but it is packages we need to put into our configuration. 
Overall, it should not matter really as far as I can see. If egg1 needs 
the contents of egg2, it stands to reason that that package includes of 
egg2 need to be pushed higher in the site.zcml that those of egg1. 
Further if we adhere to good information in the egg we should not have 
to worry that imports of packages in eggs will not work. So I tend to 
believe that some means of scoring, weighting or boosting could be used 
together with the exploration of each egg for a configuration 
(*-configure.zcml, *-meta.zcml) to autogenerate the site.zcml as the 
buildout proceeds. I see potential to also include (in the app part of 
the buildout recipe) a simple boolean value to indicate whether your app 
uses an overrides so it can write an includeOverrides into the site.zcml 
to point to an overrides.zcml in your app egg as part of the automation.

Perhaps something like this could save us poor humans the terrible and 
ominous fate of ordering all these includes, much of which is based on 
dependencies spelled out in the eggs already. I don't think we need a 
gui for this honestly - we just need some logic. In any case, I am 
certain others will likely arrive at this conclusion with eggs and zope 
given enough time. It only took me a few days of this to begin to 
consider whether there has got to be better way that can operate more 
dynamically on the app (with some better certainty that you have got 
your bases covered on dependency) given a list of eggs you are using. 
Many thanks.

Regards,
David


Martijn Faassen wrote:
> Hello,
> 
> Tres Seaver wrote:
>> David Pratt wrote:
>>> On the basis that eggs spell out dependencies, I am thinking the 
>>> inclusion of packages and their dependencies should be enough to 
>>> dictate the sequence of inclusion for package configuration (and 
>>> creation of the site.zcml) for the app buidout recipe. This could be 
>>> a option to the current manual configuration.
>>
>> - -1.  Configuration is *policy*;  implicitly wiring in the default
>> configuration for every egg on the path is not going to be an acceptable
>> default.
> 
> In the path? David didn't say in the path, did we? What about in the 
> buildout? Since my buildout already says explicitly it wants egg foo, 
> and egg foo needs egg bar, it is a major pain to have to specify the 
> ZCML for bar manually.
> 
> I don't think something being policy means it's automatically a bad 
> candidate for automation. Information about the policy may after all be 
> elsewhere in the system, for instance in a buildout.cfg.
> 
>>> Presumably, the reason folks would use this recipe is to configure 
>>> one or more of the same app. I know attempting to do this manually is 
>>> prone to errors if packages are not in the correct sequence or if you 
>>> miss the configuration of a package. Any thoughts on this. Many thanks.
>>
>> Not necessarily.  There are really two kinds of packages in play here:
>> "libraryish" packages, which supply mechanisms, and "applicationish"
>> packages, which use the mechanism according to some policy.  I would
>> argue that the only things you can safely auto-include would be the
>> 'meta.zcml', because it is policy-free.  Reusable packages need to avoid
>> imposing policy (they may *suggest* it, but they shouldn't insist).
> 
> That's where we have the concept of overridable defaults. So the default 
> should be to follow the suggestions, and there should be an option in 
> the system to override this suggestion.
> 
>> I would cheer if somebody proposed writing a UI which introspcts
>> packgaes for such suggestions, and allows the site manager to merge /
>> override them to create an appropriate 'site.zcml'.  Until then, I don't
>> want package authors dictating to site managers.
> 
> Who are these ZCML-using site managers you are speaking of? Anyway, are 
> you saying you want some form of ZCML UI to be created before *any* 
> automation can be implemented? Asking for the creation of a UI on the 
> Zope 3 mailing list is paramount to waiting forever...
> 
> If you never automate policy, you'll be writing a lot of stuff by hand. 
> That may be fine for you, but I myself would prefer to write less boring 
> code and focus on more interesting problems.
> 
> Regards,
> 
> Martijn
> 
> _______________________________________________
> Zope3-dev mailing list
> Zope3-dev at zope.org
> Unsub: 
> http://mail.zope.org/mailman/options/zope3-dev/fairwinds%40eastlink.ca
>