[Zope-dev] "ZTK" futures: one big package?

Tue May 12 03:08:05 EDT 2009

I realize now that I've neglected to give sufficient reasoning for why less 
granular packaging would be a good thing.

I've noticed that there's a common theme in Zope development, software, and 
packages that I can only describe as "power law development" or "turtles all the 
way down".  It's a bit of an antipattern, unfortunately.

I'll provide an example by way of Zope-3-the-appserver.  In an application that 
uses Zope-3-the-appserver, many individual subframeworks will be used.  For 
example, there is a traversal subframework, a security subframework, a 
cataloging subframework, and so on.

Each of these subframeworks acts as a logical unit, and through the magic of the 
component architecture, each can be replaced wholesale by registering some 
adapter.  However, each of these subframeworks tend to also have settings that 
can be configured.  For example, individual traversal steps for certain types of 
objects can be overridden by registering an adapter that *configures* the 
subframework.  In the case of Zope 3, we have a traversal situation where the 
larger traversal subframework can either be replaced wholesale via an adapter 
registration or extended piecemeal via other adapter registrations.

The problem is that the mechanism to *replace* the subframework is the same as 
the mechanism to *configure* it (both are done via adapter registration, 
sometimes even in the same file).  This is theoretically fine.  But in reality, 
it's tremendously hard for someone just walking up to a complex system like Zope 
3 to discern adapter registrations that replace subsystems from those which 
merely configure subsystems.  An inability to discern the difference leads to 
situations where people just "don't get the joke" and try to wiggle wires to 
configure-to-death a existing subsystem that's clearly suboptimal for their use 
case instead of just replacing it wholesale with a much simpler custom policy. 
They just don't know it was engineered to be replaced.  So they keep adding more 
configuration code to the existing subframework to handle various 1% edge cases. 
  Often this code makes the subframework tremendously complex, and the 
subframework grows inappropriate dependencies along the way.  *Sometimes* the 
situation gets so confusing for a new user, they just quit and go use something 
else.

This is a pattern that happens over and over again in Zope development.  In my 
personal opinion, the original error was trying to make the subframework 
configurable at all.  Instead, the subframework should be replaceable easily, 
but it should itself be relatively rigid.  At very least, for subframeworks that 
really do require extra configuration (should be very few), this configuration 
should be done via highly specialized ZCML directives (or grokkers), as opposed 
to some very general adapter registration that can't be easily discerned from 
other adapter registrations by a newbie.

If the subframeworks were more rigid (but replaceable), the *intent* of the 
subframework author could be more easily discerned, and fewer people would fall 
into the trap of adding more configuration code to a subframework instead of 
just replacing it entirely.  And fewer people would just walk away in frustration.

What does this have to do with packaging?  Well, currently, there's a dizzying 
number of packages that make up "the ZTK" (nee "Zope 3").  Each of these 
packages is a pure peer of all others in a PyPI listing with no real way to get 
a sense of their relative importance other than performing a linear audit.  Even 
if a user *does* do a linear search of all of them, it's still awful hard to 
discern for some new user which ones are "important", and which ones just happen 
to exist by some inequity of history without trying to install it.  The user 
needs to gain some holistic knowledge of the system in order to discern the 
important bits from these historical inequities.

Most new users understandably just walk away from *all* Zope packages before 
they gain this knowledge; it's just too hard for them to tell the difference 
between the truly important and reusable bits and the stuff that just happens to 
be packaged up and released but which is useless outside of some highly specific 
context.  In effect, we just don't communicate *intent* very effectively in our 
current packaging structure.

In my opinion, this is why a lot of Python developers who are otherwise very 
smart have given up on trying to use Zope packages.  The time required to figure 
out which ones are useful and which ones aren't is just too high.  It's way 
"easier" for them to write them all off wholesale and just write what they need 
from scratch or use somebody else's software.  Good developers tend to like 
small, useful libraries and frameworks.

We can ameliorate the situation in a few ways:

- We can reduce the number of distributions.

- We can make each current Zope package distribution independently useful.

My suggestion is that we do the former first in order to communicate *current 
intent* (the "state of reality right now").  We can do the latter after this in 
pieces, hopefully aided by some new developers we've picked up by making it 
easier to find useful stuff.

Once we deflate our current set of packages down to a reasonable number, the 
packages listed in PyPI will immediately start to reflect "the state of reality 
right now".  As a result, we'll hopefully be able to get some new blood in the 
form of new developers that use the smaller bits outside Zope to help us tease 
the truly independent pieces out of the larger pile.  If we do this, at no time 
after the deflation will PyPI listings ever as badly advertise "the state of 
reality" as it is advertised right now, and the community will hopefully again 
start to grow.

- C

On 5/11/09 11:11 AM, Martijn Faassen wrote:
> Hey,
>
> Chris McDonough wrote:
>
>   >  Instead, I have argued for promoting packages that have some life of
>   >  their own (independent of the rest of the pile) into subprojects that
>   >  have their own release cycles.
>
>   >  Then outside projects such as Plone and Grok could depend on
>   >  independent versions of such packages, giving them slightly more
>   >  flexibility than requiring a "version of the ZTK".
>
> We already have that flexibility today. To me, the utility of a release
> of version numbers in the ZTK does not at all exclude the potential to
> evolve the packages to more independent sub-projects.
>
>> Given that this suggestion has been met with skepticism, let me try another
>> tact.
>
> I think that's an inaccurate description of the response you got. I'm
> quite positive about trying to give as many packages as possible a life
> of their own. I don't think you got anyone else arguing against this
> point of view.
>
> I'm also quite positive that some packages are:
>
> * useful as independently distributed packages
>
> * only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they
> depend on a significant set of Zope packages.
>
> I'd like to get out of this paradigm:
>
> * the Zope packages are independent sub-projects.
>
> * therefore we cannot distribute a list of versions that work best together.
>
> And this one:
>
> * if we distribute a KGS of anything
>
> * packages in that anything aren't independently reusable automatically
> and should be merged into a ball.
>
> I'd also like to get out of the following paradigm:
>
> * the Zope packages are not independently reusable yet
>
> * therefore we should distribute them all together
>
> We're in a grey area. Some package are here, some packages are there,
> some are in between. Some packages build on other packages, but have
> clear dependency structures. Some don't have clear dependency
> structures. Some have better documentation and better focus than others.
>
> If there is to be a merging of code together, then I propose we continue
> the project where the ZMI code is merged into one or a few packages. We
> can also investigate merging 2 or 3 packages together where it seems to
> make sense, or simply moving code between packages (some code needs to
> go down the dependency list, some up).
>
>> Instead of thinking about it this way, could we think about it as
>> *deflating* the current set of zope.* packages that do not currently have a
>> meaningful life of their own into a single setuptools distribution named "ZTK".
>>    This package would include most everything in zope.app*, plus things like
>> zope.server, zope.publisher, and other things that just aren't useful outside of
>> Zope-the-appserver, or which currently just depend on "too much".
>
> -1
>
> This would make it a lot harder to:
>
> * clean up dependency relationships with the goal of creating more
> reusable code. We'd all hide them in a sumo ball again.
>
> * get rid of bits we *don't want anymore*. If I need anything in a sumo
> package I'd need *all* of it.
>
> * override individual packages with newer versions
>
> * we've done a lot of refactoring recently trying to separate the UI
> from packages. This is done by creating a *new* package, leaving the old
> package behind. We can do this, test this and release this
> package-by-package now.
>
>> Over time, we'd tease out the dependencies of packages that live in the ZTK
>> distribution, making them useful outside without any dependency on the ZTK.  The
>> names of these packages could be arbitrary (they wouldn't need to retain their
>> old "zope.*" names).  Some backwards dependency shims would be added to the ZTK
>> to prevent old imports from breaking, and the ZTK distribution would then just
>> have a dependency on the thing that got broken out.
>
> I don't like the attempt to redefine what the ZTK means to a giant ball
> of Python packages. That's implying that, say, zope.component is *not*
> in the ZTK. That's wrong.
>
> Why generate a whole lot of work for ourselves getting from where we are
> now to here? We've learned how to work with the current situation in
> 2007 already.
>
>> I'm thinking that this would simplify things greatly for people who want to be
>> consumers of "truly independent" Zope packages.  There'd be exactly N of them
>> available for download (where N is much less than 100, more like 20), plus the
>> ZTK, which would have the rest of the pile in it over time.
>
> I don't see why a big package would "simplify things greatly" for you or
> anyone else.
>
>> If someone wanted
>> to use a forked version of a package that lived in the ZTK distribution, they'd
>> either do so by teasing out the dependencies and making it "truly independent"
>> or they'd just reroll a custom version of the entire ZTK distribution.
>
> And that's easier than the current situation how? Are you really
> proposing we destroy the dependency information we've already teased out
> and then make people do the work again?
>
>> Does this make any sense?
>
> Not a lot in my book.
>
> I think an important reason why there's so much awareness of dependency
> issues in the Zope world now (and effort spent on it) is precisely
> because we released our separate packages and can see the dependency
> information clearly.
>
> Regards,
>
> Martijn
>
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists -
>   http://mail.zope.org/mailman/listinfo/zope-announce
>   http://mail.zope.org/mailman/listinfo/zope )
>