[Zope-dev] "ZTK" futures: one big package?

Tue May 12 04:44:20 EDT 2009

Hi,

I agree with the package inflation. It makes people nervous of the
complexity.
But putting everything back into a pile of mud sounds like going back from
where we came, and I this situation makes people even more nervous when they
want to look into zope.
As far as I know, the big dependency clean up is not finished yet, no? Maybe
it might be better to do that first. With a clean dependency graph, its
easier to see, which set of packages could be merged into one package.
About the configurability of all the subpackages, I remember a proposal long
time ago to reduce the number of zcml directives. This time I am in favor of
inventing some zcml directives with some care. But before doing that, it
might be better to write two or free easy tutorials about how to write
custom zcml directives and grokkers. That would result in people writing
more custom zcml directives and grokkers and get the hang of it. With the a
much bigger userbase of writers of custom zcml directives and grokkers, the
community as a whole will get a much better idea of how to use zcml, and I
think gaining that community sense first will result in some high quality
new general usable zcml directives. At the moment, and with my limited view,
it looks like most zope ppl really don't know how to write custom zcml
directives.

Just my 0.02€

Best regards,

       Patrick

On Tue, May 12, 2009 at 09:08, Chris McDonough <chrism at plope.com> wrote:

> I realize now that I've neglected to give sufficient reasoning for why less
> granular packaging would be a good thing.
>
> I've noticed that there's a common theme in Zope development, software, and
> packages that I can only describe as "power law development" or "turtles
> all the
> way down".  It's a bit of an antipattern, unfortunately.
>
> I'll provide an example by way of Zope-3-the-appserver.  In an application
> that
> uses Zope-3-the-appserver, many individual subframeworks will be used.  For
> example, there is a traversal subframework, a security subframework, a
> cataloging subframework, and so on.
>
> Each of these subframeworks acts as a logical unit, and through the magic
> of the
> component architecture, each can be replaced wholesale by registering some
> adapter.  However, each of these subframeworks tend to also have settings
> that
> can be configured.  For example, individual traversal steps for certain
> types of
> objects can be overridden by registering an adapter that *configures* the
> subframework.  In the case of Zope 3, we have a traversal situation where
> the
> larger traversal subframework can either be replaced wholesale via an
> adapter
> registration or extended piecemeal via other adapter registrations.
>
> The problem is that the mechanism to *replace* the subframework is the same
> as
> the mechanism to *configure* it (both are done via adapter registration,
> sometimes even in the same file).  This is theoretically fine.  But in
> reality,
> it's tremendously hard for someone just walking up to a complex system like
> Zope
> 3 to discern adapter registrations that replace subsystems from those which
> merely configure subsystems.  An inability to discern the difference leads
> to
> situations where people just "don't get the joke" and try to wiggle wires
> to
> configure-to-death a existing subsystem that's clearly suboptimal for their
> use
> case instead of just replacing it wholesale with a much simpler custom
> policy.
> They just don't know it was engineered to be replaced.  So they keep adding
> more
> configuration code to the existing subframework to handle various 1% edge
> cases.
>  Often this code makes the subframework tremendously complex, and the
> subframework grows inappropriate dependencies along the way.  *Sometimes*
> the
> situation gets so confusing for a new user, they just quit and go use
> something
> else.
>
> This is a pattern that happens over and over again in Zope development.  In
> my
> personal opinion, the original error was trying to make the subframework
> configurable at all.  Instead, the subframework should be replaceable
> easily,
> but it should itself be relatively rigid.  At very least, for subframeworks
> that
> really do require extra configuration (should be very few), this
> configuration
> should be done via highly specialized ZCML directives (or grokkers), as
> opposed
> to some very general adapter registration that can't be easily discerned
> from
> other adapter registrations by a newbie.
>
> If the subframeworks were more rigid (but replaceable), the *intent* of the
> subframework author could be more easily discerned, and fewer people would
> fall
> into the trap of adding more configuration code to a subframework instead
> of
> just replacing it entirely.  And fewer people would just walk away in
> frustration.
>
> What does this have to do with packaging?  Well, currently, there's a
> dizzying
> number of packages that make up "the ZTK" (nee "Zope 3").  Each of these
> packages is a pure peer of all others in a PyPI listing with no real way to
> get
> a sense of their relative importance other than performing a linear audit.
>  Even
> if a user *does* do a linear search of all of them, it's still awful hard
> to
> discern for some new user which ones are "important", and which ones just
> happen
> to exist by some inequity of history without trying to install it.  The
> user
> needs to gain some holistic knowledge of the system in order to discern the
> important bits from these historical inequities.
>
> Most new users understandably just walk away from *all* Zope packages
> before
> they gain this knowledge; it's just too hard for them to tell the
> difference
> between the truly important and reusable bits and the stuff that just
> happens to
> be packaged up and released but which is useless outside of some highly
> specific
> context.  In effect, we just don't communicate *intent* very effectively in
> our
> current packaging structure.
>
> In my opinion, this is why a lot of Python developers who are otherwise
> very
> smart have given up on trying to use Zope packages.  The time required to
> figure
> out which ones are useful and which ones aren't is just too high.  It's way
> "easier" for them to write them all off wholesale and just write what they
> need
> from scratch or use somebody else's software.  Good developers tend to like
> small, useful libraries and frameworks.
>
> We can ameliorate the situation in a few ways:
>
> - We can reduce the number of distributions.
>
> - We can make each current Zope package distribution independently useful.
>
> My suggestion is that we do the former first in order to communicate
> *current
> intent* (the "state of reality right now").  We can do the latter after
> this in
> pieces, hopefully aided by some new developers we've picked up by making it
> easier to find useful stuff.
>
> Once we deflate our current set of packages down to a reasonable number,
> the
> packages listed in PyPI will immediately start to reflect "the state of
> reality
> right now".  As a result, we'll hopefully be able to get some new blood in
> the
> form of new developers that use the smaller bits outside Zope to help us
> tease
> the truly independent pieces out of the larger pile.  If we do this, at no
> time
> after the deflation will PyPI listings ever as badly advertise "the state
> of
> reality" as it is advertised right now, and the community will hopefully
> again
> start to grow.
>
> - C
>
>
> On 5/11/09 11:11 AM, Martijn Faassen wrote:
> > Hey,
> >
> > Chris McDonough wrote:
> >
> >   >  Instead, I have argued for promoting packages that have some life of
> >   >  their own (independent of the rest of the pile) into subprojects
> that
> >   >  have their own release cycles.
> >
> >   >  Then outside projects such as Plone and Grok could depend on
> >   >  independent versions of such packages, giving them slightly more
> >   >  flexibility than requiring a "version of the ZTK".
> >
> > We already have that flexibility today. To me, the utility of a release
> > of version numbers in the ZTK does not at all exclude the potential to
> > evolve the packages to more independent sub-projects.
> >
> >> Given that this suggestion has been met with skepticism, let me try
> another
> >> tact.
> >
> > I think that's an inaccurate description of the response you got. I'm
> > quite positive about trying to give as many packages as possible a life
> > of their own. I don't think you got anyone else arguing against this
> > point of view.
> >
> > I'm also quite positive that some packages are:
> >
> > * useful as independently distributed packages
> >
> > * only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they
> > depend on a significant set of Zope packages.
> >
> > I'd like to get out of this paradigm:
> >
> > * the Zope packages are independent sub-projects.
> >
> > * therefore we cannot distribute a list of versions that work best
> together.
> >
> > And this one:
> >
> > * if we distribute a KGS of anything
> >
> > * packages in that anything aren't independently reusable automatically
> > and should be merged into a ball.
> >
> > I'd also like to get out of the following paradigm:
> >
> > * the Zope packages are not independently reusable yet
> >
> > * therefore we should distribute them all together
> >
> > We're in a grey area. Some package are here, some packages are there,
> > some are in between. Some packages build on other packages, but have
> > clear dependency structures. Some don't have clear dependency
> > structures. Some have better documentation and better focus than others.
> >
> > If there is to be a merging of code together, then I propose we continue
> > the project where the ZMI code is merged into one or a few packages. We
> > can also investigate merging 2 or 3 packages together where it seems to
> > make sense, or simply moving code between packages (some code needs to
> > go down the dependency list, some up).
> >
> >> Instead of thinking about it this way, could we think about it as
> >> *deflating* the current set of zope.* packages that do not currently
> have a
> >> meaningful life of their own into a single setuptools distribution named
> "ZTK".
> >>    This package would include most everything in zope.app*, plus things
> like
> >> zope.server, zope.publisher, and other things that just aren't useful
> outside of
> >> Zope-the-appserver, or which currently just depend on "too much".
> >
> > -1
> >
> > This would make it a lot harder to:
> >
> > * clean up dependency relationships with the goal of creating more
> > reusable code. We'd all hide them in a sumo ball again.
> >
> > * get rid of bits we *don't want anymore*. If I need anything in a sumo
> > package I'd need *all* of it.
> >
> > * override individual packages with newer versions
> >
> > * we've done a lot of refactoring recently trying to separate the UI
> > from packages. This is done by creating a *new* package, leaving the old
> > package behind. We can do this, test this and release this
> > package-by-package now.
> >
> >> Over time, we'd tease out the dependencies of packages that live in the
> ZTK
> >> distribution, making them useful outside without any dependency on the
> ZTK.  The
> >> names of these packages could be arbitrary (they wouldn't need to retain
> their
> >> old "zope.*" names).  Some backwards dependency shims would be added to
> the ZTK
> >> to prevent old imports from breaking, and the ZTK distribution would
> then just
> >> have a dependency on the thing that got broken out.
> >
> > I don't like the attempt to redefine what the ZTK means to a giant ball
> > of Python packages. That's implying that, say, zope.component is *not*
> > in the ZTK. That's wrong.
> >
> > Why generate a whole lot of work for ourselves getting from where we are
> > now to here? We've learned how to work with the current situation in
> > 2007 already.
> >
> >> I'm thinking that this would simplify things greatly for people who want
> to be
> >> consumers of "truly independent" Zope packages.  There'd be exactly N of
> them
> >> available for download (where N is much less than 100, more like 20),
> plus the
> >> ZTK, which would have the rest of the pile in it over time.
> >
> > I don't see why a big package would "simplify things greatly" for you or
> > anyone else.
> >
> >> If someone wanted
> >> to use a forked version of a package that lived in the ZTK distribution,
> they'd
> >> either do so by teasing out the dependencies and making it "truly
> independent"
> >> or they'd just reroll a custom version of the entire ZTK distribution.
> >
> > And that's easier than the current situation how? Are you really
> > proposing we destroy the dependency information we've already teased out
> > and then make people do the work again?
> >
> >> Does this make any sense?
> >
> > Not a lot in my book.
> >
> > I think an important reason why there's so much awareness of dependency
> > issues in the Zope world now (and effort spent on it) is precisely
> > because we released our separate packages and can see the dependency
> > information clearly.
> >
> > Regards,
> >
> > Martijn
> >
> > _______________________________________________
> > Zope-Dev maillist  -  Zope-Dev at zope.org
> > http://mail.zope.org/mailman/listinfo/zope-dev
> > **  No cross posts or HTML encoding!  **
> > (Related lists -
> >   http://mail.zope.org/mailman/listinfo/zope-announce
> >   http://mail.zope.org/mailman/listinfo/zope )
> >
>
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists -
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope )
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope-dev/attachments/20090512/7ac328cd/attachment-0001.html 