I was just thinking about the future of the "ZTK", and in the past I have argued for not attempting to version the entire set of packages that currently comprises Zope 3 over time as "ZTK releases". Instead, I have argued for promoting packages that have some life of their own (independent of the rest of the pile) into subprojects that have their own release cycles. Then outside projects such as Plone and Grok could depend on independent versions of such packages, giving them slightly more flexibility than requiring a "version of the ZTK". Given that this suggestion has been met with skepticism, let me try another tact. Instead of thinking about it this way, could we think about it as *deflating* the current set of zope.* packages that do not currently have a meaningful life of their own into a single setuptools distribution named "ZTK". This package would include most everything in zope.app*, plus things like zope.server, zope.publisher, and other things that just aren't useful outside of Zope-the-appserver, or which currently just depend on "too much". This "ZTK" distribution would *not* include packages that already have a life of their own outside Zope such as zope.interface, zope.component, zope.configuration, zope.proxy, ZODB, etc. These packages would continue to have their own release cycles. Over time, we'd tease out the dependencies of packages that live in the ZTK distribution, making them useful outside without any dependency on the ZTK. The names of these packages could be arbitrary (they wouldn't need to retain their old "zope.*" names). Some backwards dependency shims would be added to the ZTK to prevent old imports from breaking, and the ZTK distribution would then just have a dependency on the thing that got broken out. I'm thinking that this would simplify things greatly for people who want to be consumers of "truly independent" Zope packages. There'd be exactly N of them available for download (where N is much less than 100, more like 20), plus the ZTK, which would have the rest of the pile in it over time. If someone wanted to use a forked version of a package that lived in the ZTK distribution, they'd either do so by teasing out the dependencies and making it "truly independent" or they'd just reroll a custom version of the entire ZTK distribution. Does this make any sense? - C - C
Chris McDonough wrote:
Given that this suggestion has been met with skepticism, let me try another tact. Instead of thinking about it this way, could we think about it as *deflating* the current set of zope.* packages that do not currently have a meaningful life of their own into a single setuptools distribution named "ZTK".
This "ZTK" distribution would *not* include packages that already have a life of their own outside Zope such as zope.interface, zope.component, zope.configuration, zope.proxy, ZODB, etc. These packages would continue to have their own release cycles.
Yay! Big +1 from me...
Over time, we'd tease out the dependencies of packages that live in the ZTK distribution, making them useful outside without any dependency on the ZTK. The names of these packages could be arbitrary (they wouldn't need to retain their old "zope.*" names). Some backwards dependency shims would be added to the ZTK to prevent old imports from breaking, and the ZTK distribution would then just have a dependency on the thing that got broken out.
Well, if they just used their old zope.* names, no shims would be needed, right? If it works, don't break it so it doesn't work ;-)
I'm thinking that this would simplify things greatly for people who want to be consumers of "truly independent" Zope packages. There'd be exactly N of them available for download (where N is much less than 100, more like 20), plus the ZTK, which would have the rest of the pile in it over time. If someone wanted to use a forked version of a package that lived in the ZTK distribution, they'd either do so by teasing out the dependencies and making it "truly independent" or they'd just reroll a custom version of the entire ZTK distribution.
Does this make any sense?
yes, totally in agreement. Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On 5/10/09 8:50 PM, Chris Withers wrote:
Over time, we'd tease out the dependencies of packages that live in the ZTK distribution, making them useful outside without any dependency on the ZTK. The names of these packages could be arbitrary (they wouldn't need to retain their old "zope.*" names). Some backwards dependency shims would be added to the ZTK to prevent old imports from breaking, and the ZTK distribution would then just have a dependency on the thing that got broken out.
Well, if they just used their old zope.* names, no shims would be needed, right? If it works, don't break it so it doesn't work ;-)
When possible, sure. The situation I'm thinking about in particular when suggesting that we could rename some things is that the current *structure* of packaging is pretty messed up in some cases (zope.publisher vs. zope.app.publisher, or z3c.form vs. zope.formlib, for instance). It might make sense to reshuffle code around that currently lives in separate packages into a single more external logical library or framework that might have aspects of all of the individual packages without needing to carry along the baggage of the old packaging structure. When we move it out, it could have an arbitrary name and an arbitrary structure; shims could exist in the old locations within the ZTK to keep old code running. - C
Hey, Chris McDonough wrote:
Instead, I have argued for promoting packages that have some life of their own (independent of the rest of the pile) into subprojects that have their own release cycles.
Then outside projects such as Plone and Grok could depend on independent versions of such packages, giving them slightly more flexibility than requiring a "version of the ZTK".
We already have that flexibility today. To me, the utility of a release of version numbers in the ZTK does not at all exclude the potential to evolve the packages to more independent sub-projects.
Given that this suggestion has been met with skepticism, let me try another tact.
I think that's an inaccurate description of the response you got. I'm quite positive about trying to give as many packages as possible a life of their own. I don't think you got anyone else arguing against this point of view. I'm also quite positive that some packages are: * useful as independently distributed packages * only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they depend on a significant set of Zope packages. I'd like to get out of this paradigm: * the Zope packages are independent sub-projects. * therefore we cannot distribute a list of versions that work best together. And this one: * if we distribute a KGS of anything * packages in that anything aren't independently reusable automatically and should be merged into a ball. I'd also like to get out of the following paradigm: * the Zope packages are not independently reusable yet * therefore we should distribute them all together We're in a grey area. Some package are here, some packages are there, some are in between. Some packages build on other packages, but have clear dependency structures. Some don't have clear dependency structures. Some have better documentation and better focus than others. If there is to be a merging of code together, then I propose we continue the project where the ZMI code is merged into one or a few packages. We can also investigate merging 2 or 3 packages together where it seems to make sense, or simply moving code between packages (some code needs to go down the dependency list, some up).
Instead of thinking about it this way, could we think about it as *deflating* the current set of zope.* packages that do not currently have a meaningful life of their own into a single setuptools distribution named "ZTK". This package would include most everything in zope.app*, plus things like zope.server, zope.publisher, and other things that just aren't useful outside of Zope-the-appserver, or which currently just depend on "too much".
-1 This would make it a lot harder to: * clean up dependency relationships with the goal of creating more reusable code. We'd all hide them in a sumo ball again. * get rid of bits we *don't want anymore*. If I need anything in a sumo package I'd need *all* of it. * override individual packages with newer versions * we've done a lot of refactoring recently trying to separate the UI from packages. This is done by creating a *new* package, leaving the old package behind. We can do this, test this and release this package-by-package now.
Over time, we'd tease out the dependencies of packages that live in the ZTK distribution, making them useful outside without any dependency on the ZTK. The names of these packages could be arbitrary (they wouldn't need to retain their old "zope.*" names). Some backwards dependency shims would be added to the ZTK to prevent old imports from breaking, and the ZTK distribution would then just have a dependency on the thing that got broken out.
I don't like the attempt to redefine what the ZTK means to a giant ball of Python packages. That's implying that, say, zope.component is *not* in the ZTK. That's wrong. Why generate a whole lot of work for ourselves getting from where we are now to here? We've learned how to work with the current situation in 2007 already.
I'm thinking that this would simplify things greatly for people who want to be consumers of "truly independent" Zope packages. There'd be exactly N of them available for download (where N is much less than 100, more like 20), plus the ZTK, which would have the rest of the pile in it over time.
I don't see why a big package would "simplify things greatly" for you or anyone else.
If someone wanted to use a forked version of a package that lived in the ZTK distribution, they'd either do so by teasing out the dependencies and making it "truly independent" or they'd just reroll a custom version of the entire ZTK distribution.
And that's easier than the current situation how? Are you really proposing we destroy the dependency information we've already teased out and then make people do the work again?
Does this make any sense?
Not a lot in my book. I think an important reason why there's so much awareness of dependency issues in the Zope world now (and effort spent on it) is precisely because we released our separate packages and can see the dependency information clearly. Regards, Martijn
Hey, After reading this, I thought of one benefit that having a "larger" package would have: it's somewhat easier to refactor for dependencies, because all the code will be in a single checkout and all the tests can be run together, and the fixed release can go out as a single release. Having done some dependency refactoring I can see the benefit of that, but I also think it is not a great benefit given the capabilities of buildout-driven development and some of the tools we have (such as compattest). We'd certainly give up a lot just to gain that benefit. Regards, Martijn
A couple of things: - If it helps to think of this thing as not-the-ZTK, that's fine. I don't care what it's called. Maybe "the ZTK" is another package that depends on this big thing plus all the broken out bits. - There's very little work to doing this. You make an SVN directory that has externals pointing at each of these packages, write a setup.py that includes them all, and ship it. - I'm not proposing that we "destroy" existing dependency info. I'm just saying that sometimes it doesn't matter. Some of the bits we're trying to make useful independendently don't make sense to make independent of each other: zope.app.principalannotation, zope.app.dublincore, zope.app.ftp, zope.site, zope.copypastemove, and so forth. It seems to me very silly to continue pretending that these could ever have a meaningful life of their own. - There's nothing precious about the current packaging structure *except* for backwards compatibility. The original egg-ification of Zope 3 was a largely mechanical procedure. I hope you'd agree that if we had it all to do over again, there would be far fewer setuptools distributions; each would provide some bit of logical functionality instead of happening to be a directory in SVN. - I don't disagree that this large distribution has to *depend* on zope.component/zope.interface etc. It's just in a structure where the large distribution becomes a set of packages that can't have their dependencies teased apart (distributed as a sngle setuptools distribution), it falls out that it's "not part of the large distribution", just a dependency of it. I think at some point it's best to just declare defeat, realize that the current packaging structure and distribution strategy is very artificial, realize that the granularity that the 100+ egg strategy gets for us is way, way too granular, and package up the bits that really don't *need* to have a life of their own into some large setuptools distribution. I don't really care if that thing is called "the ZTK", just that it exists. If later it turns out that some piece of functionality in that distribution *is* useful outside, we take it out, make sure it doesn't depend on anything in the large distribution, and make the large distribution depend on it. Another thing is this: even if we're successful in teasing out dependency info so we do have a collection of truly independently useful things, after it's all over, we're going to get to a point one way or another where we make a big package of stuff that just can't have its dependencies teased apart because it *really is just one thing*. Why *not* just do it now? - C On 5/11/09 11:11 AM, Martijn Faassen wrote:
Hey,
Chris McDonough wrote:
Instead, I have argued for promoting packages that have some life of their own (independent of the rest of the pile) into subprojects that have their own release cycles.
Then outside projects such as Plone and Grok could depend on independent versions of such packages, giving them slightly more flexibility than requiring a "version of the ZTK".
We already have that flexibility today. To me, the utility of a release of version numbers in the ZTK does not at all exclude the potential to evolve the packages to more independent sub-projects.
Given that this suggestion has been met with skepticism, let me try another tact.
I think that's an inaccurate description of the response you got. I'm quite positive about trying to give as many packages as possible a life of their own. I don't think you got anyone else arguing against this point of view.
I'm also quite positive that some packages are:
* useful as independently distributed packages
* only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they depend on a significant set of Zope packages.
I'd like to get out of this paradigm:
* the Zope packages are independent sub-projects.
* therefore we cannot distribute a list of versions that work best together.
And this one:
* if we distribute a KGS of anything
* packages in that anything aren't independently reusable automatically and should be merged into a ball.
I'd also like to get out of the following paradigm:
* the Zope packages are not independently reusable yet
* therefore we should distribute them all together
We're in a grey area. Some package are here, some packages are there, some are in between. Some packages build on other packages, but have clear dependency structures. Some don't have clear dependency structures. Some have better documentation and better focus than others.
If there is to be a merging of code together, then I propose we continue the project where the ZMI code is merged into one or a few packages. We can also investigate merging 2 or 3 packages together where it seems to make sense, or simply moving code between packages (some code needs to go down the dependency list, some up).
Instead of thinking about it this way, could we think about it as *deflating* the current set of zope.* packages that do not currently have a meaningful life of their own into a single setuptools distribution named "ZTK". This package would include most everything in zope.app*, plus things like zope.server, zope.publisher, and other things that just aren't useful outside of Zope-the-appserver, or which currently just depend on "too much".
-1
This would make it a lot harder to:
* clean up dependency relationships with the goal of creating more reusable code. We'd all hide them in a sumo ball again.
* get rid of bits we *don't want anymore*. If I need anything in a sumo package I'd need *all* of it.
* override individual packages with newer versions
* we've done a lot of refactoring recently trying to separate the UI from packages. This is done by creating a *new* package, leaving the old package behind. We can do this, test this and release this package-by-package now.
Over time, we'd tease out the dependencies of packages that live in the ZTK distribution, making them useful outside without any dependency on the ZTK. The names of these packages could be arbitrary (they wouldn't need to retain their old "zope.*" names). Some backwards dependency shims would be added to the ZTK to prevent old imports from breaking, and the ZTK distribution would then just have a dependency on the thing that got broken out.
I don't like the attempt to redefine what the ZTK means to a giant ball of Python packages. That's implying that, say, zope.component is *not* in the ZTK. That's wrong.
Why generate a whole lot of work for ourselves getting from where we are now to here? We've learned how to work with the current situation in 2007 already.
I'm thinking that this would simplify things greatly for people who want to be consumers of "truly independent" Zope packages. There'd be exactly N of them available for download (where N is much less than 100, more like 20), plus the ZTK, which would have the rest of the pile in it over time.
I don't see why a big package would "simplify things greatly" for you or anyone else.
If someone wanted to use a forked version of a package that lived in the ZTK distribution, they'd either do so by teasing out the dependencies and making it "truly independent" or they'd just reroll a custom version of the entire ZTK distribution.
And that's easier than the current situation how? Are you really proposing we destroy the dependency information we've already teased out and then make people do the work again?
Does this make any sense?
Not a lot in my book.
I think an important reason why there's so much awareness of dependency issues in the Zope world now (and effort spent on it) is precisely because we released our separate packages and can see the dependency information clearly.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Chris McDonough wrote:
Another thing is this: even if we're successful in teasing out dependency info so we do have a collection of truly independently useful things, after it's all over, we're going to get to a point one way or another where we make a big package of stuff that just can't have its dependencies teased apart because it *really is just one thing*. Why *not* just do it now?
That's why I suggest a ZMI project (or doing away with the ZMI entirely if nobody's interested). That's really the main thing creating entangled dependency structures. I want to create a grokcore.view that doesn't depend on a thousand Zope packages and a ton of Zope code. That's because grokcore.view doesn't need all this stuff. Right now it gets it indirectly through tangled dependencies. We need to try porting it to the latest set of releases and see whether that helps. Once that's done, we need to cut a few more dependencies, and there we are. If all those packages are maintained as a single large ball, I cannot do this analysis. In the current setting, we have experience with this analysis, we have tools to do this analysis. Additionally, I'd need to do a lot of work separating things again in order to release grokcore.view if I were to do the work in a singly maintained ball. But you also said that there's no need to maintain this *just* as a single ball. That is, we could have something similar to the current Zope 3 project that pulls in all the other packages as externals and that we release to PyPI. I'm not sure what this gains us currently. Who would be using this package? Are you suggesting we remove all the other stuff from PyPI or something? Finally, an independent package can be useful by itself, with clear dependency structures, even though it's only useful for people who buy into quite a few of the other packages that you don't want to buy into. Such is the case for zope.site. Regards, Martijn
I realize now that I've neglected to give sufficient reasoning for why less granular packaging would be a good thing. I've noticed that there's a common theme in Zope development, software, and packages that I can only describe as "power law development" or "turtles all the way down". It's a bit of an antipattern, unfortunately. I'll provide an example by way of Zope-3-the-appserver. In an application that uses Zope-3-the-appserver, many individual subframeworks will be used. For example, there is a traversal subframework, a security subframework, a cataloging subframework, and so on. Each of these subframeworks acts as a logical unit, and through the magic of the component architecture, each can be replaced wholesale by registering some adapter. However, each of these subframeworks tend to also have settings that can be configured. For example, individual traversal steps for certain types of objects can be overridden by registering an adapter that *configures* the subframework. In the case of Zope 3, we have a traversal situation where the larger traversal subframework can either be replaced wholesale via an adapter registration or extended piecemeal via other adapter registrations. The problem is that the mechanism to *replace* the subframework is the same as the mechanism to *configure* it (both are done via adapter registration, sometimes even in the same file). This is theoretically fine. But in reality, it's tremendously hard for someone just walking up to a complex system like Zope 3 to discern adapter registrations that replace subsystems from those which merely configure subsystems. An inability to discern the difference leads to situations where people just "don't get the joke" and try to wiggle wires to configure-to-death a existing subsystem that's clearly suboptimal for their use case instead of just replacing it wholesale with a much simpler custom policy. They just don't know it was engineered to be replaced. So they keep adding more configuration code to the existing subframework to handle various 1% edge cases. Often this code makes the subframework tremendously complex, and the subframework grows inappropriate dependencies along the way. *Sometimes* the situation gets so confusing for a new user, they just quit and go use something else. This is a pattern that happens over and over again in Zope development. In my personal opinion, the original error was trying to make the subframework configurable at all. Instead, the subframework should be replaceable easily, but it should itself be relatively rigid. At very least, for subframeworks that really do require extra configuration (should be very few), this configuration should be done via highly specialized ZCML directives (or grokkers), as opposed to some very general adapter registration that can't be easily discerned from other adapter registrations by a newbie. If the subframeworks were more rigid (but replaceable), the *intent* of the subframework author could be more easily discerned, and fewer people would fall into the trap of adding more configuration code to a subframework instead of just replacing it entirely. And fewer people would just walk away in frustration. What does this have to do with packaging? Well, currently, there's a dizzying number of packages that make up "the ZTK" (nee "Zope 3"). Each of these packages is a pure peer of all others in a PyPI listing with no real way to get a sense of their relative importance other than performing a linear audit. Even if a user *does* do a linear search of all of them, it's still awful hard to discern for some new user which ones are "important", and which ones just happen to exist by some inequity of history without trying to install it. The user needs to gain some holistic knowledge of the system in order to discern the important bits from these historical inequities. Most new users understandably just walk away from *all* Zope packages before they gain this knowledge; it's just too hard for them to tell the difference between the truly important and reusable bits and the stuff that just happens to be packaged up and released but which is useless outside of some highly specific context. In effect, we just don't communicate *intent* very effectively in our current packaging structure. In my opinion, this is why a lot of Python developers who are otherwise very smart have given up on trying to use Zope packages. The time required to figure out which ones are useful and which ones aren't is just too high. It's way "easier" for them to write them all off wholesale and just write what they need from scratch or use somebody else's software. Good developers tend to like small, useful libraries and frameworks. We can ameliorate the situation in a few ways: - We can reduce the number of distributions. - We can make each current Zope package distribution independently useful. My suggestion is that we do the former first in order to communicate *current intent* (the "state of reality right now"). We can do the latter after this in pieces, hopefully aided by some new developers we've picked up by making it easier to find useful stuff. Once we deflate our current set of packages down to a reasonable number, the packages listed in PyPI will immediately start to reflect "the state of reality right now". As a result, we'll hopefully be able to get some new blood in the form of new developers that use the smaller bits outside Zope to help us tease the truly independent pieces out of the larger pile. If we do this, at no time after the deflation will PyPI listings ever as badly advertise "the state of reality" as it is advertised right now, and the community will hopefully again start to grow. - C On 5/11/09 11:11 AM, Martijn Faassen wrote:
Hey,
Chris McDonough wrote:
Instead, I have argued for promoting packages that have some life of their own (independent of the rest of the pile) into subprojects that have their own release cycles.
Then outside projects such as Plone and Grok could depend on independent versions of such packages, giving them slightly more flexibility than requiring a "version of the ZTK".
We already have that flexibility today. To me, the utility of a release of version numbers in the ZTK does not at all exclude the potential to evolve the packages to more independent sub-projects.
Given that this suggestion has been met with skepticism, let me try another tact.
I think that's an inaccurate description of the response you got. I'm quite positive about trying to give as many packages as possible a life of their own. I don't think you got anyone else arguing against this point of view.
I'm also quite positive that some packages are:
* useful as independently distributed packages
* only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they depend on a significant set of Zope packages.
I'd like to get out of this paradigm:
* the Zope packages are independent sub-projects.
* therefore we cannot distribute a list of versions that work best together.
And this one:
* if we distribute a KGS of anything
* packages in that anything aren't independently reusable automatically and should be merged into a ball.
I'd also like to get out of the following paradigm:
* the Zope packages are not independently reusable yet
* therefore we should distribute them all together
We're in a grey area. Some package are here, some packages are there, some are in between. Some packages build on other packages, but have clear dependency structures. Some don't have clear dependency structures. Some have better documentation and better focus than others.
If there is to be a merging of code together, then I propose we continue the project where the ZMI code is merged into one or a few packages. We can also investigate merging 2 or 3 packages together where it seems to make sense, or simply moving code between packages (some code needs to go down the dependency list, some up).
Instead of thinking about it this way, could we think about it as *deflating* the current set of zope.* packages that do not currently have a meaningful life of their own into a single setuptools distribution named "ZTK". This package would include most everything in zope.app*, plus things like zope.server, zope.publisher, and other things that just aren't useful outside of Zope-the-appserver, or which currently just depend on "too much".
-1
This would make it a lot harder to:
* clean up dependency relationships with the goal of creating more reusable code. We'd all hide them in a sumo ball again.
* get rid of bits we *don't want anymore*. If I need anything in a sumo package I'd need *all* of it.
* override individual packages with newer versions
* we've done a lot of refactoring recently trying to separate the UI from packages. This is done by creating a *new* package, leaving the old package behind. We can do this, test this and release this package-by-package now.
Over time, we'd tease out the dependencies of packages that live in the ZTK distribution, making them useful outside without any dependency on the ZTK. The names of these packages could be arbitrary (they wouldn't need to retain their old "zope.*" names). Some backwards dependency shims would be added to the ZTK to prevent old imports from breaking, and the ZTK distribution would then just have a dependency on the thing that got broken out.
I don't like the attempt to redefine what the ZTK means to a giant ball of Python packages. That's implying that, say, zope.component is *not* in the ZTK. That's wrong.
Why generate a whole lot of work for ourselves getting from where we are now to here? We've learned how to work with the current situation in 2007 already.
I'm thinking that this would simplify things greatly for people who want to be consumers of "truly independent" Zope packages. There'd be exactly N of them available for download (where N is much less than 100, more like 20), plus the ZTK, which would have the rest of the pile in it over time.
I don't see why a big package would "simplify things greatly" for you or anyone else.
If someone wanted to use a forked version of a package that lived in the ZTK distribution, they'd either do so by teasing out the dependencies and making it "truly independent" or they'd just reroll a custom version of the entire ZTK distribution.
And that's easier than the current situation how? Are you really proposing we destroy the dependency information we've already teased out and then make people do the work again?
Does this make any sense?
Not a lot in my book.
I think an important reason why there's so much awareness of dependency issues in the Zope world now (and effort spent on it) is precisely because we released our separate packages and can see the dependency information clearly.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
chris you have *exactly* mirrored my feelings as a longtime zope2 user/application developer getting in touch with plone3/five. now after a year digging in it I still am not free of it thanks a lot robert Chris McDonough schrieb:
I realize now that I've neglected to give sufficient reasoning for why less granular packaging would be a good thing.
I've noticed that there's a common theme in Zope development, software, and packages that I can only describe as "power law development" or "turtles all the way down". It's a bit of an antipattern, unfortunately.
I'll provide an example by way of Zope-3-the-appserver. In an application that uses Zope-3-the-appserver, many individual subframeworks will be used. For example, there is a traversal subframework, a security subframework, a cataloging subframework, and so on.
Each of these subframeworks acts as a logical unit, and through the magic of the component architecture, each can be replaced wholesale by registering some adapter. However, each of these subframeworks tend to also have settings that can be configured. For example, individual traversal steps for certain types of objects can be overridden by registering an adapter that *configures* the subframework. In the case of Zope 3, we have a traversal situation where the larger traversal subframework can either be replaced wholesale via an adapter registration or extended piecemeal via other adapter registrations.
The problem is that the mechanism to *replace* the subframework is the same as the mechanism to *configure* it (both are done via adapter registration, sometimes even in the same file). This is theoretically fine. But in reality, it's tremendously hard for someone just walking up to a complex system like Zope 3 to discern adapter registrations that replace subsystems from those which merely configure subsystems. An inability to discern the difference leads to situations where people just "don't get the joke" and try to wiggle wires to configure-to-death a existing subsystem that's clearly suboptimal for their use case instead of just replacing it wholesale with a much simpler custom policy. They just don't know it was engineered to be replaced. So they keep adding more configuration code to the existing subframework to handle various 1% edge cases. Often this code makes the subframework tremendously complex, and the subframework grows inappropriate dependencies along the way. *Sometimes* the situation gets so confusing for a new user, they just quit and go use something else.
This is a pattern that happens over and over again in Zope development. In my personal opinion, the original error was trying to make the subframework configurable at all. Instead, the subframework should be replaceable easily, but it should itself be relatively rigid. At very least, for subframeworks that really do require extra configuration (should be very few), this configuration should be done via highly specialized ZCML directives (or grokkers), as opposed to some very general adapter registration that can't be easily discerned from other adapter registrations by a newbie.
If the subframeworks were more rigid (but replaceable), the *intent* of the subframework author could be more easily discerned, and fewer people would fall into the trap of adding more configuration code to a subframework instead of just replacing it entirely. And fewer people would just walk away in frustration.
What does this have to do with packaging? Well, currently, there's a dizzying number of packages that make up "the ZTK" (nee "Zope 3"). Each of these packages is a pure peer of all others in a PyPI listing with no real way to get a sense of their relative importance other than performing a linear audit. Even if a user *does* do a linear search of all of them, it's still awful hard to discern for some new user which ones are "important", and which ones just happen to exist by some inequity of history without trying to install it. The user needs to gain some holistic knowledge of the system in order to discern the important bits from these historical inequities.
Most new users understandably just walk away from *all* Zope packages before they gain this knowledge; it's just too hard for them to tell the difference between the truly important and reusable bits and the stuff that just happens to be packaged up and released but which is useless outside of some highly specific context. In effect, we just don't communicate *intent* very effectively in our current packaging structure.
In my opinion, this is why a lot of Python developers who are otherwise very smart have given up on trying to use Zope packages. The time required to figure out which ones are useful and which ones aren't is just too high. It's way "easier" for them to write them all off wholesale and just write what they need from scratch or use somebody else's software. Good developers tend to like small, useful libraries and frameworks.
We can ameliorate the situation in a few ways:
- We can reduce the number of distributions.
- We can make each current Zope package distribution independently useful.
My suggestion is that we do the former first in order to communicate *current intent* (the "state of reality right now"). We can do the latter after this in pieces, hopefully aided by some new developers we've picked up by making it easier to find useful stuff.
Once we deflate our current set of packages down to a reasonable number, the packages listed in PyPI will immediately start to reflect "the state of reality right now". As a result, we'll hopefully be able to get some new blood in the form of new developers that use the smaller bits outside Zope to help us tease the truly independent pieces out of the larger pile. If we do this, at no time after the deflation will PyPI listings ever as badly advertise "the state of reality" as it is advertised right now, and the community will hopefully again start to grow.
- C
On 5/11/09 11:11 AM, Martijn Faassen wrote:
Hey,
Chris McDonough wrote:
Instead, I have argued for promoting packages that have some life of their own (independent of the rest of the pile) into subprojects that have their own release cycles.
Then outside projects such as Plone and Grok could depend on independent versions of such packages, giving them slightly more flexibility than requiring a "version of the ZTK".
We already have that flexibility today. To me, the utility of a release of version numbers in the ZTK does not at all exclude the potential to evolve the packages to more independent sub-projects.
Given that this suggestion has been met with skepticism, let me try another tact. I think that's an inaccurate description of the response you got. I'm quite positive about trying to give as many packages as possible a life of their own. I don't think you got anyone else arguing against this point of view.
I'm also quite positive that some packages are:
* useful as independently distributed packages
* only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they depend on a significant set of Zope packages.
I'd like to get out of this paradigm:
* the Zope packages are independent sub-projects.
* therefore we cannot distribute a list of versions that work best together.
And this one:
* if we distribute a KGS of anything
* packages in that anything aren't independently reusable automatically and should be merged into a ball.
I'd also like to get out of the following paradigm:
* the Zope packages are not independently reusable yet
* therefore we should distribute them all together
We're in a grey area. Some package are here, some packages are there, some are in between. Some packages build on other packages, but have clear dependency structures. Some don't have clear dependency structures. Some have better documentation and better focus than others.
If there is to be a merging of code together, then I propose we continue the project where the ZMI code is merged into one or a few packages. We can also investigate merging 2 or 3 packages together where it seems to make sense, or simply moving code between packages (some code needs to go down the dependency list, some up).
Instead of thinking about it this way, could we think about it as *deflating* the current set of zope.* packages that do not currently have a meaningful life of their own into a single setuptools distribution named "ZTK". This package would include most everything in zope.app*, plus things like zope.server, zope.publisher, and other things that just aren't useful outside of Zope-the-appserver, or which currently just depend on "too much". -1
This would make it a lot harder to:
* clean up dependency relationships with the goal of creating more reusable code. We'd all hide them in a sumo ball again.
* get rid of bits we *don't want anymore*. If I need anything in a sumo package I'd need *all* of it.
* override individual packages with newer versions
* we've done a lot of refactoring recently trying to separate the UI from packages. This is done by creating a *new* package, leaving the old package behind. We can do this, test this and release this package-by-package now.
Over time, we'd tease out the dependencies of packages that live in the ZTK distribution, making them useful outside without any dependency on the ZTK. The names of these packages could be arbitrary (they wouldn't need to retain their old "zope.*" names). Some backwards dependency shims would be added to the ZTK to prevent old imports from breaking, and the ZTK distribution would then just have a dependency on the thing that got broken out. I don't like the attempt to redefine what the ZTK means to a giant ball of Python packages. That's implying that, say, zope.component is *not* in the ZTK. That's wrong.
Why generate a whole lot of work for ourselves getting from where we are now to here? We've learned how to work with the current situation in 2007 already.
I'm thinking that this would simplify things greatly for people who want to be consumers of "truly independent" Zope packages. There'd be exactly N of them available for download (where N is much less than 100, more like 20), plus the ZTK, which would have the rest of the pile in it over time. I don't see why a big package would "simplify things greatly" for you or anyone else.
If someone wanted to use a forked version of a package that lived in the ZTK distribution, they'd either do so by teasing out the dependencies and making it "truly independent" or they'd just reroll a custom version of the entire ZTK distribution. And that's easier than the current situation how? Are you really proposing we destroy the dependency information we've already teased out and then make people do the work again?
Does this make any sense? Not a lot in my book.
I think an important reason why there's so much awareness of dependency issues in the Zope world now (and effort spent on it) is precisely because we released our separate packages and can see the dependency information clearly.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Hi, I agree with the package inflation. It makes people nervous of the complexity. But putting everything back into a pile of mud sounds like going back from where we came, and I this situation makes people even more nervous when they want to look into zope. As far as I know, the big dependency clean up is not finished yet, no? Maybe it might be better to do that first. With a clean dependency graph, its easier to see, which set of packages could be merged into one package. About the configurability of all the subpackages, I remember a proposal long time ago to reduce the number of zcml directives. This time I am in favor of inventing some zcml directives with some care. But before doing that, it might be better to write two or free easy tutorials about how to write custom zcml directives and grokkers. That would result in people writing more custom zcml directives and grokkers and get the hang of it. With the a much bigger userbase of writers of custom zcml directives and grokkers, the community as a whole will get a much better idea of how to use zcml, and I think gaining that community sense first will result in some high quality new general usable zcml directives. At the moment, and with my limited view, it looks like most zope ppl really don't know how to write custom zcml directives. Just my 0.02€ Best regards, Patrick On Tue, May 12, 2009 at 09:08, Chris McDonough <chrism@plope.com> wrote:
I realize now that I've neglected to give sufficient reasoning for why less granular packaging would be a good thing.
I've noticed that there's a common theme in Zope development, software, and packages that I can only describe as "power law development" or "turtles all the way down". It's a bit of an antipattern, unfortunately.
I'll provide an example by way of Zope-3-the-appserver. In an application that uses Zope-3-the-appserver, many individual subframeworks will be used. For example, there is a traversal subframework, a security subframework, a cataloging subframework, and so on.
Each of these subframeworks acts as a logical unit, and through the magic of the component architecture, each can be replaced wholesale by registering some adapter. However, each of these subframeworks tend to also have settings that can be configured. For example, individual traversal steps for certain types of objects can be overridden by registering an adapter that *configures* the subframework. In the case of Zope 3, we have a traversal situation where the larger traversal subframework can either be replaced wholesale via an adapter registration or extended piecemeal via other adapter registrations.
The problem is that the mechanism to *replace* the subframework is the same as the mechanism to *configure* it (both are done via adapter registration, sometimes even in the same file). This is theoretically fine. But in reality, it's tremendously hard for someone just walking up to a complex system like Zope 3 to discern adapter registrations that replace subsystems from those which merely configure subsystems. An inability to discern the difference leads to situations where people just "don't get the joke" and try to wiggle wires to configure-to-death a existing subsystem that's clearly suboptimal for their use case instead of just replacing it wholesale with a much simpler custom policy. They just don't know it was engineered to be replaced. So they keep adding more configuration code to the existing subframework to handle various 1% edge cases. Often this code makes the subframework tremendously complex, and the subframework grows inappropriate dependencies along the way. *Sometimes* the situation gets so confusing for a new user, they just quit and go use something else.
This is a pattern that happens over and over again in Zope development. In my personal opinion, the original error was trying to make the subframework configurable at all. Instead, the subframework should be replaceable easily, but it should itself be relatively rigid. At very least, for subframeworks that really do require extra configuration (should be very few), this configuration should be done via highly specialized ZCML directives (or grokkers), as opposed to some very general adapter registration that can't be easily discerned from other adapter registrations by a newbie.
If the subframeworks were more rigid (but replaceable), the *intent* of the subframework author could be more easily discerned, and fewer people would fall into the trap of adding more configuration code to a subframework instead of just replacing it entirely. And fewer people would just walk away in frustration.
What does this have to do with packaging? Well, currently, there's a dizzying number of packages that make up "the ZTK" (nee "Zope 3"). Each of these packages is a pure peer of all others in a PyPI listing with no real way to get a sense of their relative importance other than performing a linear audit. Even if a user *does* do a linear search of all of them, it's still awful hard to discern for some new user which ones are "important", and which ones just happen to exist by some inequity of history without trying to install it. The user needs to gain some holistic knowledge of the system in order to discern the important bits from these historical inequities.
Most new users understandably just walk away from *all* Zope packages before they gain this knowledge; it's just too hard for them to tell the difference between the truly important and reusable bits and the stuff that just happens to be packaged up and released but which is useless outside of some highly specific context. In effect, we just don't communicate *intent* very effectively in our current packaging structure.
In my opinion, this is why a lot of Python developers who are otherwise very smart have given up on trying to use Zope packages. The time required to figure out which ones are useful and which ones aren't is just too high. It's way "easier" for them to write them all off wholesale and just write what they need from scratch or use somebody else's software. Good developers tend to like small, useful libraries and frameworks.
We can ameliorate the situation in a few ways:
- We can reduce the number of distributions.
- We can make each current Zope package distribution independently useful.
My suggestion is that we do the former first in order to communicate *current intent* (the "state of reality right now"). We can do the latter after this in pieces, hopefully aided by some new developers we've picked up by making it easier to find useful stuff.
Once we deflate our current set of packages down to a reasonable number, the packages listed in PyPI will immediately start to reflect "the state of reality right now". As a result, we'll hopefully be able to get some new blood in the form of new developers that use the smaller bits outside Zope to help us tease the truly independent pieces out of the larger pile. If we do this, at no time after the deflation will PyPI listings ever as badly advertise "the state of reality" as it is advertised right now, and the community will hopefully again start to grow.
- C
On 5/11/09 11:11 AM, Martijn Faassen wrote:
Hey,
Chris McDonough wrote:
Instead, I have argued for promoting packages that have some life of their own (independent of the rest of the pile) into subprojects that have their own release cycles.
Then outside projects such as Plone and Grok could depend on independent versions of such packages, giving them slightly more flexibility than requiring a "version of the ZTK".
We already have that flexibility today. To me, the utility of a release of version numbers in the ZTK does not at all exclude the potential to evolve the packages to more independent sub-projects.
Given that this suggestion has been met with skepticism, let me try another tact.
I think that's an inaccurate description of the response you got. I'm quite positive about trying to give as many packages as possible a life of their own. I don't think you got anyone else arguing against this point of view.
I'm also quite positive that some packages are:
* useful as independently distributed packages
* only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they depend on a significant set of Zope packages.
I'd like to get out of this paradigm:
* the Zope packages are independent sub-projects.
* therefore we cannot distribute a list of versions that work best together.
And this one:
* if we distribute a KGS of anything
* packages in that anything aren't independently reusable automatically and should be merged into a ball.
I'd also like to get out of the following paradigm:
* the Zope packages are not independently reusable yet
* therefore we should distribute them all together
We're in a grey area. Some package are here, some packages are there, some are in between. Some packages build on other packages, but have clear dependency structures. Some don't have clear dependency structures. Some have better documentation and better focus than others.
If there is to be a merging of code together, then I propose we continue the project where the ZMI code is merged into one or a few packages. We can also investigate merging 2 or 3 packages together where it seems to make sense, or simply moving code between packages (some code needs to go down the dependency list, some up).
Instead of thinking about it this way, could we think about it as *deflating* the current set of zope.* packages that do not currently have a meaningful life of their own into a single setuptools distribution named "ZTK". This package would include most everything in zope.app*, plus things like zope.server, zope.publisher, and other things that just aren't useful outside of Zope-the-appserver, or which currently just depend on "too much".
-1
This would make it a lot harder to:
* clean up dependency relationships with the goal of creating more reusable code. We'd all hide them in a sumo ball again.
* get rid of bits we *don't want anymore*. If I need anything in a sumo package I'd need *all* of it.
* override individual packages with newer versions
* we've done a lot of refactoring recently trying to separate the UI from packages. This is done by creating a *new* package, leaving the old package behind. We can do this, test this and release this package-by-package now.
Over time, we'd tease out the dependencies of packages that live in the ZTK distribution, making them useful outside without any dependency on the ZTK. The names of these packages could be arbitrary (they wouldn't need to retain their old "zope.*" names). Some backwards dependency shims would be added to the ZTK to prevent old imports from breaking, and the ZTK distribution would then just have a dependency on the thing that got broken out.
I don't like the attempt to redefine what the ZTK means to a giant ball of Python packages. That's implying that, say, zope.component is *not* in the ZTK. That's wrong.
Why generate a whole lot of work for ourselves getting from where we are now to here? We've learned how to work with the current situation in 2007 already.
I'm thinking that this would simplify things greatly for people who want to be consumers of "truly independent" Zope packages. There'd be exactly N of them available for download (where N is much less than 100, more like 20), plus the ZTK, which would have the rest of the pile in it over time.
I don't see why a big package would "simplify things greatly" for you or anyone else.
If someone wanted to use a forked version of a package that lived in the ZTK distribution, they'd either do so by teasing out the dependencies and making it "truly independent" or they'd just reroll a custom version of the entire ZTK distribution.
And that's easier than the current situation how? Are you really proposing we destroy the dependency information we've already teased out and then make people do the work again?
Does this make any sense?
Not a lot in my book.
I think an important reason why there's so much awareness of dependency issues in the Zope world now (and effort spent on it) is precisely because we released our separate packages and can see the dependency information clearly.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On 5/12/09 4:44 AM, Patrick Gerken wrote:
Hi,
I agree with the package inflation. It makes people nervous of the complexity. But putting everything back into a pile of mud sounds like going back from where we came, and I this situation makes people even more nervous when they want to look into zope. As far as I know, the big dependency clean up is not finished yet, no? Maybe it might be better to do that first. With a clean dependency graph, its easier to see, which set of packages could be merged into one package.
I don't think there will ever be a point where it's "finished"; at least not in any time frame in which Zope is still relevant at the end. Sure, we can keep the current setuptools distributions and keep pulling apart their respective dependencies forever, but by the time it's all over, it just won't matter anyway; folks will be happily using "Django 3000" or "Pylons 4", which will have recreated all the features we teased out. - C
About the configurability of all the subpackages, I remember a proposal long time ago to reduce the number of zcml directives. This time I am in favor of inventing some zcml directives with some care. But before doing that, it might be better to write two or free easy tutorials about how to write custom zcml directives and grokkers. That would result in people writing more custom zcml directives and grokkers and get the hang of it. With the a much bigger userbase of writers of custom zcml directives and grokkers, the community as a whole will get a much better idea of how to use zcml, and I think gaining that community sense first will result in some high quality new general usable zcml directives. At the moment, and with my limited view, it looks like most zope ppl really don't know how to write custom zcml directives.
Just my 0.02€
Best regards,
Patrick
On Tue, May 12, 2009 at 09:08, Chris McDonough <chrism@plope.com <mailto:chrism@plope.com>> wrote:
I realize now that I've neglected to give sufficient reasoning for why less granular packaging would be a good thing.
I've noticed that there's a common theme in Zope development, software, and packages that I can only describe as "power law development" or "turtles all the way down". It's a bit of an antipattern, unfortunately.
I'll provide an example by way of Zope-3-the-appserver. In an application that uses Zope-3-the-appserver, many individual subframeworks will be used. For example, there is a traversal subframework, a security subframework, a cataloging subframework, and so on.
Each of these subframeworks acts as a logical unit, and through the magic of the component architecture, each can be replaced wholesale by registering some adapter. However, each of these subframeworks tend to also have settings that can be configured. For example, individual traversal steps for certain types of objects can be overridden by registering an adapter that *configures* the subframework. In the case of Zope 3, we have a traversal situation where the larger traversal subframework can either be replaced wholesale via an adapter registration or extended piecemeal via other adapter registrations.
The problem is that the mechanism to *replace* the subframework is the same as the mechanism to *configure* it (both are done via adapter registration, sometimes even in the same file). This is theoretically fine. But in reality, it's tremendously hard for someone just walking up to a complex system like Zope 3 to discern adapter registrations that replace subsystems from those which merely configure subsystems. An inability to discern the difference leads to situations where people just "don't get the joke" and try to wiggle wires to configure-to-death a existing subsystem that's clearly suboptimal for their use case instead of just replacing it wholesale with a much simpler custom policy. They just don't know it was engineered to be replaced. So they keep adding more configuration code to the existing subframework to handle various 1% edge cases. Often this code makes the subframework tremendously complex, and the subframework grows inappropriate dependencies along the way. *Sometimes* the situation gets so confusing for a new user, they just quit and go use something else.
This is a pattern that happens over and over again in Zope development. In my personal opinion, the original error was trying to make the subframework configurable at all. Instead, the subframework should be replaceable easily, but it should itself be relatively rigid. At very least, for subframeworks that really do require extra configuration (should be very few), this configuration should be done via highly specialized ZCML directives (or grokkers), as opposed to some very general adapter registration that can't be easily discerned from other adapter registrations by a newbie.
If the subframeworks were more rigid (but replaceable), the *intent* of the subframework author could be more easily discerned, and fewer people would fall into the trap of adding more configuration code to a subframework instead of just replacing it entirely. And fewer people would just walk away in frustration.
What does this have to do with packaging? Well, currently, there's a dizzying number of packages that make up "the ZTK" (nee "Zope 3"). Each of these packages is a pure peer of all others in a PyPI listing with no real way to get a sense of their relative importance other than performing a linear audit. Even if a user *does* do a linear search of all of them, it's still awful hard to discern for some new user which ones are "important", and which ones just happen to exist by some inequity of history without trying to install it. The user needs to gain some holistic knowledge of the system in order to discern the important bits from these historical inequities.
Most new users understandably just walk away from *all* Zope packages before they gain this knowledge; it's just too hard for them to tell the difference between the truly important and reusable bits and the stuff that just happens to be packaged up and released but which is useless outside of some highly specific context. In effect, we just don't communicate *intent* very effectively in our current packaging structure.
In my opinion, this is why a lot of Python developers who are otherwise very smart have given up on trying to use Zope packages. The time required to figure out which ones are useful and which ones aren't is just too high. It's way "easier" for them to write them all off wholesale and just write what they need from scratch or use somebody else's software. Good developers tend to like small, useful libraries and frameworks.
We can ameliorate the situation in a few ways:
- We can reduce the number of distributions.
- We can make each current Zope package distribution independently useful.
My suggestion is that we do the former first in order to communicate *current intent* (the "state of reality right now"). We can do the latter after this in pieces, hopefully aided by some new developers we've picked up by making it easier to find useful stuff.
Once we deflate our current set of packages down to a reasonable number, the packages listed in PyPI will immediately start to reflect "the state of reality right now". As a result, we'll hopefully be able to get some new blood in the form of new developers that use the smaller bits outside Zope to help us tease the truly independent pieces out of the larger pile. If we do this, at no time after the deflation will PyPI listings ever as badly advertise "the state of reality" as it is advertised right now, and the community will hopefully again start to grow.
- C
On 5/11/09 11:11 AM, Martijn Faassen wrote: > Hey, > > Chris McDonough wrote: > > > Instead, I have argued for promoting packages that have some life of > > their own (independent of the rest of the pile) into subprojects that > > have their own release cycles. > > > Then outside projects such as Plone and Grok could depend on > > independent versions of such packages, giving them slightly more > > flexibility than requiring a "version of the ZTK". > > We already have that flexibility today. To me, the utility of a release > of version numbers in the ZTK does not at all exclude the potential to > evolve the packages to more independent sub-projects. > >> Given that this suggestion has been met with skepticism, let me try another >> tact. > > I think that's an inaccurate description of the response you got. I'm > quite positive about trying to give as many packages as possible a life > of their own. I don't think you got anyone else arguing against this > point of view. > > I'm also quite positive that some packages are: > > * useful as independently distributed packages > > * only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they > depend on a significant set of Zope packages. > > I'd like to get out of this paradigm: > > * the Zope packages are independent sub-projects. > > * therefore we cannot distribute a list of versions that work best together. > > And this one: > > * if we distribute a KGS of anything > > * packages in that anything aren't independently reusable automatically > and should be merged into a ball. > > I'd also like to get out of the following paradigm: > > * the Zope packages are not independently reusable yet > > * therefore we should distribute them all together > > We're in a grey area. Some package are here, some packages are there, > some are in between. Some packages build on other packages, but have > clear dependency structures. Some don't have clear dependency > structures. Some have better documentation and better focus than others. > > If there is to be a merging of code together, then I propose we continue > the project where the ZMI code is merged into one or a few packages. We > can also investigate merging 2 or 3 packages together where it seems to > make sense, or simply moving code between packages (some code needs to > go down the dependency list, some up). > >> Instead of thinking about it this way, could we think about it as >> *deflating* the current set of zope.* packages that do not currently have a >> meaningful life of their own into a single setuptools distribution named "ZTK". >> This package would include most everything in zope.app*, plus things like >> zope.server, zope.publisher, and other things that just aren't useful outside of >> Zope-the-appserver, or which currently just depend on "too much". > > -1 > > This would make it a lot harder to: > > * clean up dependency relationships with the goal of creating more > reusable code. We'd all hide them in a sumo ball again. > > * get rid of bits we *don't want anymore*. If I need anything in a sumo > package I'd need *all* of it. > > * override individual packages with newer versions > > * we've done a lot of refactoring recently trying to separate the UI > from packages. This is done by creating a *new* package, leaving the old > package behind. We can do this, test this and release this > package-by-package now. > >> Over time, we'd tease out the dependencies of packages that live in the ZTK >> distribution, making them useful outside without any dependency on the ZTK. The >> names of these packages could be arbitrary (they wouldn't need to retain their >> old "zope.*" names). Some backwards dependency shims would be added to the ZTK >> to prevent old imports from breaking, and the ZTK distribution would then just >> have a dependency on the thing that got broken out. > > I don't like the attempt to redefine what the ZTK means to a giant ball > of Python packages. That's implying that, say, zope.component is *not* > in the ZTK. That's wrong. > > Why generate a whole lot of work for ourselves getting from where we are > now to here? We've learned how to work with the current situation in > 2007 already. > >> I'm thinking that this would simplify things greatly for people who want to be >> consumers of "truly independent" Zope packages. There'd be exactly N of them >> available for download (where N is much less than 100, more like 20), plus the >> ZTK, which would have the rest of the pile in it over time. > > I don't see why a big package would "simplify things greatly" for you or > anyone else. > >> If someone wanted >> to use a forked version of a package that lived in the ZTK distribution, they'd >> either do so by teasing out the dependencies and making it "truly independent" >> or they'd just reroll a custom version of the entire ZTK distribution. > > And that's easier than the current situation how? Are you really > proposing we destroy the dependency information we've already teased out > and then make people do the work again? > >> Does this make any sense? > > Not a lot in my book. > > I think an important reason why there's so much awareness of dependency > issues in the Zope world now (and effort spent on it) is precisely > because we released our separate packages and can see the dependency > information clearly. > > Regards, > > Martijn > > _______________________________________________ > Zope-Dev maillist - Zope-Dev@zope.org <mailto:Zope-Dev@zope.org> > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) >
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org <mailto:Zope-Dev@zope.org> http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Hey Chris, Chris McDonough wrote:
On 5/12/09 4:44 AM, Patrick Gerken wrote: [snip] I don't think there will ever be a point where it's "finished"; at least not in any time frame in which Zope is still relevant at the end. Sure, we can keep the current setuptools distributions and keep pulling apart their respective dependencies forever, but by the time it's all over, it just won't matter anyway; folks will be happily using "Django 3000" or "Pylons 4", which will have recreated all the features we teased out.
Such skepticism. Please consult the following (non-reduced) dependency graphs (of released packages): http://startifact.com/depgraphs/ I've picked a few high-level components like container and catalog. They depend on a lot, yes. Too much, yes, there are bits that can still be teased out from that graph (in particular I think we should make the container stop depending on the publisher). But they also depend on a lot because they're fairly high-level components that do quite a few things. I realize that many of these packages are not useful for a random Python developer. But I don't believe we have to ensure all our packages are useful in that way. We just have to create more of them. I think all of the ZMI stuff should either be eliminated or consolidated. That would allow us to lose zope.app.* packages. Remove them from PyPI, no, though - we already crossed that bridge and can't break everybody's code. I think these high-level components and a few more is what we can at least base a future release of Grok around. (with a compat package that pulls in a lot of the zope.app. packages to make sure the existing code doesn't break). The dependency graph will still be huge, but it won't be as crazy as it is with the current Grok release. Regards, Martijn
Chris McDonough <chrism <at> plope.com> writes:
This is a pattern that happens over and over again in Zope development. In my personal opinion, the original error was trying to make the subframework configurable at all. Instead, the subframework should be replaceable easily, but it should itself be relatively rigid. At very least, for subframeworks that really do require extra configuration (should be very few), this configuration should be done via highly specialized ZCML directives (or grokkers), as opposed to some very general adapter registration that can't be easily discerned from other adapter registrations by a newbie.
I agree very much that the various default Zope components could be much more rigid; this would make it easier to understand the application flow and better realize when to subclass or replace. If rigid means less configurable, then perhaps components could be made flexible by better adapting to the objects given to them, e.g. use the interfaces system to tell what kind of functionality objects inhibit and provide flexibility through this route. As such, instead of attempting to look up a custom traverser for an object, ask the object "do you provide your own traversal logic?". Instead of components being primarily pluggable, they could be adaptive. \malthe
On Tue, May 12, 2009 at 03:08:05AM -0400, Chris McDonough wrote:
Once we deflate our current set of packages down to a reasonable number, the packages listed in PyPI will immediately start to reflect "the state of reality right now". As a result, we'll hopefully be able to get some new blood in the form of new developers that use the smaller bits outside Zope to help us tease the truly independent pieces out of the larger pile. If we do this, at no time after the deflation will PyPI listings ever as badly advertise "the state of reality" as it is advertised right now, and the community will hopefully again start to grow.
I agree with a lot of what you wrote, but I don't understand this last bit. All the currently released packages will continue to live on pypi more or less forever, no? To paraphrase a wise man, "Releasing a package means always having to say you're sorry" ;-) -- Paul Winkler http://www.slinkp.com
On 5/12/09 12:27 PM, Paul Winkler wrote:
On Tue, May 12, 2009 at 03:08:05AM -0400, Chris McDonough wrote:
Once we deflate our current set of packages down to a reasonable number, the packages listed in PyPI will immediately start to reflect "the state of reality right now". As a result, we'll hopefully be able to get some new blood in the form of new developers that use the smaller bits outside Zope to help us tease the truly independent pieces out of the larger pile. If we do this, at no time after the deflation will PyPI listings ever as badly advertise "the state of reality" as it is advertised right now, and the community will hopefully again start to grow.
I agree with a lot of what you wrote, but I don't understand this last bit. All the currently released packages will continue to live on pypi more or less forever, no? To paraphrase a wise man, "Releasing a package means always having to say you're sorry" ;-)
I don't think so. People would need to change their code to depend on the big package, and the smaller packages would cease to have a life of their own. Having both the smaller packages and some bigger package that also contains it and needing to maintain both truly would be the worst of all possible worlds. - C
On Tue, May 12, 2009 at 12:36:06PM -0400, Chris McDonough wrote:
On 5/12/09 12:27 PM, Paul Winkler wrote:
I agree with a lot of what you wrote, but I don't understand this last bit. All the currently released packages will continue to live on pypi more or less forever, no? To paraphrase a wise man, "Releasing a package means always having to say you're sorry" ;-)
I don't think so. People would need to change their code to depend on the big package, and the smaller packages would cease to have a life of their own. Having both the smaller packages and some bigger package that also contains it and needing to maintain both truly would be the worst of all possible worlds.
I must be dense today because I still don't understand what "cease to have a life" means. Remove them from pypi? How would you avoid breaking all the current software in the world that currently depends on all those distributions? -- Paul Winkler http://www.slinkp.com
On 5/12/09 12:46 PM, Paul Winkler wrote:
On Tue, May 12, 2009 at 12:36:06PM -0400, Chris McDonough wrote:
On 5/12/09 12:27 PM, Paul Winkler wrote:
I agree with a lot of what you wrote, but I don't understand this last bit. All the currently released packages will continue to live on pypi more or less forever, no? To paraphrase a wise man, "Releasing a package means always having to say you're sorry" ;-) I don't think so. People would need to change their code to depend on the big package, and the smaller packages would cease to have a life of their own. Having both the smaller packages and some bigger package that also contains it and needing to maintain both truly would be the worst of all possible worlds.
I must be dense today because I still don't understand what "cease to have a life" means. Remove them from pypi? How would you avoid breaking all the current software in the world that currently depends on all those distributions?
If your package depends on zope.app.publisher, you get *78* eggs. Change that install_requires line in your package to "ZTK" and you get the same software. OTOH, packages that rely on things that are *truly reusable* like zope.interface, and so on will need to do nothing; those packages will continue to have a life of their own. - C
Hey, Chris McDonough wrote: [snip]
If your package depends on zope.app.publisher, you get *78* eggs.
63 eggs these days, by my measurement. Still far too many. I think with some effort we can chop off quite a few more. Look here at the main cycles in that graph (this is the cause of a lot being pulled in that shouldn't be): http://startifact.com/depgraphs/zope_app_publisher_cycles.svg It looks we need to pull more out of zope.app.component somewhere else so zope.app.publisher doesn't need to depend on zope.formlib anymore. Probably ZMI stuff holding things in. The other tricky line is the dependency of zope.app.publisher on zope.app.publication.
Change that install_requires line in your package to "ZTK" and you get the same software. OTOH, packages that rely on things that are *truly reusable* like zope.interface, and so on will need to do nothing; those packages will continue to have a life of their own.
I'm worried that one person's truly reusable is another person's package that is really useless without a huge amount of buy-in. Regards, Martijn
Chris McDonough wrote: [snip extending configuration patterns instead of replacing wholesale]
Often this code makes the subframework tremendously complex, and the subframework grows inappropriate dependencies along the way. *Sometimes* the situation gets so confusing for a new user, they just quit and go use something else.
I am curious about your use of the word "often". I think what you describe has happened for the example you give, namely the publication/traversal stack. I think it also has happened to a certain extent for the user authentication story. I'd like to identify other such nasty grown subframeworks and you seem to know about a few more. I think in part this happened because there wasn't enough documentation about wholesale replacement patterns (and not enough documentation period), and in part it happened because the APIs or component relationships were just not designed as well as we'd like (as you remark below), and in part it happened because people often don't *want* to do a wholesale replacement but exactly the kind of "only reconfigure this bit" you describe
This is a pattern that happens over and over again in Zope development. In my personal opinion, the original error was trying to make the subframework configurable at all. Instead, the subframework should be replaceable easily, but it should itself be relatively rigid. At very least, for subframeworks that really do require extra configuration (should be very few), this configuration should be done via highly specialized ZCML directives (or grokkers), as opposed to some very general adapter registration that can't be easily discerned from other adapter registrations by a newbie.
I agree that specialized ZCML directives and grokkers are a good thing. I think we had too much of a tendency to get away from those and instead generalize everything. I think we should define clear Python APIs for particular configuration concepts, not just ZCML directives or grokkers. In turn the ZCML/grokker implementations can use this Python API and be really minimal in themselves. That said, I do think there's implementation value in a general registry. It's just not a great API in most cases.
If the subframeworks were more rigid (but replaceable), the *intent* of the subframework author could be more easily discerned, and fewer people would fall into the trap of adding more configuration code to a subframework instead of just replacing it entirely. And fewer people would just walk away in frustration.
While I understand the sentiment and I agree with the general advice, I again must note that sometimes it's useful to be able to configure a subframework. "everything as normal with just a few tweaks" is after all a good way forward in many cases. The idea you are promoting is "clearly focused frameworks". They do one thing well, and are configurable in a few ways and clearly replaceable too.
What does this have to do with packaging? Well, currently, there's a dizzying number of packages that make up "the ZTK" (nee "Zope 3"). Each of these packages is a pure peer of all others in a PyPI listing with no real way to get a sense of their relative importance other than performing a linear audit.
One way to get some hints is to look at a dependency diagram and see which ones are lower down in the stack.
Even if a user *does* do a linear search of all of them, it's still awful hard to discern for some new user which ones are "important", and which ones just happen to exist by some inequity of history without trying to install it. The user needs to gain some holistic knowledge of the system in order to discern the important bits from these historical inequities.
Most new users understandably just walk away from *all* Zope packages before they gain this knowledge; it's just too hard for them to tell the difference between the truly important and reusable bits and the stuff that just happens to be packaged up and released but which is useless outside of some highly specific context. In effect, we just don't communicate *intent* very effectively in our current packaging structure.
Sure, agreed.
In my opinion, this is why a lot of Python developers who are otherwise very smart have given up on trying to use Zope packages. The time required to figure out which ones are useful and which ones aren't is just too high. It's way "easier" for them to write them all off wholesale and just write what they need from scratch or use somebody else's software. Good developers tend to like small, useful libraries and frameworks.
Agreed again.
We can ameliorate the situation in a few ways:
- We can reduce the number of distributions.
- We can make each current Zope package distribution independently useful.
My suggestion is that we do the former first in order to communicate *current intent* (the "state of reality right now"). We can do the latter after this in pieces, hopefully aided by some new developers we've picked up by making it easier to find useful stuff.
In order for your plan to work I think you'd need two things ideally: * remove a lot of the released packages from pypi. This is a no-go. People depend on these in a massive way. * at least move around the existing packages in svn.zope.org to some kind of "random crap" area so people aren't confronted with them.
Once we deflate our current set of packages down to a reasonable number, the packages listed in PyPI will immediately start to reflect "the state of reality right now". As a result, we'll hopefully be able to get some new blood in the form of new developers that use the smaller bits outside Zope to help us tease the truly independent pieces out of the larger pile. If we do this, at no time after the deflation will PyPI listings ever as badly advertise "the state of reality" as it is advertised right now, and the community will hopefully again start to grow.
How do you propose we reduce the PyPI listing without breaking a lot of code? The other practical issue is community-oriented: we have been able to break so many dependencies and have an awareness of these issues precisely because packages *were* split up. Meanwhile we've invested quite a bit of effort in making existing Zope packages at least *depend* more or less logically on other packages. That doesn't mean they're always, or even typically, independently useful in the sense you want: they're useful only if you buy into a lot of the Zope story. I hope we can reduce the buy-in more once we explode the publisher, but it'll always be that way to a certain extent. A package that depends on 10 other packages might not be considered "independently useful" by many, as it assumes quite a bit of the framework to be present. But that doesn't mean it's useless as an independent package. Regards, Martijn
On Wed, May 13, 2009 at 15:29, Martijn Faassen <faassen@startifact.com>wrote:
Chris McDonough wrote:
We can ameliorate the situation in a few ways:
- We can reduce the number of distributions.
- We can make each current Zope package distribution independently useful.
My suggestion is that we do the former first in order to communicate *current intent* (the "state of reality right now"). We can do the latter after this in pieces, hopefully aided by some new developers we've picked up by making it easier to find useful stuff.
In order for your plan to work I think you'd need two things ideally:
* remove a lot of the released packages from pypi. This is a no-go. People depend on these in a massive way.
Wouldn't it be possible to put them on a dedicated pypi? pypi.support.zope.com? I start being scared of using pypi. Best regards, Patrick
Hey, Patrick Gerken wrote: [snip]
Wouldn't it be possible to put them on a dedicated pypi? pypi.support.zope.com <http://pypi.support.zope.com>?
Yes, but not without breaking backwards compatibility with a lot of released buildout.cfg files, and having to arrange our own mirroring services and so on. (and definitely not on zope.com. zope.org, yes)
I start being scared of using pypi.
Why? Regards, Martijn
On Wed, May 13, 2009 at 16:23, Martijn Faassen <faassen@startifact.com>wrote:
I start being scared of using pypi.
Why?
Because I can imagine case where I would like to remove a package. I did just that in a totally unrelated piece of software where I wrote an extension. After a while I realized, that another extension was not all that different, so I refactored the other extension and now it contains my features as a configuration option. I was just made clear, that this might not work very well in pypi. Scared is a very strong word for what I intended to say. I do not want to abuse you here as a support forum for pypi, but would it be possible to publish a new release, but without providing the distribution files? The intention would be, not the break the installations with pinned versions, but breaking attempts to get the newest versions, because the distribution files would be missing. The people might then check the webpage of pypi, where the readme would state that the package has been discontinued and can be found in package x.y. As a next step, pypi might offer some way to ignore these packages in searches. If this would work, it could be a way to reduce packages over time (Provided the dependency reduction shows that they don't need a live on their own). Best regards, Patrick
Hi there, Patrick Gerken wrote: [snip]
I do not want to abuse you here as a support forum for pypi, but would it be possible to publish a new release, but without providing the distribution files? The intention would be, not the break the installations with pinned versions,
That would never happen anyway, unless you removed existing releases.
but breaking attempts to get the newest versions, because the distribution files would be missing.
I believe buildout has an option to prefer final releases. If you release a non-final release (beta, alpha) and people use this option they won't get your version even if they do not use a list of pinned versions. But that doesn't look like what you want.
The people might then check the webpage of pypi, where the readme would state that the package has been discontinued and can be found in package x.y. As a next step, pypi might offer some way to ignore these packages in searches.
That's like what Chris suggested with a potential supersedes and replaces metadata for packages in PyPI. PyPI doesn't really support this yet. I think you could simply release a package that issues a deprecation warning when it's in use. That way people will know to look somewhere else. Note that there's also work done on the test runner (don't know the state, Christian Theune is working on it) that can report on improved import locations for a codebase (if you were importing from zope.app.component something that's now in zope.site, it will tell you). Regards, Martijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Patrick Gerken wrote:
I start being scared of using pypi.
You should be *very* afraid of depending on PyPI for softare rolled into production. PyPI is only really useful for discovery during development. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKCu+F+gerLs4ltQ4RAnniAKDAKw0BVoBlPY/9Un3MwzRL+1fUIwCcD4tu cn4NtZQnlQbTesBeZiA6Kjs= =niRy -----END PGP SIGNATURE-----
On May 13, 2009, at 12:04 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Patrick Gerken wrote:
I start being scared of using pypi.
I wonder why.
You should be *very* afraid of depending on PyPI for softare rolled into production.
Why do you think he should be afraid? Jim -- Jim Fulton Zope Corporation
On 13.05.09 18:38, Jim Fulton wrote:
On May 13, 2009, at 12:04 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Patrick Gerken wrote:
I start being scared of using pypi.
I wonder why.
You should be *very* afraid of depending on PyPI for softare rolled into production.
Why do you think he should be afraid? Packages or releases might disappear - intentionally or unintentionally - in both cases a buildout with fixed pinned version may fail.
Andreas
On May 13, 2009, at 12:41 PM, Andreas Jung wrote:
On 13.05.09 18:38, Jim Fulton wrote:
On May 13, 2009, at 12:04 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Patrick Gerken wrote:
I start being scared of using pypi.
I wonder why.
You should be *very* afraid of depending on PyPI for softare rolled into production.
Why do you think he should be afraid? Packages or releases might disappear - intentionally or unintentionally - in both cases a buildout with fixed pinned version may fail.
That's a minor issue at this point, because: - We now know not to remove releases. - If you are using something in production, you should archive the necessary source releases, using a tool like zc.sourcerelease. IOW, you shouldn't do production deployments using a dynamic assembly mechanism. Jim -- Jim Fulton Zope Corporation
On 13.05.09 18:44, Jim Fulton wrote:
On May 13, 2009, at 12:41 PM, Andreas Jung wrote:
On 13.05.09 18:38, Jim Fulton wrote:
On May 13, 2009, at 12:04 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Patrick Gerken wrote:
I start being scared of using pypi.
I wonder why.
You should be *very* afraid of depending on PyPI for softare rolled into production.
Why do you think he should be afraid? Packages or releases might disappear - intentionally or unintentionally - in both cases a buildout with fixed pinned version may fail.
That's a minor issue at this point, because:
- We now know not to remove releases.
Jup, we know but some package maintainers outside the Zope world don't.
- If you are using something in production, you should archive the necessary source releases, using a tool like zc.sourcerelease.
One option or Tres solution: having a dedicated local index on a per-project basis or a local egg server or a (partial) local PyPI mirror. Andreas
On May 13, 2009, at 12:47 PM, Andreas Jung wrote:
On 13.05.09 18:44, Jim Fulton wrote:
On May 13, 2009, at 12:41 PM, Andreas Jung wrote:
On 13.05.09 18:38, Jim Fulton wrote:
On May 13, 2009, at 12:04 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Patrick Gerken wrote:
I start being scared of using pypi.
I wonder why.
You should be *very* afraid of depending on PyPI for softare rolled into production.
Why do you think he should be afraid? Packages or releases might disappear - intentionally or unintentionally - in both cases a buildout with fixed pinned version may fail.
That's a minor issue at this point, because:
- We now know not to remove releases.
Jup, we know but some package maintainers outside the Zope world don't.
- If you are using something in production, you should archive the necessary source releases, using a tool like zc.sourcerelease.
One option or Tres solution: having a dedicated local index on a per- project basis or a local egg server or a (partial) local PyPI mirror.
That's an option. It takes a lot of work. I don't have a problem with people doing that. I just don't like this meme of "fearing" pypi. Jim -- Jim Fulton Zope Corporation
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
- We now know not to remove releases.
Not everybody does: I've seen folks *recently* re-upload a changed release without bumping the version number; and "we" is a much narrower set than the set of all PyPI maintainers.
- If you are using something in production, you should archive the necessary source releases, using a tool like zc.sourcerelease.
IOW, you shouldn't do production deployments using a dynamic assembly mechanism.
Which is exaclt what I said:
You should be *very* afraid of depending on PyPI for softare rolled into production.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKCwBE+gerLs4ltQ4RAmdnAKDKM8hJietF8FVHZZJ8sn2iP7HFRwCfUt8s loL+AU9xuY5x1vl/D43akGg= =uHh7 -----END PGP SIGNATURE-----
On May 13, 2009, at 1:15 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jim Fulton wrote:
- We now know not to remove releases.
Not everybody does: I've seen folks *recently* re-upload a changed release without bumping the version number; and "we" is a much narrower set than the set of all PyPI maintainers.
Well, at some point you have to take into account the skills of the maintainers when considering whether to use a package. I personally haven't been burned by this, so I hardly think this is a cause for "fear".
- If you are using something in production, you should archive the necessary source releases, using a tool like zc.sourcerelease.
IOW, you shouldn't do production deployments using a dynamic assembly mechanism.
Which is exaclt what I said:
You should be *very* afraid of depending on PyPI for softare rolled into production.
I don't consider the 2 statements to be the same. I had a feeling that that was what you meant, at least on some level. I use PyPI when creating source releases. I use source releases (actually binary rpms built from source rpms built from source releases) for deployment. The impression I think you're giving is that people should avoid PyPI and need to build their own indexes and I just don't agree with that. Jim -- Jim Fulton Zope Corporation
On 13.05.09 20:16, Jim Fulton wrote:
I don't consider the 2 statements to be the same. I had a feeling that that was what you meant, at least on some level.
I use PyPI when creating source releases. I use source releases (actually binary rpms built from source rpms built from source releases) for deployment.
The impression I think you're giving is that people should avoid PyPI and need to build their own indexes and I just don't agree with that.
I have to second Tres. If your deployment is critical, one should not depend on an external dependency (like PyPI) that might change (changing or disappearing release - we had such an issue several times) or that may become unavailable (that's why we're having PyPI mirror (internal and external). Andreas
Hey Chris, What about the following alternative suggestion to alleviate some of the underlying issues you point out. I agree we are signaling badly which packages are interesting to newcomers and outsiders and which packages aren't. In part we've already done the damage with the packages in the "zope.*" namespace. I think we shouldn't try to put humpty-dumpty back together again marketing-wise by removing those packages a few years after their release, and whether this is worth the effort (and I see negative drawbacks to doing so). What we can do is start to clearly indicate on top of a package's documentation whether it's intended to be independently reusable outside of a Zope context or not. We should do this on pypi, but we should also back this up by writing narrative documentation for those packages we *do* think are independently reusable by a wide audience. I think this should be done by starting 'doc' directories in those packages and putting in sphinx-based documentation. The next step would be to go to our "non-reusable" packages and start writing narrative docs for that, ideally with example projects. If we pick a few likely candidates and do some more refactoring we may be able to salvage them into reusable packages and we can declare them as such. As indications I propose: "This package is intended to be independently reusable in any Python project. It is maintained by the Zope Toolkit project." (with hopefully appended: "For more documentation on this see <narrative docs>.") "This package is at present not reusable without depending on a large chunk of the Zope Toolkit and its assumptions. It is maintained by the Zope Toolkit project." We can also add 'reusable' to the metadata tags in PyPI in addition to this. Regards, Martijn
The implementation details are over my head, but what SchoolTool needs is a middle ground between one big package and a giant pile of little ones, because our primary deployment strategy is via Linux distribution packaging, in Debian/Ubuntu in particular. Currently, Fabio maintains an official monolithic package for Debian, and we generate a giant pile of .debs from eggs and keep them in a PPA. Because there is administrative overhead with every package, even if we can eventually come to consensus on the desirability or pushing 70 tiny packages into the Debian pipeline, it may never be practical. So a single, community-recognized ZTF that would be the foundation of manageable packaging for Linux distros would be a huge win for us. --Tom
Tom Hoffman wrote:
The implementation details are over my head, but what SchoolTool needs is a middle ground between one big package and a giant pile of little ones, because our primary deployment strategy is via Linux distribution packaging, in Debian/Ubuntu in particular.
Currently, Fabio maintains an official monolithic package for Debian, and we generate a giant pile of .debs from eggs and keep them in a PPA. Because there is administrative overhead with every package, even if we can eventually come to consensus on the desirability or pushing 70 tiny packages into the Debian pipeline, it may never be practical.
So a single, community-recognized ZTF that would be the foundation of manageable packaging for Linux distros would be a huge win for us.
That's a good point. I can imagine this giant .deb is created from a combination of a KGS list of versions and downloading them together. But having a clearer picture of which packages we truly mark as reusable outside of Zope would also help inform the decisions on what to package for debian as seperate packages, and what to package as a whole. Regards, Martijn
On 5/13/09 10:34 AM, Martijn Faassen wrote:
Hey Chris,
What about the following alternative suggestion to alleviate some of the underlying issues you point out.
I agree we are signaling badly which packages are interesting to newcomers and outsiders and which packages aren't.
In part we've already done the damage with the packages in the "zope.*" namespace. I think we shouldn't try to put humpty-dumpty back together again marketing-wise by removing those packages a few years after their release, and whether this is worth the effort (and I see negative drawbacks to doing so).
I'd hope you'd agree that given a perfect world where packaging structure backwards compatibility was not a concern: - The original distribution structure was a mistake. - Changing it would be a bugfix. That said, given your other arguments in prior mails today, I'll give up agitating for any packaging changes on this maillist, because it's pretty much impossible to argue against the article of faith that there is some presumed majority of thousands-of-people-who-depend-on-those-packages-as-distributed-now-and-whom-will-forever-want-to-do-so-and-whose-world-will-explode-if-we-take-them-away. Maybe when setuptools grows "provides" and "obsoletes" setup parameters (ala RPM), this particular problem can be solved better technically.
What we can do is start to clearly indicate on top of a package's documentation whether it's intended to be independently reusable outside of a Zope context or not. We should do this on pypi, but we should also back this up by writing narrative documentation for those packages we *do* think are independently reusable by a wide audience. I think this should be done by starting 'doc' directories in those packages and putting in sphinx-based documentation.
The next step would be to go to our "non-reusable" packages and start writing narrative docs for that, ideally with example projects. If we pick a few likely candidates and do some more refactoring we may be able to salvage them into reusable packages and we can declare them as such.
As indications I propose:
"This package is intended to be independently reusable in any Python project. It is maintained by the Zope Toolkit project."
(with hopefully appended: "For more documentation on this see<narrative docs>.")
"This package is at present not reusable without depending on a large chunk of the Zope Toolkit and its assumptions. It is maintained by the Zope Toolkit project."
We can also add 'reusable' to the metadata tags in PyPI in addition to this.
I think this is a reasonable workaround if the packaging structure does not change. Thanks for the responses, - C
Hey, Chris McDonough wrote: [snip]
I'd hope you'd agree that given a perfect world where packaging structure backwards compatibility was not a concern:
- The original distribution structure was a mistake.
- Changing it would be a bugfix.
I think we should've gone for an approach where we slowly peeled off independent packages from a big ball in iterative fashion, instead of the giant explosion we ended up with. (assuming the tools allow us to do so) Whether changing it back now would be a bug fix: I don't know, for two reasons: * we have the ability now to do fine-grained bugfix and feature releases of individual packages without having to coordinate all code. This of course is also a minus, sometimes. * more nebulous: I do find that the explosion, for all its flaws, helped us with identifying bad dependencies. Peeling off packages would allow us to do this too, however.
That said, given your other arguments in prior mails today, I'll give up agitating for any packaging changes on this maillist, because it's pretty much impossible to argue against the article of faith that there is some presumed majority of thousands-of-people-who-depend-on-those-packages-as-distributed-now-and-whom-will-forever-want-to-do-so-and-whose-world-will-explode-if-we-take-them-away.
meta: I don't like how you say that this is an article of faith, because you seem to imply that we're superstitious with this. Concretely I have quite a few codebases around that depend on the current package list being present. They'd stop working if we suddenly withdrew these packages from PyPI. I think there are quite a few others in the same position.
Maybe when setuptools grows "provides" and "obsoletes" setup parameters (ala RPM), this particular problem can be solved better technically.
Yes, something like that would probably help. [snip]
As indications I propose:
"This package is intended to be independently reusable in any Python project. It is maintained by the Zope Toolkit project."
(with hopefully appended: "For more documentation on this see<narrative docs>.")
"This package is at present not reusable without depending on a large chunk of the Zope Toolkit and its assumptions. It is maintained by the Zope Toolkit project."
We can also add 'reusable' to the metadata tags in PyPI in addition to this.
I think this is a reasonable workaround if the packaging structure does not change.
I'll start putting up a few of these notifications today. Regards, Martijn
On 5/13/09 1:22 PM, Martijn Faassen wrote:
That said, given your other arguments in prior mails today, I'll give up agitating for any packaging changes on this maillist, because it's pretty much impossible to argue against the article of faith that there is some presumed majority of thousands-of-people-who-depend-on-those-packages-as-distributed-now-and-whom-will-forever-want-to-do-so-and-whose-world-will-explode-if-we-take-them-away.
meta: I don't like how you say that this is an article of faith, because you seem to imply that we're superstitious with this.
No one on this list has yet claimed that such a repackaging would harm them irreparably personally. So I assume we're catering to the great silent majority. Without identifying the actual size of that group, we'll be unable to ever make any change because it might harm "someone". I consider this a pretty untenable place to be in over the long haul.
Concretely I have quite a few codebases around that depend on the current package list being present. They'd stop working if we suddenly withdrew these packages from PyPI. I think there are quite a few others in the same position.
You (in particular) could almost certainly keep these working by setting up a private index. Which you should probably do anyway, or at least be using zc.sourcerelease or so, for actual repeatability for important systems that need to keep running forever and cannot be changed. IOW, the union of these two sets is the null set: - people for whom repeatable builds is an absolute requirement - people who use PyPI to build their systems in production IMO, PyPI should be treated as a place to advertise your software; not to use as a must-be-up-all-the-time and must-give-me-a-repeatable-build-forever system.
We can also add 'reusable' to the metadata tags in PyPI in addition to this. I think this is a reasonable workaround if the packaging structure does not change.
I'll start putting up a few of these notifications today.
OK. - C
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris McDonough wrote:
On 5/13/09 10:34 AM, Martijn Faassen wrote:
Hey Chris,
What about the following alternative suggestion to alleviate some of the underlying issues you point out.
I agree we are signaling badly which packages are interesting to newcomers and outsiders and which packages aren't.
In part we've already done the damage with the packages in the "zope.*" namespace. I think we shouldn't try to put humpty-dumpty back together again marketing-wise by removing those packages a few years after their release, and whether this is worth the effort (and I see negative drawbacks to doing so).
I'd hope you'd agree that given a perfect world where packaging structure backwards compatibility was not a concern:
- The original distribution structure was a mistake.
- Changing it would be a bugfix.
That said, given your other arguments in prior mails today, I'll give up agitating for any packaging changes on this maillist, because it's pretty much impossible to argue against the article of faith that there is some presumed majority of thousands-of-people-who-depend-on-those-packages-as-distributed-now-and-whom-will-forever-want-to-do-so-and-whose-world-will-explode-if-we-take-them-away.
Maybe when setuptools grows "provides" and "obsoletes" setup parameters (ala RPM), this particular problem can be solved better technically.
What we can do is start to clearly indicate on top of a package's documentation whether it's intended to be independently reusable outside of a Zope context or not. We should do this on pypi, but we should also back this up by writing narrative documentation for those packages we *do* think are independently reusable by a wide audience. I think this should be done by starting 'doc' directories in those packages and putting in sphinx-based documentation.
The next step would be to go to our "non-reusable" packages and start writing narrative docs for that, ideally with example projects. If we pick a few likely candidates and do some more refactoring we may be able to salvage them into reusable packages and we can declare them as such.
As indications I propose:
"This package is intended to be independently reusable in any Python project. It is maintained by the Zope Toolkit project."
(with hopefully appended: "For more documentation on this see<narrative docs>.")
"This package is at present not reusable without depending on a large chunk of the Zope Toolkit and its assumptions. It is maintained by the Zope Toolkit project."
We can also add 'reusable' to the metadata tags in PyPI in addition to this.
I think this is a reasonable workaround if the packaging structure does not change.
I think we need to clarify terms / triage the sets of packages we are talking about: Truly reusable - -------------- Things like the "bicycle seat toolkit" group (zope.interface, zope.component, etc.). These things ideally already have in-package narrative docs, excellent test coverage, and minimal dependencies. In particular, they *cannot* depend on anything in one of the other sets. Wannabe reusable - ---------------- Candidates for promotion to the "truly reusable" set, but which aren't yet ready. E.g., they might be missing docs, or have a few remaining dependencies which need breaking, or their testing is incomplete. The first pass for identifying this set should be the dependecies of the trunks of Zope2 and Grok. A package *cannot* be in this set if it depends on any package the "never" set. Never-gonna-be reusable - ----------------------- One (or maybe more?) sets of packages which exist only to support a given application (e.g., the Zope3 ZMI). Each set should ideally be released as a "mudball" distribution: I will note that the eggified Zope2 is one of these. For the first cut, anything in zope.app would be in this pile. The goal would be for the ZTK to consist only of reusable packages, and for Zope2 and Grok to depend only on pacakges in the ZTK. By this set of criteria, the urgent priority for Zope2 would be to break the expliti dependency on the following packages: - zope.app.appsetup (Zope2.Startup, Zope.App.Startup) - zope.app.component (?) - zope.app.container (Products.Five.browser.adding) - zope.app.form (Products.Five.form.*) - zope.app.pagetemplate (Products.PageTemplates.Expressions, Products.Five.browser.pagetemplatefile, Products.Five.browser.metaconfigure) - zope.app.publication (ZPublisher.BaseRequest, Products.Five.component) - zope.app.publisher (ZPublisher.BaseRequest, Products.Five.browser.*, Products.Five.viewlet.metaconfigure, Products.Five.form.metaconfigure, Products.Five.fivedirectives) - zope.app.schema (Products.Five) And the transitive dependencies on these packages (some of : - zope.app.applicationcontrol (zope.traversing, zope.app.publication) - zope.app.basicskin (zope.app.form) - zope.app.debug (zope.app.testing) - zope.app.dependable (zope.container, zope.app.testing) - zope.app.exception (zope.app.publication) - zope.app.http (zope.app.publication) - zope.app.interface (zope.app.component) - zope.app.localpermission (zope.app.security) - zope.app.security (zope.viewlet, zope.traversing, zope.testbrowser, zope.app.*) - zope.app.testing (zope.viewlet, zope.container, zope.copypastemve, zope.error, zope.dublincore, zope.formlib, zope.traversing, zope.testbrowser, zope.site, zope.app.*) I'm not enough up on Grok to do a similar analysis there. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKCxDa+gerLs4ltQ4RAiIlAJ9o6ZYOiIxoLug25Rd+0OOch04nBQCeLVNm stlvpyUMz+OXGXQbsfPFjGQ= =ujuf -----END PGP SIGNATURE-----
Hey there, Tres Seaver wrote: [snip]
I think we need to clarify terms / triage the sets of packages we are talking about:
Sure, agreed, though I think we can already work with 'reusable' and 'not reusable' right now as hints to users. The 'not reusable' group consists of 'wannabe reusable' and 'implausible it'll ever be reusable'. I also expect we'll end up with some packages that have reasonable dependencies but still a *lot* of dependencies - does that mean they're reusable or not? If not, what if Grok still needs one? I'd be happy to see someone triage the existing set of packages in the categories Tres proposes.
- zope.app.container (Products.Five.browser.adding)
Should be easy enough to move to zope.container, hopefully.
- zope.app.form (Products.Five.form.*)
This actually isn't that bad in the reusability department; it's mostly a bunch of widgets. Not that it's the be all end all form library by a long shot, and most of our libraries shouldn't be depending on it, but it's not a bad thing to have it in the ZTK for now. [snip]
I'm not enough up on Grok to do a similar analysis there.
Grok is mostly split up into grokcore.* packages which list explicit dependencies in their setup.py. I think grokcore.view is holding in the most dependencies at the moment. Regards, Martijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martijn Faassen wrote:
Tres Seaver wrote:
I think we need to clarify terms / triage the sets of packages we are talking about:
Sure, agreed, though I think we can already work with 'reusable' and 'not reusable' right now as hints to users. The 'not reusable' group consists of 'wannabe reusable' and 'implausible it'll ever be reusable'.
I'm worried that if we don't target the "wannabes" now, we'll end up making a lot of work for ourselves *and* for people consuming the packages later.
I also expect we'll end up with some packages that have reasonable dependencies but still a *lot* of dependencies - does that mean they're reusable or not? If not, what if Grok still needs one?
*Many* dependencies makes it harder to reuse, but not possible. *Any* dependencies on the pile we deem "never-gonna-happen" tars the dependent with the same brush.
I'd be happy to see someone triage the existing set of packages in the categories Tres proposes.
- zope.app.container (Products.Five.browser.adding)
Should be easy enough to move to zope.container, hopefully.
I moved the IAdding interface to zope.browser, which contains *only* interfaces at this point (and therefore no dependencies beyond zope.interface). I also moved IView and IBrowserView there from zope.publisher.interfaces.browser.
- zope.app.form (Products.Five.form.*)
This actually isn't that bad in the reusability department; it's mostly a bunch of widgets. Not that it's the be all end all form library by a long shot, and most of our libraries shouldn't be depending on it, but it's not a bad thing to have it in the ZTK for now.
It should get renamed, then (zope.formwidgets, or something). The 'zope.app' stuff is presumptively "never-gonna-happen." Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKC089+gerLs4ltQ4RAosKAKCnDYja0uYLf6wKancg+EJGIM5YpQCfS/PF zCxlVo1hL+iSLc7yMaQjtrg= =LwuV -----END PGP SIGNATURE-----
participants (11)
-
Andreas Jung -
Chris McDonough -
Chris Withers -
Jim Fulton -
Malthe Borch -
Martijn Faassen -
Patrick Gerken -
Paul Winkler -
robert rottermann -
Tom Hoffman -
Tres Seaver