Hi there. I have been developing with zope3 for about 4 years and would like to see zope continue in a healthy way into the future. The last couple of years particularly have brought significant change in how we deploy zope particularly with wsgi with or without the zodb. In addition, there is a increasing plethora of lightweight frameworks emerging to compete with mind share and feel zope is loosing ground in this respect. I am feeling increasing pressure and frustration to re-examine what I am doing. Zope has a wonderful code base but it is spread through many packages in the form of dependencies. As a result, a small app in a working z3 setup can start off at almost 50MB while the similar app on a competitive framework may be as little as 15 - 20 MB. To some extent, there is complexity in the politics of change needed since zope is largely consumed as packages by z2 (Plone). So the impetus for change may be less than favorable for those consuming packages in Plone as opposed to a developer interested in creating larger scale apps purely from zope 3 and other python packages. The key concern is dependencies. There have been efforts I realize to settle some of this over the past but in reality the volume of zope packages that comed together for a base build is 'pulling in the world' as i have heard it referred to many times. The testing setup is another major factor in this and the changes controversial over the eliminating the testing framework as a dependency of zope eggs. I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality. bfg is a relatively new framework that builds on wsgi and zope technologies but is an example of what can be achieved if you consume only what you need. It is attractive in a number of respects for zope developers since it offers simplicity and development speed with a lightweight footprint. I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies. The grokcore packages can help with the simplicity development but do little for the dependency issues. I think there are couple of options. One option would be to set about on a course of change to do something about this with the existing codebase. Another option is to create a core of leaner packages that could result in a much smaller, tighter core that can be competitive with the changing python landscape. bfg is currently taking the option of selectively forking some of the packages such as zope.catalog as repoze.catalog for example. Personally, I would like to see these changes occur in some way within zope. In any case I am interested in hearing from folks about what can or ought to be done or whether there is interest in this direction. Many thanks. Regards David
On Wed, Sep 3, 2008 at 02:54, David Pratt <fairwinds.dp@gmail.com> wrote:
In any case I am interested in hearing from folks about what can or ought to be done or whether there is interest in this direction. Many thanks.
That the packages are too dependent on each other today, and that this means a base installation of Zope3 is too big is well known. So I think I can definitely say "Yes" to that answer. -- Lennart Regebro: Zope and Plone consulting. http://www.colliberty.com/ +33 661 58 14 64
--On 3. September 2008 08:04:00 +0200 Lennart Regebro <regebro@gmail.com> wrote:
On Wed, Sep 3, 2008 at 02:54, David Pratt <fairwinds.dp@gmail.com> wrote:
In any case I am interested in hearing from folks about what can or ought to be done or whether there is interest in this direction. Many thanks.
That the packages are too dependent on each other today, and that this means a base installation of Zope3 is too big is well known. So I think I can definitely say "Yes" to that answer.
This is a big issue? I don't think so. Disks are cheap and usually you don't get in touch with the dependent modules under the hood - except for debugging :-) -aj
Hi Andreas
Betreff: Re: [Zope-dev] Dependencies and future of zope 3
--On 3. September 2008 08:04:00 +0200 Lennart Regebro <regebro@gmail.com> wrote:
On Wed, Sep 3, 2008 at 02:54, David Pratt <fairwinds.dp@gmail.com> wrote:
In any case I am interested in hearing from folks about what can or ought to be done or whether there is interest in this direction. Many thanks.
That the packages are too dependent on each other today, and that this means a base installation of Zope3 is too big is well known. So I think I can definitely say "Yes" to that answer.
This is a big issue? I don't think so. Disks are cheap and usually you don't get in touch with the dependent modules under the hood - except for debugging :-)
That's not so simple, we have some bad dependencies which get touched. Because some packages offer components which we use in other packages, but the core implementation of such a package get replaced by another package. e.g. IPasswordManager in zope.app.authentication get used but not the IAuthentication utility if you use z3c.authentication. Or ITerms from zope.app.form get used if you like to implement a widget in z3c.form but you don't need to use something else from zope.app.form. I think the cleanup isn't really needed for zope packages itself. It's more a question how other can reuse small parts of our component architecture without to load everything. My personal meaning is, we already have a component architecture but we need to split it in a different way into reusable components. Such a split could probably not be done earlier because we didn't see all the usecases. But now since we have grok, repoze and z3c we have many more options to reuse other components and this makes it much clearer what we have to provide as reusable and what not. Regards Roger Ineichen
-aj
--On 3. September 2008 12:17:56 +0200 Roger Ineichen <dev@projekt01.ch> wrote:
My personal meaning is, we already have a component architecture but we need to split it in a different way into reusable components. Such a split could probably not be done earlier because we didn't see all the usecases. But now since we have grok, repoze and z3c we have many more options to reuse other components and this makes it much clearer what we have to provide as reusable and what not.
Sounds like a task for someone to build a dependency graph in order to generate a visualization in order to figure out where to break the dependency chain. Andreas
Hi Andreas
Betreff: Re: AW: [Zope-dev] Dependencies and future of zope 3
[...]
Sounds like a task for someone to build a dependency graph in order to generate a visualization in order to figure out where to break the dependency chain.
Marius implemented such an incredible tool already. See in action, it's really great: http://mg.pov.lt/blog/2007/09 http://wiki.zope.org/zope3/PackageDependenciesTool Regards Roger Ineichen
Andreas
On Wed, Sep 03, 2008 at 12:50:06PM +0200, Andreas Jung wrote:
--On 3. September 2008 12:17:56 +0200 Roger Ineichen <dev@projekt01.ch> wrote:
My personal meaning is, we already have a component architecture but we need to split it in a different way into reusable components. Such a split could probably not be done earlier because we didn't see all the usecases. But now since we have grok, repoze and z3c we have many more options to reuse other components and this makes it much clearer what we have to provide as reusable and what not.
Sounds like a task for someone to build a dependency graph in order to generate a visualization in order to figure out where to break the dependency chain.
http://pypi.python.org/pypi/tl.eggdeps is a tool that extracts dependency graphs from egg metadata and produces a .dot file http://code.google.com/p/jrfonseca/wiki/XDot is an interactive .dot viewer (PyGtk + Cairo) http://pypi.python.org/pypi/gtkeggdeps builds on top of tl.eggdeps and expresses the graph as a tree Unfortunately even these tools didn't help much in untangling the big core dependency cycle, during the Foliage sprint last year. Marius Gedminas -- Computo, ergo sum. -- Curt Suplee
Roger, you make excellent sense here. The other issue of course is the testing setup. So there is potential to operate here on a few levels to achieve something that makes much better sense for moving forward. Roger Ineichen wrote:
I think the cleanup isn't really needed for zope packages itself. It's more a question how other can reuse small parts of our component architecture without to load everything.
My personal meaning is, we already have a component architecture but we need to split it in a different way into reusable components. Such a split could probably not be done earlier because we didn't see all the usecases. But now since we have grok, repoze and z3c we have many more options to reuse other components and this makes it much clearer what we have to provide as reusable and what not.
Regards Roger Ineichen
-aj
On Wed, Sep 03, 2008 at 08:18:11AM +0200, Andreas Jung wrote:
--On 3. September 2008 08:04:00 +0200 Lennart Regebro <regebro@gmail.com> wrote:
That the packages are too dependent on each other today, and that this means a base installation of Zope3 is too big is well known. So I think I can definitely say "Yes" to that answer.
This is a big issue?
Yes.
I don't think so. Disks are cheap and usually you don't get in touch with the dependent modules under the hood - except for debugging :-)
It is not a disk space issue, it's a conceptual issue. Marius Gedminas -- Voodoo Programming: Things programmers do that they know shouldn't work but they try anyway, and which sometimes actually work, such as recompiling everything. -- Karl Lehenbauer
Andreas Jung wrote:
This is a big issue? I don't think so. Disks are cheap and usually you don't get in touch with the dependent modules under the hood - except for debugging :-)
I don't agree. I think making the dependencies fewer would result is easier re-use of bits of "zope", which grows their community and hopefully ups their quality. If this goes well, we might even see robust seperation of packets making the KGS unnecessary... Chris - well, I can always dream ;-) -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Hi David, David Pratt wrote:
I am feeling increasing pressure and frustration to re-examine what I am doing. Zope has a wonderful code base but it is spread through many packages in the form of dependencies. As a result, a small app in a working z3 setup can start off at almost 50MB while the similar app on a competitive framework may be as little as 15 - 20 MB.
Are you worried about disk space? Memory footprint?
I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality.
zope.component, at least, is one of the packages that *does* work without "the world". :)
bfg is a relatively new framework that builds on wsgi and zope technologies but is an example of what can be achieved if you consume only what you need.
True. I'd say that repoze.bfg is very much part of the Zope world, though. It's an example of what Zope (and it's splitting of things into many packages) has made possible.
It is attractive in a number of respects for zope developers since it offers simplicity and development speed with a lightweight footprint.
Yep. It's nice. :)
I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies.
Reducing unneeded dependencies would indeed be a good architectural goal. However, I'm not sure that having a few extra packages today is stopping people from getting things done.
I think there are couple of options. One option would be to set about on a course of change to do something about this with the existing codebase. Another option is to create a core of leaner packages that could result in a much smaller, tighter core that can be competitive with the changing python landscape.
I'm not sure that another "armageddon" of Zope - starting it all again in search of something "better" - will serve anybody or go particularly far. I don't think that's what bfg is doing; I think it's using the power of Zope 3 and the CA to selectively swap out the bits it doesn't like for new bits. I see that as Zope delivering, not Zope failing.
bfg is currently taking the option of selectively forking some of the packages such as zope.catalog as repoze.catalog for example. Personally, I would like to see these changes occur in some way within zope.
+1, but only where it actually makes sense. I'm not sure about repoze.catalog... but quite often, you may get a repoze.* that's just a wrapper around a zope.* package to make it easier to integrate with a particular framework (bfg). That's the way re-use normally happens, I think. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
Hi Martin. The concern is building high volume applications using z3, the memory footprint for virtual hosting, and the unnecessary code that adds to burden of managing security. **I only want the code I use**. I agree that the current situation does not stop folks from getting things done but overall z3 as a development platform is looking not so attractive for these reasons. It is analogous to packing two suitcases of clothes for a trip and finding you just needed a change of underwear and a shirt. Frankly is just getting difficult to accept the status quo anymore so hoping folks can get behind this sort of effort. I am trying to avoid the need for selective forking that Chris has found necessary to make progress with bfg. I want to continue using zope since these things are a big factor for the factors I stated. Martin Aspeli wrote:
Hi David,
David Pratt wrote:
I am feeling increasing pressure and frustration to re-examine what I am doing. Zope has a wonderful code base but it is spread through many packages in the form of dependencies. As a result, a small app in a working z3 setup can start off at almost 50MB while the similar app on a competitive framework may be as little as 15 - 20 MB.
Are you worried about disk space? Memory footprint?
I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality.
zope.component, at least, is one of the packages that *does* work without "the world". :)
bfg is a relatively new framework that builds on wsgi and zope technologies but is an example of what can be achieved if you consume only what you need.
True. I'd say that repoze.bfg is very much part of the Zope world, though. It's an example of what Zope (and it's splitting of things into many packages) has made possible.
It is attractive in a number of respects for zope developers since it offers simplicity and development speed with a lightweight footprint.
Yep. It's nice. :)
I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies.
Reducing unneeded dependencies would indeed be a good architectural goal. However, I'm not sure that having a few extra packages today is stopping people from getting things done.
I think there are couple of options. One option would be to set about on a course of change to do something about this with the existing codebase. Another option is to create a core of leaner packages that could result in a much smaller, tighter core that can be competitive with the changing python landscape.
I'm not sure that another "armageddon" of Zope - starting it all again in search of something "better" - will serve anybody or go particularly far. I don't think that's what bfg is doing; I think it's using the power of Zope 3 and the CA to selectively swap out the bits it doesn't like for new bits. I see that as Zope delivering, not Zope failing.
bfg is currently taking the option of selectively forking some of the packages such as zope.catalog as repoze.catalog for example. Personally, I would like to see these changes occur in some way within zope.
+1, but only where it actually makes sense. I'm not sure about repoze.catalog... but quite often, you may get a repoze.* that's just a wrapper around a zope.* package to make it easier to integrate with a particular framework (bfg). That's the way re-use normally happens, I think.
Martin
On Wed, Sep 3, 2008 at 8:40 AM, David Pratt <fairwinds.dp@gmail.com> wrote:
I am trying to avoid the need for selective forking that Chris has found necessary to make progress with bfg. I want to continue using zope [...]
+1 Experimental forks to help determine what refactoring need to be done in the "mother" package are fine, but I hope that the findings of Plone, Grok, and repoze/bfg can all be folded back in. Finding agreement on what changes to make may take work, but the code base will benefit immensely from it. -- Benji York Senior Software Engineer Zope Corporation
Benji York wrote:
On Wed, Sep 3, 2008 at 8:40 AM, David Pratt <fairwinds.dp@gmail.com> wrote:
I am trying to avoid the need for selective forking that Chris has found necessary to make progress with bfg. I want to continue using zope [...]
+1 Experimental forks to help determine what refactoring need to be done in the "mother" package are fine, but I hope that the findings of Plone, Grok, and repoze/bfg can all be folded back in.
Agreed with this. We want Zope 3 packages to move forward, so I'm very glad that David took up this discussion. It's important we develop a bit of vision here, some guidelines, and a plan on how to get there step by step. Note that Grok hasn't been forking Zope 3 packages. We've built a few packages on top of Zope 3 that are now reusable with straight Zope 3 too, to wit, grokcore.component, grokcore.view and grokcore.security and soon grokcore.formlib. Grok has its own approaches of course, but one thing we spent quite a bit of time on is to be good Zope 3 citizens. Grok 0.14 will be built on top of these grokcore.*, and we took pains to make these compatible with straight Zope 3 projects as well. This means that if you want Grok-style configuration of adapters, views and utilities in your Zope 3 project or library you can use these projects. I have a few z3c packages sitting around that I hope to convert to use these once Grok 0.14 is released. These packages are already finding some uptake in Zope 2 projects as well. It's been interesting to see how the requirements to reuse bits of Grok in Zope 3 and Zope 2 have been pulling togeter to help factor these packages out. I think the only bit that you can really consider a 'fork' is grokproject itself, which is like an improved zopeproject. If someone wants to take it up, we could start factoring out a common core there as well. Regards, Martijn
Hi Martijn. As a side note I have found immense value in the effort to split out the grok packages as it is has been very useful in my own development. I have been looking for you on irc to discuss this further to create a grokcore.traverser package and another package to abstract grok.Model (that depends more upon grokcore.component), grok.Container and grok.Application. This abstraction paves the way for general usage of megrok.rdb, megrok.rdb, and megrok.trails without the grok dependency and can bring the general usage of the model concept into regular z3. You would not believe how much this can reduce the volume of your package with these things. My preference is not to develop in grok, but at the same time these packages are excellent as I can selectively use them to reduce configuration and volume in my packages and not loose anything in the process so it is very much appreciated what you have done here. Martijn Faassen wrote:
Benji York wrote:
On Wed, Sep 3, 2008 at 8:40 AM, David Pratt <fairwinds.dp@gmail.com> wrote:
I am trying to avoid the need for selective forking that Chris has found necessary to make progress with bfg. I want to continue using zope [...] +1 Experimental forks to help determine what refactoring need to be done in the "mother" package are fine, but I hope that the findings of Plone, Grok, and repoze/bfg can all be folded back in.
Agreed with this. We want Zope 3 packages to move forward, so I'm very glad that David took up this discussion. It's important we develop a bit of vision here, some guidelines, and a plan on how to get there step by step.
Note that Grok hasn't been forking Zope 3 packages. We've built a few packages on top of Zope 3 that are now reusable with straight Zope 3 too, to wit, grokcore.component, grokcore.view and grokcore.security and soon grokcore.formlib. Grok has its own approaches of course, but one thing we spent quite a bit of time on is to be good Zope 3 citizens.
Grok 0.14 will be built on top of these grokcore.*, and we took pains to make these compatible with straight Zope 3 projects as well. This means that if you want Grok-style configuration of adapters, views and utilities in your Zope 3 project or library you can use these projects. I have a few z3c packages sitting around that I hope to convert to use these once Grok 0.14 is released. These packages are already finding some uptake in Zope 2 projects as well. It's been interesting to see how the requirements to reuse bits of Grok in Zope 3 and Zope 2 have been pulling togeter to help factor these packages out.
I think the only bit that you can really consider a 'fork' is grokproject itself, which is like an improved zopeproject. If someone wants to take it up, we could start factoring out a common core there as well.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Wednesday 03 September 2008, David Pratt wrote:
I am trying to avoid the need for selective forking that Chris has found necessary to make progress with bfg. I want to continue using zope since these things are a big factor for the factors I stated.
I do not think that this community is so hard headed that forking is necessary. As far as I remember, I have not seen any serious commitment on their side. Many of us are interested in decreasing the dependency tree, so solutions will be found. It is just so much easier to fork. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Stephan Richter wrote:
On Wednesday 03 September 2008, David Pratt wrote:
I am trying to avoid the need for selective forking that Chris has found necessary to make progress with bfg. I want to continue using zope since these things are a big factor for the factors I stated.
I do not think that this community is so hard headed that forking is necessary. As far as I remember, I have not seen any serious commitment on their side. Many of us are interested in decreasing the dependency tree, so solutions will be found. It is just so much easier to fork.
I'm not sure what "committment" and "their side" implies here, but it's probably a good idea to explain what "selective fork" refers to in this context. The package that David referred to as a fork (because I mentioned to him offhand that it was) was "repoze.catalog" (http://static.repoze.org/catalogdocs/). This is a package that is modeled on zope.app.catalog, but which relies on only zope.index. In reality, it's maybe less of a fork than it is a reaction to a divergence of goals. There's no way I could have gotten what I needed *without* abandoning any hope of pushing the changes up to zope.app.catalog because it's likely that nobody but me cares about the use case: zope.app.catalog relies on "the world" because it wants to hook in tightly with Zope containers, local site managers, object events and so on. I didn't want any of that; I just wanted something that I could hand some text and a docid and have the results queryable later, making the responsibility of the catalog to later only hand me back a set of docids based on the query parameters. It'd be absurd to pull down more than zope.index and ZODB for this purpose, which is exactly what the repoze.catalog package relies on. Sometimes such divergences are just necessary. Take for example, Zope 3, which was a "fork" of Zope 2 in basically the same way that repoze.catalog is a fork of zope.app.catalog. You might argue that it should have retained the "Zope" brand (it hasn't). But then again, it's is not really any more a fork than, say, z3c.form is of zope.formlib, and neither does that. You could also maybe argue that it should live in the Zope SVN repository (it doesn't). But then again, neither does Plone or hundreds of other derivative projects. If we unhose the dependency tree a bit, we might be able to avoid "silly" forks of things, but we'll never be able to mend the divergence of goals and natural progress. I'm not terribly optimistic that the existing community has the will or the time to do the necessary unhosing under the "Zope" flag; it will be extremely painful and time-consuming, and a lot of it will imply similar "forks" which will also need to be maintained even while the old code lives on. These forks won't be in the interest of folks who just want to use Zope 2 and Zope 3 wholesale, which appears to be a solid majority right now given the reluctance to make changes to existing packages. - C
On Sep 3, 2008, at 3:57 AM, Martin Aspeli wrote:
I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality.
zope.component, at least, is one of the packages that *does* work without "the world". :)
Only partially and only because I did something I really didn't want to do, which was to employ extras. ...
I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies.
Reducing unneeded dependencies would indeed be a good architectural goal. However, I'm not sure that having a few extra packages today is stopping people from getting things done.
I think there is a distaste for having lots of extra packages around. This isn't very important to me, but it really bugs some folks. Jim -- Jim Fulton Zope Corporation
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
On Sep 3, 2008, at 3:57 AM, Martin Aspeli wrote:
I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality. zope.component, at least, is one of the packages that *does* work without "the world". :)
Only partially and only because I did something I really didn't want to do, which was to employ extras.
...
I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies. Reducing unneeded dependencies would indeed be a good architectural goal. However, I'm not sure that having a few extra packages today is stopping people from getting things done.
I think there is a distaste for having lots of extra packages around. This isn't very important to me, but it really bugs some folks.
Extra dependencies impose burdens on every *client* of the careless pacakge: - Everybody has to download and store the pacakge, which wouldn't be so bad for one-time use, but lots of times "rebuild the world" (including blowing away caches) can be a useful strategy. - The cognitive load is non-trivial, even in the mythical universe where every package has readable and useful documentation: not needing to consider a package's documentation is better than reading it, for the case that it is truly unneeded. - Debugging is tougher, especially in the face of auto-included ZCML. - Auditing the dependent application is harder when there are not- really-needed pacakges in the mix. - Runtime footprint issues (RAM usage, startup time) are also worth onsidering. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIvqvD+gerLs4ltQ4RAmi7AKC6IQg6lhIRIPCEtBQupDH3mx8alwCfTgOk /JbClwZ/OalaVMdv6jYsxNI= =+Uyq -----END PGP SIGNATURE-----
Previously Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jim Fulton wrote:
On Sep 3, 2008, at 3:57 AM, Martin Aspeli wrote:
I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality. zope.component, at least, is one of the packages that *does* work without "the world". :)
Only partially and only because I did something I really didn't want to do, which was to employ extras.
...
I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies. Reducing unneeded dependencies would indeed be a good architectural goal. However, I'm not sure that having a few extra packages today is stopping people from getting things done.
I think there is a distaste for having lots of extra packages around. This isn't very important to me, but it really bugs some folks.
Extra dependencies impose burdens on every *client* of the careless pacakge:
- Everybody has to download and store the pacakge, which wouldn't be so bad for one-time use, but lots of times "rebuild the world" (including blowing away caches) can be a useful strategy.
- The cognitive load is non-trivial, even in the mythical universe where every package has readable and useful documentation: not needing to consider a package's documentation is better than reading it, for the case that it is truly unneeded.
- Debugging is tougher, especially in the face of auto-included ZCML.
- Auditing the dependent application is harder when there are not- really-needed pacakges in the mix.
- Runtime footprint issues (RAM usage, startup time) are also worth onsidering.
- The chance of a conflict when one of the many packages in your project sees a new release increases reapidly Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Hi David
Betreff: [Zope-dev] Dependencies and future of zope 3
Hi there. I have been developing with zope3 for about 4 years and would like to see zope continue in a healthy way into the future. The last couple of years particularly have brought significant change in how we deploy zope particularly with wsgi with or without the zodb. In addition, there is a increasing plethora of lightweight frameworks emerging to compete with mind share and feel zope is loosing ground in this respect.
I am feeling increasing pressure and frustration to re-examine what I am doing. Zope has a wonderful code base but it is spread through many packages in the form of dependencies. As a result, a small app in a working z3 setup can start off at almost 50MB while the similar app on a competitive framework may be as little as 15 - 20 MB. To some extent, there is complexity in the politics of change needed since zope is largely consumed as packages by z2 (Plone). So the impetus for change may be less than favorable for those consuming packages in Plone as opposed to a developer interested in creating larger scale apps purely from zope 3 and other python packages.
The key concern is dependencies. There have been efforts I realize to settle some of this over the past but in reality the volume of zope packages that comed together for a base build is 'pulling in the world' as i have heard it referred to many times. The testing setup is another major factor in this and the changes controversial over the eliminating the testing framework as a dependency of zope eggs.
I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality.
bfg is a relatively new framework that builds on wsgi and zope technologies but is an example of what can be achieved if you consume only what you need. It is attractive in a number of respects for zope developers since it offers simplicity and development speed with a lightweight footprint. I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies. The grokcore packages can help with the simplicity development but do little for the dependency issues.
I think there are couple of options. One option would be to set about on a course of change to do something about this with the existing codebase. Another option is to create a core of leaner packages that could result in a much smaller, tighter core that can be competitive with the changing python landscape. bfg is currently taking the option of selectively forking some of the packages such as zope.catalog as repoze.catalog for example. Personally, I would like to see these changes occur in some way within zope. In any case I am interested in hearing from folks about what can or ought to be done or whether there is interest in this direction. Many thanks.
+1 I fully agree. I put the dependency cleanup on my task list and started the last couple days with reviewing the zope core packages. I think everybody whould be happy if we provide less dependencies. But if it comes to move things arround we really have a lot of work with convince everybody. It whould really help if we could build a team of developers which volunteer to review such cleanup work. That makes it easier to make decisions and avoids that people get stocked with their cleanup work. Is someone willing to help doing that task? Regards Roger Ineichen
Regards David _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Hi there, Roger Ineichen wrote: [snip]
Is someone willing to help doing that task?
I'm very interested in this topic as well, especially from the perspective of Grok of course. There are many strategies to go ahead in doing this. I'll list just one observation I've had here. One observation is that the pattern of '.browser' subpackages tends to expand the dependency structure significantly. Often you want to use non-browser functionality and don't care about the UI that ships with .browser. At the same time .browser tends to add dependencies to the overall package. Other times (such as for zope.app.form.browser) the main reusable functionality of a package is actually almost completely in the .browser sub package. It might be nicer to flatten the namespace then and move things from .browser into the main package. It might therefore make sense to review packages one by one, and see whether zope.foo.browser can be factored out into a zope.fooui package or something like that. Of course the question remains how we can get from A to B without a major breakage in backwards compatibility then. Regards, Martijn
Hi Martijn
Betreff: Re: [Zope-dev] Dependencies and future of zope 3
Hi there, Roger Ineichen wrote: [snip]
Is someone willing to help doing that task?
I'm very interested in this topic as well, especially from the perspective of Grok of course.
That whould be great. I'll let you know if I have something ready for review or like to discuss new topics about dependency refactoring.
There are many strategies to go ahead in doing this. I'll list just one observation I've had here.
One observation is that the pattern of '.browser' subpackages tends to expand the dependency structure significantly. Often you want to use non-browser functionality and don't care about the UI that ships with .browser. At the same time .browser tends to add dependencies to the overall package.
Other times (such as for zope.app.form.browser) the main reusable functionality of a package is actually almost completely in the .browser sub package. It might be nicer to flatten the namespace then and move things from .browser into the main package.
It might therefore make sense to review packages one by one, and see whether zope.foo.browser can be factored out into a zope.fooui package or something like that. Of course the question remains how we can get from A to B without a major breakage in backwards compatibility then.
Good point, that will be a lot of work. The menu/menu item supported by the ZCML directive and the browser:page built in menu option is also a nesty part if it comes to dependencies. Right now I use zc.configuration to exclude every browser configuration from the zope.app* core packages and use my own management views if I need them. But I'm looking forward to use the zope.* packages without any exclude tweak. Regards Roger Ineichen
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Hey Martijn. These are good ideas. I also find myself importing a package for some interfaces which sort of sucks too and which there were perhaps a better solution for. Martijn Faassen wrote:
Hi there, Roger Ineichen wrote: [snip]
Is someone willing to help doing that task?
I'm very interested in this topic as well, especially from the perspective of Grok of course.
There are many strategies to go ahead in doing this. I'll list just one observation I've had here.
One observation is that the pattern of '.browser' subpackages tends to expand the dependency structure significantly. Often you want to use non-browser functionality and don't care about the UI that ships with .browser. At the same time .browser tends to add dependencies to the overall package.
Other times (such as for zope.app.form.browser) the main reusable functionality of a package is actually almost completely in the .browser sub package. It might be nicer to flatten the namespace then and move things from .browser into the main package.
It might therefore make sense to review packages one by one, and see whether zope.foo.browser can be factored out into a zope.fooui package or something like that. Of course the question remains how we can get from A to B without a major breakage in backwards compatibility then.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Wednesday 03 September 2008, Martijn Faassen wrote:
One observation is that the pattern of '.browser' subpackages tends to expand the dependency structure significantly. Often you want to use non-browser functionality and don't care about the UI that ships with .browser. At the same time .browser tends to add dependencies to the overall package.
Other times (such as for zope.app.form.browser) the main reusable functionality of a package is actually almost completely in the .browser sub package. It might be nicer to flatten the namespace then and move things from .browser into the main package.
It might therefore make sense to review packages one by one, and see whether zope.foo.browser can be factored out into a zope.fooui package or something like that. Of course the question remains how we can get from A to B without a major breakage in backwards compatibility then.
Jim must have read your message with a big smile on his face. He was arguing for this approach of flat package names about 2-3 years ago and I shot that proposal down. I hate when I only realize design mistakes years after Jim does. ;-\ For several packages we took the following approach. Most packages that have browser packages are in zope.app; for example, zope.app.folder (we did not convert this package yet). We then took the API and moved it to zope.folder. We imported the API in zope.app.folder again to maintain BBB. This way zope.folder has the minimal deps and zope.app.folder contains the browser code. BTW, zope.app.form is a big exception. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
On Wed, Sep 3, 2008 at 11:41 AM, Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote:
For several packages we took the following approach. Most packages that have browser packages are in zope.app; for example, zope.app.folder (we did not convert this package yet). We then took the API and moved it to zope.folder.
Maybe we should create a new namespace package for "browser" code. How about "zope.browser"? -- Benji York Senior Software Engineer Zope Corporation
On Wed, Sep 3, 2008 at 1:47 PM, Benji York <benji@zope.com> wrote:
Maybe we should create a new namespace package for "browser" code.
How about "zope.browser"?
Heh. That's just sooo long. Perhaps it should be "zobr". :-) -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Benji York wrote:
On Wed, Sep 3, 2008 at 11:41 AM, Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote:
For several packages we took the following approach. Most packages that have browser packages are in zope.app; for example, zope.app.folder (we did not convert this package yet). We then took the API and moved it to zope.folder.
Maybe we should create a new namespace package for "browser" code.
How about "zope.browser"?
-0 My general sentiment is against creating more structure than we already have. Flat is better than nested. IMHO it's perfectly ok to have the Python APIs in zope.foo and browser code in zope.app.foo. I think sooner than later people won't want to the zope.app.* stuff anyway.
On Wednesday 03 September 2008, Philipp von Weitershausen wrote:
My general sentiment is against creating more structure than we already have. Flat is better than nested. IMHO it's perfectly ok to have the Python APIs in zope.foo and browser code in zope.app.foo. I think sooner than later people won't want to the zope.app.* stuff anyway.
Well, I don't. :-) I only use the default browser code because I have to due to dependencies. But with z3c.form and ZAM I have better alternatives already. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Hi there, On Wed, Sep 3, 2008 at 7:55 PM, Philipp von Weitershausen <philipp@weitershausen.de> wrote:
Benji York wrote: [snip]
Maybe we should create a new namespace package for "browser" code.
How about "zope.browser"?
My general sentiment is against creating more structure than we already have. Flat is better than nested. IMHO it's perfectly ok to have the Python APIs in zope.foo and browser code in zope.app.foo. I think sooner than later people won't want to the zope.app.* stuff anyway.
Yeah, I'm leaning towards this approach as well. It's less to move around. We should not be afraid to move stuff from zope.app.* into zope.* though if it's clear it's "browser-related" but still not a UI. Say, absolute URL generation. Otherwise we'll never get rid of some of the zope.app.* stuff. Regards, Martijn
Hi
Betreff: Re: [Zope-dev] Dependencies and future of zope 3
On Wed, Sep 3, 2008 at 11:41 AM, Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote:
For several packages we took the following approach. Most packages that have browser packages are in zope.app; for example, zope.app.folder (we did not convert this package yet). We then took the API and moved it to zope.folder.
Maybe we should create a new namespace package for "browser" code.
How about "zope.browser"?
Most packages which are interesting for reuse provide more or less only ZMI related views. What about zope.zmi if they provide views for the ZMI. This views are allmost unuseable outside the ZMI (know as Rotterdam skin) Regards Roger Ineichen
-- Benji York Senior Software Engineer Zope Corporation _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Hey Roger. Sounds reasonable to me. Can we also discuss the potential of only including testing setup for dev eggs and removing testing as part of a release when the eggs are packaged to pypi or other repository for consumption. Besides loosing the dependency, this makes for happier folks external to zope that consume our eggs. While I personally do not like the contributor agreement, I am willing to sign to help out to work with you and others to get this settled. I am busy just like anyone else, but this stuff with the dependencies has to end now. Weve been with eggs for more than a couple years, progress has been made but it has been slow. Seriously, let's see what we can do to. The browser packages are a good place to start. Testing another. Third would be seriously examining dependencies of core again once this is done. Fourth might be tackling some of the zope.xxx zope.app.xxx relationships. Some of the stale packages in the main repository and placing them at another location if they are unmaintained might also be in order. If we want to folks to use zope we need to be friendly to wsgi with or without a zodb and show both sides of the coin - that CA + choice of backend + zope security + choice of traversal method (with publisher) == interesting, productive, mature, dynamic and efficient. On Wed, Sep 3, 2008 at 3:06 PM, Roger Ineichen <dev@projekt01.ch> wrote:
Hi
Betreff: Re: [Zope-dev] Dependencies and future of zope 3
On Wed, Sep 3, 2008 at 11:41 AM, Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote:
For several packages we took the following approach. Most packages that have browser packages are in zope.app; for example, zope.app.folder (we did not convert this package yet). We then took the API and moved it to zope.folder.
Maybe we should create a new namespace package for "browser" code.
How about "zope.browser"?
Most packages which are interesting for reuse provide more or less only ZMI related views.
What about zope.zmi if they provide views for the ZMI. This views are allmost unuseable outside the ZMI (know as Rotterdam skin)
Regards Roger Ineichen
-- Benji York Senior Software Engineer Zope Corporation _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Wednesday 03 September 2008, David Pratt wrote:
Hey Roger. Sounds reasonable to me. Can we also discuss the potential of only including testing setup for dev eggs and removing testing as part of a release when the eggs are packaged to pypi or other repository for consumption.
-1. Without the testing environment available in the packages, we could not: (a) Test the packages using the KGS. (b) Verify that a package works within a given environment. Both of those use cases are extremely important to us.
While I personally do not like the contributor agreement, I am willing to sign to help out to work with you and others to get this settled. I am busy just like anyone else, but this stuff with the dependencies has to end now. Weve been with eggs for more than a couple years, progress has been made but it has been slow. Seriously, let's see what we can do to.
We are using eggs for about 1.5 years and really did not have a sensible configuration until the KGS came around less than a year ago. But other than that, I am all for less dependencies. Roger and I have worked on this a lot already. (ZAM was developed out of a need to have the least amount of dependencies while still having the ability to use a management interface.) Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Hi David
Betreff: Re: [Zope-dev] Dependencies and future of zope 3
Hey Roger. Sounds reasonable to me. Can we also discuss the potential of only including testing setup for dev eggs and removing testing as part of a release when the eggs are packaged to pypi or other repository for consumption.
I guess we do not have tets eggs. What do you mean with test eggs. I think extras_require test is a pattern which let's you use the extras or not if you use an egg. By default an egg has only dependencies the defined packages in install_requires. Or are I'm wrong?
Besides loosing the dependency, this makes for happier folks external to zope that consume our eggs.
While I personally do not like the contributor agreement, I am willing to sign to help out to work with you and others to get this settled. I am busy just like anyone else, but this stuff with the dependencies has to end now. Weve been with eggs for more than a couple years, progress has been made but it has been slow. Seriously, let's see what we can do to.
Cool any help is welcome.
The browser packages are a good place to start. Testing another. Third would be seriously examining dependencies of core again once this is done. Fourth might be tackling some of the zope.xxx zope.app.xxx relationships. Some of the stale packages in the main repository and placing them at another location if they are unmaintained might also be in order.
I think we should start with identify the hard core dependencies and list them in a proposal or another document in the zope wiki. Anybody can list their ideas of what should be done and list ideas how we can solve the problems. We also can use that paper for vote about the different refactorings. Such a proposal/paper could also be usefull for others which don't read each mail. We have different kind of refactorings which all solve some problems. I think we should not start with the browser views. There are some core dependencies we need to cleanup first. Right now I'm working forward with small refactorings which solve some dependencies to zope.app.form (ITerms) and zope.app.authentication (IPaswordManager). After that, my goal is to work on the testing framework, offering a clean testing (skin) layer, which should make it possible to write functional tests without to use the basic, default or rotterdam skin and the zope.app.authentication package. I guess that's what the repoze people need to have too. Your help is defently very welcome. Go ahead with the contributor agreement sing up and let Jim know that I volunteer for you.
If we want to folks to use zope we need to be friendly to wsgi with or without a zodb and show both sides of the coin - that CA + choice of backend + zope security + choice of traversal method (with publisher) == interesting, productive, mature, dynamic and efficient.
Sounds interesting but let's put that on the todo later list. Regards Roger Ineichen _____________________________ END OF MESSAGE
On Wed, Sep 3, 2008 at 3:06 PM, Roger Ineichen <dev@projekt01.ch> wrote:
Hi
Betreff: Re: [Zope-dev] Dependencies and future of zope 3
On Wed, Sep 3, 2008 at 11:41 AM, Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote:
For several packages we took the following approach. Most packages that have browser packages are in zope.app; for example, zope.app.folder (we did not convert this package yet). We then took the API and moved it to zope.folder.
Maybe we should create a new namespace package for "browser" code.
How about "zope.browser"?
Most packages which are interesting for reuse provide more or less only ZMI related views.
What about zope.zmi if they provide views for the ZMI. This views are allmost unuseable outside the ZMI (know as Rotterdam skin)
Regards Roger Ineichen
-- Benji York Senior Software Engineer Zope Corporation _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Roger, what you say makes good sense. I will get agreement signed and sent and off to Jim. I am much more optimistic than I have been for a long time. This stuff has really been bothering me since I am concerned about efficient wsgi virtual host deployments and zope is unnecessarily heavy. Personally I would like to see a core zope install with a footprint of no more that 20MB with just essential packages. I am a believer in zope and I am encouraged by the support for change. I also realize some of this will be disruptive but it is necessary. A wiki page will be helpful to communicate and get the best ideas for moving ahead. There are a number of good folks here that understand the circumstances so we have an excellent opportunity to act on this. On Wed, Sep 3, 2008 at 9:09 PM, Roger Ineichen <dev@projekt01.ch> wrote:
Hi David
Betreff: Re: [Zope-dev] Dependencies and future of zope 3
Hey Roger. Sounds reasonable to me. Can we also discuss the potential of only including testing setup for dev eggs and removing testing as part of a release when the eggs are packaged to pypi or other repository for consumption.
I guess we do not have tets eggs. What do you mean with test eggs.
I think extras_require test is a pattern which let's you use the extras or not if you use an egg. By default an egg has only dependencies the defined packages in install_requires. Or are I'm wrong?
Besides loosing the dependency, this makes for happier folks external to zope that consume our eggs.
While I personally do not like the contributor agreement, I am willing to sign to help out to work with you and others to get this settled. I am busy just like anyone else, but this stuff with the dependencies has to end now. Weve been with eggs for more than a couple years, progress has been made but it has been slow. Seriously, let's see what we can do to.
Cool any help is welcome.
The browser packages are a good place to start. Testing another. Third would be seriously examining dependencies of core again once this is done. Fourth might be tackling some of the zope.xxx zope.app.xxx relationships. Some of the stale packages in the main repository and placing them at another location if they are unmaintained might also be in order.
I think we should start with identify the hard core dependencies and list them in a proposal or another document in the zope wiki. Anybody can list their ideas of what should be done and list ideas how we can solve the problems. We also can use that paper for vote about the different refactorings.
Such a proposal/paper could also be usefull for others which don't read each mail.
We have different kind of refactorings which all solve some problems. I think we should not start with the browser views. There are some core dependencies we need to cleanup first.
Right now I'm working forward with small refactorings which solve some dependencies to zope.app.form (ITerms) and zope.app.authentication (IPaswordManager).
After that, my goal is to work on the testing framework, offering a clean testing (skin) layer, which should make it possible to write functional tests without to use the basic, default or rotterdam skin and the zope.app.authentication package.
I guess that's what the repoze people need to have too.
Your help is defently very welcome. Go ahead with the contributor agreement sing up and let Jim know that I volunteer for you.
If we want to folks to use zope we need to be friendly to wsgi with or without a zodb and show both sides of the coin - that CA + choice of backend + zope security + choice of traversal method (with publisher) == interesting, productive, mature, dynamic and efficient.
Sounds interesting but let's put that on the todo later list.
Regards Roger Ineichen _____________________________ END OF MESSAGE
On Wed, Sep 3, 2008 at 3:06 PM, Roger Ineichen <dev@projekt01.ch> wrote:
Hi
Betreff: Re: [Zope-dev] Dependencies and future of zope 3
On Wed, Sep 3, 2008 at 11:41 AM, Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote:
For several packages we took the following approach. Most packages that have browser packages are in zope.app; for example, zope.app.folder (we did not convert this package yet). We then took the API and moved it to zope.folder.
Maybe we should create a new namespace package for "browser" code.
How about "zope.browser"?
Most packages which are interesting for reuse provide more or less only ZMI related views.
What about zope.zmi if they provide views for the ZMI. This views are allmost unuseable outside the ZMI (know as Rotterdam skin)
Regards Roger Ineichen
-- Benji York Senior Software Engineer Zope Corporation _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Hi there, Roger Ineichen wrote: [snip]
We have different kind of refactorings which all solve some problems. I think we should not start with the browser views. There are some core dependencies we need to cleanup first.
Right now I'm working forward with small refactorings which solve some dependencies to zope.app.form (ITerms) and zope.app.authentication (IPaswordManager).
I think most of our progress will be by incremental step-by-step refactoring, so I'm glad to hear you're working on this. Regards, Martijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Pratt wrote:
Hey Roger. Sounds reasonable to me. Can we also discuss the potential of only including testing setup for dev eggs and removing testing as part of a release when the eggs are packaged to pypi or other repository for consumption.
Besides loosing the dependency, this makes for happier folks external to zope that consume our eggs.
- -1 to removing the tests from distributions (which should nearly always be 'sdists', rather than eggs). Consumers should be *able* to run tests, if desired, but they shouldn't have to pay the price for the testrunner unless they choose too: "pay for what you eat." The change under discussion about testing is whether or not to remove the test requirements as part of 'install_requires' in setup.py. Some folks have objecctions to using the setuptools-provided 'tests_require' field, although I think the argument there is weak: packages which spell 'tests_require' and 'test_suite' in their setup can be tested trivially with 'python setup.py test', which seems a win to me. I'm not volunteering to write it, but it strikes me as odd that folks haven't morphed zc.buildout (and / or associated recipes) to use this field. I *did* write a setuptools add on which saved the 'tests_require' info into the EGG_INFO directory: that package could be used to capture the metadata during installation of packages, for consumption by a testrunner later. I discovered today I think the time is ripe for a "blank buffer" rewrite of the testrunner: it is so full of "twisty passages" that my confidence in its own internal correctness is seriously depleted. It has too many features, and too many incompatible ways to run it: I would love a simpler version, whose discovery logic used egg metadata instead of (mal)heuristics (e.g., ordering of '--test-path' and '--package-path' arguments can make some tests unfindable).
While I personally do not like the contributor agreement, I am willing to sign to help out to work with you and others to get this settled. I am busy just like anyone else, but this stuff with the dependencies has to end now. Weve been with eggs for more than a couple years, progress has been made but it has been slow. Seriously, let's see what we can do to.
The browser packages are a good place to start. Testing another. Third would be seriously examining dependencies of core again once this is done. Fourth might be tackling some of the zope.xxx zope.app.xxx' relationships. Some of the stale packages in the main repository and placing them at another location if they are unmaintained might also be in order.
If we want to folks to use zope we need to be friendly to wsgi with or without a zodb and show both sides of the coin - that CA + choice of backend + zope security + choice of traversal method (with publisher) == interesting, productive, mature, dynamic and efficient.
Stripping away inessentials is always hard, with layering and good dependency management the only sane path forward. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIvy3M+gerLs4ltQ4RAkFFAKDbUVHergvl7EllWy9uP02odeerfwCgryNb qL7CY7DAVGqeEUKxpnBJ5CY= =NUwO -----END PGP SIGNATURE-----
On Wednesday 03 September 2008, Tres Seaver wrote:
I'm not volunteering to write it, but it strikes me as odd that folks haven't morphed zc.buildout (and / or associated recipes) to use this field. I *did* write a setuptools add on which saved the 'tests_require' info into the EGG_INFO directory: that package could be used to capture the metadata during installation of packages, for consumption by a testrunner later.
It might be that nobody was aware of this. The last discussion I read ended in "EGG_INFO does not record the test_require data". This is good news. I think the people most interested in stripping dependencies should add a feature to zc.buildout making this info usable as a first step. Once an example has been given on how to use this new pattern, packages can be quickly updated to use it. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
On Sep 3, 2008, at 8:37 PM, Tres Seaver wrote:
The change under discussion about testing is whether or not to remove the test requirements as part of 'install_requires' in setup.py. Some folks have objecctions to using the setuptools-provided 'tests_require' field, although I think the argument there is weak: packages which spell 'tests_require' and 'test_suite' in their setup can be tested trivially with 'python setup.py test', which seems a win to me.
I agree that this is a win. It is not a big win IMO, as I prefer a separate test runner. It would be far more useful to me if the data were captured as part of egg info.
I'm not volunteering to write it, but it strikes me as odd that folks haven't morphed zc.buildout (and / or associated recipes) to use this field. I *did* write a setuptools add on which saved the 'tests_require' info into the EGG_INFO directory: that package could be used to capture the metadata during installation of packages, for consumption by a testrunner later.
That's very good. It would be much much much better if setuptools would do this by default. Now that setuptools' development is opening up, perhaps we can do this. The setuptools add-on doesn't help when you don't have source releases and many interesting packages get distributed as eggs on Windows.
I discovered today I think the time is ripe for a "blank buffer" rewrite of the testrunner: it is so full of "twisty passages" that my confidence in its own internal correctness is seriously depleted. It has too many features,
I doubt it. I find the vast majority of it's features very useful. I agree that the test discovery is ripe for a revamp.
and too many incompatible ways to run it: I would love a simpler version, whose discovery logic used egg metadata instead of (mal)heuristics (e.g., ordering of '--test-path' and '--package-path' arguments can make some tests unfindable).
I'm very open to revamping test discovery. I'm opposed to a rewrite. I sure as heck wouldn't be willing to spend time on a rewrite. I wouldn't be opposed to replacing out test runner with a bunch of nose plugins that added missing features. I have almost zero interest in using setuptools test runner during development. Jim -- Jim Fulton Zope Corporation
Hi there, On Thu, Sep 4, 2008 at 4:29 PM, Jim Fulton <jim@zope.com> wrote: [snip]
I discovered today I think the time is ripe for a "blank buffer" rewrite of the testrunner: it is so full of "twisty passages" that my confidence in its own internal correctness is seriously depleted. It has too many features,
I doubt it. I find the vast majority of it's features very useful. I agree that the test discovery is ripe for a revamp.
Note also that Christian Theune also started a major cleanup/reorganization of the testrunner some months ago. Christian can tell you more about the status and future plans for further cleanups. Regards, Martijn
On Thu, 2008-09-04 at 16:33 +0200, Martijn Faassen wrote:
Note also that Christian Theune also started a major cleanup/reorganization of the testrunner some months ago. Christian can tell you more about the status and future plans for further cleanups.
The code became more readable in many places so the test runner became easier to change. The test discovery is one of the remaining really dirty places in there and I'd be happy to see this improved. I'd also be happy to help with this. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
On Sep 3, 2008, at 8:37 PM, Tres Seaver wrote:
The change under discussion about testing is whether or not to remove the test requirements as part of 'install_requires' in setup.py. Some folks have objecctions to using the setuptools-provided 'tests_require' field, although I think the argument there is weak: packages which spell 'tests_require' and 'test_suite' in their setup can be tested trivially with 'python setup.py test', which seems a win to me.
I agree that this is a win. It is not a big win IMO, as I prefer a separate test runner.
It would be far more useful to me if the data were captured as part of egg info.
I'm not volunteering to write it, but it strikes me as odd that folks haven't morphed zc.buildout (and / or associated recipes) to use this field. I *did* write a setuptools add on which saved the 'tests_require' info into the EGG_INFO directory: that package could be used to capture the metadata during installation of packages, for consumption by a testrunner later.
That's very good. It would be much much much better if setuptools would do this by default. Now that setuptools' development is opening up, perhaps we can do this.
'egginfo' uses the established "setuptools plugin" API: I don't have commit access, but I would be fine with having it merged. In the meantime, including it within 'zc.buildout' environments would make the information widely available.
The setuptools add-on doesn't help when you don't have source releases and many interesting packages get distributed as eggs on Windows.
'egginfo' includes the test info in bdist_egg zipfiles if it is present on the system where the egg is built.
I discovered today I think the time is ripe for a "blank buffer" rewrite of the testrunner: it is so full of "twisty passages" that my confidence in its own internal correctness is seriously depleted. It has too many features,
I doubt it. I find the vast majority of it's features very useful. I agree that the test discovery is ripe for a revamp.
Can you truly say that reading the code doesn't make your eyes cross? Here are a couple I noted while debugging broken discovery yesterday: - Extensive use of generators, in spite of the fact that we always consume them all. It feels like a "here's a new language feature, let's use it" choice, and makes debugging the code harder. I doubt very much that the decision was driven by an actual constraint (the only one I can imagine is RAM consumption). - '--path' vs '--test-path' vs. '--package-path' interact in unpredictable ways, based on ordering.
and too many incompatible ways to run it: I would love a simpler version, whose discovery logic used egg metadata instead of (mal)heuristics (e.g., ordering of '--test-path' and '--package-path' arguments can make some tests unfindable).
I'm very open to revamping test discovery.
I'm opposed to a rewrite. I sure as heck wouldn't be willing to spend time on a rewrite. I wouldn't be opposed to replacing out test runner with a bunch of nose plugins that added missing features. I have almost zero interest in using setuptools test runner during development.
I typically use 'setup.py test' in two cases: - When evaluating a new package, I create a virtualenv and run 'setup.py develop' and 'setup.py test' using the virtualenv's python. Packages whose tests run clealy in a sandbox get a higher "score" than those which don't: for one thing, it guarantees that their dependencies are accurate. - When building packages which I hope will be used outsize of Zopeland, I want to make the tests accessible to folks who don't use zc.buildout or zope.testing. I tend to skip running 'setup.py test' only in projects where I am writing custom software, and where the project is already committed to another testrunner. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIv/4u+gerLs4ltQ4RAp+8AJ9iTagUjGOdkgYGIA7xGqeyFu6X/wCcD74d 6LZrrCg9OsDb8+FYHG2rVmM= =d2kX -----END PGP SIGNATURE-----
David Pratt wrote at 2008-9-3 20:32 -0300:
Can we also discuss the potential of only including testing setup for dev eggs and removing testing as part of a release when the eggs are packaged to pypi or other repository for consumption.
-1. This would really only save disk space.... -- Dieter
Roger Ineichen wrote: [snip]
Most packages which are interesting for reuse provide more or less only ZMI related views.
What about zope.zmi if they provide views for the ZMI. This views are allmost unuseable outside the ZMI (know as Rotterdam skin)
Why not simply leave the ZMI stuff in zope.app.* for now? It means less to move around, so more likely it'll get done and less chance it'll break. I.e. the eventual goal would be that the zope.app.* packages *only* contain the remaining ZMI stuff. Regards, Martijn
Hi Martijn
Betreff: Re: [Zope-dev] Dependencies and future of zope 3
Roger Ineichen wrote: [snip]
Most packages which are interesting for reuse provide more or less only ZMI related views.
What about zope.zmi if they provide views for the ZMI. This views are allmost unuseable outside the ZMI (know as Rotterdam skin)
Why not simply leave the ZMI stuff in zope.app.* for now? It means less to move around, so more likely it'll get done and less chance it'll break. I.e. the eventual goal would be that the zope.app.* packages *only* contain the remaining ZMI stuff.
I can absolutly agree on that. Regards Roger Ineichen _____________________________ END OF MESSAGE
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Hey, On Wed, Sep 3, 2008 at 5:41 PM, Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote: [snip]
Jim must have read your message with a big smile on his face. He was arguing for this approach of flat package names about 2-3 years ago and I shot that proposal down. I hate when I only realize design mistakes years after Jim does. ;-\
I only get the idea a few years after Jim gets them. I don't know what's better. :)
For several packages we took the following approach. Most packages that have browser packages are in zope.app; for example, zope.app.folder (we did not convert this package yet). We then took the API and moved it to zope.folder. We imported the API in zope.app.folder again to maintain BBB. This way zope.folder has the minimal deps and zope.app.folder contains the browser code.
I guess that's indeed a reasonable way to move forward. The idea would be that we can move forward on the zope.* package and probably retire some of the zope.app.* packages eventually.
BTW, zope.app.form is a big exception.
Agreed. Just wanted to mention the exception too. There are some other publisher related packages that also have significant not-really-UI stuff in .browser. Regards, Martijn
Hi Roger. Great. I am willing to help with this. I understand the politics of change and feel there is most likely less impetus for change for those consuming packages as opposed to folks like yourself or I that use zope 3 as our framework. This is something that has to happen. The situation has gone on too long and the answer has been to exclude configuration. This is only a partial solution at best. Roger Ineichen wrote:
Hi David
Betreff: [Zope-dev] Dependencies and future of zope 3
Hi there. I have been developing with zope3 for about 4 years and would like to see zope continue in a healthy way into the future. The last couple of years particularly have brought significant change in how we deploy zope particularly with wsgi with or without the zodb. In addition, there is a increasing plethora of lightweight frameworks emerging to compete with mind share and feel zope is loosing ground in this respect.
I am feeling increasing pressure and frustration to re-examine what I am doing. Zope has a wonderful code base but it is spread through many packages in the form of dependencies. As a result, a small app in a working z3 setup can start off at almost 50MB while the similar app on a competitive framework may be as little as 15 - 20 MB. To some extent, there is complexity in the politics of change needed since zope is largely consumed as packages by z2 (Plone). So the impetus for change may be less than favorable for those consuming packages in Plone as opposed to a developer interested in creating larger scale apps purely from zope 3 and other python packages.
The key concern is dependencies. There have been efforts I realize to settle some of this over the past but in reality the volume of zope packages that comed together for a base build is 'pulling in the world' as i have heard it referred to many times. The testing setup is another major factor in this and the changes controversial over the eliminating the testing framework as a dependency of zope eggs.
I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality.
bfg is a relatively new framework that builds on wsgi and zope technologies but is an example of what can be achieved if you consume only what you need. It is attractive in a number of respects for zope developers since it offers simplicity and development speed with a lightweight footprint. I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies. The grokcore packages can help with the simplicity development but do little for the dependency issues.
I think there are couple of options. One option would be to set about on a course of change to do something about this with the existing codebase. Another option is to create a core of leaner packages that could result in a much smaller, tighter core that can be competitive with the changing python landscape. bfg is currently taking the option of selectively forking some of the packages such as zope.catalog as repoze.catalog for example. Personally, I would like to see these changes occur in some way within zope. In any case I am interested in hearing from folks about what can or ought to be done or whether there is interest in this direction. Many thanks.
+1
I fully agree. I put the dependency cleanup on my task list and started the last couple days with reviewing the zope core packages.
I think everybody whould be happy if we provide less dependencies. But if it comes to move things arround we really have a lot of work with convince everybody.
It whould really help if we could build a team of developers which volunteer to review such cleanup work. That makes it easier to make decisions and avoids that people get stocked with their cleanup work.
Is someone willing to help doing that task?
Regards Roger Ineichen
Regards David _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Some high-level remarks: I agree with your sentiments. I too would like to see Zope 3 technology become more usable for lightweight applications. I'd like to see the existing code base evolve in that direction. Unfortunately, Zope 3 evolved as a monolithic development tree. Tendrils formed between packages that should have been independent. There was no incentive to keep things cleanly separated. I'm certain that this is fixable, but it will take a lot of work. I think this is happening slowly. Many of us have "day jobs" and it's hard to make this a priority. Jim On Sep 2, 2008, at 8:54 PM, David Pratt wrote:
Hi there. I have been developing with zope3 for about 4 years and would like to see zope continue in a healthy way into the future. The last couple of years particularly have brought significant change in how we deploy zope particularly with wsgi with or without the zodb. In addition, there is a increasing plethora of lightweight frameworks emerging to compete with mind share and feel zope is loosing ground in this respect.
I am feeling increasing pressure and frustration to re-examine what I am doing. Zope has a wonderful code base but it is spread through many packages in the form of dependencies. As a result, a small app in a working z3 setup can start off at almost 50MB while the similar app on a competitive framework may be as little as 15 - 20 MB. To some extent, there is complexity in the politics of change needed since zope is largely consumed as packages by z2 (Plone). So the impetus for change may be less than favorable for those consuming packages in Plone as opposed to a developer interested in creating larger scale apps purely from zope 3 and other python packages.
The key concern is dependencies. There have been efforts I realize to settle some of this over the past but in reality the volume of zope packages that comed together for a base build is 'pulling in the world' as i have heard it referred to many times. The testing setup is another major factor in this and the changes controversial over the eliminating the testing framework as a dependency of zope eggs.
I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality.
bfg is a relatively new framework that builds on wsgi and zope technologies but is an example of what can be achieved if you consume only what you need. It is attractive in a number of respects for zope developers since it offers simplicity and development speed with a lightweight footprint. I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies. The grokcore packages can help with the simplicity development but do little for the dependency issues.
I think there are couple of options. One option would be to set about on a course of change to do something about this with the existing codebase. Another option is to create a core of leaner packages that could result in a much smaller, tighter core that can be competitive with the changing python landscape. bfg is currently taking the option of selectively forking some of the packages such as zope.catalog as repoze.catalog for example. Personally, I would like to see these changes occur in some way within zope. In any case I am interested in hearing from folks about what can or ought to be done or whether there is interest in this direction. Many thanks.
Regards David _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Jim Fulton Zope Corporation
Hi Jim. Here is an idea I have that can help bring perspective and change. I propose that if we had the efforts of a few developers to work on a single reference application, and the eyes of others willing to inspect the package we could all benefit. The idea would be to make the reference application as lightweight as possible and work backwards so that we can measure change. This would be a simple wsgi application. I propose we use cluemapper since it is simple, small and would take little time. We can create the reference app with different backends so we can see effect of zodb also. The idea would be to use the reference to expose the issues, propose and make changes, and measure the impact of changes we are making. We also see how competitive we are compared to equivalent application on another framework in terms of no of app files, RAM consumption, no of packages, or other measures that would be important to developers. We can target the dependencies from the perspective of the impact it is having on something real as opposed to perceived. A second benefit is that we can use the application to educate folks on simple and lightweight zope development with wsgi. Jim Fulton wrote:
Some high-level remarks:
I agree with your sentiments. I too would like to see Zope 3 technology become more usable for lightweight applications. I'd like to see the existing code base evolve in that direction.
Unfortunately, Zope 3 evolved as a monolithic development tree. Tendrils formed between packages that should have been independent. There was no incentive to keep things cleanly separated.
I'm certain that this is fixable, but it will take a lot of work. I think this is happening slowly. Many of us have "day jobs" and it's hard to make this a priority.
Jim
On Sep 2, 2008, at 8:54 PM, David Pratt wrote:
Hi there. I have been developing with zope3 for about 4 years and would like to see zope continue in a healthy way into the future. The last couple of years particularly have brought significant change in how we deploy zope particularly with wsgi with or without the zodb. In addition, there is a increasing plethora of lightweight frameworks emerging to compete with mind share and feel zope is loosing ground in this respect.
I am feeling increasing pressure and frustration to re-examine what I am doing. Zope has a wonderful code base but it is spread through many packages in the form of dependencies. As a result, a small app in a working z3 setup can start off at almost 50MB while the similar app on a competitive framework may be as little as 15 - 20 MB. To some extent, there is complexity in the politics of change needed since zope is largely consumed as packages by z2 (Plone). So the impetus for change may be less than favorable for those consuming packages in Plone as opposed to a developer interested in creating larger scale apps purely from zope 3 and other python packages.
The key concern is dependencies. There have been efforts I realize to settle some of this over the past but in reality the volume of zope packages that comed together for a base build is 'pulling in the world' as i have heard it referred to many times. The testing setup is another major factor in this and the changes controversial over the eliminating the testing framework as a dependency of zope eggs.
I guess the simple solution is well it you don't like it, use the another framework. Its not quite that simple since I am extremely fond of the CA architecture and have a strong desire to continue with it in some form or another into the future. I think what I am sensing more than anything is a need for zope to adapt a changing reality.
bfg is a relatively new framework that builds on wsgi and zope technologies but is an example of what can be achieved if you consume only what you need. It is attractive in a number of respects for zope developers since it offers simplicity and development speed with a lightweight footprint. I believe much of what is being accomplished in bfg could be accomplished in zope if it were tighter and we could focus on a leaner core of packages void of the large number of dependencies. The grokcore packages can help with the simplicity development but do little for the dependency issues.
I think there are couple of options. One option would be to set about on a course of change to do something about this with the existing codebase. Another option is to create a core of leaner packages that could result in a much smaller, tighter core that can be competitive with the changing python landscape. bfg is currently taking the option of selectively forking some of the packages such as zope.catalog as repoze.catalog for example. Personally, I would like to see these changes occur in some way within zope. In any case I am interested in hearing from folks about what can or ought to be done or whether there is interest in this direction. Many thanks.
Regards David _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Jim Fulton Zope Corporation
participants (18)
-
Andreas Jung -
Benji York -
Chris McDonough -
Chris Withers -
Christian Theune -
David Pratt -
Dieter Maurer -
Fred Drake -
Jim Fulton -
Lennart Regebro -
Marius Gedminas -
Martijn Faassen -
Martin Aspeli -
Philipp von Weitershausen -
Roger Ineichen -
Stephan Richter -
Tres Seaver -
Wichert Akkerman