zope.publisher dependencies
I've been working on the dependencies to and from zope.publisher. Refining the dependencies should make it easier to integrate zope.pipeline when it's ready. I've noticed that nearly all packages that depend on zope.publisher depend only on a few pieces of it: - zope.publisher.interfaces - zope.publisher.browser.Browser{View|Page} - zope.publisher.browser.TestRequest One simple, low-risk refactoring I would like to do is move zope.publisher.interfaces into its own package, make zope.publisher a namespace package, and make zope.publisher depend on zope.publisher.interfaces. The __init__.py in zope.publisher is already empty, so I expect the namespace conversion to be safe. Then I'd like to refine the dependency list of various packages that only require zope.publisher.interfaces. Any objections? It is less clear what we should do with BrowserView and BrowserPage. They depend on zope.location, unlike the rest of zope.publisher, so they don't really fit there. Perhaps those two belong in a new package, "zope.publisher.browserbase". There is also the tiny new "zope.browser" package. Would it make sense to move them there? (It's hard to tell what the intent of the new package is.) I'd love to hear other suggestions. As for TestRequest, I could update the setup.py of various packages that currently depend on zope.publisher just for TestRequest. I would make zope.publisher a test-only requirement. Shane
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Shane Hathaway wrote:
I've been working on the dependencies to and from zope.publisher. Refining the dependencies should make it easier to integrate zope.pipeline when it's ready.
I've noticed that nearly all packages that depend on zope.publisher depend only on a few pieces of it:
- zope.publisher.interfaces
- zope.publisher.browser.Browser{View|Page}
- zope.publisher.browser.TestRequest
One simple, low-risk refactoring I would like to do is move zope.publisher.interfaces into its own package, make zope.publisher a namespace package, and make zope.publisher depend on zope.publisher.interfaces. The __init__.py in zope.publisher is already empty, so I expect the namespace conversion to be safe. Then I'd like to refine the dependency list of various packages that only require zope.publisher.interfaces. Any objections?
+1.
It is less clear what we should do with BrowserView and BrowserPage. They depend on zope.location, unlike the rest of zope.publisher, so they don't really fit there. Perhaps those two belong in a new package, "zope.publisher.browserbase". There is also the tiny new "zope.browser" package. Would it make sense to move them there? (It's hard to tell what the intent of the new package is.) I'd love to hear other suggestions.
zope.browser is supposed to be a zero-dependency spot for commonly-used interfaces: This package provides shared zope browser components without other dependencies. So I wouldn't move anything depending on zope.location into that package. Your new package idea sounds better.
As for TestRequest, I could update the setup.py of various packages that currently depend on zope.publisher just for TestRequest. I would make zope.publisher a test-only requirement.
Frankly, any code using a testing stub which is that heavyweight needs to be refactored. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJpAOY+gerLs4ltQ4RAjp/AJ9sbBrxvOrWkjFuypP7/16n75uUkwCgvtZW 3J0s+vDo0p1nxkxhtrFbS/A= =FUZq -----END PGP SIGNATURE-----
On Feb 24, 2009, at 9:26 AM, Tres Seaver wrote: ...
As for TestRequest, I could update the setup.py of various packages that currently depend on zope.publisher just for TestRequest. I would make zope.publisher a test-only requirement.
Frankly, any code using a testing stub which is that heavyweight needs to be refactored.
There's nothing all that heavyweight about TestRequest. Jim -- Jim Fulton Zope Corporation
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
On Feb 24, 2009, at 9:26 AM, Tres Seaver wrote: ...
As for TestRequest, I could update the setup.py of various packages that currently depend on zope.publisher just for TestRequest. I would make zope.publisher a test-only requirement. Frankly, any code using a testing stub which is that heavyweight needs to be refactored.
There's nothing all that heavyweight about TestRequest.
- - It derives from BrowserRequest, which means carrying along a *lot* of extra implementation baggage. Tests which use this class, rather than stubbing out a dummy request which provides only the API required by the application-under-test, will tend to grow unexpected / undesirable tentacles to the request implementation. - - Using TestRequest involves pulling in all of zope.publisher, a *big* dependency; Shane wants to reduce such dependencies. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJpBxw+gerLs4ltQ4RAosKAKDNmJoShgxtFrhi3rVFYqa3H+ncVACgmGU8 TOcde0xx65K1KeIopuy3hpk= =/+UR -----END PGP SIGNATURE-----
On Feb 24, 2009, at 11:12 AM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jim Fulton wrote:
On Feb 24, 2009, at 9:26 AM, Tres Seaver wrote: ...
As for TestRequest, I could update the setup.py of various packages that currently depend on zope.publisher just for TestRequest. I would make zope.publisher a test-only requirement. Frankly, any code using a testing stub which is that heavyweight needs to be refactored.
There's nothing all that heavyweight about TestRequest.
- - It derives from BrowserRequest, which means carrying along a *lot* of extra implementation baggage. Tests which use this class, rather than stubbing out a dummy request which provides only the API required by the application-under-test, will tend to grow unexpected / undesirable tentacles to the request implementation.
- - Using TestRequest involves pulling in all of zope.publisher, a *big* dependency; Shane wants to reduce such dependencies.
OK, I don't agree that zope.publisher is a big dependency, especially for code that is meant to run in the context of it. Any view (or resource) code, which is the only code who's tests would need zope.publisher, will be used in together with zope.publisher in practice. Jim -- Jim Fulton Zope Corporation
On Tuesday 24 February 2009, Jim Fulton wrote:
- - Using TestRequest involves pulling in all of zope.publisher, a *big* dependency; Shane wants to reduce such dependencies.
OK, I don't agree that zope.publisher is a big dependency, especially for code that is meant to run in the context of it. Any view (or resource) code, which is the only code who's tests would need zope.publisher, will be used in together with zope.publisher in practice.
Yep, I agree. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Jim Fulton wrote:
On Feb 24, 2009, at 11:12 AM, Tres Seaver wrote:
- - Using TestRequest involves pulling in all of zope.publisher, a *big* dependency; Shane wants to reduce such dependencies.
OK, I don't agree that zope.publisher is a big dependency, especially for code that is meant to run in the context of it.
I think it's a very subjective and relative measure. Some people call 15 packages a very big dependency. Some measure it in terms of actual features you get and think it's small. Hanno P.S. See http://hannosch.eu/dependencies/zope/zope.publisher.svg for the dependency graph ;)
Hi there, Hanno Schlichting wrote: [snip]
P.S. See http://hannosch.eu/dependencies/zope/zope.publisher.svg for the dependency graph ;)
That's a cool resource! (the general dependencies folder there) Are you removing indirect dependencies before generating the graphs? I think it is valuable to actually consider the graphs *without* such removal being done. I think doing so hides the true complexity of the dependency situation and can therefore be very misleading. At least I recall the true dependency graphs of something like zope.formlib look a lot more hairy than this: http://hannosch.eu/dependencies/zope/zope.formlib.svg I highly recommend the use graphviz's sccmap to detect clusters of strongly connected components. Circular dependencies are one of our true problems and this tool can help you identify them. It would be nice if we could have a dependency checking service that could inform us when we're going in the wrong direction; in particular when we create strongly connected clusters. Regards, Martijn
Martijn Faassen wrote:
Hanno Schlichting wrote: [snip]
P.S. See http://hannosch.eu/dependencies/zope/zope.publisher.svg for the dependency graph ;)
That's a cool resource! (the general dependencies folder there)
Are you removing indirect dependencies before generating the graphs?
Yep. Those are the version with transitive dependencies removed. I work on this mostly from a Plone perspective, where the full versions are just utterly unreadable. But for actual code refactoring and dependency reduction on the zope.* level the full versions are indeed more useful. I added them now. The full horror of formlib is available as http://hannosch.eu/dependencies/zope/zope.formlib-full.svg
I highly recommend the use graphviz's sccmap to detect clusters of strongly connected components. Circular dependencies are one of our true problems and this tool can help you identify them.
I'll check that out, thanks. Hanno
On Feb 24, 2009, at 12:33 PM, Hanno Schlichting wrote:
P.S. See http://hannosch.eu/dependencies/zope/zope.publisher.svg for the dependency graph ;)
That's cool, although wildly inaccurate. One of the things wrong with zope.publisher is that it depends on too many other things. It would be useful to try to clean that up. A while ago, I proposed a stripped down zope.publisher (zope.pub or something) that had a lot less in it, http://mail.zope.org/pipermail/zope-dev/2008-March/031170.html . I never found time to do this though. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
On Feb 24, 2009, at 12:33 PM, Hanno Schlichting wrote:
P.S. See http://hannosch.eu/dependencies/zope/zope.publisher.svg for the dependency graph ;)
That's cool, although wildly inaccurate.
What's wildly inaccurate about it? Missing transitive dependencies or something else? Regards, Martijn
On Feb 24, 2009, at 5:19 PM, Martijn Faassen wrote:
Jim Fulton wrote:
On Feb 24, 2009, at 12:33 PM, Hanno Schlichting wrote:
P.S. See http://hannosch.eu/dependencies/zope/zope.publisher.svg for the dependency graph ;)
That's cool, although wildly inaccurate.
What's wildly inaccurate about it? Missing transitive dependencies or something else?
The graph only shows direct dependencies on zope.i18n and zope.security, but there are many other direct dependencies. Jim -- Jim Fulton Zope Corporation
Hey, On Tue, Feb 24, 2009 at 11:26 PM, Jim Fulton <jim@zope.com> wrote: [snip]
The graph only shows direct dependencies on zope.i18n and zope.security, but there are many other direct dependencies.
Ah, agreed, yes, I think this is because of the transitive dependency functionality removal somehow, even though it seems to remove more than just these. Hanno's now also generating the real graphs, though: http://hannosch.eu/dependencies/zope/zope.publisher-full.svg Hanno, would you consider also generating graphs for the grokcore.* packages? Regards, Martijn
Martijn Faassen wrote:
Hanno, would you consider also generating graphs for the grokcore.* packages?
Can you point me to a buildout or virtualenv-friendly way of getting an environment with those? Than it should be rather trivial to do for me. Hanno
Hanno Schlichting wrote:
Martijn Faassen wrote:
Hanno, would you consider also generating graphs for the grokcore.* packages?
Can you point me to a buildout or virtualenv-friendly way of getting an environment with those? Than it should be rather trivial to do for me.
I'm not sure what your question is. The various grokcore.* packages are on svn.zope.org and can be installed like any others. They're also published on PyPI. Regards, Martijn
Martijn Faassen wrote:
Hey,
On Tue, Feb 24, 2009 at 11:26 PM, Jim Fulton <jim@zope.com> wrote: [snip]
The graph only shows direct dependencies on zope.i18n and zope.security, but there are many other direct dependencies.
Ah, agreed, yes, I think this is because of the transitive dependency functionality removal somehow, even though it seems to remove more than just these. Hanno's now also generating the real graphs, though:
http://hannosch.eu/dependencies/zope/zope.publisher-full.svg
I see in that graph a number of dependencies that are pulled in just for specifications. For instance, zope.publisher doesn't really need the Location class, it only needs ILocation. Just brainstorming, but I wonder if we shouldn't split at least the following packages into specification and implementation packages: - zope.location - zope.security - zope.i18n - zope.publisher - zope.component That way various packages could use i18n interfaces without pulling in pytz, or could use location interfaces without pulling in zope.proxy, and so on. Brainstorming deeper: we could apply a naming convention where the specification package is named with the suffix "spec", so zope.location would be split into zope.location and zope.locationspec. Shane
On Tuesday 24 February 2009, Shane Hathaway wrote:
Brainstorming deeper: we could apply a naming convention where the specification package is named with the suffix "spec", so zope.location would be split into zope.location and zope.locationspec.
what about zope.ilocation? Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Stephan Richter wrote:
On Tuesday 24 February 2009, Shane Hathaway wrote:
Brainstorming deeper: we could apply a naming convention where the specification package is named with the suffix "spec", so zope.location would be split into zope.location and zope.locationspec.
what about zope.ilocation?
Maybe. I'd lean toward "zope.locationspec" because it would appear right after zope.location in a sorted list, making it more apparent that they are related. Shane
On Wed, Feb 25, 2009 at 04:05, Shane Hathaway <shane@hathawaymix.org> wrote:
Stephan Richter wrote:
On Tuesday 24 February 2009, Shane Hathaway wrote:
Brainstorming deeper: we could apply a naming convention where the specification package is named with the suffix "spec", so zope.location would be split into zope.location and zope.locationspec.
what about zope.ilocation?
Maybe. I'd lean toward "zope.locationspec" because it would appear right after zope.location in a sorted list, making it more apparent that they are related.
zope.location.interfaces? -- Lennart Regebro: Pythonista, Barista, Notsotrista. http://regebro.wordpress.com/ +33 661 58 14 64
On Wed, Feb 25, 2009 at 12:39 PM, Lennart Regebro <regebro@gmail.com> wrote:
On Wed, Feb 25, 2009 at 04:05, Shane Hathaway <shane@hathawaymix.org> wrote:
Stephan Richter wrote:
On Tuesday 24 February 2009, Shane Hathaway wrote:
Brainstorming deeper: we could apply a naming convention where the specification package is named with the suffix "spec", so zope.location would be split into zope.location and zope.locationspec.
what about zope.ilocation?
Maybe. I'd lean toward "zope.locationspec" because it would appear right after zope.location in a sorted list, making it more apparent that they are related.
zope.location.interfaces?
This will not make any change in dependency graph unless zope.location become a namespace package. -- Baiju M
On Wed, Feb 25, 2009 at 08:46, Baiju M <mbaiju@zeomega.net> wrote:
This will not make any change in dependency graph unless zope.location become a namespace package.
Yeah, that was what I was thinking, but I just realized that it might be tricky to have both a zope.location and a zope.location.interfaces. I haven't tried. Anyway, it was just an idea. -- Lennart Regebro: Pythonista, Barista, Notsotrista. http://regebro.wordpress.com/ +33 661 58 14 64
Hi there, In general splitting up interfaces from implementations would help pluggability. That said, I'm not very much in favor of such a split up of so many packages at the moment, as the case for pluggability will need to be made first for each new package created. I think we should focus on reducing circular dependencies and pluggability improvements where there are concrete benefits, before we start to think of more general patterns. I'd therefore like to see more concrete proposals (or actual implementation) for rearranging of code in packages instead of general pattern proposals. I know my discussion of splitting off ZCML implementations out of packages foundered because many people would only see the general proposal as acceptable but then found it *unacceptable* because it'd generate so many new packages. :) Regards, Martijn
Jim Fulton wrote:
The graph only shows direct dependencies on zope.i18n and zope.security, but there are many other direct dependencies.
Right, that's what http://hannosch.eu/dependencies/zope/zope.publisher-full.svg is for. Now showing the full set of direct dependencies. I find both the reduced and full version to give useful information depending on the question one has. All those graphs actually do have a meaningful color coding as well: If a node has a yellow background it is a direct dependency of the package in question. Hanno
On Feb 24, 2009, at 3:08 AM, Shane Hathaway wrote:
I've been working on the dependencies to and from zope.publisher. Refining the dependencies should make it easier to integrate zope.pipeline when it's ready.
Can you elaborate on this a bit?
I've noticed that nearly all packages that depend on zope.publisher depend only on a few pieces of it:
- zope.publisher.interfaces
- zope.publisher.browser.Browser{View|Page}
- zope.publisher.browser.TestRequest
I'd like to turn this around a little bit. Only browser-based code should depend on zope.publisher. This seems like a very reasonable dependency. It's like wxwindows UI code depending on wxwindows. Maybe the browser code should be factored out of the packages that depend on zoep.publisher so that only *that* code has the dependency and non-browser code doesn't. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
On Feb 24, 2009, at 3:08 AM, Shane Hathaway wrote:
I've been working on the dependencies to and from zope.publisher. Refining the dependencies should make it easier to integrate zope.pipeline when it's ready.
Can you elaborate on this a bit?
He has, though perhaps not in an obvious place for you: http://shane.willowrise.com/archives/repozublisher/ http://shane.willowrise.com/archives/redesign-of-zopepublisher/ http://shane.willowrise.com/archives/zopepipeline/ http://shane.willowrise.com/archives/limits-of-zopepipeline/
I've noticed that nearly all packages that depend on zope.publisher depend only on a few pieces of it:
- zope.publisher.interfaces
- zope.publisher.browser.Browser{View|Page}
- zope.publisher.browser.TestRequest
I'd like to turn this around a little bit. Only browser-based code should depend on zope.publisher. This seems like a very reasonable dependency. It's like wxwindows UI code depending on wxwindows. Maybe the browser code should be factored out of the packages that depend on zoep.publisher so that only *that* code has the dependency and non-browser code doesn't.
Shane, how integrated is code that relies on the pieces in zope.publisher you identified into their own packages? I have the impression it'll be much harder to go that way than factor bits out of zope.publisher instead. Especially as zope.publisher contains stuff that Shane has no use for with zope.pipeline, and we'd still be pulling it in if we didn't do the refactoring. We got two kinds of browser-based code we should distinguish between: * ZMI code * framework code that supports the browser To get rid of ZMI code, a pattern that works fairly well is to refactor everything *else* out of the package and leave the ZMI code in its original location, with backwards compatibility imports in place. zope.app.* packages frequently can get this kind of treatment. Other framework code that supports the browser is much like any other framework code. Some packages need to be aware of the browser in order to perform their role as framework component at all. If those packages can rely on *less* code that would be an improvement. I'm not very much in favor of making these sub-packages of zope.publisher though, as to me a sub-package structure tends to make an implication that it relies on the outer package, which in this case it doesn't. I'd rather see zope.browser take this role. Perhaps the current zope.browser package doesn't have the right name? Regards, Martijn
On Feb 24, 2009, at 11:46 AM, Martijn Faassen wrote:
Jim Fulton wrote:
On Feb 24, 2009, at 3:08 AM, Shane Hathaway wrote:
I've been working on the dependencies to and from zope.publisher. Refining the dependencies should make it easier to integrate zope.pipeline when it's ready.
Can you elaborate on this a bit?
He has, though perhaps not in an obvious place for you:
http://shane.willowrise.com/archives/repozublisher/
http://shane.willowrise.com/archives/redesign-of-zopepublisher/
I disagree strongly with many of the assertions made in these articles. (I can't judge the pipeline proposal, since it is only fleshed out in code.) While I do think zope.publisher has some problems, they aren't the same problems that shane sees. ...
I'd like to turn this around a little bit. Only browser-based code should depend on zope.publisher. This seems like a very reasonable dependency. It's like wxwindows UI code depending on wxwindows. Maybe the browser code should be factored out of the packages that depend on zoep.publisher so that only *that* code has the dependency and non-browser code doesn't.
Shane, how integrated is code that relies on the pieces in zope.publisher you identified into their own packages? I have the impression it'll be much harder to go that way than factor bits out of zope.publisher instead.
I'd like to see see some specific examples. In general, in Zope 3, we've advocated a separation of model and application code from presentation code. Presentation code is going to depend on whatever presentation framework it uses.
Especially as zope.publisher contains stuff that Shane has no use for with zope.pipeline, and we'd still be pulling it in if we didn't do the refactoring.
I'm not sure why this matters. BTW, I suspect we're more concerned about odd dependencies *of* zope.publisher, like zope.location. It might be better to focus on some of those. I'd also be in favor of separating out less central parts, like support for xml-rpc.
We got two kinds of browser-based code we should distinguish between:
* ZMI code
There shouldn't be anything in zope.publisher that depends on the ZMI.
* framework code that supports the browser
To get rid of ZMI code, a pattern that works fairly well is to refactor everything *else* out of the package and leave the ZMI code in its original location, with backwards compatibility imports in place. zope.app.* packages frequently can get this kind of treatment.
Other framework code that supports the browser is much like any other framework code. Some packages need to be aware of the browser in order to perform their role as framework component at all. If those packages can rely on *less* code that would be an improvement.
I'm not sure what you're saying. If an application package has presentation code mixed into it and if there is a desire to use that application code in a context without presentation, I'd separate the presentation code from the application code. The presentation code would depend on zope.publisher. The application code wouldn't.
I'm not very much in favor of making these sub-packages of zope.publisher though, as to me a sub-package structure tends to make an implication that it relies on the outer package, which in this case it doesn't. I'd rather see zope.browser take this role. Perhaps the current zope.browser package doesn't have the right name?
I don't know what you mean by "these" above. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
I disagree strongly with many of the assertions made in these articles. (I can't judge the pipeline proposal, since it is only fleshed out in code.) While I do think zope.publisher has some problems, they aren't the same problems that shane sees.
What are the problems with zope.publisher as you see it? Shane
On Feb 24, 2009, at 12:59 PM, Shane Hathaway wrote:
Jim Fulton wrote:
I disagree strongly with many of the assertions made in these articles. (I can't judge the pipeline proposal, since it is only fleshed out in code.) While I do think zope.publisher has some problems, they aren't the same problems that shane sees.
What are the problems with zope.publisher as you see it?
- It has too many dependencies. Obviously. - It has too much in it. This contributes to the large number of dependencies. - I think the attempt to support multiple protocols through different request types has turned out to be a mistake. I think it was a reasonable approach to try, but I don't think it turned out well, in part because it seems too heavy. Last year, I implemented some Ajax support framework without anything like the complexity that's in the xml-rpc support. I did end up defining a new request marker type to facilitate component lookup, but I was able to do this in a very light- weight way. - It's not well enough documented. While I think there's merit in doing some things at the WSGI level, I remain pretty happy with the publication interface for separatating generic publisher functions from application policies. I which the use of this API was better communicated and understood. A less major complaint is some baggage from the past. There are a number of request features that I never use and tend to forget about. The biggest of these is the special form data unmarshalling and url manipulation support. (I was amused to read in your introduction to your pipeline proposal that people wanted to know the answer to the question: "When does Zope respect the :method form variable?". :) Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
- It's not well enough documented. While I think there's merit in doing some things at the WSGI level, I remain pretty happy with the publication interface for separatating generic publisher functions from application policies. I which the use of this API was better communicated and understood.
I hope you're not asking me to write documentation for zope.publisher :-), because I only understand the mechanics. The overall scope and purpose is cloudy to me. In particular, I don't understand how the publication interface is actually generic. Does it fit the needs of anything other than Zope?
A less major complaint is some baggage from the past. There are a number of request features that I never use and tend to forget about. The biggest of these is the special form data unmarshalling and url manipulation support. (I was amused to read in your introduction to your pipeline proposal that people wanted to know the answer to the question: "When does Zope respect the :method form variable?". :)
FWIW, that particular functionality has been pulled out twice now, both in repoze.monty and zope.httpform. As a baby step, we could make zope.publisher depend on zope.httpform. (I made zope.httpform without knowing repoze.monty already existed, but zope.httpform has more tests and interfaces and it's hosted on svn.zope.org, so I think zope.httpform is worth keeping.) Shane
On Mar 6, 2009, at 2:18 AM, Shane Hathaway wrote:
Jim Fulton wrote:
- It's not well enough documented. While I think there's merit in doing some things at the WSGI level, I remain pretty happy with the publication interface for separatating generic publisher functions from application policies. I which the use of this API was better communicated and understood.
I hope you're not asking me to write documentation for zope.publisher :-),
I wasn't asking you to do anything. :)
because I only understand the mechanics. The overall scope and purpose is cloudy to me. In particular, I don't understand how the publication interface is actually generic. Does it fit the needs of anything other than Zope?
I believe so. I think there are enough hooks there for pretty much any application.
A less major complaint is some baggage from the past. There are a number of request features that I never use and tend to forget about. The biggest of these is the special form data unmarshalling and url manipulation support. (I was amused to read in your introduction to your pipeline proposal that people wanted to know the answer to the question: "When does Zope respect the :method form variable?". :)
FWIW, that particular functionality has been pulled out twice now, both in repoze.monty and zope.httpform. As a baby step, we could make zope.publisher depend on zope.httpform. (I made zope.httpform without knowing repoze.monty already existed, but zope.httpform has more tests and interfaces and it's hosted on svn.zope.org, so I think zope.httpform is worth keeping.)
I'd rather leave zope.publsiher more or less alone, but develop a new thing that has the basic/core functionality we need and refactor zope.publisher to use that. I'd also like to use or be compatible with WebOb on that. I'd prefer to do this at PyCon where I'll have time to pay attention and where there will be higher bandwidth communication. I'm sympathetic to Martijn's desire not to slow you down, it's not clear to me why you need changes to zope.publisher. If it's just a matter of reducing dependencies, I don't think that's a blocker for any work you might want to do. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
I'd rather leave zope.publsiher more or less alone, but develop a new thing that has the basic/core functionality we need and refactor zope.publisher to use that. I'd also like to use or be compatible with WebOb on that. I'd prefer to do this at PyCon where I'll have time to pay attention and where there will be higher bandwidth communication.
I look forward to PyCon! :-) Shane
Hey, Jim Fulton wrote: [snip]
I'd rather leave zope.publsiher more or less alone, but develop a new thing that has the basic/core functionality we need and refactor zope.publisher to use that.
I had the impression Shane was doing that; i.e. building zope.pipeline factoring bits out of the publisher but not really changing the publisher very much itself, creating a new structure instead. Shane, is that correct?
I'd also like to use or be compatible with WebOb on that. I'd prefer to do this at PyCon where I'll have time to pay attention and where there will be higher bandwidth communication.
+1 on the WebOb compatibility. I imagine it should be possible to reconstruct our request and response interfaces on top of WebOb's. We should also think of ways so that code can get the WebOb request directly instead of having to go through these adapters. I'd also be much in favor of reusing WebOb's implementation, not just the APIs. I think it could help us replace part of the publisher.
I'm sympathetic to Martijn's desire not to slow you down, it's not clear to me why you need changes to zope.publisher. If it's just a matter of reducing dependencies, I don't think that's a blocker for any work you might want to do.
Thanks. I'm very much in favor of you guys talking at PyCon if that's the best way to make progress. I'm also looking forward to seeing Shane again at PyCon after meeting him so briefly all these years ago when I was still young and nimble. So I'm glad to hear he's going to make it! I'm also happy I'll get to see you again as well! Regards, Martijn
Martijn Faassen wrote:
Jim Fulton wrote: [snip]
I'd rather leave zope.publsiher more or less alone, but develop a new thing that has the basic/core functionality we need and refactor zope.publisher to use that.
I had the impression Shane was doing that; i.e. building zope.pipeline factoring bits out of the publisher but not really changing the publisher very much itself, creating a new structure instead. Shane, is that correct?
Yes. I think Jim and I are saying the same thing there. The current code in zope.pipeline doesn't reflect that thinking, though, because we really need to divide zope.publisher somehow before a project like this makes sense.
+1 on the WebOb compatibility. I imagine it should be possible to reconstruct our request and response interfaces on top of WebOb's. We should also think of ways so that code can get the WebOb request directly instead of having to go through these adapters. I'd also be much in favor of reusing WebOb's implementation, not just the APIs. I think it could help us replace part of the publisher.
That should be a great thing to work on in a sprint.
Thanks. I'm very much in favor of you guys talking at PyCon if that's the best way to make progress. I'm also looking forward to seeing Shane again at PyCon after meeting him so briefly all these years ago when I was still young and nimble. So I'm glad to hear he's going to make it! I'm also happy I'll get to see you again as well!
I'll be the one who arrives Monday afternoon without much hair on top. :-) Shane
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Shane Hathaway wrote:
Jim Fulton wrote:
- It's not well enough documented. While I think there's merit in doing some things at the WSGI level, I remain pretty happy with the publication interface for separatating generic publisher functions from application policies. I which the use of this API was better communicated and understood.
I hope you're not asking me to write documentation for zope.publisher :-), because I only understand the mechanics. The overall scope and purpose is cloudy to me. In particular, I don't understand how the publication interface is actually generic. Does it fit the needs of anything other than Zope?
A less major complaint is some baggage from the past. There are a number of request features that I never use and tend to forget about. The biggest of these is the special form data unmarshalling and url manipulation support. (I was amused to read in your introduction to your pipeline proposal that people wanted to know the answer to the question: "When does Zope respect the :method form variable?". :)
FWIW, that particular functionality has been pulled out twice now, both in repoze.monty and zope.httpform. As a baby step, we could make zope.publisher depend on zope.httpform. (I made zope.httpform without knowing repoze.monty already existed, but zope.httpform has more tests and interfaces and it's hosted on svn.zope.org, so I think zope.httpform is worth keeping.)
Yup. repoze.monty should become just a BBB wrapper. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJsUNS+gerLs4ltQ4RAlsAAKCQNH3YTAb4NjDq4VOW/ZxU6/pQLgCgzGBW pMIHEqDDxxUno1GdRRx3sEw= =E5aV -----END PGP SIGNATURE-----
Hey, On Tue, Feb 24, 2009 at 6:39 PM, Jim Fulton <jim@zope.com> wrote: [Shane's blog entries]
I disagree strongly with many of the assertions made in these articles. (I can't judge the pipeline proposal, since it is only fleshed out in code.) While I do think zope.publisher has some problems, they aren't the same problems that shane sees.
That's not a very constructive statement. :) Which assertions do you disagree with, and why? And what problems do you think the publication machinery of Zope has? The main problem I have with the zope publication machinery is that after all these years of using it I *still* get lost in it regularly. A more regular architecture that can be traced more easily would not only allow better understanding on my part, but might also allow us to more easily selectively replace or remove bits of it.
Especially as zope.publisher contains stuff that Shane has no use for with zope.pipeline, and we'd still be pulling it in if we didn't do the refactoring.
I'm not sure why this matters.
If Grok wants to switch to zope.pipeline it'd be pulling in code that would never be used. That code is a harmful distraction.
BTW, I suspect we're more concerned about odd dependencies *of* zope.publisher, like zope.location. It might be better to focus on some of those.
Some of the bits that Shane is talking about *are* those dependencies on zope.location. zope.location is a pretty central bit that ends up being used by huge parts of the framework. We've already regularized its dependencies a few weeks ago so it's not as terrible to depend on it as it used to be.
To get rid of ZMI code, a pattern that works fairly well is to refactor everything *else* out of the package and leave the ZMI code in its original location, with backwards compatibility imports in place. zope.app.* packages frequently can get this kind of treatment.
Other framework code that supports the browser is much like any other framework code. Some packages need to be aware of the browser in order to perform their role as framework component at all. If those packages can rely on *less* code that would be an improvement.
I'm not sure what you're saying.
I'm just trying to make clear we have framework code that builds on bits of zope.publisher that might not need to build on as much of it as it does now. I think framework code that is presentation-related and code that is there to build a specific user interface the ZMI can receive a different treatment in dependency analysis.
I'm not very much in favor of making these sub-packages of zope.publisher though, as to me a sub-package structure tends to make an implication that it relies on the outer package, which in this case it doesn't. I'd rather see zope.browser take this role. Perhaps the current zope.browser package doesn't have the right name?
I don't know what you mean by "these" above.
The three bits Shane is talking about: - zope.publisher.interfaces - zope.publisher.browser.Browser{View|Page} - zope.publisher.browser.TestRequest I'd prefer them to be off in a package by themselves if they can be instead of being sub-packages of zope.publisher, since they don't depend on zope.publisher. Regards, Martijn
Martijn Faassen wrote:
The main problem I have with the zope publication machinery is that after all these years of using it I *still* get lost in it regularly. A more regular architecture that can be traced more easily would not only allow better understanding on my part, but might also allow us to more easily selectively replace or remove bits of it.
+1. As I recall, we tried to build a regular architecture in zope.publisher using the IPublication interface, but the publisher machinery is still painfully difficult to understand without extensive study. I think a pipeline design will make the publisher a lot easier for everyone to understand because the pipeline design seems to keep concerns closer together. For example, I've made a "traversal" module in zope.pipeline which has nearly all of the traversal logic in one place and almost nothing else. Its code came from at least 4 scattered modules. Now, in theory, when people want to understand traversal, they will usually only need to understand one module. Shane
On Feb 24, 2009, at 1:55 PM, Shane Hathaway wrote:
Martijn Faassen wrote:
The main problem I have with the zope publication machinery is that after all these years of using it I *still* get lost in it regularly. A more regular architecture that can be traced more easily would not only allow better understanding on my part, but might also allow us to more easily selectively replace or remove bits of it.
+1. As I recall, we tried to build a regular architecture in zope.publisher using the IPublication interface, but the publisher machinery is still painfully difficult to understand without extensive study.
Maybe, but I find that people confuse the machinery in zope.publisher with a bunch of additional and very confusing machinery in various zope.app packages. The publisher itself is pretty simple. I think this is illustrated by paste.txt in the zope.publisher package. That isn't to say there might not be better models. Hopefully, I'll find time to study your pipeline ideas. I wish there was a proposal I could read rather than reading code. Jim -- Jim Fulton Zope Corporation
Jim Fulton wrote:
Maybe, but I find that people confuse the machinery in zope.publisher with a bunch of additional and very confusing machinery in various zope.app packages. The publisher itself is pretty simple. I think this is illustrated by paste.txt in the zope.publisher package. That isn't to say there might not be better models.
I see what you're saying and I agree that the zope.publisher package by itself is fairly simple. However, zope.publisher by itself isn't very useful (except in tests), yet zope.publisher doesn't provide all the kinds of extension facilities that other packages require. This has led to messy code, starting with zope.app.publication. Furthermore, I'm not sure anyone can foresee all the ways people need to extend zope.publisher. I think we need a more powerful abstraction.
Hopefully, I'll find time to study your pipeline ideas. I wish there was a proposal I could read rather than reading code.
I am close to being able to formulate a proposal. The concept is simple: move most publisher functionality into WSGI pipeline stages, similar to Repoze. WSGI pipeline stages are very simple and powerful. I had to work out a way to build pipelines with replaceable stages, but I think I've almost solved that. Shane
Jim Fulton wrote:
On Feb 24, 2009, at 3:08 AM, Shane Hathaway wrote:
I've been working on the dependencies to and from zope.publisher. Refining the dependencies should make it easier to integrate zope.pipeline when it's ready.
Can you elaborate on this a bit?
I've been discussing zope.pipeline on my blog. I am attempting to rebuild the publisher to make it easier to understand and customize. I think it's working out pretty well. zope.pipeline is intended to replace most of the implementation code in zope.publisher and zope.app.publication. To accomplish that, it looks like I ought to separate the implementations in zope.publisher from the specifications. Separating the zope.publisher.interfaces package would mostly accomplish that. After thinking this over last night, I realize that the idea to move BrowserView, BrowserPage, and TestRequest is driven by the desire to clarify the dependency graph. That's more complex than what I'm trying to do and I don't think I need to do that for zope.pipeline, so I'm going to withdraw from that part of the discussion for now.
I've noticed that nearly all packages that depend on zope.publisher depend only on a few pieces of it:
- zope.publisher.interfaces
- zope.publisher.browser.Browser{View|Page}
- zope.publisher.browser.TestRequest
I'd like to turn this around a little bit. Only browser-based code should depend on zope.publisher. This seems like a very reasonable dependency. It's like wxwindows UI code depending on wxwindows. Maybe the browser code should be factored out of the packages that depend on zoep.publisher so that only *that* code has the dependency and non-browser code doesn't.
I think things are already pretty well factored in that sense. Take zope.formlib, for example. It wouldn't make sense to separate zope.formlib into an abstract package and a browser package, because zope.formlib only makes sense for browsers. Shane
Hey, Shane Hathaway wrote: [snip]
After thinking this over last night, I realize that the idea to move BrowserView, BrowserPage, and TestRequest is driven by the desire to clarify the dependency graph. That's more complex than what I'm trying to do and I don't think I need to do that for zope.pipeline, so I'm going to withdraw from that part of the discussion for now.
If you can make progress without doing so now, by all means. I think we do need to get back to this though eventually. It'd be bad to pull in code that actually doesn't get used when you use zope.pipeline. Regards, Martijn
On Feb 24, 2009, at 12:44 PM, Shane Hathaway wrote:
Jim Fulton wrote:
On Feb 24, 2009, at 3:08 AM, Shane Hathaway wrote:
I've been working on the dependencies to and from zope.publisher. Refining the dependencies should make it easier to integrate zope.pipeline when it's ready. Can you elaborate on this a bit?
I've been discussing zope.pipeline on my blog.
I just looked and I don't see many specifics. I think I have to look at the code and I don't have time for that atm.
I am attempting to rebuild the publisher to make it easier to understand and customize. I think it's working out pretty well. zope.pipeline is intended to replace most of the implementation code in zope.publisher and zope.app.publication.
If it is mixing those 2, them I'm not too impressed and I think those two packages have very different concerns. In any case, if what you you really want to do is to standardize some common APIs that a number of different publishing implementations can use, I'm fine with that. It should be a new package that has just those APIs and that zope.publisher could import. Jim -- Jim Fulton Zope Corporation
On Tuesday 24 February 2009, Shane Hathaway wrote:
I've noticed that nearly all packages that depend on zope.publisher depend only on a few pieces of it:
- zope.publisher.interfaces
Can you give examples?
- zope.publisher.browser.Browser{View|Page}
- zope.publisher.browser.TestRequest
Packages that depend on those classes usually more or less implicitly depend on zope.publisher. So the split might be arbitrary for this example.
One simple, low-risk refactoring I would like to do is move zope.publisher.interfaces into its own package, make zope.publisher a namespace package, and make zope.publisher depend on zope.publisher.interfaces. The __init__.py in zope.publisher is already empty, so I expect the namespace conversion to be safe. Then I'd like to refine the dependency list of various packages that only require zope.publisher.interfaces. Any objections?
I want to see some motivation, because I fail to see how this helps.
It is less clear what we should do with BrowserView and BrowserPage. They depend on zope.location, unlike the rest of zope.publisher, so they don't really fit there. Perhaps those two belong in a new package, "zope.publisher.browserbase".
I do agree moving BrowserView and BrowserPage out of the publisher because they introduce the zope.location dependency.
There is also the tiny new "zope.browser" package. Would it make sense to move them there? (It's hard to tell what the intent of the new package is.) I'd love to hear other suggestions.
I think the purpose of the package is still defining itself. I think it will be defined by the things that we move into it. I am very tempted to say that it is a good home for BrowserView and BrowserPage.
As for TestRequest, I could update the setup.py of various packages that currently depend on zope.publisher just for TestRequest. I would make zope.publisher a test-only requirement.
TestRequest does not add any additional dependencies to the system, so what's the point? It will depend on zope.publisher.browser anyways. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Stephan Richter wrote:
On Tuesday 24 February 2009, Shane Hathaway wrote:
I've noticed that nearly all packages that depend on zope.publisher depend only on a few pieces of it:
- zope.publisher.interfaces
Can you give examples?
- zope.publisher.browser.Browser{View|Page}
- zope.publisher.browser.TestRequest
Packages that depend on those classes usually more or less implicitly depend on zope.publisher. So the split might be arbitrary for this example.
My understanding is that Shane is working on an alternative publisher, zope.pipeline, that doesn't need this other code. Is that correct, Shane? Regards, Martijn
On Tuesday 24 February 2009, Martijn Faassen wrote:
Packages that depend on those classes usually more or less implicitly depend on zope.publisher. So the split might be arbitrary for this example.
My understanding is that Shane is working on an alternative publisher, zope.pipeline, that doesn't need this other code. Is that correct, Shane?
I see. He only needs the interfaces, so that he can write compatible code. And he is doing some rest-like stuff, so he does not need browser. I would only factor out the browser code, if it introduces additional dependencies. In general I am worried that we are creating too many packages. However, here is my order of importance: 1. Minimize dependencies. 2. Minimize packages. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
Hey, On Tue, Feb 24, 2009 at 6:33 PM, Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote:
In general I am worried that we are creating too many packages. However, here is my order of importance:
1. Minimize dependencies. 2. Minimize packages.
+1 I think on the longer term better dependencies can allow us to remove a great number of packages from the Zope Framework, so I'm not too worried about the minimization of packages right now. To properly reduce dependencies you need to go about it in an intelligent way anyway and do some actual analysis of code, so the new packages that are created tend to make sense. Fundamental goals should be: * Make the code more comprehensible * Make the code more reusable The dependency refactoring is not only an effort to make the dependencies between packages a nicer tree (and thus encourage reuse of more packages and easier understandability), but also results in trees that have less code altogether. I think the last point is very important. Less code might not be reflected directly in the amount of packages, but is a huge win anyway. Less code makes the code that remains typically far more understandable and because it pulls in less baggage also more reusable. Less code should be evaluated for the whole framework, but also for each node in the dependency tree. As an example, zope.site is a new package but contains bits of zope.app.component and zope.app.folder both. Together with the creation of zope.container and moving stuff from zope.app.component to zope.security, we can probably eventually get rid of zope.app.component altogether, along with zope.app.folder and zope.app.container. That is a net minimization of packages, and what's more important, also less code to have to understand. Regards, Martijn
Hey Shane, +1 on separating out zope.publisher.interfaces, as it seems low-hanging fruit. Shane Hathaway wrote:
It is less clear what we should do with BrowserView and BrowserPage. They depend on zope.location, unlike the rest of zope.publisher, so they don't really fit there. Perhaps those two belong in a new package, "zope.publisher.browserbase". There is also the tiny new "zope.browser" package. Would it make sense to move them there? (It's hard to tell what the intent of the new package is.) I'd love to hear other suggestions.
Perhaps zope.browser is the most straightforward location, especially given the names of the views involved. Even if zope.browser has unclear intent now it'll gain more clear intent from this. That said, zope.browser currently doesn't depend on zope.location and it would need to gain this as a dependency.
As for TestRequest, I could update the setup.py of various packages that currently depend on zope.publisher just for TestRequest. I would make zope.publisher a test-only requirement.
I would prefer if we could make TestRequest also go somewhere else (zope.browser?) instead of making zope.publisher a test-only requirement. While that step would be an improvement, I think the greater improvement would be to find a way to reduce test-only requirements. Regards, Martijn
participants (8)
-
Baiju M -
Hanno Schlichting -
Jim Fulton -
Lennart Regebro -
Martijn Faassen -
Shane Hathaway -
Stephan Richter -
Tres Seaver