Hi all, I've put up a draft of a zope.pipeline proposal: http://wiki.zope.org/zope3/ZopePipeline The proposal is intended to explain my thoughts on the subject more thoroughly. Shane
2009/2/25 Shane Hathaway <shane@hathawaymix.org>:
Hi all,
I've put up a draft of a zope.pipeline proposal:
http://wiki.zope.org/zope3/ZopePipeline
The proposal is intended to explain my thoughts on the subject more thoroughly.
I like the idea, as it definetely cleans up request processing and publication related things in Zope, so if you'll manage to keep full backward compatibility, a strong +1 on that. :) The pipeline way makes request processing process much more pluggable and componentized. Also, how easy is to integrate existing non-zopeish WSGI middlewares into the zope.pipeline? Like some resource injectors or XHTML slimmers and so on. It would be really great to be able to do that with single ZCML directive. -- WBR, Dan Korostelev
Dan Korostelev wrote:
Also, how easy is to integrate existing non-zopeish WSGI middlewares into the zope.pipeline? Like some resource injectors or XHTML slimmers and so on. It would be really great to be able to do that with single ZCML directive.
You can do that with two ZCML directives. I probably need to elaborate on the directives in the proposal. Shane
Shane Hathaway wrote:
Dan Korostelev wrote:
Also, how easy is to integrate existing non-zopeish WSGI middlewares into the zope.pipeline? Like some resource injectors or XHTML slimmers and so on. It would be really great to be able to do that with single ZCML directive.
You can do that with two ZCML directives. I probably need to elaborate on the directives in the proposal.
Agreed, I'd like to hear more about these. Regards, Martijn
Hi Shane,
Thanks for putting this up! In general, I think your goals are very worthy. I hope we'll end up with more re-usable end/middleware that can be used by others, including Zope 2 applications like Plone, as a result of this. Unifying the Zope 2 and Zope 3 publishers a bit more would also be quite interesting. In the proposal, you say: "The zope.pipeline project is also working out how to use ZCML and the Zope component architecture to build a WSGI pipeline that developers can easily modify to suit their applications." I'm used to using Paste Deploy ini files to configure a WSGI pipeline. Is this simply an alternative to that? If so, do we really need our own alternative, or could we try to use the Paste Deploy stuff directly? I am a little worried about the conceptual overhead of having both the ZCA and the WSGI pipeline provide "swappability" services to application builders. It feels like those two things overlap somewhat in scope. Also, looking at your endware, there are some that seem to overlap with Repoze stuff. Is this on purpose? I think the relationship with Repoze should be made more clear in the proposal. virtual_host -- is this the same as repoze.vhm? retry -- is this the same as repoze.retry? create_request -- should this maybe have some compatibility with WebOb requests? switch_pipeline -- could this be made non-Zope specific? It sounds useful. log -- both repoze and paste have logging middleware, IIRC open_root -- I thought repoze had something similar, but I may be wrong clean_transaction -- is this not the same as repoze.tm2? set_site -- sounds useful event -- also sounds useful; I've had use cases other than setSite() that require pre- and post-traversal event handle_error -- again, I thought Repoze had something like this end_transaction -- sounds like the other end of repoze.tm2 authenticate -- sounds like repoze.who? fix_relative_links -- sounds generally useful outside Zope as well Cheers, Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin Aspeli wrote:
Thanks for putting this up! In general, I think your goals are very worthy. I hope we'll end up with more re-usable end/middleware that can be used by others, including Zope 2 applications like Plone, as a result of this. Unifying the Zope 2 and Zope 3 publishers a bit more would also be quite interesting.
In the proposal, you say:
"The zope.pipeline project is also working out how to use ZCML and the Zope component architecture to build a WSGI pipeline that developers can easily modify to suit their applications."
I'm used to using Paste Deploy ini files to configure a WSGI pipeline. Is this simply an alternative to that? If so, do we really need our own alternative, or could we try to use the Paste Deploy stuff directly?
I am a little worried about the conceptual overhead of having both the ZCA and the WSGI pipeline provide "swappability" services to application builders. It feels like those two things overlap somewhat in scope.
Compared to the paste.ini format, I find ZCA + ZCML both too flexible (I don't need type-based lookup) and not flexible enough (I can't configure policy values) for the task of configuring a WSGI pipeline.
Also, looking at your endware, there are some that seem to overlap with Repoze stuff. Is this on purpose? I think the relationship with Repoze should be made more clear in the proposal.
virtual_host -- is this the same as repoze.vhm?
repoze.vhm allows two different modes for spelling the virtual host: the "classic" Zope2 spelling, and a spelling which uses extra headers added by the proxy. In both cases, it works by fixing up the WSGI environment such that the "stock" behavior is correct, without requiring the traversal majyk performed by the VHM.
retry -- is this the same as repoze.retry?
Probably. repoze.retry allows configuring both the number of retries and the kind of exception in the .ini file. The exception type makes it possible to handle stuff like PostgreSQL's "serialization error" with retry.
create_request -- should this maybe have some compatibility with WebOb requests?
This feature is provided by the "endpoint" machinery in repoze.zope2 and repoze.bfg.
switch_pipeline -- could this be made non-Zope specific? It sounds useful.
This might be like the Paste 'composite' / "pants leg" dispatch, or like Routes, depending. repoze.bfg optionally allows using Routes before / in place of traversal from the single root: http://static.repoze.org/bfgdocs/narr/urldispatch.html
log -- both repoze and paste have logging middleware, IIRC
repoze just uses the paste logging stuff for the access log. repoze.debug supplies the equivalent of the "big M" / trace log, as well as a postmortem debugger, and the moral equivalent of the Zope2 Control Panel's "Debug" tab.
open_root -- I thought repoze had something similar, but I may be wrong
This feature is provided by the "endpoint" machinery in repoze.zope2 and repoze.bfg. E.g., see the probably over-geeral framework in repoze.obob (repoze.zope2 works by supplying a different "helper" to obob): http://svn.repoze.org/repoze.obob/trunk/repoze/obob/publisher.py The repoze.bfg traversal machinery is documented here: http://static.repoze.org/bfgdocs/narr/traversal.html
clean_transaction -- is this not ithe same as repoze.tm2?
I can't imagine a need for more than one WSGI middleware integration with the 'transaction' package. Shane's proposal here is for a layer which starts and aborts the transaction at a given layer in the traversal stack: commit would have to happen at another layer further to the "right." I don't know why Shane would split the two apart.
set_site -- sounds useful
This feature is provided by the "endpoint" machinery in repoze.bfg; repoze.zope2 just uses the stock traversal-based hook.
event -- also sounds useful; I've had use cases other than setSite() that require pre- and post-traversal event
I actually think that the case for such events is dubious, given that you configure the pipeline yourself, and can thus do the same integration via middleware as you would with events, without the downside of needing to suppress unwanted event handlers.
handle_error -- again, I thought Repoze had something like this
repoze.errorlog supplies a Zope-like "error logger" which runs from middleware. You can inspect tracebacks TTW, as with the Zope2 "tool.", but it doesn't live in the ZODB.
end_transaction -- sounds like the other end of repoze.tm2
Again, I don't know why they are split.
authenticate -- sounds like repoze.who?
That is certainly the job done by repoze.who. Moving it out to middleware eases sharing authentication across multiple applications. repoze.what is an *authorization* middleware / framework which has been adopted for use in TurboGears2: it extends repoze.who to provide for a quasi-declarative security model for applications.
fix_relative_links -- sounds generally useful outside Zope as well
I have written a couple of such bits as one-offs, but never thought I had anything like a general solution. Composability means that we can more easily *avoid* the temptation to generalize prematurely (as with events). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJpXPY+gerLs4ltQ4RAt5cAKCp+RTbLilJ5eWZAdsuIW75kw0BWgCg0Vlz Dp03UOd0NomDlSg9yaq7ot8= =JxK8 -----END PGP SIGNATURE-----
Martin Aspeli wrote:
I'm used to using Paste Deploy ini files to configure a WSGI pipeline. Is this simply an alternative to that? If so, do we really need our own alternative, or could we try to use the Paste Deploy stuff directly?
Yes, you can just use Paste Deploy instead of the ZCML-based pipeline builder. However, the pipeline you get with Paste Deploy will not vary according to the request type, so you won't get the publisher's special handling for non-browser methods like PUT and DELETE, nor XML-RPC support, etc. OTOH, I'm sure people can find creative ways to make Paste Deploy vary the pipeline according to the request type.
I am a little worried about the conceptual overhead of having both the ZCA and the WSGI pipeline provide "swappability" services to application builders. It feels like those two things overlap somewhat in scope.
They do overlap, but the ZCA-configured pipeline can easily vary according to the request type. I need to add more info about this to the proposal.
Also, looking at your endware, there are some that seem to overlap with Repoze stuff. Is this on purpose? I think the relationship with Repoze should be made more clear in the proposal.
I don't know what the relationship is yet because I've chosen to only get code from Zope, for now, since I'm trying to maintain compatibility.
virtual_host -- is this the same as repoze.vhm?
Could be. Not sure.
retry -- is this the same as repoze.retry?
I think it might be different, because this version of retry resets the environment rather than the request. It has to be in the pipeline before the create_request application. This design makes it possible to eliminate the complex retry code from Zope requests.
create_request -- should this maybe have some compatibility with WebOb requests?
I've looked at WebOb, and my impression is that Zope requests and WebOb requests serve completely different purposes. A Zope request is essentially an input decoder and information holder. A WebOb request is a whole framework, AFAICT. That said, I expect zope.pipeline to give people a greater opportunity to prove I'm wrong about things like that.
switch_pipeline -- could this be made non-Zope specific? It sounds useful.
I need to understand better what you mean by non-Zope specific. How would you use it?
log -- both repoze and paste have logging middleware, IIRC
Sure. I haven't bothered to write this app at all since I know lots of people have written such things.
open_root -- I thought repoze had something similar, but I may be wrong
I think this one is significantly different from Repoze. It would be good to communicate about stuff like this at a sprint.
clean_transaction -- is this not the same as repoze.tm2?
No. To mimic the current Zope publisher, we need to commit the transaction shortly after the "call" application is finished, but then a lot of things can still happen before the response leaves the server, so we need to make sure any open transaction is aborted before letting the open_root application close the database. This application is very small. It might make sense to have the open_root application do its job instead.
handle_error -- again, I thought Repoze had something like this
This error handling is very different from Repoze, I think. It's going to handle errors the Zope way, by looking up error handlers in the database. I imagine a lot of people will want to remove or customize it.
end_transaction -- sounds like the other end of repoze.tm2
This is like repoze.tm2, but it will also annotate the transaction. Note that the code in my sandbox does not yet reflect this thinking.
authenticate -- sounds like repoze.who?
I doubt it. This does the IAuthentication dance, allowing authentication to be added during traversal.
fix_relative_links -- sounds generally useful outside Zope as well
I wish, but this application depends on Zope's traversal logic to determine the canonical URL of an object. I don't think the same traversal information will be present in most frameworks. OTOH, I imagine fix_relative_links could be rewritten as a more broadly useful thing, if someone can figure out how without breaking performance. Shane
Hey, Shane Hathaway wrote:
Martin Aspeli wrote: [snip]
create_request -- should this maybe have some compatibility with WebOb requests?
I've looked at WebOb, and my impression is that Zope requests and WebOb requests serve completely different purposes. A Zope request is essentially an input decoder and information holder. A WebOb request is a whole framework, AFAICT.
As far as I know WebOb provides a request and response implementation on top of WSGI. I guess that makes it similar to what zope.publisher does. Using WebOb straight away won't be very easy as the API isn't compatible with that of Zope's request object. You could adapt one to the other, but that won't gain you much. It should be fairly straightforward, I hope, to obtain a WebOb-style request instead of a Zope-style request in application code.
That said, I expect zope.pipeline to give people a greater opportunity to prove I'm wrong about things like that.
On the longer term I'd like to experiment with a WebOb-based Grok and this should allow a way in. But that's definitely long-term thinking. Regards, Martijn
Shane Hathaway wrote:
Martin Aspeli wrote:
I'm used to using Paste Deploy ini files to configure a WSGI pipeline. Is this simply an alternative to that? If so, do we really need our own alternative, or could we try to use the Paste Deploy stuff directly?
Yes, you can just use Paste Deploy instead of the ZCML-based pipeline builder. However, the pipeline you get with Paste Deploy will not vary according to the request type, so you won't get the publisher's special handling for non-browser methods like PUT and DELETE, nor XML-RPC support, etc.
You can configure multiple pipelines in a Paste Deploy file. It's up to the fronting web server to decide which 'pipeline app' to invoke.
OTOH, I'm sure people can find creative ways to make Paste Deploy vary the pipeline according to the request type.
The Paste#composite web app basically does this based on inbound subpath. It's not exactly hard to do in WSGI.
I am a little worried about the conceptual overhead of having both the ZCA and the WSGI pipeline provide "swappability" services to application builders. It feels like those two things overlap somewhat in scope.
They do overlap, but the ZCA-configured pipeline can easily vary according to the request type. I need to add more info about this to the proposal.
If this is the only reason to use the ZCA to register the pipeline, then maybe there's no need: only a fronting piece of middleware that picks the right pipeline. Of course, there may be other things that make using the ZCA attractive.
Also, looking at your endware, there are some that seem to overlap with Repoze stuff. Is this on purpose? I think the relationship with Repoze should be made more clear in the proposal.
I don't know what the relationship is yet because I've chosen to only get code from Zope, for now, since I'm trying to maintain compatibility.
I think it'd be a shame to produce a lot of new middleware that overlaps with the Repoze middleware, when both are born out of Zope. At least, we should try to re-use Repoze stuff wherever we can, rather than roll our own.
virtual_host -- is this the same as repoze.vhm?
Could be. Not sure.
I suspect they are.
retry -- is this the same as repoze.retry?
I think it might be different, because this version of retry resets the environment rather than the request. It has to be in the pipeline before the create_request application. This design makes it possible to eliminate the complex retry code from Zope requests.
Ok.
create_request -- should this maybe have some compatibility with WebOb requests?
I've looked at WebOb, and my impression is that Zope requests and WebOb requests serve completely different purposes. A Zope request is essentially an input decoder and information holder. A WebOb request is a whole framework, AFAICT.
Yes, kind of. I'm not sure using it would make sense for Zope. However, lots of other middleware does use WebOb requests, so it'd be useful to think about and document how they relate, if nothing else.
That said, I expect zope.pipeline to give people a greater opportunity to prove I'm wrong about things like that.
;-)
switch_pipeline -- could this be made non-Zope specific? It sounds useful.
I need to understand better what you mean by non-Zope specific. How would you use it?
I'd imagine, to have multiple pipelines set up and have this middleware know which one to invoke. At the end of the day, to one piece of middleware, the 'pipeline' is merely one or possibly many callables that invoke the next piece of middleware/application.
open_root -- I thought repoze had something similar, but I may be wrong
I think this one is significantly different from Repoze. It would be good to communicate about stuff like this at a sprint.
Yes I think they'd be different, reading Tres' reply.
clean_transaction -- is this not the same as repoze.tm2?
No. To mimic the current Zope publisher, we need to commit the transaction shortly after the "call" application is finished, but then a lot of things can still happen before the response leaves the server, so we need to make sure any open transaction is aborted before letting the open_root application close the database.
Why is it desirable to do things in this way? I do find it kind of confusing/error prone that we have two pieces of middleware: one that opens the transaction and another that closes it, with ordering dependencies between the two.
This application is very small. It might make sense to have the open_root application do its job instead.
Sure.
authenticate -- sounds like repoze.who?
I doubt it. This does the IAuthentication dance, allowing authentication to be added during traversal.
repoze.who seems to be turning into something of a widely used standard. I think it'd be worth looking into whether it can be used 'upstream' and something else could do the IAuthentication stuff based on what repoze.who does. Not that I understand what the IAuthentication dance is. ;-)
fix_relative_links -- sounds generally useful outside Zope as well
I wish, but this application depends on Zope's traversal logic to determine the canonical URL of an object. I don't think the same traversal information will be present in most frameworks.
Ok.
OTOH, I imagine fix_relative_links could be rewritten as a more broadly useful thing, if someone can figure out how without breaking performance.
It may not be necessary, of course. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
Martin Aspeli wrote:
Shane Hathaway wrote:
Martin Aspeli wrote:
clean_transaction -- is this not the same as repoze.tm2? No. To mimic the current Zope publisher, we need to commit the transaction shortly after the "call" application is finished, but then a lot of things can still happen before the response leaves the server, so we need to make sure any open transaction is aborted before letting the open_root application close the database.
Why is it desirable to do things in this way? I do find it kind of confusing/error prone that we have two pieces of middleware: one that opens the transaction and another that closes it, with ordering dependencies between the two.
I'm trying to express something different, actually. The first app, "clean_transaction", only exists to cancel accidental writes. The Zope publisher does this today. The actual transaction management is in the second application, which I now realize is misnamed as "end_transaction" because it both begins and ends transactions.
repoze.who seems to be turning into something of a widely used standard. I think it'd be worth looking into whether it can be used 'upstream' and something else could do the IAuthentication stuff based on what repoze.who does.
Ok, that changes my perspective a bit. Shane
Hi there, One issue I have with using paste deploy's pipeline configuration for "endware" is that such configuration sometimes really wants to be part of a library. I.e. I don't want to configure a tower of endwares each time I write an application, I want to reuse some premade configuration that comes along with some released library. This as opposed to middleware. The middleware is generic across WSGI servers and the person who decides to use it is frequently the developer. That said, there's likely a grey area between the two. Perhaps there's a way you can use paste's mechanism and ship it in a library. If not, you'll end up stacking together WSGI's in Python code. I can then see the benefit in a declarative version of this that can ship along with a library. With the right application of grokkers that could probably be made to look very compact too. :) Regards, Martijn
Hi Shane
Betreff: [Zope-dev] Zope.pipeline proposal
Hi all,
I've put up a draft of a zope.pipeline proposal:
http://wiki.zope.org/zope3/ZopePipeline
The proposal is intended to explain my thoughts on the subject more thoroughly.
Do you know something about the performance of WSGI? I whould be happy to see some perfomance tests comparing WSGI with other server concepts. Regards Roger Ineichen
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Roger Ineichen wrote:
Hi Shane
Betreff: [Zope-dev] Zope.pipeline proposal
Hi all,
I've put up a draft of a zope.pipeline proposal:
http://wiki.zope.org/zope3/ZopePipeline
The proposal is intended to explain my thoughts on the subject more thoroughly.
Do you know something about the performance of WSGI?
I whould be happy to see some perfomance tests comparing WSGI with other server concepts.
Chris McDonough wrote up some "complexity" benchmarks here: - http://plope.com/whats_your_web_framework_doing - http://plope.com/whatsitdoing2 We did some work early on in the repoze effort comparing the various WSGI servers: http://lists.repoze.org/pipermail/repoze-dev/2007-September/000487.html Note that the relative differences between the various servers drop off steeply once the application begins doing any real work (even the Z2 quickstart page). This is because requests-per-second is actually a deceptive measure: we should probably switch to talking "microseconds per request" instead. In general, if you need full-on backward compatibility with the existing behavior of Zope2 / Zope3 / Grok, switching to a paste-driven WSGI pipeline doesn't gain you much speed (but it is not a loss, either). If, for a given application, you can relax the BBB requirement, then some performance wins are available via WSGI which can't be made in the monolithic publisher (dropping out features by removing the middleware layer). Real speed wins only come from more radical simplifications: for instance, repoze.bfg drops the adapter-based traversal semantics in favor of using only __getitem__. The fact that the BFG "hello world" app is ~20 times faster[1] than the Grok "hello world" is due to such simplifications. [1] 163 us/req for the BFG vs.3012 us/req for Grok in Chris' profiling, the difference is more marked with profiling turned off. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJpXd3+gerLs4ltQ4RAiQXAJ9/RP7UWcLZ/lCW0jDdlTCqko8cgACdHhoC yx1PS0mMm7eCNrtvAr3QMYY= =hfNz -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tres Seaver wrote:
Roger Ineichen wrote:
Hi Shane
Betreff: [Zope-dev] Zope.pipeline proposal
Hi all,
I've put up a draft of a zope.pipeline proposal:
http://wiki.zope.org/zope3/ZopePipeline
The proposal is intended to explain my thoughts on the subject more thoroughly. Do you know something about the performance of WSGI?
I whould be happy to see some perfomance tests comparing WSGI with other server concepts.
Sorry for following up to myself: I had meant to include links to the recent "songlist" thread which examined "raw" WSGI vs. various frameworks: - - The original, "raw WSGI" writeup: http://www.eflorenzano.com/blog/post/writing-blazing-fast-infinitely-scalabl... - - Martijn Faassen's Grok take: http://faassen.n--tree.net/blog/view/weblog/2009/01/10/0 - - Carlos de la Guardia's repoze.bfg writeup: http://blog.delaguardia.com.mx/index.php?op=ViewArticle&articleId=119&blogId... Please note the comparison Carlos makes between "raw" WSGI on his box, using the default Paste http server (1780 requests/second) vs. the repoze.bfg version with the same server (1060 requests/second). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJpYPX+gerLs4ltQ4RAiihAJwNxxxiU3L8c2aBCyE1D6qu3CfHYQCgySlm LpYSFDGcULBueL/9+Oz2OjE= =4a6d -----END PGP SIGNATURE-----
Hey, Tres Seaver wrote: [snip]
In general, if you need full-on backward compatibility with the existing behavior of Zope2 / Zope3 / Grok, switching to a paste-driven WSGI pipeline doesn't gain you much speed (but it is not a loss, either). If, for a given application, you can relax the BBB requirement, then some performance wins are available via WSGI which can't be made in the monolithic publisher (dropping out features by removing the middleware layer).
As for Grok I hope we can break some backwards compatibility and get some larger performance speedups. We definitely need to aggressively keep moving forward in this area. Not even primarily for speed gains but also for comprehensibility; I find Chris's "what's it doing" report far more worrying than differences in speed at this point: http://plope.com/whatsitdoing2 This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area.
Real speed wins only come from more radical simplifications: for instance, repoze.bfg drops the adapter-based traversal semantics in favor of using only __getitem__. The fact that the BFG "hello world" app is ~20 times faster[1] than the Grok "hello world" is due to such simplifications.
In a hello world the overhead of the publisher dominates in a way that in normal applications it probably doesn't, so what are "real speed wins", as usual, will depend. Improvements in template language implementations might gain us more. That said, recent profiling of Grok with chameleon make that a small component too, so perhaps it *is* the publisher. Oh the joys of profiling! :) Regards, Martijn
Hi all
Betreff: Re: [Zope-dev] Zope.pipeline proposal
Hey,
Tres Seaver wrote: [snip]
In general, if you need full-on backward compatibility with the existing behavior of Zope2 / Zope3 / Grok, switching to a paste-driven WSGI pipeline doesn't gain you much speed (but it is not a loss, either). If, for a given application, you can relax the BBB requirement, then some performance wins are available via WSGI which can't be made in the monolithic publisher (dropping out features by removing the middleware layer).
As for Grok I hope we can break some backwards compatibility and get some larger performance speedups. We definitely need to aggressively keep moving forward in this area. Not even primarily for speed gains but also for comprehensibility; I find Chris's "what's it doing" report far more worrying than differences in speed at this point:
http://plope.com/whatsitdoing2
This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area.
I absolutly agree! As far as I can see, the repoze sample doesn't open a ZODB which makes it not really comparable. Regards Roger Ineichen
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Roger Ineichen wrote:
Hi all
Betreff: Re: [Zope-dev] Zope.pipeline proposal
Hey,
Tres Seaver wrote: [snip]
In general, if you need full-on backward compatibility with the existing behavior of Zope2 / Zope3 / Grok, switching to a paste-driven WSGI pipeline doesn't gain you much speed (but it is not a loss, either). If, for a given application, you can relax the BBB requirement, then some performance wins are available via WSGI which can't be made in the monolithic publisher (dropping out features by removing the middleware layer). As for Grok I hope we can break some backwards compatibility and get some larger performance speedups. We definitely need to aggressively keep moving forward in this area. Not even primarily for speed gains but also for comprehensibility; I find Chris's "what's it doing" report far more worrying than differences in speed at this point:
http://plope.com/whatsitdoing2
This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area.
I absolutly agree!
As far as I can see, the repoze sample doesn't open a ZODB which makes it not really comparable.
I think you've made Chris' point for him: nothing about the application being tested *requires* that there be a ZODB connection open; Grok's design forces opening one unconditionally, which is a layer of complexity which *can't be turned off.* The "conceptual" overhead of each of the frameworks is at least as important as the performance overhead. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJpdHk+gerLs4ltQ4RAutNAJ91GlaX91Kl4hzKlv9NKUezrdK4zQCgyLBL We7uIeSpZ+KPepKU3/eCey8= =U1s/ -----END PGP SIGNATURE-----
Hi Tres
http://plope.com/whatsitdoing2
This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area.
I absolutly agree!
As far as I can see, the repoze sample doesn't open a ZODB which makes it not really comparable.
I think you've made Chris' point for him: nothing about the application being tested *requires* that there be a ZODB connection open; Grok's design forces opening one unconditionally, which is a layer of complexity which *can't be turned off.* The "conceptual" overhead of each of the frameworks is at least as important as the performance overhead.
Yes, that's true. But I like to see the real performance win within a *real* application. My applications normaly do not only show a hello world page ;-) I also like to see how much the repoze method calls will grow if the authentication, transaction, ZODB etc is involved. I'm pretty sure that it will much less then 20 times like in the given test. I'll also be happy if we will gain a 2 time perfomance factor in a real comparable test app. I've developed a prototyping package in my own personal repos which tries to setup a site within some objects which I use with the old zope server and a paste WSGI server setup. This let us run performance tests against the ZODB. Probably I should add that to the zope repos and we could start adding all our different publishing implementations. This whould let us compare the same things in a real world use case. Regards Roger Ineichen
Roger Ineichen wrote:
Hi Tres
http://plope.com/whatsitdoing2
This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area. I absolutly agree!
As far as I can see, the repoze sample doesn't open a ZODB which makes it not really comparable. I think you've made Chris' point for him: nothing about the application being tested *requires* that there be a ZODB connection open; Grok's design forces opening one unconditionally, which is a layer of complexity which *can't be turned off.* The "conceptual" overhead of each of the frameworks is at least as important as the performance overhead.
Yes, that's true. But I like to see the real performance win within a *real* application. My applications normaly do not only show a hello world page ;-)
I also like to see how much the repoze method calls will grow if the authentication, transaction, ZODB etc is involved. I'm pretty sure that it will much less then 20 times like in the given test. I'll also be happy if we will gain a 2 time perfomance factor in a real comparable test app.
I've developed a prototyping package in my own personal repos which tries to setup a site within some objects which I use with the old zope server and a paste WSGI server setup. This let us run performance tests against the ZODB. Probably I should add that to the zope repos and we could start adding all our different publishing implementations. This whould let us compare the same things in a real world use case.
While I think that would be a good thing, I do want to mention that it's not really the point of the "whatsitdoing" benchmark. Repoze stuff typically tries to simplify each component in the set of components used to service a goal, sometimes at the expense of at least some backwards compatibility or excessive configurability. On the other hand, Zope3 and Grok tend to keep backwards compatibility and configurability sometimes at the expense of verbosity and extra runtime expense. They are just somewhat different goals. Repoze stuff therefore has the major advantage of not needing to carry around 6-10 years of backwards compatibility baggage; it owes any 20-20 hindsight in these cases of course directly to Zope. In the meantime, we are proud of the work we've been doing to simplify Zopelike publishing, and the "whatsitdoing" benchmark was really just a mechanism to toot that particular horn. I'm not sure how to toot that horn without comparing it to Zope, so while I see that it has ruffled some feathers, I'll stand by the benchmark as a useful comparison. I'm pretty sure the stuff would fare pretty well on some other less objective benchmark (like "developer productvity" as you seem to be advocating). FTR, the "publishing implementation" you speak of above in the case of repoze is is really a different framework altogether: http://static.repoze.org/bfgdocs/ . - C
Hi Chris
Betreff: Re: [Zope-dev] Zope.pipeline proposal
Roger Ineichen wrote:
Hi Tres
http://plope.com/whatsitdoing2
This is why zope.pipeline is such an important effort to me. Not that it will immediately make things better, but it would hopefully open up a path to move the Zope Framework forward in this area. I absolutly agree!
As far as I can see, the repoze sample doesn't open a ZODB which makes it not really comparable. I think you've made Chris' point for him: nothing about the application being tested *requires* that there be a ZODB connection open; Grok's design forces opening one unconditionally, which is a layer of complexity which *can't be turned off.* The "conceptual" overhead of each of the frameworks is at least as important as the performance overhead.
Yes, that's true. But I like to see the real performance win within a *real* application. My applications normaly do not only show a hello world page ;-)
I also like to see how much the repoze method calls will grow if the authentication, transaction, ZODB etc is involved. I'm pretty sure that it will much less then 20 times like in the given test. I'll also be happy if we will gain a 2 time perfomance factor in a real comparable test app.
I've developed a prototyping package in my own personal repos which tries to setup a site within some objects which I use with the old zope server and a paste WSGI server setup. This let us run performance tests against the ZODB. Probably I should add that to the zope repos and we could start adding all our different publishing implementations. This whould let us compare the same things in a real world use case.
While I think that would be a good thing, I do want to mention that it's not really the point of the "whatsitdoing" benchmark.
Repoze stuff typically tries to simplify each component in the set of components used to service a goal, sometimes at the expense of at least some backwards compatibility or excessive configurability. On the other hand, Zope3 and Grok tend to keep backwards compatibility and configurability sometimes at the expense of verbosity and extra runtime expense. They are just somewhat different goals. Repoze stuff therefore has the major advantage of not needing to carry around 6-10 years of backwards compatibility baggage; it owes any 20-20 hindsight in these cases of course directly to Zope.
In the meantime, we are proud of the work we've been doing to simplify Zopelike publishing, and the "whatsitdoing" benchmark was really just a mechanism to toot that particular horn. I'm not sure how to toot that horn without comparing it to Zope, so while I see that it has ruffled some feathers, I'll stand by the benchmark as a useful comparison. I'm pretty sure the stuff would fare pretty well on some other less objective benchmark (like "developer productvity" as you seem to be advocating).
FTR, the "publishing implementation" you speak of above in the case of repoze is is really a different framework altogether: http://static.repoze.org/bfgdocs/ .
I see what you mean. I really have to deep into the repoze packages. It defently looks very promising! btw, Don't get me wrong. I really appreciate your great work. It's one of this pieces we need for make better progress. Regards Roger Ineichen
- C
Chris McDonough wrote: [snip]
While I think that would be a good thing, I do want to mention that it's not really the point of the "whatsitdoing" benchmark.
Right, agreed. I think it's more important to make the Zope Framework more comprehensible than it is to improve its performance. Its performance is already fine for many purposes and it'll be much easier to improve once the structure is more comprehensible anyway. We have currently two efforts going on to increase comprehensibility of the Zope Framework: * cutting away code that isn't needed or is unnecessarily pulled in. This is the dependency refactoring work. * Shane's zope.pipeline work. I'm pretty pleased we have all of this going, as long as we keep it up. :) That we need to clean up the framework became clear to me much before Chris started pointing out things in the "whatsitdoing" benchmark though, but earlier discussions about Repoze (in particular with Tres at DZUG last september) did help shape my ideas.
Repoze stuff typically tries to simplify each component in the set of components used to service a goal, sometimes at the expense of at least some backwards compatibility or excessive configurability. On the other hand, Zope3 and Grok tend to keep backwards compatibility and configurability sometimes at the expense of verbosity and extra runtime expense. They are just somewhat different goals. Repoze stuff therefore has the major advantage of not needing to carry around 6-10 years of backwards compatibility baggage; it owes any 20-20 hindsight in these cases of course directly to Zope.
I do believe we can move the existing Zope Framework forward quite a bit in the right direction too. Perhaps we'll have to give up some backwards compatibility, but we can probably keep most of the code working. We haven't been afraid to break things occasionally with Grok but we've found that with good upgrade notes we can keep the pain to a minimum. This work should also result in more reusable components for Repoze and more use of Repoze components within code that uses the Zope Framework (such as Grok), so that should make us all happy. :) Luckily we're not starting with a very bad situation here. We have the benefit that the Zope Framework (and Grok) are made out of pieces that are at least *somewhat* tractable and replaceable. Regards, Martijn
Roger Ineichen wrote:
Do you know something about the performance of WSGI?
I whould be happy to see some perfomance tests comparing WSGI with other server concepts.
WSGI is extremely lightweight, so WSGI itself isn't going to affect performance. The WSGI servers I know about are reasonably fast. Shane
Shane Hathaway wrote:
Roger Ineichen wrote:
Do you know something about the performance of WSGI?
I whould be happy to see some perfomance tests comparing WSGI with other server concepts.
WSGI is extremely lightweight, so WSGI itself isn't going to affect performance. The WSGI servers I know about are reasonably fast.
What's the overhead of a WSGI middleware? Is the overhead cost in the same order of magnitude as a simple function call with a return value or is there something inherently more complex going on?
From my current experience I usually classify things into the groups of:
1. attribute access 2. function calls 3. adapter lookups Going from one of these groups into the next one has a very noticeable performance cost involved, if used on the critical path of the application. The same goes for "contract checks" in terms of: 1. getattr(object, 'foo') 2. isinstance(object, Foo) 3. IFoo.providedBy(object) If you want to design things for astonishing speed, you need to make trade-offs in terms of configurablity ;) Hanno
Hanno Schlichting wrote:
What's the overhead of a WSGI middleware? Is the overhead cost in the same order of magnitude as a simple function call with a return value or is there something inherently more complex going on?
A WSGI middleware app is simply a callable thing that calls the next app, so the overhead is literally the same as a simple function call. There won't be security proxies or any other such thing getting in the way.
From my current experience I usually classify things into the groups of:
1. attribute access 2. function calls 3. adapter lookups
Going from one of these groups into the next one has a very noticeable performance cost involved, if used on the critical path of the application.
The current publisher is chock full of function calls and adapter lookups along the critical path. I expect the zope.pipeline project to open everyone's eyes about just how much we're already doing in the critical path. I think much of it is frequently not needed at all. Shane
Hanno Schlichting wrote:
Shane Hathaway wrote:
Roger Ineichen wrote:
Do you know something about the performance of WSGI?
I whould be happy to see some perfomance tests comparing WSGI with other server concepts. WSGI is extremely lightweight, so WSGI itself isn't going to affect performance. The WSGI servers I know about are reasonably fast.
What's the overhead of a WSGI middleware? Is the overhead cost in the same order of magnitude as a simple function call with a return value or is there something inherently more complex going on?
No. A WSGI app is basically just a callable. Of course, the app can do lots of crazy things. There may also be some setup cost if you use things like Paste Deploy, but that's a one-time thing. zope.pipeline may of course do lots of crazy things in its apps. ;-) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
Shane Hathaway wrote:
I've put up a draft of a zope.pipeline proposal:
http://wiki.zope.org/zope3/ZopePipeline
The proposal is intended to explain my thoughts on the subject more thoroughly.
+1 for the project. I think this will make it easier to reason about what goes on in Zope and help us evolve the publisher forward. It's also high time the Zope Framework started to adopt concepts from Repoze; it shouldn't only flow one way. :) One area that I'd like to see support for is some easy way to turn off security proxies. It's rumored there is such a way but with Grok, we ended up ripping them off repeatedly anyway. Am I right in that it should be possible to put a WSGI endware on top of this whole stack that does an explicit security check? Regards, Martijn
2009/2/25 Martijn Faassen <faassen@startifact.com>:
One area that I'd like to see support for is some easy way to turn off security proxies. It's rumored there is such a way but with Grok, we ended up ripping them off repeatedly anyway. Am I right in that it should be possible to put a WSGI endware on top of this whole stack that does an explicit security check?
I think so. Currently, the main entry point for security proxies is the "get root" method of the publication, so if you'll use modified publication object that don't wrap root object in the ProxyFactory, you'll rip most of them. However, some things, like trusted adapters rewrap objects using ProxyFactory, so, maybe we could add some modifier to the ProxyFactory function that just makes it return object as is w/o wrapping. This way we could turn off proxies globally without need to modify code that uses ProxyFactory. -- WBR, Dan Korostelev
participants (8)
-
Chris McDonough -
Dan Korostelev -
Hanno Schlichting -
Martijn Faassen -
Martin Aspeli -
Roger Ineichen -
Shane Hathaway -
Tres Seaver