Zope 2 WSGI investigation
Hi, There are three known WSGI implementations of the Zope 2 publisher. I've had a look at them and made some notes about what I think provides the best story: ## Zope 2.13 WSGIPublisher Pros: * Allows distributed transaction management with repoze.tm2 * Allows distributed retry with repoze.retry * Ships with Zope * Quite simple Cons: * Requires repoze.tm2 and repoze.rety * Does not properly emit publication events - possible patch in https://gist.github.com/1548061 * Does not do error handling or exception views * Claims not to properly implement streaming (though there is some code for it) * Probably less well tested than infrae.wsgi and repoze.zope2 (at least there is zero documentation) ## infrae.wsgi Pros: * Clean and well documented * Properly emits publication events * Supports streaming * Supports simplified virtual hosting with X-VHM-Host * Supports exception handling / error views * Reportedly has significant production use Cons: * Not 100% compatible (but close and fixable) - fix to make plone.transformchain work is here: https://gist.github.com/1547328 * Unnecessary five.grok dependency (but easy to rewrite to use ZCML registration) * No support for middleware transaction and retry management, so these can't be distributed across a WSGI pipeline * Error logging will not support ZMI error_log and assumes single process * Error handling is slightly different to standard publisher's exception views, and also does not honour existing standard_error_message etc ## repoze.zope2 Pros: * Clean and well documented * Reimplements and simplifies the BaseRequest.traverse() code, with comments * Supports distributed transaction management and retry Cons: * Replicates a lot of Zope startup code * Has now-unnecessary code to manage instances and configuration * repoze.obob abstraction is unnecessary since nothing else uses this * Does not emit publication events - possible patch in http://bugs.repoze.org/issue181 * Does not do error handling or exception views * Problems with file resources (does not properly traverse to browserDefault() result) -- possible patch in http://bugs.repoze.org/issue64 * Requires various middleware (repoze.tm, repoze.retry, repoze.vhm) ## Suggested approach going forward * Integrate infrae.wsgi into Zope 2 * Remove its five.grok dependency * Use the same exception-views protocol as ZPublisher (mainly, that the view name is ``index.html``) * Stop using __ 'private' variables in response.py to make it easier to work with * Add some BBB support for existing error logging and error messages Thoughts? Martin
On 01/01/2012 08:39 PM, Martin Aspeli wrote:
Hi,
There are three known WSGI implementations of the Zope 2 publisher. I've had a look at them and made some notes about what I think provides the best story:
## Zope 2.13 WSGIPublisher
Pros:
* Allows distributed transaction management with repoze.tm2 * Allows distributed retry with repoze.retry * Ships with Zope * Quite simple
Cons:
* Requires repoze.tm2 and repoze.rety
Why is that a con? I use repoze.tm2 and repoze.retry with all my pyramid projects and they work beautifully. Wichert.
On 2 January 2012 08:50, Wichert Akkerman <wichert@wiggy.net> wrote:
On 01/01/2012 08:39 PM, Martin Aspeli wrote:
Hi,
There are three known WSGI implementations of the Zope 2 publisher. I've had a look at them and made some notes about what I think provides the best story:
## Zope 2.13 WSGIPublisher
Pros:
* Allows distributed transaction management with repoze.tm2 * Allows distributed retry with repoze.retry * Ships with Zope * Quite simple
Cons:
* Requires repoze.tm2 and repoze.rety
Why is that a con? I use repoze.tm2 and repoze.retry with all my pyramid projects and they work beautifully.
Only insofar as it means you have to have these in your WSGI pipeline or it won't work, so there are more places things can go wrong. You'll note that I also consider not supporting such things a con for infrae.wsgi. I wouldn't get too hung up on it, I was mainly just trying to bring out the differences. It'd be nice if it wasn't a hard requirement, though. Martin
On Mon, 2012-01-02 at 10:39 +0000, Martin Aspeli wrote:
On 2 January 2012 08:50, Wichert Akkerman <wichert@wiggy.net> wrote:
On 01/01/2012 08:39 PM, Martin Aspeli wrote:
Hi,
There are three known WSGI implementations of the Zope 2 publisher. I've had a look at them and made some notes about what I think provides the best story:
## Zope 2.13 WSGIPublisher
Pros:
* Allows distributed transaction management with repoze.tm2 * Allows distributed retry with repoze.retry * Ships with Zope * Quite simple
Cons:
* Requires repoze.tm2 and repoze.rety
Why is that a con? I use repoze.tm2 and repoze.retry with all my pyramid projects and they work beautifully.
Only insofar as it means you have to have these in your WSGI pipeline or it won't work, so there are more places things can go wrong.
You'll note that I also consider not supporting such things a con for infrae.wsgi. I wouldn't get too hung up on it, I was mainly just trying to bring out the differences. It'd be nice if it wasn't a hard requirement, though.
FWIW, when I wrote repoze.tm2 I did not know that the transaction module already supported retry, so at least repoze.rety should die in favor of logic in something-like-repoze.tm2 that looks a little more like this: https://github.com/Pylons/pyramid_tm/blob/master/pyramid_tm/__init__.py#L49 (that's obviously not WSGI middleware, I'm just showing what the general logic should look like) - C
Martin _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
On 3 January 2012 06:39, Chris McDonough <chrism@plope.com> wrote:
On Mon, 2012-01-02 at 10:39 +0000, Martin Aspeli wrote:
On 2 January 2012 08:50, Wichert Akkerman <wichert@wiggy.net> wrote:
On 01/01/2012 08:39 PM, Martin Aspeli wrote:
Hi,
There are three known WSGI implementations of the Zope 2 publisher. I've had a look at them and made some notes about what I think provides the best story:
## Zope 2.13 WSGIPublisher
Pros:
* Allows distributed transaction management with repoze.tm2 * Allows distributed retry with repoze.retry * Ships with Zope * Quite simple
Cons:
* Requires repoze.tm2 and repoze.rety
Why is that a con? I use repoze.tm2 and repoze.retry with all my pyramid projects and they work beautifully.
Only insofar as it means you have to have these in your WSGI pipeline or it won't work, so there are more places things can go wrong.
You'll note that I also consider not supporting such things a con for infrae.wsgi. I wouldn't get too hung up on it, I was mainly just trying to bring out the differences. It'd be nice if it wasn't a hard requirement, though.
FWIW, when I wrote repoze.tm2 I did not know that the transaction module already supported retry, so at least repoze.rety should die in favor of logic in something-like-repoze.tm2 that looks a little more like this:
https://github.com/Pylons/pyramid_tm/blob/master/pyramid_tm/__init__.py#L49
(that's obviously not WSGI middleware, I'm just showing what the general logic should look like)
Am I right in thinking Pyramid no longer uses repoze.tm2 or a middleware approach? What was the rationale for that design decision? Martin
On Tue, 2012-01-03 at 07:32 +0000, Martin Aspeli wrote:
On 3 January 2012 06:39, Chris McDonough <chrism@plope.com> wrote:
On Mon, 2012-01-02 at 10:39 +0000, Martin Aspeli wrote:
On 2 January 2012 08:50, Wichert Akkerman <wichert@wiggy.net> wrote:
On 01/01/2012 08:39 PM, Martin Aspeli wrote:
Hi,
There are three known WSGI implementations of the Zope 2 publisher. I've had a look at them and made some notes about what I think provides the best story:
## Zope 2.13 WSGIPublisher
Pros:
* Allows distributed transaction management with repoze.tm2 * Allows distributed retry with repoze.retry * Ships with Zope * Quite simple
Cons:
* Requires repoze.tm2 and repoze.rety
Why is that a con? I use repoze.tm2 and repoze.retry with all my pyramid projects and they work beautifully.
Only insofar as it means you have to have these in your WSGI pipeline or it won't work, so there are more places things can go wrong.
You'll note that I also consider not supporting such things a con for infrae.wsgi. I wouldn't get too hung up on it, I was mainly just trying to bring out the differences. It'd be nice if it wasn't a hard requirement, though.
FWIW, when I wrote repoze.tm2 I did not know that the transaction module already supported retry, so at least repoze.rety should die in favor of logic in something-like-repoze.tm2 that looks a little more like this:
https://github.com/Pylons/pyramid_tm/blob/master/pyramid_tm/__init__.py#L49
(that's obviously not WSGI middleware, I'm just showing what the general logic should look like)
Am I right in thinking Pyramid no longer uses repoze.tm2 or a middleware approach? What was the rationale for that design decision?
You're right, Pyramid scaffolding no longer supplies repoze.tm2 or any other WSGI middleware component for handling transaction commits/aborts. Instead, it uses a Pyramid "tween", which is sort of like internal Pyramid middleware inasmuch as its a user-manageable functional composition terminating at the application. See http://www.slideshare.net/aconrad/alex-conrad-pyramid-tweens-ploneconf-2011 Tweens can be reordeded arbitrarily, and the Pyramid exception handling logic (which locates and executes a Pyramid view when an application exception is raised) is itself a tween. The rationale is that application-specific error logic often needs to be executed after the transaction has been committed or aborted due to an exception. For example, if an exception is raised by an application, you'd like the system to abort the transaction, then potentially display a custom internal server error page (e.g. twitter fail-whale). People want to be able to write the error logic within their existing Pyramid development environment (where they have access to templating systems, a real request object, and possibly their database and other niceties) instead of as WSGI middleware higher in the stack than the transaction manager. The pyramid debug toolbar (the thing that shows the pretty debug pages described at http://docs.pylonsproject.org/projects/pyramid_debugtoolbar/en/latest/) is also a tween, which means it has access to the Pyramid environment that WSGI middleware could not (such as access to the registry, allowing us to display view-configuration information, information about routes, etc). For us, this replaced our use of WebError, much as our use of pyramid_tm replaced our use of repoze.tm2, for mostly the same reasons. FWIW, I believe Tres still disagrees strongly with this decision. - C
On 3 January 2012 08:01, Chris McDonough <chrism@plope.com> wrote:
Am I right in thinking Pyramid no longer uses repoze.tm2 or a middleware approach? What was the rationale for that design decision?
You're right, Pyramid scaffolding no longer supplies repoze.tm2 or any other WSGI middleware component for handling transaction commits/aborts. Instead, it uses a Pyramid "tween", which is sort of like internal Pyramid middleware inasmuch as its a user-manageable functional composition terminating at the application. See http://www.slideshare.net/aconrad/alex-conrad-pyramid-tweens-ploneconf-2011
Tweens can be reordeded arbitrarily, and the Pyramid exception handling logic (which locates and executes a Pyramid view when an application exception is raised) is itself a tween.
The rationale is that application-specific error logic often needs to be executed after the transaction has been committed or aborted due to an exception. For example, if an exception is raised by an application, you'd like the system to abort the transaction, then potentially display a custom internal server error page (e.g. twitter fail-whale). People want to be able to write the error logic within their existing Pyramid development environment (where they have access to templating systems, a real request object, and possibly their database and other niceties) instead of as WSGI middleware higher in the stack than the transaction manager.
I mostly agree with this, and for this reason I think the ZPublisher.Pubevents (not quite as good as tweens, I admit) and exception views are useful. With a middleware solution, you may find yourself passing application state in the wsgi environ which seems wrong.
FWIW, I believe Tres still disagrees strongly with this decision.
I can certainly see the use case for distributing transactions across middleware and/or composite apps, which is why it may make sense for this to be opt-in. Martin
Op 1 jan 2012, om 20:39 heeft Martin Aspeli het volgende geschreven:
Hi,
Hello,
There are three known WSGI implementations of the Zope 2 publisher. I've had a look at them and made some notes about what I think provides the best story:
## Zope 2.13 WSGIPublisher
[...]
## infrae.wsgi
Pros:
* Clean and well documented * Properly emits publication events * Supports streaming
Those two points are features I use actively in Silva, and where motivation for me to work on my implementation. (I had a look before to repoze.zope2 and the default Zope 2 support, back in 2.12).
* Supports simplified virtual hosting with X-VHM-Host
That is not completely true. I support setting the hostname, however to set a virtual path, you need to do this during traversing, which is done in BaseRequest, that I don't change (because it is a big large blob of code where you cannot really plug anything in it, or change only a few line in it without changing everything). In production we use mod_rewrite to rewrite the URL with an old VirtualHostMonster url and pass it to mod_wsgi with the help of the flags PT. When I boot Zope in infrae.wsgi, I don't do anything that is the responsibility of the server, like setup up user, logs, signals and so on, so that doesn't interfere with the WSGI server, so this works well with all the WSGI servers, including mod_wsgi (that was on of the issues with the default Zope 2 WSGI support).
* Supports exception handling / error views * Reportedly has significant production use
We have customers now using for more than one year in production, for large sites (30GB+ Data.fs) without problems.
Cons:
* Not 100% compatible (but close and fixable) - fix to make plone.transformchain work is here: https://gist.github.com/1547328
There are a couple of methods on the response that would probably a bit more of work in order to be a bit more compatible with what Zope 2 does. I just implemented what was required to get the ZMI and Silva work correctly.
* Unnecessary five.grok dependency (but easy to rewrite to use ZCML registration)
This is only used to register default error views (and the error log page).
* No support for middleware transaction and retry management, so these can't be distributed across a WSGI pipeline * Error logging will not support ZMI error_log and assumes single process
As matter of fact, I have more detailed log than the default Zope 2 error logging, that include lot of request information. I use after the filesystem logs afterward to investigate issues instead of going into of the ZMI. This is because we often don't have access to the ZMI of our customers, however they can send us the logs in case of problems. We use Paster as well in the stack, and use it to configure the Python logging process, you can configure there a logging to syslog, and after having a central syslog server. I think it is more professional like this, than using a tool in Zope's UI.
* Error handling is slightly different to standard publisher's exception views, and also does not honour existing standard_error_message etc
I did this mostly because for each error I wanted to know what was the last valid context I had before I got an error. This information is lost in how Zope 2 does it, because mostly the error handling goes through at least 5 different hooks. After that was to get something much more simple. [...]
## Suggested approach going forward
* Integrate infrae.wsgi into Zope 2
If that happens, like I said, we would probably need a bit more of work to be compatible with the standard Zope 2 response (for instance setHeader is aliased to addHeader, but they should behave a bit differently). I can help on my free time to work on any official WSGI support in Zope 2, and simplification of the publication process.
* Remove its five.grok dependency
That would be no problem.
* Use the same exception-views protocol as ZPublisher (mainly, that the view name is ``index.html``)
We could update it to be more compatible with Zope 2, and use the Zope 2 error log, but that would need some cleanup work in this last one which is overcomplicated for what it does.
* Stop using __ 'private' variables in response.py to make it easier to work with * Add some BBB support for existing error logging and error messages
Sure. Regards, Sylvain, -- Sylvain Viollon -- Infrae t +31 10 243 7051 -- http://infrae.com Hoevestraat 10 3033GC Rotterdam -- The Netherlands
On Jan 3, 2012, at 09:34 , Sylvain Viollon wrote:
We use Paster as well in the stack, and use it to configure the Python logging process, you can configure there a logging to syslog, and after having a central syslog server. I think it is more professional like this, than using a tool in Zope's UI.
Logging to syslog usually leads to problems due to the maximum message size for syslog entries. Error log messages with a traceback tend to be cut off and information lost. I'm not saying a ZMI tool is ideal, but syslog has pitfalls. jens
On 01/03/2012 10:04 AM, Jens Vagelpohl wrote:
On Jan 3, 2012, at 09:34 , Sylvain Viollon wrote:
We use Paster as well in the stack, and use it to configure the Python logging process, you can configure there a logging to syslog, and after having a central syslog server. I think it is more professional like this, than using a tool in Zope's UI. Logging to syslog usually leads to problems due to the maximum message size for syslog entries. Error log messages with a traceback tend to be cut off and information lost.
Don't log the entire error as one syslog entry, but use one syslog entry for every line in the error log. That is a standard way to dealing with this kind of thing. Wichert.
On 3 January 2012 08:34, Sylvain Viollon <sylvain@infrae.com> wrote:
Op 1 jan 2012, om 20:39 heeft Martin Aspeli het volgende geschreven:
Hi,
Hello,
There are three known WSGI implementations of the Zope 2 publisher. I've had a look at them and made some notes about what I think provides the best story:
## Zope 2.13 WSGIPublisher
[...]
## infrae.wsgi
Pros:
* Clean and well documented * Properly emits publication events * Supports streaming
Those two points are features I use actively in Silva, and where motivation for me to work on my implementation. (I had a look before to repoze.zope2 and the default Zope 2 support, back in 2.12).
* Supports simplified virtual hosting with X-VHM-Host
That is not completely true. I support setting the hostname, however to set a virtual path, you need to do this during traversing, which is done in BaseRequest, that I don't change (because it is a big large blob of code where you cannot really plug anything in it, or change only a few line in it without changing everything).
In production we use mod_rewrite to rewrite the URL with an old VirtualHostMonster url and pass it to mod_wsgi with the help of the flags PT.
What advantage is there to setting the X-VHM-Host header over just setting the Host header? Laurence
Op 8 jan 2012, om 16:36 heeft Laurence Rowe het volgende geschreven:
On 3 January 2012 08:34, Sylvain Viollon <sylvain@infrae.com> wrote:
Op 1 jan 2012, om 20:39 heeft Martin Aspeli het volgende geschreven:
[...]
* Supports simplified virtual hosting with X-VHM-Host
That is not completely true. I support setting the hostname, however to set a virtual path, you need to do this during traversing, which is done in BaseRequest, that I don't change (because it is a big large blob of code where you cannot really plug anything in it, or change only a few line in it without changing everything).
In production we use mod_rewrite to rewrite the URL with an old VirtualHostMonster url and pass it to mod_wsgi with the help of the flags PT.
What advantage is there to setting the X-VHM-Host header over just setting the Host header?
You can't set a virtual path with the Host header, it is not valid if you want to follow the RFC. After as well, you can include the port in it, but I think you are not sure if you can't really know if you should create https URL or not, you don't have this information (this is not required only to be on port 443, and if it is I am not even sure the port is correctly added in the Host header). I think it would be safer to have a separate header for all this information, and not touching the Host headers, that applications might expect to be correct. Regards, Sylvain, http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23 http://www.ietf.org/rfc/rfc2818.txt -- Sylvain Viollon -- Infrae t +31 10 243 7051 -- http://infrae.com Hoevestraat 10 3033GC Rotterdam -- The Netherlands
On Mon, Jan 09, 2012 at 10:01:29AM +0100, Sylvain Viollon wrote:
Op 8 jan 2012, om 16:36 heeft Laurence Rowe het volgende geschreven:
On 3 January 2012 08:34, Sylvain Viollon <sylvain@infrae.com> wrote:
Op 1 jan 2012, om 20:39 heeft Martin Aspeli het volgende geschreven:
[...]
* Supports simplified virtual hosting with X-VHM-Host
That is not completely true. I support setting the hostname, however to set a virtual path, you need to do this during traversing, which is done in BaseRequest, that I don't change (because it is a big large blob of code where you cannot really plug anything in it, or change only a few line in it without changing everything).
In production we use mod_rewrite to rewrite the URL with an old VirtualHostMonster url and pass it to mod_wsgi with the help of the flags PT.
What advantage is there to setting the X-VHM-Host header over just setting the Host header?
You can't set a virtual path with the Host header, it is not valid if you want to follow the RFC. After as well, you can include the port in it, but I think you are not sure if you can't really know if you should create https URL or not, you don't have this information (this is not required only to be on port 443, and if it is I am not even sure the port is correctly added in the Host header).
In the WSGI world it's usual to use the X-Forwarded-Scheme header to determine the scheme (http or https), AFAIU. See e.g. http://pythonpaste.org/wsgiproxy/#the-proxy http://pythonpaste.org/deploy/modules/config.html#paste.deploy.config.Prefix... http://packages.python.org/Deliverance/proxy.html http://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/deployment...
I think it would be safer to have a separate header for all this information, and not touching the Host headers, that applications might expect to be correct.
Regards,
Sylvain,
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.23 http://www.ietf.org/rfc/rfc2818.txt
Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
Op 9 jan 2012, om 12:44 heeft Marius Gedminas het volgende geschreven:
On Mon, Jan 09, 2012 at 10:01:29AM +0100, Sylvain Viollon wrote:
You can't set a virtual path with the Host header, it is not valid if you want to follow the RFC. After as well, you can include the port in it, but I think you are not sure if you can't really know if you should create https URL or not, you don't have this information (this is not required only to be on port 443, and if it is I am not even sure the port is correctly added in the Host header).
In the WSGI world it's usual to use the X-Forwarded-Scheme header to determine the scheme (http or https), AFAIU.
I have nothing against using those headers, even if there disparities between some of the implementations.
See e.g. http://pythonpaste.org/wsgiproxy/#the-proxy http://pythonpaste.org/deploy/modules/config.html#paste.deploy.config.Prefix... http://packages.python.org/Deliverance/proxy.html http://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/deployment...
Regards, Sylvain, -- Sylvain Viollon -- Infrae t +31 10 243 7051 -- http://infrae.com Hoevestraat 10 3033GC Rotterdam -- The Netherlands
On 1/11/12 5:59 AM, Sylvain Viollon wrote:
Op 9 jan 2012, om 12:44 heeft Marius Gedminas het volgende geschreven:
On Mon, Jan 09, 2012 at 10:01:29AM +0100, Sylvain Viollon wrote:
You can't set a virtual path with the Host header, it is not valid if you want to follow the RFC. After as well, you can include the port in it, but I think you are not sure if you can't really know if you should create https URL or not, you don't have this information (this is not required only to be on port 443, and if it is I am not even sure the port is correctly added in the Host header).
In the WSGI world it's usual to use the X-Forwarded-Scheme header to determine the scheme (http or https), AFAIU.
I have nothing against using those headers, even if there disparities between some of the implementations.
See e.g. http://pythonpaste.org/wsgiproxy/#the-proxy http://pythonpaste.org/deploy/modules/config.html#paste.deploy.config.Prefix... http://packages.python.org/Deliverance/proxy.html http://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/deployment...
FWIW I tried to get Zope2's WSGI working recently with repoze.vhm and failed: http://stackoverflow.com/questions/8865349/mod-wsgi-zope2-plone-virtual-host..., though the concept of using HTTP_X_VHM_* seemed to make sense at the time. Alex
Regards,
Sylvain,
-- Alex Clark · http://pythonpackages.com
participants (8)
-
Alex Clark -
Chris McDonough -
Jens Vagelpohl -
Laurence Rowe -
Marius Gedminas -
Martin Aspeli -
Sylvain Viollon -
Wichert Akkerman