Hi. I've been attempting to develop products with Zope 2.9, and am finding it increasingly difficult and slow to do so as Product refreshing no longer works. Google shows up a few results from mailing lists with somewhat negative responses (The gist I'm getting is that no-one 'in the know' wants to worry about fixing it, as they don't see a real need for it). However, as a zope restart seems to take a good 5-10 seconds on my high end workstation (AthlonX2 4800+) with all the essentiall products installed (Plone, archetypes etc, plus my application), it's starting to become a major problem. as an example, I spent about three hours today debugging a problem, 90% of which was wasted waiting for zope restarts after making small changes and adding trace print statements. If I was able to refresh the product I would likely have taken less than half an hour. Another major issue I've discovered is Zope hosting. We reguarly deploy custom sites on shared zope environments, and having to restart an entire server just to update one product severely breaks our uptime guarentee. I'm afraid we are going to have no choice but to either drop back to Zope 2.8, or ditch Zope entirely- something I'd prefer not to have to do with our large codebase. I'm hoping I can garner some support to get the zope developers to fix the problem. Failing that, ANY information pointing me in the right direction would be helpful, as I'm more than happy to fix it myself and submit a patch. If anyone else is finding this a problem, please reply in this thread. :-) Thanks in advance. James.
--On 7. August 2006 21:47:37 +1000 James Davies <zope-list@foomatic.net> wrote:
Hi. I've been attempting to develop products with Zope 2.9, and am finding it increasingly difficult and slow to do so as Product refreshing no longer works.
Google shows up a few results from mailing lists with somewhat negative responses (The gist I'm getting is that no-one 'in the know' wants to worry about fixing it, as they don't see a real need for it).
Refreshing was always a hack and never something I would call a feature. It was always a hack for *development* purposes but not for production.
However, as a zope restart seems to take a good 5-10 seconds on my high end workstation (AthlonX2 4800+) with all the essentiall products installed (Plone, archetypes etc, plus my application), it's starting to become a major problem.
You might blame Plone for shipping for three tons of frameworks?
as an example, I spent about three hours today debugging a problem, 90% of which was wasted waiting for zope restarts after making small changes and adding trace print statements. If I was able to refresh the product I would likely have taken less than half an hour.
I understand and fully agree with you but things are as they are. Even worser....Refresh does not work Five...
Another major issue I've discovered is Zope hosting. We reguarly deploy custom sites on shared zope environments, and having to restart an entire server just to update one product severely breaks our uptime guarentee.
Refresh only works in debug-mode. You want to tell us that you are running a production site in debug-mode?
I'm afraid we are going to have no choice but to either drop back to Zope 2.8, or ditch Zope entirely- something I'd prefer not to have to do with our large codebase.
I'm hoping I can garner some support to get the zope developers to fix the problem. Failing that, ANY information pointing me in the right direction would be helpful, as I'm more than happy to fix it myself and submit a patch.
Dieter Maurer wrote a RefreshTool product lately (not sure if this would solve your problem and not sure if he released it). I doubt that the Refresh situation will change in the near future. -aj
Andreas Jung wrote:
James Davies wrote:
Another major issue I've discovered is Zope hosting. We reguarly deploy custom sites on shared zope environments, and having to restart an entire server just to update one product severely breaks our uptime guarentee.
Refresh only works in debug-mode. You want to tell us that you are running a production site in debug-mode?
Since when does refresh only work in debug mode? Admittedly we are still on Zope 2.7.x, but it works in non-debug mode for me. When I need to update skin folders on our production server then I do that by forcing the skin product to refresh. For other products I restart the servers, but as Reinoud points out if you have multiple ZEO clients that doesn't result in any downtime (we currently have 6 ZEO clients) although it will result in a slower response for the first few hits after the restart.
--On 7. August 2006 12:42:38 +0000 Duncan Booth <duncan.booth@suttoncourtenay.org.uk> wrote:
Andreas Jung wrote:
James Davies wrote:
Another major issue I've discovered is Zope hosting. We reguarly deploy custom sites on shared zope environments, and having to restart an entire server just to update one product severely breaks our uptime guarentee.
Refresh only works in debug-mode. You want to tell us that you are running a production site in debug-mode?
Since when does refresh only work in debug mode? Admittedly we are still on Zope 2.7.x, but it works in non-debug mode for me.
Possibly I haven't used Refresh since Zope 2.7 :-) I considered it always to be a hack and it raised more problems than it solved. -aj
Andreas Jung wrote at 2006-8-7 13:56 +0200:
... Refreshing was always a hack and never something I would call a feature. It was always a hack for *development* purposes but not for production.
We are speaking about development, not about production.
... Refresh does not work Five...
Do you understand why? In my view, refreshing should work with Five. For this to work, we almost sure need to tell Five that the product was refreshed such that Five can reprocess the "*.zcml" files for the product.
...
Another major issue I've discovered is Zope hosting. We reguarly deploy custom sites on shared zope environments, and having to restart an entire server just to update one product severely breaks our uptime guarentee.
Refresh only works in debug-mode. You want to tell us that you are running a production site in debug-mode?
I think you err, Andreas. Refresh always works, at least the explicit refresh (do not know about the automatic one which I have never used).
...
I'm hoping I can garner some support to get the zope developers to fix the problem.
I am a great fan of refresh and would try to get it fixed if we had a need for it. But, we decided recently, to stick with Zope 2.8.1 until Zope 2.11 is released. This means, it may take some time that I need to get active... Nevertheless, it may come sooner as some external developpers are a fan of Five and want to install a newer version. As we all expect Five to be the reason that refresh no longer works, this might force me to look into this problem earlier than I currently think.
... Dieter Maurer wrote a RefreshTool product lately (not sure if this would solve your problem and not sure if he released it).
The refresh tool is employer's work and therefore not released. Furthermore, it solves a problem with dependent product refreshs but I am not sure whether it will solve the Five problem -- maybe, if it would be sufficient to refresh Five as well. @James: you may try this: refresh your product, then refresh Five. If this succeeds, then a technique similar to the RefreshTool may be sufficient. -- Dieter
--On 7. August 2006 20:30:11 +0200 Dieter Maurer <dieter@handshake.de> wrote:
Andreas Jung wrote at 2006-8-7 13:56 +0200:
... Refreshing was always a hack and never something I would call a feature. It was always a hack for *development* purposes but not for production.
We are speaking about development, not about production.
We're also talking about techniques to avoid downtime. So we are are also speaking of production :-)
... Refresh does not work Five...
Do you understand why?
No, and I did not try.
In my view, refreshing should work with Five.
For this to work, we almost sure need to tell Five that the product was refreshed such that Five can reprocess the "*.zcml" files for the product.
I think this is basically a Zope 3 issue and not a Zope 2 issue. There were also some approaches in Z3 to make browser views somehow refreshable but I nothing that appeared in a release so far.
...
Another major issue I've discovered is Zope hosting. We reguarly deploy custom sites on shared zope environments, and having to restart an entire server just to update one product severely breaks our uptime guarentee.
Refresh only works in debug-mode. You want to tell us that you are running a production site in debug-mode?
I think you err, Andreas.
See my later posting.
Refresh always works, at least the explicit refresh (do not know about the automatic one which I have never used).
...
I'm hoping I can garner some support to get the zope developers to fix the problem.
I am a great fan of refresh and would try to get it fixed if we had a need for it.
Nevertheless, it may come sooner as some external developpers are a fan of Five and want to install a newer version. As we all expect Five to be the reason that refresh no longer works, this might force me to look into this problem earlier than I currently think.
This is basically an issue when you work with Plone. Loading the complete boilerplate really takes ages. An instance running on a decent machine will start fast when you work with CMF & friends. -aj
Andreas Jung wrote at 2006-8-7 20:43 +0200:
...
Nevertheless, it may come sooner as some external developpers are a fan of Five and want to install a newer version. As we all expect Five to be the reason that refresh no longer works, this might force me to look into this problem earlier than I currently think.
This is basically an issue when you work with Plone. Loading the complete boilerplate really takes ages. An instance running on a decent machine will start fast when you work with CMF & friends.
We know that already AT (and CMF) cause slow startup. Probably, Plone will make it worse but even without it, we want something comparable to "refresh". -- Dieter
Aloha,
Refresh always works, at least the explicit refresh (do not know about the automatic one which I have never used).
I'm on 2.7 and yes, the explicit and *very* useful refresh tab in the ZMI seems to work fine in production mode also. I do not remember fondly the old days (2.4, 2.5) of having to restart zope for every little product change/tweak/debug effort during development, or when doing a minor product upgrade to production server. I was going to upgrade to 2.9 very soon but now may go only to 2.8. I don't have the technical chops to work on a refresh solution in 2.9+, however I heartily support such efforts from those of you who also see it as an important issue. cheers, John S. -- John Schinnerer - MA, Whole Systems Design ------------------------------------------ - Eco-Living - Whole Systems Design Services People - Place - Learning - Integration john@eco-living.net http://eco-living.net
The real effort towards making a "better refresh" should likely be spent at the level of the Python interpreter. The dynamic nature of Python is the thing that allows for a "refresh" in the first place, but the implementation of Python object references limits its usefulness. The implementation of "reload" could be vastly improved by changing Python internally to allow for a sort of mark-and-sweep reload that could traverse an object graph full of references between an object and the modules that it is declared in. Anything else will likely be too fragile to work in all cases. On Aug 7, 2006, at 4:36 PM, John Schinnerer wrote:
Aloha,
Refresh always works, at least the explicit refresh (do not know about the automatic one which I have never used).
I'm on 2.7 and yes, the explicit and *very* useful refresh tab in the ZMI seems to work fine in production mode also.
I do not remember fondly the old days (2.4, 2.5) of having to restart zope for every little product change/tweak/debug effort during development, or when doing a minor product upgrade to production server.
I was going to upgrade to 2.9 very soon but now may go only to 2.8. I don't have the technical chops to work on a refresh solution in 2.9+, however I heartily support such efforts from those of you who also see it as an important issue.
cheers, John S.
--
John Schinnerer - MA, Whole Systems Design ------------------------------------------ - Eco-Living - Whole Systems Design Services People - Place - Learning - Integration john@eco-living.net http://eco-living.net _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Since I never knew the refresh tab was a fragile hack, it always worked fine for me... :-) Are you saying that a "really good" solution for this is in the hands of python developers, not zope developers? In that case it seems far off, if ever, since zope is only one of many many applications using python. I would settle for another fragile hack that works, I guess. The volume of posts so far would seem to indicate that it's not a trivial topic... John S. Chris McDonough wrote:
The real effort towards making a "better refresh" should likely be spent at the level of the Python interpreter. The dynamic nature of Python is the thing that allows for a "refresh" in the first place, but the implementation of Python object references limits its usefulness.
The implementation of "reload" could be vastly improved by changing Python internally to allow for a sort of mark-and-sweep reload that could traverse an object graph full of references between an object and the modules that it is declared in. Anything else will likely be too fragile to work in all cases.
On Aug 7, 2006, at 4:36 PM, John Schinnerer wrote:
Aloha,
Refresh always works, at least the explicit refresh (do not know about the automatic one which I have never used).
I'm on 2.7 and yes, the explicit and *very* useful refresh tab in the ZMI seems to work fine in production mode also.
I do not remember fondly the old days (2.4, 2.5) of having to restart zope for every little product change/tweak/debug effort during development, or when doing a minor product upgrade to production server.
I was going to upgrade to 2.9 very soon but now may go only to 2.8. I don't have the technical chops to work on a refresh solution in 2.9+, however I heartily support such efforts from those of you who also see it as an important issue.
cheers, John S.
--
John Schinnerer - MA, Whole Systems Design ------------------------------------------ - Eco-Living - Whole Systems Design Services People - Place - Learning - Integration john@eco-living.net http://eco-living.net _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- John Schinnerer - MA, Whole Systems Design ------------------------------------------ - Eco-Living - Whole Systems Design Services People - Place - Learning - Integration john@eco-living.net http://eco-living.net
On Aug 7, 2006, at 5:57 PM, John Schinnerer wrote:
Since I never knew the refresh tab was a fragile hack, it always worked fine for me... :-)
Bliss! I'm happy for you! FWIW, you've just had it nail you now, whereas I'd had it nail me four years ago, and swore it off then. ;-)
Are you saying that a "really good" solution for this is in the hands of python developers, not zope developers?
Yes.
In that case it seems far off, if ever, since zope is only one of many many applications using python.
There are many non-Zope applications that could benefit from better reload support in Python including any application that could benefit from running continuously without restart. Improving Python this way is a "rising tide floats all boats" sort of thing, maybe worthy of some sort of sprint at PyCon or something like it.
I would settle for another fragile hack that works, I guess. The volume of posts so far would seem to indicate that it's not a trivial topic...
Honestly I have no idea how trivial or nontrivial fixing the current Refresh implementation is, and frankly (shame on me, yes, I'm selfish, I'm sorry) I don't care. It doesn't hurt me because I don't develop using all of Plone often, so my startup times are reasonably acceptable. Even if I did develop using all of Plone often, I'd likely try to find the root causes of the slow startup and fix them or fix Python rather than fixing the Zope Rrefresh implementation. I realize that by having that position, I'm likely in the minority as a consumer, but unfortunately for those consumers, my position is likely shared by the majority of producers (core developers). ;-) - C
On 8/7/06, Dieter Maurer <dieter@handshake.de> wrote:
For this to work, we almost sure need to tell Five that the product was refreshed such that Five can reprocess the "*.zcml" files for the product.
Yeah, and we don't, and ZCML isn't designed for reprocessing, so I'm not at all sure it's easy (but it may be). I haven't looked deeply into this, I try to be testdriven, and then refresh is not a problem. :) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
On Mon, Aug 07, 2006 at 09:47:37PM +1000, James Davies wrote:
Another major issue I've discovered is Zope hosting. We reguarly deploy custom sites on shared zope environments, and having to restart an entire server just to update one product severely breaks our uptime guarentee.
If you use a ZEO setup you can restart the ZEO clients one-by-one. Works for me without downtime... -- __________________________________________________ "Nothing is as subjective as reality" Reinoud van Leeuwen reinoud.v@n.leeuwen.net http://www.xs4all.nl/~reinoud __________________________________________________
We have Zeo, but running a seperate instance of zope for each site is still too much (We probably host around 50 sites at the moment, and growing. The only economical way to do this is to put multiple sites in one zope instance). On 8/7/06, Reinoud van Leeuwen <reinoud.v@n.leeuwen.net> wrote:
On Mon, Aug 07, 2006 at 09:47:37PM +1000, James Davies wrote:
Another major issue I've discovered is Zope hosting. We reguarly deploy custom sites on shared zope environments, and having to restart an entire server just to update one product severely breaks our uptime guarentee.
If you use a ZEO setup you can restart the ZEO clients one-by-one. Works for me without downtime...
-- __________________________________________________ "Nothing is as subjective as reality" Reinoud van Leeuwen reinoud.v@n.leeuwen.net http://www.xs4all.nl/~reinoud __________________________________________________ _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Mon, Aug 07, 2006 at 10:12:42PM +1000, James Davies wrote:
We have Zeo, but running a seperate instance of zope for each site is still too much (We probably host around 50 sites at the moment, and growing. The only economical way to do this is to put multiple sites in one zope instance).
Still you can shutdown 1 ZEO client; update products there and start it up again... There will be some performance degradation but the sites stay up... (and I hope you've got enough ZEO clients so that bringing 1 down will not cause severe problems...) -- __________________________________________________ "Nothing is as subjective as reality" Reinoud van Leeuwen reinoud.v@n.leeuwen.net http://www.xs4all.nl/~reinoud __________________________________________________
Reinoud van Leeuwen wrote at 2006-8-7 14:15 +0200:
On Mon, Aug 07, 2006 at 10:12:42PM +1000, James Davies wrote:
We have Zeo, but running a seperate instance of zope for each site is still too much (We probably host around 50 sites at the moment, and growing. The only economical way to do this is to put multiple sites in one zope instance).
Still you can shutdown 1 ZEO client; update products there and start it up again... There will be some performance degradation but the sites stay up...
You need to restart all ZEO clients such that they see the new product all. Furthermore, your session data must be stored persistently for this to work (which is considerably less efficient that storing them in RAM). -- Dieter
This isn't really a solution. Clients will still lose session data, which is unacceptable. -James. On Monday 07 August 2006 22:15, Reinoud van Leeuwen wrote:
On Mon, Aug 07, 2006 at 10:12:42PM +1000, James Davies wrote:
We have Zeo, but running a seperate instance of zope for each site is still too much (We probably host around 50 sites at the moment, and growing. The only economical way to do this is to put multiple sites in one zope instance).
Still you can shutdown 1 ZEO client; update products there and start it up again... There will be some performance degradation but the sites stay up... (and I hope you've got enough ZEO clients so that bringing 1 down will not cause severe problems...)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 James Davies wrote:
This isn't really a solution. Clients will still lose session data, which is unacceptable.
-James.
On Monday 07 August 2006 22:15, Reinoud van Leeuwen wrote:
On Mon, Aug 07, 2006 at 10:12:42PM +1000, James Davies wrote:
We have Zeo, but running a seperate instance of zope for each site is still too much (We probably host around 50 sites at the moment, and growing. The only economical way to do this is to put multiple sites in one zope instance). Still you can shutdown 1 ZEO client; update products there and start it up again... There will be some performance degradation but the sites stay up... (and I hope you've got enough ZEO clients so that bringing 1 down will not cause severe problems...)
You have that point-of-failure anyway if you are using the RAM-based sessions provided by Zope's core sessioning. If availability is important to you, then you *need* to be using ZEO and a load balancer, at which point you also have to look at a more scalable / available sessioning solution. There are a couple of alternatives out there: - SQL-based session implementaitons (including some which will drop into your application "seamlessly"), e.g.: http://www.zope.org/Members/anthony/software/SQLSession (really old, might need some tweaking). - A memcached-based solutions, e.g.: http://agendaless.com/Members/tseaver/software/mcdutils/ Refresh is dangerous because it mostly works, and because when it breaks, the breakage surfaces as heisenbugs. I therefore have a rule about supporting folks who use refresh: if they can't reproduce the problem with refresh disabled, I can't help them. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE2Lqw+gerLs4ltQ4RAv6bAKCr7ooZQq0ZEYoxw3J6uq9bKrgGOACgp5is cmP3uyTUBCOMk8Mv0Xk24R8= =a6Ui -----END PGP SIGNATURE-----
... You have that point-of-failure anyway if you are using the RAM-based sessions provided by Zope's core sessioning. If availability is important to you, then you *need* to be using ZEO and a load balancer, at which point you also have to look at a more scalable / available sessioning solution.
We are using ZEO, a load balancer and nevertheless have sessions in RAM. True, it is less safe than with persistent sessions -- but it is also more efficient. We do this by ensuring that independent of the load balancer's choice a request for a given session goes to that Zope instance that has this session. -- Dieter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dieter Maurer wrote:
... You have that point-of-failure anyway if you are using the RAM-based sessions provided by Zope's core sessioning. If availability is important to you, then you *need* to be using ZEO and a load balancer, at which point you also have to look at a more scalable / available sessioning solution.
We are using ZEO, a load balancer and nevertheless have sessions in RAM.
True, it is less safe than with persistent sessions -- but it is also more efficient.
We do this by ensuring that independent of the load balancer's choice a request for a given session goes to that Zope instance that has this session.
Then you have the problem the orignal poster was trying to avoid: you can't restart an appserver without losing session data for some users. Externalizing the session storage removes the appserver as a point-of-failure for user sessions. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFE2iTZ+gerLs4ltQ4RAn3HAKDNYodL3ENdV/jrQZhmEEtMkIQnbwCfbT+5 xH3krOiXo7KRd5G+DiEp92I= =WLfz -----END PGP SIGNATURE-----
On 8/7/06, James Davies <zope-list@foomatic.net> wrote:
Hi. I've been attempting to develop products with Zope 2.9, and am finding it increasingly difficult and slow to do so as Product refreshing no longer works.
Google shows up a few results from mailing lists with somewhat negative responses (The gist I'm getting is that no-one 'in the know' wants to worry about fixing it, as they don't see a real need for it).
Yeah, too bad, but making it better is complicated, and it can never be perfect anyway.
Another major issue I've discovered is Zope hosting. We reguarly deploy custom sites on shared zope environments, and having to restart an entire server just to update one product severely breaks our uptime guarentee.
You mean you can't allow the server even to be down for the less than a minute it takes to restart it? I would honestly be very interested in knowing what kind of systems you run that has that sort of requirements. But in any case, as mentioned by others here, multiple ZEO-clients and load balancing fixes this for most cases.
I'm hoping I can garner some support to get the zope developers to fix the problem.
Well, I'm sure it can be improved by throwing money on the right people... ;) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
On Tuesday 08 August 2006 01:34, you wrote:
You mean you can't allow the server even to be down for the less than a minute it takes to restart it? I would honestly be very interested in knowing what kind of systems you run that has that sort of requirements.
Are you serious? People pay large sums of money for us to host there critical business infrastructure. If we have to restart zope for any reason, people get apache proxy errors and lose session data. -James.
On 8/8/06, James Davies <zope-list@foomatic.net> wrote:
People pay large sums of money for us to host there critical business infrastructure. If we have to restart zope for any reason, people get apache proxy errors and lose session data.
Well, right the session data would be annoying. But still, most companies do accept that you schedule upgrades, and I still would like to know what kind of business can't accept a one-minute scheduled outage from time to time. I sure never have met one. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
FWIW, the session data problem can be solved with the use of a ZODB session data container that is not stored in RAM (e.g. an SDC that is not /temp_folder/session_data), or a reimplentation of a session manager that uses a relational database or the filesystem. The proxy error issue should be solved by using multiple Zope servers and a load balancer which allows you to remove each from the service pool before restarting it. - C On Tue, 2006-08-08 at 12:24 +0200, Lennart Regebro wrote:
On 8/8/06, James Davies <zope-list@foomatic.net> wrote:
People pay large sums of money for us to host there critical business infrastructure. If we have to restart zope for any reason, people get apache proxy errors and lose session data.
Well, right the session data would be annoying. But still, most companies do accept that you schedule upgrades, and I still would like to know what kind of business can't accept a one-minute scheduled outage from time to time. I sure never have met one.
James Davies wrote at 2006-8-7 21:47 +1000:
I've been attempting to develop products with Zope 2.9, and am finding it increasingly difficult and slow to do so as Product refreshing no longer works.
What happens when you try to refresh a product? Does it happen only when the product uses "Five"? Or can even products not using "Five" not be refreshed? -- Dieter
Dieter Maurer wrote:
James Davies wrote at 2006-8-7 21:47 +1000:
I've been attempting to develop products with Zope 2.9, and am finding it increasingly difficult and slow to do so as Product refreshing no longer works.
What happens when you try to refresh a product?
Nothing. No errors, no refresh.
Does it happen only when the product uses "Five"? I've only tried refreshing a non-Five product in Zope 2.9 and it doesn't work. Or can even products not using "Five" not be refreshed? Yes.
A side note... I've followed this thread in the mailinglist and it seems that people get sidetracked into maintaining the session state and/or restarting zope via zeo clients. What REALLY concerns me is development. Not production. If I had to restart Zope every time I make a change I want to test I'd loose several hours worth of valuable work time every week. Plus every time you have to wait for Zope to restart there's a silent moment of waiting which can be distracting so that you loose the flow. Very unproductive. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
----- Original Message ----- From: "Peter Bengtsson" <peter@fry-it.com> To: "Dieter Maurer" <dieter@handshake.de> Cc: <zope@zope.org> Sent: Wednesday, August 16, 2006 5:47 AM Subject: Re: [Zope] Zope 2.9 Product Refreshing
Dieter Maurer wrote:
James Davies wrote at 2006-8-7 21:47 +1000:
I've been attempting to develop products with Zope 2.9, and am finding it increasingly difficult and slow to do so as Product refreshing no longer works.
What happens when you try to refresh a product?
Nothing. No errors, no refresh.
Does it happen only when the product uses "Five"? I've only tried refreshing a non-Five product in Zope 2.9 and it doesn't work. Or can even products not using "Five" not be refreshed? Yes.
A side note... I've followed this thread in the mailinglist and it seems that people get sidetracked into maintaining the session state and/or restarting zope via zeo clients. What REALLY concerns me is development. Not production. If I had to restart Zope every time I make a change I want to test I'd loose several hours worth of valuable work time every week. Plus every time you have to wait for Zope to restart there's a silent moment of waiting which can be distracting so that you loose the flow. Very unproductive.
I am developing under zope 2.9.2 and have adopted a slightly different approach which minimizes the impact of the 'broken refresh': I only use Products for class definitions, all other code goes into external methods (which are automatically 'refreshed' if you have debug turned on). There are no delays in development using this approach (the code/test cycle is immediate), but whether or not you can use this methodology depends on your overall design approach. Jonathan
Peter Bengtsson wrote at 2006-8-16 10:47 +0100:
Dieter Maurer wrote:
James Davies wrote at 2006-8-7 21:47 +1000:
I've been attempting to develop products with Zope 2.9, and am finding it increasingly difficult and slow to do so as Product refreshing no longer works.
What happens when you try to refresh a product?
Nothing. No errors, no refresh.
Then you should look in the SVN repository how the "refresh" code changed. In Zope 2.8, refresh moves all modules starting with "Products.<product_to_be_refreshed>" out of "sys.modules" (and into a safe place) and then reimports the product. If there is an exception, the saved modules are restored. Otherwise, the ZODB is told to discard the connection cache when the connection is opened for the next time. With this code, "no errors, no refresh" is impossible (you get either an error or a refesh). Of course, someone may have turned the refresh into a "noop". Then, you can try to reinstate the former code and see what happens. -- Dieter
What happens when you try to refresh a product?
Nothing. No errors, no refresh.
Then you should look in the SVN repository how the "refresh" code changed.
In Zope 2.8, refresh moves all modules starting with "Products.<product_to_be_refreshed>" out of "sys.modules" (and into a safe place) and then reimports the product.
If there is an exception, the saved modules are restored. Otherwise, the ZODB is told to discard the connection cache when the connection is opened for the next time.
With this code, "no errors, no refresh" is impossible (you get either an error or a refesh). No really. My last comment was a big ambiguous. The Refresh tab of the Control_Panel is does say "Product refreshed. (2006-08-18 18:33)" But it actually fails. It gets more complicated...
Of course, someone may have turned the refresh into a "noop". Then, you can try to reinstate the former code and see what happens.
I've now done a lot of searching and diffs on the difference between some zope286 and some zope294 code. The modules I've looked at are: ZODB.Connection.resetCaches() - no difference ZODB.Connection - BIG difference (but is it applicable?) OFS.Application - instead of zLOG->logger, tiny difference in raise App.RefreshFuncs - instead of zLOG->logger I've also compared the difference between python2.3 and python2.4's __import__ builtin function and I can't see any difference. On that note, I started my zope286 with python2.4 but refreshing was still possible so zope2.9 requiring python2.4 does not seem to be the reason. I'm quite stuck right now. I haven't yet really dug into how sys.modules work. The Zope code doesn't seem to have changed inside the function import_product() of OFS.Application. One last odd clue, when I refresh my product (eg. MyProduct) in zope 2.9.4, actual something happens. My product is very very simple and it's one main class has a method called index_html() that looks like this: from DateTime import DateTime def index_html(self, REQUEST, RESPONSE): """ doc str """ return str(DateTime()) It works fine. After I manually refresh the product through the Control_Panel, I get an error in index_html() because now 'DateTime' has become None and thus can't be called with DateTime(). Does that help you help me? -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
Peter Bengtsson wrote at 2006-8-18 18:53 +0100:
... One last odd clue, when I refresh my product (eg. MyProduct) in zope 2.9.4, actual something happens. My product is very very simple and it's one main class has a method called index_html() that looks like this:
from DateTime import DateTime def index_html(self, REQUEST, RESPONSE): """ doc str """ return str(DateTime())
It works fine. After I manually refresh the product through the Control_Panel, I get an error in index_html() because now 'DateTime' has become None and thus can't be called with DateTime(). Does that help you help me?
Global variables apparently becoming "None" is a sign that you are using an old (pre refresh) version of a function ("index_html" in your case). When a module is released, some Python magic arranges that all its variables are rebound to "None". When the function accesses one of its globals, it then appears as "None". If you have accessed the "index_html" via a persistent instance, then the "resetCaches" seems not to have done what we expect (it should have caused the connection cache to be dropped and a new instance loaded from the storage with the new class definition). -- Dieter
Dieter Maurer wrote:
Peter Bengtsson wrote at 2006-8-18 18:53 +0100:
... One last odd clue, when I refresh my product (eg. MyProduct) in zope 2.9.4, actual something happens. My product is very very simple and it's one main class has a method called index_html() that looks like this:
from DateTime import DateTime def index_html(self, REQUEST, RESPONSE): """ doc str """ return str(DateTime())
It works fine. After I manually refresh the product through the Control_Panel, I get an error in index_html() because now 'DateTime' has become None and thus can't be called with DateTime(). Does that help you help me?
Global variables apparently becoming "None" is a sign that you are using an old (pre refresh) version of a function ("index_html" in your case).
When a module is released, some Python magic arranges that all its variables are rebound to "None".
When the function accesses one of its globals, it then appears as "None".
If you have accessed the "index_html" via a persistent instance, then the "resetCaches" seems not to have done what we expect (it should have caused the connection cache to be dropped and a new instance loaded from the storage with the new class definition).
The function resetCaches() and self._resetCache() in ZODB.Connection hasn't changed from zope 2.8 to zope 2.9. ==== the code ========= global_reset_counter = 0 def resetCaches(): """Causes all connection caches to be reset as connections are reopened. Zope's refresh feature uses this. When you reload Python modules, instances of classes continue to use the old class definitions. To use the new code immediately, the refresh feature asks ZODB to clear caches by calling resetCaches(). When the instances are loaded by subsequent connections, they will use the new class definitions. """ global global_reset_counter global_reset_counter += 1 class Connection(ExportImport, object): .... def _resetCache(self): """Creates a new cache, discarding the old one. See the docstring for the resetCaches() function. """ self._reset_counter = global_reset_counter self._invalidated.clear() cache_size = self._cache.cache_size self._cache = cache = PickleCache(self, cache_size) ======================== -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
Peter Bengtsson wrote at 2006-8-21 10:13 +0100:
... global variables turn to 'None' after refresh ...
If you have accessed the "index_html" via a persistent instance, then the "resetCaches" seems not to have done what we expect (it should have caused the connection cache to be dropped and a new instance loaded from the storage with the new class definition).
The function resetCaches() and self._resetCache() in ZODB.Connection hasn't changed from zope 2.8 to zope 2.9.
Nevertheless, your "index_html" seems to be old. Apparently, the old module instance was dropped (this dropping caused the global variable to get 'None'). You should now have a new module instance in its place in "sys.modules" (reflecting your code change). If the ZODB connection cache was replaced by a new one, then the self of your "index_html" is newly loaded from the storage. This should give it the new class definition (where "index_html" references the new module instance). Something along this line appears to be wrong. You need to investigate what it is. -- Dieter
participants (11)
-
Andreas Jung -
Chris McDonough -
Dieter Maurer -
Duncan Booth -
James Davies -
John Schinnerer -
Jonathan -
Lennart Regebro -
Peter Bengtsson -
Reinoud van Leeuwen -
Tres Seaver