Product refreshing in Zope 2.9
It appears that product refreshing is futile in Zope 2.9 which is according to zope.org the current stable version :( Think again. According to: http://comments.gmane.org/gmane.comp.web.zope.z3base.five/977 Philipp von Weitershausen informs us that if you have a product that uses Five in zope2 and you try to refresh the product it simply won't work. Your only hope is to write tests which I'm not interested in. I only write unit tests from non-trivial things that requires thinking. If I decide to include one more template or a new non-trivial function I don't want to have to restart Zope. But, in that example they are using Five and I'm not. I've just got a Zope 2.9 install and a python product that I started on before Five even existed. Does that give us any hope for non-Five python products? Anybody? I've tried pressing the little "Refresh" button in Control_Panel and sure enough, the product is reloaded (I put some print statements in the .py file) but not refreshed. If it's just a bug I'm willing to help out with getting it fixed. Peter -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
Peter Bengtsson wrote at 2006-1-27 16:42 +0000:
... But, in that example they are using Five and I'm not. I've just got a Zope 2.9 install and a python product that I started on before Five even existed. Does that give us any hope for non-Five python products? Anybody?
I cannot believe that they were able to break refreshability of "normal" (non Five) products. Refreshing essentially works by removing the modules belonging to the product from "sys.module" and then reimporting the product. This may fail when product defined objects are held in registries (such as adapters) and are not overridden when the product is reimported. Maybe, such cases occur often with Five. Without Five, it is rather rare (product objects are often registered but usually, the reimport overrides the old no longer working entries).
I've tried pressing the little "Refresh" button in Control_Panel and sure enough, the product is reloaded (I put some print statements in the .py file) but not refreshed.
"reloaded but not refreshed" means what? -- Dieter
On 1/27/06, Dieter Maurer <dieter@handshake.de> wrote:
Peter Bengtsson wrote at 2006-1-27 16:42 +0000:
... But, in that example they are using Five and I'm not. I've just got a Zope 2.9 install and a python product that I started on before Five even existed. Does that give us any hope for non-Five python products? Anybody?
I cannot believe that they were able to break refreshability of "normal" (non Five) products.
Refreshing essentially works by removing the modules belonging to the product from "sys.module" and then reimporting the product.
Ok. Still quite clueless on how to debug it. I'll see what I can do with some greps and various print statements.
This may fail when product defined objects are held in registries (such as adapters) and are not overridden when the product is reimported. Maybe, such cases occur often with Five.
Without Five, it is rather rare (product objects are often registered but usually, the reimport overrides the old no longer working entries).
I've tried pressing the little "Refresh" button in Control_Panel and sure enough, the product is reloaded (I put some print statements in the .py file) but not refreshed.
"reloaded but not refreshed" means what?
Suppose my code looks like this:: class MyProduct(Folder): blablabla print "class MyProduct has just been reloaded" Then I start zope with ./bin/runzopt, debug-mode off, make a change in the product and press the Refresh button in the Control_Panel and notice how it prints 'class MyProduct has just been reloaded' to stdout but the change I've made does not happen. A quick restart of Zope makes the change happen. That's what I meant. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
Peter Bengtsson schrieb:
Suppose my code looks like this::
class MyProduct(Folder): blablabla print "class MyProduct has just been reloaded"
Then I start zope with ./bin/runzopt, debug-mode off, make a change in the product and press the Refresh button in the Control_Panel and notice how it prints 'class MyProduct has just been reloaded' to stdout but the change I've made does not happen. A quick restart of Zope makes the change happen. That's what I meant.
Better your zLog or log instead of print. The refresh works most of the times - but its really a very bad hack and does not garanty for anything. If it does not work for your product you need to restart. Regards Tino
On 1/30/06, Tino Wildenhain <tino@wildenhain.de> wrote:
Peter Bengtsson schrieb:
Suppose my code looks like this::
class MyProduct(Folder): blablabla print "class MyProduct has just been reloaded"
Then I start zope with ./bin/runzopt, debug-mode off, make a change in the product and press the Refresh button in the Control_Panel and notice how it prints 'class MyProduct has just been reloaded' to stdout but the change I've made does not happen. A quick restart of Zope makes the change happen. That's what I meant.
Better your zLog or log instead of print.
The refresh works most of the times - but its really a very bad hack and does not garanty for anything. If it does not work for your product you need to restart.
That's not good enough. The only thing that changed between my development environments was the version of zope and then it stopped working. Python product refresh is crucial to zope2 and this problem needs to be sorted out. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
On 30 Jan 2006, at 15:01, Peter Bengtsson wrote:
That's not good enough. The only thing that changed between my development environments was the version of zope and then it stopped working. Python product refresh is crucial to zope2 and this problem needs to be sorted out.
I'm calling bullshit on this one. How is it "crucial"? A product that introduces its own strange problems cannot be called "crucial". Unless you're now signing up to support it and keep it working. jens
On 1/30/06, Jens Vagelpohl <jens@dataflake.org> wrote:
On 30 Jan 2006, at 15:01, Peter Bengtsson wrote:
That's not good enough. The only thing that changed between my development environments was the version of zope and then it stopped working. Python product refresh is crucial to zope2 and this problem needs to be sorted out.
I'm calling bullshit on this one. How is it "crucial"? A product that introduces its own strange problems cannot be called "crucial". Unless you're now signing up to support it and keep it working.
It's not bullshit. Bullshit is usually when you say something that isn't true. It's not a strange product. It's quite simple in fact. Not being able to refresh without restarting means that I can't use Zope 2.9 for python product development. It could just be something weird with my setup, but it could also be a bug in zope 2.9 which I'm happy to try to solve by participating in the discussion. PS. I'm working on a dummy product that other zope core developers can download that I can refer to for pointing out the bug. Watch this space. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
On 1/30/06, Peter Bengtsson <peter@fry-it.com> wrote:
It's not a strange product. It's quite simple in fact. Not being able to refresh without restarting means that I can't use Zope 2.9 for python product development.
Explain "Can't". I have developed complex products where the Refresh hasn't worked since it was integrated into Zope core, which was like, 2.4 or something. I can still use Zope for Python product development. Yes, a working refresh would have made it faster, no doubt. But it's in no means impossible.
It could just be something weird with my setup, but it could also be a bug in zope 2.9 which I'm happy to try to solve by participating in the discussion.
Yeah, well I would be very happy if the refresh could be improved, but sadly I don't understand it well enough, and those who do doesn't seem to care about improving it. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
--On 30. Januar 2006 17:39:11 +0000 Peter Bengtsson <peter@fry-it.com> wrote:
It's not a strange product. It's quite simple in fact. Not being able to refresh without restarting means that I can't use Zope 2.9 for python product development.
Why is refresh a requirement for doing Zope development? I am doing Zope development since five or six years and I have never done any development using Refresh...so there must be something very special with your development approach?! -aj
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Peter Bengtsson wrote:
On 1/30/06, Tino Wildenhain <tino@wildenhain.de> wrote:
Peter Bengtsson schrieb:
Suppose my code looks like this::
class MyProduct(Folder): blablabla print "class MyProduct has just been reloaded"
Then I start zope with ./bin/runzopt, debug-mode off, make a change in the product and press the Refresh button in the Control_Panel and notice how it prints 'class MyProduct has just been reloaded' to stdout but the change I've made does not happen. A quick restart of Zope makes the change happen. That's what I meant.
Better your zLog or log instead of print.
The refresh works most of the times - but its really a very bad hack and does not garanty for anything. If it does not work for your product you need to restart.
That's not good enough. The only thing that changed between my development environments was the version of zope and then it stopped working. Python product refresh is crucial to zope2 and this problem needs to be sorted out.
I don't agree that refresh is "crucial": refresh has always had "best effort" semantics, which leads many experienced Zope2 developers never to use it at all. Through bitter experience, they have learned that when refresht fails (silently), they have found that the hours spent figuring out why the application's behavior changed are much more expensive than the time it takes enter '<Ctrl-C><UpArrow><Enter>' in the zopectl shell. You'd be better off lobbying for faster restart times, including especially the third party products you use (a "stock" Zope restarts *very* quickly). Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFD3kjz+gerLs4ltQ4RAjI4AKDTUztzqizXkq1lha3ybk4mYfm3zgCeIzBV 01ChW3psdF86+Mp/90Cj3zk= =hrPZ -----END PGP SIGNATURE-----
For those interested I've put together a little basic zope python product here: http://www.peterbe.com/zope/2.9-refresh-reproduction/MyProduct.tgz To reproduce the refreshing problem: install this product, start zope, change something in the index_html() method, go to /Control_Panel/Products/MyProduct/manage_refresh, press refresh, and notice how the change doesn't take effect, restart zope and notice how that's the only way to refresh. About the importance of this... See below
That's not good enough. The only thing that changed between my development environments was the version of zope and then it stopped working. Python product refresh is crucial to zope2 and this problem needs to be sorted out.
I don't agree that refresh is "crucial": refresh has always had "best effort" semantics, which leads many experienced Zope2 developers never to use it at all. Through bitter experience, they have learned that when refresht fails (silently), they have found that the hours spent figuring out why the application's behavior changed are much more expensive than the time it takes enter '<Ctrl-C><UpArrow><Enter>' in the zopectl shell.
I've got fresh install of zope2.9 on my half-decent thinkpad with few but very light products (no CMF, Plone or TextIndexNG). A restart takes about 3-5 seconds buy human stopwatch. A product refresh in the Control Panel takes less than 1 second. The whole story is that I never use the Control_Panel manually; I have a spinning script that uses inotify on changes and visit the control panel programmatically using ZPublisher.Client I don't use auto-refresh on changes because of the risk of getting those strange ValueError's from the session machinery. And by the way, a zope restart makes you loose your SESSION. Point is, to **ME** product refreshing is much much more productive that zope restarts.
You'd be better off lobbying for faster restart times, including especially the third party products you use (a "stock" Zope restarts *very* quickly).
-- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
On 1/30/06, Peter Bengtsson <peter@fry-it.com> wrote:
class MyProduct(Folder): blablabla print "class MyProduct has just been reloaded"
Then I start zope with ./bin/runzopt, debug-mode off, make a change in the product and press the Refresh button in the Control_Panel and notice how it prints 'class MyProduct has just been reloaded' to stdout but the change I've made does not happen. A quick restart of Zope makes the change happen. That's what I meant.
You also need to reload all products that import this product, and (as I think was mentioned before) if you use any kind of global registy it will most likely fail. Personally I haven't had a situation trivial enough for refresh to work for years... -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
Lennart Regebro wrote at 2006-1-30 15:18 +0100:
... Personally I haven't had a situation trivial enough for refresh to work for years...
As I reported earlier, all that is needed is a small tool to allows to register product dependencies and that, when one product is refreshed, automatically refreshes all dependent products as well. Works very well and speeds up product development significantly. -- Dieter
On 1/30/06, Dieter Maurer <dieter@handshake.de> wrote:
Lennart Regebro wrote at 2006-1-30 15:18 +0100:
... Personally I haven't had a situation trivial enough for refresh to work for years...
As I reported earlier, all that is needed is a small tool to allows to register product dependencies and that, when one product is refreshed, automatically refreshes all dependent products as well. Works very well and speeds up product development significantly.
Uhm. But the refresh allows you to do that. I just usually doesn't get it to work. And now with Five, zcml becomes a refresh problem as well, so it's only half the solution for me anyway. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
Peter Bengtsson wrote at 2006-1-30 13:57 +0000:
...
"reloaded but not refreshed" means what?
Suppose my code looks like this::
class MyProduct(Folder): blablabla print "class MyProduct has just been reloaded"
Then I start zope with ./bin/runzopt, debug-mode off, make a change in the product and press the Refresh button in the Control_Panel and notice how it prints 'class MyProduct has just been reloaded' to stdout but the change I've made does not happen. A quick restart of Zope makes the change happen. That's what I meant.
Maybe, they forgot to flush the ZODB caches (in this case, old objects remain in the ZODB cache which still reference the old classes) or the "resetCaches" function no longer works in ZODB 3.6? Have a look at the sources... -- Dieter
I've noticed another strange behaviour with the Acquisition module in Zope 2.9 that might give us some clues as to why refreshing doesn't work. Imagine some code that looks like this:: from Acquisition import aq_parent class MyProduct(Folder): def index_html(self): print type(aq_parent) is None return "Hello World!" The first time I run this, it prints "False" to stdout because ac_parent is of course a function. BUT, if I refresh the product (no change made to the source) and run this index_html() again it instead prints "True" because ac_parent has become None. Isn't that odd? Again, this is easily reproducable with http://www.peterbe.com/zope/2.9-refresh-reproduction/MyProduct.tgz Any idea anyone? On 1/27/06, Dieter Maurer <dieter@handshake.de> wrote:
Peter Bengtsson wrote at 2006-1-27 16:42 +0000:
... But, in that example they are using Five and I'm not. I've just got a Zope 2.9 install and a python product that I started on before Five even existed. Does that give us any hope for non-Five python products? Anybody?
I cannot believe that they were able to break refreshability of "normal" (non Five) products.
Refreshing essentially works by removing the modules belonging to the product from "sys.module" and then reimporting the product.
This may fail when product defined objects are held in registries (such as adapters) and are not overridden when the product is reimported. Maybe, such cases occur often with Five.
Without Five, it is rather rare (product objects are often registered but usually, the reimport overrides the old no longer working entries).
I've tried pressing the little "Refresh" button in Control_Panel and sure enough, the product is reloaded (I put some print statements in the .py file) but not refreshed.
"reloaded but not refreshed" means what?
-- Dieter
-- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
Peter Bengtsson wrote at 2006-1-30 18:54 +0000:
I've noticed another strange behaviour with the Acquisition module in Zope 2.9 that might give us some clues as to why refreshing doesn't work.
Imagine some code that looks like this::
from Acquisition import aq_parent
class MyProduct(Folder): def index_html(self): print type(aq_parent) is None return "Hello World!"
The first time I run this, it prints "False" to stdout because ac_parent is of course a function. BUT, if I refresh the product (no change made to the source) and run this index_html() again it instead prints "True" because ac_parent has become None. Isn't that odd?
Indeed, especially because no type can be "None"! It is known that refreshing (more generally reloading a Python module) can cause objects apparently becoming "None". This happens because the variables defined in the old module instance are rebound to None when the module is released. This would explain that "aq_parent" in your "index_html" would appear as None if you happen to access an object still associated with the old class. However, the type of "None" is "NoneType" and not "None". I do not yet know what magic can cause "type(aq_parent)" (which usually is "<type 'builtin_function_or_method'>) to become "None". As written earlier: refreshing is essentially a Python feature (delete a set of modules from "sys.modules", then reimport some of them again) rather than a Zope feature. It should be possible to reproduce your weird observations with Python operations only. I tried but was not yet able to reproduce it (I still use Zope 2.8 (but that should not matter) and Python 2.4.1). -- Dieter
On 1/30/06, Dieter Maurer <dieter@handshake.de> wrote:
Peter Bengtsson wrote at 2006-1-30 18:54 +0000:
I've noticed another strange behaviour with the Acquisition module in Zope 2.9 that might give us some clues as to why refreshing doesn't work.
Imagine some code that looks like this::
from Acquisition import aq_parent
class MyProduct(Folder): def index_html(self): print type(aq_parent) is None return "Hello World!"
The first time I run this, it prints "False" to stdout because ac_parent is of course a function. BUT, if I refresh the product (no change made to the source) and run this index_html() again it instead prints "True" because ac_parent has become None. Isn't that odd?
Indeed, especially because no type can be "None"!
Oops! A typo. It should be:: def index_html(self): print aq_parent is None return "Hello World!"
It is known that refreshing (more generally reloading a Python module) can cause objects apparently becoming "None". This happens because the variables defined in the old module instance are rebound to None when the module is released.
This would explain that "aq_parent" in your "index_html" would appear as None if you happen to access an object still associated with the old class. However, the type of "None" is "NoneType" and not "None".
As written earlier: refreshing is essentially a Python feature (delete a set of modules from "sys.modules", then reimport some of them again) rather than a Zope feature. It should be possible to reproduce your weird observations with Python operations only. I tried but was not yet able to reproduce it (I still use Zope 2.8 (but that should not matter) and Python 2.4.1).
I noticed that since zope 2.8, the function manage_performRefresh() hasn't changed. The only difference I can see is that (my) zope 2.8.5 comes with ZODB 3.4.2 and my zope 2.9.0 comes with ZODB 3.6.0. The code is this:: if RefreshFuncs.performFullRefresh(self._p_jar, self.id): from ZODB import Connection Connection.resetCaches() # Clears cache in future connections. message = 'Product refreshed.' There is no code difference in RefreshFuncs.performFullRefresh() between zope 2.8.5 and zope 2.9.0. If someone can guide me I'd like to dig deeper into ZODB.Connection.resetCaches() to see if there's be a big difference in version 3.6 that might have caused my beloved refresh to stop working. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
Peter Bengtsson wrote at 2006-1-31 10:50 +0000:
... if RefreshFuncs.performFullRefresh(self._p_jar, self.id): from ZODB import Connection Connection.resetCaches() # Clears cache in future connections.
The call above is responsible to clear the ZODB caches. After your problem description corrections, it appears as if you were still accessing an old object. Two possibilities: * "resetCaches" no longer works reliable * you access the object from somewhere else then the ZODB cache (e.g. you store a persistent object on module or class level)
... If someone can guide me I'd like to dig deeper into ZODB.Connection.resetCaches() to see if there's be a big difference in version 3.6 that might have caused my beloved refresh to stop working.
"resetCaches" will almost surely only increment a number. The true cache flush happens in "Connection._setDB". Look there. -- Dieter
participants (7)
-
Andreas Jung -
Dieter Maurer -
Jens Vagelpohl -
Lennart Regebro -
Peter Bengtsson -
Tino Wildenhain -
Tres Seaver