Hi all, I have some problems adding properties trough a python script def publishcontainer(container,path): ov=map(lambda x:(x.id,x), container.objectValues()) ov.sort() for (oid,obj) in ov: if not (hasattr(obj,"isPortalContent") or obj.isPrincipiaFolderish): continue id="" if hasattr(obj,"isPortalContent"): print "ID: ",path+"/"+str(obj.id) try: context.manage_addProperty('prop', 0, 'boolean') print "prop added<br>" except: print "adding prop to",str(obj.id),"failed<br>" if hasattr(obj,"objectValues"): print publishcontainer(obj,path+"/"+str(obj.id)) return printed The problem is not getting the property added, it does, but editing it afterwards in the ZMI is impossible. When I hit the Properties tab of the item, the newly added property is not on the page. However when I call the prop with a dtml-var it is rendered. I'm at a loss here. Can someone help me out? Roel ----------------------------------------------------- Make cheaper and longer calls thanks to Scarlet Phone More info at http://www.scarlet.be/nl/consumer/phone/
rv000299 wrote at 2004-10-15 12:24 +0200:
... ov=map(lambda x:(x.id,x), container.objectValues())
You get the same effect with: ov = container.objectItems()
ov.sort() for (oid,obj) in ov: if not (hasattr(obj,"isPortalContent") or obj.isPrincipiaFolderish):
Be aware of acquisition. Both "hasattr" and attribute lookup may acquire and thus do not what you except. Use: "obj.aq_inner.aq_explicit" instead of "obj" at places where you want to avoid acquisition.
... try: context.manage_addProperty('prop', 0, 'boolean') print "prop added<br>" except: print "adding prop to",str(obj.id),"failed<br>"
Do not use "try: ... except:...". It is dangerous. It can easily lead to ZODB inconsistencies...
... The problem is not getting the property added, it does, but editing it afterwards in the ZMI is impossible. When I hit the Properties tab of the item, the newly added property is not on the page. However when I call the prop with a dtml-var it is rendered.
You add the property to "context" but, probably, you wanted it to get added to "obj"? -- Dieter
On Fri, 2004-10-15 at 19:32 +0200, Dieter Maurer wrote:
... try: context.manage_addProperty('prop', 0, 'boolean') print "prop added<br>" except: print "adding prop to",str(obj.id),"failed<br>"
Do not use "try: ... except:...". It is dangerous. It can easily lead to ZODB inconsistencies...
not exactly good news to me.... can you point out some document explaining this? can I raise exceptions from a python script? thanks massimo
On Fri, 2004-10-15 at 19:32 +0200, Dieter Maurer wrote:
... try: context.manage_addProperty('prop', 0, 'boolean') print "prop added<br>" except: print "adding prop to",str(obj.id),"failed<br>"
Do not use "try: ... except:...". It is dangerous. It can easily lead to ZODB inconsistencies...
not exactly good news to me....
can you point out some document explaining this?
mailing list archives? in general, don't use bare excepts; catch specific errors instead. Or, you can use a bare except IF you re-raise it; this is handy if you want to do some special logging or some such.
can I raise exceptions from a python script?
sure.
massimop@users.berlios.de wrote at 2004-10-16 12:23 +0200:
...
Do not use "try: ... except:...". It is dangerous. It can easily lead to ZODB inconsistencies...
not exactly good news to me....
can you point out some document explaining this?
It follows from common sense (and therefore need no documentation ;-) Python's exceptions are abortive: as soon an exceptin is raise, the normal flow of control is aborted. When you have modified persistent state before the exception occured, the transaction must usually be aborted, to get the modifications reverted. If instead, the transaction is committed, the (often) partial change is made persistent. Usually, the Zope framework does this for you. However, when you catch an exception, the framework may not learn about the problem and commit rather than abort. As I wrote, this can easily lead to corruption of persistent data.
can I raise exceptions from a python script?
Yes. -- Dieter
I believe that "try: ... except:..." code is only really dangerous if you do not specify the type of Exceptions that will be caught. The data inconsistencies become possible when something in the try statement raises a ConflictError, and your "naked" except statement catches it. It is generally good coding practice to always explicitly specify what Exception types you are catching. Assuming that there is only one possible reason that a given section of code might throw an exception, and therefore failing to specify what type of Exception it is that you are handling, can lead to some very hard to debug problems in general, and inconsistent data states with Zope in particular. --Sean
-----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org]On Behalf Of Dieter Maurer Sent: Sunday, October 17, 2004 4:44 PM To: massimop@users.berlios.de Cc: zope list Subject: Re: [Zope] adding properties trough pythonscript
massimop@users.berlios.de wrote at 2004-10-16 12:23 +0200:
...
Do not use "try: ... except:...". It is dangerous. It can easily lead to ZODB inconsistencies...
not exactly good news to me....
can you point out some document explaining this?
It follows from common sense (and therefore need no documentation ;-)
Python's exceptions are abortive: as soon an exceptin is raise, the normal flow of control is aborted.
When you have modified persistent state before the exception occured, the transaction must usually be aborted, to get the modifications reverted. If instead, the transaction is committed, the (often) partial change is made persistent.
Usually, the Zope framework does this for you. However, when you catch an exception, the framework may not learn about the problem and commit rather than abort.
As I wrote, this can easily lead to corruption of persistent data.
can I raise exceptions from a python script?
Yes.
-- Dieter _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Sean Hastings wrote at 2004-10-18 01:58 -0400:
I believe that "try: ... except:..." code is only really dangerous if you do not specify the type of Exceptions that will be caught.
I believe you are wrong: Catching *ANY* exception can cause inconsistencies in persistent data. It does (almost surely) when the "try" block modified persistent state before the exception occurred (and you did not ensure that the transaction is aborted). -- Dieter
On Oct 18, 2004, at 1:36 PM, Dieter Maurer wrote:
Sean Hastings wrote at 2004-10-18 01:58 -0400:
I believe that "try: ... except:..." code is only really dangerous if you do not specify the type of Exceptions that will be caught.
I believe you are wrong:
Catching *ANY* exception can cause inconsistencies in persistent data.
It does (almost surely) when the "try" block modified persistent state before the exception occurred (and you did not ensure that the transaction is aborted).
I have a large body of code written in the manner that Sean is describing, and after some time maintaining it, I'm now starting to lean towards what Dieter is saying. I have a system with an automated data import subsystem (for import of "wire feeds" and such) , an event channel alerting various subsystems about new changes throughout the system, and a web page creation subsystem that create special styles of pages and layout based on any criteria that it can query from the surrounding environment (for example, if it is creating a new page based on a politics story, it can add a related items link list, etc.) I've seen a few cases where the following happens, * A new story comes into the system from the feeds subsystem. * The feeds subsystem sends a "new content" message the event channel. * The event channel invokes page creation subsystem creates a new page. (doing a little "self._setObject( id, page );page = self._getOb( id )" dance to get a wrapped object instance and at that point altering the persistent state) * The page creation system then starts adding items to the page. One of them throws an exception, that is caught in the event channel that invoked it. The end result is a page that is missing some of its critical components. I'm starting to think that in this case, the code that handles the event channel should be less specific, rather than more explicit about the exceptions it should catch, along with a subtransaction abort and commit for each message handler it invokes. ( I definitely don't think the exception should bubble up any higher and abort the feeds import, because the other content coming through the feeds may be more important than this one story. Maybe lower into the page creation into the page creation subsystem might give more control over the error handling.) It definitely doesn't help that Python's dynamic nature here makes it very hard to predict which errors might be occurring. (for example, I noticed that the date parsing code for the DateTime object in 2.8a now gives a consistent exception for syntactically invalid input. Up to now, you could get some odd mix of DateTime.DateTime.DateError, DateTime.DateTime.TimeError, IndexError, KeyError, TypeError, or whatever else may be wrong with the input. Specific try:except:s are hard to develop when you have no specific enumeration of the errors that might be emitted for any particular method. (Don't get my wrong and take that as a condemnation of the python language as a whole. Its just one of those things where the more possibilities that you have before you, the more ways you can fail.)
I've seen a few cases where the following happens, * A new story comes into the system from the feeds subsystem.
* The feeds subsystem sends a "new content" message the event channel.
* The event channel invokes page creation subsystem creates a new page. (doing a little "self._setObject( id, page );page = self._getOb( id )" dance to get a wrapped object instance and at that point altering the persistent state)
* The page creation system then starts adding items to the page. One of them throws an exception, that is caught in the event channel that invoked it.
The end result is a page that is missing some of its critical components.
And this is not just a case where the missing components would have been added after the point in the control flow where the Exception was thrown, but before it was caught?
On Oct 19, 2004, at 2:29 PM, Sean Hastings wrote:
And this is not just a case where the missing components would have been added after the point in the control flow where the Exception was thrown, but before it was caught?
Essentially it is, but the end result is that you have a half-constructed object in the persistent store, which is what I thought Dieter meant by "inconsistencies in persistent data". If the steps could be re-arranged a bit so that the page isn't attached to the parent folder until it is fully constructed, there wouldn't be a problem. The page would either fully exist in the persistent store or be garbage collected. Since the page is first made persistent before the other steps, nearly anything other than letting an exception bubble all the way up to ZPublisher will commit the transaction and keep a reference to the page object. One other option is to use _delOb to try to undo the attachment of the new page to its folder, but there is no reason to be sure that _delOb will not fail.
Dieter, It may be that I do not understand some voodoo going on in Zope control flow - by what mechanism would either of the code examples below end up with foo and foo2 in a state inconsistent from the values passed in the last transaction to finish?
Catching *ANY* exception can cause inconsistencies in persistent data.
def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" self.foo = REQUEST.get('foo','') try: self.foo2 = int(REQUEST.get('foo2',0) except ValueError, e: self.foo2 = 0
It does (almost surely) when the "try" block modified persistent state before the exception occurred (and you did not ensure that the transaction is aborted).
def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" try: self.foo = REQUEST.get('foo','') self.foo2 = int(REQUEST.get('foo2',0) except ValueError, e: self.foo2 = 0 I am sure that you know more about both Python and Zope than I do, but I really don't want to believe that the design of Zope prohibits the use of such a fundamental part of the python language as raising and catching exceptions. --Sean
-----Original Message----- From: Dieter Maurer [mailto:dieter@handshake.de] Sent: Monday, October 18, 2004 1:36 PM To: Sean Hastings Cc: massimop@users.berlios.de; zope list Subject: RE: [Zope] adding properties trough pythonscript
Sean Hastings wrote at 2004-10-18 01:58 -0400:
I believe that "try: ... except:..." code is only really dangerous if you do not specify the type of Exceptions that will be caught.
I believe you are wrong:
Catching *ANY* exception can cause inconsistencies in persistent data.
It does (almost surely) when the "try" block modified persistent state before the exception occurred (and you did not ensure that the transaction is aborted).
-- Dieter
Sean Hastings wrote at 2004-10-19 14:20 -0400:
It may be that I do not understand some voodoo going on in Zope control flow - by what mechanism would either of the code examples below end up with foo and foo2 in a state inconsistent from the values passed in the last transaction to finish?
Catching *ANY* exception can cause inconsistencies in persistent data.
The "any" refers to "any type of exception" (i.e. not only "ConflictError"). It does (of course) not mean that any exception catching does (with necessity) result in inconsistent persistent data. That's why there is a "can" in the sentence above...
def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" self.foo = REQUEST.get('foo','') try: self.foo2 = int(REQUEST.get('foo2',0) except ValueError, e: self.foo2 = 0
It does (almost surely) when the "try" block modified persistent state before the exception occurred (and you did not ensure that the transaction is aborted).
Please reread paragraph above (which you quoted from my previous message). *carefully*. In your example, the exception occurred *before* you modified persistent state.... Let's extend your example a bit: try: foo2 = REQUEST.get('foo2',0) self.foo2_str= foo2 self.foo2 = int(foo2) except ValueError: ... And unless you do special things, "foo2_str" and "foo2" are no longer consistent...
... I am sure that you know more about both Python and Zope than I do, but I really don't want to believe that the design of Zope prohibits the use of such a fundamental part of the python language as raising and catching exceptions.
Catching exceptions is dangerous all over Python! That's why Python provides the "try: ... finally:...". It allows you to clean up inconsistencies caused by exceptions -- if you care about them. In a normal Python process, data's life is limited -- all objects die when the process stops. Zope makes the danger far greater -- because it introduces persistent state -- objects that live (in principle) forever. Inconsitencies become by that far more serious. This is in fact not special to Zope but to any system with persistent data. E.g. if you catch exceptions carelessly (and later commit) in an application having persistent data in a relational database, you run the same risk... -- Dieter
Catching *ANY* exception can cause inconsistencies in persistent data.
The "any" refers to "any type of exception" (i.e. not only "ConflictError"). It does (of course) not mean that any exception catching does (with necessity) result in inconsistent persistent data. That's why there is a "can" in the sentence above...
def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" self.foo = REQUEST.get('foo','') try: self.foo2 = int(REQUEST.get('foo2',0) except ValueError, e: self.foo2 = 0
Then could I say that this would be an example in which catching SOME exception CAN NOT result in inconsistent data?
On Oct 21, 2004, at 11:18 AM, Sean Hastings wrote:
def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" self.foo = REQUEST.get('foo','') try: self.foo2 = int(REQUEST.get('foo2',0) except ValueError, e: self.foo2 = 0
Then could I say that this would be an example in which catching SOME exception CAN NOT result in inconsistent data?
Depends on the connection between "foo" and "foo2". Maybe 0 is a good default for foo2 if it is not present, but the client wouldn't want it reset to 0 if it was supplied and incorrect. For example imagine the input coming from an HTML form snippet like this. The option for "Shoes" has a small typo. Item: <select name="foo2"> <option value="0">Skirt</option> <option value="1">Jeans</option> <option value="2">Overcoat</option> <option value="e3">Shoes</option> </select> Color: <input type="text" name="foo" size="40" /> You wouldn't want the shoes converted to a skirt just because of a typo in the form. def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" try: newfoo = REQUEST.get('foo','') newfoo2 = int(REQUEST.get('foo2',0) except ValueError, e: raise InvalidInput(e) else: self.foo = newfoo self.foo2 = newfoo2 and have something at a higher level catch InvalidInput and report it to the user. Or maybe in this case, you might be able to report the error within .fooEdit because the changes to the ZODB are moved after the rest of the processing. But that is someone depending on implicit knowledge that .get() and int() won't change the persistent state.
An error in the business logic is not a data inconsistency - assume for this example that 0 is a fine and correct value for a non integer entry. I am just worried about stuff that Zope does on the back end, for example raising ConflictErrors, etc. - Dieter seemed to indicate that any use of try and except could cause such problems. I thought that as long as I don't have any naked except statements that will accidentally catch ConflictErrors, that I only have to worry about what I see in my code, but when I said that, Dieter seemed to indicate that I was wrong - however, his latest response would seem to indicate that I was right. So now I am confused.
-----Original Message----- From: Andrew Langmead [mailto:alangmead@boston.com] Sent: Thursday, October 21, 2004 12:07 PM To: Sean Hastings Cc: zope list Subject: Re: [Zope] adding properties trough pythonscript
On Oct 21, 2004, at 11:18 AM, Sean Hastings wrote:
def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" self.foo = REQUEST.get('foo','') try: self.foo2 = int(REQUEST.get('foo2',0) except ValueError, e: self.foo2 = 0
Then could I say that this would be an example in which catching SOME exception CAN NOT result in inconsistent data?
Depends on the connection between "foo" and "foo2". Maybe 0 is a good default for foo2 if it is not present, but the client wouldn't want it reset to 0 if it was supplied and incorrect.
For example imagine the input coming from an HTML form snippet like this. The option for "Shoes" has a small typo.
Item: <select name="foo2"> <option value="0">Skirt</option> <option value="1">Jeans</option> <option value="2">Overcoat</option> <option value="e3">Shoes</option> </select> Color: <input type="text" name="foo" size="40" />
You wouldn't want the shoes converted to a skirt just because of a typo in the form.
def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" try: newfoo = REQUEST.get('foo','') newfoo2 = int(REQUEST.get('foo2',0) except ValueError, e: raise InvalidInput(e) else: self.foo = newfoo self.foo2 = newfoo2
and have something at a higher level catch InvalidInput and report it to the user. Or maybe in this case, you might be able to report the error within .fooEdit because the changes to the ZODB are moved after the rest of the processing. But that is someone depending on implicit knowledge that .get() and int() won't change the persistent state.
Sean Hastings wrote at 2004-10-21 11:18 -0400:
...
def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" self.foo = REQUEST.get('foo','') try: self.foo2 = int(REQUEST.get('foo2',0) except ValueError, e: self.foo2 = 0
Then could I say that this would be an example in which catching SOME exception CAN NOT result in inconsistent data?
You can :-) -- Dieter
Cool! I already have a whole bunch of code that uses this sort of error checking, and I wanted to make sure that as long as I was careful to specify the errors I am trapping that I don't have to rewrite it all. Thanks.
-----Original Message----- From: Dieter Maurer [mailto:dieter@handshake.de] Sent: Thursday, October 21, 2004 1:44 PM To: Sean Hastings Cc: massimop@users.berlios.de; zope list Subject: RE: [Zope] adding properties trough pythonscript
Sean Hastings wrote at 2004-10-21 11:18 -0400:
...
def fooEdit(self,REQUEST): "foo is a string - foo2 is an int" self.foo = REQUEST.get('foo','') try: self.foo2 = int(REQUEST.get('foo2',0) except ValueError, e: self.foo2 = 0
Then could I say that this would be an example in which catching SOME exception CAN NOT result in inconsistent data?
You can :-)
-- Dieter
Sean Hastings wrote at 2004-10-21 16:22 -0400:
Cool!
I already have a whole bunch of code that uses this sort of error checking, and I wanted to make sure that as long as I was careful to specify the errors I am trapping that I don't have to rewrite it all.
Hopefully, you observe that it does *NOT* depend on the type of errors you are trapping but whether or not you changed (in the try block) persistent state *before* the exception occured (in this block). -- Dieter
Hopefully, you observe that it does *NOT* depend on the type of errors you are trapping but whether or not you changed (in the try block) persistent state *before* the exception occurred (in this block).
I do not yet understanding that part. My second example was: 1 def fooEdit(self,REQUEST): 2 "foo is a string - foo2 is an int" 3 try: 4 self.foo = REQUEST.get('foo','') 5 self.foo2 = int(REQUEST.get('foo2',0) 6 except ValueError, e: 7 self.foo2 = 0 Here I am modifying the persistent object's "foo" property in line 4, then trapping an exception that the "int" function could raise in line 5. Unless line 4 can also raise a ValueError (and I don't think it can), why am I not safe in assuming one of these three cases? A. A ValueError raised in line 5 will leave "foo" set properly from the REQUEST, while "foo2" gets set to 0. B. Some other Exception raised in line 4 or 5 will cause Zope to roll back the transaction and no properties are changed on this try. C. No Exceptions occur, and leave both properties are set correctly from the REQUEST. --Sean
-----Original Message----- From: Dieter Maurer [mailto:dieter@handshake.de] Sent: Friday, October 22, 2004 1:40 PM To: Sean Hastings Cc: massimop@users.berlios.de; zope list Subject: RE: [Zope] adding properties trough pythonscript
Sean Hastings wrote at 2004-10-21 16:22 -0400:
Cool!
I already have a whole bunch of code that uses this sort of error checking, and I wanted to make sure that as long as I was careful to specify the errors I am trapping that I don't have to rewrite it all.
-- Dieter
Sean Hastings wrote at 2004-10-22 14:53 -0400:
Hopefully, you observe that it does *NOT* depend on the type of errors you are trapping but whether or not you changed (in the try block) persistent state *before* the exception occurred (in this block).
I do not yet understanding that part. My second example was:
1 def fooEdit(self,REQUEST): 2 "foo is a string - foo2 is an int" 3 try: 4 self.foo = REQUEST.get('foo','') 5 self.foo2 = int(REQUEST.get('foo2',0) 6 except ValueError, e: 7 self.foo2 = 0
Here I am modifying the persistent object's "foo" property in line 4, then trapping an exception that the "int" function could raise in line 5. Unless line 4 can also raise a ValueError (and I don't think it can), why am I not safe in assuming one of these three cases?
A. A ValueError raised in line 5 will leave "foo" set properly from the REQUEST, while "foo2" gets set to 0.
B. Some other Exception raised in line 4 or 5 will cause Zope to roll back the transaction and no properties are changed on this try.
C. No Exceptions occur, and leave both properties are set correctly from the REQUEST.
You can assume this. Apparently, you are sure that "A" is acceptable. Carefull consideration is necessary for such cases. If there were some dependancy between "foo" and "foo2", an invariant might get violated. In your case, this seems to be not the case. -- Dieter
On Sun, 2004-10-17 at 22:44 +0200, Dieter Maurer wrote:
massimop@users.berlios.de wrote at 2004-10-16 12:23 +0200:
...
Do not use "try: ... except:...". It is dangerous. It can easily lead to ZODB inconsistencies...
not exactly good news to me....
can you point out some document explaining this?
It follows from common sense (and therefore need no documentation ;-)
Python's exceptions are abortive: as soon an exceptin is raise, the normal flow of control is aborted.
When you have modified persistent state before the exception occured, the transaction must usually be aborted, to get the modifications reverted. If instead, the transaction is committed, the (often) partial change is made persistent.
Usually, the Zope framework does this for you. However, when you catch an exception, the framework may not learn about the problem and commit rather than abort.
As I wrote, this can easily lead to corruption of persistent data. yes, it looks sound...:)
thanks massimo
participants (6)
-
Andrew Langmead -
Dieter Maurer -
massimop＠users.berlios.de -
pw_lists＠slinkp.com -
rv000299 -
Sean Hastings