Trying to trap ConflictError
I am trying to trap the following error: Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module transaction._transaction, line 380, in commit Module transaction._transaction, line 378, in commit Module transaction._transaction, line 433, in _commitResources Module ZODB.Connection, line 484, in commit Module ZODB.Connection, line 526, in _commit Module ZODB.Connection, line 554, in _store_objects Module tempstorage.TemporaryStorage, line 200, in store ConflictError: database conflict error (oid 0x39b0, class BTrees._OOBTree.OOBucket, serial this txn started with 0x0366974da4fd2288 2006-07-04 13:33:38.669252, serial currently committed 0x0366974da54d9fcc 2006-07-04 13:33:38.742942) in an external method, as follows: folder = self.unrestrictedTraverse(TEMPFOLDERIMAGES, None) try: folder.manage_addImage(imageId, imageData) except: <handle error message and return gracefully code> But the try/except block is not catching the error! (the error continues to show up in error_log) (note: I started by trying "except ConflictError:", but that was not working, so I went to the bare try/except) The traceback does not show where the error originates in the external method, but there is only one place where the external methods writes to TemporaryStorage. Does this indicate that the error is not bubbling up properly? Any ideas as to why the error trapping is not trapping? (running Zope 2.9.2, debug=on) Thanks, Jonathan
First, you should not - ever - catch a ConflictError. It is essential for ConflictErrors to be propagated to the ZPublisher. Second, the conflict is detected at commit time, which happens at the very end of the REQUEST. This likely is a while after your method has executed. Stefan On 4. Jul 2006, at 15:56, Jonathan wrote:
I am trying to trap the following error:
Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module transaction._transaction, line 380, in commit Module transaction._transaction, line 378, in commit Module transaction._transaction, line 433, in _commitResources Module ZODB.Connection, line 484, in commit Module ZODB.Connection, line 526, in _commit Module ZODB.Connection, line 554, in _store_objects Module tempstorage.TemporaryStorage, line 200, in store ConflictError: database conflict error (oid 0x39b0, class BTrees._OOBTree.OOBucket, serial this txn started with 0x0366974da4fd2288 2006-07-04 13:33:38.669252, serial currently committed 0x0366974da54d9fcc 2006-07-04 13:33:38.742942)
in an external method, as follows:
folder = self.unrestrictedTraverse(TEMPFOLDERIMAGES, None) try: folder.manage_addImage(imageId, imageData) except: <handle error message and return gracefully code>
But the try/except block is not catching the error! (the error continues to show up in error_log) (note: I started by trying "except ConflictError:", but that was not working, so I went to the bare try/except)
The traceback does not show where the error originates in the external method, but there is only one place where the external methods writes to TemporaryStorage. Does this indicate that the error is not bubbling up properly?
Any ideas as to why the error trapping is not trapping? (running Zope 2.9.2, debug=on)
-- Anything that, in happening, causes something else to happen, causes something else to happen. --Douglas Adams
----- Original Message ----- From: "Stefan H. Holek" <stefan@epy.co.at> To: "Jonathan" <dev101@magma.ca> Cc: <zope@zope.org> Sent: Tuesday, July 04, 2006 11:02 AM Subject: Re: [Zope] Trying to trap ConflictError
First, you should not - ever - catch a ConflictError. It is essential for ConflictErrors to be propagated to the ZPublisher.
This seems to be an extremely 'user-unfriendly' approach. In my use case the error is being generated when an image object is being written to a temporary folder. If, for any reason, the image object is not able to be stored it would seem prudent to be able to handle this condition within the application software.
Second, the conflict is detected at commit time, which happens at the very end of the REQUEST. This likely is a while after your method has executed.
Stefan
Ok, this I can understand. Is there any kind of 'post processing' mechanism/hooks that would enable application level software (external methods) to check for and handle errors at this point? Jonathan
On 4. Jul 2006, at 15:56, Jonathan wrote:
I am trying to trap the following error:
Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit Module transaction._manager, line 96, in commit Module transaction._transaction, line 380, in commit Module transaction._transaction, line 378, in commit Module transaction._transaction, line 433, in _commitResources Module ZODB.Connection, line 484, in commit Module ZODB.Connection, line 526, in _commit Module ZODB.Connection, line 554, in _store_objects Module tempstorage.TemporaryStorage, line 200, in store ConflictError: database conflict error (oid 0x39b0, class BTrees._OOBTree.OOBucket, serial this txn started with 0x0366974da4fd2288 2006-07-04 13:33:38.669252, serial currently committed 0x0366974da54d9fcc 2006-07-04 13:33:38.742942)
in an external method, as follows:
folder = self.unrestrictedTraverse(TEMPFOLDERIMAGES, None) try: folder.manage_addImage(imageId, imageData) except: <handle error message and return gracefully code>
But the try/except block is not catching the error! (the error continues to show up in error_log) (note: I started by trying "except ConflictError:", but that was not working, so I went to the bare try/except)
The traceback does not show where the error originates in the external method, but there is only one place where the external methods writes to TemporaryStorage. Does this indicate that the error is not bubbling up properly?
Any ideas as to why the error trapping is not trapping? (running Zope 2.9.2, debug=on)
-- Anything that, in happening, causes something else to happen, causes something else to happen. --Douglas Adams
--On 4. Juli 2006 12:58:24 -0400 Jonathan <dev101@magma.ca> wrote:
----- Original Message ----- From: "Stefan H. Holek" <stefan@epy.co.at> To: "Jonathan" <dev101@magma.ca> Cc: <zope@zope.org> Sent: Tuesday, July 04, 2006 11:02 AM Subject: Re: [Zope] Trying to trap ConflictError
First, you should not - ever - catch a ConflictError. It is essential for ConflictErrors to be propagated to the ZPublisher.
This seems to be an extremely 'user-unfriendly' approach.
In my use case the error is being generated when an image object is being written to a temporary folder. If, for any reason, the image object is not able to be stored it would seem prudent to be able to handle this condition within the application software.
If you catch ConflictError then you have to perform conflict resolution on your own. But as Stefan wrote: you should *never* swallow ConflictErrors. I thionk there are one or two ZODB article available from zope.org explaining how to implement conflict resolution properly. -aj
Johnathan, I have been lurking, following your thread, but not paying full attention. Excuse me if thes points have been made before. There are many long threads on Conflict Errors. You should read them. Note that there have been significant changes in management strategy beginning with Zope 2.9.X. A "Conflict Error" is not really an error in the usual sense; it is an indication of conflicting transactions in the ZODB, that is, transactions have violated the optimistic locking used by the ZODB. In general, Conflict Errors are resolved without problem. The resolution usually involves rolling back the older transaction and committing the newer transaction. When the actions associated with the rolled back transaction are not themselves fully transactional (for example, a file write) there can be unanticipated side effects. Your conflicts appear to be in Temporary Storage which is used for Session Variables. The version that ships with Zope is flexible and complex and, IMHO, overkill. Moreover, it has some unexpected properties, the most significant is that, even though session variables are stored in RAM, it occasionally uses the ZODB to manage session variable expiration. You can often decrease the number of session variable induced conflicts by increasing the lifetime of session variables. --
----- Original Message ----- From: "Dennis Allison" <allison@shasta.stanford.edu> To: "Jonathan" <dev101@magma.ca> Cc: <zope@zope.org> Sent: Tuesday, July 04, 2006 1:27 PM Subject: Re: [Zope] Trying to trap ConflictError
Johnathan,
I have been lurking, following your thread, but not paying full attention. Excuse me if thes points have been made before.
There are many long threads on Conflict Errors. You should read them. Note that there have been significant changes in management strategy beginning with Zope 2.9.X.
A "Conflict Error" is not really an error in the usual sense; it is an indication of conflicting transactions in the ZODB, that is, transactions have violated the optimistic locking used by the ZODB. In general, Conflict Errors are resolved without problem. The resolution usually involves rolling back the older transaction and committing the newer transaction. When the actions associated with the rolled back transaction are not themselves fully transactional (for example, a file write) there can be unanticipated side effects.
Your conflicts appear to be in Temporary Storage which is used for Session Variables. The version that ships with Zope is flexible and complex and, IMHO, overkill. Moreover, it has some unexpected properties, the most significant is that, even though session variables are stored in RAM, it occasionally uses the ZODB to manage session variable expiration. You can often decrease the number of session variable induced conflicts by increasing the lifetime of session variables.
As both you and Andreas have recommended, I have started researching 'ZODB Conflict Resolution' and if I come up with generic/reusable external method application-level solution I will post it. By way of providing more information: I am not using sessions. I am using TemporaryStorage to hold small, custom-made, short-lived image objects (the use case requires that each user views a static image with a custom overlay, and that the user accesses this customized image via an html img tag). The custom images are viewed only once, and then can be deleted. Hence the use of TemporaryStorage (to eliminate many disk writes and ZODB bloating). The image objects stored in TemporaryStorage are never overwritten, edited etc (which is causing me some confusion as to why the conflict errors are occurring), they are simply created, stored in the TemporaryStorage folder (actually a BTreeFolder2 folder contained within the TemporaryStorage folder) and then 2 minutes later are deleted. I am guessing that the conflicts are occuring as the TemporaryStorage is trying to delete old images while new image object 'add requests' are piling up, but I have yet to determine the cause of the conflict with 100% certainty (conflicts only occur under heavy loading, and not always in the same place in the load testing). Thanks again for everyone's help! Jonathan
You did not mention what LOG LEVEL you were using. I would suggest setting it to BLATHER (you are using Zope 2.9.2 I believe -- the log systems is in transition but I think BLATHER is appropriate for 2.9.2). This will allow you to look at the transactions with a bit finer grainularlity. On Tue, 4 Jul 2006, Jonathan wrote:
----- Original Message ----- From: "Dennis Allison" <allison@shasta.stanford.edu> To: "Jonathan" <dev101@magma.ca> Cc: <zope@zope.org> Sent: Tuesday, July 04, 2006 1:27 PM Subject: Re: [Zope] Trying to trap ConflictError
Johnathan,
I have been lurking, following your thread, but not paying full attention. Excuse me if thes points have been made before.
There are many long threads on Conflict Errors. You should read them. Note that there have been significant changes in management strategy beginning with Zope 2.9.X.
A "Conflict Error" is not really an error in the usual sense; it is an indication of conflicting transactions in the ZODB, that is, transactions have violated the optimistic locking used by the ZODB. In general, Conflict Errors are resolved without problem. The resolution usually involves rolling back the older transaction and committing the newer transaction. When the actions associated with the rolled back transaction are not themselves fully transactional (for example, a file write) there can be unanticipated side effects.
Your conflicts appear to be in Temporary Storage which is used for Session Variables. The version that ships with Zope is flexible and complex and, IMHO, overkill. Moreover, it has some unexpected properties, the most significant is that, even though session variables are stored in RAM, it occasionally uses the ZODB to manage session variable expiration. You can often decrease the number of session variable induced conflicts by increasing the lifetime of session variables.
As both you and Andreas have recommended, I have started researching 'ZODB Conflict Resolution' and if I come up with generic/reusable external method application-level solution I will post it.
By way of providing more information:
I am not using sessions. I am using TemporaryStorage to hold small, custom-made, short-lived image objects (the use case requires that each user views a static image with a custom overlay, and that the user accesses this customized image via an html img tag). The custom images are viewed only once, and then can be deleted. Hence the use of TemporaryStorage (to eliminate many disk writes and ZODB bloating). The image objects stored in TemporaryStorage are never overwritten, edited etc (which is causing me some confusion as to why the conflict errors are occurring), they are simply created, stored in the TemporaryStorage folder (actually a BTreeFolder2 folder contained within the TemporaryStorage folder) and then 2 minutes later are deleted. I am guessing that the conflicts are occuring as the TemporaryStorage is trying to delete old images while new image object 'add requests' are piling up, but I have yet to determine the cause of the conflict with 100% certainty (conflicts only occur under heavy loading, and not always in the same place in the load testing).
Thanks again for everyone's help!
Jonathan
--
You might also want to use the conflict resulution hook. The reason you don't see the Conflict Error exception is that it is caught and triggers a conflict resolution process. On Tue, 4 Jul 2006, Jonathan wrote:
----- Original Message ----- From: "Dennis Allison" <allison@shasta.stanford.edu> To: "Jonathan" <dev101@magma.ca> Cc: <zope@zope.org> Sent: Tuesday, July 04, 2006 1:27 PM Subject: Re: [Zope] Trying to trap ConflictError
Johnathan,
I have been lurking, following your thread, but not paying full attention. Excuse me if thes points have been made before.
There are many long threads on Conflict Errors. You should read them. Note that there have been significant changes in management strategy beginning with Zope 2.9.X.
A "Conflict Error" is not really an error in the usual sense; it is an indication of conflicting transactions in the ZODB, that is, transactions have violated the optimistic locking used by the ZODB. In general, Conflict Errors are resolved without problem. The resolution usually involves rolling back the older transaction and committing the newer transaction. When the actions associated with the rolled back transaction are not themselves fully transactional (for example, a file write) there can be unanticipated side effects.
Your conflicts appear to be in Temporary Storage which is used for Session Variables. The version that ships with Zope is flexible and complex and, IMHO, overkill. Moreover, it has some unexpected properties, the most significant is that, even though session variables are stored in RAM, it occasionally uses the ZODB to manage session variable expiration. You can often decrease the number of session variable induced conflicts by increasing the lifetime of session variables.
As both you and Andreas have recommended, I have started researching 'ZODB Conflict Resolution' and if I come up with generic/reusable external method application-level solution I will post it.
By way of providing more information:
I am not using sessions. I am using TemporaryStorage to hold small, custom-made, short-lived image objects (the use case requires that each user views a static image with a custom overlay, and that the user accesses this customized image via an html img tag). The custom images are viewed only once, and then can be deleted. Hence the use of TemporaryStorage (to eliminate many disk writes and ZODB bloating). The image objects stored in TemporaryStorage are never overwritten, edited etc (which is causing me some confusion as to why the conflict errors are occurring), they are simply created, stored in the TemporaryStorage folder (actually a BTreeFolder2 folder contained within the TemporaryStorage folder) and then 2 minutes later are deleted. I am guessing that the conflicts are occuring as the TemporaryStorage is trying to delete old images while new image object 'add requests' are piling up, but I have yet to determine the cause of the conflict with 100% certainty (conflicts only occur under heavy loading, and not always in the same place in the load testing).
Thanks again for everyone's help!
Jonathan
--
Jonathan wrote at 2006-7-4 13:46 -0400:
... The image objects stored in TemporaryStorage are never overwritten, edited etc (which is causing me some confusion as to why the conflict errors are occurring)
The error message tells your that the conflict is not caused by the modification of the image but the container holding your image. "BTreeFolder" already do conflict resolution -- and reduce the conflict probability by about a factor of 30 to 100 (depending on type). However, if several threads should try to add elements with the same id, then the conflict resolution cannot work. Maybe, that happens in your case? -- Dieter
----- Original Message ----- From: <dieter@handshake.de> To: "Jonathan" <dev101@magma.ca> Cc: "Dennis Allison" <allison@shasta.stanford.edu>; <zope@zope.org> Sent: Wednesday, July 05, 2006 2:28 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote at 2006-7-4 13:46 -0400:
... The image objects stored in TemporaryStorage are never overwritten, edited etc (which is causing me some confusion as to why the conflict errors are occurring)
The error message tells your that the conflict is not caused by the modification of the image but the container holding your image.
"BTreeFolder" already do conflict resolution -- and reduce the conflict probability by about a factor of 30 to 100 (depending on type).
However, if several threads should try to add elements with the same id, then the conflict resolution cannot work. Maybe, that happens in your case?
The id for the image object is generated by the following code: def MakeId(): time.sleep(.01) ts = string.replace(str(time.time()),'.', '') # remove floating point '.' char return ts[-10:] So there should not be any collision of ids. I am still investing, and have found errors like: 2006-07-04T14:48:12 ERROR ZODB.Connection Couldn't load state for 0x1c Traceback (most recent call last): File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 732, in setstate self._setstate(obj) File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 768, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File "/usr/local/Zope-2.9.2/lib/python/tempstorage/TemporaryStorage.py", line 104, in load s=self._index[oid] KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c' in the event.log. From my googling research, it appears that these type of errors may be due to insufficient memory (but I am not sure and am still investigating to see if this is the cause of the conflict errors). Thanks to all for the suggestions and assistance! Jonathan
Jonathan wrote:
----- Original Message ----- From: <dieter@handshake.de> To: "Jonathan" <dev101@magma.ca> Cc: "Dennis Allison" <allison@shasta.stanford.edu>; <zope@zope.org> Sent: Wednesday, July 05, 2006 2:28 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote at 2006-7-4 13:46 -0400:
... The image objects stored in TemporaryStorage are never overwritten, edited etc (which is causing me some confusion as to why the conflict errors are occurring)
The error message tells your that the conflict is not caused by the modification of the image but the container holding your image.
"BTreeFolder" already do conflict resolution -- and reduce the conflict probability by about a factor of 30 to 100 (depending on type).
However, if several threads should try to add elements with the same id, then the conflict resolution cannot work. Maybe, that happens in your case?
The id for the image object is generated by the following code:
def MakeId(): time.sleep(.01) ts = string.replace(str(time.time()),'.', '') # remove floating point '.' char return ts[-10:]
So there should not be any collision of ids.
I am still investing, and have found errors like:
Jonathan, I wouldn't feel safe with that. Why not append a few random digits just to be safe? David
----- Original Message ----- From: "David H" <bluepaul@earthlink.net> To: "Jonathan" <dev101@magma.ca> Cc: <dieter@handshake.de>; <zope@zope.org> Sent: Wednesday, July 05, 2006 3:00 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote:
----- Original Message ----- From: <dieter@handshake.de> To: "Jonathan" <dev101@magma.ca> Cc: "Dennis Allison" <allison@shasta.stanford.edu>; <zope@zope.org> Sent: Wednesday, July 05, 2006 2:28 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote at 2006-7-4 13:46 -0400:
... The image objects stored in TemporaryStorage are never overwritten, edited etc (which is causing me some confusion as to why the conflict errors are occurring)
The error message tells your that the conflict is not caused by the modification of the image but the container holding your image.
"BTreeFolder" already do conflict resolution -- and reduce the conflict probability by about a factor of 30 to 100 (depending on type).
However, if several threads should try to add elements with the same id, then the conflict resolution cannot work. Maybe, that happens in your case?
The id for the image object is generated by the following code:
def MakeId(): time.sleep(.01) ts = string.replace(str(time.time()),'.', '') # remove floating point '.' char return ts[-10:]
So there should not be any collision of ids.
I am still investing, and have found errors like:
Jonathan,
I wouldn't feel safe with that. Why not append a few random digits just to be safe?
I am relatively new to python, so if there is something problematic with my use of the time module to create unique ids, please let me know! (just adding random digits seems a bit kludgy?!) Thanks, Jonathan
Jonathan wrote:
----- Original Message ----- From: "David H" <bluepaul@earthlink.net> To: "Jonathan" <dev101@magma.ca> Cc: <dieter@handshake.de>; <zope@zope.org> Sent: Wednesday, July 05, 2006 3:00 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote:
----- Original Message ----- From: <dieter@handshake.de> To: "Jonathan" <dev101@magma.ca> Cc: "Dennis Allison" <allison@shasta.stanford.edu>; <zope@zope.org> Sent: Wednesday, July 05, 2006 2:28 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote at 2006-7-4 13:46 -0400:
... The image objects stored in TemporaryStorage are never overwritten, edited etc (which is causing me some confusion as to why the conflict errors are occurring)
The error message tells your that the conflict is not caused by the modification of the image but the container holding your image.
"BTreeFolder" already do conflict resolution -- and reduce the conflict probability by about a factor of 30 to 100 (depending on type).
However, if several threads should try to add elements with the same id, then the conflict resolution cannot work. Maybe, that happens in your case?
The id for the image object is generated by the following code:
def MakeId(): time.sleep(.01) ts = string.replace(str(time.time()),'.', '') # remove floating point '.' char return ts[-10:]
So there should not be any collision of ids.
I am still investing, and have found errors like:
Jonathan,
I wouldn't feel safe with that. Why not append a few random digits just to be safe?
I am relatively new to python, so if there is something problematic with my use of the time module to create unique ids, please let me know! (just adding random digits seems a bit kludgy?!)
Thanks,
Jonathan
Hi Jonathan, I figured that if conflicts occur because mutliple processes are requesting some service at the same time +- delta then simply generating a file name based on the time was suspect. David David
----- Original Message ----- From: "David H" <bluepaul@earthlink.net> To: "Jonathan" <dev101@magma.ca> Cc: <zope@zope.org> Sent: Wednesday, July 05, 2006 3:22 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote:
----- Original Message ----- From: "David H" <bluepaul@earthlink.net> To: "Jonathan" <dev101@magma.ca> Cc: <dieter@handshake.de>; <zope@zope.org> Sent: Wednesday, July 05, 2006 3:00 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote:
----- Original Message ----- From: <dieter@handshake.de> To: "Jonathan" <dev101@magma.ca> Cc: "Dennis Allison" <allison@shasta.stanford.edu>; <zope@zope.org> Sent: Wednesday, July 05, 2006 2:28 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote at 2006-7-4 13:46 -0400:
... The image objects stored in TemporaryStorage are never overwritten, edited etc (which is causing me some confusion as to why the conflict errors are occurring)
The error message tells your that the conflict is not caused by the modification of the image but the container holding your image.
"BTreeFolder" already do conflict resolution -- and reduce the conflict probability by about a factor of 30 to 100 (depending on type).
However, if several threads should try to add elements with the same id, then the conflict resolution cannot work. Maybe, that happens in your case?
The id for the image object is generated by the following code:
def MakeId(): time.sleep(.01) ts = string.replace(str(time.time()),'.', '') # remove floating point '.' char return ts[-10:]
So there should not be any collision of ids.
I am still investing, and have found errors like:
Jonathan,
I wouldn't feel safe with that. Why not append a few random digits just to be safe?
I am relatively new to python, so if there is something problematic with my use of the time module to create unique ids, please let me know! (just adding random digits seems a bit kludgy?!)
Thanks,
Jonathan
Hi Jonathan,
I figured that if conflicts occur because mutliple processes are requesting some service at the same time +- delta then simply generating a file name based on the time was suspect.
Good thought! I had not considered problems arising from multiple processes. I have now included a random 3 digit suffix as part of the id generation routine. On the downside, I am still getting conflict errors (back to the investigation... sigh). Thanks David, Jonathan
Jonathan wrote at 2006-7-5 14:40 -0400:
... The id for the image object is generated by the following code:
def MakeId(): time.sleep(.01) ts = string.replace(str(time.time()),'.', '') # remove floating point '.' char return ts[-10:]
So there should not be any collision of ids.
You should verify this. I doubt that you are right, as both the scheduling as well as the clock granularity are discrete (and rather coarse). I suggest you use either the mechanism of * the "BrowserIdManager"::it combines a time with a random number * the "ZCatalog": it chooses per thread a random number and then assignes sequentially, until a conflict happens. Then it restarts (choosing a random number...)
I am still investing, and have found errors like:
2006-07-04T14:48:12 ERROR ZODB.Connection Couldn't load state for 0x1c Traceback (most recent call last): File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 732, in setstate self._setstate(obj) File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 768, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File "/usr/local/Zope-2.9.2/lib/python/tempstorage/TemporaryStorage.py", line 104, in load s=self._index[oid] KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c'
in the event.log.
Apparently, you have more than a single type of error... Are you the person that used "try: ... except: ..."? The non-deterministic inconsistencies caused by such constructs can look like what you decribe above... -- Dieter
----- Original Message ----- From: <dieter@handshake.de> To: "Jonathan" <dev101@magma.ca> Cc: "Dennis Allison" <allison@shasta.stanford.edu>; <zope@zope.org> Sent: Wednesday, July 05, 2006 3:25 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote at 2006-7-5 14:40 -0400:
... The id for the image object is generated by the following code:
def MakeId(): time.sleep(.01) ts = string.replace(str(time.time()),'.', '') # remove floating point '.' char return ts[-10:]
So there should not be any collision of ids.
You should verify this. I doubt that you are right, as both the scheduling as well as the clock granularity are discrete (and rather coarse).
I suggest you use either the mechanism of
* the "BrowserIdManager"::it combines a time with a random number
* the "ZCatalog": it chooses per thread a random number and then assignes sequentially, until a conflict happens. Then it restarts (choosing a random number...)
I am still investing, and have found errors like:
2006-07-04T14:48:12 ERROR ZODB.Connection Couldn't load state for 0x1c Traceback (most recent call last): File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 732, in setstate self._setstate(obj) File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 768, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File "/usr/local/Zope-2.9.2/lib/python/tempstorage/TemporaryStorage.py", line 104, in load s=self._index[oid] KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c'
in the event.log.
Apparently, you have more than a single type of error...
Are you the person that used "try: ... except: ..."?
The non-deterministic inconsistencies caused by such constructs can look like what you decribe above...
I was originally using the try/except to trap the conflict error, but as a result of being educated as to the futility of that approach I have since removed the try/except. I have also included a random 3 digit suffix in the id generation routine to address the issue of muliple processes potentially generating the same id. However, the conflict errors remain, so I am back to investigating... Thanks for the help, Jonathan
On 7/6/06, Jonathan <dev101@magma.ca> wrote:
I am still investing, and have found errors like:
2006-07-04T14:48:12 ERROR ZODB.Connection Couldn't load state for 0x1c Traceback (most recent call last): File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 732, in setstate self._setstate(obj) File "/usr/local/Zope-2.9.2/lib/python/ZODB/Connection.py", line 768, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File "/usr/local/Zope-2.9.2/lib/python/tempstorage/TemporaryStorage.py", line 104, in load s=self._index[oid] KeyError: '\x00\x00\x00\x00\x00\x00\x00\x1c'
in the event.log. From my googling research, it appears that these type of errors may be due to insufficient memory (but I am not sure and am still investigating to see if this is the cause of the conflict errors).
TempoaryStorage.py has a few tuning constants. The above error reads as though RECENTLY_GC_OIDS_LEN is too small for your application. It may well be that the default tuning of TempoaryStorage does not match your application very well. It would be useful to try FileStorage to help decide on where to focus attention. If FileStorage behaves a lot better then that also suggests that TempoaryStorage is miss-tuned for your application. Cheers Michael
----- Original Message ----- From: "Michael Dunstan" <michael.dunstan@gmail.com> To: "Jonathan" <dev101@magma.ca> Cc: <zope@zope.org> Sent: Wednesday, July 05, 2006 5:15 PM Subject: Re: [Zope] Trying to trap ConflictError
TempoaryStorage.py has a few tuning constants. The above error reads as though RECENTLY_GC_OIDS_LEN is too small for your application. It may well be that the default tuning of TempoaryStorage does not match your application very well.
It would be useful to try FileStorage to help decide on where to focus attention. If FileStorage behaves a lot better then that also suggests that TempoaryStorage is miss-tuned for your application.
Thanks for the idea Micheal... as a test I changed from TemporaryStorage to FileStorage to see if the ConflictErrors would 'go away'. However, the ConflictErrors still occur! This leads me to believe that the problem is not related to TemporaryStorage. Further investigation led to the discovery that ConflictErrors are being raised by TemporaryStorage and FileStorage in the same 'store' routines. Here is the relevant code from TemporaryStorage: def store(self, oid, serial, data, version, transaction): if transaction is not self._transaction: raise POSException.StorageTransactionError(self, transaction) self._lock_acquire() try: if self._index.has_key(oid): oserial=self._index[oid] if serial != oserial: newdata = self.tryToResolveConflict( oid, oserial, serial, data) if not newdata: raise POSException.ConflictError( oid=oid, serials=(oserial, serial), data=data) # ***** Conflict Error raised here **** else: data = newdata else: oserial = serial newserial=self._tid self._tmp.append((oid, data)) return serial == oserial and newserial or ResolvedSerial finally: self._lock_release()
From what I understand from the above code, the error is being raised because the oid already exists and the object pointed to by the old oid is different from the object pointed to by the new oid, hence the conflict error.
I haven't been able to determine how the oid's are generated, other than "little-endian 64-bit unsigned integers that will be assigned more or less sequentially", but I can't figure out how the same oid is being used twice (on the other hand I could be totally off-base as I am really way over my head here!). Once again, any and all ideas, comments, suggestions are greatly appreciated! Jonathan
On 7/7/06, Jonathan <dev101@magma.ca> wrote:
I haven't been able to determine how the oid's are generated, other than "little-endian 64-bit unsigned integers that will be assigned more or less sequentially", but I can't figure out how the same oid is being used twice (on the other hand I could be totally off-base as I am really way over my head here!).
An oid is an identifier for a persistent state of an object. As the state of an object evolves from transaction to transaction it gets a new oid for each state. Two concurrent transactions that involve the same object will start with the same state, i.e. the same oid, for that object. Some detail about how BTrees behave can be found in http://www.zope.org/Wikis/ZODB/BTreeConflictResolution Also might be illustrative to run fsdump to see what is happening in your transactions. See http://www.zope.org/Wikis/ZODB/FileStorageBackup. You'll need FileStorage for that. And that only shows you details of successful transactions. cheers michael
On 7/7/06, Michael Dunstan <michael.dunstan@gmail.com> wrote:
An oid is an identifier for a persistent state of an object. As the state of an object evolves from transaction to transaction it gets a new oid for each state. Two concurrent transactions that involve the same object will start with the same state, i.e. the same oid, for that object.
Sorry. Ignore that. :-( Note to self, don't even read email before morning caffeine intake.
Jonathan wrote at 2006-7-6 09:53 -0400:
...
From what I understand from the above code, the error is being raised because the oid already exists and the object pointed to by the old oid is different from the object pointed to by the new oid, hence the conflict error.
You interpretation is slightly wrong. What really differs is not the object (or "oid") but the timestamp. Each persistent object has a timestamp "_p_serial" which identifies the transaction that made the last change to the object. When an object is loaded from the ZODB, its timestamp is recorded (in "_p_serial"). When later a change to the object should be commited, the recorded timestamp is compared to the object's current timestamp in the ZODB. If the timestamps differ, then the object was modified by a concurrent transaction and our modification is likely to be based on stale information. As a result a "ConflictError" is raised. -- Dieter
----- Original Message ----- From: <dieter@handshake.de> To: "Jonathan" <dev101@magma.ca> Cc: "Michael Dunstan" <michael.dunstan@gmail.com>; <zope@zope.org> Sent: Thursday, July 06, 2006 2:56 PM Subject: Re: [Zope] Trying to trap ConflictError
Jonathan wrote at 2006-7-6 09:53 -0400:
...
From what I understand from the above code, the error is being raised because the oid already exists and the object pointed to by the old oid is different from the object pointed to by the new oid, hence the conflict error.
You interpretation is slightly wrong. What really differs is not the object (or "oid") but the timestamp.
Each persistent object has a timestamp "_p_serial" which identifies the transaction that made the last change to the object.
When an object is loaded from the ZODB, its timestamp is recorded (in "_p_serial"). When later a change to the object should be commited, the recorded timestamp is compared to the object's current timestamp in the ZODB. If the timestamps differ, then the object was modified by a concurrent transaction and our modification is likely to be based on stale information. As a result a "ConflictError" is raised.
Thanks Dieter, this got me pointed in the right direction... The problem was due to multiple processes simultaneously accessing a 'MakeId' routine (as some had suggested earlier in this investigation!). Even though the routine returned a 10 digit id consisting of a 7 digit timestamp + 3 random digits, there were still id collisions. Changing the MakeId routine to return a 10 character string (all 10 characters randomly generated) eliminated the ConflictErrors that were occurring during heavy load testing. Thanks to everyone for the great assistance!!! Jonathan
Jonathan wrote at 2006-7-4 09:56 -0400:
... Traceback (innermost last): Module Zope2.App.startup, line 173, in zpublisher_exception_hook Module ZPublisher.Publish, line 121, in publish Module Zope2.App.startup, line 240, in commit ... Module ZODB.Connection, line 554, in _store_objects Module tempstorage.TemporaryStorage, line 200, in store ConflictError: database conflict error (oid 0x39b0, class BTrees._OOBTree.OOBucket, serial this txn started with 0x0366974da4fd2288 2006-07-04 13:33:38.669252, serial currently committed 0x0366974da54d9fcc 2006-07-04 13:33:38.742942)
in an external method, as follows:
folder = self.unrestrictedTraverse(TEMPFOLDERIMAGES, None) try: folder.manage_addImage(imageId, imageData) except: <handle error message and return gracefully code>
Such "try: ... except: ..." are *VERY* dangerous. You can get non deterministic problems with inconsistencies in your persistant storages -- very difficult to analyse and to understand. Never use "try: ... except: ..."! Put your "graceful code" in "standard_error_message".
... But the try/except block is not catching the error! (the error continues to show up in error_log) (note: I started by trying "except ConflictError:", but that was not working, so I went to the bare try/except)
The traceback tells you that the error does not come from the "manage_addImage" but from the "commit" at the end of request processing.
The traceback does not show where the error originates in the external method
because, it does not originate in the external method...
but there is only one place where the external methods writes to TemporaryStorage.
When you change a persistent object, you do not write to the storage but to a copy of your object in RAM only. After the request is completely processed, the changes done during the request are committed (or aborted, in case of an exception). Only during the commit are your modifications stored to the storage. In this phase, conflicts are recognized as well and cause the transaction to be aborted rather than committed. -- Dieter
participants (7)
-
Andreas Jung -
David H -
Dennis Allison -
dieter＠handshake.de -
Jonathan -
Michael Dunstan -
Stefan H. Holek