ZODB.POSException.ConflictError (in custommade product)
Hello, I have created a small tool for CMF to count how many times a specific object has been viewed. It's very simple. It works by a call I have added to my main_template.pt that calls the tool with the URL of recent object and counts up by 1. Now from time to time we see the following error: Site Error An error was encountered while publishing this resource. ZODB.POSException.ConflictError Sorry, a site error occurred. Traceback (innermost last): Module ZPublisher.Publish, line 150, in publish_module Module Products.Localizer, line 58, in new_publish Module ZPublisher.Publish, line 127, in publish Module Products.Localizer, line 58, in new_publish Module ZPublisher.Publish, line 127, in publish Module Products.Localizer, line 58, in new_publish Module ZPublisher.Publish, line 127, in publish Module Products.Localizer, line 58, in new_publish Module ZPublisher.Publish, line 122, in publish Module Zope.App.startup, line 142, in zpublisher_exception_hook Module ZPublisher.Publish, line 102, in publish Module Zope.App.startup, line 200, in commit Module ZODB.Transaction, line 235, in commit Module ZODB.Transaction, line 349, in _commit_objects Module ZODB.Connection, line 300, in commit ConflictError: database conflict error (oid 000000000001c065, class Products.CMFCounter.CMFCounterTool.CMFCounterTool) What is causing this error ?? Have I made some mistakes in the code of my tool ? Any hints - because the error is really annoying in a production site ;-) Greetings, Gitte Wange
Gitte Wange wrote at 2003-10-25 13:39 +0200:
I have created a small tool for CMF to count how many times a specific object has been viewed. ... Now from time to time we see the following error: ZODB.POSException.ConflictError
When you put counters in the ZODB, you should use "BTrees.Length" instances. They are small persistent objects with conflict resolution (no ZODB write conflicts) and "independence" (no ZODB read conflicts). When you count in an integer object attribute, each increment writes the the complete object. This may fast increase your ZODB and lead to "ConflictError"s. -- Dieter
At 11:03 26-10-2003, Dieter Maurer wrote:
Gitte Wange wrote at 2003-10-25 13:39 +0200:
I have created a small tool for CMF to count how many times a specific object has been viewed. ... Now from time to time we see the following error: ZODB.POSException.ConflictError
When you put counters in the ZODB, you should use "BTrees.Length" instances. They are small persistent objects with conflict resolution (no ZODB write conflicts) and "independence" (no ZODB read conflicts).
Arh ... At this moment I just have a dictionary with URL as key and count as value. count is a simple integer. So instead I should save value as an instance of BTrees.Length() ? Or just I make something else than a dictionary ?
When you count in an integer object attribute, each increment writes the the complete object. This may fast increase your ZODB and lead to "ConflictError"s.
Sounds just like what I need to do :-) Greetings, Gitte Wange
Gitte Wange wrote at 2003-10-26 11:25 +0100:
At 11:03 26-10-2003, Dieter Maurer wrote:
Gitte Wange wrote at 2003-10-25 13:39 +0200:
I have created a small tool for CMF to count how many times a specific object has been viewed. ... Now from time to time we see the following error: ZODB.POSException.ConflictError
When you put counters in the ZODB, you should use "BTrees.Length" instances. They are small persistent objects with conflict resolution (no ZODB write conflicts) and "independence" (no ZODB read conflicts).
Arh ... At this moment I just have a dictionary with URL as key and count as value. count is a simple integer. So instead I should save value as an instance of BTrees.Length() ? Or just I make something else than a dictionary ?
If your counter is maintained in the ZODB, the counter should be a "BTrees.Length" instance. Otherwise, you can use whatever data structure seems best for you. -- Dieter
At 18:34 26-10-2003, Dieter Maurer wrote:
Gitte Wange wrote at 2003-10-26 11:25 +0100:
At 11:03 26-10-2003, Dieter Maurer wrote:
Gitte Wange wrote at 2003-10-25 13:39 +0200:
I have created a small tool for CMF to count how many times a specific object has been viewed. ... Now from time to time we see the following error: ZODB.POSException.ConflictError
When you put counters in the ZODB, you should use "BTrees.Length" instances. They are small persistent objects with conflict resolution (no ZODB write conflicts) and "independence" (no ZODB read conflicts).
Arh ... At this moment I just have a dictionary with URL as key and count as value. count is a simple integer. So instead I should save value as an instance of BTrees.Length() ? Or just I make something else than a dictionary ?
If your counter is maintained in the ZODB, the counter should be a "BTrees.Length" instance. Otherwise, you can use whatever data structure seems best for you.
Sorry Dieter if I'm slow on this but I just don't get it ... At this moment I have a dictionary looking like this: views = {'http://mypage.com/frontpage': 10, 'http://mypage.com/article2': 3} This get's saved into ZODB and - as I understand - causes the ConflictErrors ? Instead I should use some sort of BTrees.Length object ... and then my question is: Should the dictionary look like this instead: views = {'http://mypage.com/frontpage': BTrees.Length(10), 'http://mypage.com/article2': BTrees.Length(3)} ?? Or is it the entire dictionary that should be some kind of BTrees.Length() ?? Greetings, Gitte Wange
At 23:14 26-10-2003, you wrote:
At 18:34 26-10-2003, Dieter Maurer wrote:
Gitte Wange wrote at 2003-10-26 11:25 +0100:
At 11:03 26-10-2003, Dieter Maurer wrote:
Gitte Wange wrote at 2003-10-25 13:39 +0200:
I have created a small tool for CMF to count how many times a specific object has been viewed. ... Now from time to time we see the following error: ZODB.POSException.ConflictError
When you put counters in the ZODB, you should use "BTrees.Length" instances. They are small persistent objects with conflict resolution (no ZODB write conflicts) and "independence" (no ZODB read conflicts).
Arh ... At this moment I just have a dictionary with URL as key and count as value. count is a simple integer. So instead I should save value as an instance of BTrees.Length() ? Or just I make something else than a dictionary ?
If your counter is maintained in the ZODB, the counter should be a "BTrees.Length" instance. Otherwise, you can use whatever data structure seems best for you.
Sorry Dieter if I'm slow on this but I just don't get it ... At this moment I have a dictionary looking like this: views = {'http://mypage.com/frontpage': 10, 'http://mypage.com/article2': 3} This get's saved into ZODB and - as I understand - causes the ConflictErrors ?
Instead I should use some sort of BTrees.Length object ... and then my question is: Should the dictionary look like this instead: views = {'http://mypage.com/frontpage': BTrees.Length(10), 'http://mypage.com/article2': BTrees.Length(3)} ?? Or is it the entire dictionary that should be some kind of BTrees.Length() ??
No one with a hint on this ?? Greetings, Gitte Wange
Gitte Wange wrote at 2003-10-26 23:14 +0100:
At 18:34 26-10-2003, Dieter Maurer wrote:
If your counter is maintained in the ZODB, the counter should be a "BTrees.Length" instance. Otherwise, you can use whatever data structure seems best for you.
Sorry Dieter if I'm slow on this but I just don't get it ... At this moment I have a dictionary looking like this: views = {'http://mypage.com/frontpage': 10, 'http://mypage.com/article2': 3} This get's saved into ZODB and - as I understand - causes the ConflictErrors ?
I said: your counter (any one of them) in the ZODB should become a "BTrees.Length.Length" instance. I thought, this were pretty clear.
... Or is it the entire dictionary that should be some kind of BTrees.Length() ??
A dictionary cannot become a "BTrees.Length.Length" instance. This would have been pretty clear, if you had a look at this class ;-) Often, it is less work to look at the code than to ask a question on the mailing list ;-) That said: you may want to replace the dictionary by a "BTrees.OOBTree.OOBTree" instance. This way, the conflict resolution of "OOBTrees" may prevent (most) concurrent writes to the mapping (when you add new entries to the mapping) causing exceptions. It may not be necessary, though, as you probably rarely add new entries (usually, you will count in already existing counters). -- Dieter
Tirsdag 28 oktober 2003 09:13 skrev Dieter Maurer:
Gitte Wange wrote at 2003-10-26 23:14 +0100:
At 18:34 26-10-2003, Dieter Maurer wrote:
If your counter is maintained in the ZODB, the counter should be a "BTrees.Length" instance. Otherwise, you can use whatever data structure seems best for you.
Sorry Dieter if I'm slow on this but I just don't get it ... At this moment I have a dictionary looking like this: views = {'http://mypage.com/frontpage': 10, 'http://mypage.com/article2': 3} This get's saved into ZODB and - as I understand - causes the ConflictErrors ?
I said: your counter (any one of them) in the ZODB should become a "BTrees.Length.Length" instance. I thought, this were pretty clear.
Now it's pretty clear :-) I updated the counts to be a BTrees.Length instance instead of just an integer. Looks like it helped but we still see the error sometimes (not as often as before). The code for updating the counts look like this: def addCount(self, url): """ Add a new count for an url """ counts = self._counts if not url in counts.keys(): counts[url] = Length(0) counter = counts[url] counter.change(1) counts[url] = counter self._counts = counts self._p_changed = 1 Guess the error is raised when assigning the counts to self._counts. I have experienced before that dictinaries don't survive a restart of Zope if not saving them as above but maybe this can work: def addCount(self, url): """ Add a new count for an url """ if not url in self._counts.keys(): counts = self._counts counts[url] = Length(0) self._counts = counts self._p_changed = 1 counter = self._counts[url] counter.change(1) Would that work ?
That said: you may want to replace the dictionary by a "BTrees.OOBTree.OOBTree" instance. This way, the conflict resolution of "OOBTrees" may prevent (most) concurrent writes to the mapping (when you add new entries to the mapping) causing exceptions. It may not be necessary, though, as you probably rarely add new entries (usually, you will count in already existing counters).
It's very rarely that I will add new entries - all existing pages already exist in the tool. -- Gitte Wange Technical Manager Email: gitte@mmmanager.org Web: http://www.mmmanager.org Tlf: +45 36 46 20 02
We'll tell you when we try out the code you've written ;-) Ahha, the classic open-source sanity pill/ thorn in my side.... What You Want Is What You Should Code Yourself ( WYWIWYSCY )
-- Tom Smith: Zope Mailing List, About using ZODB as ODBC data source.
Gitte Wange wrote at 2003-11-3 10:12 +0100:
....
I said: your counter (any one of them) in the ZODB should become a "BTrees.Length.Length" instance. I thought, this were pretty clear.
Now it's pretty clear :-) I updated the counts to be a BTrees.Length instance instead of just an integer. Looks like it helped but we still see the error sometimes (not as often as before). The code for updating the counts look like this:
def addCount(self, url): """ Add a new count for an url """ counts = self._counts if not url in counts.keys(): counts[url] = Length(0) counter = counts[url] counter.change(1) counts[url] = counter self._counts = counts self._p_changed = 1
Guess the error is raised when assigning the counts to self._counts. I have experienced before that dictinaries don't survive a restart of Zope if not saving them as above but maybe this can work:
Yes, probably. Make your "_counts" an "BTrees.OOBTree.OOBTree" instance (which is itself a persistent object). You can then drop "self._counts= counts" and "self._p_changed = 1".
... def addCount(self, url): """ Add a new count for an url """ if not url in self._counts.keys(): counts = self._counts counts[url] = Length(0) self._counts = counts self._p_changed = 1 counter = self._counts[url] counter.change(1)
Would that work ?
Yes.
That said: you may want to replace the dictionary by a "BTrees.OOBTree.OOBTree" instance. This way, the conflict resolution of "OOBTrees" may prevent (most) concurrent writes to the mapping (when you add new entries to the mapping) causing exceptions. It may not be necessary, though, as you probably rarely add new entries (usually, you will count in already existing counters).
It's very rarely that I will add new entries - all existing pages already exist in the tool.
Then, you *can* stay with a dictionary. But an "OOBTree" would still give you easier code (no "_p_changed = 1"). -- Dieter
participants (2)
-
Dieter Maurer -
Gitte Wange