ConflictError (ID-Counter for all Documents)
I am developing a simple Document Management System with a python product. All documents are in one folder. If a new document gets created I do the following: root.last_id=root.last_id+1 If I stress test this with httperf I get ConflictErrors. I searched and found two solutions: Catch the Exception, and retry it. Use PCounter from ZODB/test/ConflictResolution.py Are there other, better solutions? Which one is better? thomas
On Thu, 24 Jan 2002 11:59:18 +0100, Thomas Guettler <zopestoller@thomas-guettler.de> wrote:
Catch the Exception, and retry it.
Dont catch it yourself. Zope is already doing this, and only shows the client an error message if it still fails after 3 attempts.
Use PCounter from ZODB/test/ConflictResolution.py
No, for two reasons: That class makes a good counter, but it is no use if you want to use the counter to generate unique ids. It works by allowing two transactions to proceed simultaneously with the same count value, then incrementing the count by two instead of one. Secondly, it wouldnt help even if it did generate unique ids. You would still get a conflict from the folder object. The easiest solution is to use only one publisher thread. That means no concurrency, no conflicts, but limited scalability. You best solution is: 1. Use a BTreeFolder (download it from www.zope.org), which allows for concurrent writes. It wont eliminate the need for retries, but it should decrease the probability of a failure after the three retries to an acceptable level. 2. Use a different method of generating ids, to avoid the problem where two concurrent writes always generate the same id. The easiest solution is random ids. Toby Dickenson tdickenson@geminidataloggers.com
Toby Dickenson wrote:
On Thu, 24 Jan 2002 11:59:18 +0100, Thomas Guettler <zopestoller@thomas-guettler.de> wrote:
Catch the Exception, and retry it.
Dont catch it yourself. Zope is already doing this, and only shows the client an error message if it still fails after 3 attempts.
Use PCounter from ZODB/test/ConflictResolution.py
No, for two reasons:
That class makes a good counter, but it is no use if you want to use the counter to generate unique ids. It works by allowing two transactions to proceed simultaneously with the same count value, then incrementing the count by two instead of one.
Secondly, it wouldnt help even if it did generate unique ids. You would still get a conflict from the folder object.
The easiest solution is to use only one publisher thread. That means no concurrency, no conflicts, but limited scalability.
Would this mean that if someone is downloading a 100MB file all other request have to wait? I already asked this in an other thread: How can I get somethink like a syncronized block in Java? syncronize{ last_id=last_id+1 }
You best solution is:
1. Use a BTreeFolder (download it from www.zope.org), which allows for concurrent writes. It wont eliminate the need for retries, but it should decrease the probability of a failure after the three retries to an acceptable level.
I downloaded it. But it doesn't allow me to _setObject() of an int or a string.
2. Use a different method of generating ids, to avoid the problem where two concurrent writes always generate the same id. The easiest solution is random ids.
I now use BTreeFolder to store all documents. The ID of the documents is created with DateTime().millis() This works much better than before, but still gives me some conflicts when I add the new document to the folder: Sometimes I get a strange error: 2002-01-24T17:14:44 ERROR(200) ZODB Couldn't load state for '\x00\x00\x00\x00\x0 0\x00!\x0b' Traceback (innermost last): File F:\ZOPE-2~1\lib\python\ZODB\Connection.py, line 446, in setstate AttributeError: 'None' object has no attribute 'load'
Ahhh. Threading issues. Can you try something? Look a lib/python/ts_regex.py I think that this is the locking you are looking for. If this works, it could be the answer to our unique_id problems. I see a Zope3 utility coming: unique id number generator. Troy -- Troy Farrell Developer Entheos Software mailto:troy@entheossoft.com http://www.entheossoft.com Thomas Guettler wrote:
Toby Dickenson wrote:
On Thu, 24 Jan 2002 11:59:18 +0100, Thomas Guettler <zopestoller@thomas-guettler.de> wrote:
Catch the Exception, and retry it.
Dont catch it yourself. Zope is already doing this, and only shows the client an error message if it still fails after 3 attempts.
Use PCounter from ZODB/test/ConflictResolution.py
No, for two reasons:
That class makes a good counter, but it is no use if you want to use the counter to generate unique ids. It works by allowing two transactions to proceed simultaneously with the same count value, then incrementing the count by two instead of one.
Secondly, it wouldnt help even if it did generate unique ids. You would still get a conflict from the folder object.
The easiest solution is to use only one publisher thread. That means no concurrency, no conflicts, but limited scalability.
Would this mean that if someone is downloading a 100MB file all other request have to wait?
I already asked this in an other thread: How can I get somethink like a syncronized block in Java?
syncronize{ last_id=last_id+1 }
You best solution is:
1. Use a BTreeFolder (download it from www.zope.org), which allows for concurrent writes. It wont eliminate the need for retries, but it should decrease the probability of a failure after the three retries to an acceptable level.
I downloaded it. But it doesn't allow me to _setObject() of an int or a string.
2. Use a different method of generating ids, to avoid the problem where two concurrent writes always generate the same id. The easiest solution is random ids.
I now use BTreeFolder to store all documents. The ID of the documents is created with DateTime().millis() This works much better than before, but still gives me some conflicts when I add the new document to the folder:
Sometimes I get a strange error: 2002-01-24T17:14:44 ERROR(200) ZODB Couldn't load state for '\x00\x00\x00\x00\x0 0\x00!\x0b' Traceback (innermost last): File F:\ZOPE-2~1\lib\python\ZODB\Connection.py, line 446, in setstate AttributeError: 'None' object has no attribute 'load'
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
I haven't followed this thread, so feel free to shoot me down - but what's wrong with making an autoincrement product which spits out [number++] each time it is called? seb On Fri, 2002-01-25 at 03:38, Troy Farrell wrote:
Ahhh. Threading issues. Can you try something? Look a lib/python/ts_regex.py I think that this is the locking you are looking for. If this works, it could be the answer to our unique_id problems. I see a Zope3 utility coming: unique id number generator.
Troy
-- Troy Farrell Developer Entheos Software mailto:troy@entheossoft.com http://www.entheossoft.com
Thomas Guettler wrote:
Toby Dickenson wrote:
On Thu, 24 Jan 2002 11:59:18 +0100, Thomas Guettler <zopestoller@thomas-guettler.de> wrote:
Catch the Exception, and retry it.
Dont catch it yourself. Zope is already doing this, and only shows the client an error message if it still fails after 3 attempts.
Use PCounter from ZODB/test/ConflictResolution.py
No, for two reasons:
That class makes a good counter, but it is no use if you want to use the counter to generate unique ids. It works by allowing two transactions to proceed simultaneously with the same count value, then incrementing the count by two instead of one.
Secondly, it wouldnt help even if it did generate unique ids. You would still get a conflict from the folder object.
The easiest solution is to use only one publisher thread. That means no concurrency, no conflicts, but limited scalability.
Would this mean that if someone is downloading a 100MB file all other request have to wait?
I already asked this in an other thread: How can I get somethink like a syncronized block in Java?
syncronize{ last_id=last_id+1 }
You best solution is:
1. Use a BTreeFolder (download it from www.zope.org), which allows for concurrent writes. It wont eliminate the need for retries, but it should decrease the probability of a failure after the three retries to an acceptable level.
I downloaded it. But it doesn't allow me to _setObject() of an int or a string.
2. Use a different method of generating ids, to avoid the problem where two concurrent writes always generate the same id. The easiest solution is random ids.
I now use BTreeFolder to store all documents. The ID of the documents is created with DateTime().millis() This works much better than before, but still gives me some conflicts when I add the new document to the folder:
Sometimes I get a strange error: 2002-01-24T17:14:44 ERROR(200) ZODB Couldn't load state for '\x00\x00\x00\x00\x0 0\x00!\x0b' Traceback (innermost last): File F:\ZOPE-2~1\lib\python\ZODB\Connection.py, line 446, in setstate AttributeError: 'None' object has no attribute 'load'
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
On Fri, Jan 25, 2002 at 10:05:21AM +0000, seb bacon wrote:
I haven't followed this thread, so feel free to shoot me down - but what's wrong with making an autoincrement product which spits out [number++] each time it is called?
The Problem is: last_id=last_id+1 1. Thread1 reads last_id=4 2. Thread2 reads last_id=4 3. Thread1 writes last_id=4+1 4. Thread2 writes last_id=4+1 --> two IDs with the same number -- Thomas Guettler <guettli@thomas-guettler.de> http://www.thomas-guettler.de
Thomas Guettler writes:
On Fri, Jan 25, 2002 at 10:05:21AM +0000, seb bacon wrote: The Problem is: last_id=last_id+1 From Python 2.1 on, you can use:
last_id+= 1 Not sure, whether this is thread safe. It might be implemented in a single bytecode instruction. Then it would be... Dieter
On Thu, 24 Jan 2002 18:16:19 +0100, Thomas Guettler <zopestoller@thomas-guettler.de> wrote:
The easiest solution is to use only one publisher thread. That means no concurrency, no conflicts, but limited scalability.
Would this mean that if someone is downloading a 100MB file all other request have to wait?
no, medusa takes care of sending responses back to a slow client. However, it would block if one of your methods made a slow call to a backend database, for example.
I already asked this in an other thread: How can I get somethink like a syncronized block in Java?
syncronize{ last_id=last_id+1 }
Zope's concurrency model is very different; I suggest you look at the documentation on www.zope.org. As for *why* its different.... how would you implement that syncronize block across multiple servers? Toby Dickenson tdickenson@geminidataloggers.com
participants (6)
-
Dieter Maurer -
seb bacon -
Thomas Guettler -
Thomas Guettler -
Toby Dickenson -
Troy Farrell