Here is the example I was thinking about. 1) Suppose an online store is selling product X and at a given moment its inventory is n. Two shoppers hit the buy button at about the same time. Then either: a) No conflict is detected because both threads set the inventory value to the same value, n-1. This would cause a problem if n = 1 Or: b) A conflict is detected when the last thread writes because the inventory is not n as it was when he read it. request is resubmitted but again another thread has changed made the value dirty- thread starves. -----Original Message----- From: Chris McDonough [mailto:chrism@zope.com] Sent: Thursday, November 01, 2001 6:50 AM To: Clark OBrien; k_vertigo@yahoo.com; zope@zope.org Subject: Re: [Zope] zope operations atomic? What is "the dirty read problem"? ----- Original Message ----- From: "Clark OBrien" <COBrien@isis-server.vuse.vanderbilt.edu> To: "'Chris McDonough'" <chrism@zope.com>; "Clark OBrien" <COBrien@isis-server.vuse.vanderbilt.edu>; <k_vertigo@yahoo.com>; <zope@zope.org> Sent: Thursday, November 01, 2001 9:39 AM Subject: RE: [Zope] zope operations atomic?
Chris, Thanks for the explanation. I don't see how ApplicationLevelConflictResolution can handle the dirty read problem though.
-----Original Message----- From: Chris McDonough [mailto:chrism@zope.com] Sent: Wednesday, October 31, 2001 3:11 PM To: Clark OBrien; k_vertigo@yahoo.com; zope@zope.org Subject: Re: [Zope] zope operations atomic?
Example: To count the number of files uploaded to my site I set an index as an attribute to a Folder. These are large files so they take some time to upload. What happens after two people upload at once does my index only get rolled back or do the files uploaded also get rolled back?
If the uploads finish at the same time (meaning the transactions will likely commit at the same time), a ConflictError is raised, and one of the requests is retried.
(Note that the file needn't be uploaded from the remote browser again; Zope keeps the request around -- which includes the cgi data, which includes your file's data -- to retry with.)
The uploader never notices any of this, except perhaps the site will become sluggish.
None of this has anything to do with the counter in your example, except that a ConflictError is more likely to be raised when changing "hotspot" items like a counter.
The example you provide of a counter is one that is problematic for ZODB for two reasons:
1. Multiple simultaneous commits which include counter incrementing can cause conflicts
2. The undoability of FileStorage causes the Data.fs file to grow on every successful transaction.
These problems are orthogonal to each other; they are not related to each other in any way except that they both occur in the default Zope configuration. The first is solved via application level conflict resolution
(http://www.zope.org/Members/jim/ZODB/ApplicationLevelConflictResoluti
on) as well as a storage which supports it, the second is solved by using a nonundoing storage.
So in the example you gave, you're most worried about problem #1, conflicts. To solve this particular problem, you would implement a "smart" counter that did not make itself a hotspot. There is actually already one of these that ships with Zope. It's in the BTrees.Length module, and it's a counter that does app-level conflict resolution. It's also a good module to take a look at if you're interested in seeing how conflict resolution works.
Note that none of this is terribly different AFAICT than how most relational databases implement multithreadedness save for the fact that ZODB uses optimistic concurrency instead of pessimistic, allowing the app developer to be able to handle conflict scenarios in order to maximize performance.
HTH,
- C
even more generally the problem can be stated as transaction A starts reads a bunch of objects, transaction B starts and will read a set of objects that intersects with transaction A. transaction A commits changing objects that B is about to read (after B has read some objects). B finishes, and has possibly commited or done operations on data that is no longer in sync because the set of data may not have been consistent for its operation, hence the dirty read. this is a more interesting and IMO legitimate concern, i haven't done any testing recently, but i believe as it currently stands this should throw a ConflictError (i'm not sure, though.). Jim and pythonlabs have talked about setting up some sort MVCC for the zodb, whereby transaction B will see a snapshot of objects as it existed when it began. if you're interested in this further i would send mail to the zodb-dev@zope.org list, referencing this thread. cheers kapil --- Clark OBrien <COBrien@isis-server.vuse.vanderbilt.edu> wrote:
Here is the example I was thinking about.
1) Suppose an online store is selling product X and at a given moment its inventory is n. Two shoppers hit the buy button at about the same time.
Then either:
a) No conflict is detected because both threads set the inventory value to the same value, n-1. This would cause a problem if n = 1
Or: b) A conflict is detected when the last thread writes because the inventory is not n as it was when he read it. request is resubmitted but again another thread has changed made the value dirty- thread starves.
-----Original Message----- From: Chris McDonough [mailto:chrism@zope.com] Sent: Thursday, November 01, 2001 6:50 AM To: Clark OBrien; k_vertigo@yahoo.com; zope@zope.org Subject: Re: [Zope] zope operations atomic?
What is "the dirty read problem"?
__________________________________________________ Do You Yahoo!? Make a great connection at Yahoo! Personals. http://personals.yahoo.com
AFAIK, a read conflict is raised in this circumstance, so there can't be a dirty read. But again, since requests which raise conflicts are retried, there is never a consistency problem. - C Kapil Thangavelu wrote:
even more generally the problem can be stated as transaction A starts reads a bunch of objects, transaction B starts and will read a set of objects that intersects with transaction A. transaction A commits changing objects that B is about to read (after B has read some objects). B finishes, and has possibly commited or done operations on data that is no longer in sync because the set of data may not have been consistent for its operation, hence the dirty read.
this is a more interesting and IMO legitimate concern, i haven't done any testing recently, but i believe as it currently stands this should throw a ConflictError (i'm not sure, though.). Jim and pythonlabs have talked about setting up some sort MVCC for the zodb, whereby transaction B will see a snapshot of objects as it existed when it began. if you're interested in this further i would send mail to the zodb-dev@zope.org list, referencing this thread.
cheers
kapil
--- Clark OBrien <COBrien@isis-server.vuse.vanderbilt.edu> wrote:
Here is the example I was thinking about.
1) Suppose an online store is selling product X and at a given moment its inventory is n. Two shoppers hit the buy button at about the same time.
Then either:
a) No conflict is detected because both threads set the inventory value to the same value, n-1. This would cause a problem if n = 1
Or: b) A conflict is detected when the last thread writes because the inventory is not n as it was when he read it. request is resubmitted but again another thread has changed made the value dirty- thread starves.
-----Original Message----- From: Chris McDonough [mailto:chrism@zope.com] Sent: Thursday, November 01, 2001 6:50 AM To: Clark OBrien; k_vertigo@yahoo.com; zope@zope.org Subject: Re: [Zope] zope operations atomic?
What is "the dirty read problem"?
__________________________________________________ Do You Yahoo!? Make a great connection at Yahoo! Personals. http://personals.yahoo.com
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
participants (3)
-
Chris McDonough -
Clark OBrien -
Kapil Thangavelu