I will convert my example to code and post the results. I did not understand case b). In particular what does "ghost out" mean. -----Original Message----- From: Chris McDonough [mailto:chrism@digicool.com] Sent: Thursday, November 01, 2001 2:01 PM To: Clark OBrien Cc: 'k_vertigo@yahoo.com'; 'zope@zope.org' Subject: Re: [Zope] zope operations atomic? Clark OBrien wrote:
It is what happens with this retrying that interests me. As the example given below shows, this retry may never succeed. Thus long running operations may never complete and are effectively starved by faster running operations.
Assume in Zope I have a folder structure like this.
Folder-1 -Folder-2 -Folder-3 .... .. Folder-1000
Suppose I have a script traverseFolder(root) that starts at a given root and traverses sub-folders adding the attribute foo.
If I continuously call traversefolder(Folder-1000) how could traverseFolder(Folder-1) ever complete. I mean, by the time traverseFolder(Folder-1) could complete traverseFolder(Folder-1000) would have already committed several times. What would happen with the request that started traverseFolder(Folder-1), would it keep being retried ad infinitum. It is basically starved out of contention by a faster running operations.
It does get interesting when you consider that the longer-running transaction might always tend to lose on read conflicts because: a) read conflicts can't be resolved. b) the longer-running transaction *might* "ghost out" Folder-1000 to conserve RAM during the traversal after the first (aborted) commit. If b) is true in your example for every run of traverseFolder(Folder-1), your contention that it will never commit might prove correct. It'd be slightly interesting to try it to see what the behavior actually is. Would you be willing to do so? In any case, we have an answer to this problem if you're willing to lose a bit of consistency. CoreSessionTracking's LowConflictConnection class comes in handy here. If your application is very write-intensive and you've carefully coded a massive hotspot ala your example into it, you can turn off read conflicts by using this class at the expense of some consistency. With read conflicts turned off, I'm positive your example will eventually resolve itself. Maybe it'll take a few minutes of retries, but it will eventually finish. I say that because the longer running of the two scripts will eventually be able to do the commit because it's statistically as likely to "win" a commit as the shorter-running script when there's a write conflict; it just has fewer opportunities to do so. And actually if you used a special FooFolder instead of a Folder for this demonstration, the only thing that changed was foo, and the value of foo was often the same in both connections on the object upon which the transaction conflicted, you could resolve most of the conflicts that could potentially occur here (save for the read conflicts) by giving it a _p_resolveConflict that looked something like this: class FooFolder: def _p_resolveConflict(self, old, saved, new): marker = [] for k,v in saved.items(): if new.get(k, marker) != v: return None for k,v in new.items(): if saved.get(k, marker) != v: return None return new However, the real answer is: Dont design your application like this if you can help it. This is not a good pattern. It's best to avoid hotspots like Folder-1000 in your example. It's no different than continually beating the snot out of a field in a row in a relational database table with writes from multiple threads, where one of the threads is running an overnight transaction and the others are just incrementing the field every second. You have the same consistency, contention, and timeliness issues there, AFAICT, except that it's expressed in terms of pages, locks, and dirty reads. (I'm sure there's an Oracle person waiting around to "WRONG!" me to death, however. ;-) - C
"ghost out" means "deactivate" the object, meaning essentially that we dump all the objects' state out of memory except what is necessary to load it back in from disk. Clark OBrien wrote:
I will convert my example to code and post the results. I did not understand case b). In particular what does "ghost out" mean.
-----Original Message----- From: Chris McDonough [mailto:chrism@digicool.com] Sent: Thursday, November 01, 2001 2:01 PM To: Clark OBrien Cc: 'k_vertigo@yahoo.com'; 'zope@zope.org' Subject: Re: [Zope] zope operations atomic?
Clark OBrien wrote:
It is what happens with this retrying that interests me. As the example given below shows, this retry may never succeed. Thus long running operations may never complete and are effectively starved by faster running operations.
Assume in Zope I have a folder structure like this.
Folder-1 -Folder-2 -Folder-3 .... .. Folder-1000
Suppose I have a script traverseFolder(root) that starts at a given root and traverses sub-folders adding the attribute foo.
If I continuously call traversefolder(Folder-1000) how could traverseFolder(Folder-1) ever complete. I mean, by the time traverseFolder(Folder-1) could complete traverseFolder(Folder-1000) would have already committed several times. What would happen with the request that started traverseFolder(Folder-1), would it keep being retried ad infinitum. It is basically starved out of contention by a faster running operations.
It does get interesting when you consider that the longer-running transaction might always tend to lose on read conflicts because:
a) read conflicts can't be resolved. b) the longer-running transaction *might* "ghost out" Folder-1000 to conserve RAM during the traversal after the first (aborted) commit.
If b) is true in your example for every run of traverseFolder(Folder-1), your contention that it will never commit might prove correct. It'd be slightly interesting to try it to see what the behavior actually is. Would you be willing to do so?
In any case, we have an answer to this problem if you're willing to lose a bit of consistency. CoreSessionTracking's LowConflictConnection class comes in handy here. If your application is very write-intensive and you've carefully coded a massive hotspot ala your example into it, you can turn off read conflicts by using this class at the expense of some consistency.
With read conflicts turned off, I'm positive your example will eventually resolve itself. Maybe it'll take a few minutes of retries, but it will eventually finish. I say that because the longer running of the two scripts will eventually be able to do the commit because it's statistically as likely to "win" a commit as the shorter-running script when there's a write conflict; it just has fewer opportunities to do so.
And actually if you used a special FooFolder instead of a Folder for this demonstration, the only thing that changed was foo, and the value of foo was often the same in both connections on the object upon which the transaction conflicted, you could resolve most of the conflicts that could potentially occur here (save for the read conflicts) by giving it a _p_resolveConflict that looked something like this:
class FooFolder: def _p_resolveConflict(self, old, saved, new): marker = [] for k,v in saved.items(): if new.get(k, marker) != v: return None for k,v in new.items(): if saved.get(k, marker) != v: return None return new
However, the real answer is: Dont design your application like this if you can help it. This is not a good pattern. It's best to avoid hotspots like Folder-1000 in your example. It's no different than continually beating the snot out of a field in a row in a relational database table with writes from multiple threads, where one of the threads is running an overnight transaction and the others are just incrementing the field every second. You have the same consistency, contention, and timeliness issues there, AFAICT, except that it's expressed in terms of pages, locks, and dirty reads. (I'm sure there's an Oracle person waiting around to "WRONG!" me to death, however. ;-)
- C
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
--- Chris McDonough <chrism@digicool.com> wrote:
"ghost out" means "deactivate" the object, meaning essentially that we dump all the objects' state out of memory except what is necessary to load it back in from disk.
chris, i've never been clear on the difference in state between a ghost and an unloaded object on disk? could you elaborate on this. thanks kapil __________________________________________________ Do You Yahoo!? Find a job, post your resume. http://careers.yahoo.com
chris,
i've never been clear on the difference in state between a ghost and an unloaded object on disk? could you elaborate on this.
They are the same, AFAIK.. at least I've never needed to make the distinction before? -- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
participants (3)
-
Chris McDonough -
Clark OBrien -
Kapil Thangavelu