Zope 2.8.4 ZODB 3.4.2 I have a dtml method which is an obvious hotspot. It puts up navigation wigetry and is rendered on nearly every page. The logs show it generates a large number of conflicts. The conflicts are in TemporaryStrorage and so are due to session variables. The session variables are accessed with short Script(Python) functions one call per session variable used: getSessionVariable( var ), Session variable hold mostly data that is constant throughout the session. There are about twenty values, ten of which are referenced in the navigation method. The structure of the method is simple enough: there is a large <dtml-let> block which populates local variables with data from the session variables <dtml-let foobar="getSessionVariable('foobar')" ... > with the body of the <dtml-let> containing 300 lines or so of dtml, control code, sql queries, and occasional dtml-let blocks to hold query results and computed values. So, this navigation method is clearly a good candidate for refactoring, but the question is "What's the best way to refactor it?". I still don't have an intuitve understanding of the conflict mechanism in practice. For example, Why is the conflict error reported for the navigation dtml method rather than for the getSessionVariable python script which actually references the SESSION object where the conflict occurs? A conflict error gets raised whenever another thread references the same session's SESSION object. Apparently the MVCC read-read conflict resolution fails for session variables although I am not sure why. The DEBUG logs indicate that read-read conflicts found. It would be helpful to understand why. When the conflict is resolved, one is backed out (everything is transactional) and restarted. After three retries, it is considered an error. This processing is handled automatically by Zope. At one point, I had considered writing a custom conflict resolution function where resolution assumed independence of individual session variables. After some thought and experimentation, I concluded that was a bad idea because it could break synchronization assertions between session variables (e.g., X is twice Y). Absent a magical conflict resolution method, the best approach would seem to be to minimize the potential for conflicts. Intuition says that the simplest way to do this would be to have the navigation dtml method make a single call to a python script, getSessionVariables, which would return a copy of the session variables. The copy could then be unpacked and processed without any potential conflict errors.
The structure of the method is simple enough: there is a large <dtml-let> block which populates local variables with data from the session variables
<dtml-let foobar="getSessionVariable('foobar')" ...
with the body of the <dtml-let> containing 300 lines or so of dtml, control code, sql queries, and occasional dtml-let blocks to hold query results and computed values.
If this is the same getSessionVariable as you reported before (a Python Script), note that each call that you make to it has overhead. It might be better to create a single Python script to return all of the values that you need from the session rather than calling this over and over again. If that doesn't work, it might be even better to just do SESSION.get('foobar') instead of calling a Python Script for every value.
So, this navigation method is clearly a good candidate for refactoring, but the question is "What's the best way to refactor it?".
Making it do its job faster will likely help. Likewise,
I still don't have an intuitve understanding of the conflict mechanism in practice. For example, Why is the conflict error reported for the navigation dtml method rather than for the getSessionVariable python script which actually references the SESSION object where the conflict occurs?
Write conflicts happen for a transaction. In Zope, there is one transaction per request. So it reports the URL of the method that was called as the thing that caused the conflict (because, basically, it was). A conflict is not "noticed" until a transaction is committed; ZODB has no idea what caused the data structure to change in a conflicting way within "application code".
A conflict error gets raised whenever another thread references the same session's SESSION object.
Are you using frames by any chance? It would be understandable that you're seeing an increased number conflicts if you have an application in which many threads try to access the same session object at once due to a browser making multiple writes to the same session object because of frames.
Apparently the MVCC read-read conflict resolution fails for session variables although I am not sure why.
Can we see one of the conflict tracebacks? You should be seeing only write conflicts. Read conflict errors are reported in a traceback with a ReadConflictError message (conflicts reported as only ConflictError are write conflicts).
The DEBUG logs indicate that read-read conflicts found. It would be helpful to understand why.
Yes. A traceback would help.
When the conflict is resolved, one is backed out (everything is transactional) and restarted. After three retries, it is considered an error. This processing is handled automatically by Zope.
That's right.
At one point, I had considered writing a custom conflict resolution function where resolution assumed independence of individual session variables. After some thought and experimentation, I concluded that was a bad idea because it could break synchronization assertions between session variables (e.g., X is twice Y).
That's a good conclusion.
Absent a magical conflict resolution method, the best approach would seem to be to minimize the potential for conflicts. Intuition says that the simplest way to do this would be to have the navigation dtml method make a single call to a python script, getSessionVariables, which would return a copy of the session variables. The copy could then be unpacked and processed without any potential conflict errors.
Heh. Good thinking! ;-) Really, anything that makes your transaction finish faster is the right thing (and good coding practice to boot). Maybe not a silver bullet especially if you're using frames but it's a start. - C
Chris, thanks again. You've clarified several issues. Interlinear comments below, original text edited with a heavy hand for brevity... On Wed, 21 Dec 2005, Chris McDonough wrote:
If this is the same getSessionVariable as you reported before (a Python Script), note that each call that you make to it has overhead. It might be better to create a single Python script to return all of the values that you need from the session rather than calling this over and over again. If that doesn't work, it might be even better to just do SESSION.get('foobar') instead of calling a Python Script for every value.
Good point. No overhead is probably the right thing especially since there are no subtransactions.
I still don't have an intuitve understanding of the conflict mechanism in practice. For example, Why is the conflict error reported for the navigation dtml method rather than for the getSessionVariable python script which actually references the SESSION object where the conflict occurs?
Write conflicts happen for a transaction. In Zope, there is one transaction per request. So it reports the URL of the method that was called as the thing that caused the conflict (because, basically, it was). A conflict is not "noticed" until a transaction is committed; ZODB has no idea what caused the data structure to change in a conflicting way within "application code".
Ah so desu. That's the conceptual hook I was missing. Only one transaction per request and no subtransactions! A transaction is processing initiated by a client request or a redirect. (Anything else?) A transaction has its own REQUEST and RESPONSE objects.
A conflict error gets raised whenever another thread references the same session's SESSION object.
Are you using frames by any chance? It would be understandable that you're seeing an increased number conflicts if you have an application in which many threads try to access the same session object at once due to a browser making multiple writes to the same session object because of frames.
Yes, we do use frames. We manage rendering explicitly with a bit of Javascript magic so the full frameset rarely gets rerendered. But, it certainly could be a contributing factor.
Apparently the MVCC read-read conflict resolution fails for session variables although I am not sure why.
Can we see one of the conflict tracebacks? You should be seeing only write conflicts. Read conflict errors are reported in a traceback with a ReadConflictError message (conflicts reported as only ConflictError are write conflicts).
I'll try to find a few tracebacks. --
Dennis Allison wrote:
Ah so desu. That's the conceptual hook I was missing. Only one transaction per request and no subtransactions!
ZODB substransactions won't help you here in the slightest...
A transaction is processing initiated by a client request or a redirect. (Anything else?) A transaction has its own REQUEST and RESPONSE objects.
Wrong way round. Each REQUEST has one RESPONSE associated with it, and the publisher does either a transaction commit, if no errors occur, or a transaction abort, if errors occur, once it has processed a REQUEST. Read conflict errors can be raised at any time, but shouldn't really occur once you have MVCC working. Write conflict errors only happen during transaction commit. The publisher retries the whole request if a conflict errors of either sort occurs. If the retry fails 3 times, it gives up, aborts the transaction and reports the conflict error to the end user by way of writing to the response object. hth, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris W, The issue here has had to do with session variables and their iteraction with the persistence mechanism and conflicts and multiple threads for the same session. Chris McDonough has pointed out that session variables can cause writes and write conflicts even if the only Zope level access is a read. I thought MVCC works out of the box for Zope 2.8.4 which uses ZODB 2.3.4. Am I wrong? The real surprise was that you can get a write conflict from a pair of session variable reads! Sorry, I wasn't all that clear when I had my epiphany about REQUEST and transactions. You said it much more clearly and precisely. Since conflict errors are discovered and managed when the publisher commits, there is not a whole lot one can do, in terms of code organization, to minimize the potential for conflicts. Thanks for your insight. On Wed, 21 Dec 2005, Chris Withers wrote:
Dennis Allison wrote:
Ah so desu. That's the conceptual hook I was missing. Only one transaction per request and no subtransactions!
ZODB substransactions won't help you here in the slightest...
A transaction is processing initiated by a client request or a redirect. (Anything else?) A transaction has its own REQUEST and RESPONSE objects.
Wrong way round. Each REQUEST has one RESPONSE associated with it, and the publisher does either a transaction commit, if no errors occur, or a transaction abort, if errors occur, once it has processed a REQUEST. Read conflict errors can be raised at any time, but shouldn't really occur once you have MVCC working. Write conflict errors only happen during transaction commit. The publisher retries the whole request if a conflict errors of either sort occurs. If the retry fails 3 times, it gives up, aborts the transaction and reports the conflict error to the end user by way of writing to the response object.
hth,
Chris
--
On Dec 21, 2005, at 12:40 PM, Dennis Allison wrote:
Chris W,
The issue here has had to do with session variables and their iteraction with the persistence mechanism and conflicts and multiple threads for the same session. Chris McDonough has pointed out that session variables can cause writes and write conflicts even if the only Zope level access is a read. I thought MVCC works out of the box for Zope 2.8.4 which uses ZODB 2.3.4. Am I wrong?
It does. But MVCC does not prevent write conflicts. Only read conflicts.
The real surprise was that you can get a write conflict from a pair of session variable reads!
That is surprising, no doubt. ;-)
On 12/22/05, Dennis Allison <allison@shasta.stanford.edu> wrote:
The issue here has had to do with session variables and their iteraction with the persistence mechanism and conflicts and multiple threads for the same session. Chris McDonough has pointed out that session variables can cause writes and write conflicts even if the only Zope level access is a read. I thought MVCC works out of the box for Zope 2.8.4 which uses ZODB 2.3.4. Am I wrong?
The real surprise was that you can get a write conflict from a pair of session variable reads!
The missing detail here is that reading a session object causes a write to the database to update the last access time for that session object. So what looks like a plain old read to the application code above, in-fact includes a write just below the surface. (This is a design choice so that stale session objects can be removed.) There are implementation details of TransientObject that attempt to mitigate some of the pain: - The access time has a WRITEGRANULARITY. That can help a bit by avoiding writing altogether some of the time. But still there is need to write (and so potential contention) each time a boundary of that granularity is crossed. - And _p_resolveConflict of TransientObject will most often pick a state to avoid a conflict error. When it does there is the performance cost of writing to database but there will be no ConflictError. However, I've suggested in the past that you should comment out _p_resolveConflict while debugging consistency problems. What is the current state of your _p_resolveConflict now? _p_resolveConflict includes a DIWM'ly component. Perhaps you still need to avoid that component of _p_resolveConflict? In that case a _p_resolveConflict could be engineered that can resolve the case of just updates to the access time? But still bails out when presented with two states that have explicit updates to maintain consistency. Though that's all just conjecture till we see your tracebacks for your conflicts. michael
Michael -- You raise an interesting point. We use sessions to hold the volatile state for each user in a highly interactive, highly personalized site. A session lasts from login until logout or until there has been no activity for a long time. Perhaps, if I were to simply excise the session timeout mechanism and the write of the last access time all would be well. We do not use session timeouts and could easily handle the recovery in a different way. I have changed the WRITEGRANULARITY and other parameters to be more comensurate with our long session times. I have begun to think that the Zope session variable mechanism, as implemented, addresses a different part of the storage spectrum than we need. A small number of per user parameters (3 to 10) are used by almost every request. Most of the parameters are set at user login and are not changed for the duration of the session; sessions can last a long time, up to ten hours, say, Some of the parameters change more rapidly, for example, those that capture the current navigation state. It may be that we need two different kinds of session data, one for login-static data and another for dynamic data. I'm still working on getting some detailed traceback information, difficult when the problems occur only infrequently on loaded production systems with a 24x7 service profile. We are moving to Zope 2.8.5 to take advantage of the better conflict reporting. On Thu, 22 Dec 2005, Michael Dunstan wrote:
On 12/22/05, Dennis Allison <allison@shasta.stanford.edu> wrote:
The issue here has had to do with session variables and their iteraction with the persistence mechanism and conflicts and multiple threads for the same session. Chris McDonough has pointed out that session variables can cause writes and write conflicts even if the only Zope level access is a read. I thought MVCC works out of the box for Zope 2.8.4 which uses ZODB 2.3.4. Am I wrong?
The real surprise was that you can get a write conflict from a pair of session variable reads!
The missing detail here is that reading a session object causes a write to the database to update the last access time for that session object. So what looks like a plain old read to the application code above, in-fact includes a write just below the surface. (This is a design choice so that stale session objects can be removed.)
There are implementation details of TransientObject that attempt to mitigate some of the pain:
- The access time has a WRITEGRANULARITY. That can help a bit by avoiding writing altogether some of the time. But still there is need to write (and so potential contention) each time a boundary of that granularity is crossed.
- And _p_resolveConflict of TransientObject will most often pick a state to avoid a conflict error. When it does there is the performance cost of writing to database but there will be no ConflictError. However, I've suggested in the past that you should comment out _p_resolveConflict while debugging consistency problems. What is the current state of your _p_resolveConflict now? _p_resolveConflict includes a DIWM'ly component. Perhaps you still need to avoid that component of _p_resolveConflict? In that case a _p_resolveConflict could be engineered that can resolve the case of just updates to the access time? But still bails out when presented with two states that have explicit updates to maintain consistency.
Though that's all just conjecture till we see your tracebacks for your conflicts.
michael
--
On Dec 21, 2005, at 2:17 PM, Dennis Allison wrote:
I have begun to think that the Zope session variable mechanism, as implemented, addresses a different part of the storage spectrum than we need. A small number of per user parameters (3 to 10) are used by almost every request. Most of the parameters are set at user login and are not changed for the duration of the session;
These would be better implemented as "member data" (ala CMF's member data tool). - C
The missing detail here is that reading a session object causes a write to the database to update the last access time for that session object. So what looks like a plain old read to the application code above, in-fact includes a write just below the surface. (This is a design choice so that stale session objects can be removed.)
Some time ago (search the archives) I had problems with high rate of write conflict errors and then Michael Dunstan has written: "One option to reduce the rate of this write conflict is to tune the session machinery to suit. For example use session-resolution-seconds of say 300 seconds." It helped me a lot - almost no conflict errors :) Maybe you should try to change this setting in zope.conf and see what will happen. Im curious if there are side effects of setting high value to this variable? I haven't noticed any... -- Maciej Wisniowski
The side effect is that sessions will live longer than the specified period, up to 300 seconds longer if session-resolution-seconds is set to 30 seconds. In my case, setting it higher helps, b ut does not eliminate the problems. On Wed, 21 Dec 2005, Maciej Wisniowski wrote:
The missing detail here is that reading a session object causes a write to the database to update the last access time for that session object. So what looks like a plain old read to the application code above, in-fact includes a write just below the surface. (This is a design choice so that stale session objects can be removed.)
Some time ago (search the archives) I had problems with high rate of write conflict errors and then Michael Dunstan has written:
"One option to reduce the rate of this write conflict is to tune the session machinery to suit. For example use session-resolution-seconds of say 300 seconds."
It helped me a lot - almost no conflict errors :) Maybe you should try to change this setting in zope.conf and see what will happen.
Im curious if there are side effects of setting high value to this variable? I haven't noticed any...
--
Dennis Allison wrote:
The issue here has had to do with session variables and their iteraction with the persistence mechanism and conflicts and multiple threads for the same session. Chris McDonough has pointed out that session variables can cause writes and write conflicts even if the only Zope level access is a read.
Indeed, they have to track expirey and the like. This is why "sessions are evil" (tm)
I thought MVCC works out of the box for Zope 2.8.4 which uses ZODB 2.3.4. Am I wrong?
Nope, it works fine, we've seen all our read conflicts go away :-)
Sorry, I wasn't all that clear when I had my epiphany about REQUEST and transactions. You said it much more clearly and precisely. Since conflict errors are discovered and managed when the publisher commits, there is not a whole lot one can do, in terms of code organization, to minimize the potential for conflicts.
Oh, there's plenty, I can assure you ;-) Ditching your frames is the 1st place I'd start... any scenario where several frames/iframes/whatever try to manipulate the same session variable (ie: per user) is doomed to failure ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
One more thought --- On Wed, 21 Dec 2005, Chris McDonough wrote:
The structure of the method is simple enough: there is a large <dtml-let> block which populates local variables with data from the session variables
<dtml-let foobar="getSessionVariable('foobar')" ...
with the body of the <dtml-let> containing 300 lines or so of dtml, control code, sql queries, and occasional dtml-let blocks to hold query results and computed values.
If this is the same getSessionVariable as you reported before (a Python Script), note that each call that you make to it has overhead. It might be better to create a single Python script to return all of the values that you need from the session rather than calling this over and over again. If that doesn't work, it might be even better to just do SESSION.get('foobar') instead of calling a Python Script for every value. ... Write conflicts happen for a transaction. In Zope, there is one transaction per request.
There are two distinct sections to the navigation_box transaction. One where the session variables are read and a second where they are used. To minimize conflicts, the what is now a single tranaction should be split into two separate transactions. Any thought as to how to do that? If navigation_box were broken into two separate methods, say nav_box1 and nav_box2, how does nav_box1 commit itself and then transfer control and data (a session variable snapshot) to nav_box2 as a new transaction? I would guess that if nav_box1 redirects to nav_box2 a new transaction is initiated and the old one committed. Is that correct? And is there a better way to get the same effect?
There are two distinct sections to the navigation_box transaction. One where the session variables are read and a second where they are used. To minimize conflicts, the what is now a single tranaction should be split into two separate transactions.
Any thought as to how to do that? If navigation_box were broken into two separate methods, say nav_box1 and nav_box2, how does nav_box1 commit itself and then transfer control and data (a session variable snapshot) to nav_box2 as a new transaction?
I would guess that if nav_box1 redirects to nav_box2 a new transaction is initiated and the old one committed. Is that correct? And is there a better way to get the same effect?
Hi You've written in your first post that you're dealing with read conflict errors so I think transaction commiting is not the point here - it would cause write conflict. Am I right? I think that few simple things you can do is using external method instead of your's python script getSessionVariable (to make it faster) and redesigning it's code that it returns (at once) for example a dictionary filled with requested values from session. Maybe you can use external method to generate whole navigation. The best thing would be to write product but this is a bit more work, and I think external methods are much faster to check in your situation. Other thing you can do is to use ZopeProfiler to see exactly what is called when the page is requested and how long it takes. Other quick solution, which may not be appropiate in your case, is to render navigation box once and store whole the generated code in session. -- Maciej Wisniowski
Write conflicts happen for a transaction. In Zope, there is one transaction per request.
There are two distinct sections to the navigation_box transaction.
There's a transaction for something named "navigation_box"? Is this an IFRAME?
One where the session variables are read and a second where they are used. To minimize conflicts, the what is now a single tranaction should be split into two separate transactions.
Are you maybe misusing the term "transaction" here? If this logic all happens in the course of a single request, it all happens in the same transaction unless that request causes, say, the reload of a frame. There is a concept of a transaction here. But it doesn't have anything to do with what happens during the course of a single request unless you explicitly try to control transactions, which is almost never a good idea.
Any thought as to how to do that? If navigation_box were broken into two separate methods, say nav_box1 and nav_box2, how does nav_box1 commit itself and then transfer control and data (a session variable snapshot) to nav_box2 as a new transaction?
I suspect I don't understand what this would achieve. Having more transactions will cause more conflicts. Conflicts happen as a result of conflicting changes in two transactions.
I would guess that if nav_box1 redirects to nav_box2 a new transaction is initiated and the old one committed. Is that correct? And is there a better way to get the same effect?
I have no idea, sorry. I think maybe what you might want to do here is to not use builtin Zope sessions. ;-) Zope sessions rely on ZODB and are transactional. If they didn't rely on ZODB, you wouldn't be getting conflict errors. If they weren't transactional, you probably wouldn't notice. Might be time to cut bait here. We've been talking about this for months. ;-) I think someone wrote a relational database backend for the sessioning API some time ago. You may want to give that a shot. - C
Thanks again Chris for the helpful comments. The navigation_box, in this context is just a table which is rendered into a frame in our standard frameset. It is not an iframe. In the sense I used them below, a transaction and a request are the same thing. This follows from the fact that each request is managed as a single transaction. Perhaps it would have been a better choice of words to use "request". I suggested breaking the request into two requests as one way to help manage conflicts. Only the first part of the the split request would reference the session variables, so the window of opportunity for a session variable conflict would be smaller even though the number of requests is larger. And, sigh, you are probably right--it may be time to abandon the standard release session implementation and roll our own. On Wed, 21 Dec 2005, Chris McDonough wrote:
Write conflicts happen for a transaction. In Zope, there is one transaction per request.
There are two distinct sections to the navigation_box transaction.
There's a transaction for something named "navigation_box"? Is this an IFRAME?
One where the session variables are read and a second where they are used. To minimize conflicts, the what is now a single tranaction should be split into two separate transactions.
Are you maybe misusing the term "transaction" here? If this logic all happens in the course of a single request, it all happens in the same transaction unless that request causes, say, the reload of a frame.
There is a concept of a transaction here. But it doesn't have anything to do with what happens during the course of a single request unless you explicitly try to control transactions, which is almost never a good idea.
Any thought as to how to do that? If navigation_box were broken into two separate methods, say nav_box1 and nav_box2, how does nav_box1 commit itself and then transfer control and data (a session variable snapshot) to nav_box2 as a new transaction?
I suspect I don't understand what this would achieve. Having more transactions will cause more conflicts. Conflicts happen as a result of conflicting changes in two transactions.
I would guess that if nav_box1 redirects to nav_box2 a new transaction is initiated and the old one committed. Is that correct? And is there a better way to get the same effect?
I have no idea, sorry.
I think maybe what you might want to do here is to not use builtin Zope sessions. ;-) Zope sessions rely on ZODB and are transactional. If they didn't rely on ZODB, you wouldn't be getting conflict errors. If they weren't transactional, you probably wouldn't notice.
Might be time to cut bait here. We've been talking about this for months. ;-) I think someone wrote a relational database backend for the sessioning API some time ago. You may want to give that a shot.
- C
--
On Dec 21, 2005, at 10:32 AM, Dennis Allison wrote:
Thanks again Chris for the helpful comments.
The navigation_box, in this context is just a table which is rendered into a frame in our standard frameset. It is not an iframe.
So you do use frames! That's a huge clue. I wish I didn't feel like I had to drag that out of you. :-(
In the sense I used them below, a transaction and a request are the same thing. This follows from the fact that each request is managed as a single transaction. Perhaps it would have been a better choice of words to use "request".
I suggested breaking the request into two requests as one way to help manage conflicts. Only the first part of the the split request would reference the session variables, so the window of opportunity for a session variable conflict would be smaller even though the number of requests is larger.
This is probably not really a reasonable thing to do. A request is generated by a user agent. With a lot of deep dark magic, you *can* generate a request from within Zope that makes it appear that it comes from the outside, but it's not easy and unhelpful here. Remember what I said about the "write on read" pattern of sessions? it doesn't matter if you're "only reading" session data. There still may be writes happening. Breaking things up into more requests will not help. What you want to do instead is *not access the same session from two different requests at the same time*. You probably know this, but when a frameset is used, the browser generates N requests... one per frame. These requests happen at about "the same time" whenever the frameset is accessed by a user. Moreover, since the requests come from the same browser, the session identifier for both requests is the same. By accessing session data from code within more than one frame, you are essentially creating a situation where at least two different threads will always be accessing the very same session object whenever the frameset is rendered. For as many frames as are in the frameset, if the code that renders each frame access the session, a transaction will touch the same session data object and perhaps some shared housekeeping data structures. It's as if you're pressing the "refresh" button in rapid succession on a page that accesses session data. This is prime area for a conflict.
And, sigh, you are probably right--it may be time to abandon the standard release session implementation and roll our own.
This is still a reasonable thing to do, but if you can get away with using sessions in only *one* of the pages of your frameset, things will get much better. - C
Chris McDonough schrieb: ..
This is still a reasonable thing to do, but if you can get away with using sessions in only *one* of the pages of your frameset, things will get much better.
Or even drop frames completely. They are too much pain for too less gain imho.
Chris, You asked about frames a while back and I responded in the affirmative. I am sure I mentioned that we use frames and framesets and explained that we use a bit of Javascript to manage loading individual frames rather than loading the entire frameset whenever possible. Still, your point about the multiple threads due to the requests from individual frames when a frameset gets rendered is a valid one and one I have not given the weight it deserves. Issues around multiple threads and concurrency is going to be more and more of a problem as we move to smarter clients and AJAX-like implementations. It looks like there are several small things I can do to improve the present system's conflict rate, but, absent some major structural changes, I won't be able to drive them to O(zero). Fortunately, while the level of conflicts is high, they are infrequent relative to the number of hits served, and most are resolved by the normal Zope conflict resolution mechanism. On Wed, 21 Dec 2005, Chris McDonough wrote:
On Dec 21, 2005, at 10:32 AM, Dennis Allison wrote:
Thanks again Chris for the helpful comments.
The navigation_box, in this context is just a table which is rendered into a frame in our standard frameset. It is not an iframe.
So you do use frames! That's a huge clue. I wish I didn't feel like I had to drag that out of you. :-(
In the sense I used them below, a transaction and a request are the same thing. This follows from the fact that each request is managed as a single transaction. Perhaps it would have been a better choice of words to use "request".
I suggested breaking the request into two requests as one way to help manage conflicts. Only the first part of the the split request would reference the session variables, so the window of opportunity for a session variable conflict would be smaller even though the number of requests is larger.
This is probably not really a reasonable thing to do. A request is generated by a user agent. With a lot of deep dark magic, you *can* generate a request from within Zope that makes it appear that it comes from the outside, but it's not easy and unhelpful here. Remember what I said about the "write on read" pattern of sessions? it doesn't matter if you're "only reading" session data. There still may be writes happening. Breaking things up into more requests will not help.
What you want to do instead is *not access the same session from two different requests at the same time*. You probably know this, but when a frameset is used, the browser generates N requests... one per frame. These requests happen at about "the same time" whenever the frameset is accessed by a user. Moreover, since the requests come from the same browser, the session identifier for both requests is the same.
By accessing session data from code within more than one frame, you are essentially creating a situation where at least two different threads will always be accessing the very same session object whenever the frameset is rendered. For as many frames as are in the frameset, if the code that renders each frame access the session, a transaction will touch the same session data object and perhaps some shared housekeeping data structures. It's as if you're pressing the "refresh" button in rapid succession on a page that accesses session data. This is prime area for a conflict.
And, sigh, you are probably right--it may be time to abandon the standard release session implementation and roll our own.
This is still a reasonable thing to do, but if you can get away with using sessions in only *one* of the pages of your frameset, things will get much better.
- C
--
On Dec 21, 2005, at 11:38 AM, Dennis Allison wrote:
Chris,
You asked about frames a while back and I responded in the affirmative. I am sure I mentioned that we use frames and framesets and explained that we use a bit of Javascript to manage loading individual frames rather than loading the entire frameset whenever possible.
Eek. Sorry, I don't remember this. Context is hard to keep over the course of several weeks/months, of course. And my brain is going with my age. I don't mean to be a nag (really, I understand this is hard), but I asked two emails ago "do you use frames?" and you didn't answer that question directly in your response at all; only by way of teasing it out of you by asking if it was an IFRAME in a followup was the fact that this app is a frame-based one re-revealed to me. In general, it would be very helpful if you could answer the questions that are asked when they're asked even if they reveal the fact that I'm a doddering old man and can't remember things from moment to moment, Otherwise it's difficult (and honestly a bit frustrating) to try to help you. In any case, typically the problem with frames only reveals itself when the entire frameset is loaded or the body of one frame causes the other frames to load. Your javascript may not be helping here... it could be hurting. For example, if a user reloads the "main" frame (say as a result of a POST request) and by doing so, during its re- rendering, it causes two other frames to refresh themselves due to an inline script function in the body of the main frame, that's the moral equivalent of reloading the frameset entirely. You have a tiny bit more control, e.g. maybe you could set a Javscript timer to reload the other frames in a "staggered" way where you reload one, then the other after a certain number of seconds. But otherwise it's the same problem. Hard to know what's going on there.
Still, your point about the multiple threads due to the requests from individual frames when a frameset gets rendered is a valid one and one I have not given the weight it deserves.
Issues around multiple threads and concurrency is going to be more and more of a problem as we move to smarter clients and AJAX-like implementations.
"Smart clients" perhaps. But Ajax is supposed to allow you to do interesting things *without* reloading the page, and because (to my knowledge) no current Javascript implementation can make a request asynchronously, so you can only make one request at a time from within a given browser document. So using Ajax shouldn't cause *more* conflicts; it should cause fewer. Unless of course your Ajax calls emanate from different frames (different DOM "documents"). Then you're back in the same boat (depending on the Javascript implementation, I suppose). It's really a problem with frames (or more generally, multiple documents at the same time) rather than with Ajax.
It looks like there are several small things I can do to improve the present system's conflict rate, but, absent some major structural changes, I won't be able to drive them to O(zero). Fortunately, while the level of conflicts is high, they are infrequent relative to the number of hits served, and most are resolved by the normal Zope conflict resolution mechanism.
Well, that's good. Does that mean I'm off the hook? ;-) - C
On Wed, 21 Dec 2005, Chris McDonough wrote:
On Dec 21, 2005, at 11:38 AM, Dennis Allison wrote:
Chris,
You asked about frames a while back and I responded in the affirmative. I am sure I mentioned that we use frames and framesets and explained that we use a bit of Javascript to manage loading individual frames rather than loading the entire frameset whenever possible.
Eek. Sorry, I don't remember this. Context is hard to keep over the course of several weeks/months, of course. And my brain is going with my age. I don't mean to be a nag (really, I understand this is hard), but I asked two emails ago "do you use frames?" and you didn't answer that question directly in your response at all; only by way of teasing it out of you by asking if it was an IFRAME in a followup was the fact that this app is a frame-based one re-revealed to me. In general, it would be very helpful if you could answer the questions that are asked when they're asked even if they reveal the fact that I'm a doddering old man and can't remember things from moment to moment, Otherwise it's difficult (and honestly a bit frustrating) to try to help you.
Sorry if I appeared unresponsive--the fact that we use frames is hardly a secret. I suppose that it would be helpful to make up soem summary of features for ongoing threads like this one.
In any case, typically the problem with frames only reveals itself when the entire frameset is loaded or the body of one frame causes the other frames to load. Your javascript may not be helping here... it could be hurting. For example, if a user reloads the "main" frame (say as a result of a POST request) and by doing so, during its re- rendering, it causes two other frames to refresh themselves due to an inline script function in the body of the main frame, that's the moral equivalent of reloading the frameset entirely. You have a tiny bit more control, e.g. maybe you could set a Javscript timer to reload the other frames in a "staggered" way where you reload one, then the other after a certain number of seconds. But otherwise it's the same problem. Hard to know what's going on there.
By convention the primary content frame does not cause other frames to re-render but there are multiple implementors and creaping featurism which may have changed that. The navigation_box frame we have been discussion gets re-rendered on a navigation change (duh) and may initiate re-rendering of a tabs_frame. Except for the full frameset rendering, we are likely to be fairly conflict clean. I can look in the logs and tell... Might be interesting.
Still, your point about the multiple threads due to the requests from individual frames when a frameset gets rendered is a valid one and one I have not given the weight it deserves.
Issues around multiple threads and concurrency is going to be more and more of a problem as we move to smarter clients and AJAX-like implementations.
"Smart clients" perhaps. But Ajax is supposed to allow you to do interesting things *without* reloading the page, and because (to my knowledge) no current Javascript implementation can make a request asynchronously, so you can only make one request at a time from within a given browser document. So using Ajax shouldn't cause *more* conflicts; it should cause fewer. Unless of course your Ajax calls emanate from different frames (different DOM "documents"). Then you're back in the same boat (depending on the Javascript implementation, I suppose). It's really a problem with frames (or more generally, multiple documents at the same time) rather than with Ajax.
I did not know that XMLHttp interactions were asynchronous but serialized. Most browsers can make multiple simultaneous requests so I'd assumed that XMLHttp would just grab a connection and go and that multiple requests can be in progress simultaneously.
It looks like there are several small things I can do to improve the present system's conflict rate, but, absent some major structural changes, I won't be able to drive them to O(zero). Fortunately, while the level of conflicts is high, they are infrequent relative to the number of hits served, and most are resolved by the normal Zope conflict resolution mechanism.
Well, that's good. Does that mean I'm off the hook? ;-)
Not really off the hook as issues still remain and you are a valuable resource. :-) I need to spend some time thinking about the issues an do some cautious experimentation and testing.
On Dec 21, 2005, at 1:07 PM, Dennis Allison wrote:
Sorry if I appeared unresponsive--the fact that we use frames is hardly a secret. I suppose that it would be helpful to make up soem summary of features for ongoing threads like this one.
It was a secret to me (again, perhaps because of my inability to keep a thought in my head for more than an hour, for better or worse) or I honestly I wouldn't have asked again. Just answering the questions as they're asked would really, really be the best tact here, because configurations change, code changes, and so on. What was gospel a month ago might not be the case now. So it's important to just answer the questions even if it seems utterly redundant to do so. Again, I'm not chiding you here, I realize this is hard stuff, but I need you to *help me help you* to get this put to bed.
By convention the primary content frame does not cause other frames to re-render but there are multiple implementors and creaping featurism which may have changed that. The navigation_box frame we have been discussion gets re-rendered on a navigation change (duh) and may initiate re-rendering of a tabs_frame. Except for the full frameset rendering, we are likely to be fairly conflict clean. I can look in the logs and tell... Might be interesting.
Yep.. particularly if you see a conflict only in requests that directly follow a request for the frameset URL. If that's the case, unless people constantly reload the frameset, you should be OK.
I did not know that XMLHttp interactions were asynchronous but serialized. Most browsers can make multiple simultaneous requests so I'd assumed that XMLHttp would just grab a connection and go and that multiple requests can be in progress simultaneously.
After going and reading the docs for at least one implementation of XMLHttpRequest, I think you're right. There are some asynchronous things going on here. So I take it back. ;-)
Well, that's good. Does that mean I'm off the hook? ;-)
Not really off the hook as issues still remain and you are a valuable resource. :-) I need to spend some time thinking about the issues an do some cautious experimentation and testing.
Heh. - C
participants (6)
-
Chris McDonough -
Chris Withers -
Dennis Allison -
Maciej Wisniowski -
Michael Dunstan -
Tino Wildenhain