Hi all, I've a product, http://zetadb.sourceforge.net, that uses SESSION extensively. I've just discover, unhappiness, that there are some problems when SESSION object is changed at the same time in two diferent python scripts (using frames). The two frames are changing diferent keys of the SESSION object, but at the end the whole SESSION object is saved, I think, so one of the two objects losses its SESSION data Anybody knows any workarround about this? Thanks in advance Santi Camps
Santi Camps wrote at 2003-11-24 23:14 +0100:
I've a product, http://zetadb.sourceforge.net, that uses SESSION extensively. I've just discover, unhappiness, that there are some problems when SESSION object is changed at the same time in two diferent python scripts (using frames). The two frames are changing diferent keys of the SESSION object, but at the end the whole SESSION object is saved, I think, so one of the two objects losses its SESSION data
It should *NOT* loose its session data but get a "ConflictError" and automatically retry the request. That's normal ZODB behaviour. Search for "Application specific conflict resolution" for a (complex, in your case) way to work around this. A partial workaround may be to put your various keys into persistent subobjects. -- Dieter
Santi Camps wrote at 2003-11-24 23:14 +0100:
I've a product, http://zetadb.sourceforge.net, that uses SESSION extensively. I've just discover, unhappiness, that there are some problems when SESSION object is changed at the same time in two diferent python scripts (using frames). The two frames are changing diferent keys of the SESSION object, but at the end the whole SESSION object is saved, I think, so one of the two objects losses its SESSION data
It should *NOT* loose its session data but get a "ConflictError" and automatically retry the request.
That's normal ZODB behaviour.
Search for "Application specific conflict resolution" for a (complex, in your case) way to work around this.
A partial workaround may be to put your various keys into persistent subobjects.
I'm trying to understand how Conflict Resolution works. The first step will be change my DirectoryStorage for a FileStorage and try again. As you say, I seems that Zope retries the request 3 times when a ConflictError is raised, so it should work in my case. If not, I will try to debug and see how not, or perhaps disabling low conflict flag. If, at the end, it countinues crashing, I will store my data in persistent subobjects. Thanks a lot Santi Camps
santi@zetadb.com wrote at 2003-11-26 11:54 +0100:
... I'm trying to understand how Conflict Resolution works. The first step will be change my DirectoryStorage for a FileStorage and try again.
This will have no influence to your session problems. Sessions are (by default) put into a RAM based storage independent from your main storage ("DirectoryStorage" or "FileStorage").
As you say, I seems that Zope retries the request 3 times when a ConflictError is raised, so it should work in my case.
Indeed.
If not, I will try to debug and see how not, or perhaps disabling low conflict flag.
It is unlikely that the the low conflict connection causes your problem. "Low conflict" affects the following situation: You try to load an object (because it is not in your ZODB cache) and get informed that is has been modified (by another commited transaction) since your transaction started. With a normal ZODB connection, you will get a "ReadConflictError" as adding the object to your ZODB cache would make it inconsistent: some objects would have the state from your transactions start and others from a later date. With a low conflict connection, the inconsistency is accepted and no "ReadConflictError" is raised. As you described, you loose the session object. This is not easily understood as the effect of a cache inconsistency. -- Dieter
It is unlikely that the the low conflict connection causes your problem. "Low conflict" affects the following situation:
You try to load an object (because it is not in your ZODB cache) and get informed that is has been modified (by another commited transaction) since your transaction started.
With a normal ZODB connection, you will get a "ReadConflictError" as adding the object to your ZODB cache would make it inconsistent: some objects would have the state from your transactions start and others from a later date. With a low conflict connection, the inconsistency is accepted and no "ReadConflictError" is raised.
As you described, you loose the session object. This is not easily understood as the effect of a cache inconsistency.
Sure. Disabling Low conflict hasn't solved anything. I'm not able to find where is the problem. I have not enought knowledge of Zope internals. I will try to store my data in persisten subobjects, as you said. One thing is true: Changing session object simultaniously in various frames doesn't work fine. Sometimes session data is not the espected becouse another frame has erased your changes in the session object. I think this is a serious inconvenient in front of other platforms.
From my point of view, it would be better to have SESSION out of the transaction but with changes stored inmediatly. You don't?
Perhaps some day I will be able to implement this for myself. Until that, do you think I should enter a bug, or this cann't be considered as a bug? Thanks for your help Santi Camps
I'm not able to find where is the problem. I have not enought knowledge of Zope internals. I will try to store my data in persisten subobjects, as you said.
Finally I've found a better and easy partial workarround. I will explain it here so other users in similar situation can know. It consist in to have multiple Transitient Object Containers and multiple Session Data Managers. For instance, in my case: Folder1/some_scripts Folder1/session_data (a Transitient Object Container) Folder1/session_data_manager (a Session Data Manager using previous object and referenced in REQUEST as SESSION_FOLDER1) Folder2/some_scripts Folder2/session_data (a Transitient Object Container) Folder2/session_data_manager (a Session Data Manager using previous object and referenced in REQUEST as SESSION_FOLDER2) Then, scripts in Folder1 can use SESSION_FOLDER1 to store its session data, scripts in Folder2 can use SESSION_FOLDER2 to do the same. To use SESSSION_FOLDER1 from scripts in Folder2, a simple traversal will work fine. Using this solution, my simultanious frames are working using session with no problems. Cheers Santi Camps http://zetadb.sourceforge.net
Santi Camps wrote at 2003-11-27 17:41 +0100:
... One thing is true: Changing session object simultaniously in various frames doesn't work fine. Sometimes session data is not the espected becouse another frame has erased your changes in the session object.
I think this is a serious inconvenient in front of other platforms.
This is a general problem of concurrently updating the same object. Zope's "Session" mechanism is not that bad in this respect (it is in some way better than that for other plattforms). When it works as advertised (I see it working this way; you do not), then concurrent modification will result in a "ConflictError" with automatic request retrial. This provides for consistency despite concurrent updates. A *BIG* problem is the large granularity of a Zope session. It is essentially one big monolitic object (with respect to concurrent update). This makes "ConflictError" much more likely. I have never seen disappearing sessions -- but I must admit that we use our own session implementation - deriving from Zope's session but getting rid of the over-complex session management code. This code caused many "inconsistency" errors for the session management data structures; which is a bad sign. The complexity comes from the fact that it tries to scale well for large numbers of sessions. Our simplified code scales not that well but can easily cope with several thousand (locally maintained) sessions we sometimes see.
From my point of view, it would be better to have SESSION out of the transaction but with changes stored inmediatly. You don't?
No: you would face the general inconsistency problem caused by concurrent updates of an object. Analysing and fixing such problems is a nightmare (they occur non-deterministically, they are extremely difficult to reproduce and analyse).
Perhaps some day I will be able to implement this for myself.
No need. There is a product called "ZSession". It maintains session data in RAM and (I think) allows threads to read/write sessions asynchronously. It is probably no longer maintained, now that Zope comes with Sessions in the core distribution. But that need not be a big problem. Should it break in some future Zope version (which may be possible), the fixes should be quite easy and straight forward.
Until that, do you think I should enter a bug, or this cann't be considered as a bug?
When you see session objects disappearing (and they have not expired), then this is a bug! As said: The advertised behaviour when several requests modify the same session object is that all but the first committing request get a "ConflictError" and are retried. Everything else is a bug. Another note: it is known (though we do not yet fully understand why) that sessions behave non-deterministically (disappear, revert to old values) when they are read/written in "standard_error_message" (Zope's error handling hook for failing requests). "standard_error_message" is called after the transaction has been aborted -- thus any (transactional) effect from the request has already been wiped out. Accessing "Session" there may read old values or create a new (empty) session. Apparently, and this is what we do not yet understand, such phony sessions seem to be able to spill over into later requests. One hypothesis was that it were caused by Zope forgetting to abort/commit the transaction implicitly created for "standard_error_message" execution (this definitely is a bug!). But the transaction *is* aborted later, before it is being used in another request -- maybe this happens then in the wrong context? I will not see problems caused by this "failing to commit/abort error transaction" bug because I fixed our Zope as soon as the bug became known. I am not sure whether the bug is fixed in Zope 2.6.2. That's a disadvantage of maintaining our own Zope version - I do not know and can not easily check the distributed code any longer. -- Dieter
I have never seen disappearing sessions -- but I must admit that we use our own session implementation - deriving from Zope's session but getting rid of the over-complex session management code.
I've not explained very well. Session object doesn't disappears, just some data in session object does it.
(they occur non-deterministically, they are extremely difficult to reproduce and analyse).
As said: The advertised behaviour when several requests modify the same session object is that all but the first committing request get a "ConflictError" and are retried. Everything else is a bug.
I'm agree that, if it works fine, the retrieing request method is a good solution. My problem is quasi deterministic, it crashes most of times :-( I will try to provide a example.zexp when entering the bug so it could be easier to reproduce.
Another note: it is known (though we do not yet fully understand why) that sessions behave non-deterministically (disappear, revert to old values) when they are read/written in "standard_error_message" (Zope's error handling hook for failing requests).
I'm not using SESSION in standard_error_message, but symptoms are very similar. Santi Camps
Hi Santi,
I've a product, http://zetadb.sourceforge.net, that uses SESSION extensively. I've just discover, unhappiness, that there are some problems when SESSION object is changed at the same time in two diferent python scripts (using frames). The two frames are changing diferent keys of the SESSION object, but at the end the whole SESSION object is saved, I think, so one of the two objects losses its SESSION data
Anybody knows any workarround about this?
I guess this happens because the SESSION is stored with a low conflict resolution in the TempFolder. You should be able to switch the conflict resolution in the temp folder to something more strict, which may increase the number conflict errors a lot. There have been a thread which may tell You how to do this on this list in March(?) of this year; seaching the mailing list archive for "LowConflictConnection" or something similar may help. Cheers, Clemens
participants (4)
-
Clemens Robbenhaar -
Dieter Maurer -
Santi Camps -
santi＠zetadb.com