CST 0.9 Problems under load?
Zopers and Chris, I have written a little zope app that allows people to search an oracle database thru-the-web. Relevant details: Zope 2.3.3 Win NT DC Oracle 1 CST 0.9 - used in just about every request as it provided shopping basket functionality Session timeout set to 20 mins Using ZServer (i.e. no apache/squid in front) P2 450MHz Can't remember how much RAM, but we didn't use it all. [From Analog] Day: reqs: pages: Wed: 133586: 2458: Thu: 48310: 831: Fri: 24890: 438: We launched the site last Wednesday (10am), and the little box it was running on was a bit overwhelmed. We started to see KeyErrors relating to the session data manager (more on that below), so backed off to an earlier search app that had no CST component. CPU had been running at 100% almost solidly for about half an hour, but we still had half our RAM free. At this stage, we weren't logging the errors. After the load died down a bit (the first hour was far and away the worst), we switched back to the CST'ified app. Server load was lower, and although we still weren't logging, we got the impression that the errors had gone. Sadly, the KeyErrors were spotted again. At this point, I setup standard_error_message to email me with the REQUEST and traceback whenever it caught a KeyError. That began at 5pm Wednesday. From then until 1pm Friday, I received 174 error messages of the following form: ----- <snipped REQUEST> Traceback (innermost last): File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171, in publish File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in mapply (Object: results) File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112, in call_object (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line 339, in eval (Object: sdm.getSessionData().has_key('your_results')) (Info: sdm) File <string>, line 0, in ? File C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag er.py, line 264, in getSessionData (Object: sdm) File C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag er.py, line 375, in _createSessionDataObject (Object: sdm) File C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta iner.py, line 191, in set (Object: /accom2/search/sdm) File C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta iner.py, line 344, in __setitem__ (Object: /accom2/search/sdm) KeyError: 64396498A0K2Wz9uuoA ----- As far as I can tell, the only difference between each of the 174 errors is a different key value (corresponding to different sessions I presume). I also received 4 of this sort: ----- Traceback (innermost last): File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171, in publish File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in mapply (Object: results) File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112, in call_object (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line 339, in eval (Object: sdm.getSessionData().has_key('your_results')) (Info: sdm) File <string>, line 0, in ? File C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag er.py, line 258, in getSessionData (Object: sdm) File C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag er.py, line 390, in _getSessionDataObject (Object: sdm) File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 508, in setstate File C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p y, line 156, in load (Object: SessionStorage) KeyError: ----- <No key value given!?> As I said, the relevant REQUEST objects for each of these errors if they can help shed any light on the problems I'm seeing... just ask. In addition, I also managed to salvage this traceback from about 10.15am on Wednesday, when the CST site was getting hammered. I only saw it once, but wasn't logging at that stage... ----- Error Type: TypeError Error Value: function requires exactly 1 argument; 2 given Traceback (innermost last): File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 223, in publish_module File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 187, in publish File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 221, in zpublisher_exception_hook (Object: Traversable) File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 175, in publish File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 235, in commit File C:\PROGRA~1\zope233\lib\python\ZODB\Transaction.py, line 300, in commit File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 377, in commit (Info: (('BTrees.OOBTree', 'OOBTree'), '\000\000\000\000\000\000\000\020', '')) File C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p y, line 182, in store (Object: SessionStorage) File C:\PROGRA~1\zope233\lib\python\ZODB\ConflictResolution.py, line 177, in tryToResolveConflict (Object: SessionStorage) TypeError: (see above) ----- I don't really no how to try to reproduce these errors short of getting a few thousand people to access the site at the same time again. The CST site is still up and in use under much lighter load. I haven't received any of these errors since Friday lunch time, so it really does seem to be a load issue. I don't have much Zen, so to me it looks like the first error is CST not being able to find (losing?) some session information. The second one is a bit of a mystery to me, and the third looks like it might just be a bug in the way the CST code calls ConflictResolution.<some class>.tryToResolveConflict() I'd really appreciate any enlightenment anyone could give about these errors, and how to fix them. cheers tim
Damn. I wish I had an easy answer but I don't. Terry Kerr came up with an important bugfix for CST 0.9 lately, and I'm sure if I dug in to my inbox I could find a few others that folks have submitted. These may or may not solve your particular issues. CST 0.9 seems to have some fundamental problems with supporting load that are proving very difficult to pin down because they are particularly subtle and only show up in situations that are irreplicable. I'm afraid for now that I'm going to have to suggest that you either wait for CST 1.0 (which has not ETA as of now) or try the Zope 2.5 Sessioning machinery. Sorry, - C ----- Original Message ----- From: "Tim Hicks" <tim@sitefusion.co.uk> To: <zope@zope.org> Cc: "Chris McDonough" <chrism@zope.com>; "Paul Browning" <paul.browning@bristol.ac.uk> Sent: Sunday, February 17, 2002 1:28 PM Subject: CST 0.9 Problems under load?
Zopers and Chris,
I have written a little zope app that allows people to search an oracle database thru-the-web. Relevant details:
Zope 2.3.3 Win NT DC Oracle 1 CST 0.9 - used in just about every request as it provided shopping basket functionality Session timeout set to 20 mins Using ZServer (i.e. no apache/squid in front) P2 450MHz Can't remember how much RAM, but we didn't use it all. [From Analog] Day: reqs: pages: Wed: 133586: 2458: Thu: 48310: 831: Fri: 24890: 438:
We launched the site last Wednesday (10am), and the little box it was running on was a bit overwhelmed. We started to see KeyErrors relating to the session data manager (more on that below), so backed off to an earlier search app that had no CST component. CPU had been running at 100% almost solidly for about half an hour, but we still had half our RAM free. At this stage, we weren't logging the errors. After the load died down a bit (the first hour was far and away the worst), we switched back to the CST'ified app. Server load was lower, and although we still weren't logging, we got the impression that the errors had gone. Sadly, the KeyErrors were spotted again. At this point, I setup standard_error_message to email me with the REQUEST and traceback whenever it caught a KeyError. That began at 5pm Wednesday. From then until 1pm Friday, I received 174 error messages of the following form:
----- <snipped REQUEST> Traceback (innermost last): File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171, in publish File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in mapply (Object: results) File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112, in call_object (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line 339, in eval (Object: sdm.getSessionData().has_key('your_results')) (Info: sdm) File <string>, line 0, in ? File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
er.py, line 264, in getSessionData (Object: sdm) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
er.py, line 375, in _createSessionDataObject (Object: sdm) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta
iner.py, line 191, in set (Object: /accom2/search/sdm) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta
iner.py, line 344, in __setitem__ (Object: /accom2/search/sdm) KeyError: 64396498A0K2Wz9uuoA ----- As far as I can tell, the only difference between each of the 174 errors is a different key value (corresponding to different sessions I presume).
I also received 4 of this sort:
----- Traceback (innermost last): File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171, in publish File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in mapply (Object: results) File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112, in call_object (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line 339, in eval (Object: sdm.getSessionData().has_key('your_results')) (Info: sdm) File <string>, line 0, in ? File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
er.py, line 258, in getSessionData (Object: sdm) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
er.py, line 390, in _getSessionDataObject (Object: sdm) File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 508, in setstate File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p
y, line 156, in load (Object: SessionStorage) KeyError: ----- <No key value given!?>
As I said, the relevant REQUEST objects for each of these errors if they can help shed any light on the problems I'm seeing... just ask.
In addition, I also managed to salvage this traceback from about 10.15am on Wednesday, when the CST site was getting hammered. I only saw it once, but wasn't logging at that stage...
----- Error Type: TypeError Error Value: function requires exactly 1 argument; 2 given
Traceback (innermost last): File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 223, in publish_module File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 187, in publish File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 221, in zpublisher_exception_hook (Object: Traversable) File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 175, in publish File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 235, in commit File C:\PROGRA~1\zope233\lib\python\ZODB\Transaction.py, line 300, in commit File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 377, in commit (Info: (('BTrees.OOBTree', 'OOBTree'), '\000\000\000\000\000\000\000\020', '')) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p
y, line 182, in store (Object: SessionStorage) File C:\PROGRA~1\zope233\lib\python\ZODB\ConflictResolution.py, line 177, in tryToResolveConflict (Object: SessionStorage) TypeError: (see above) -----
I don't really no how to try to reproduce these errors short of getting a few thousand people to access the site at the same time again. The CST site is still up and in use under much lighter load. I haven't received any of these errors since Friday lunch time, so it really does seem to be a load issue.
I don't have much Zen, so to me it looks like the first error is CST not being able to find (losing?) some session information. The second one is a bit of a mystery to me, and the third looks like it might just be a bug in the way the CST code calls ConflictResolution.<some class>.tryToResolveConflict()
I'd really appreciate any enlightenment anyone could give about these errors, and how to fix them.
cheers
tim
----- Original Message ----- From: "Chris McDonough" <chrism@zope.com> To: "Tim Hicks" <tim@sitefusion.co.uk>; <zope@zope.org> Cc: "Paul Browning" <paul.browning@bristol.ac.uk> Sent: Sunday, February 17, 2002 10:27 PM Subject: Re: CST 0.9 Problems under load?
Damn. I wish I had an easy answer but I don't.
Damn ;-)
Terry Kerr came up with an important bugfix for CST 0.9 lately, and I'm sure if I dug in to my inbox I could find a few others that folks have submitted. These may or may not solve your particular issues. CST 0.9 seems to have some fundamental problems with supporting load that are proving very difficult to pin down because they are particularly subtle and only show up in situations that are irreplicable.
I did a quick search on the nip list archive, but couldn't find Terry Kerr's patch. Where can I find it? I'd like to give the patches a try. The only problem is, the server probably won't see the same kind of load until this time next year. Do you have any good ways of programmatically loading the server and invoking the session machinery? Is this where 'ab' comes in?
I'm afraid for now that I'm going to have to suggest that you either wait for CST 1.0 (which has not ETA as of now) or try the Zope 2.5 Sessioning machinery.
OK, fair enough. I realise that CST has turned into a bit of a side-project for you now. If I manage to reproduce the same sort of results using the 2.5.0 session machinery, will I stand a better chance of seeing a fix? I'd love to be able to help, but I'm well over my head at the moment. cheers tim
Sorry,
- C
----- Original Message ----- From: "Tim Hicks" <tim@sitefusion.co.uk> To: <zope@zope.org> Cc: "Chris McDonough" <chrism@zope.com>; "Paul Browning" <paul.browning@bristol.ac.uk> Sent: Sunday, February 17, 2002 1:28 PM Subject: CST 0.9 Problems under load?
Zopers and Chris,
I have written a little zope app that allows people to search an oracle database thru-the-web. Relevant details:
Zope 2.3.3 Win NT DC Oracle 1 CST 0.9 - used in just about every request as it provided shopping basket functionality Session timeout set to 20 mins Using ZServer (i.e. no apache/squid in front) P2 450MHz Can't remember how much RAM, but we didn't use it all. [From Analog] Day: reqs: pages: Wed: 133586: 2458: Thu: 48310: 831: Fri: 24890: 438:
We launched the site last Wednesday (10am), and the little box it was running on was a bit overwhelmed. We started to see KeyErrors relating to the session data manager (more on that below), so backed off to an earlier search app that had no CST component. CPU had been running at 100% almost solidly for about half an hour, but we still had half our RAM free. At this stage, we weren't logging the errors. After the load died down a bit (the first hour was far and away the worst), we switched back to the CST'ified app. Server load was lower, and although we still weren't logging, we got the impression that the errors had gone. Sadly, the KeyErrors were spotted again. At this point, I setup standard_error_message to email me with the REQUEST and traceback whenever it caught a KeyError. That began at 5pm Wednesday. From then until 1pm Friday, I received 174 error messages of the following form:
----- <snipped REQUEST> Traceback (innermost last): File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171, in publish File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in mapply (Object: results) File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112, in call_object (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line 339, in eval (Object: sdm.getSessionData().has_key('your_results')) (Info: sdm) File <string>, line 0, in ? File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
er.py, line 264, in getSessionData (Object: sdm) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
er.py, line 375, in _createSessionDataObject (Object: sdm) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta
iner.py, line 191, in set (Object: /accom2/search/sdm) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataConta
iner.py, line 344, in __setitem__ (Object: /accom2/search/sdm) KeyError: 64396498A0K2Wz9uuoA ----- As far as I can tell, the only difference between each of the 174 errors is a different key value (corresponding to different sessions I presume).
I also received 4 of this sort:
----- Traceback (innermost last): File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 171, in publish File C:\PROGRA~1\zope233\lib\python\ZPublisher\mapply.py, line 160, in mapply (Object: results) File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 112, in call_object (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 189, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: results) File C:\PROGRA~1\zope233\lib\python\OFS\DTMLMethod.py, line 182, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_String.py, line 540, in __call__ (Object: standard_html_header) File C:\PROGRA~1\zope233\lib\python\DocumentTemplate\DT_Util.py, line 339, in eval (Object: sdm.getSessionData().has_key('your_results')) (Info: sdm) File <string>, line 0, in ? File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
er.py, line 258, in getSessionData (Object: sdm) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionDataManag
er.py, line 390, in _getSessionDataObject (Object: sdm) File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 508, in setstate File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p
y, line 156, in load (Object: SessionStorage) KeyError: ----- <No key value given!?>
As I said, the relevant REQUEST objects for each of these errors if they can help shed any light on the problems I'm seeing... just ask.
In addition, I also managed to salvage this traceback from about 10.15am on Wednesday, when the CST site was getting hammered. I only saw it once, but wasn't logging at that stage...
----- Error Type: TypeError Error Value: function requires exactly 1 argument; 2 given
Traceback (innermost last): File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 223, in publish_module File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 187, in publish File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 221, in zpublisher_exception_hook (Object: Traversable) File C:\PROGRA~1\zope233\lib\python\ZPublisher\Publish.py, line 175, in publish File C:\PROGRA~1\zope233\lib\python\Zope\__init__.py, line 235, in commit File C:\PROGRA~1\zope233\lib\python\ZODB\Transaction.py, line 300, in commit File C:\PROGRA~1\zope233\lib\python\ZODB\Connection.py, line 377, in commit (Info: (('BTrees.OOBTree', 'OOBTree'), '\000\000\000\000\000\000\000\020', '')) File
C:\PROGRA~1\zope233\lib\python\Products\CoreSessionTracking\SessionStorage.p
y, line 182, in store (Object: SessionStorage) File C:\PROGRA~1\zope233\lib\python\ZODB\ConflictResolution.py, line 177, in tryToResolveConflict (Object: SessionStorage) TypeError: (see above) -----
I don't really no how to try to reproduce these errors short of getting a few thousand people to access the site at the same time again. The CST site is still up and in use under much lighter load. I haven't received any of these errors since Friday lunch time, so it really does seem to be a load issue.
I don't have much Zen, so to me it looks like the first error is CST not being able to find (losing?) some session information. The second one is a bit of a mystery to me, and the third looks like it might just be a bug in the way the CST code calls ConflictResolution.<some class>.tryToResolveConflict()
I'd really appreciate any enlightenment anyone could give about these errors, and how to fix them.
cheers
tim
I did a quick search on the nip list archive, but couldn't find Terry Kerr's patch. Where can I find it? I'd like to give the patches a try. The only problem is, the server probably won't see the same kind of load until this time next year. Do you have any good ways of programmatically loading the server and invoking the session machinery? Is this where 'ab' comes in?
Terry's message to me on the subject is attached. It doesn't sound like the same problem, though. :-( There is a module named "Sessions.stresstests.stresstestMultiThread.py" in the Zope 2.5 sessioning stuff that tries to put the sessioning machinery under load. It might be adapted for CST. If you could create the same problem with "ab", that would be just as valid a test. Anything to recreate the error condition(s).
I'm afraid for now that I'm going to have to suggest that you either wait for CST 1.0 (which has not ETA as of now) or try the Zope 2.5 Sessioning machinery.
OK, fair enough. I realise that CST has turned into a bit of a side-project for you now. If I manage to reproduce the same sort of results using the 2.5.0 session machinery, will I stand a better chance of seeing a fix? I'd love to be able to help, but I'm well over my head at the moment.
If there was a replicable test case that reproduced a specific failure for CST 0.9, I'd almost certainly be able to fix it in a reasonable amount of time. But I'm unable to take action on "it does X under nonspecific conditions" types of bugreports because I can't even begin to estimate how long it would take to guess the conditions under which the error is raised. There's an equal chance of seeing a fix for either CST or Zope 2.5 out of me the moment, which is just about zero for at least the next two weeks. :-( That said, I believe the sessioning stuff in Zope 2.5 is very stable. I want to write a compatibility layer between its API and the CST API and release *that* as "CST 1.0 " (for use in Zope older than 2.5) as soon as I can (if that's even possible). I've realized that I cannot support both in good faith and I need to find a way to merge the two codebases somehow. If I find anything out, I'll send mail to the list. Sorry again! - C Terry's message follows: Chris, The problem I was having with the DB connection locking last week is solved. The patch I explained to you did not fix the problem on our live server, even though it did stop the hang for the example on my workstation, so obviosly the problem was not related. The problem on our live server was with CST 0.9. I noticed many of the following error, but always ignored them and assumed they were harmless as advised by somebody on zope-dev a while agao 2001-11-16T06:46:22 ERROR(200) ZODB Close callback failed for <Products.CoreSessionTracking.SessionDataManager.ConnectionCloser instance at 0x880680c> Traceback (innermost last): File /usr/local/zope/Zope-2.4.3-src/lib/python/ZODB/Connection.py, line 283, in close File /usr/local/zope/Zope-2.4.3-src/lib/python/Products/CoreSessionTracking/Sessi onDataManager.py, line 581, in __call__ KeyError: _v_container I noticed that there were exactly 5 of these between each server hang...so I assume that each one was killing a thread. So I patched CST 0.9 to prevent the exception, and the server has gone from restarting every 20 minutes, to no restarts in 3 days (since I did the patch) *** SessionDataManager.py Fri Oct 5 15:54:29 2001 --- /home/zope/instance/Products/CoreSessionTracking/SessionDataManager.py Fri Feb 1 16:02:10 2002 *************** *** 578,584 **** """ This is called by the 'parent' connection's onCloseCallback handler, so that our 'spawned' connection will be closed when our parent connection is closed. """ ! del self.sdm._v_container if self.conn is not None: self.conn.close() self.conn=None --- 579,586 ---- """ This is called by the 'parent' connection's onCloseCallback handler, so that our 'spawned' connection will be closed when our parent connection is closed. """ ! if hasattr(self.sdm, '_v_container'): ! del self.sdm._v_container if self.conn is not None: self.conn.close() self.conn=None Is there going to be another release of CST, or is session tracking in zope 2.5 it? terry
I did a quick search on the nip list archive, but couldn't find Terry Kerr's patch. Where can I find it? I'd like to give the patches a try. The only problem is, the server probably won't see the same kind of load until this time next year. Do you have any good ways of programmatically loading the server and invoking the session machinery? Is this where 'ab' comes in?
Terry's message to me on the subject is attached. It doesn't sound like the same problem, though. :-(
Agreed.
There is a module named "Sessions.stresstests.stresstestMultiThread.py" in the Zope 2.5 sessioning stuff that tries to put the sessioning machinery under load. It might be adapted for CST.
If you could create the same problem with "ab", that would be just as valid a test. Anything to recreate the error condition(s).
I'm afraid for now that I'm going to have to suggest that you either wait for CST 1.0 (which has not ETA as of now) or try the Zope 2.5 Sessioning machinery.
Have decided to switch to 2.5.x and see what happens. Will also put some time into trying to reliably reproduce the errors with ab and/or stresstestMultiThread.py. Will post again if anything interestings turns up. FWIW, I'm not sure any more that it is a load issue... I just received more keyerror reports at a time when the server is highly unlikely to be under strain.
OK, fair enough. I realise that CST has turned into a bit of a side-project for you now. If I manage to reproduce the same sort of results using the 2.5.0 session machinery, will I stand a better chance of seeing a fix? I'd love to be able to help, but I'm well over my head at the moment.
If there was a replicable test case that reproduced a specific failure for CST 0.9, I'd almost certainly be able to fix it in a reasonable amount of time. But I'm unable to take action on "it does X under nonspecific conditions" types of bugreports because I can't even begin to estimate how long it would take to guess the conditions under which the error is raised.
Understood. Now I've got my homework, I'll make a stab :-)
There's an equal chance of seeing a fix for either CST or Zope 2.5 out of me the moment, which is just about zero for at least the next two weeks. :-( That said, I believe the sessioning stuff in Zope 2.5 is very stable. I want to write a compatibility layer between its API and the CST API and release *that* as "CST 1.0 " (for use in Zope older than 2.5) as soon as I can (if that's even possible). I've realized that I cannot support both in good faith and I need to find a way to merge the two codebases somehow.
That sounds interesting.
If I find anything out, I'll send mail to the list.
Sorry again!
Thanks for your time, tim
participants (2)
-
Chris McDonough -
Tim Hicks