[Zope-Checkins] CVS: Zope/lib/python/Products/Sessions/stresstests - stresstestMultiThread.py:1.3

[Chris McDonough]

Update of /cvs-repository/Zope/lib/python/Products/Sessions/stresstests
In directory cvs.zope.org:/tmp/cvs-serv20290/stresstests

Modified Files:
	stresstestMultiThread.py 
Log Message:
New TransientObjectContainer implementation.

Changes:

 - More stable under high usage, especially in the face of
   situations under which there are many ZODB conflict
   errors. The previous implementation had stability problems
   when many conflict errors were encountered; especially
   conflicts that were generated as a result of a simultaneous
   change to a subobject of the TOC (such as in the case of a Zope
   application which makes heavy use of both frames and
   sessions).
   
 - More conflict-resistant.  Instead of ignoring the likelihood
   that multiple threads will attempt to perform the same actions
   simultaneously in methods of the TOC (which often causes
   conflicts), the new implementation attempts to avoid conflicts
   by employing a chance-based housekeeping model.  In this model,
   one thread is "elected" by chance to do the kinds of tasks that
   cause the most conflicts.

 - Now uses a "timeslice" based model instead of a "ring" based
   model.  This also helps cut down on conflicts and makes
   the code slighly less obfuscated (not much, though! ;-)

 - Quite a few more comments in the code.

 - Changes to the sessioning stresstest (which exposed the
   bug that made me reimplement the TOC in the first place).

 - Updates to unit tests.

 - A "HowTransienceWorks.stx" document which attempts to
   explain how the code works.  It's not stellar, but
   it's a start.

 - Changes to the TransientObject class that the TOC
   hands out (typically as a "session data object"), in order
   to make invalidation less Rube-Goldberg-ish.

The structure of the TOC object has changed enough that in order to
maintain b/w compatibility, an in-place upgrade of "old" instances
is implied by running them with this code.   "Upgraded" instances
are not backwards-incompatible, however, so folks can hopefully
move back and forth between Zope versions without much hassle.




=== Zope/lib/python/Products/Sessions/stresstests/stresstestMultiThread.py 1.2 => 1.3 ===
     TransientObjectContainer, TransientObject
 from Products.TemporaryFolder.TemporaryFolder import MountedTemporaryFolder
-from ZODB.POSException import InvalidObjectReference, ConflictError
+from Products.TemporaryFolder.LowConflictConnection import LowConflictConnection
+from ZODB.Connection import Connection
+from ZODB.POSException import InvalidObjectReference, ConflictError, ReadConflictError, BTreesConflictError
 from DateTime import DateTime
 from unittest import TestCase, TestSuite, TextTestRunner, makeSuite
 import time, threading, random
@@ -32,6 +34,7 @@
 from ZODB.DemoStorage import DemoStorage
 from OFS.Application import Application
 import sys
+from zLOG import log_time
 sys.setcheckinterval(200)
 
 tf_name = 'temp_folder'
@@ -65,7 +68,7 @@
     bidmgr = BrowserIdManager(idmgr_name)
     tf = MountedTemporaryFolder(tf_name, title="Temporary Folder")
     toc = TransientObjectContainer(toc_name, title='Temporary '
-        'Transient Object Container', timeout_mins=20)
+        'Transient Object Container', timeout_mins=1)
     session_data_manager=SessionDataManager(id='session_data_manager',
         path='/'+tf_name+'/'+toc_name, title='Session Data Manager')
 
@@ -89,7 +92,7 @@
     get_transaction().commit()
 
 class TestMultiThread(TestCase):
-    def testNonOverlappingBrowserIds(self):
+    def testOverlappingBrowserIds(self):
         readers = []
         writers = []
         readiters = 20
@@ -115,10 +118,7 @@
         while threading.activeCount() > 1:
             time.sleep(1)
         
-        for thread in readers:
-            assert thread.out == [], thread.out
-
-    def testOverlappingBrowserIds(self):
+    def testNonOverlappingBrowserIds(self):
         readers = []
         writers = []
         readiters = 20
@@ -144,9 +144,6 @@
         while threading.activeCount() > 1:
             time.sleep(1)
         
-        for thread in readers:
-            assert thread.out == [], thread.out
-
 class BaseReaderWriter(threading.Thread):
     def __init__(self, db, iters, sdm_name):
         self.conn = db.open()
@@ -166,13 +163,23 @@
                 try:
                     self.run1()
                     return
+                except ReadConflictError:
+                    print "read conflict"
+                except BTreesConflictError:
+                    print "btrees conflict"
                 except ConflictError:
-                    i = i + 1
-                    #print "conflict %d" % i
-                    if i > 3: raise
+                    print "general conflict"
+                except:
+                    get_transaction().abort()
+                    print log_time()
+                    raise
+                i = i + 1
+                get_transaction().abort()
+                time.sleep(random.randrange(5) * .1)
         finally:
             self.conn.close()
             del self.app
+            print i
             
 class ReaderThread(BaseReaderWriter):
     def run1(self):