Threads, Locks, Volatility, and Angst
Greetings. I'm looking for a little wisdom -- specifically, about thread locks and Zope. I'm developing a Python-based product. One of the classes needs to hand out unique IDs to clients as they connect for the first time. I generate these by keeping an index integer in the instance that starts at zero, incrementing it by one for every new client, and using the new value for the unique ID. In order to prevent race conditions when two clients connect almost simultaneously, I have another instance variable in my class that holds a threading.lock(). I call acquire() on that lock just before the increment-and-report code, and release() just after. So far so good. That seemed to work just fine yesterday when I coded it. Then I shut down Zope overnight and started it up again today. Now, I cannot create a new instance of the class without getting the Zope error "UnpickleableError: Cannot pickle <type 'thread.lock'> objects". (I'd swear it was working yesterday, though.) I thought "Okay, no problem. I'll just make the lock variable volatile, since I don't really care whether it persists when Zope shuts down." So I renamed the lock to begin with '_v_'. That solves my "UnpickleableError" problem nicely. Unfortunately, it introduces a different problem: the code that calls acquire() on the lock now throws "AttributeError: 'NoneType' object has no attribute 'acquire'". I'm sure I initialize that variable to a threading.Lock() in the object's __init__ method. So now I'm worried that Zope is doing all kinds of pickling-unpickling activity behind the scenes, and anything I have that's volatile can disappear without warning. Are there any thread-wise Zopistas out there who can steer me right? Thanks, ..Ian -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- Dr. Ian Beatty beatty@physics.umass.edu Physics Education Research Group voice: 413.545.9483 Department of Physics fax: 413.545.4884 Univ. of Massachusetts AIM: (available upon request) Amherst, MA 01003-4525 USA http://umperg.physics.umass.edu/ -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- --
Do you use this unique ID only as some kind of session ID? If you only need these IDs during a session (don't have to store them in ZODB), you can create a thread safe class for managing them and then use an instant of this class as a module global variable in your product. AFAIR, this is the way how exUserFolder implements it's usercache. You can get a clue by examining these files: http://cvs.sourceforge.net/viewcvs.py/*checkout*/exuserfolder/exUserFold er/UserCache/UserCache.py?content-type=text%2Fplain&rev=1.16 http://cvs.sourceforge.net/viewcvs.py/*checkout*/exuserfolder/exUserFold er/exUserFolder.py?content-type=text%2Fplain&rev=1.91 Regards, Sandor
-----Original Message----- From: zope-dev-bounces@zope.org [mailto:zope-dev-bounces@zope.org] On Behalf Of Ian Beatty Sent: Wednesday, March 31, 2004 4:26 PM To: zope-dev@zope.org Subject: [Zope-dev] Threads, Locks, Volatility, and Angst
Greetings.
I'm looking for a little wisdom -- specifically, about thread locks and Zope.
I'm developing a Python-based product. One of the classes needs to hand out unique IDs to clients as they connect for the first time. I generate these by keeping an index integer in the instance that starts at zero, incrementing it by one for every new client, and using the new value for the unique ID. In order to prevent race conditions when two clients connect almost simultaneously, I have another instance variable in my class that holds a threading.lock(). I call acquire() on that lock just before the increment-and-report code, and release() just after. So far so good.
That seemed to work just fine yesterday when I coded it. Then I shut down Zope overnight and started it up again today. Now, I cannot create a new instance of the class without getting the Zope error "UnpickleableError: Cannot pickle <type 'thread.lock'> objects". (I'd swear it was working yesterday, though.)
I thought "Okay, no problem. I'll just make the lock variable volatile, since I don't really care whether it persists when Zope shuts down." So I renamed the lock to begin with '_v_'. That solves my "UnpickleableError" problem nicely. Unfortunately, it introduces a different problem: the code that calls acquire() on the lock now throws "AttributeError: 'NoneType' object has no attribute 'acquire'". I'm sure I initialize that variable to a threading.Lock() in the object's __init__ method. So now I'm worried that Zope is doing all kinds of pickling-unpickling activity behind the scenes, and anything I have that's volatile can disappear without warning.
Are there any thread-wise Zopistas out there who can steer me right?
Thanks,
..Ian
-- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- Dr. Ian Beatty beatty@physics.umass.edu Physics Education Research Group voice: 413.545.9483 Department of Physics fax: 413.545.4884 Univ. of Massachusetts AIM: (available upon request) Amherst, MA 01003-4525 USA http://umperg.physics.umass.edu/ -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- --
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On 2004-03-31 5:42 PM, zope@netchan.cotse.net is reputed to have said:
Do you use this unique ID only as some kind of session ID?
Yes, exactly.
If you only need these IDs during a session (don't have to store them in ZODB), you can create a thread safe class for managing them and then use an instant of this class as a module global variable in your product. AFAIR, this is the way how exUserFolder implements it's usercache. You can get a clue by examining these files...
I'll check them out. Thanks. ..Ian -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- Dr. Ian Beatty beatty@physics.umass.edu Physics Education Research Group voice: 413.545.9483 Department of Physics fax: 413.545.4884 Univ. of Massachusetts AIM: (available upon request) Amherst, MA 01003-4525 USA http://umperg.physics.umass.edu/ -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- -- --- --
Ian Beatty wrote:
On 2004-03-31 5:42 PM, zope@netchan.cotse.net is reputed to have said:
Do you use this unique ID only as some kind of session ID?
Yes, exactly.
If you only need these IDs during a session (don't have to store them in ZODB), you can create a thread safe class for managing them and then use an instant of this class as a module global variable in your product. AFAIR, this is the way how exUserFolder implements it's usercache. You can get a clue by examining these files...
I'll check them out. Thanks.
The root folder of your Zope probably already has an object called 'browser_id_manager' whose job it is to generate such IDs cheaply; I would recommend using it, instead. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
On Thu, 2004-04-01 at 00:25, Ian Beatty wrote:
I thought "Okay, no problem. I'll just make the lock variable volatile, since I don't really care whether it persists when Zope shuts down." So I renamed the lock to begin with '_v_'. That solves my "UnpickleableError" problem nicely. Unfortunately, it introduces a different problem: the code that calls acquire() on the lock now throws "AttributeError: 'NoneType' object has no attribute 'acquire'". I'm sure I initialize that variable to a threading.Lock() in the object's __init__ method. So now I'm worried that Zope is doing all kinds of pickling-unpickling activity behind the scenes, and anything I have that's volatile can disappear without warning.
I'm not sure about this right now, but iirc _v_ attributes are not stored until "zope shutdown" but until the object is reloaded from the ZODB (for whatever reason it went out of the cache). This basically means no availability longer than until the end of one request can be guaranteed. Christian -- Christian Theune, gocept gmbh & co. kg http://www.gocept.com - ct@gocept.com fon: 03496 3099112 fax: 03496 3099118 mobile: 0179 7808366
Christian Theune wrote:
On Thu, 2004-04-01 at 00:25, Ian Beatty wrote:
I thought "Okay, no problem. I'll just make the lock variable volatile, since I don't really care whether it persists when Zope shuts down." So I renamed the lock to begin with '_v_'. That solves my "UnpickleableError" problem nicely. Unfortunately, it introduces a different problem: the code that calls acquire() on the lock now throws "AttributeError: 'NoneType' object has no attribute 'acquire'". I'm sure I initialize that variable to a threading.Lock() in the object's __init__ method. So now I'm worried that Zope is doing all kinds of pickling-unpickling activity behind the scenes, and anything I have that's volatile can disappear without warning.
I'm not sure about this right now, but iirc _v_ attributes are not stored until "zope shutdown" but until the object is reloaded from the ZODB (for whatever reason it went out of the cache). This basically means no availability longer than until the end of one request can be guaranteed.
'_v_' variablews are not stored *at all* in the ZODB; they are only useful for caching an expensive computed value. In fact, the ZODB does not even guarantee that volatile won't go away during a single request, although objects are not normally ghosted until the end of a transaction. Any use of volatiles needs careful thought. An example:; class Bar: def __init__( self ): """ do something really expensive here. """ class Foo( Persistent ): _v_bar = None def _getBar( self ): result = self._v_bar if result is None: result = self._v_bar = Bar() return result # Always use self._getBar() instead of self._v_bar..... I don't think volatility is a good match for locks, which are tasked with ensuring consistency. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
participants (4)
-
Christian Theune -
Ian Beatty -
Tres Seaver -
zope＠netchan.cotse.net