Sharing global data between threads / locking a method
I have a synkronisation script that I run every 10 minutes via wget from a cron job. Sometimes the script runs longer than 10 minutes. In that case I would like to return a page to wget, but not run the actual script. So in a external method/module I have a function like this: BUSY_STATE = 0 def sync_in_progress(busy=None): global BUSY_STATE if busy is None: return BUSY_STATE else: BUSY_STATE = busy The idea is that BUSY_STATE is a global value shared between all threads, and if the sync_test() function below is called while it is allready running in another thread, it will just return 'Sync allready in progress'. I then wrap the actual sync code like this: def long_test_function(): # just to kill time t = Timer('Loop time') for i in xrange(10**7): d = 7*8 e = 7*8 print t.time() return d def sync_test(self): if not sync_in_progress(): sync_in_progress(1) long_test_function() # placeholder for the real sync code sync_in_progress(0) return 'Sync done' else: return 'Sync allready in progress' But it seems that the new method merely waits until the first one has completed. So it allways returns 'Sync done' and calls long_test_function() What am I misunderstanding here? Isn't it the right way to share global data? Or is there some kind of locking going on under my nose that I am to blind to see. -- hilsen/regards Max M, Denmark http://www.mxm.dk/ IT's Mad Science
I expect that your 'global BUSY_STATE' statement is only creating a global variable within the context of a 'program execution'. In general terms, a global variable is only accessible to routines within a single running program. Two programs, running simultaneously, will each have their own versions of the global variable (even if the global variable has the same name). If this were not the case then any two programs running at the same time (on the same computer), with matching variable names, would change the contents of the variables in both programs -> resulting in unintended/uncontrollable program behaviour! If you have two copies of your external method running (as a result of overlapping wget calls) each external method will run as a stand-alone program and will not know about any other external methods that are running. A possible solution: create a property field on the folder where the external methods are stored. Have your external method update this property field when the external method starts and again when it exits. This way you can test whether or not the external method is currently in operation. You could store this property field on a temp_folder for faster performance. hth Jonathan ----- Original Message ----- From: "Max M" <maxm@mxm.dk> To: <zope@zope.org> Sent: Monday, June 27, 2005 9:53 AM Subject: [Zope] Sharing global data between threads / locking a method
I have a synkronisation script that I run every 10 minutes via wget from a cron job.
Sometimes the script runs longer than 10 minutes.
In that case I would like to return a page to wget, but not run the actual script.
So in a external method/module I have a function like this:
BUSY_STATE = 0 def sync_in_progress(busy=None): global BUSY_STATE if busy is None: return BUSY_STATE else: BUSY_STATE = busy
The idea is that BUSY_STATE is a global value shared between all threads, and if the sync_test() function below is called while it is allready running in another thread, it will just return 'Sync allready in progress'.
I then wrap the actual sync code like this:
def long_test_function(): # just to kill time t = Timer('Loop time') for i in xrange(10**7): d = 7*8 e = 7*8 print t.time() return d
def sync_test(self): if not sync_in_progress(): sync_in_progress(1) long_test_function() # placeholder for the real sync code sync_in_progress(0) return 'Sync done' else: return 'Sync allready in progress'
But it seems that the new method merely waits until the first one has completed.
So it allways returns 'Sync done' and calls long_test_function()
What am I misunderstanding here? Isn't it the right way to share global data? Or is there some kind of locking going on under my nose that I am to blind to see.
--
hilsen/regards Max M, Denmark
http://www.mxm.dk/ IT's Mad Science
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Jonathan wrote:
A possible solution: create a property field on the folder where the external methods are stored. Have your external method update this property field when the external method starts and again when it exits. This way you can test whether or not the external method is currently in operation. You could store this property field on a temp_folder for faster performance.
Not sure if this will work. If I understand correctly how does Zope/ZODB transaction machinery works, if you set a property on some persistent object the change will be visible only after the transaction has been commited, wich usually happens at the end of the request, so other threads/requests will always get a "long_process_finished" value for the property. Regarding Max M question, a simple solution could be to use a file (filesystem, not zope) as lock. HTH -- //// (@ @) ----------------------------oOO----(_)----OOo-------------------------- <> Ojo por ojo y el mundo acabara ciego /\ Alexis Roda - Universitat Rovira i Virgili - Reus, Tarragona (Spain) -----------------------------------------------------------------------
Max M wrote at 2005-6-27 15:53 +0200:
... So in a external method/module I have a function like this:
BUSY_STATE = 0 def sync_in_progress(busy=None): global BUSY_STATE if busy is None: return BUSY_STATE else: BUSY_STATE = busy
Note that this is likely to fail. The module containing an External Method is maintained in the ZODB cache. As a consequence, each worker gets its own copy and you cannot synchronize via global variables of such modules. Use a true Python module (note that Zope does not import the source file of an External Method; therefore, it is not inside a module in the Python sense) when you need synchronization via global module level variables. -- Dieter
Dieter Maurer wrote:
Max M wrote at 2005-6-27 15:53 +0200:
... So in a external method/module I have a function like this:
BUSY_STATE = 0 def sync_in_progress(busy=None): global BUSY_STATE if busy is None: return BUSY_STATE else: BUSY_STATE = busy
Note that this is likely to fail.
The module containing an External Method is maintained in the ZODB cache. As a consequence, each worker gets its own copy and you cannot synchronize via global variables of such modules.
Use a true Python module (note that Zope does not import the source file of an External Method; therefore, it is not inside a module in the Python sense) when you need synchronization via global module level variables.
I wrote a small tool and ended up with this: BUSY_STATE = 0 def sync_in_progress(busy=None): global BUSY_STATE print '----------------' print 'BUSY_STATE:', BUSY_STATE, 'busy:', busy print '' if not busy is None: BUSY_STATE = busy return BUSY_STATE class Syncer(UniqueObject, PropertyManager, SimpleItem.SimpleItem, ActionProviderBase): def redirect(self, url): self.REQUEST.RESPONSE.redirect(url) def reset(self): "reset" return repr(sync_in_progress(0)) def sync(self): "Syncs" print '##################' print 'sync start' if not sync_in_progress(): sync_in_progress(1) self.redirect('%s/sync_action' % self.absolute_url()) else: return 'SYNC: in progress' def sync_action(self): "sync_action" # do stuff sync_in_progress(1) self.sync_calendar() self.sync_email() sync_in_progress(0) return 'SYNC: done' It doesn't seem to work without the redirect. Which is the reason for having both a sync and a sync_action method. -- hilsen/regards Max M, Denmark http://www.mxm.dk/ IT's Mad Science
participants (4)
-
Dieter Maurer -
Jonathan -
Max M -
SER.RI-TIC-Alexis Roda