[ZODB-Dev] RE:[Zope-dev] Thread safety in Zope 2 (was: [Zope-dev] zope 2 and non-multithread safe extensions) extensions)

[Norfleet, Sheppard S.]

Jim,

I know this is an old thread, but I have related question...

I am making a application level replication product for my company, and the
product spawns a daemon thread to look for incoming transactions.   I have
gotten rid of all my database conflict errors and tested concurrency locks
pretty rigorously.  The daemon thread opens its own database connection and
creates a new context for the product, and any change to the product is
fully bracketed by a python RLock.  

I say its thread safe, but actually after a period of time it stops being
so.  Since the original presentation thread has joined, I am wondering that
when another presention thread is created that references the product, that
Zope is creating a whole new RLock.  If this is the case, how would I go
about creating a singleton RLock object?

P.S. Zope really should have a services thread that it offers up to
products.  I saw ThreadedASync but it looks like its not the answer. Sigh.
Maybe I should write a proposal for one... 

Regards,

Shep


********************************************************************
Anthony Baxter wrote:
> 
> How should a Product that's not thread-safe indicate this to Zope 2?

It doesn't.  It must be thread safe.
 
> Say, for the sake of argument, that I had an (unreleased) SNMP product
> that wasn't thread-safe. How will this play in Zope2?

It will work incorrectly unless the number of application treads
is limited to one (e.g. 'python z2.py -t1').

This topic deserves some more discussion.  Which is the topic of this
note. :)

1. How can you tell if a product is thread safe?

   If a Zope product defines *persistent* objects and 
   those objects don't use any *mutable* gloal or shared variables
   (such as class attributes or default arguments), then the product
   is thread safe! *Pay close attention to this. It could let
   you off the hook!* :)

   A key feature of Zope 2 (ZODB 3) is that (a copy of) a 
   persistent object is never accessed by more than one thread
   (unless the programmer goes way out of thier way).  Each thread
   uses it's own database connection(s).  Each connection has it's
   own copies of persistent objects.  This feature provides a major 
   simplification for application developers.

   If a product *does* make use of mutable shared data, then the product
   must take steps to assure that they are used safely.  Some key examples
   of shared data include:

     - Variables defined in modules (globals),

     - Variables defined in classes,

     - Default arguments to Python functions.

   Note that a mutable global variable can be thread safe without
   special care in some cases.  For example, thanks to the methods
   append and pop, lists can be used as thread-safe stacks and 
   queues without any extra locks, because they are protected by the global
   interpreter lock.

2. What can I do if my product is not thread safe?

   I can think of two choices:

     a. Make it thread safe,

     b. Tell my "customers" that it can only be used in
        Zope applications with a single application thread.

3. How do I achiev thread safety for my objects?

   If they are persistent objects, you don't need to.
   If they are non-persistent objects that are *only*
   referenced by persistent objects, you also don't need to worry.

   This is a big topic, but in many cases, simple approaches
   can be used.

   First, you can assure that only one thread can call operations
   on your objects at one time and that data are accessed only through
   operations.  If your objects are of extension types, then the Python
   global interpreter lock protects operations on them (unless they release
the
   lock).  If your objects are written in Python, you can protect them by
   mixing in the 'Synchronized' class from the 'Sync' module.

   The Synchronized class provides a per-instance lock that gets 
   automatically acquired and released on entry to and exit from 
   methods on your objects.

   Second, you should, if possible, provide a thread-safe API for your
objects.
   In a thread-safe API, the operations are self-contained.  The meaning of 
   an operation depends on the history of operation calls.  An example 
   of a non-thread-safe API is the regex API.  Regex objects have a 
   'group' method whose results depend on results of previous 'search' and
   'match' calls.  

   If you can't change the API for an object to make it thread safe, then
you
   can often create a wrapper for the object that is thread safe (see for
example, 
   lib/python/ts_regex.py) or make sure that you only make thread safe
calls.
   In many places in Zope, we make sure we only make thread safe calls on
regex objects
   by storing references to the thread-safe methods rather than to compiled
   regular expressions themselves.

   If the approach sketched above doesn't work for you, then you need to
   put on your thinking cap and get up to speed on thread programming.

Jim
************************************************************************