[ZODB-Dev] question about connections

[Tim Peters]

[Victor Safronovich]
> Hello zodb-dev!
>
> I have Zope 2.6.1 with ZODB 3.1.5 ( I know this is old versions ).

Too old ;-)  ZODB 3.1.5 was released in February of 2004, was the last
release in the 3.1 line, and it would be a real pain to try to identify the
critical bugs fixed in later ZODBs that remained broken in the 3.1 line.

> I have scheduler in separate thread with code
>
> class EventDispatcher( threading.Thread ):
>     running = False
>     def __init__( self, scheduler ):
>         self.scheduler_path = scheduler.getPhysicalPath()
>         threading.Thread.__init__( self, name='Scheduler_%s' %
scheduler.getSchedulerName() )
>
>     def run( self ):
>         """
>             Start as main schedule thread.
>         """
>         LOG( Config.ProductName,
>              TRACE,
>              'Started dispatcher thread for scheduler %s' % '/'.join(
self.scheduler_path ),
>            )
>
>         app = Zope.bobo_application()
>         try:
>             self.running = True
>             while self.running:
>                 get_transaction().begin()
>                 try:
>                     scheduler = app.unrestrictedTraverse(
self.scheduler_path )
>                     scheduler.dispatchEvents()
>                     get_transaction().commit()
>                 except Exception:
>                     get_transaction().abort()
>                 threading.Event().wait( random() )

Heh.  Is there a secret agenda here, or is that line an elaborate way to
spell "time.sleep(random())"?


>         finally:
>             app._p_jar.close()
>
>             LOG( Config.ProductName,
>                  TRACE,
>                  'Terminating dispatcher thread for scheduler %s' %
'/'.join( self.scheduler_path ),
>                 )
>
>     def terminate( self ):
>         self.running = False
>         threading.Event().wait( random() )
>
> But connection to the ZODB in the app object some times does`t see
> modifications from the others connections :(.

Sorry, offhand I don't see "a good reason" for that either.  One
possibility:  if, over a stretch of time, no _modifications_ are made to any
persistent objects by a thread, then the Connection used by the thread never
registers with the transaction manager, and then the transaction manager
doesn't tell the Connection to process invalidations at transaction
boundaries (the transaction manager literally has no idea that a Connection
exists unless and until an object loaded from that Connection gets
modified).

All ZODBs before 3.3 work that way.  As an extreme example, if one of your
threads never modifies a persistent object, it will never see changes made
by other threads.

If that's the case, you can worm around it by explicitly calling the
Connection's .sync() method.  For example, call app._p_jar.sync() instead of
get_transaction().begin().  Give it a try -- it might fix it.

This shouldn't be needed at or after ZODB 3.3.

> This repaired in this way
>     def run( self ):
>         """
>             Start as main schedule thread.
>         """
>         LOG( Config.ProductName,
>              TRACE,
>              'Started dispatcher thread for scheduler %s' % '/'.join(
self.scheduler_path ),
>            )
>
>         self.running = True
>         while self.running:
>             app = Zope.bobo_application()

This is the only material change, right (you moved this line into the loop)?


>             try:
>                 get_transaction().begin()
>                 try:
>                     scheduler = app.unrestrictedTraverse(
self.scheduler_path )
>                     scheduler.dispatchEvents()
>                     get_transaction().commit()
>                 except Exception:
>                     get_transaction().abort()
>                 threading.Event().wait( random() )
>             finally:
>                 app._p_jar.close()
>
>         LOG( Config.ProductName,
>               TRACE,
>               'Terminating dispatcher thread for scheduler %s' % '/'.join(
self.scheduler_path ),
>             )
>
> But  this  create  new  ZODB-connection every time, is this bad or good?

It doesn't actually create a new Connection.  When a Connection is closed,
it doesn't really go away.  It's added to a pool of available Connections
maintained by the DB object; it doesn't even lose its memory cache.
DB.open() takes an already-existing Connection from its pool, if its pool
isn't empty.  It's only when that pool is empty (for example, the first time
DB.open() is called) that it needs to create a new Connection.  So, e.g.,
doing

    cn = db.open()
    cn.close()

in a loop is inexpensive (assuming nobody else is mucking with db, db.open()
there returns the same Connection object every time).

Anyway, one thing (re)open'ing a Connection does is process invalidation
messages, regardless of whether any objects from the Connection have been
modified, so that (re)opening appears to fix your problem is consistent with
the hope that calling Connection.sync() could fix your problem too.

In any cae, re-opening Connections isn't expensive.

> And Why one   connection   in   the   first   example   not  see
> modifications from the other connections?

Not enough info to say for sure; gave my best guess above.

> Is this a ZODB 3.1.5 problem or my?

Same answer ;-)