[Zope-DB] Zope database connectivity

[Dieter Maurer]

Maciej Wisniowski wrote at 2006-9-27 22:40 +0200:
> ...
>>   It can lead to nasty, apparently non-deterministic effects.
>>   You may lose your database connection mid transaction
>>   and part of what you think is a transaction (i.e. atomical) can be lost.
>>  
>>
>What are the circumstances under which this can occur? I've found something
>similiar (?) when I was dealing with stored procedures in DCOracle2:

Several events must occur together such that this happens (that's
why the problem is not yet fixed).

  While the ZODB does not promise any definite lifetime for
  "_v_" variables (they may disappear at any time),
  the current implementation calls cache garbage collection
  only at savepoint and transaction boundaries.
  During cache garbage collection objects may be deactivated
  which causes their content (including "_v_" variables) to be deleted.
  While the ZODB provides other means to deactivate
  objects than cache garbage collection, there are rarely used
  (especially on a DA object).
  Thus, one condition is that savepoints (formerly "subtransaction"s)
  are used.
  However, some Zope operations, e.g. large cataloguing operations
  use savepoints (in order to reduce memory footprint by
  early cache garbage collection), without you beeing aware of.

  There is not problem when the DA is deactivated when it was
  not used in the same transaction. If however, it is use in this
  transaction, then it is used rather recently. Cache garbage collection
  first deactivates the least recently used objects.
  A second condition is therefore, that your transaction accesses
  huge numbers of objects (more than the cache size) since it accessed
  the DA and then it performs a savepoint.

  If these conditions are met, you will lose your connection. Still
  nothing bad happens as the transaction still holds a reference to it
  and may eventually commit/abort it at transaction end time.

  If however, the same request accesses the DA again, a new connection
  will be opened. This new connection will not see the effects
  performed via the former connection and may deadlock on
  access to resources locked by the former connection.
  In these rare cases, you will have problems.
  And as they are very rare and apparently non-deterministic,
  they are almost impossible to analyse (and quite difficult to
  understand).
  

>When you have a request that causes the new _v_database_connection to be
>created and this new connection is used to execute stored procedure which
>causes exception. Then... transaction is aborted and resource
>managers' abort() functions are called.
>I've found that newly created _v_database_connection was destroyed
>before(!) ResourceManager's (used to handle relational db connection)
>abort() function was called, so there was no rollback to database... huh...

This might be possible but need not be a problem:

  The order in which operations occur at commit time can be controlled
  via the resource manager's "sortKey" (or similarly spelled).

  As the sort order between ZODB resources and relational
  resources is not really relevant, it may well be
  that the ZODB connection is aborted before the relational connection.

  In this case, you may see the loss of the "_v_" variable
  (caused either by the cache garbage collection or the invalidation
  of the DA object) before the relational "abort".

  However, the transaction holds an additional (beside the "_v_" attribute)
  reference to the connection and can commit/abort it despite
  the loss of the "_v_" attribute.



-- 
Dieter