[Zope-dev] DA Roadmap
Christopher Petrilli
petrilli@digicool.com
Thu, 29 Jul 1999 12:33:56 -0400
Gang,
Below is the Database Adapter roadmap that I've just put up on the web site.
It covers both where we are, how we measure where we are, and where we're
going. Some of you might find the section dealing with the future of ODBC
drivers interesting.
Please let me know directly any feedback.
Chris
Roadmap for Zope Database Adapters
Overview
The past few weeks, having brought the Zope 2 release to a beta level,
have brought to the attention of everyone that the current design and
architecture of various DAs is incompatible with use in a fully
threaded environment. This document hopefully will clarify some of
what the various level of compatibility is, as well as comments on
individual DAs, and their future.
DA Thread Safety Levels
In order to quantify in a uniform manner the compatibility of various
DAs with Zope2, we have developed the following "levels" for database
adapters to implement.
Level 1
All database adapters which implement this level of compatibility
must run in a single thread as they have no level of thread safety.
This currently applies to nearly all DAs. This means in ZServer that
Zope runs in a single thread.
Level 2
Level 2 database adapters must be safe to use in a multi-threaded
Zope instance. This can take multiple behavior patterns:
o Internal locking/mutexes for single DA access
o Global locking through the transaction manager if appropriate
While a sub-optimal solution, it should be usable on any application
that has minimal, or infrequent, database access. This means that
all access patterns will be serialized.
In addition, all DAs at this level *must* release the Python global
interpreter lock.
Level 3
A database adapter implementing Level 3 compliance must be fully
safe running in a multi-threaded Zope instance. In addition, it
must support:
o Multiple connections in the same thread
o Multiple connections in different threads running simultaneously
Note that there are instances (see the CLI information below) under
which the DA might implement Level 3, but the underlying client
libraries are somewhat different.
Transaction Manager Issues
Zope has the concept of a Transaction Manager which deals with the
integration of Zope transactions and transactions in an external data
source.
New Semantics
With the introduction of Zope 2, there are new Transaction Manager
semantics, described as below. If you wish to bring a DA into line
with the new specification, these methods are what you must
implement.
o 'self._begin' - This is called at the beginning of every
transaction. Depending on your database, and its implementation
of "implicit transactions", you may or may not have to do
anything here besides a 'pass'.
o 'self._finish' - This is called when a transaction is finished,
or committed.
o 'self._abort' - Obviously this is called when the transaction
involved has been aborted. It should *always* roll back the
transaction in the database.
Integration Requirements
Not all DAs talk to data stores which have transactional semantics
These Database Adapters should not implement integration with the
Transaction Manager unless it makes complete sense. However, all DAs
which deal with data stores which do have a transactional semantics
should integrate completely with the Transaction Manager.
Relational Database Adapters Available
OracleDA (v2.0.0)
The current Oracle DA is based on the 'dcoracle' package, and is not
currently thread safe. It operates at L1 currently, although L2 is
trivial.
*Near term progress.* Before the final release of Zope 2.0, the
Oracle DA will be renamed to ZOracleDA, and will be brought up to
Level 3 compliance.
*Long term progress.* Plans have been made to split the ZOracleDA
into two seperate DAs:
o ZOracle7DA - Compatible with Oracle 7 OCI interfaces.
o ZOracle8DA - Comaptible with Oracle 8 OCI interfaces. This will
also introduce some level of support for LOBs.
*Support availability.* Commercial support is available. Note that
this is support for the DA, Digital Creations will not be selling
general support for 'dcoracle'.
ZSybaseDA (v2.0.0)
The current internal Digital Creations implementation of a Sybase DA
(using the 'ctsybase' Python level interface) is operating at Level
1, with some work having been made at making it Level 2. Currently
it has a Sybase-level memory leak when at Level 2.
*Near term progress.* Before the final release of Zope 2.0, the
ZSybaseDA will be brought up to Level 2 DA. This is being forced by
customer requirements.
*Long term progress.* While we desire to bring the ZSybaseDA up to
full Level 3 compiance exists, it will not happen until customer
requirements have driven it.
*Support availability.* Commercial support is available.
MySQL Database Adapter (v1.1.3)
The current MySQL adapter is at least Level 1 compliant, however, it
does not currently implement the new transaction manager symantics.
*Near term progress.* We would very much like to transfer ownership
of the MySQL DA to someone who has a desire to keep it maintained.
*Long term progress.* None.
*Support availability.* None.
SOLID Database Adapter (v2.0.0)
Due to the recent change in SOLID Technologies marketing direction
and a clarification from their staff that they are no longer
interested in the "stand alone" database market, we have deprecated
support for this module.
*Near term progress.* None.
*Long term progress.* We think if desired, this could be implemented
in the upcoming replacement rather than in a seperate module.
*Support availability.* None.
ODBC Database Adatper (v3.0.3)
The current ODBC DA is not compliant with any level of Zope 2
Database Adapter thread safety. In addition, it is not integrated
with the new transaction amanger symantics. This is a serious
problem.
*Near term progress.* By the final release of Zope 2.0, the ODBC DA
will be modified to be at a minimum Level 2 compliant, based on the
available "thunk" mechanism as well as transaction manager
integration.
*Long term progress.* See information below on the future of a X/Open
CLI interface DA.
*Support availability.* Commercial support is available.
SQL/CLI Database Adapter
In the process of reviewing the desirability of an IBM DB2 database
adapter, it has come to our attention that both ODBC and DB2 are
derived from the same X/Open and ISO specification called the "Call
Level Interface", and are both compatible at that level. Because of
this, we will be transitioning to a new design for the ODBC DA, as well
as integrating support for IBM DB2 into the same DA.
Background on SQL/CLI
The X/Open Company and the SQL Access Group jointly defined a
specification for a callable SQL interface reffered to as the *X/Open
Call Level Interface*. The goal of this interface is to increase the
portability of applications by enabling them to become independent of
any one database vendor's programming interface. Most of the X/Open
Call Level Interface specification has been accepted as part of the
ISO Call Level Interface International Specification (ISO/IEF
9075-3:1995 SQL/CLI).
Microsoft developed a callable SQL interface called Open Database
Connectivity (ODBC) for Microsoft operating systems based on a
preliminary draft of the X/Open CLI. Version 3 of ODBC conforms to
almost all of the ISO SQL/CLI. ODBC 3.0 does contain considerable
functionality that is not part of the International Standard; much of
this is being added to the next draft of that standard.
A Merged DA
This new DA will be called the CLIDAzope :-) Actually, we've not come
up with a name, but it will be based purely on the CLI specification
intially, with small subclasses to deal with ODBC and DB2
individually. This will allow us to leverage two seperate
environments at the same time. Eventually we will integrate with
DB2s excellent (Oracle are you listening? :-) LOB support, and work
to integrate with the support in Microsoft SQLServer 7 wherever
possible.
Relation to UNIX Platforms
Many people have asked about ODBC support on UNIX, and this is the
solution. What this provides is the ability to link directly with an
SQL/CLI compatible library without having to deal with the issues of
an ODBC Driver Manager. In addition, we hope that this will also
reduce the performance impact of the driver manager.
Non-Relational Database Adapters
In addition to integration with relational databases, there are also
various DAs which do not talk to "relational" data stores. These
include:
o LDAP
o Ultraseek
o IMAP
These databases are covered by the same thread-safety levels, but may
or may not be integrated with the transaction manager depending on the
paradigm that they are constructed inside.
Deprecated Database Adapter Issues
As you will note, we have deprecated several database adapters which we
have handled the distribution of, and therefore they are now
orphaned. In order to help people who currently depend on them, we
are offering them to the community to take up the torch of support.
Toward that end we are willing to:
o Transfer code ownership to a new "champion," so long as the license
remains Open Source.
o Provide CVS repository space for the management of the project.
Please let us know if you have the time and desire to take up the
banner of these DAs.
The Kludge of the Century
While we understand that it is desirable to have everything run
beautifiully in a fully threaded environment, it is also unrealistic to
assume that this will all be done within the next 48 hours, and so we
have provided a klduge to get around this issue. This does not escape
the requirement for TM integration, it simply fixes the problem with
threading in DAs that don't support threading yet, but do use the
Transaction Manager.
In order to bring your DA into line, you will need to subclass your
'DB' class as follows::
import Shared.DB.ZRDB.THUNK
class DB(Shared.DB.ZRDB.THUNK):
...
The result of this "thunk" is to create a DA that is single-threaded
through the transaction manager. More so, *all* DAs which are
"thunked" are single threaded with each other. This means that a MySQL
DA will wait on a Sybase DA, and vice versa. While this is
sub-optimal, it is at least functional, which is more than many DAs
currently can claim under Zope 2.
Support Issues
Where support has been discussed, please understand that we will try
and provide support on an as available basis to all comers, however we
can not guarantee support response times and effectiveness to any
non-paying customers. This should be taken into account when
selecting a DA.
When commercial support is offered it is offered in multiple fashions:
o Time & materials
o Per-incident basis
o Yearly contract
Please contact Digital Creations for more information about this
option.
--
| Christopher Petrilli Python Powered Digital Creations, Inc.
| petrilli@digicool.com http://www.digicool.com