The transaction manager here: http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collec... Has support for TPC and savepoints. It is tested and works. The only issue with sqlite is that the default in the branch is to use two-phase commit and sqlite does not support this. Apart from that the module is completely self contained. The public interfaces to it is the two functions at the bottom, join_transaction and dirty_session. The idea of the dirty_session stuff is to prevent unnecessary commits when data is only read. Session writes trigger the 'dirtying' automatically with a SessionExtension: http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collec... The downside of this approach is that the when you interact with the connection directly to modify data you must call dirty_session yourself. If this is a problem for you then you may set up the transaction to be in the dirty state initially. Thread based session management is used (scoped_session). There is fairly extensive testing of this and the transaction support here: http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collec... I think it would be worth factoring out the threaded session support as well as the transactions. @Christian: I'm not quite sure what you mean by: - Database triggers as events I don't see how you could reasonably get a database trigger on the db server to instantiate a zope event on the db client. Perhaps the database could be persuaded to make a request to a URL on a trigger, but this seems a bit slow and maybe not a great solution. - Making SA 0.4 work with security proxies I don't know anything about security proxies in zope 3, but surely this is just a matter of what the base class is / the container implementation? I don't see why an SQL Alchemy mapped object would be any different to any other object. @Martin: Should one phase commit be set as the default to make it easier to work with sqlite (and mssql)? Probably yes. Should the default be for sessions to start out `active` or `dirty`? If they start out `dirty` then existing 1.0 code should work as before. It'll probably be next week before I manage to get this tidied up and merged back to trunk. @everyone: If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module. Laurence Martin Aspeli wrote:

Martijn Faassen wrote:

...

Hi there,

[I originally picked this up on a thread on zope3-users, but this deserves its own thread here]

There are at least three approaches to SQLAlchemy integration with Zope:

* z3c.zalchemy (Christian Theune)

* z3c.sqlalchemy (Andreas Jung)

* collective.lead (Laurence Rowe)

All of these are in various states of brokenness. z3c.zalchemy doesn't work with SQLAlchemy trunk. collective.lead works with it, but only if you check out a particular branch, and not with sqlite. Quite possibly z3c.sqlalchemy has a release that actually works. One out of three is not bad... :)

Then there's also mentions about WSGI-based integration, and I think in Plone, Alchemist probably also does its own integration...

There must be a reason for this proliferation of approaches. What is it? We all get along, don't we? I know that the various packages are taking code and approaches from each other too.

Can't we work together more and at least come up with *one* package that works? Perhaps factor out some low-level commonality than then all share? Criticize one of the other packages until you're satisfied, and then retire your own package perhaps? I know the various packages add on their own approaches to configuration and might offer higher level container approaches. Those could be in different packages, sharing a foundation.

In the end, I hope we will end up with just *one* integration layer, that is released, that works with Zope 2 and Zope 3 and a recent release of SQLAlchemy, that is documented, and that people know about. We can then offer packages on top of this that offer extra features.

I'm all for one integration layer, even though I spurred one of the above (collective.lead). I would ask that we be very conservative, though. collective.lead wants to provide:

- automatic transaction integration with ZODB transactions - an easy way to register your database connections - an easy way to look those up as utilities and get an ORM session

Everything else (e.g. Zope ORM integration, configuration abstractions, etc) should be in some other package.

collective.lead has a reasonable degree of traction in the Plone universe, at least, but should work just fine with plain Zope 3. We have other things that work on top it (mercury, rope, probably others).

The branch that works with trunk also has some carefully worked out TPC support, which is difficult to get right, though I'm disappointed that we seem to have broken sqlite integration. Hopefully it's an easy fix, though.

I'm CC'ing Laurence to ask:

- can we get a release soon? - can we fix the sqlite integration?

Cheers, Martin

Andreas Jung wrote:

--On 9. April 2008 14:15:38 +0100 Laurence Rowe <l@lrowe.co.uk> wrote:

...

@everyone:

If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module.

I would be happy with such a solution. As said, I am not into the datamanager-building business and more interested in application-integration issues. The basic requirements I have for our applications:

- having the same session within one ZODB transaction (per thread) - need to get hold of the underlying database connection for a given session

I think that's very much the business collective.lead wants to be in. :) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book

On Wed, Apr 09, 2008 at 05:47:06PM +0200, Andreas Jung wrote:

--On 9. April 2008 14:15:38 +0100 Laurence Rowe <l@lrowe.co.uk> wrote:

...

@everyone:

If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module.

I would be happy with such a solution. As said, I am not into the datamanager-building business and more interested in application-integration issues. The basic requirements I have for our applications:

- having the same session within one ZODB transaction (per thread) - need to get hold of the underlying database connection for a given session

Just adding my few requirements: - Integration into transactions. - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. - Have a connection available as IZopeDatabaseAdapter utility (for BBB) -- Brian Sutherland

On Thu, Apr 10, 2008 at 07:31:54PM +0200, Andreas Jung wrote:

--On 10. April 2008 19:10:49 +0200 Brian Sutherland <brian@vanguardistas.net> wrote:

...

Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility.

Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration.

What defines if something is the instance configuration? Isn't ZCML just another format of configuration file? I agree it's evil to hardcode database connections in the application as they must be under the control of the sysadmin. But that doesn't preclude them from being expressed in zcml. I *do* like the idea of using one and only one file format to configure applications. We also run the in a few different environments, the /etc/myserver/site.zcml is different on a per instance basis and looks like this: <configure xmlns="http://namespaces.zope.org/zope" xmlns:rdb="http://namespaces.zope.org/rdb"> <include package="mymodule" file="configure.zcml" /> <rdb:provideConnection name="admin" component="psycopg2da.adapter.Psycopg2Adapter" dsn="dbi://prototype" /> </configure> But, I'm diverging, my point is that it would be nice to be able to have SQLAlchemy connections available as IZopeDatabaseAdapter utilities to allow a migration path. Whether they are defined in or out of ZCML, I just don't care. -- Brian Sutherland

--On 10. April 2008 21:29:43 +0200 Brian Sutherland <brian@vanguardistas.net> wrote:

On Thu, Apr 10, 2008 at 07:31:54PM +0200, Andreas Jung wrote:

...

--On 10. April 2008 19:10:49 +0200 Brian Sutherland <brian@vanguardistas.net> wrote:

...

Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility.

Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration.

What defines if something is the instance configuration?

Isn't ZCML just another format of configuration file? I agree it's evil to hardcode database connections in the application as they must be under the control of the sysadmin. But that doesn't preclude them from being expressed in zcml.

The point is: we don't need yet another configuration language for the configuration of a site. ZCML should be used for the purpose it has been intended for: for the configuration of the components of your site (easily spoken). This kind of configuration is static if you setup a site for project X. But settings like ports, paths etc. often differ between dev, test and production environments. We already have ZConfig as standard cofniguration language for instance specific configuration aspects. In addition: we often do use environment variables and .ini files for additional configurations. Introducing another configuration language is stupid. Especially because system administrators or co-workers know little or nothing about ZCML. This is an unnecessary introduction of complexity. Setting an environment variable with a DSN is much easier then having to fiddle around with ZCML and XML...totally unnecessary overhead....keep it simple ...using ZCML in this configuration context is only for XML masochists. Andreas

--On 11. April 2008 10:09:16 +0200 Brian Sutherland <brian@vanguardistas.net> wrote:

On Fri, Apr 11, 2008 at 05:07:56AM +0200, Andreas Jung wrote:

...

--On 10. April 2008 21:29:43 +0200 Brian Sutherland <brian@vanguardistas.net> wrote:

...

What defines if something is the instance configuration?

Isn't ZCML just another format of configuration file? I agree it's evil to hardcode database connections in the application as they must be under the control of the sysadmin. But that doesn't preclude them from being expressed in zcml.

The point is: we don't need yet another configuration language for the configuration of a site. ZCML should be used for the purpose it has been intended for: for the configuration of the components of your site (easily spoken). This kind of configuration is static if you setup a site for project X. But settings like ports, paths etc. often differ between dev, test and production environments.

We already have ZConfig as standard cofniguration language for instance specific configuration aspects. In addition: we often do use environment variables and .ini files for additional configurations. Introducing another configuration language is stupid. Especially because system administrators or co-workers know little or nothing about ZCML. This is an unnecessary introduction of complexity. Setting an environment variable with a DSN is much easier then having to fiddle around with ZCML and XML...totally unnecessary overhead....keep it simple ...using ZCML in this configuration context is only for XML masochists.

Unfortunately, it doesn't seem an option, right now, to avoid using ZCML to setup database connections. So I'm not sure how you want to implement your ideas?

I have no need for ideas to be implemented. We're using SA within in several large projects based on z3c.sqlalchemy. And nothing beats the simplicity of of an .ini file or an environment variable.

In general, I think if you want your configuration expressed in ZConfig and available as a utility, you have a problem.

I'm not sure of: * How do I register a utility in ZConfig?

z3c.sqlalchemy provides a method for registering a SA connection as a named utility. This code is usually called during the startup phase of Zope within the related initialize() methds of our products. The DSN are acquired from well-known environment variables...this just works - easy, simple, reliable.

* How will the above registration work properly in functional tests (especially with tearing down and setting up layers)?

Functional tests in our company are tests done by humans. Why? Our applications are very complex with lots of dependencies. Not speaking of lots of involved databases, big storages.. Andreas

Brian Sutherland wrote: [snip]

Unfortunately, it doesn't seem an option, right now, to avoid using ZCML to setup database connections. So I'm not sure how you want to implement your ideas?

collective.lead can do it without ZCML; you set up the right connection by implementing a 'url' property on a (global) utility. This to me seems to be flexible, as the code could get the connection from anywhere. Regards, Martijn

On 11.04.2008, at 10:09, Brian Sutherland wrote:

Unfortunately, it doesn't seem an option, right now, to avoid using ZCML to setup database connections. So I'm not sure how you want to implement your ideas?

+1 k, i didn't read the whole thread, but we (lovely systems) use zope.storm but i think its the same story somehow and we do all instance specific configuration in zcml - also database uris. we have a base.zcml config for all instances and include this it and do all instance specific configuration in an instance specific zcml file. this file is then used with the site-definition otpion in zope.conf. i can see no reason why we should avoid zcml except in environments where there is no zcml at all. although it is a few lines of code to get db uris from somewhere else, and this could also be application code regards, bernd

On Friday 11 April 2008, Bernd Dorn wrote:

although it is a few lines of code to get db uris from somewhere else,   and this could also be application code

if you use paster, you can stick options into your pater ini script. We are currently doing this for a setting in our app. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"

Martijn Faassen wrote:

Andreas Jung wrote:

...

--On 10. April 2008 19:10:49 +0200 Brian Sutherland <brian@vanguardistas.net> wrote:

...

Just adding my few requirements: - Integration into the component architecture in such a way that I can specify the db connection parameters in ZCML and that database reflection still works. I want to be able to get a handle on SQLAlchemy things by calling getUtility. Please *NO* database specific configurations within ZCML. We're running applications in up three or four different environments and I don't want to maintain instance specific configurations within ZCML. Either we pass such informations through environment variables or the database connections parameters are read from a configuration within the instance configuration.

As long as it can also be specified in the code, that way people can build UIs to set up the connection. Perhaps that's what you mean by a "configuration within the instance configuration".

The way I use collective.lead in my book is to have it look up the database settings in a local utility. That utility is editable via a control panel page in Plone. I suspect that it'd be quite easy to do something similar where the settings were deferred to a global utility configured with ZCML or looked up in a config file of some sort. Once the config is available, it's all about calling getUtility() to get hold of a database connection/session. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book

Andreas Jung <lists <at> zopyx.com> writes:

...

The way I use collective.lead in my book is to have it look up the database settings in a local utility. That utility is editable via a control panel page in Plone. I suspect that it'd be quite easy to do something similar where the settings were deferred to a global utility configured with ZCML or looked up in a config file of some sort.

Once the config is available, it's all about calling getUtility() to get hold of a database connection/session.

Possibly a sane and clean solution but way of over-designed. I am interested in simple and robust solutions that work and not in a solution that adds more and more unnecessary layers just the sake of having them

collective.lead doesn't enforce this style at all. If you want TTW configurable connections, though, it's a good pattern to use. All collective.lead does is: - Set up database connections and engines properly - Hook into transaction machinery so you don't need to do your own transaction management - Let you look up a database session/connection as a utility You need to register that utility, of course. Lead has a base class that means you just fill in a few template methods (database url, and some callbacks for configuring tables and orm classes). If you want some kind of config-file based or ZCML based version, you'd just construct an appropriate class and register it as a utility on startup. Lead doesn't want to enforce one particular connection parameter configuration model or config file format. That should be the job of a higher level package that is concerned only with getting these parameters, not with the database setup itself. However, you probably still need some imperative code to set up the tables and mappers. Martin

On Wed, Apr 9, 2008 at 9:15 AM, Laurence Rowe <l@lrowe.co.uk> wrote:

The transaction manager here:

http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collec...

Has support for TPC and savepoints. It is tested and works. The only issue with sqlite is that the default in the branch is to use two-phase commit and sqlite does not support this. Apart from that the module is completely self contained.

The public interfaces to it is the two functions at the bottom, join_transaction and dirty_session. The idea of the dirty_session stuff is to prevent unnecessary commits when data is only read. Session writes trigger the 'dirtying' automatically with a SessionExtension:

http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collec...

The downside of this approach is that the when you interact with the connection directly to modify data you must call dirty_session yourself. If this is a problem for you then you may set up the transaction to be in the dirty state initially.

Thread based session management is used (scoped_session). There is fairly extensive testing of this and the transaction support here:

http://svn.plone.org/svn/collective/collective.lead/branches/elro-tpc/collec...

I think it would be worth factoring out the threaded session support as well as the transactions.

+1, the collective.lead transaction support looks nice.

@Christian:

I'm not quite sure what you mean by:

- Database triggers as events

I don't see how you could reasonably get a database trigger on the db server to instantiate a zope event on the db client. Perhaps the database could be persuaded to make a request to a URL on a trigger, but this seems a bit slow and maybe not a great solution.

- Making SA 0.4 work with security proxies

I don't know anything about security proxies in zope 3, but surely this is just a matter of what the base class is / the container implementation? I don't see why an SQL Alchemy mapped object would be any different to any other object.

its more to do with the fact that proxies are restrictive by default, and the sa api likes to monkey attributes on instances. christian kicked off some nice work with the sa developers at pycon to get this behavior configurable in sa ( instrumentation branch) which obviate the need to remove the proxies before calling into sa apis on an object.

@Martin:

Should one phase commit be set as the default to make it easier to work with sqlite (and mssql)? Probably yes.

Should the default be for sessions to start out `active` or `dirty`? If they start out `dirty` then existing 1.0 code should work as before.

It'll probably be next week before I manage to get this tidied up and merged back to trunk.

@everyone:

If we can all agree to use the same basic session and transaction management then we should probably push for it to be included as a sqlalchemy extension module.

sounds good, or failing that (due to additional dependencies), a common base package. cheers, kapil

Martin Aspeli wrote:

Laurence Rowe wrote:

...

Should one phase commit be set as the default to make it easier to work with sqlite (and mssql)? Probably yes.

Ideally we'd guess based on the URL scheme but allow it to be set explicitly, IMHO. Single phase would be the fallback, I guess.

I don't like the idea of guessing too much, but it could be done in the __init__. There might be good reasons to choose one phase commit even for a two phase commit supporting database (i.e. when you only are talking to a single database, it is more efficient). Given that a single database is the common case I am tempted to say just make 1PC the default.

...

Should the default be for sessions to start out `active` or `dirty`? If they start out `dirty` then existing 1.0 code should work as before.

What are the implications of having them start out dirty?

Unnecessary commits to the database for read only transactions (i.e. the current situation) Laurence

Martijn Pieters wrote:

On Tue, Apr 8, 2008 at 11:54 PM, Martijn Faassen <faassen@startifact.com> wrote:

...

All of these are in various states of brokenness. z3c.zalchemy doesn't work with SQLAlchemy trunk. collective.lead works with it, but only if you check out a particular branch, and not with sqlite. Quite possibly z3c.sqlalchemy has a release that actually works. One out of three is not bad... :)

We are going into production with collective.lead and are committed to releasing the 1.0 branch as 1.0. sqlite works just great for us, we use it to run unit tests and for developers that just need to adjust the styling and such. The production environment will run against Oracle.

The elro-tpc branch (which, I was told, is the future) doesn't appear to work with sqlite out of the box. I'm curious how to do functional tests with collective.lead - I'd like to have some real easy way to get the database set up and tore down down in my tests. Are you doing this? Regards, Martijn

Martin Aspeli wrote: [snip]

It's never particularly pleasant.

I think we need to make sure we make this as easy and pleasant as possible though. This means at the very least some document that tells you what to do. :) Regards, Martijn

Hi there, On Thu, Apr 10, 2008 at 7:04 PM, Brian Sutherland <brian@vanguardistas.net> wrote:

On Wed, Apr 09, 2008 at 12:51:53PM -0400, Kapil Thangavelu wrote:

...

that sound good, i'd like to see a common base layer, providing transaction support, simple containers.

Actually, containers are one of the things I think should be explicitly excluded from a base SQLAlchemy integration package.

You could do the same thing by IPublishTraverse plugins as well.

Agreed. I think the common base layer should do the minimum transaction integration stuff and leave any policy to layers on top. Hopefully we'll arrive at a commonly used package for containers too at some point, but we'll see about that then. Regards, Martijn

--On 9. April 2008 20:28:18 +0200 Dieter Maurer <dieter@handshake.de> wrote:

Martijn Faassen wrote at 2008-4-8 23:54 +0200:

...

... There are at least three approaches to SQLAlchemy integration with Zope: .... Quite possibly z3c.sqlalchemy has a release that actually works.

The current release is not very robust against some kinds of temporary problems. When e.g. the database is restarted, "z3c.sqlalchemy" may enter a state where all database interactions raise an exception until Zope is restarted.

This is very likely an issue of the underlying SA implementation. None of the existing wrappers deal with this kind of problems. In fact I've never seen such issues within applications under my control :-)

-aj