Hi there, Today I had a discussion with Jasper Spaans about how to go about improving megrok.rdb, Grok's relational database integration which aims to integrate Grok with SQLAlchemy. We started developing megrok.rdb at the Grokkerdam sprint a few weeks ago. We reread the discussion surrounding zope.sqlalchemy for ideas on how to go about integration and configuration. I think these discussions reach wider than just Grok's concerns. Note that I'm not proposing we fold any of these ideas into zope.sqlalchemy itself, which should remain as free of policy as possible; it could become (yet another) extension. Let me run our current thinking by the list. What would be nice in Zope applications (and we think would be good for Grok) would be per-instance database configuration. That is, we want to be able to install multiple instances of the same application and then configure each of them with a different database URN and it should all work, each talking to their own database. Michael Bayer's suggestion involves the use of scoped sessions. He proposed the following code: Session = scoped_session() # start of request engine = get_appropriate_engine() Session(bind=engine) try: # do request finally: Session.remove() Let's go through the steps. First it makes a scoped session object, it then configures it with the right engine at the start of the request (it can do this on a per-class level), and then at the end of the request it removes the Session again, which results in the actual session being closed. Our get_appropriate_engine() would probably look the engine up as a local utility, as Laurence suggested. There is a bit of question about engine configuration, though. If we want to support the use case of looking up the engine URL in a persistent datastore (for instance one URL per location), we have a question of ordering. We cannot do it too early; at the start of the transaction there isn't a ZODB yet to talk to so we can't look up a local utility. We can try doing it just in time: _Session = scoped_session() def Session(*args, **kw): engine = get_appropriate_engine() _Session.bind(bind=engine) return _Session(*args, **kw) Here get_appropriate_engine() could do a component.getUtility() and look up the engine for us, possibly in an application-local way. There's still the question of how this engine got configured in the first place. How does it know the database URL? How does the engine get created after the database URL is known (this might be quite late in the game; it could be stored in the ZODB). It then starts to look more and more attractive to do something similar like collective.lead's IDatabase utility, which can be stored persistently in the ZODB and has a getEngine() method which actually gets the engine (creating it if necessary). If we use sqlalchemy.ext.declarative, we also need to make the declarative extension of SQLALchemy load up the tables at the right point in time. We would also like a way to hook into matters and register some of our own tables and mappers manually. We figured perhaps the utility could fire an event that you can then write a handler for. This way there's less need to subclass the utility just to change some configuration (this is what collective.lead currently requires you to do). If a persistent local utility is in play, it shouldn't fire the configuration event during its own creation, as that would mean it'd only be fired once ever. We want to fire it just after engine creation. I guess the database utility can remain quite simple. Its main tasks would be: * allow access to the engine (creating it the first time) * fire the event for additional configuration when the engine is first created * maintain or somehow obtain the database URL. This could be retrieved from the ZODB if it's a local utility, or it could be hardcoded into a global utility, or it could be retrieved from some config file by a global utility. We could have an expanded variety which also configures things using the SQLAlchemy declarative extension. We still have the question of the 'remove()' bit in Michael's code. We looked at ScopedSession's remove() method, and it looks like it removes the session from the thread-local storage, and it actually closes the session. Closing the session should be taken care of: zope.sqlalchemy's integration with Zope's transaction machinery will close the session. What about the registry cleanup that remove() appears to do? Is this currently being done by zope.sqlalchemy? Should it be? Anyway, a whole lot of abstract talk. I still hope to get some feedback on this. Regards, Martijn
We need to differentiate between the interface for session configuration and session usage from an application. For session usage I think it is fairly simple. We should define an ISessionContext interface such that: class ISessionContext(Interface): def __call__(): "return a session in this context" A future version of collective.lead could implement an ISessionContext. Client code however should have a cleaner interface, a plain ISession. This is accessed through a lookup on the context, translated into a simple adapter: def session_adapter(context): session_context = queryUtility(ISessionContext, context=context, default=None) if session_context is not None: return session_context() This will allow application code to do something like: session = ISession(context) ob = session.query(MyClass) Of course it would be possible to register a ScopedSession globally as such a utility, but more usually a local utility should be registered. (I haven't though about the consequences of this in pre-traversal, before the site and local utilities are set up) session.remove() is not important, sessions are closed by the zope.sqlalchemy datamanager and closed sessions are recyclable. Presumably the session object would be referred to by a volatile attribute on the local utility and the session would be GC'd along with the local utility object itself. Table creation is another matter that I don't think too important. Implicit creation of tables seems wrong, instead tables should only be created explicitly, by the use clicking a button in the zope web interface (or automatically on adding an application). An exception to this is sqlalchemy in memory databases, which must be created on first access. Session configuration would be somewhat similar to collective.lead currently (registering one as a local utility). Laurence Martijn Faassen wrote:
Hi there,
Today I had a discussion with Jasper Spaans about how to go about improving megrok.rdb, Grok's relational database integration which aims to integrate Grok with SQLAlchemy. We started developing megrok.rdb at the Grokkerdam sprint a few weeks ago. We reread the discussion surrounding zope.sqlalchemy for ideas on how to go about integration and configuration. I think these discussions reach wider than just Grok's concerns. Note that I'm not proposing we fold any of these ideas into zope.sqlalchemy itself, which should remain as free of policy as possible; it could become (yet another) extension.
Let me run our current thinking by the list.
What would be nice in Zope applications (and we think would be good for Grok) would be per-instance database configuration. That is, we want to be able to install multiple instances of the same application and then configure each of them with a different database URN and it should all work, each talking to their own database.
Michael Bayer's suggestion involves the use of scoped sessions. He proposed the following code:
Session = scoped_session()
# start of request engine = get_appropriate_engine() Session(bind=engine) try: # do request finally: Session.remove()
Let's go through the steps. First it makes a scoped session object, it then configures it with the right engine at the start of the request (it can do this on a per-class level), and then at the end of the request it removes the Session again, which results in the actual session being closed.
Our get_appropriate_engine() would probably look the engine up as a local utility, as Laurence suggested. There is a bit of question about engine configuration, though.
If we want to support the use case of looking up the engine URL in a persistent datastore (for instance one URL per location), we have a question of ordering. We cannot do it too early; at the start of the transaction there isn't a ZODB yet to talk to so we can't look up a local utility. We can try doing it just in time:
_Session = scoped_session()
def Session(*args, **kw): engine = get_appropriate_engine() _Session.bind(bind=engine) return _Session(*args, **kw)
Here get_appropriate_engine() could do a component.getUtility() and look up the engine for us, possibly in an application-local way. There's still the question of how this engine got configured in the first place. How does it know the database URL? How does the engine get created after the database URL is known (this might be quite late in the game; it could be stored in the ZODB). It then starts to look more and more attractive to do something similar like collective.lead's IDatabase utility, which can be stored persistently in the ZODB and has a getEngine() method which actually gets the engine (creating it if necessary).
If we use sqlalchemy.ext.declarative, we also need to make the declarative extension of SQLALchemy load up the tables at the right point in time.
We would also like a way to hook into matters and register some of our own tables and mappers manually. We figured perhaps the utility could fire an event that you can then write a handler for. This way there's less need to subclass the utility just to change some configuration (this is what collective.lead currently requires you to do). If a persistent local utility is in play, it shouldn't fire the configuration event during its own creation, as that would mean it'd only be fired once ever. We want to fire it just after engine creation.
I guess the database utility can remain quite simple. Its main tasks would be:
* allow access to the engine (creating it the first time)
* fire the event for additional configuration when the engine is first created
* maintain or somehow obtain the database URL. This could be retrieved from the ZODB if it's a local utility, or it could be hardcoded into a global utility, or it could be retrieved from some config file by a global utility.
We could have an expanded variety which also configures things using the SQLAlchemy declarative extension.
We still have the question of the 'remove()' bit in Michael's code. We looked at ScopedSession's remove() method, and it looks like it removes the session from the thread-local storage, and it actually closes the session.
Closing the session should be taken care of: zope.sqlalchemy's integration with Zope's transaction machinery will close the session. What about the registry cleanup that remove() appears to do? Is this currently being done by zope.sqlalchemy? Should it be?
Anyway, a whole lot of abstract talk. I still hope to get some feedback on this.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote:
We need to differentiate between the interface for session configuration and session usage from an application.
For session usage I think it is fairly simple. We should define an ISessionContext interface such that:
class ISessionContext(Interface): def __call__(): "return a session in this context"
+lots (I was thinking about proposing an interface called ISessionMaker doing much the same thing) I'm not sure what "in this context" means?
A future version of collective.lead could implement an ISessionContext. Client code however should have a cleaner interface, a plain ISession. This is accessed through a lookup on the context, translated into a simple adapter:
def session_adapter(context): session_context = queryUtility(ISessionContext, context=context,
Why call queryUtility with the context keyword?
default=None) if session_context is not None: return session_context()
This will allow application code to do something like:
session = ISession(context) ob = session.query(MyClass)
This really confuses me. What is the context? Does it have any meaning? Or is it just a shorter way to write: session = getUtility(ISessionContext)() Does the value of context have an effect on what you get from the ISession adaptation?
Of course it would be possible to register a ScopedSession globally as such a utility, but more usually a local utility should be registered.
Depends what you're doing. If you are running without a ZODB, you have mostly just global utilities. It would be a pity if zope.sqlalchemy started to depend on the ZODB.
(I haven't though about the consequences of this in pre-traversal, before the site and local utilities are set up)
session.remove() is not important, sessions are closed by the zope.sqlalchemy datamanager and closed sessions are recyclable. Presumably the session object would be referred to by a volatile attribute on the local utility and the session would be GC'd along with the local utility object itself.
Table creation is another matter that I don't think too important. Implicit creation of tables seems wrong, instead tables should only be created explicitly, by the use clicking a button in the zope web interface (or automatically on adding an application). An exception to this is sqlalchemy in memory databases, which must be created on first access.
Session configuration would be somewhat similar to collective.lead currently (registering one as a local utility).
Laurence
Martijn Faassen wrote:
Hi there,
Today I had a discussion with Jasper Spaans about how to go about improving megrok.rdb, Grok's relational database integration which aims to integrate Grok with SQLAlchemy. We started developing megrok.rdb at the Grokkerdam sprint a few weeks ago. We reread the discussion surrounding zope.sqlalchemy for ideas on how to go about integration and configuration. I think these discussions reach wider than just Grok's concerns. Note that I'm not proposing we fold any of these ideas into zope.sqlalchemy itself, which should remain as free of policy as possible; it could become (yet another) extension.
Let me run our current thinking by the list.
What would be nice in Zope applications (and we think would be good for Grok) would be per-instance database configuration. That is, we want to be able to install multiple instances of the same application and then configure each of them with a different database URN and it should all work, each talking to their own database.
Michael Bayer's suggestion involves the use of scoped sessions. He proposed the following code:
Session = scoped_session()
# start of request engine = get_appropriate_engine() Session(bind=engine) try: # do request finally: Session.remove()
Let's go through the steps. First it makes a scoped session object, it then configures it with the right engine at the start of the request (it can do this on a per-class level), and then at the end of the request it removes the Session again, which results in the actual session being closed.
Our get_appropriate_engine() would probably look the engine up as a local utility, as Laurence suggested. There is a bit of question about engine configuration, though.
If we want to support the use case of looking up the engine URL in a persistent datastore (for instance one URL per location), we have a question of ordering. We cannot do it too early; at the start of the transaction there isn't a ZODB yet to talk to so we can't look up a local utility. We can try doing it just in time:
_Session = scoped_session()
def Session(*args, **kw): engine = get_appropriate_engine() _Session.bind(bind=engine) return _Session(*args, **kw)
Here get_appropriate_engine() could do a component.getUtility() and look up the engine for us, possibly in an application-local way. There's still the question of how this engine got configured in the first place. How does it know the database URL? How does the engine get created after the database URL is known (this might be quite late in the game; it could be stored in the ZODB). It then starts to look more and more attractive to do something similar like collective.lead's IDatabase utility, which can be stored persistently in the ZODB and has a getEngine() method which actually gets the engine (creating it if necessary).
If we use sqlalchemy.ext.declarative, we also need to make the declarative extension of SQLALchemy load up the tables at the right point in time.
We would also like a way to hook into matters and register some of our own tables and mappers manually. We figured perhaps the utility could fire an event that you can then write a handler for. This way there's less need to subclass the utility just to change some configuration (this is what collective.lead currently requires you to do). If a persistent local utility is in play, it shouldn't fire the configuration event during its own creation, as that would mean it'd only be fired once ever. We want to fire it just after engine creation.
I guess the database utility can remain quite simple. Its main tasks would be:
* allow access to the engine (creating it the first time)
* fire the event for additional configuration when the engine is first created
* maintain or somehow obtain the database URL. This could be retrieved from the ZODB if it's a local utility, or it could be hardcoded into a global utility, or it could be retrieved from some config file by a global utility.
We could have an expanded variety which also configures things using the SQLAlchemy declarative extension.
We still have the question of the 'remove()' bit in Michael's code. We looked at ScopedSession's remove() method, and it looks like it removes the session from the thread-local storage, and it actually closes the session.
Closing the session should be taken care of: zope.sqlalchemy's integration with Zope's transaction machinery will close the session. What about the registry cleanup that remove() appears to do? Is this currently being done by zope.sqlalchemy? Should it be?
Anyway, a whole lot of abstract talk. I still hope to get some feedback on this.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Brian Sutherland
Brian Sutherland wrote:
On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote:
We need to differentiate between the interface for session configuration and session usage from an application.
For session usage I think it is fairly simple. We should define an ISessionContext interface such that:
class ISessionContext(Interface): def __call__(): "return a session in this context"
+lots
(I was thinking about proposing an interface called ISessionMaker doing much the same thing)
I'm not sure what "in this context" means?
The context of this application instance. Say you have two instances of an application, one pointed at database A, another at database B. It is possible to involve both applications in a single request / transaction. You must be sure that you are working with the correct session.
A future version of collective.lead could implement an ISessionContext. Client code however should have a cleaner interface, a plain ISession. This is accessed through a lookup on the context, translated into a simple adapter:
def session_adapter(context): session_context = queryUtility(ISessionContext, context=context,
Why call queryUtility with the context keyword?
default=None) if session_context is not None: return session_context()
This will allow application code to do something like:
session = ISession(context) ob = session.query(MyClass)
This really confuses me. What is the context? Does it have any meaning? Or is it just a shorter way to write:
session = getUtility(ISessionContext)()
Does the value of context have an effect on what you get from the ISession adaptation?
Yes, as it translates directly to the getUtility lookup. It ensures that lookups in /appA go to a local utility defined in /appA and lookups in /appB go to a local utility defined in /appB.
Of course it would be possible to register a ScopedSession globally as such a utility, but more usually a local utility should be registered.
Depends what you're doing. If you are running without a ZODB, you have mostly just global utilities.
It would be a pity if zope.sqlalchemy started to depend on the ZODB.
Wihout ZODB and zope.app.component it seems unlikely that you would be able to register a local utility, the idea of ISessionContext is so that you might be able to register a ScopedSession as a global utility too.
(I haven't though about the consequences of this in pre-traversal, before the site and local utilities are set up)
session.remove() is not important, sessions are closed by the zope.sqlalchemy datamanager and closed sessions are recyclable. Presumably the session object would be referred to by a volatile attribute on the local utility and the session would be GC'd along with the local utility object itself.
Table creation is another matter that I don't think too important. Implicit creation of tables seems wrong, instead tables should only be created explicitly, by the use clicking a button in the zope web interface (or automatically on adding an application). An exception to this is sqlalchemy in memory databases, which must be created on first access.
Session configuration would be somewhat similar to collective.lead currently (registering one as a local utility).
Laurence
Martijn Faassen wrote:
Hi there,
Today I had a discussion with Jasper Spaans about how to go about improving megrok.rdb, Grok's relational database integration which aims to integrate Grok with SQLAlchemy. We started developing megrok.rdb at the Grokkerdam sprint a few weeks ago. We reread the discussion surrounding zope.sqlalchemy for ideas on how to go about integration and configuration. I think these discussions reach wider than just Grok's concerns. Note that I'm not proposing we fold any of these ideas into zope.sqlalchemy itself, which should remain as free of policy as possible; it could become (yet another) extension.
Let me run our current thinking by the list.
What would be nice in Zope applications (and we think would be good for Grok) would be per-instance database configuration. That is, we want to be able to install multiple instances of the same application and then configure each of them with a different database URN and it should all work, each talking to their own database.
Michael Bayer's suggestion involves the use of scoped sessions. He proposed the following code:
Session = scoped_session()
# start of request engine = get_appropriate_engine() Session(bind=engine) try: # do request finally: Session.remove()
Let's go through the steps. First it makes a scoped session object, it then configures it with the right engine at the start of the request (it can do this on a per-class level), and then at the end of the request it removes the Session again, which results in the actual session being closed.
Our get_appropriate_engine() would probably look the engine up as a local utility, as Laurence suggested. There is a bit of question about engine configuration, though.
If we want to support the use case of looking up the engine URL in a persistent datastore (for instance one URL per location), we have a question of ordering. We cannot do it too early; at the start of the transaction there isn't a ZODB yet to talk to so we can't look up a local utility. We can try doing it just in time:
_Session = scoped_session()
def Session(*args, **kw): engine = get_appropriate_engine() _Session.bind(bind=engine) return _Session(*args, **kw)
Here get_appropriate_engine() could do a component.getUtility() and look up the engine for us, possibly in an application-local way. There's still the question of how this engine got configured in the first place. How does it know the database URL? How does the engine get created after the database URL is known (this might be quite late in the game; it could be stored in the ZODB). It then starts to look more and more attractive to do something similar like collective.lead's IDatabase utility, which can be stored persistently in the ZODB and has a getEngine() method which actually gets the engine (creating it if necessary).
If we use sqlalchemy.ext.declarative, we also need to make the declarative extension of SQLALchemy load up the tables at the right point in time.
We would also like a way to hook into matters and register some of our own tables and mappers manually. We figured perhaps the utility could fire an event that you can then write a handler for. This way there's less need to subclass the utility just to change some configuration (this is what collective.lead currently requires you to do). If a persistent local utility is in play, it shouldn't fire the configuration event during its own creation, as that would mean it'd only be fired once ever. We want to fire it just after engine creation.
I guess the database utility can remain quite simple. Its main tasks would be:
* allow access to the engine (creating it the first time)
* fire the event for additional configuration when the engine is first created
* maintain or somehow obtain the database URL. This could be retrieved from the ZODB if it's a local utility, or it could be hardcoded into a global utility, or it could be retrieved from some config file by a global utility.
We could have an expanded variety which also configures things using the SQLAlchemy declarative extension.
We still have the question of the 'remove()' bit in Michael's code. We looked at ScopedSession's remove() method, and it looks like it removes the session from the thread-local storage, and it actually closes the session.
Closing the session should be taken care of: zope.sqlalchemy's integration with Zope's transaction machinery will close the session. What about the registry cleanup that remove() appears to do? Is this currently being done by zope.sqlalchemy? Should it be?
Anyway, a whole lot of abstract talk. I still hope to get some feedback on this.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Sat, May 24, 2008 at 09:30:18PM +0100, Laurence Rowe wrote:
Brian Sutherland wrote:
On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote:
We need to differentiate between the interface for session configuration and session usage from an application.
For session usage I think it is fairly simple. We should define an ISessionContext interface such that:
class ISessionContext(Interface): def __call__(): "return a session in this context"
+lots
(I was thinking about proposing an interface called ISessionMaker doing much the same thing)
I'm not sure what "in this context" means?
The context of this application instance. Say you have two instances of an application, one pointed at database A, another at database B. It is possible to involve both applications in a single request / transaction. You must be sure that you are working with the correct session.
Are the instances you are talking about persistent objects in the ZODB?
A future version of collective.lead could implement an ISessionContext. Client code however should have a cleaner interface, a plain ISession. This is accessed through a lookup on the context, translated into a simple adapter:
def session_adapter(context): session_context = queryUtility(ISessionContext, context=context,
Why call queryUtility with the context keyword?
default=None) if session_context is not None: return session_context()
This will allow application code to do something like:
session = ISession(context) ob = session.query(MyClass)
This really confuses me. What is the context? Does it have any meaning? Or is it just a shorter way to write:
session = getUtility(ISessionContext)()
Does the value of context have an effect on what you get from the ISession adaptation?
Yes, as it translates directly to the getUtility lookup. It ensures that lookups in /appA go to a local utility defined in /appA and lookups in /appB go to a local utility defined in /appB.
I've been burned by using context as a keyword to getUtility before. When your context doesn't have a proper __parent__ pointer, the default IComponentLookup adapter falls back to the global site manager and ignores the local one. That causes no end of confusion and hard to debug bugs as people will call ISession on objects with and without __parent__ pointers and then wonder why it fails. Perhaps a more robust way is to rely on the site stored while traversing in zope.app.component.hooks? For example: * appA implements ISite and has a local ISessionContext utility * The path appA is traversed over (i.e. the url path is /appA/foo) * queryUtility(ISessionContext) then returns the local utility from appA without the need to specify a "context" (thanks to the magic in zope.app.component.hooks) I think the only situation where you really want to specify a "context" for queryUtility is when you want to access a local utility registered in /appB from a URL path like /appA/foo. That seems pretty rare and almost broken. You could implement a function like: def query_session(name=u''): return queryUtility(ISessionContext, name=name)() That does the right thing in almost all cases (i.e. uses the closest local site), and is much more robust.
Of course it would be possible to register a ScopedSession globally as such a utility, but more usually a local utility should be registered.
Depends what you're doing. If you are running without a ZODB, you have mostly just global utilities.
It would be a pity if zope.sqlalchemy started to depend on the ZODB.
Wihout ZODB and zope.app.component it seems unlikely that you would be able to register a local utility, the idea of ISessionContext is so that you might be able to register a ScopedSession as a global utility too.
I very much like the idea of ISessionContext! I'm just not sure about how you suggest using it in client code. -- Brian Sutherland
Brian Sutherland wrote:
On Sat, May 24, 2008 at 09:30:18PM +0100, Laurence Rowe wrote:
Brian Sutherland wrote:
On Fri, May 23, 2008 at 11:39:39PM +0100, Laurence Rowe wrote:
We need to differentiate between the interface for session configuration and session usage from an application.
For session usage I think it is fairly simple. We should define an ISessionContext interface such that:
class ISessionContext(Interface): def __call__(): "return a session in this context" +lots
(I was thinking about proposing an interface called ISessionMaker doing much the same thing)
I'm not sure what "in this context" means? The context of this application instance. Say you have two instances of an application, one pointed at database A, another at database B. It is possible to involve both applications in a single request / transaction. You must be sure that you are working with the correct session.
Are the instances you are talking about persistent objects in the ZODB?
Yes, I'm assuming that the application instances live in the ZODB and are local site managers for this example.
A future version of collective.lead could implement an ISessionContext. Client code however should have a cleaner interface, a plain ISession. This is accessed through a lookup on the context, translated into a simple adapter:
def session_adapter(context): session_context = queryUtility(ISessionContext, context=context, Why call queryUtility with the context keyword?
default=None) if session_context is not None: return session_context()
This will allow application code to do something like:
session = ISession(context) ob = session.query(MyClass) This really confuses me. What is the context? Does it have any meaning? Or is it just a shorter way to write:
session = getUtility(ISessionContext)()
Does the value of context have an effect on what you get from the ISession adaptation? Yes, as it translates directly to the getUtility lookup. It ensures that lookups in /appA go to a local utility defined in /appA and lookups in /appB go to a local utility defined in /appB.
I've been burned by using context as a keyword to getUtility before. When your context doesn't have a proper __parent__ pointer, the default IComponentLookup adapter falls back to the global site manager and ignores the local one. That causes no end of confusion and hard to debug bugs as people will call ISession on objects with and without __parent__ pointers and then wonder why it fails.
Perhaps a more robust way is to rely on the site stored while traversing in zope.app.component.hooks?
For example: * appA implements ISite and has a local ISessionContext utility * The path appA is traversed over (i.e. the url path is /appA/foo) * queryUtility(ISessionContext) then returns the local utility from appA without the need to specify a "context" (thanks to the magic in zope.app.component.hooks)
I think the only situation where you really want to specify a "context" for queryUtility is when you want to access a local utility registered in /appB from a URL path like /appA/foo. That seems pretty rare and almost broken.
You could implement a function like:
def query_session(name=u''): return queryUtility(ISessionContext, name=name)()
That does the right thing in almost all cases (i.e. uses the closest local site), and is much more robust.
I fear that if we rely on the site manager set during traversal, then applications will rely on that when looking up their session in application code, and it will be impossible to involve objects from different application instances in the same transaction. getUtility already looks up in the global component registry if a site manager cannot be found in the context's parents (or where __parent__ is missing). I guess this concept could be extended with something like: def session_adapter(context): smgr = getSiteManager(context) if smgr is getGlobalSiteManager(): smgr = None return getUtility(ISessionContext, context=smgr)() This would defer the bugs to the point you start working with two instances of an application in a single request. I wouldn't consider accessing objects from /appA and /appB in the same request broken, for someone coming from a ZODB background it would seems quite reasonable.
Of course it would be possible to register a ScopedSession globally as such a utility, but more usually a local utility should be registered. Depends what you're doing. If you are running without a ZODB, you have mostly just global utilities.
It would be a pity if zope.sqlalchemy started to depend on the ZODB. Wihout ZODB and zope.app.component it seems unlikely that you would be able to register a local utility, the idea of ISessionContext is so that you might be able to register a ScopedSession as a global utility too.
I very much like the idea of ISessionContext!
I'm just not sure about how you suggest using it in client code.
I definitely want to avoid a dependency on ZODB or zope.app.component for zope.sqlalchemy, but I would like to find some way of allowing the possibility of local utility registrations so that you don't hit a brick wall when you start invloving multiple application instances in a single request. I guess I'm suggesting that applications whose root lives in the ZODB should be local site managers and have a local ISessionContext utility registered. Applications that live solely on the file system should just register a ScopedSession as the ISessionContext utility globally. If that code wanted to allow for the possibility of more than one application instance, then it could register an ISessionContext as a named utility and register an ISession adapter that had logic like: def session_adapter(context): name = get_my_application_id(context) return getUtility(ISessionContext, name)() Code that needs access to a session object could then use the session = ISession(context) pattern in all the above cases. Laurence
Laurence Rowe wrote: [snip]
I fear that if we rely on the site manager set during traversal, then applications will rely on that when looking up their session in application code, and it will be impossible to involve objects from different application instances in the same transaction.
As I pointed out earlier in the thread, all *other* getUtility calls in a typical application don't use an explicit context anyway. If you're going to call from one application into another without changing your site explicitly before you do the call, you're going to be in for a world of trouble *anyway*. Let's teach people to use manual setSite() when they have two cooperating applications and forget about explicitly setting the context. [snip]
I wouldn't consider accessing objects from /appA and /appB in the same request broken, for someone coming from a ZODB background it would seems quite reasonable.
In fact, Zope 3 hasn't supported this pattern for years now. If appA calls a method in appB which looks up a local utility that only is registered for appB, that local utility *won't be found*. You need to explicitly use setSite() before you make the call in order to make this work properly. Unless appB *only* uses explicit contexts everywhere to look up its utilities (and adapters, and views, actually, as these can also be overridden in a local site manager!), and you really can't be counting on that. I think it's a bad idea to have one pattern for ZODB use (where we use local utilities and site managers and so on) and another one for non-ZODB use. SQLAlchemy already supports much of our use case with scoped sessions. The open problem is how to set up the right engine for the current session, and we can use utility lookup for that. Local site managers are not ZODB-bound as far as I know. Regards, Martijn
Laurence Rowe wrote:
Brian Sutherland wrote: [snip]
I'm not sure what "in this context" means?
The context of this application instance. Say you have two instances of an application, one pointed at database A, another at database B. It is possible to involve both applications in a single request / transaction. You must be sure that you are working with the correct session.
I should've read the whole thread before I asked the same question as Brian. I think the use case of involving two applications in the same transaction is a exceptional use case, and therefore passing in context manually each and every time isn't warranted. I'd prefer it if we simply used Zope 3's existing mechanisms for this, where getUtility allows the explicit passing of a context attribute *if desired*, instead of wrapping it in an adapter and requiring an explicit context just for that. Passing in context all the time would theoretically allow existing code to work unchanged when two applications are involved in the same transaction. They'd both get the right session. I'll note that Zope 3 itself doesn't actually provide for this use case for its own local utility system - it encourages people to look up utility *without* context, and thus if two sites are in play (especially when non-nested), quite unexpected things can happen when one calls into the other. For that reason, I argued against the thread-local changes that Jim made years ago. Jim wisely went ahead and did it anyway, and I've since come to appreciate the alteration, as it just makes life a lot easier for the programmer not to have to worry about context all the time. I never actually saw much unexpected happen in practice. So for the same reason I propose we follow Zope's lead and don't place a burden on all developers for this rather theoretical use case.
Of course it would be possible to register a ScopedSession globally as such a utility, but more usually a local utility should be registered.
Depends what you're doing. If you are running without a ZODB, you have mostly just global utilities.
It would be a pity if zope.sqlalchemy started to depend on the ZODB.
Wihout ZODB and zope.app.component it seems unlikely that you would be able to register a local utility, the idea of ISessionContext is so that you might be able to register a ScopedSession as a global utility too.
local utilities are not necessarily persistent, so you could still have local sites with local site managers without the ZODB. I'd argue in this case just using SQLAlchemy's ScopedSession mechanism works fine for scoping sessions however, and we don't go play with utilities for sessions, just for engines. Regards, Martijn
Laurence Rowe wrote:
We need to differentiate between the interface for session configuration and session usage from an application.
Session configuration? I'm talking about engine configuration. A session doesn't need to be configured, except with a session, I think, which the ScopedSession machinery allows you to do. If you can make the engine be the right one in the appropriate context it should only be a matter of configuring the session in that context.
For session usage I think it is fairly simple. We should define an ISessionContext interface such that:
class ISessionContext(Interface): def __call__(): "return a session in this context"
A future version of collective.lead could implement an ISessionContext.
Client code however should have a cleaner interface, a plain ISession. This is accessed through a lookup on the context, translated into a simple adapter:
def session_adapter(context): session_context = queryUtility(ISessionContext, context=context, default=None) if session_context is not None: return session_context()
This will allow application code to do something like:
session = ISession(context) ob = session.query(MyClass)
I don't understand what the point of doing this is. What is 'context' here? Why am I adapting context? My context is normally thread-local and implicit. This has been the way to approach context in Zope 3 for a long time now (though explicit context can still be passed through getUtility it's rarely done). And again, I think *engine* should be in context, not sessions. ScopedSession, a standard SQLAlchemy mechanism, should be used for session context-specific session access. Why the introduction of ISessionContext and ISession interfaces, and an adapter lookup that looks up a utility and then *still* I haven't seen the code that actually configures the engine? My aim was to try to stick to SQLAlchemy patterns for solving this problem where we had no reason to diverge from them, and our use case, I take it, is what ScopedSession was designed for. [snip]
session.remove() is not important, sessions are closed by the zope.sqlalchemy datamanager and closed sessions are recyclable.
That's good to seee confirmed. I thought it was that way reading the code, but I wanted to make sure.
Presumably the session object would be referred to by a volatile attribute on the local utility and the session would be GC'd along with the local utility object itself.
Are you talking about a persistent local utility? Which one? Would this mean that the session needs to be re-created each time the ZODB swaps out the object with the volatile attribute? I thought the session was intended to be recreated each request, does it make any sense to cache them between requests?
Table creation is another matter that I don't think too important.
I think it's important to get it right for Grok. We're using the declarative extension and still want to support hand-created tables as well. There are various scenarios surrounding table creation, either not doing it at all ever, or spelling them out by hand in Python, or by inlining them into the classes as with the declarative extension.
Implicit creation of tables seems wrong, instead tables should only be created explicitly, by the use clicking a button in the zope web interface (or automatically on adding an application).
Automatic on adding an application is a good point to do it, as that way we don't bother the person who creates the application too much (first create the database.. then install the application, then go to this screen and create the tables). There's another feature to automatic table creation though: when I'm developing and I don't care about the data yet, all I need to do now when I change the schema is throw away the database and create it again. I found this very convenient while developing with collective.lead. We need to have something that is at least as convenient for this use case (common during initial development).
An exception to this is sqlalchemy in memory databases, which must be created on first access.
Don't know what these are?
Session configuration would be somewhat similar to collective.lead currently (registering one as a local utility).
Before we talk more about session configuration, please explain why we're not talking about engine configuration. :) Regards, Martijn
Martijn Faassen wrote:
Laurence Rowe wrote:
We need to differentiate between the interface for session configuration and session usage from an application.
Session configuration? I'm talking about engine configuration. A session doesn't need to be configured, except with a session, I think, which the ScopedSession machinery allows you to do. If you can make the engine be the right one in the appropriate context it should only be a matter of configuring the session in that context.
For session usage I think it is fairly simple. We should define an ISessionContext interface such that:
class ISessionContext(Interface): def __call__(): "return a session in this context"
A future version of collective.lead could implement an ISessionContext.
Client code however should have a cleaner interface, a plain ISession. This is accessed through a lookup on the context, translated into a simple adapter:
def session_adapter(context): session_context = queryUtility(ISessionContext, context=context, default=None) if session_context is not None: return session_context()
This will allow application code to do something like:
session = ISession(context) ob = session.query(MyClass)
I don't understand what the point of doing this is.
What is 'context' here? Why am I adapting context? My context is normally thread-local and implicit. This has been the way to approach context in Zope 3 for a long time now (though explicit context can still be passed through getUtility it's rarely done).
And again, I think *engine* should be in context, not sessions. ScopedSession, a standard SQLAlchemy mechanism, should be used for session context-specific session access.
Why the introduction of ISessionContext and ISession interfaces, and an adapter lookup that looks up a utility and then *still* I haven't seen the code that actually configures the engine? My aim was to try to stick to SQLAlchemy patterns for solving this problem where we had no reason to diverge from them, and our use case, I take it, is what ScopedSession was designed for.
[snip]
session.remove() is not important, sessions are closed by the zope.sqlalchemy datamanager and closed sessions are recyclable.
That's good to seee confirmed. I thought it was that way reading the code, but I wanted to make sure.
Presumably the session object would be referred to by a volatile attribute on the local utility and the session would be GC'd along with the local utility object itself.
Are you talking about a persistent local utility? Which one? Would this mean that the session needs to be re-created each time the ZODB swaps out the object with the volatile attribute? I thought the session was intended to be recreated each request, does it make any sense to cache them between requests?
Table creation is another matter that I don't think too important.
I think it's important to get it right for Grok. We're using the declarative extension and still want to support hand-created tables as well. There are various scenarios surrounding table creation, either not doing it at all ever, or spelling them out by hand in Python, or by inlining them into the classes as with the declarative extension.
Implicit creation of tables seems wrong, instead tables should only be created explicitly, by the use clicking a button in the zope web interface (or automatically on adding an application).
Automatic on adding an application is a good point to do it, as that way we don't bother the person who creates the application too much (first create the database.. then install the application, then go to this screen and create the tables).
There's another feature to automatic table creation though: when I'm developing and I don't care about the data yet, all I need to do now when I change the schema is throw away the database and create it again. I found this very convenient while developing with collective.lead. We need to have something that is at least as convenient for this use case (common during initial development).
An exception to this is sqlalchemy in memory databases, which must be created on first access.
Don't know what these are?
Those with a url like 'sqlite://:memory:'
Session configuration would be somewhat similar to collective.lead currently (registering one as a local utility).
Before we talk more about session configuration, please explain why we're not talking about engine configuration. :)
Engine configuration is a subset of session configuration. You cannot have a single ScopedSession for a package if you want to have multiple instances of that application. We must work with unbound metadata if we have this goal. That implies that we must use bound sessions, a session associated with a particular engine (actually it could be more complex than this, you can associate particular classes/tables with particular engines within a session). The simplest way to achieve this is to associate an engine with a single (scoped) session, and create the engine based on some configuration infomation used to instantiate the scoped session / thing that implements ISessionContext. In collective.tin I got bored passing around IDatabase utility names everywhere I needed to access a session, and created an IDatabase adapter that looked up the IDatabase on the __parent__ object. This would then be resolved to an actual utility lookup when it reached the top level. On a second look, this is entirely unnecessary... It is possible to retrieve the session for a mapped object using: session = sqlalchemy.orm.session.Session.object_session(obj) I'm not sure whether it would be a good idea to wrap this in a session property, or just register it as an adapter. The only other object that would need access to the session (either as a property or through adaption) would be the application instance root object. Something like: @adapter(MyApp) @provider(ISession) def root_session(context): return context._sessioncontext() And the simplest persistent session context might be: class PersistentSessionContext(persistent): implements(ISessionContext) def __init__(self, url, twophase=False, engine_kw={}, session_kw={}): self.url = url self.twophase = twophase self.engine_kw = engine_kw self.session_kw = session_kw def __call__(self): session = getattr(self._v_session, None) if session is None: engine = getattr(self._v_engine, None) if engine is None: engine = self._v_engine = create_engine( self.url, **self.engine_kw) session = self._v_session = create_session( bind=engine, twophase=self.twophase, **self.session_kw) return session A more complex scheme might maintain a dict of ScopedSessions keyed by application path, outside of the object cache. You could also ensure that only a single engine is created for a given set of arguments, but this seems overkill Everything would then get a session consistently with a call to ISession(self) or ISession(self.context) in the case of views. No parent pointers involved. We do still need to setup parent pointers though for grok.url and IAbsoluteURL to work. It looks fairly easy to add location information to the containers themselves: from sqlalchemy.orm.collections import MappedCollection, collection_adapter class LocatedCollection(MappedCollection): @property def __name__(self): return collection_adapter(self).attr.key @property def __parent__(self): return collection_adapter(self).owner_state.obj() But locating mapped objects themselves is more complex, they could exist in more than one collection at once. Perhaps A LocationProxy could solve this. Laurence
Laurence Rowe wrote:
Martijn Faassen wrote: [snip]
Before we talk more about session configuration, please explain why we're not talking about engine configuration. :)
Engine configuration is a subset of session configuration. You cannot have a single ScopedSession for a package if you want to have multiple instances of that application.
I don't understand then what Mike Bayer wrote before: Mike Bayer wrote:
If you are running different instances each connected to a different engine within one process you wouldn't need any awareness of engines at the object level (therefore no entity_name) and also no engine proxying - you should have separate Session instances for each "process" managed by scoped_session(), which was designed to handle this. Multiple apps on one codebase within one process was an original requirement of Pylons as well, though nobody has ever used it.
That seemed to suggest to me that scoped sessions were an appropriate solution. Anyway, back to you: [snip]
I'm not sure whether it would be a good idea to wrap this in a session property, or just register it as an adapter. The only other object that would need access to the session (either as a property or through adaption) would be the application instance root object. Something like:
@adapter(MyApp) @provider(ISession) def root_session(context): return context._sessioncontext()
This looks to me like it could be a simple function that looks up a local utility instead: def session(): return component.getUtility(ISessionContext)() We get the right context from the site that way. I don't see the point in trying to re-implement local utilities with adapters while zope.component already does something quite similar for you. That said, I still have hope we can used ScopedSession and forgo a utility lookup here, see below...
And the simplest persistent session context might be:
class PersistentSessionContext(persistent): implements(ISessionContext)
def __init__(self, url, twophase=False, engine_kw={}, session_kw={}): self.url = url self.twophase = twophase self.engine_kw = engine_kw self.session_kw = session_kw
def __call__(self): session = getattr(self._v_session, None) if session is None: engine = getattr(self._v_engine, None) if engine is None: engine = self._v_engine = create_engine( self.url, **self.engine_kw) session = self._v_session = create_session( bind=engine, twophase=self.twophase, **self.session_kw) return session
Doesn't ScopedSession already take care of this using ScopedRegistry? Perhaps a more clever scopefunc is necessary to introduce a per-instance scope? Right now it's only per-thread scope. scopefunc could also do a local utility lookup that gets it a way to uniquely identify the current application (not sure what would be best, object id? zodb id? a unique string generated for each installed application?). Something like this: def scopefunc(): return (component.getUtility(ISessionSiteScope).applicationScope(), thread.getindent()) Session = scoped_session(sessionmaker(autoflush=True), scopefunc)
A more complex scheme might maintain a dict of ScopedSessions keyed by application path, outside of the object cache. You could also ensure that only a single engine is created for a given set of arguments, but this seems overkill
Everything would then get a session consistently with a call to ISession(self) or ISession(self.context) in the case of views. No parent pointers involved.
I still don't understand why this is nicer than a local utility lookup. I understand the two cooperating applications use case, but that can be easily provided for with using setSite(), just like you *already* need to do to make everything work right with the other local utilities that this application might be using. Above in my scopefunc example I assume that setSite has been set up appropriately. If you don't use the ZODB at all, you could still set up local sites (I'm not sure how hard this is, but it *should* be possible; zope.component has no knowledge of persistence), or if it's just one app per zope instance, set up a global ISessionSiteScope utility.
We do still need to setup parent pointers though for grok.url and IAbsoluteURL to work. It looks fairly easy to add location information to the containers themselves:
(note that grok.url uses IAbsoluteURL, so we just care about IAbsoluteURL) So far this isn't a particular problem in Grok as traversal uses zope.location.located() to locate everything. That said, if you want to directly access an object by accessing ORM-mapped attributes and then get a URL for that object, this won't work. Since it *does* work when you use the ZODB, it'd be nice if it also worked properly with sqlalchemy. Hopefully we can indeed make containers behave the right way by making our own MappedCollection.
from sqlalchemy.orm.collections import MappedCollection,
collection_adapter
class LocatedCollection(MappedCollection):
@property def __name__(self): return collection_adapter(self).attr.key
@property def __parent__(self): return collection_adapter(self).owner_state.obj()
But locating mapped objects themselves is more complex, they could exist in more than one collection at once. Perhaps A LocationProxy could solve this.
LocationProxy (using zope.location.located()) has worked quite well for me so far. Regards, Martijn
Martijn Faassen wrote:
Laurence Rowe wrote:
Martijn Faassen wrote: [snip]
Before we talk more about session configuration, please explain why we're not talking about engine configuration. :)
Engine configuration is a subset of session configuration. You cannot have a single ScopedSession for a package if you want to have multiple instances of that application.
I don't understand then what Mike Bayer wrote before:
Mike Bayer wrote:
If you are running different instances each connected to a different engine within one process you wouldn't need any awareness of engines at the object level (therefore no entity_name) and also no engine proxying - you should have separate Session instances for each "process" managed by scoped_session(), which was designed to handle this. Multiple apps on one codebase within one process was an original requirement of Pylons as well, though nobody has ever used it.
That seemed to suggest to me that scoped sessions were an appropriate solution.
Anyway, back to you:
[snip]
I'm not sure whether it would be a good idea to wrap this in a session property, or just register it as an adapter. The only other object that would need access to the session (either as a property or through adaption) would be the application instance root object. Something like:
@adapter(MyApp) @provider(ISession) def root_session(context): return context._sessioncontext()
This looks to me like it could be a simple function that looks up a local utility instead:
def session(): return component.getUtility(ISessionContext)()
We get the right context from the site that way. I don't see the point in trying to re-implement local utilities with adapters while zope.component already does something quite similar for you. That said, I still have hope we can used ScopedSession and forgo a utility lookup here, see below...
And the simplest persistent session context might be:
class PersistentSessionContext(persistent): implements(ISessionContext)
def __init__(self, url, twophase=False, engine_kw={}, session_kw={}): self.url = url self.twophase = twophase self.engine_kw = engine_kw self.session_kw = session_kw
def __call__(self): session = getattr(self._v_session, None) if session is None: engine = getattr(self._v_engine, None) if engine is None: engine = self._v_engine = create_engine( self.url, **self.engine_kw) session = self._v_session = create_session( bind=engine, twophase=self.twophase, **self.session_kw) return session
Doesn't ScopedSession already take care of this using ScopedRegistry?
Perhaps a more clever scopefunc is necessary to introduce a per-instance scope? Right now it's only per-thread scope. scopefunc could also do a local utility lookup that gets it a way to uniquely identify the current application (not sure what would be best, object id? zodb id? a unique string generated for each installed application?).
Something like this:
def scopefunc(): return (component.getUtility(ISessionSiteScope).applicationScope(), thread.getindent())
Session = scoped_session(sessionmaker(autoflush=True), scopefunc)
A more complex scheme might maintain a dict of ScopedSessions keyed by application path, outside of the object cache. You could also ensure that only a single engine is created for a given set of arguments, but this seems overkill
Everything would then get a session consistently with a call to ISession(self) or ISession(self.context) in the case of views. No parent pointers involved.
I still don't understand why this is nicer than a local utility lookup. I understand the two cooperating applications use case, but that can be easily provided for with using setSite(), just like you *already* need to do to make everything work right with the other local utilities that this application might be using. Above in my scopefunc example I assume that setSite has been set up appropriately.
That would be fine if you had the same configuration across all sessions (e.g. they all connected to the same engine / database) or each session was configured at the start of every request. Presumably we will want to connect different application instances to different databases. This means that if we have a central register of ScopedSessions, then we must key it by some unique application id (path I guess). def application_session(app): try: return registry[app.getPath()]() except KeyError: return registry.setdefault(app.getPath(), scoped_session(sessionmaker(**app.getConfiguration())))() My point about using adapters, or indeed properties to access the session, is that the only object needing to access the session which cannot look it up directly with Session.object_session(object) is the application root object. To me it seems simpler to do this than to register utilities. Also it would be nice to have a consistent way to lookup the session.
If you don't use the ZODB at all, you could still set up local sites (I'm not sure how hard this is, but it *should* be possible; zope.component has no knowledge of persistence), or if it's just one app per zope instance, set up a global ISessionSiteScope utility.
We do still need to setup parent pointers though for grok.url and IAbsoluteURL to work. It looks fairly easy to add location information to the containers themselves:
(note that grok.url uses IAbsoluteURL, so we just care about IAbsoluteURL)
So far this isn't a particular problem in Grok as traversal uses zope.location.located() to locate everything. That said, if you want to directly access an object by accessing ORM-mapped attributes and then get a URL for that object, this won't work. Since it *does* work when you use the ZODB, it'd be nice if it also worked properly with sqlalchemy.
Hopefully we can indeed make containers behave the right way by making our own MappedCollection.
from sqlalchemy.orm.collections import MappedCollection,
collection_adapter
class LocatedCollection(MappedCollection):
@property def __name__(self): return collection_adapter(self).attr.key
@property def __parent__(self): return collection_adapter(self).owner_state.obj()
But locating mapped objects themselves is more complex, they could exist in more than one collection at once. Perhaps A LocationProxy could solve this.
LocationProxy (using zope.location.located()) has worked quite well for me so far.
That sounds promising then. Laurence
Hey Laurence, [cc-ed Mike in case he hadn't seen this thread and has comments] Thanks for the continued back and forth; I'm learning about SQLAlchemy and hopefully my pushing helps you improve your ideas too. Laurence Rowe wrote:
Martijn Faassen wrote: [using scoped sessions] That would be fine if you had the same configuration across all sessions (e.g. they all connected to the same engine / database) or each session was configured at the start of every request. Presumably we will want to connect different application instances to different databases.
Mike Bayer suggested configuring the session with the proper engine at the start of each request in his example that I quote at the start of the thread. How expensive actually is it to configure a session with an engine? I'm still getting back to combining scoped sessions and utilities, though. We need to do two things: * convince scoped session to return the session appropriate for our application. We can do this by introducing a custom scopefunc that takes application into account (by looking up a utility in it that knows what scope the application has) * convince the session factory to look up application specific configuration information the first time a new scope is entered.
This means that if we have a central register of ScopedSessions, then we must key it by some unique application id (path I guess).
def application_session(app): try: return registry[app.getPath()]() except KeyError: return registry.setdefault(app.getPath(), scoped_session(sessionmaker(**app.getConfiguration())))()
I don't understand, why is a further registry necessary? What is wrong with the modified scopefunc I suggested (using path or whatever you prefer to key it)? Then you don't need a registry of ScopedSessions, ScopedSession *is* that registry... def our_scope_func(): return (component.getUtility(ISessionSiteScope).applicationScope(), thread.getindent()) where applicationScope could be the path or the unique id or whatever would be best for the application, as long as it's unique per app. Then we also introduce a custom session factory that does a utility lookup for its configuration (I'm handwaving hopefully non-essential details here): def our_session_factory(): config = component.getUtility(ISessionSiteScope).configuration() return sessionmaker(**config)() Session = scoped_session(our_session_factory, our_scope_func) ScopedSession should then take care of the rest. ISessionSiteScope is looked up each time you instantiate Session. 'configuration()' is only called the *first* time you instantiate a session in a particular scope, after that the configuration is cached.
My point about using adapters, or indeed properties to access the session, is that the only object needing to access the session which cannot look it up directly with Session.object_session(object) is the application root object. To me it seems simpler to do this than to register utilities. Also it would be nice to have a consistent way to lookup the session.
I think we can agree that the ideal way to look up the session would be to allow the following in applications, right? from zope.rdbintegration import Session session = Session() and then session would always be an appropriately configured session for the current scope. Now please explain to me why what I sketched out above can't work. :) (note that if you want two cooperating applications each using their own session, you're required to use setSite() before you call into the second application. This is required anyway to make other local utility lookups work properly for that other application, so this is not an extra burden on developers) Regards, Martijn
On Thu, May 29, 2008 at 07:10:16PM +0200, Martijn Faassen wrote:
[snip]
I'm not sure whether it would be a good idea to wrap this in a session property, or just register it as an adapter. The only other object that would need access to the session (either as a property or through adaption) would be the application instance root object. Something like:
@adapter(MyApp) @provider(ISession) def root_session(context): return context._sessioncontext()
This looks to me like it could be a simple function that looks up a local utility instead:
def session(): return component.getUtility(ISessionContext)()
We get the right context from the site that way. I don't see the point in trying to re-implement local utilities with adapters while zope.component already does something quite similar for you. That said, I still have hope we can used ScopedSession and forgo a utility lookup here, see below...
+lots Perhaps you may want the utility name as a keyword. I'll also note that doing it this way does not preclude using the ISession adapter in future. But, using the ISession adapter now means never being able to get rid of it again.
If you don't use the ZODB at all, you could still set up local sites (I'm not sure how hard this is, but it *should* be possible;
I've done it as a work around to override a global utility forced upon me. It's not too difficult.
We do still need to setup parent pointers though for grok.url and IAbsoluteURL to work. It looks fairly easy to add location information to the containers themselves:
(note that grok.url uses IAbsoluteURL, so we just care about IAbsoluteURL)
I'll note that there is a difference between where things are published (IAbsoluteURL) and where you should go to find their canonical representation. One problem I've found with using IAbsoluteURL and location proxies is getting the object directly from the database. In this situation you don't have an LocationProxy and IAbsoluteURL doesn't work. It's pretty nasty when IAbsoluteURL sometimes works and sometimes doesn't for what essentially is the same object. -- Brian Sutherland
Brian Sutherland wrote: [snip]
I'll note that there is a difference between where things are published (IAbsoluteURL) and where you should go to find their canonical representation.
One problem I've found with using IAbsoluteURL and location proxies is getting the object directly from the database. In this situation you don't have an LocationProxy and IAbsoluteURL doesn't work.
It's pretty nasty when IAbsoluteURL sometimes works and sometimes doesn't for what essentially is the same object.
Agreed, we should ideally always get our objects from the db already properly wrapped in a location proxy (or providing ILocation directly). I think we can do this with containers based on relations; those containers are under our control after all. Is a direct one on one relation possible in SQLAlchemy? I.e. an object foo that has an attribute 'bar' that is another ORM-ed object. If so we also need to take care of that case somehow. This is the "it should work like ZODB-backed objects" pattern I've been trying to follow as a design guideline. Since for contained objects I can always get the URL, it should work that way for RDB-based contained objects too. Regards, Martijn
On Fri, May 30, 2008 at 03:49:46PM +0200, Martijn Faassen wrote:
Brian Sutherland wrote: [snip]
I'll note that there is a difference between where things are published (IAbsoluteURL) and where you should go to find their canonical representation.
One problem I've found with using IAbsoluteURL and location proxies is getting the object directly from the database. In this situation you don't have an LocationProxy and IAbsoluteURL doesn't work.
It's pretty nasty when IAbsoluteURL sometimes works and sometimes doesn't for what essentially is the same object.
Agreed, we should ideally always get our objects from the db already properly wrapped in a location proxy (or providing ILocation directly). I think we can do this with containers based on relations; those containers are under our control after all.
I've just decided to jettison IAbsoluteURL and make a new interface ICanonicalURL. The adapters for ICanonicalURL are available anywhere without specially wrapping the object in a location proxy.
Is a direct one on one relation possible in SQLAlchemy? I.e. an object foo that has an attribute 'bar' that is another ORM-ed object. If so we also need to take care of that case somehow.
This was a problem with sqlos containers as well. It gets nasty quickly. Say you have 2 containers, of people and addresses. Also people can have an address. So you can get to an address in 2 different ways: /adresses/address1 /people/person1/address1 What should the url for address1 be? I'd vote for /adresses/address1. But due to the LocationProxy, you'll get different results for different IAbsoluteURL calls depending on how you traversed to the object.
This is the "it should work like ZODB-backed objects" pattern I've been trying to follow as a design guideline. Since for contained objects I can always get the URL, it should work that way for RDB-based contained objects too.
I'm not sure it's worthwhile following the "it should work like ZODB-backed objects" pattern.
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Brian Sutherland
On Fri, May 30, 2008 at 1:40 PM, Brian Sutherland <brian@vanguardistas.net> wrote:
I've just decided to jettison IAbsoluteURL and make a new interface ICanonicalURL. The adapters for ICanonicalURL are available anywhere without specially wrapping the object in a location proxy.
IAbsoluteURL doesn't neccesarily require a location proxy, it's just that the default implementations require objects to implement ILocation. No need to define your own interface, just override the IAbsoluteURL implementation(s) you don't like to behave differently (or better yet, don't let them get registered in the first place). -- Benji York Senior Software Engineer Zope Corporation
On Fri, May 30, 2008 at 01:55:44PM -0400, Benji York wrote:
On Fri, May 30, 2008 at 1:40 PM, Brian Sutherland <brian@vanguardistas.net> wrote:
I've just decided to jettison IAbsoluteURL and make a new interface ICanonicalURL. The adapters for ICanonicalURL are available anywhere without specially wrapping the object in a location proxy.
IAbsoluteURL doesn't neccesarily require a location proxy, it's just that the default implementations require objects to implement ILocation.
No need to define your own interface, just override the IAbsoluteURL implementation(s) you don't like to behave differently (or better yet, don't let them get registered in the first place).
One thing I don't like about the default implementation is that if you override IAbsoluteURL for a container, it has no effect for contained objects. I suppose there's a good reason for that -- efficiency -- but it often makes custom IAbsoluteURL adapters pointless. LocationProxy helps then. Marius Gedminas -- Photons have energy, and trying to cram too many into too small of a space can cause a black hole to form, which is, needless to say, not a desirable trait for an optical computer. -- http://scottaaronson.com/blog/?p=261#comment-13693
Brian Sutherland wrote: [snip]
This was a problem with sqlos containers as well. It gets nasty quickly. Say you have 2 containers, of people and addresses. Also people can have an address.
So you can get to an address in 2 different ways:
/adresses/address1 /people/person1/address1
What should the url for address1 be? I'd vote for /adresses/address1. But due to the LocationProxy, you'll get different results for different IAbsoluteURL calls depending on how you traversed to the object.
I've seen this complaint coming from quite a few people a number of times, so this needs some thinking about. One interesting observation is that if you do this in the ZODB you'd get the exact same problem, and then people use things like zc.shortcut to get around it. So, in the ZODB, we typically don't give the same object two URLs. I guess one difference here is that it seems more common in the RDB scenario to have multiple ways to reach the same object, and even if you just publish one, you can get to the actual object in multiple ways. In this case it'd be nice to convince IAbsoluteURL to somehow know which URL option is the best, or alternatively, give the object somehow its 'canonical' location (parent, etc) even if you don't get it that way.
This is the "it should work like ZODB-backed objects" pattern I've been trying to follow as a design guideline. Since for contained objects I can always get the URL, it should work that way for RDB-based contained objects too.
I'm not sure it's worthwhile following the "it should work like ZODB-backed objects" pattern.
The nice property of ZODB-backed objects is that they're just plain Python objects, and it's also familiar to me, so that's why I look for special reasons to diverge from it. Regards, Martijn
On Fri, May 30, 2008 at 2:35 PM, Martijn Faassen <faassen@startifact.com> wrote:
I guess one difference here is that it seems more common in the RDB scenario to have multiple ways to reach the same object, and even if you just publish one, you can get to the actual object in multiple ways.
In this case it'd be nice to convince IAbsoluteURL to somehow know which URL option is the best, or alternatively, give the object somehow its 'canonical' location (parent, etc) even if you don't get it that way.
I have an app that allows the same object to be reachable via several URLs. When we wanted one of the URLs to be preferred (none are really "cononical" in my case), then we did just as you suggest above and added a way for the IAbsoluteURL adapter(s) to know which URL was preferred. It worked out quite well. -- Benji York Senior Software Engineer Zope Corporation
Benji York wrote:
On Fri, May 30, 2008 at 2:35 PM, Martijn Faassen <faassen@startifact.com> wrote:
I guess one difference here is that it seems more common in the RDB scenario to have multiple ways to reach the same object, and even if you just publish one, you can get to the actual object in multiple ways.
In this case it'd be nice to convince IAbsoluteURL to somehow know which URL option is the best, or alternatively, give the object somehow its 'canonical' location (parent, etc) even if you don't get it that way.
I have an app that allows the same object to be reachable via several URLs. When we wanted one of the URLs to be preferred (none are really "cononical" in my case), then we did just as you suggest above and added a way for the IAbsoluteURL adapter(s) to know which URL was preferred. It worked out quite well.
Cool. Where does the adapter get the information to make a decision from? I mean, how does it know to use URL A and not URL B? Regards, Martijn
On Fri, May 30, 2008 at 3:03 PM, Martijn Faassen <faassen@startifact.com> wrote:
Benji York wrote:
I have an app that allows the same object to be reachable via several URLs. When we wanted one of the URLs to be preferred (none are really "cononical" in my case), then we did just as you suggest above and added a way for the IAbsoluteURL adapter(s) to know which URL was preferred. It worked out quite well.
Cool. Where does the adapter get the information to make a decision from? I mean, how does it know to use URL A and not URL B?
Nothing fancy: we adapt the object to (names changed to clarify the example) IPreferredLocations to get back a list ordered by how preferred the locations are (from most-preferred to least), we then simply iterate over the list (it's always short) until we find the first place the object is visible, and use it. The location ordering is specified by the user (more or less, there are details I'm leaving out). -- Benji York Senior Software Engineer Zope Corporation
On May 28, 2008, at 7:09 PM, Laurence Rowe wrote:
Engine configuration is a subset of session configuration. You cannot have a single ScopedSession for a package if you want to have multiple instances of that application. We must work with unbound metadata if we have this goal. That implies that we must use bound sessions, a session associated with a particular engine (actually it could be more complex than this, you can associate particular classes/tables with particular engines within a session).
<chiming in, although I haven't read the whole thread....> I would agree that "bound" metadata is a little cumbersome in an environment that wants to switch engines within a single process. "Bound" metadata has always given us a lot of headaches because it's just *so* convenient that we just can't get rid of it, but at the same time people are just so tripped up by it, thinking it's necessary to make anything happen (it's not). Binding sessions allows better control of connection/transaction scope (since the Session relates to transactions, MetaData does not), so it's probabably the way to go in an environment that has a lot of things going on. As far as ScopedSession, it's really just a thread local variable. Like any global threadlocal, you can stick whatever you want on it at the start of a request, and tear it down at the end. Then again it's also reasonable that you might want to have individual ScopedSessions for each application instance within a multi-app process, that way the burden of setting up/tearing down is reduced or removed. A single ScopedSession for a multi-app process is like a one-dimensional approach where both current thread and current app are identified by the current "get()" of the registry; a collection of ScopedSessions is more like a two-dimensional approach where the first level of registry (i.e. which ScopedSession do I choose) distinguishes between app instance, and the second level (i.e. what Session is bound to this thread ID) distinguishes between threads. All of that said I think it can work either way but I think the latter approach might have the explicitness you're looking for.
Hey, Michael Bayer wrote: [snip discussion on BoundMetadata I don't comprehend yet but probably should :)]
As far as ScopedSession, it's really just a thread local variable. Like any global threadlocal, you can stick whatever you want on it at the start of a request, and tear it down at the end.
Is it really needed to set things up at the start and the request and tear things down on the end, or can a session be retained between requests? I.e. is the intent to recreate a Session class each time a request is issued?
Then again it's also reasonable that you might want to have individual ScopedSessions for each application instance within a multi-app process, that way the burden of setting up/tearing down is reduced or removed.
This indicates to me is possible to retain a Session object between two requests? I.e. it's a worthwhile goal to reduce the amount of Session configuration going on, right?
A single ScopedSession for a multi-app process is like a one-dimensional approach where both current thread and current app are identified by the current "get()" of the registry; a collection of ScopedSessions is more like a two-dimensional approach where the first level of registry (i.e. which ScopedSession do I choose) distinguishes between app instance, and the second level (i.e. what Session is bound to this thread ID) distinguishes between threads.
All of that said I think it can work either way but I think the latter approach might have the explicitness you're looking for.
I'm trying to understand why you think so. I am looking for a way to let developers use SQLAlchemy in a straightforward way. They should be able to import 'Session', instantiate session in their app, and everything works as expected. The framework takes care of making you get the appropriately configured Session. That underneath sessions are scoped per thread and per app, and configured per app, is something the developer who handles the session object ideally should not have to worry about. Since I hope we can actually make this work, why do you think an explicit approach with a custom registry which then contains scoped sessions would be better? The less of our own new registries we invent the better, I'd say. That said, I guess you could make the same semantics work if you make Session be some factory that uses a nested registry. (I'd still absolutely like to avoid passing in context explicitly each time you need a session; it puts unnecessary burden on the developer and promises to make multi-application interactions easier while it actually doesn't do so) Regards, Martijn
On May 30, 2008, at 9:44 AM, Martijn Faassen wrote:
Hey,
Michael Bayer wrote:
[snip discussion on BoundMetadata I don't comprehend yet but probably should :)]
As far as ScopedSession, it's really just a thread local variable. Like any global threadlocal, you can stick whatever you want on it at the start of a request, and tear it down at the end.
Is it really needed to set things up at the start and the request and tear things down on the end, or can a session be retained between requests? I.e. is the intent to recreate a Session class each time a request is issued?
in the 0.4 series, the Session is "sticky" in that whatever is in it retains the state that was last loaded - 0.4 does not go back to the database to re-fetch the state of an object unless its explicitly told to do so. When an HTTP request ends, assuming all objects in the session are marked as "clean", they're weakly referenced and will fall out of scope assuming those objects were only referenced by the request. So in that sense, you can just leave the Session alone and it'll work just fine for the next request...but any pending changes in the Session that weren't flushed for some reason would get carried over, as well as anything that might be strongly referenced elsewhere. So we always recommended in 0.4 to at least issue a session.clear() at the end of a request to just empty it out (but you can still reuse that session). Other options included scopedsession.remove() which tears the whole session down, the advantage there being that the new request could configure the next session differently (as in, binding it to something else). in 0.5, Session has been changed to be less reluctant to go and "re- fetch" data (which is contrary to the particular background I came from, but since then I've learned to see a broader scope of use cases). In 0.5, after a commit() or rollback(), the Session still may be holding on to objects, but their state is expired such that it will all be re-loaded upon access, and all "pending" and "deleted" states are reverted. So 0.5's Session, when configured in the default way, makes it very hard to get at "stale" state, so in that sense you can just re-use a session wihtout worrying much about what may have been left over.
Then again it's also reasonable that you might want to have individual ScopedSessions for each application instance within a multi-app process, that way the burden of setting up/tearing down is reduced or removed.
This indicates to me is possible to retain a Session object between two requests? I.e. it's a worthwhile goal to reduce the amount of Session configuration going on, right?
Its not a strong argument either way to reuse a session or just make a new one on each request. In 0.4, making a brand new session on each request does have a "cleaner" feel to it since theres no chance of any state left hanging around. Its not an expensive operation.
A single ScopedSession for a multi-app process is like a one- dimensional approach where both current thread and current app are identified by the current "get()" of the registry; a collection of ScopedSessions is more like a two-dimensional approach where the first level of registry (i.e. which ScopedSession do I choose) distinguishes between app instance, and the second level (i.e. what Session is bound to this thread ID) distinguishes between threads. All of that said I think it can work either way but I think the latter approach might have the explicitness you're looking for.
I'm trying to understand why you think so. I am looking for a way to let developers use SQLAlchemy in a straightforward way. They should be able to import 'Session', instantiate session in their app, and everything works as expected. The framework takes care of making you get the appropriately configured Session.
they can in fact do this without any issue. The question is, when someone writes the call "s = Session()" three times within one request, do you want each "s" to all reference the *same* set of objects ? that was the issue scoped_session() was meant to solve. Configuration is easy since you just configure a "Session" callable for that application. But you guys have this issue of "multiple applications in the same process" at play, which each talk to a different database. So thats where a decision has to be made how to deal with that complexity, either configure engine on one scoped_session per request, or configure multiple scoped_sessions, one per engine. The latter approach seems "easier" to me since you don't actually have to do anything on a per-request basis assuming 0.5 usage. Neither of these make any difference to the end user who sees the exact same usage pattern.
(I'd still absolutely like to avoid passing in context explicitly each time you need a session; it puts unnecessary burden on the developer and promises to make multi-application interactions easier while it actually doesn't do so)
so one scoped_session() per app instance/engine seems like the easiest way to do this.
Martijn Faassen wrote:
Hi there,
Today I had a discussion with Jasper Spaans about how to go about improving megrok.rdb, Grok's relational database integration which aims to integrate Grok with SQLAlchemy. We started developing megrok.rdb at the Grokkerdam sprint a few weeks ago. We reread the discussion surrounding zope.sqlalchemy for ideas on how to go about integration and configuration. I think these discussions reach wider than just Grok's concerns. Note that I'm not proposing we fold any of these ideas into zope.sqlalchemy itself, which should remain as free of policy as possible; it could become (yet another) extension.
Let me run our current thinking by the list.
What would be nice in Zope applications (and we think would be good for Grok) would be per-instance database configuration. That is, we want to be able to install multiple instances of the same application and then configure each of them with a different database URN and it should all work, each talking to their own database.
Instance meaning Zope instance? In that case I'd rather like to have the configuration in zope.conf (even for classic zope2 with database adaptor) since it makes it both easy to change w/o relying on a running client (think of user auth via database and then chaning the session...) and also for security (currently if you get into management account you can see login details to the database which isnt always desired - as well as ability to open arbitrary tcp connections from zope) The implementation should be fairly easy and requires no special tricks. Cheers Tino
Hey, On Mon, May 26, 2008 at 9:45 PM, Tino Wildenhain <tino@wildenhain.de> wrote:
Martijn Faassen wrote: [snip]
What would be nice in Zope applications (and we think would be good for Grok) would be per-instance database configuration. That is, we want to be able to install multiple instances of the same application and then configure each of them with a different database URN and it should all work, each talking to their own database.
Instance meaning Zope instance?
No, multiple instances of an application into the same Zope. Sorry about the confusing terminology, I didn't do it. :) Regards, Martijn
participants (7)
-
Benji York -
Brian Sutherland -
Laurence Rowe -
Marius Gedminas -
Martijn Faassen -
Michael Bayer -
Tino Wildenhain