Hi there, I'd like to announce my contribution for the expanding list of options for SQLAlchemy integration for Zope 3. I've just implemented a package called z3c.saconfig which implements a utility-based way to set up SQLAlchemy's scoped session, as discussed recently on this. The package currently offers a way to configure SQLAlchemy sessions with a utility. This allows SQLAlchemy sessions to potentially differ per application. Currently if you would want to implement such utilities you're still on your own, but I intend to add support for this soon. Hermann What is in there already is support for a global utility as discussed with Brian Sutherland. I intend to add support for a local utility soon, inspired by some code sent to me by Hermann Himmelbauer. Anyway, the README.txt should explain more: http://svn.zope.org/z3c.saconfig/trunk/src/z3c/saconfig/README.txt Feedback is welcome! Regards, Martijn
Hey, Martijn Faassen wrote:
I intend to add support for a local utility soon, inspired by some code sent to me by Hermann Himmelbauer.
This is now in there. It only looks faintly like Hermann's code, but it was still very useful. You can register an engine factory globally or locally. This engine factory can be called to create an engine, but it'll return the same engine again when called again. This way, engines are shared and engine pooling works. To make this work for your application/framework you need to subclass SiteScopedSession and implement siteScopeFunc. You can then register an instance of that subclass globally (or locally if you should so desire, but globally is usually enough). The README was updated to reflect the new situation: http://svn.zope.org/z3c.saconfig/trunk/src/z3c/saconfig/README.txt This now works against the trunk of zope.sqlalchemy as well, as Laurence merged my branch. I do think my code currently requires SQLAlchemy 0.5beta1 or higher. Overall I'm quite pleased by this code. It's straightforward, and leaves to SQLAlchemy as much as possible, and just passes configuration parameters for engines and sessions along if you define them for the utilities. It doesn't offer any user interface code or schemas itself; that's up to the application or framework developer. It's also flexible. Configuration information could be retrieved from any place a developer would like; hardcoded in Python, or the ZODB. I think it wouldn't be hard to write custom utilities that look up configuration in configuration files such as zope.conf or ZCML as well. One thing I'm not sure about is using _v_engine to store the engine in a persistent local utility. What if the local utility gets evicted from the ZODB cache? It'd need to recreate the engine. Anyway, I've left the policy on how to cache the engine overridable. I'm curious about better ideas. I considered using a global dictionary with the cached engines in there, as nothing could get just disappear from that, but I'm not entirely sure that'd be thread safe. There is a chance that an engine gets recreated if two threads were to be writing to it at the same time. Perhaps some thread locking code is required. Opinions? Feedback is again welcome! Regards, Martijn
Am Freitag, 20. Juni 2008 00:14 schrieb Martijn Faassen:
Hey,
Martijn Faassen wrote:
I intend to add support for a local utility soon, inspired by some code sent to me by Hermann Himmelbauer.
This is now in there. It only looks faintly like Hermann's code, but it was still very useful.
Thanks for the flowers. :-)
This now works against the trunk of zope.sqlalchemy as well, as Laurence merged my branch. I do think my code currently requires SQLAlchemy 0.5beta1 or higher.
Does it really? Because in the README.txt doctests, there seems to be a "session.save()" directive that is AFAIK deprecated by "session.add()".
Overall I'm quite pleased by this code. It's straightforward, and leaves to SQLAlchemy as much as possible, and just passes configuration parameters for engines and sessions along if you define them for the utilities.
Yes, that's also my overall impression.
It doesn't offer any user interface code or schemas itself; that's up to the application or framework developer. It's also flexible. Configuration information could be retrieved from any place a developer would like; hardcoded in Python, or the ZODB. I think it wouldn't be hard to write custom utilities that look up configuration in configuration files such as zope.conf or ZCML as well.
In this part, I'd like to have something in the package as well, at least, some guidance of how to do this. Best Regards, Hermann -- hermann@qwer.tk GPG key ID: 299893C7 (on keyservers) FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7
Hermann Himmelbauer wrote:
Am Freitag, 20. Juni 2008 00:14 schrieb Martijn Faassen: [snip]
This now works against the trunk of zope.sqlalchemy as well, as Laurence merged my branch. I do think my code currently requires SQLAlchemy 0.5beta1 or higher.
Does it really? Because in the README.txt doctests, there seems to be a "session.save()" directive that is AFAIK deprecated by "session.add()".
Interesting, I didn't get any deprecation warnings from SQLAlchemy. I'm using autocommit=False, and in the 0.4 era that's transactional=True. [snip]
It doesn't offer any user interface code or schemas itself; that's up to the application or framework developer. It's also flexible. Configuration information could be retrieved from any place a developer would like; hardcoded in Python, or the ZODB. I think it wouldn't be hard to write custom utilities that look up configuration in configuration files such as zope.conf or ZCML as well.
In this part, I'd like to have something in the package as well, at least, some guidance of how to do this.
As long as it doesn't add a lot of dependencies. The package's dependencies are pretty small right now, and I tried to keep it that way. Even though I needed to test setSite() and such, I chose not to depend on zope.app.component as that pulls in the universe. Anyway, it could always be done in another package that builds on z3c.saconfig. Thanks for the feedback! Regards, Martijn
Am Donnerstag, 19. Juni 2008 20:51 schrieb Martijn Faassen:
Hi there,
I'd like to announce my contribution for the expanding list of options for SQLAlchemy integration for Zope 3.
I've just implemented a package called z3c.saconfig which implements a utility-based way to set up SQLAlchemy's scoped session, as discussed recently on this.
The package currently offers a way to configure SQLAlchemy sessions with a utility. This allows SQLAlchemy sessions to potentially differ per application. Currently if you would want to implement such utilities you're still on your own, but I intend to add support for this soon.
Yes, this looks very fine. I also find the documentation in README.txt explanative and well written. However, I have the following thoughts: 1) Why do you need to specify what interface the factory provides, such as here: component.provideUtility(engine_factory, provides=IEngineFactory) component.provideUtility(utility, provides=IScopedSession) Why can't the utilities provide the interface out of the box? 2) I'd suggest to depict the case where an engine is bound to the session via the "bind=" parameter, as not all of us are that advanced in SA, thus it may be helpful. Moreover, you later on write that "setting up an engine factory is not actually necessary", so the reader may ask himself why the engine utility makes sense at all. Perhaps it would be best to sketch the most simple case, with the bind parameter first, then explain what the shortcomings of this case are, and then introduce the engine utility. 3) I'd suggest to explain the part, where you do a "from z3c.saconfig import Session; session = Session()" a little, and line out that it's used in SQLAlchemy style, e.g.: "After registering the session utility, one can import the Session class vom z3c.saconfig, which offers the same capabilities as a common SQLAlchemy session class." 4) In the site examples, it reads: sm1 = site1.getSiteManager() sm1.registerUtility(engine_factory1, provided=IEngineFactory) Why is it now "provided" instead of "provides"? Is this a typo or something specific? 5) For the siteScopeFunc part, it would be best if there would already be a generic one in the SiteScopedSession class, although I don't know if this would be possible. However, this would make things simpler for beginners. Later on I suggest to explain that it's possible to overwrite this method and what it's for. The missing bits in this module seem to be: 1) Some way to update database parameters, e.g. change your engine: In many web applications, database setup is done by the user during installation (e.g. PHProjekt and many others). The user has some install wizard and inputs the database parameters here, moreover he can change them later on via a web frontend. I think there should be some solution/guideline that aids the programmer in this part. What I can think of is: - Simply reregister the engine utility with new parameters - Provide some "updateEngine" helper function, that queries a site/global registry for an IEngineUtility and alters the database parameters there - Somthing like I suggested in my code, where there are specific configuration properties in the utility, that, if changed, recreate the engine. However, it has to be take special care that when changing the engine, open sessions are not somehow corrupted. 2) Basically, I can think of 4 main scenarious for a Zope3 + SA integration: a) 1 database per Zope3 instance b) 1 database per Site c) n databases per Zope3 instance d) n databases per Zope3 Site I suggest to outline that in the beginning of the README.txt along with some introductory words and explain that the setup for these cases differ. Maybe I'd include the case in the heading, such as GloballyScopedSession (for 1 DB per Zope3 instance) ======================================= (a) and (b) seem to be most common and are covered in z3c.saconfig, but (c) and (d) seem to be missing. I don't know how common they are and if it makes sense to put a lot of effort into this. But maybe there's a solution via named utilities? Anyway, I'd outline that scenarious (c) and (d) are missing in this package, so that people know that right away. All in all, thanks a lot for your efforts, this package will be very helpful to others who also need to integrate their application with a RDB! Best Regards, Hermann -- hermann@qwer.tk GPG key ID: 299893C7 (on keyservers) FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7
Hey, Hermann Himmelbauer wrote: [snip]
1) Why do you need to specify what interface the factory provides, such as here:
component.provideUtility(engine_factory, provides=IEngineFactory) component.provideUtility(utility, provides=IScopedSession)
Why can't the utilities provide the interface out of the box?
They do, but then registration will only do the right thing if your utility only implements a single interface. I also think this makes the examples slightly easier to read. Anyway, it's not so important as normally registrations would take place from ZCML or using Grok.
2) I'd suggest to depict the case where an engine is bound to the session via the "bind=" parameter, as not all of us are that advanced in SA, thus it may be helpful. Moreover, you later on write that "setting up an engine factory is not actually necessary", so the reader may ask himself why the engine utility makes sense at all.
Yes, I wasn't sure about this one. Perhaps I should show this example, though it'll expand the example quite a bit and I'm not sure how it's helpful. I think we should encourage people to use the utility style registration for the engine.
Perhaps it would be best to sketch the most simple case, with the bind parameter first, then explain what the shortcomings of this case are, and then introduce the engine utility.
Yes, perhaps. I'm not sure whether that's a good idea in a tutorial; one that shows examples we don't want to encourage first, or the right example right away. Perhaps this could be done in an extra .txt file where we go into more detail about various options.
3) I'd suggest to explain the part, where you do a "from z3c.saconfig import Session; session = Session()" a little, and line out that it's used in SQLAlchemy style, e.g.: "After registering the session utility, one can import the Session class vom z3c.saconfig, which offers the same capabilities as a common SQLAlchemy session class."
You're right, that could use some more information. I've expanded the text.
4) In the site examples, it reads: sm1 = site1.getSiteManager() sm1.registerUtility(engine_factory1, provided=IEngineFactory)
Why is it now "provided" instead of "provides"? Is this a typo or something specific?
It's an annoying inconsistency in the zope.component APIs. It's not a typo. Again this is an API that at least Grok hides away for you.
5) For the siteScopeFunc part, it would be best if there would already be a generic one in the SiteScopedSession class, although I don't know if this would be possible. However, this would make things simpler for beginners. Later on I suggest to explain that it's possible to overwrite this method and what it's for.
I haven't found it easy for z3c.saconfig, as I tried to avoid dependencies on things like zope.traversing (which again pull in the world), or the ZODB. My intent is for z3c.saconfig to be foundational, but that other frameworks will need to fill in some more of the holes. My aim is to use this with megrok.rdb, and this will certainly offer a Grok-specific way to distinguish between applications.
The missing bits in this module seem to be:
1) Some way to update database parameters, e.g. change your engine: In many web applications, database setup is done by the user during installation (e.g. PHProjekt and many others). The user has some install wizard and inputs the database parameters here, moreover he can change them later on via a web frontend. I think there should be some solution/guideline that aids the programmer in this part.
I agree that this is still a feature that's missing and should be carefully tested. I'd like to avoid putting knowledge about user interfaces or the exact specification of SQLAlchemy parameters in z3c.saconfig though. I'd like to offer an infrastructure to reconfigure the engine and then make sure the reconfigured engine gets used, but only the minimal one. Again it's the task of applications or frameworks that build on top of this to use this infrastructure.
What I can think of is:
- Simply reregister the engine utility with new parameters
Hm, reregistering a local utility on the fly is rather hard-core, I'd like to avoid that and allow modification of engine utilities instead.
- Provide some "updateEngine" helper function, that queries a site/global registry for an IEngineUtility and alters the database parameters there - Somthing like I suggested in my code, where there are specific configuration properties in the utility, that, if changed, recreate the engine.
Yes, I saw the property approach. I didn't want to go for the property approach, as I thought if you set 3 properties at once, the engine is disposed 3 times. What I want to offer is an API on IScopedSession and IEngineFactory that allows you to replace the engine parameters. This should dispose the engine, and then allow for a new engine to be created. I already have a sketch of the code on IEngineFactory, taken from your code. That isn't enough though, as the scoped session machinery will cache sessions and engines indefinitely. We also need an API on IScopedSession therefore that can trigger a particular engine (or set of engines) to be throw out from the scoped session. All of this needs careful unit testing. :)
However, it has to be take special care that when changing the engine, open sessions are not somehow corrupted.
Hm, do you have any details of how this can happen and can be avoided?
2) Basically, I can think of 4 main scenarious for a Zope3 + SA integration:
a) 1 database per Zope3 instance b) 1 database per Site c) n databases per Zope3 instance d) n databases per Zope3 Site
I suggest to outline that in the beginning of the README.txt along with some introductory words and explain that the setup for these cases differ.
I've added some text to the introduction and adjusted the headings for case a) and b).
(a) and (b) seem to be most common and are covered in z3c.saconfig, but (c) and (d) seem to be missing.
c) isn't missing, as b) allows a form of c), right? The idea is that if you want multiple databases in an instance, you'd set up multiple sites. d) is indeed not covered. I think it's hard to cover in the current design. Laurence Rowe is interested in this use case I believe, and perhaps we can find ways to make it possible, but I suspect it might have to look quite different and use adapters instead of the Session class to get sessions. This because to know which database applies to which object you need to know more than just what site you are in, but also what object you're dealing with. We could eventually consider expanding z3c.saconfig with a an adapter-driven approach as well to cover this use case. [snip]
All in all, thanks a lot for your efforts, this package will be very helpful to others who also need to integrate their application with a RDB!
Thanks again for your helpful code, and thanks for your extensive feedback! Regards, Martijn
Am Freitag, 20. Juni 2008 13:59 schrieb Martijn Faassen:
Hey,
Hermann Himmelbauer wrote: [snip]
1) Why do you need to specify what interface the factory provides, such as here:
component.provideUtility(engine_factory, provides=IEngineFactory) component.provideUtility(utility, provides=IScopedSession)
Why can't the utilities provide the interface out of the box?
They do, but then registration will only do the right thing if your utility only implements a single interface. I also think this makes the examples slightly easier to read. Anyway, it's not so important as normally registrations would take place from ZCML or using Grok.
Ah, ok, I see.
Perhaps it would be best to sketch the most simple case, with the bind parameter first, then explain what the shortcomings of this case are, and then introduce the engine utility.
Yes, perhaps. I'm not sure whether that's a good idea in a tutorial; one that shows examples we don't want to encourage first, or the right example right away.
Perhaps this could be done in an extra .txt file where we go into more detail about various options.
+1
5) For the siteScopeFunc part, it would be best if there would already be a generic one in the SiteScopedSession class, although I don't know if this would be possible. However, this would make things simpler for beginners. Later on I suggest to explain that it's possible to overwrite this method and what it's for.
I haven't found it easy for z3c.saconfig, as I tried to avoid dependencies on things like zope.traversing (which again pull in the world), or the ZODB. My intent is for z3c.saconfig to be foundational, but that other frameworks will need to fill in some more of the holes. My aim is to use this with megrok.rdb, and this will certainly offer a Grok-specific way to distinguish between applications.
Ah, yes, dependency. Ok, then it should not be included out of the box.
The missing bits in this module seem to be:
1) Some way to update database parameters, e.g. change your engine: In many web applications, database setup is done by the user during installation (e.g. PHProjekt and many others). The user has some install wizard and inputs the database parameters here, moreover he can change them later on via a web frontend. I think there should be some solution/guideline that aids the programmer in this part.
I agree that this is still a feature that's missing and should be carefully tested.
I'd like to avoid putting knowledge about user interfaces or the exact specification of SQLAlchemy parameters in z3c.saconfig though. I'd like to offer an infrastructure to reconfigure the engine and then make sure the reconfigured engine gets used, but only the minimal one. Again it's the task of applications or frameworks that build on top of this to use this infrastructure.
What I can think of is:
- Provide some "updateEngine" helper function, that queries a site/global registry for an IEngineUtility and alters the database parameters there - Somthing like I suggested in my code, where there are specific configuration properties in the utility, that, if changed, recreate the engine.
Yes, I saw the property approach. I didn't want to go for the property approach, as I thought if you set 3 properties at once, the engine is disposed 3 times.
Yes, true, that's not that pretty. However, this has no real bad consequences, I think. A simple "updateEngineParameters" function would also be an option, btw.
What I want to offer is an API on IScopedSession and IEngineFactory that allows you to replace the engine parameters. This should dispose the engine, and then allow for a new engine to be created. I already have a sketch of the code on IEngineFactory, taken from your code. That isn't enough though, as the scoped session machinery will cache sessions and engines indefinitely. We also need an API on IScopedSession therefore that can trigger a particular engine (or set of engines) to be throw out from the scoped session.
Yes, that looks nice.
However, it has to be take special care that when changing the engine, open sessions are not somehow corrupted.
Hm, do you have any details of how this can happen and can be avoided?
I'm not sure about that. I just thought: What happens if someone actually uses a session and the engine is concurrently deleted/recreated from the EngineUtility?
2) Basically, I can think of 4 main scenarious for a Zope3 + SA integration:
a) 1 database per Zope3 instance b) 1 database per Site c) n databases per Zope3 instance d) n databases per Zope3 Site
I suggest to outline that in the beginning of the README.txt along with some introductory words and explain that the setup for these cases differ.
I've added some text to the introduction and adjusted the headings for case a) and b).
Perfect, right.
(a) and (b) seem to be most common and are covered in z3c.saconfig, but (c) and (d) seem to be missing.
c) isn't missing, as b) allows a form of c), right? The idea is that if you want multiple databases in an instance, you'd set up multiple sites.
Hmmm, yes, somehow.
d) is indeed not covered. I think it's hard to cover in the current design. Laurence Rowe is interested in this use case I believe, and perhaps we can find ways to make it possible, but I suspect it might have to look quite different and use adapters instead of the Session class to get sessions. This because to know which database applies to which object you need to know more than just what site you are in, but also what object you're dealing with.
Well, I basically thought about registering engines as named utilities. So, someone would fetch a session explicitly referring to this named utility, e.g.: session = Session('MyDatabase1') And this named utility could also be global or local. So, this is what I thought about (c) and (d).
We could eventually consider expanding z3c.saconfig with a an adapter-driven approach as well to cover this use case.
It seems you think about the idea to bind classes to specific databases. While this is an interesting approach, I'm unsure if this will be practically useful. The point is, that it may happen, that one class may perfectly match to two databases, e.g. when two databases have a table with the same table definitions. So, if someone want's to e.g. migrate data, things could become complicated for the adapter-driven approach. Maybe it's best to wait and see if there is demand in these approaches in this issue. Best Regards, Hermann -- hermann@qwer.tk GPG key ID: 299893C7 (on keyservers) FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7
On Thu, Jun 19, 2008 at 08:51:17PM +0200, Martijn Faassen wrote:
Hi there,
I'd like to announce my contribution for the expanding list of options for SQLAlchemy integration for Zope 3.
Looks good, I'd like to vote for moving some parts of it to zope.sqlalchemy. Namely z3c.saconfig.interfaces.IScopedSession and the entirety of z3c.saconfig.scopedsession. Also for this problem: # XXX what happens if EngineFactory were to be evicted from the ZODB # cache? def getCached(self): return getattr(self, '_v_engine', None) I think you could use the same mechanism found in zope.app.cache.ram. I.e. store the engines in a module level global dictionary. Then use some clever way (with a counter and time) to figure out a unique key for your local utility (and persistently store the key). -- Brian Sutherland
Hi there, Brian Sutherland wrote: [snip]
Also for this problem:
# XXX what happens if EngineFactory were to be evicted from the ZODB # cache? def getCached(self): return getattr(self, '_v_engine', None)
I think you could use the same mechanism found in zope.app.cache.ram. I.e. store the engines in a module level global dictionary. Then use some clever way (with a counter and time) to figure out a unique key for your local utility (and persistently store the key).
Ah, thanks for that tip. Does zope.app.cache.ram deal with threading issues? Ah, yes, I see it uses a lock. Anyway, a patch would be welcome. :) Regards, Martijn
On Fri, Jun 20, 2008 at 02:01:52PM +0200, Martijn Faassen wrote:
Hi there,
Brian Sutherland wrote: [snip]
Also for this problem:
# XXX what happens if EngineFactory were to be evicted from the ZODB # cache? def getCached(self): return getattr(self, '_v_engine', None)
I think you could use the same mechanism found in zope.app.cache.ram. I.e. store the engines in a module level global dictionary. Then use some clever way (with a counter and time) to figure out a unique key for your local utility (and persistently store the key).
Ah, thanks for that tip. Does zope.app.cache.ram deal with threading issues? Ah, yes, I see it uses a lock. Anyway, a patch would be welcome. :)
Committed, please feel free to revert or modify as you feel:)
Regards,
Martijn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Brian Sutherland
Hey, On Sat, Jun 21, 2008 at 9:14 PM, Brian Sutherland <brian@vanguardistas.net> wrote: [snip]
Ah, thanks for that tip. Does zope.app.cache.ram deal with threading issues? Ah, yes, I see it uses a lock. Anyway, a patch would be welcome. :)
Committed, please feel free to revert or modify as you feel:)
Thanks for this! Looks good! Looking at zope.app.cache.ram inspired me to think about a way to encapsulate all this storing-in-ram in a special kind of property. This should hopefully make this kind of code easer to write in the future. Regards, Martijn
participants (3)
-
Brian Sutherland -
Hermann Himmelbauer -
Martijn Faassen