Recently people were commenting on the recent session tracking HOWTO, which stores sessions in files inside a directory. This would be made much easier if there was some API for creating a single connection to a new ZODB; your code would have to retrieve the relevant root object, but you could at least count on it being cleaned up nicely when the server is shut down. Another application for this is my ODP code, which is an amount of data so large that you probably don't want to mix it in with the rest of your database. If we have a powerful object persistency system in the form of the ZODB, why not use it? In the ODP code release, I have an external method which looks like this: from ZODB import FileStorage, DB # Cache of ZODB objects ZODBs = {} def get_odp(filename): if ZODBs.has_key( filename ): db = ZODBs[filename] else: # Open ZODB fs = FileStorage.FileStorage(filename, read_only = 1) db = DB( fs ) ZODBs[ filename ] = db conn = db.open() root = conn.root() return root['ODP'] This may well be leaking connections; I'm not sure. Why not add some function to the Zope API which does this job? You would provide it with a Storage class (so you can use a non-versioning storage for transient sessions) and an argument tuple for the Storage class; if the Storage had been previously opened, it's returned from a cache. Invocation would be something like: conn = Zope.App.openExtraZODB( FileStorage, ('/tmp/mystorage.fs',) ) root = conn.root() ... whatever ... Thoughts? I'm happy to write such a function, if someone will tell me where it should go. -- A.M. Kuchling http://starship.python.net/crew/amk/ It was a wasted life, but God forbid that one should be hard upon it, or upon anything in this world that is not deliberately and coldly wrong . . . -- Charles Dickens, in a letter to his friend John Forster.
On Tue, 19 Oct 1999, Andrew M. Kuchling wrote: [snip]
conn = Zope.App.openExtraZODB( FileStorage, ('/tmp/mystorage.fs',) ) root = conn.root() ... whatever ...
Thoughts? I'm happy to write such a function, if someone will tell me where it should go.
This is cool! What would be even better is an example of a Folderish mixin which instead of storing its sub-objects within the main db it stores it within an external ZODB. This would be effectively "mounting" a filesystem within another but still keeping them separate. Anyone know how to do this? Taking this one-step-further we could enhance Products such that Products using this mixin will automatically store the contents of stuff within its Control Panel folder (like ZClasses) in a ZODB within its real world directory. This means we can just zip it up to get both external Python code AND the ZClasses that it uses. If I knew how the ZODB system works I'd help out ;). Cheers, Anthony Pfrunder
Anthony Pfrunder wrote:
On Tue, 19 Oct 1999, Andrew M. Kuchling wrote:
[snip]
conn = Zope.App.openExtraZODB( FileStorage, ('/tmp/mystorage.fs',) ) root = conn.root() ... whatever ...
Thoughts? I'm happy to write such a function, if someone will tell me where it should go.
This is cool! What would be even better is an example of a Folderish mixin which instead of storing its sub-objects within the main db it stores it within an external ZODB. This would be effectively "mounting" a filesystem within another but still keeping them separate. Anyone know how to do this?
This is something we plan to support at some point, possibly soon. It's somewhat complicated because you have to manage the connections to the various databases. The connection management is rather tricky because of the object references accross connections. An object in one connection cache may reference an object in another connection.
Taking this one-step-further we could enhance Products such that Products using this mixin will automatically store the contents of stuff within its Control Panel folder (like ZClasses) in a ZODB within its real world directory. This means we can just zip it up to get both external Python code AND the ZClasses that it uses. If I knew how the ZODB system works I'd help out ;).
Through the web products already zip up database objects, including ZClasses, and code for you when you create an object distribution. Currently, the data gets imported into the destination Zope. We are considering having products contain read-only databases that get used directly, rather than getting imported. This would allow some interesting optimizations and would almost certainly be easier to integrate. Stay tuned. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
"Andrew M. Kuchling" wrote:
Recently people were commenting on the recent session tracking HOWTO, which stores sessions in files inside a directory. This would be made much easier if there was some API for creating a single connection to a new ZODB; your code would have to retrieve the relevant root object, but you could at least count on it being cleaned up nicely when the server is shut down. Another application for this is my ODP code, which is an amount of data so large that you probably don't want to mix it in with the rest of your database. If we have a powerful object persistency system in the form of the ZODB, why not use it?
In the ODP code release, I have an external method which looks like this:
from ZODB import FileStorage, DB
# Cache of ZODB objects ZODBs = {}
def get_odp(filename): if ZODBs.has_key( filename ): db = ZODBs[filename] else: # Open ZODB fs = FileStorage.FileStorage(filename, read_only = 1) db = DB( fs ) ZODBs[ filename ] = db
conn = db.open() root = conn.root() return root['ODP']
This may well be leaking connections; I'm not sure.
It is. After your through with the object, you should close the connection. Also, ExternalMethod files are not really modules and are not good places to put things like your ZODB cache.
Why not add some function to the Zope API which does this job?
Which job? What part do you want to automate?
You would provide it with a Storage class (so you can use a non-versioning storage for transient sessions) and an argument tuple for the Storage class; if the Storage had been previously opened, it's returned from a cache. Invocation would be something like:
conn = Zope.App.openExtraZODB( FileStorage, ('/tmp/mystorage.fs',) ) root = conn.root() ... whatever ...
What do you intend to do with the objects you get this way? Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
Jim Fulton writes:
It is. After your through with the object, you should close the connection.
Maybe it should return the connection instead, then, so the caller can retrieve the root object, do whatever with it, and then close the connection.
Which job? What part do you want to automate?
Checking whether the Storage for this ZODB has already been opened, and ensuring that everything is neatly cleaned up when you're done.
What do you intend to do with the objects you get this way?
Whatever you like -- call methods on them, mostly, and possibly modify them. The two applications I have in mind are, first, the massive Open Directory Project tree, and second, persistent session tracking that would survive a Zope crash or system reboot (needed when people are assembling sequences of process fabrication steps). In the first case, the database is read-only; in the second case, a non-versioning storage can be used, because we don't want to track every single edit. -- A.M. Kuchling http://starship.python.net/crew/amk/ Mind you, I'm not wild about computers myself, but they are a tool. If you have a tool, it's stupid not to use it. -- The Doctor, in "Inferno"
"Andrew M. Kuchling" wrote:
Jim Fulton writes:
What do you intend to do with the objects you get this way?
Whatever you like -- call methods on them, mostly, and possibly modify them. The two applications I have in mind are, first, the massive Open Directory Project tree, and second, persistent session tracking that would survive a Zope crash or system reboot (needed when people are assembling sequences of process fabrication steps). In the first case, the database is read-only; in the second case, a non-versioning storage can be used, because we don't want to track every single edit.
Very interesting.. I just want to chime in here saying that this sounds interesting. :) AOL-ly yours, Martijn
"Andrew M. Kuchling" wrote:
Jim Fulton writes:
It is. After your through with the object, you should close the connection.
Maybe it should return the connection instead, then, so the caller can retrieve the root object, do whatever with it, and then close the connection.
Take a look at the ZODB.ZApplication module. (And the ZPublisher package in the ZODB model.) It illustrates an approach for automagic connection management.
Which job? What part do you want to automate?
Checking whether the Storage for this ZODB has already been opened, and ensuring that everything is neatly cleaned up when you're done.
What do you intend to do with the objects you get this way?
Whatever you like -- call methods on them, mostly, and possibly modify them.
If you modify them, then you need to interact with teh transaction machinery. You have to define transaction boundaries yourself, or you need to let Zope do it. If you let Zope do it, then you need to keep the connection open until the end of the request, or at least until the end of the transaction. There's an (untested) hook for keeping a connection open until the end of the current transaction. See the documentation for the database open method. ZApplication illustrates how to keep a connection open until the end of a request.
The two applications I have in mind are, first, the massive Open Directory Project tree, and second, persistent session tracking that would survive a Zope crash or system reboot (needed when people are assembling sequences of process fabrication steps). In the first case, the database is read-only; in the second case, a non-versioning storage can be used, because we don't want to track every single edit.
We plan to provide the ability to combine multiple databases in a single object space. There are alot of subtle issues with doing this correctly. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
participants (4)
-
Andrew M. Kuchling -
Anthony Pfrunder -
Jim Fulton -
Martijn Faassen