I have a product which needs to maintain a connection to a 300MB database file (Metakt, if you must know). The initial connection takees about 3 seconds, due to the size of the file, so I wanted to make a product that will keep the connection open, and re-use it. I made a python base class which opens the connection, but I get the following error from zope when it tries to pickle the object into the ZODB. Traceback (innermost last): File /home/ultraseek/Zope-2.1.6-src/lib/python/ZPublisher/Publish.py, line 214, in publish_module File /home/ultraseek/Zope-2.1.6-src/lib/python/ZPublisher/Publish.py, line 179, in publish File /home/ultraseek/Zope-2.1.6-src/lib/python/Zope/__init__.py, line 202, in zpublisher_exception_hook (Object: RoleManager) File /home/ultraseek/Zope-2.1.6-src/lib/python/ZPublisher/Publish.py, line 169, in publish File /home/ultraseek/Zope-2.1.6-src/lib/python/ZODB/Transaction.py, line 237, in commit File /home/ultraseek/Zope-2.1.6-src/lib/python/ZODB/Connection.py, line 325, in commit (Info: (('*yr88KIATmOLZfiZXPxWNeA==', 'ODP'), '\000\000\000\000\000\000\000\262', '')) UnpickleableError: (see above) I don't actually need the database connection itself to be in the ZODB, as it will not be open the next time zope restarts, anyway, but I would like it to remain resident in memory so that I can use it without reconnecting on every request. Additonally, it would be gat to be able to detect when the connection has gone down, and re-create it. Any tips? I am sure it can be done, but I have no idea where to even start, as far as adding something to the global namespace --sam
Sam Gendler wrote:
I have a product which needs to maintain a connection to a 300MB database file (Metakt, if you must know). The initial connection takees about 3 seconds, due to the size of the file, so I wanted to make a product that will keep the connection open, and re-use it.
Cool - Metakit would be really useful for a lot of stuff (sessions, caching). Any chance for a public release? As for your question - you can store the connection in a volatile variable by prefixing it's name with _v_, e.g. _v_mk_connection. You then have to reconnect only when the object is loaded into memory from the ZODB. You do this in the __setstate__ method: def __setstate__(self, state): """ initialize cache when object gets unpickled from zodb """ # if you don't do this horrible things might happen: Persistent.__setstate__(self, state) #create volatile variables: _v_mkconnection = self.mkconnect() Check out ZCatalog for a real example. -- Itamar S.T. itamars@ibm.net
[Sam Gendler, on Sun, 09 Apr 2000] :: I have a product which needs to maintain a connection to a 300MB :: database file (Metakt, if you must know). The initial connection takees :: about 3 seconds, due to the size of the file, so I wanted to make a :: product that will keep the connection open, and re-use it. This is timely. I'm evaluating Metakit for a project right now. This is currently implemented as a Python CGI, with responses appended to flat files which are FTP'd to a mainframe nightly for batch processing. Now, we want to add a local store which can be queried ad hoc for marketing purposes. (Zope may or may not be part of the picture.) Is concurrency an issue with your application? If so, how are you dealing with it? In our case, many simultaneous users could fill out forms at the same time.
Patrick Phalen wrote:
[Sam Gendler, on Sun, 09 Apr 2000] :: I have a product which needs to maintain a connection to a 300MB :: database file (Metakt, if you must know). The initial connection takees :: about 3 seconds, due to the size of the file, so I wanted to make a :: product that will keep the connection open, and re-use it.
This is timely. I'm evaluating Metakit for a project right now. This is currently implemented as a Python CGI, with responses appended to flat files which are FTP'd to a mainframe nightly for batch processing. Now, we want to add a local store which can be queried ad hoc for marketing purposes. (Zope may or may not be part of the picture.)
Is concurrency an issue with your application? If so, how are you dealing with it? In our case, many simultaneous users could fill out forms at the same time.
Currently, I am preloading a database with data from the Open Directory Project, and then accessing the Metakit from zope in read only mode. I have yet to really delve into the Metakit code to see if I need to worry about concurrency when reading, but if so, I guess I will have no choice but to wrap my class' get function with a mutex. Actually, I was forced into Metakit, as I am using Andrew Kuchling's Oedipus product, but I fully intend to migrate it over to BerkeleyDB (sleepycat), in order to take advantage of remote acess, thread safety, etc. Oops, the girlfriend is dragging me out of the house. Checkout out Andrew's product from starship.python.org (search for his name, or look for Oedipus). You will only need the code in database.py, which has a wrapper class for Metakit. --sam
On Sun, 9 Apr 2000, Sam Gendler wrote:
I made a python base class which opens the connection, but I get the following error from zope when it tries to pickle the object into the ZODB.
I don't actually need the database connection itself to be in the ZODB, as it will not be open the next time zope restarts, anyway, but I would like it to remain resident in memory so that I can use it without
Sounds like you need to store the actual metakit connection handle in a volatile attribute ('_v_myconnection' instead of just 'myconnection'). These attributes are not stored in the ZODB and destroyed if/when the object is swapped out of the ZODB memory cache. Note that you will have one connection per Zope thread. If metakit isn't thread safe, you will need to store the connection as a package global or class attribute and provide appropriate locking mechanisms. The connection can be reopened whenever the class is loaded into memory using your products __setstate__ method: def _gethandle(self): if self._v_handle is None or notvalid(self._v_handle): self._v_handle = metakitconnect() return self._v_handle def __setstate__(self,state): "Initialize on reload from ZODB" self._v_handle = None self._gethandle() -- ___ // Zen (alias Stuart Bishop) Work: zen@cs.rmit.edu.au // E N Senior Systems Alchemist Play: zen@shangri-la.dropbear.id.au //__ Computer Science, RMIT WWW: http://www.cs.rmit.edu.au/~zen
Is there an opposite function to __setstate__, which would allow me to close the connection before it gets blown away. I am fairly certain that the connection will close during garbage collection, but it is better to be safe than sorry. Also, what are the valid values for the state variable. I will check the ZQR after I send this, and let you know if I have found the answers. If you don't hear from me, I am still in need. --sam
def _gethandle(self): if self._v_handle is None or notvalid(self._v_handle): self._v_handle = metakitconnect() return self._v_handle
def __setstate__(self,state): "Initialize on reload from ZODB" self._v_handle = None self._gethandle()
-- ___ // Zen (alias Stuart Bishop) Work: zen@cs.rmit.edu.au // E N Senior Systems Alchemist Play: zen@shangri-la.dropbear.id.au //__ Computer Science, RMIT WWW: http://www.cs.rmit.edu.au/~zen
On Sun, 09 Apr 2000 22:56:05 -0700, Sam Gendler <sgendler@silcom.com> wrote:
Is there an opposite function to __setstate__, which would allow me to close the connection before it gets blown away.
You are looking for __del__ Toby Dickenson tdickenson@geminidataloggers.com
Toby Dickenson wrote:
On Sun, 09 Apr 2000 22:56:05 -0700, Sam Gendler <sgendler@silcom.com> wrote:
Is there an opposite function to __setstate__, which would allow me to close the connection before it gets blown away.
You are looking for __del__
Toby Dickenson tdickenson@geminidataloggers.com
Actually, I was thinking about a function that gets called just before the object gets pickled for insertion into the ZODB, which would allow me to set a 'connected' variable to some meaningfule value. Setting a variable just before garbage collection is fairly meaningless. The database connection already has a close call in the destructor. --sam
On Sun, 09 Apr 2000 22:56:05 -0700, Sam Gendler <sgendler@silcom.com> wrote:
Is there an opposite function to __setstate__, which would allow me to close the connection before it gets blown away.
On Mon, 10 Apr 2000 00:48:36 -0700, Sam Gendler <sgendler@silcom.com> wrote:
Actually, I was thinking about a function that gets called just before the object gets pickled for insertion into the ZODB, which would allow me to set a 'connected' variable to some meaningfule value.
From what youve said, I'm confused about _why_ you want this. But, the method you are looking for is __getstate__
Toby Dickenson tdickenson@geminidataloggers.com
participants (5)
-
Itamar Shtull-Trauring -
Patrick Phalen -
Sam Gendler -
Stuart 'Zen' Bishop -
Toby Dickenson