Dear Zopistas I'm trying to work out what the rules are for global (that is, module-level) variables in a module that contains external methods. The scenario: we have a single module (.py file) in the Extensions subdirectory of the instance directory, which contains all our external methods plus many helper methods. Amongst other things, this module contains a lot of methods that perform database queries to an associated MySQL database. The same module is also used by non-Zope Python tasks, so the queries are all direct, not via Zope SQL products. As part of the ongoing optimization of the code, I've been looking at having a single MySQLdb connection per Zope thread, and re-using this for all SQL queries. My understanding was that if I declare a connection variable at module level, like so: databaseConnection = None I could then initialize it in a method that's called at the start of handling every request, but re-use the value between requests. However, it *appears* that the value of the databaseConnection variable is (nearly) always None when any new request starts, as though the module were being reloaded afresh for every one. Thus there seems to be no way to keep per-thread module-level variables. This is on our test platform, where the Zope instance is running in debug-mode. We've also noticed an odd connection leak on our production server, where Zope is not running in debug-mode. It *appears* as though the __del__ methods of MySQLdb.Connection objects are not being called (and thus the connections are remaining open indefinitely). We've recently started wrapping Connection objects in a class of our own, and tracing __init__ and __del__ calls for that also appears to bear out that theory. All this on Zope 2.7.3, Python 2.3.4, Linux. So; is there a quick-n-easy summary of how module-level variables behave in ExtensionMethod modules? Is there any reason that, in a non-debug environment, the __del__ methods of simple Python objects may not be called? Regards to all Ben Last
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ben Last (Zope) wrote: | I'm trying to work out what the rules are for global (that is, | module-level) variables in a module that contains external methods. | | The scenario: we have a single module (.py file) in the Extensions | subdirectory of the instance directory, which contains all our external | methods plus many helper methods. Amongst other things, this module | contains a lot of methods that perform database queries to an associated | MySQL database. The same module is also used by non-Zope Python tasks, | so the queries are all direct, not via Zope SQL products. | | As part of the ongoing optimization of the code, I've been looking at | having a single MySQLdb connection per Zope thread, and re-using this | for all SQL queries. My understanding was that if I declare a | connection variable at module level, like so: | | databaseConnection = None If this *were* a true module (see below), you would not be getting per-thread semantics from it; you would need to keep a mapping keyed by thread ID to get such semantics. | I could then initialize it in a method that's called at the start of | handling every request, but re-use the value between requests. However, | it *appears* that the value of the databaseConnection variable is | (nearly) always None when any new request starts, as though the module | were being reloaded afresh for every one. Thus there seems to be no way | to keep per-thread module-level variables. | | This is on our test platform, where the Zope instance is running in | debug-mode. | | We've also noticed an odd connection leak on our production server, | where Zope is not running in debug-mode. It *appears* as though the | __del__ methods of MySQLdb.Connection objects are not being called (and | thus the connections are remaining open indefinitely). We've recently | started wrapping Connection objects in a class of our own, and tracing | __init__ and __del__ calls for that also appears to bear out that theory. | | All this on Zope 2.7.3, Python 2.3.4, Linux. | | So; is there a quick-n-easy summary of how module-level variables behave | in ExtensionMethod modules? Is there any reason that, in a non-debug | environment, the __del__ methods of simple Python objects may not be | called? | | Regards to all The files containing ExternalMethod implementations are not "modules", really, and so your expectations about module-level globals are not being met. Instead, the EM machinery "execs" the text of the EM file in a custom namespacee, and then extracts the code object from it. I think you are headed into "build a product" territory here, unless you want your EM to monkey-patch its cache into an existing module somewhere. Tres. - -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFB/7yiGqWXf00rNCgRAmA/AJwMeG9ESBmk0xls8FLeYXWR4GRBOQCdGJUN 123OT26+4dvVAguLuJ5k8so= =cMTR -----END PGP SIGNATURE-----
Tres Seaver wrote:
If this *were* a true module (see below), you would not be getting per-thread semantics from it; you would need to keep a mapping keyed by thread ID to get such semantics. I'd been told a while back that "each Zope thread has its own namespaces and therefore its own copy of any given module-level variable". Is this false (given what you say below about ExternalMethod implementations not being real modules)?
The files containing ExternalMethod implementations are not "modules", really, and so your expectations about module-level globals are not being met. Instead, the EM machinery "execs" the text of the EM file in a custom namespacee, and then extracts the code object from it.
I think you are headed into "build a product" territory here, unless you want your EM to monkey-patch its cache into an existing module somewhere. Ouch. "Build a product" isn't an option for us, for a whole bunch of reasons. I guess one way to deal with this, then, would be to put the module-level objects in *another* module and import *that*?
Does each Zope thread get its own copy of an imported module, or would I need to use something like Dieter's SharedResource to keep, say, a dict of connections keyed by thread id? Many thanks for your help Regards Ben
Hi, I'm looking to implement a product to index a list of non Zope sites (ideally with ZCatalog) and then make them available for searching via Zope. Anyone done (or tried) something like this before? In 2002, Agata posted the following similar question, but received no response: http://mail.zope.org/pipermail/zope-dev/2002-September/017373.html I have some budget to develop it if necessary (or I can just implement something like ASPSeek.org) but wondering if anyone else had tackeled a similar issue and what their conclusion was. Cheers, Duane
----- Original Message ----- From: "Duane Raymond" <duane.raymond@gmail.com>
I'm looking to implement a product to index a list of non Zope sites (ideally with ZCatalog) and then make them available for searching via Zope. Anyone done (or tried) something like this before?
Have a look at www.shopwiser.com The back-end crawls web sites and then loads the data into a zcatalog. The front-end enables users to search the zcatalog and displays/formats the results. This is an in-house project which was developed using dtml, scripts and external methods (ie. its not a 'product'). So the short answer is 'yes, it can be done! (and has been many times i am sure)'. Jonathan
From: "Duane Raymond"
I'm looking to implement a product to index a list of non Zope sites (ideally with ZCatalog) and then make them available for searching via Zope. Anyone done (or tried) something like this before? From: "Jonathan Hobbs" Have a look at www.shopwiser.com
The back-end crawls web sites and then loads the data into a zcatalog. The front-end enables users to search the zcatalog and displays/formats the results. This is an in-house project which was developed using dtml, scripts and external methods (ie. its not a 'product').
Any chance this 'code' could be shared for the purposes of turning it into a product? Anyone else done this? Cheers, Duane
On Wed, Feb 02, 2005 at 01:53:55PM +0000, Duane Raymond wrote:
Hi,
I'm looking to implement a product to index a list of non Zope sites (ideally with ZCatalog) and then make them available for searching via Zope. Anyone done (or tried) something like this before?
This is about indexing data from an RDBMS, but it might be helpful: http://zope.org/Members/rbickers/cataloganything unfortunately there are some formatting errors, maybe the wrong flavor of STX processing is being applied to it. -- Paul Winkler http://www.slinkp.com
participants (5)
-
Ben Last (Zope) -
Duane Raymond -
Jonathan Hobbs -
Paul Winkler -
Tres Seaver