Re: [Zope] Re: Global variables in ExternalMethod modules
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ben Last wrote: |> If this *were* a true module (see below), you would not be getting |> per-thread semantics from it; you would need to keep a mapping keyed by |> thread ID to get such semantics. | | I'd been told a while back that "each Zope thread has its own namespaces | and therefore its own copy of any given module-level variable". Is this | false (given what you say below about ExternalMethod implementations not | being real modules)? Persistent objects are held in per-thread caches; module-level stuff is ~ *not* part of that cache. People sometimes fake out thread-level caches by assigning "volatile" attributes to persistent objects (attributes whose names start with '_v_'); the downside is that such attributes go away whenever the persistent object is "ghostified", which may occur at unpredictable times (e.g., at transaction or subtransaction boundaries). |> The files containing ExternalMethod implementations are not "modules", |> really, and so your expectations about module-level globals are not |> being met. Instead, the EM machinery "execs" the text of the EM file in |> a custom namespacee, and then extracts the code object from it. |> |> I think you are headed into "build a product" territory here, unless you |> want your EM to monkey-patch its cache into an existing module somewhere. | | Ouch. "Build a product" isn't an option for us, for a whole bunch of | reasons. I guess one way to deal with this, then, would be to put the | module-level objects in *another* module and import *that*? | | Does each Zope thread get its own copy of an imported module, or would I | need to use something like Dieter's SharedResource to keep, say, a dict | of connections keyed by thread id? Threads which use "normal" import statements (rather than hacking around with '__import__') won't have separate copies of Python modules. I don't know that product, but it sounds like it might be what you need. Remind me again what you need a per-thread cache for? Could you scribble on the REQUEST object instead? Tres. - -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFB/8UZGqWXf00rNCgRAkdzAJ4/qFY6fMLFn82s+gWl9PVGh3OjGACdEcpH BCjKHO/frTyksThe+y7EMQU= =PL45 -----END PGP SIGNATURE-----
Tres Seaver wrote:
Persistent objects are held in per-thread caches; module-level stuff is ~ *not* part of that cache. People sometimes fake out thread-level caches by assigning "volatile" attributes to persistent objects (attributes whose names start with '_v_'); the downside is that such attributes go away whenever the persistent object is "ghostified", which may occur at unpredictable times (e.g., at transaction or subtransaction boundaries). Thanks very much for this, Tres; much appreciated. I've sorted the problem out thus: Added a module called ThreadShared which provides a thread-local storage dict, obtainable by any thread calling ThreadShared.getTLS() I've then used that to store the per-thread objects I wanted (MySQLdb.Connection objects, primarily) This is working very well now from within ExternalMethods.
Remind me again what you need a per-thread cache for? Could you scribble on the REQUEST object instead? As above; initially for MySQLdb.Connection objects (actually, we have a class that wraps them to make us more db-independent). I do also scribble on the REQUEST object with a _v_ attribute because that makes obtaining the per-thread connection somewhat faster as the request code calls the various database methods. But the connections need to persist between requests (to save continual database re-connection), so I needed somewhere thread-local to store them.
We also have a large number of configuration settings that are read at startup; I've just added code to cache these with the date and time they were read, allowing the cached versions to be used until they expire and are re-read. The database load has been significantly reduced with all of this. Regards Ben
Ben Last (Zope) wrote:
Tres Seaver wrote:
Persistent objects are held in per-thread caches; module-level stuff is ~ *not* part of that cache. People sometimes fake out thread-level caches by assigning "volatile" attributes to persistent objects (attributes whose names start with '_v_'); the downside is that such attributes go away whenever the persistent object is "ghostified", which may occur at unpredictable times (e.g., at transaction or subtransaction boundaries).
Thanks very much for this, Tres; much appreciated. I've sorted the problem out thus: Added a module called ThreadShared which provides a thread-local storage dict, obtainable by any thread calling ThreadShared.getTLS() I've then used that to store the per-thread objects I wanted (MySQLdb.Connection objects, primarily) This is working very well now from within ExternalMethods.
I am glad to hear that.
Remind me again what you need a per-thread cache for? Could you scribble on the REQUEST object instead?
As above; initially for MySQLdb.Connection objects (actually, we have a class that wraps them to make us more db-independent). I do also scribble on the REQUEST object with a _v_ attribute because that makes obtaining the per-thread connection somewhat faster as the request code calls the various database methods. But the connections need to persist between requests (to save continual database re-connection), so I needed somewhere thread-local to store them.
We also have a large number of configuration settings that are read at startup; I've just added code to cache these with the date and time they were read, allowing the cached versions to be used until they expire and are re-read. The database load has been significantly reduced with all of this.
OK. You probably know this already, but ZMySQLDA provides a persistent object which has pretty much exactly the same semantics: it holds all the configuration data for the connection, and is responsible for holding open the actual connection across requests. Because it is persistent, you end up (on a busy site) with as many open connections as you have worker threads. I don't know what made that object unsuitable for your needs, but almost all Zope RDBMS adapters work the same way. Tres. -- =============================================================== Tres Seaver tseaver@zope.com Zope Corporation "Zope Dealers" http://www.zope.com
Tres Seaver wrote:
OK. You probably know this already, but ZMySQLDA provides a persistent object which has pretty much exactly the same semantics: it holds all the configuration data for the connection, and is responsible for holding open the actual connection across requests. Yes, we've investigated ZMySQLDA, but our general policy is to keep our Python code independent rather than rely on Zope objects. This isn't to denigrate Zope, or the quality of its constituent products, but reflects the experiences we've had in building and maintaining a system based upon it, together with a desire to remove database dependencies from the Zope (presentation) layer altogether, and keep them in the business-logic (in ExternalMethods and PythonScripts). We've also wrapped the Connection objects to allow us to do things like divert the connection from the primary to secondary db server if the first one fails; this, as far as we could see, isn't easily achievable if we let Zope products handle the connections.
Because it is persistent, you end up (on a busy site) with as many open connections as you have worker threads. That's exactly the situation we have now, which is working very well. One can tune the MySQLdb server to drop connections after an amount of idle time, and the class in which we wrap the actual MySQLdb.Connection performs a ping() when a Connection is reused, waking the connection up again. Thus the number of simultaneous connections tends to follow the load on the site.
Regards Ben
participants (2)
-
Ben Last (Zope) -
Tres Seaver