Thanks for that. Its a bit ironic though, I had was hoping that i could gain 'relatively safe Zope execution context (automatic transaction management, replicated data per thread, conflict resolution)' : S I have an amount of side dbms processing that can start as soon as a session is started. It just sits stupidly in a dtml-method and is called <dtml-var'd> by a document. Its table intensive and I've noted that Zope crashes more. I have some strange goings on with my setup and this was one thought I had test what the problem is. i've seen in my logs that an opening request that is bumped by a second coming in before its finsihed can take out zope and give 200 - ' ' (as in 200 http with no data) for a few hours and then 503 eventually when zope realizes. It could be anything from the PoPy drivers ./configure 's we had to 'de-linux' for irix or it could be that my site traverses are funny somewhere (about 6 levels per page hit with session, client, interface, content layers). Can you tell me any tests that I can use to check into this? Its really a general reliability problem. The site doesnt get outrageous response, but it in the 100s per day. Thanks to Dieter or anyone who may have some advice, Paz -----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Dieter Maurer Sent: Tuesday, June 12, 2001 10:18 PM To: Paul Zwarts Cc: Zope Subject: Re: [Zope] Asynchronous dtml? Paul Zwarts writes:

If you have time consuming dtml in a method, is it possible to call the DTML method and allow it to run by itself (in paralell), where its completion being independant from the page being written to the client? This will not be easy:

what should your DTML do? 1. build the response to the request? In this case, you would need to wait for the method finishing before the request is completed 2. change the data? in this case, you need a ZODB connection. These connections are the limiting factor for the number of Zope threads. As I understand, there are up to seven connections. In principle, you can spawn a new thread for asnychronous DTML execution. But you must be very careful, as you may leave the relatively safe Zope execution context (automatic transaction management, replicated data per thread, conflict resolution). Dieter _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

albert boulanger writes:

Here is another example use for something like this. The Plumtree Portal Server, to build the content of a mypage, will use multiple threads to collect the content for the table cells of the mypage from what they term gadget servers. It is important to collect this content in parallel otherwise the time to build can be really bad especially if multiple sources time out which can occur more often than one would expect. If one were to build mypages for CMF for example and rely on external content provision (say stock quotes, etc) for any of the sources, one would need something like this. Your use case should be okay:

Threads become only problematic, if they access a database (including the ZODB) (or global data, but that is not Zope specific). If they do access a database, they need to play well with the transaction and persistence machinery of Zope: Zope associates a transaction with each thread. If you create a new thread, you will get a new transaction. The transaction is accessed with "get_transaction()". Zope registers objects with the transaction. These objects are examined when the transaction is commited or aborted in order to make potential changes persistent or discard them, respectively. ZODB objects are registered when they are changed, many other objects, e.g. DatabaseConnections, when they are accessed. Thus, if you change or access transaction aware objects, you may get a pending state, unless you call "get_transaction().commit()" or "get_transaction().abort()" at the end of your thread. Zope associates a ZODB connection with each HTTP request. This is done during traversal of the root object: ZODB.ZApplication.ZApplicationWrapper.__bobo_traverse__ (db.open(version)). This connection is used to load objects on demand from the ZODB. It is propagated from an object to its persistent children during loading of the object. Thus, if you pass a Zope object to a new thread, the new thread uses the original thread's connection when it accesses (children) of the object or modifies the object itself. Apparently, the connection does not protect itself against concurrent use by different threads. This is quite understandable as such protection would cause a significant performance penalty. It is quite likely, that you get non-deterministic difficult to explain and analyse problems when you access the connection from different threads. I expect that even read access is dangerous. You can create a new connection in your thread and then access an object identified by an absolute URL "url" through import Zope root= Zope.app() object= root.unrestrictedTraverse(url) Note, however, that ZODB connections are pooled with a limited pool size (7 or 3 (for a version)). If you try to create more connections your thread blocks until there is an available connection. This may hit ZServer, too, when all available connections are used. Thus, your site may freeze, if you are not careful. Dieter

On Tuesday 12 June 2001 13:17, Dieter Maurer wrote:

...

Paul Zwarts writes:

...

If you have time consuming dtml in a method, is it possible to call the DTML method and allow it to run by itself (in paralell), where its completion being independant from the page being written to the client?

This will not be easy:

what should your DTML do?

1. build the response to the request? In this case, you would need to wait for the method finishing before the request is completed

2. change the data?

in this case, you need a ZODB connection. These connections are the limiting factor for the number of Zope threads. As I understand, there are up to seven connections.

In principle, you can spawn a new thread for asnychronous DTML execution. But you must be very careful, as you may leave the relatively safe Zope execution context (automatic transaction management, replicated data per thread, conflict resolution).

its possible to write async dtml method without spawning a new thread by integrating with asyncore. although its tricky you can still get transaction integrity by having by having a before async call transaction and another transaction when the method returns (it will probably get serviced by another thread). except for some really extreme cases this is not nesc and normally the problem is better solved by other means. one case that does come to mind, is when you have potentially long external interaction with foriegn servers. if anyone's interested in some sample code, feel free to email me. cheers kapil