Re: [Zope] Asynchronous DTML
Thanks Chris, Anybody else able to shed some light on my problem? If the promises of .NET are realised we should be doing a lot more calling out to external servers like this from Zope. ---------------------------------------------------------------------------- --------------------------------- W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz "making decisions easy" ---------------------------------------------------------------------------- --------------------------------- ----- Original Message ----- From: "Chris McDonough" <chrism@digicool.com> To: "W. Robert Kellock" <sales@creditscore.co.nz> Sent: Friday, 5 October 2001 14:32 Subject: Re: [Zope] Asynchronous DTML
Hmm... not exactly a socket programming guru here.. would you mind sending this to the list? I wonder if there's a system call somewhere in here that locks a mutex. I'm not sure but someone else might know.
Just for a sanity check, you might want to write an external method like:
import thread, time, zLOG
def test(self): ident = thread.get_ident() zLOG.LOG('test', 0, "starting in %s at %s" % (ident, time.time())) time.sleep(10) zLOG.LOG('test', 0, " ending in %s at %s" % (ident, time.time()))
Then fire off a few requests to it to prove to yourself that it's not serializing the calls (if no time periods in the log overlap, it's serializing). Youc an see the log by starting Zope up like this:
./start STUPID_LOG_FILE=debug.log
- C
W. Robert Kellock wrote:
The following external method definitely blocks on my RedHat 7.1 Linux box. Any ideas?
from socket import * import time import string
def GetBureau(self,host,port,request,terminator): s = socket(AF_INET,SOCK_STREAM) s.connect((host,port)) s.setblocking(0) #non-blocking to allow us to time out s.send(request) start = time.time() data = '' buffer = '' cont = 1 while cont: cont = 0 try: buffer = s.recv(256) except: pass if len(buffer): cont = 1 last = string.count(buffer,terminator) if (last > 0): cont = 0 buffer = buffer[:string.find(buffer,terminator) + len(terminator)] data = data + buffer buffer = '' else: if (start + 60 > time.time()):# time out if no response cont = 1 else: if (len(data) > 0): pass else: data = '' s.close() return data
-------------------------------------------------------------------------- --
--------------------------------- W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz "making decisions easy"
-------------------------------------------------------------------------- --
---------------------------------
----- Original Message ----- From: "Chris McDonough" <chrism@digicool.com> To: "W. Robert Kellock" <sales@creditscore.co.nz> Cc: <zope@zope.org> Sent: Friday, 5 October 2001 13:56 Subject: Re: [Zope] Asynchronous DTML
Zope is multithreaded and there are typically no locks on any resources that prevents one thread from operating independently of another, so what you want is likely available out of the box.
W. Robert Kellock wrote:
Hello,
Does anyone know how to prevent an external method from blocking?
What I need is an external method that contacts an external server and returns the result to be displayed in my dtml page. It's quite alright for the person requesting the page to wait for the response. What I don't want is for other users to be blocked while the external server formulates it's response to that user.
From the archives it seems I need to piggyback onto Zopes asyncore / asynchat to achieve this, but I'm not sure how.
Thanks.
--------------------------------------------------------------------------
-----------------------------------
W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz <http://www.creditscore.co.nz> "making decisions easy"
--------------------------------------------------------------------------
-----------------------------------
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
On Thursday 04 October 2001 07:32 pm, W. Robert Kellock wrote:
Thanks Chris,
Anybody else able to shed some light on my problem?
there are some cases where blocking a thread can be problematic, talking for long periods of times to external servers can limit your throughput. if you're blocking for 20-30s waiting on an external server and you have 4 threads, and you have 3hits/sec, than most of your users will never see your site. i've done some experiments with using allowing async calls to be initiated to external servers from dtml, its possible, its not pretty and has some caveats that means its not for the faint of heart. also for most of the use cases there might well be an alternate way to structure the system that will be better. indeed if you expect alot of this type of thing bumping up your thread count and using zeo would be an easy way out. the biggest caveat with using async in zope that gets initiated from a web request is you basically lose your transaction (not in the sense of an abort) because zope transactions are request associated (well publishing more specifically) and thread specific. while your conn to the external server is still in asyncore loop, the original web request and publisher will have commited the transaction, even if you stopped that, when you enter asyncore the thread doing the select poll is a different thread and since transactions are keyed to threads you'd get a different transaction. (and its also probably not a good idea to do any long post result processing from here as holding the medusa thread messes with the dispatching of i/o and publishing requests back to zope.) so if you're don't really care about transactions and you're not doing any heavy post request processing than sure its doable... btw. if you want to learn more about (async) network programming i can recommend some resources www.kegel.com/c10k.html www.nightmare.com # the home of asyncore /medusa richard stevens network programming # the bible of a network programmer patterns of software architecture vol II patterns for network and concurrent programs if this hasn't scared you away... well than its probably possible to do the whole thing transactionally. jon heinz (sp?) posted a transaction replacement that he was using for integrating with corba that allows more explicit transaction control. never said it would be easy...
If the promises of .NET are realised we should be doing a lot more calling out to external servers like this from Zope.
if you're interested in this you should check out the web-services project on dev.zope.org, and bug certain people to keep it up to date:) cheers kapil thangavelu
----- Original Message ----- From: "Chris McDonough" <chrism@digicool.com> To: "W. Robert Kellock" <sales@creditscore.co.nz> Sent: Friday, 5 October 2001 14:32 Subject: Re: [Zope] Asynchronous DTML
Hmm... not exactly a socket programming guru here.. would you mind sending this to the list? I wonder if there's a system call somewhere in here that locks a mutex. I'm not sure but someone else might know.
Just for a sanity check, you might want to write an external method like:
import thread, time, zLOG
def test(self): ident = thread.get_ident() zLOG.LOG('test', 0, "starting in %s at %s" % (ident, time.time())) time.sleep(10) zLOG.LOG('test', 0, " ending in %s at %s" % (ident, time.time()))
Then fire off a few requests to it to prove to yourself that it's not serializing the calls (if no time periods in the log overlap, it's serializing). Youc an see the log by starting Zope up like this:
./start STUPID_LOG_FILE=debug.log
- C
W. Robert Kellock wrote:
The following external method definitely blocks on my RedHat 7.1 Linux
box.
Any ideas?
from socket import * import time import string
def GetBureau(self,host,port,request,terminator): s = socket(AF_INET,SOCK_STREAM) s.connect((host,port)) s.setblocking(0) #non-blocking to allow us to time out s.send(request) start = time.time() data = '' buffer = '' cont = 1 while cont: cont = 0 try: buffer = s.recv(256) except: pass if len(buffer): cont = 1 last = string.count(buffer,terminator) if (last > 0): cont = 0 buffer = buffer[:string.find(buffer,terminator) + len(terminator)] data = data + buffer buffer = '' else: if (start + 60 > time.time()):# time out if no response cont = 1 else: if (len(data) > 0): pass else: data = '' s.close() return data
------------------------------------------------------------------------- -
participants (2)
-
kapil thangavelu -
W. Robert Kellock