Hello, Does anyone know how to prevent an external method from blocking? What I need is an external method that contacts an external server and returns the result to be displayed in my dtml page. It's quite alright for the person requesting the page to wait for the response. What I don't want is for other users to be blocked while the external server formulates it's response to that user.
From the archives it seems I need to piggyback onto Zopes asyncore / asynchat to achieve this, but I'm not sure how.
Thanks. ------------------------------------------------------------------------------------------------------------- W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz "making decisions easy" -------------------------------------------------------------------------------------------------------------
Zope is multithreaded and there are typically no locks on any resources that prevents one thread from operating independently of another, so what you want is likely available out of the box. W. Robert Kellock wrote:
Hello,
Does anyone know how to prevent an external method from blocking?
What I need is an external method that contacts an external server and returns the result to be displayed in my dtml page. It's quite alright for the person requesting the page to wait for the response. What I don't want is for other users to be blocked while the external server formulates it's response to that user.
From the archives it seems I need to piggyback onto Zopes asyncore / asynchat to achieve this, but I'm not sure how.
Thanks.
------------------------------------------------------------------------------------------------------------- W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz <http://www.creditscore.co.nz> "making decisions easy" -------------------------------------------------------------------------------------------------------------
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
+-------[ Chris McDonough ]---------------------- | Zope is multithreaded and there are typically no locks on any resources | that prevents one thread from operating independently of another, so | what you want is likely available out of the box. What about DNS lookups? -- Totally Holistic Enterprises Internet| | Andrew Milton The Internet (Aust) Pty Ltd | | ACN: 082 081 472 ABN: 83 082 081 472 | M:+61 416 022 411 | Carpe Daemon PO Box 837 Indooroopilly QLD 4068 |akm@theinternet.com.au|
It depends on the host system's gethostbyaddr, I suppose. Is there a reason you can't concurrently do more than one DNS lookup that you can think of? Andrew Kenneth Milton wrote:
+-------[ Chris McDonough ]---------------------- | Zope is multithreaded and there are typically no locks on any resources | that prevents one thread from operating independently of another, so | what you want is likely available out of the box.
What about DNS lookups?
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
+-------[ Chris McDonough ]---------------------- | It depends on the host system's gethostbyaddr, I suppose. Is there a | reason you can't concurrently do more than one DNS lookup that you can | think of? Well first you're in a system call, so your whole process will block while it's inside kernel space, especially on those systems whose kernels don't have threaded kernels. If it takes two minutes for your systemcall to complete, then your process will block for two minutes. gethostbyname also uses static storage to return results. This is why Netscape, Squid, Mozilla et. al have DNS helper applications that do DNS lookups in a seperate forked process (not thread). They also have alarms around it to interrupt the call after about 30 seconds, since it's probably not coming back if it takes that long. You have to explicitly use gethostbyname_r to get the MP-Safe resolver, I don't know if Python does this (I'm not a python internals guru), that's why I ask. Calling _r routines doesn't guarantee that you'll get concurrent lookups either, it just means that multiple threads calling gethostbyname_r won't trash each other. -- Totally Holistic Enterprises Internet| | Andrew Milton The Internet (Aust) Pty Ltd | | ACN: 082 081 472 ABN: 83 082 081 472 | M:+61 416 022 411 | Carpe Daemon PO Box 837 Indooroopilly QLD 4068 |akm@theinternet.com.au|
Well, this is all news to me. ;-) Maybe this has something to do with the problem that kicked off this thread. I thought that the BSD calls had implied threadsafety. But I guess they don't from your description. But in this case, if there was no lock inside the system call, why do the calls serialize? You would think that they'd go right ahead and make a mess of things, but explicitly *not* block if they weren't threadsafe. All of this is kinda dumb. I want a system to think about all this for me and do the right thing. I've paid my dues already. ;-) - C Andrew Kenneth Milton wrote:
+-------[ Chris McDonough ]---------------------- | It depends on the host system's gethostbyaddr, I suppose. Is there a | reason you can't concurrently do more than one DNS lookup that you can | think of?
Well first you're in a system call, so your whole process will block while it's inside kernel space, especially on those systems whose kernels don't have threaded kernels. If it takes two minutes for your systemcall to complete, then your process will block for two minutes. gethostbyname also uses static storage to return results. This is why Netscape, Squid, Mozilla et. al have DNS helper applications that do DNS lookups in a seperate forked process (not thread). They also have alarms around it to interrupt the call after about 30 seconds, since it's probably not coming back if it takes that long.
You have to explicitly use gethostbyname_r to get the MP-Safe resolver, I don't know if Python does this (I'm not a python internals guru), that's why I ask. Calling _r routines doesn't guarantee that you'll get concurrent lookups either, it just means that multiple threads calling gethostbyname_r won't trash each other.
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
+-------[ Chris McDonough ]---------------------- | Well, this is all news to me. ;-) Maybe this has something to do with | the problem that kicked off this thread. That's what I figured d;) | I thought that the BSD calls had implied threadsafety. But I guess they | don't from your description. But in this case, if there was no lock | inside the system call, why do the calls serialize? All (userland) threads live in the same context as your process. When you enter kernel space, you're off the run queue. Deschedule the *context* you block all threads. This is why, in general, userland thread suck eggs like grandma. There's two different types of threads (that we're concerned with) a) Userland threads -- live in the context of your process b) Kernel threads -- individually schedulable threads (to the kernel) There are obviously different strains and hybrids, but, let's keep it simple d8) If you can get one of b) you can do all your lookups inside that without blocking your main context. There aren't a lot of b) implementations around, which is why a seperate process is spawned just for lookups. | You would think | that they'd go right ahead and make a mess of things, but explicitly | *not* block if they weren't threadsafe. They do. If you have two threads doing lookups (that we know are going to return quickly), they'll happily trash each others data as fast as you can do the queries (unless you use the _r ones of course) It doesn't matter if a system call is threadsafe, going into the kernel blocks your current context. Most of the time syscalls are quick. Looking up non-existant domain names, or IPs that don't resolve can take up to two minutes to timeout. -- Totally Holistic Enterprises Internet| | Andrew Milton The Internet (Aust) Pty Ltd | | ACN: 082 081 472 ABN: 83 082 081 472 | M:+61 416 022 411 | Carpe Daemon PO Box 837 Indooroopilly QLD 4068 |akm@theinternet.com.au|
Hello, I have now established that Chris is right, there are "no locks... and ...what I want is likely available out of the box". The issue of my external method importing the sockets library has nothing to do with it. What is going on is that the external method blocks if it is invoked from a sub-folder, but not from my top level folder. My folder structure is as follows: Finance - external method doesn't block. Finance/Loan - external method blocks (I am using GenericUserFolder here and on all sub-folders). I am now almost certain that the problem is connected with the GenericUserFolder product, since the method doesn't block if I try to get into another instance of the management interface, but it does block any of my users who are logged on via GenericUserFolder. ---------------------------------------------------------------------------- --------------------------------- W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz "making decisions easy" ---------------------------------------------------------------------------- --------------------------------- ----- Original Message ----- From: "Chris McDonough" <chrism@digicool.com> To: "W. Robert Kellock" <sales@creditscore.co.nz> Cc: <zope@zope.org> Sent: Friday, 5 October 2001 13:56 Subject: Re: [Zope] Asynchronous DTML
Zope is multithreaded and there are typically no locks on any resources that prevents one thread from operating independently of another, so what you want is likely available out of the box.
W. Robert Kellock wrote:
Hello,
Does anyone know how to prevent an external method from blocking?
What I need is an external method that contacts an external server and returns the result to be displayed in my dtml page. It's quite alright for the person requesting the page to wait for the response. What I don't want is for other users to be blocked while the external server formulates it's response to that user.
From the archives it seems I need to piggyback onto Zopes asyncore / asynchat to achieve this, but I'm not sure how.
Thanks.
--------------------------------------------------------------------------
W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz <http://www.creditscore.co.nz> "making decisions easy"
--------------------------------------------------------------------------
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
Weird. Does Generic User Folder use a SQL database to store user info in your app? Is it possible that it's only a single-threaded database adapter in that case? W. Robert Kellock wrote:
Hello,
I have now established that Chris is right, there are "no locks... and ...what I want is likely available out of the box". The issue of my external method importing the sockets library has nothing to do with it.
What is going on is that the external method blocks if it is invoked from a sub-folder, but not from my top level folder.
My folder structure is as follows:
Finance - external method doesn't block. Finance/Loan - external method blocks (I am using GenericUserFolder here and on all sub-folders).
I am now almost certain that the problem is connected with the GenericUserFolder product, since the method doesn't block if I try to get into another instance of the management interface, but it does block any of my users who are logged on via GenericUserFolder. ---------------------------------------------------------------------------- --------------------------------- W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz "making decisions easy" ---------------------------------------------------------------------------- ---------------------------------
----- Original Message ----- From: "Chris McDonough" <chrism@digicool.com> To: "W. Robert Kellock" <sales@creditscore.co.nz> Cc: <zope@zope.org> Sent: Friday, 5 October 2001 13:56 Subject: Re: [Zope] Asynchronous DTML
Zope is multithreaded and there are typically no locks on any resources that prevents one thread from operating independently of another, so what you want is likely available out of the box.
W. Robert Kellock wrote:
Hello,
Does anyone know how to prevent an external method from blocking?
What I need is an external method that contacts an external server and returns the result to be displayed in my dtml page. It's quite alright for the person requesting the page to wait for the response. What I don't want is for other users to be blocked while the external server formulates it's response to that user.
From the archives it seems I need to piggyback onto Zopes asyncore / asynchat to achieve this, but I'm not sure how.
Thanks.
--------------------------------------------------------------------------
-----------------------------------
W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz <http://www.creditscore.co.nz> "making decisions easy"
--------------------------------------------------------------------------
-----------------------------------
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
Chris, I think you could be right. I'm trying to see if I can replicate the problem using Login Manager right now. The database adapter I'm using is gvibDA for Interbase / Firebird.
From previous posts I had concluded that gvibDA was threaded, but perhaps it isn't after all :-( If it is single-threaded are there any other possibilities for a Linux user utilising Interbase / Firebird?
--------------------------------- W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz "making decisions easy" ---------------------------------------------------------------------------- --------------------------------- ----- Original Message ----- From: "Chris McDonough" <chrism@digicool.com> To: "W. Robert Kellock" <sales@creditscore.co.nz> Cc: <zope@zope.org> Sent: Saturday, 6 October 2001 17:49 Subject: Re: [Zope] Asynchronous DTML
Weird.
Does Generic User Folder use a SQL database to store user info in your app? Is it possible that it's only a single-threaded database adapter in that case?
W. Robert Kellock wrote:
Hello,
I have now established that Chris is right, there are "no locks... and ...what I want is likely available out of the box". The issue of my external method importing the sockets library has nothing to do with it.
What is going on is that the external method blocks if it is invoked from a sub-folder, but not from my top level folder.
My folder structure is as follows:
Finance - external method doesn't block. Finance/Loan - external method blocks (I am using GenericUserFolder here and on all sub-folders).
I am now almost certain that the problem is connected with the GenericUserFolder product, since the method doesn't block if I try to get into another instance of the management interface, but it does block any of my users who are logged on via GenericUserFolder.
-------------------------------------------------------------------------- --
--------------------------------- W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz "making decisions easy"
-------------------------------------------------------------------------- --
---------------------------------
----- Original Message ----- From: "Chris McDonough" <chrism@digicool.com> To: "W. Robert Kellock" <sales@creditscore.co.nz> Cc: <zope@zope.org> Sent: Friday, 5 October 2001 13:56 Subject: Re: [Zope] Asynchronous DTML
Zope is multithreaded and there are typically no locks on any resources that prevents one thread from operating independently of another, so what you want is likely available out of the box.
W. Robert Kellock wrote:
Hello,
Does anyone know how to prevent an external method from blocking?
What I need is an external method that contacts an external server and returns the result to be displayed in my dtml page. It's quite alright for the person requesting the page to wait for the response. What I don't want is for other users to be blocked while the external server formulates it's response to that user.
From the archives it seems I need to piggyback onto Zopes asyncore / asynchat to achieve this, but I'm not sure how.
Thanks.
On 7 Oct 2001 at 13:04, W. Robert Kellock wrote:
I think you could be right. I'm trying to see if I can replicate the problem using Login Manager right now. The database adapter I'm using is gvibDA for Interbase / Firebird.
From previous posts I had concluded that gvibDA was threaded, but perhaps it isn't after all :-( If it is single-threaded are there any other possibilities for a Linux user utilising Interbase / Firebird?
gvib uses Thunked_TM, so it's essentially single threaded during a database request. Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements
Hello, I'm glad to report success at last! A heart felt thanks goes out to all those who have helped me. The problem is in Generic User Folder somewhere and has gone away now that I have replaced it with Login Manager. A point with Login Manager that had me stumped for quite a while, you must place your ZSQL Methods SQL_check_user, SQL_get_password and SQL_get_roles underneath the Generic User Source NOT in the acl_users folder when following Jok's HowTo http://www.zope.org/Members/jok/SQL_based_LoginManager although eventually I noticed http://lists.zope.org/pipermail/zope/2001-January/081474.html which put me right (this IS in Jok's HowTo I know ;-} ). Finally a question about gvibDA Brad you said that "gvib uses Thunked_TM, so it's essentially single threaded during a database request." By this do you mean that the DA itself only allows one database request at a time, but because Zope is handling the threading issues above it we end up with several instances of the DA when there is heavy traffic? ---------------------------------------------------------------------------- --------------------------------- W.Robert Kellock Ph: +64 3 326 6115 Sales Manager Fax: +64 3 326 6115 Credit Systems Control Ltd Web: www.creditscore.co.nz "making decisions easy" ---------------------------------------------------------------------------- --------------------------------- ----- Original Message ----- From: "Brad Clements" <bkc@murkworks.com> To: "W. Robert Kellock" <sales@creditscore.co.nz>; <zope@zope.org> Sent: Monday, 8 October 2001 05:48 Subject: Re: [Zope] Asynchronous DTML
On 7 Oct 2001 at 13:04, W. Robert Kellock wrote:
I think you could be right. I'm trying to see if I can replicate the problem using Login Manager right now. The database adapter I'm using is gvibDA for Interbase / Firebird.
From previous posts I had concluded that gvibDA was threaded, but perhaps it isn't after all :-( If it is single-threaded are there any other possibilities for a Linux user utilising Interbase / Firebird?
gvib uses Thunked_TM, so it's essentially single threaded during a database request.
Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements
On 8 Oct 2001 at 21:44, W. Robert Kellock wrote:
Finally a question about gvibDA
Brad you said that "gvib uses Thunked_TM, so it's essentially single threaded during a database request." By this do you mean that the DA itself only allows one database request at a time, but because Zope is handling the threading issues above it we end up with several instances of the DA when there is heavy traffic?
No, you still have one DA that you instantiated.. just means only one thread can be processing a request at a time. We have used guf and also tried LoginManager both going to Interbase for authentication. We also used Interbase for SQLSession and for regular storage for ZSQL on our server. Based on Bob T's advice (author of gvibDA), we actually created 3 Gvib connections, using one for SQLSession, one for LoginManager and one for everything else. SQLSession docs suggest doing this too. We're now switching to LDAPUserFolder and are planning to drop SQLSession for CST. Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements
participants (4)
-
Andrew Kenneth Milton -
Brad Clements -
Chris McDonough -
W. Robert Kellock