[Zope] thread-safe xml request client

[Scott Burton]

>I can't tell *for sure* from the code you've posted, but if you are
working with a mutable global you should be ok by using the thread lock.

Mutable global data... I guess I am not too clear on what that is. I may
be completely off-base on why I am using threads. From the Thread Safety
article I took "Variables defined in classes" to be any variable. Here
is my use of the product class:

security.declareProtected('run')
    def run(self,itemList,
                            baseURL,
                            path,
                            body,
                            headers = {"Content-type":
"application/x-www-form-urlencoded", "Accept": "application/xml"}):
        "Thread wrapper"
        mutex.acquire()
        pkgs = sortPackages(itemList)
        "send the xml request over https"
        from SendRequest_threaded import Retriever
        fetch = Retriever(baseURL, path, body, headers)
        XMLresults = fetch.XMLData
        mutex.release()

Excuse my lack of knowledge in this. This is my first time writing a
product. :)

>The other threads (depending on how you write the code) will *also* all
potentially block - waiting for the lock. >
>The end result is that all of the server threads are stuck :(

This is what I figured is happening. Which makes sense cause if one
thread has locked the function the others must wait for it to return
something. I guess I can use the timeout in join to abort the thread if
it takes too long? Problem is, it still isn't able to handle multiple
client calls in a timely fashion. It is more like a thread que.


>A way around this (if the model fits for your app), is to expose the
global data through an object that only updates 
>the data at certain intervals, and preferably does so in a way that
spawns a thread that does not interfere with app 
>threads (IOW, use a lock to update the data, but get it in a non-app
thread)


So use time.sleep() to run the main app thread with a lock to change the
variables? How would I spawn a thread that will let Zope continue with
its threads? I guess I would like to know a little more about how Zope
calls products and if the global interpreter lock locks all threads
until a call is completed how do I stop that? I am not saving anything
to the ZODB.

Thanks for you help Brian

Scott



-----Original Message-----
From: zope-admin@zope.org [mailto:zope-admin@zope.org] On Behalf Of
Brian Lloyd
Sent: Wednesday, May 14, 2003 6:50 AM
To: Scott Burton; zope@zope.org
Subject: RE: [Zope] thread-safe xml request client


> I realized a few parts of my product are mutable so I wrapped those 
> methods in a thread as such:
> lock = Lock()
> class HTTPSRequester(SimpleItemWithProperties, Thread):
>     def __init__(self):
>         Thread.__init__(self)
>
> "mutable method"
> def run(self):
>     lock.acquire()
>     self.fetch()
>     lock.release()
>
>
> where fetch() is the method with mutable data.
>
> Here are my questions regarding this setup...
> 1. Is this product actually considered thread-safe now?

Depends a bit on what you mean by "thread-safe" :)

The "Thread Safety In Zope 2" that you mentioned is mostly concerned
with data integrity. By that measure, I can't tell *for sure* from the
code you've posted, but if you are working with a mutable global you
should be ok by using the thread lock.

The real issue here is the fact that it looks like you are doing a
possibly long blocking operation (an http request). The problem with
that is that you can end up with a non-responsive site, and the lock
won't help there. For ex: one thread will block making the request. The
other threads (depending on how you write the code) will *also* all
potentially block - waiting for the lock. The end result is that all of
the server threads are stuck :(

A way around this (if the model fits for your app), is to expose the
global data through an object that only updates the data at certain
intervals, and preferably does so in a way that spawns a thread that
does not interfere with app threads (IOW, use a lock to update the data,
but get it in a non-app thread).


> 2. The fetch method makes a https socket connection using httplib. The

> first time it runs when zope is restarted it takes a VERY long time to

> make the socket connection(like 20 -30 seconds) and usually returns an

> empty string response from the server. The processor utilization goes 
> up to 100% the entire time. However, the second request takes about 
> 1-2 seconds and returns
> the correct info. I have no idea why it does this and would really
like to
> know why.

Dunno on that :(

> 3. Is there any way I can create a thread or process that has all of 
> the modules I use opened and ready to use. I figure this might 
> eliminate some overhead on loading all of the modules per execution.

> 4. I have also noticed that when a user calls this product, Zope will 
> stop responding until the response is returned. I would like to have 
> the product be able to handle multiple simultaneous requests at once.

I'd bet this is due to the "thread pile-up" mentioned above.



Brian Lloyd        brian@zope.com
V.P. Engineering   540.361.1716
Zope Corporation   http://www.zope.com



_______________________________________________
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )