[Zope-dev] Runaway processes

[Stephan Richter]

Hi Jim,

first of all, thanks a lot for your quick response.

On Thursday 06 December 2007, Jim Fulton wrote:
> - You should avoid runaway processes. :)  I'm actually quite serious.

Yes, we are trying hard. :-)

We are currently using lovely.remotetask to export those calls even to a 
different server, where we run the code in forked subprocesses.

After having played for a day with this problem, I came to the following 
conclusion: Python assumes that all used C-libraries are wrapped in a way 
that they are non-blocking and cannot lock up. When forking, you better know 
what you are doing.

Based on your comment, I think you agree. :-)

> - You can run multiple processes and monitor their progress -- killing  
> processes that are stuck.

I think this is a really good idea that requires little software and not much 
setup either.

> zc.z3monitor provides some output that makes this pretty straightforward.

Wow, zc.z3monitor -- and zc.ngi which I looked at too -- are very cool. How 
useful! I have been following the checkins and had an idea of what it was 
about, but it is definitely cooler than I thought. :-)

However, some code calls a blocking operation, zc.z3monitor will be locked up 
as well. I guess then I only have to check whether I get connectivity or 
not. ;-)

> - It might be interesting to see if Java or .Net give better control  
> over threads.  If they do, then this might make Zope ports to Jython  
> or IronPython more interesting. (People who get upset by the Python  
> GIL should already find these platforms interesting.)

Java has a lot more control over threads, but I still have found complaints 
that they left out some features for portability reasons. I could not 
immediately find an answer on whether blocking, run-away C-calls are handled 
correctly. I talked to Roy Mathew, a once at a time Java guru, and he said 
that Java has a time-limit it gives each thread for doing some work. (I am 
not supposed to quote him on that. ;-) So if that is true, then Java does not 
have the problem.

Roy did mention, though, that debugging locked threads in Java is a common 
skill after you reached a certain level of Java Zen.

I looked yesterday quiet a bit at Win32 threads. That part of the Win32 kernel 
seems pretty well thought out and from what I can tell you can have quiet a 
bit of control over the threads. Again, I am not sure how hanging threads are 
handled.

BTW, there is a nice article on Python threads here (probably nothing new for 
you, Jim):

http://linuxgazette.net/107/pai.html

Regards,
Stephan
-- 
Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training