[Zope] Hanging Zope

[Chris Kratz]

We have been having intermittent problems with zope hanging in both production 
and development.  For what it's worth, it seems as if this problem started to 
occur since we upgraded to zope 2.7.3 (python 2.3.4) and Apelib 1.0.  
Previously, we were using a standard zope data.fs.  We converted to Apelib 
for other reasons and now are starting to see these intermittent lockups.  
When it locks up, it is completely locked and has to be killed with a kill 
-9.  Here is the pertinant output from ps -Aflwm.

0 - 501       2838 20340  0   -   - - 88939 -      11:42 pts/5    
00:00:06 ./python ../lib/python/Zope/Startup/run.py -C ../etc/zope.conf
0 S 501          -     -  0  76   0 -     - futex_ 11:42 -        00:00:06 -
1 S 501          -     -  0  76   0 -     - futex_ 11:43 -        00:00:00 -

Zope is not spinning, it's just locked in a futex.

I have tried to do use the debugging a spinning zope howto. 
(http://www.zope.org/Members/4am/debugspinningzope), but that doesn't work.  
I am assuming it is because the python binary has been stripped.

Are there any other tricks to find where or what is locking the threads?  Even 
though I'm running in debug mode, killing  does not produce a traceback.  Is 
there any way to force threads to dump a traceback on exit?

-Chris

--- debugging a spinning zope attempt

# cat Z2.pid
2838
# gdb ../bin/python
(gdb) attach 2838
...snip
(gdb) info threads
  26 Thread 1122839472 (LWP 2845)  0xffffe410 in ?? ()
  25 Thread 1131232176 (LWP 2846)  0xffffe410 in ?? ()
  24 Thread 1139624880 (LWP 2847)  0xffffe410 in ?? ()
  23 Thread 1148017584 (LWP 2848)  0xffffe410 in ?? ()
  22 Thread 1156410288 (LWP 2849)  0xffffe410 in ?? ()
  21 Thread 1164802992 (LWP 2850)  0xffffe410 in ?? ()
  20 Thread 1173195696 (LWP 2851)  0xffffe410 in ?? ()
  19 Thread 1181588400 (LWP 2852)  0xffffe410 in ?? ()
  18 Thread 1189981104 (LWP 2853)  0xffffe410 in ?? ()
  17 Thread 1198373808 (LWP 2854)  0xffffe410 in ?? ()
  16 Thread 1206766512 (LWP 2855)  0xffffe410 in ?? ()
  15 Thread 1215159216 (LWP 2856)  0xffffe410 in ?? ()
  14 Thread 1223551920 (LWP 2857)  0xffffe410 in ?? ()
  13 Thread 1231944624 (LWP 2858)  0xffffe410 in ?? ()
  12 Thread 1240337328 (LWP 2859)  0xffffe410 in ?? ()
  11 Thread 1248730032 (LWP 2860)  0xffffe410 in ?? ()
  10 Thread 1257122736 (LWP 2861)  0xffffe410 in ?? ()
  9 Thread 1265515440 (LWP 2862)  0xffffe410 in ?? ()
  8 Thread 1273908144 (LWP 2863)  0xffffe410 in ?? ()
  7 Thread 1282300848 (LWP 2864)  0xffffe410 in ?? ()
  6 Thread 1290693552 (LWP 2865)  0xffffe410 in ?? ()
  5 Thread 1299086256 (LWP 2866)  0xffffe410 in ?? ()
  4 Thread 1307478960 (LWP 2867)  0xffffe410 in ?? ()
  3 Thread 1315871664 (LWP 2868)  0xffffe410 in ?? ()
  2 Thread 1324264368 (LWP 2869)  0xffffe410 in ?? ()
  1 Thread 1077192064 (LWP 2838)  0xffffe410 in ?? ()
(gdb) thread 1
[Switching to thread 1 (Thread 1077192064 (LWP 2838))]#0  0xffffe410 in ?? ()
(gdb) call PyRun_SimpleString("import sys, traceback; 
sys.stderr=open('/tmp/tb','w',0); traceback.print_stack()")
No symbol "PyRun_SimpleString" in current context.