I thought a recent operating system upgrade (FreeBSD 5.4 to 6.1) fixed my problems with Zope. Unfortunately, once I re-compiled Python et al (to remove dependencies on the old libraries), my problems with Zope deadlocking recurred. I tried to use a combination of tools to try to discover the root cause of this problem, but nothing seems to work. What am I missing, and what else can I try? This system is: Nokia IP440 (Pentium II-333MHz, 256 MB RAM) FreeBSD 6.1-RELEASE-p1 Python 2.3.5 Zope 2.8.6 Plone 2.1.2 Trace and event logs: I set up the trace log to save all messages, and the last few lines of the access, event, and trace logs are: Z2.log: 127.0.0.1 - Anonymous [15/Jun/2006:22:52:54 -0400] "GET /VirtualHostBase/http/web.irtnog.org:80/irtnog/VirtualHostRoot/irtnog/he lpcenter_icon.gif HTTP/1.1" 200 1322 "http://web.irtnog.org/howtos-orig/freebsd-pf-pppoe" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)" event.log: 2006-06-15T22:40:28 INFO Plone Debug Error exceptions.AttributeError on getBeginAndEndTimes while rendering portlet here/portlet_simpleblog/macros/portletBlogFull_local trace.log: B 221198124 2006-06-15T22:52:54 GET /VirtualHostBase/http/web.irtnog.org:80/irtnog/VirtualHostRoot/Members/x enophon/photos/washdc-jan-2006/photoalbum_view?b_start:int=20 I 221198124 2006-06-15T22:52:54 0 DeadlockDebugger + threadframe: Once the Zope process deadlocks, it ceases to respond on port 8080 (the configured listener port). Thus it is not possible to obtain a thread dump from Zope using this package. Lsof: I checked the list of open files on the Zope process but didn't see anything out of the ordinary (attached as "zope-lsof.txt") Strace/Truss/Ktrace: I traced the process using ktrace, dumped it with "kdump -H" to get thread IDs, and got the following information: 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL 7505 100075 python2.3 PSIG SIGSEGV SIG_DFL I traced the process using truss and got the following information: SIGNAL 11 (SIGSEGV) SIGNAL 11 (SIGSEGV) SIGNAL 11 (SIGSEGV) SIGNAL 11 (SIGSEGV) SIGNAL 11 (SIGSEGV) SIGNAL 11 (SIGSEGV) Strace showed the same information: --- SIGSEGV (Segmentation fault: 11) --- --- SIGSEGV (Segmentation fault: 11) --- --- SIGSEGV (Segmentation fault: 11) --- --- SIGSEGV (Segmentation fault: 11) --- --- SIGSEGV (Segmentation fault: 11) --- --- SIGSEGV (Segmentation fault: 11) --- --- SIGSEGV (Segmentation fault: 11) --- --- SIGSEGV (Segmentation fault: 11) --- GDB + attach + info threads (https://engineering.purdue.edu/ECN/Resources/KnowledgeBase/Docs/2006051 8104722): The faulting thread seems to be in the middle of pthread_mutexattr_init. (gdb) info threads * 7 LWP 100075 0x2820e5c2 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 6 Thread 0x8111000 (sleeping) 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 5 Thread 0xa318a00 (LWP 100062) 0x2821546b in pthread_testcancel () from /usr/lib/libpthread.so.2 4 Thread 0x991ac00 (runnable) 0x00000000 in ?? () 3 Thread 0x9062400 (runnable) 0x0cea3a2c in ?? () 2 Thread 0x9a37a00 (sleeping) 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 1 Thread 0x9a35800 (sleeping) 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 None of the threads are in the sigsuspend, poll, select, or kill state, although it's possible that the sleeping state is equivalent to sigsuspend. At a whim, I tried stepping into the process, but this didn't do anything other than show the SEGV on the console. (gdb) call PyRun_SimpleString("import string,sys,os,ZServer; sys.stdout=open('/tmp/urls','w',0); print string.join(map(lambda a:str(getattr(a,'env',{}).get('PATH_INFO','')), sys.modules['asyncore'].socket_map.values()), '\\n')") Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x9062400 (LWP 100075)] 0x0cea3a2c in ?? () The program being debugged was signaled while in a function called from GDB. And: (gdb) call PyRun_SimpleString("import sys, traceback;sys.stderr=open('/tmp/tb','w',0); traceback.print_stack()") [Switching to Thread 0x9062400 (LWP 100114)] Cannot set lwp 100114 registers: Invalid argument Cannot set lwp 100114 registers: Invalid argument -- jsoffron: I'm generally pretty high on national defense... Mr. Bad Example: Careful...it's a gateway policy. Before you know it, you'll be mainlining the hard stuff like trade agreements. jsoffron: Too late...I've been freebasing Nafta all day... Sweet, sweet NAFTA. - As seen on Slashdot
Matthew X. Economou wrote at 2006-6-16 09:10 -0400:
I thought a recent operating system upgrade (FreeBSD 5.4 to 6.1) fixed my problems with Zope. Unfortunately, once I re-compiled Python et al (to remove dependencies on the old libraries), my problems with Zope deadlocking recurred. I tried to use a combination of tools to try to discover the root cause of this problem, but nothing seems to work. What am I missing, and what else can I try?
Your description seems to match an error case, I have seen under Python 2.3.4 with Linux 2.4 kernel: When a SIGSEGV occurs, then Zope's main thread (the ZServer thread) dies (and Zope therefore stops responding) but the other threads remain alive and keep all sockets open. The primary culprit was Python, although the Linux thread implementation had to help to expose the bug in this way. I had thought that the problem were fixed in Python 2.3.5 (which you use). Thus, maybe, your problem is different from the one outlined above. Use OS means to check in what state the various Zope threads are and whether the primary thread still exists. If this is not the problem cause, you can attach your Zope process with "gdb" and analyse the state -- especially that of the main thread. There is a howto around how to do that. -- Dieter
Dieter, I tried the whole gdb threads debugging thing, with the following (nil) results. Any other ideas? A lot of the instructions online seem to be Linux-centric, so I'm at a loss as how to proceed: (gdb) info threads * 7 LWP 100099 0x2820e544 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 6 Thread 0x8111000 (sleeping) 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 5 Thread 0xa358400 (LWP 100098) 0x2821546b in pthread_testcancel () from /usr/lib/libpthread.so.2 4 Thread 0x9918200 (sleeping) 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 3 Thread 0x9064400 (sleeping) 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 2 Thread 0x9064e00 (runnable) 0x0807b36f in PyType_IsSubtype () 1 Thread 0x9a32200 (runnable) 0x08b53622 in ?? () (gdb) thread 2 [Switching to thread 2 (Thread 0x9064e00 (runnable))]#0 0x0807b36f in PyType_IsSubtype () (gdb) call PyRun_SimpleString("import sys, traceback; sys.stderr=open('/tmp/tb','w',0); traceback.print_stack()") Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x9a32200 (LWP 100099)] 0x08b53622 in ?? () The program being debugged was signaled while in a function called from GDB. GDB remains in the frame where the signal was received. To change this behavior use "set unwindonsignal on" Evaluation of the expression containing the function (malloc) will be abandoned. (gdb) call PyRun_SimpleString("import sys, traceback; sys.stderr=open('/tmp/tb','w',0); traceback.print_stack()") Cannot set lwp 100099 registers: Invalid argument Cannot set lwp 100099 registers: Invalid argument -- jsoffron: I'm generally pretty high on national defense... Mr. Bad Example: Careful...it's a gateway policy. Before you know it, you'll be mainlining the hard stuff like trade agreements. jsoffron: Too late...I've been freebasing Nafta all day... Sweet, sweet NAFTA. - As seen on Slashdot
-----Original Message----- From: Dieter Maurer [mailto:dieter@handshake.de] Sent: Friday, June 16, 2006 2:36 PM To: Matthew X. Economou Cc: zope@zope.org Subject: Re: [Zope] DeadlockDebugger revisited
Matthew X. Economou wrote at 2006-6-16 09:10 -0400:
I thought a recent operating system upgrade (FreeBSD 5.4 to 6.1) fixed my problems with Zope. Unfortunately, once I re-compiled Python et al (to remove dependencies on the old libraries), my problems with Zope deadlocking recurred. I tried to use a combination of tools to try to discover the root cause of this problem, but nothing seems to work. What am I missing, and what else can I try?
Your description seems to match an error case, I have seen under Python 2.3.4 with Linux 2.4 kernel:
When a SIGSEGV occurs, then Zope's main thread (the ZServer thread) dies (and Zope therefore stops responding) but the other threads remain alive and keep all sockets open.
The primary culprit was Python, although the Linux thread implementation had to help to expose the bug in this way.
I had thought that the problem were fixed in Python 2.3.5 (which you use). Thus, maybe, your problem is different from the one outlined above. Use OS means to check in what state the various Zope threads are and whether the primary thread still exists.
If this is not the problem cause, you can attach your Zope process with "gdb" and analyse the state -- especially that of the main thread. There is a howto around how to do that.
-- Dieter
This time, when it locked, I thought to try a backtrace. (gdb) info threads * 7 LWP 100098 0x2820e544 in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 6 Thread 0x8111000 (sleeping) 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 5 Thread 0xa1e1a00 (LWP 100047) 0x2821546b in pthread_testcancel () from /usr/lib/libpthread.so.2 4 Thread 0x8c91e00 (runnable) 0x08098bc0 in _PyEval_SliceIndex () 3 Thread 0x8c92c00 (runnable) 0x081008e0 in PyModule_Type () 2 Thread 0x9061e00 (sleeping) 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 1 Thread 0x991ac00 (sleeping) 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 (gdb) thread 2 [Switching to thread 2 (Thread 0x9061e00 (sleeping))]#0 0x2820df0f in pthread_mutexattr_init () from /usr/lib/libpthread.so.2 (gdb) call PyRun_SimpleString("import sys, traceback; sys.stderr=open('/tmp/tb','w',0); traceback.print_stack()") Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x8c91e00 (LWP 100098)] 0x08098bc0 in _PyEval_SliceIndex () The program being debugged was signaled while in a function called from GDB. GDB remains in the frame where the signal was received. To change this behavior use "set unwindonsignal on" Evaluation of the expression containing the function (malloc) will be abandoned. (gdb) bt #0 0x08098bc0 in _PyEval_SliceIndex () #1 0x0809ba98 in _PyEval_SliceIndex () #2 0x0809c420 in PyEval_EvalCodeEx () #3 0x080cd2d2 in PyFunction_SetClosure () #4 0x0805973c in PyObject_Call () #5 0x0805f192 in PyMethod_New () #6 0x0805973c in PyObject_Call () #7 0x28465dbd in load_binpersid () from /usr/local/lib/python2.3/lib-dynload/cPickle.so #8 0x28466eea in load () from /usr/local/lib/python2.3/lib-dynload/cPickle.so #9 0x0809b310 in _PyEval_SliceIndex () #10 0x0809ba98 in _PyEval_SliceIndex () #11 0x0809ba98 in _PyEval_SliceIndex () #12 0x0809ba98 in _PyEval_SliceIndex () #13 0x0809c420 in PyEval_EvalCodeEx () #14 0x080cd2d2 in PyFunction_SetClosure () #15 0x0805973c in PyObject_Call () #16 0x0805f192 in PyMethod_New () #17 0x0805973c in PyObject_Call () #18 0x08059911 in PyObject_CallMethod () #19 0x28498671 in unghostify () from /usr/local/www/Zope28/lib/python/persistent/cPersistence.so #20 0x28499aff in Per_setstate () from /usr/local/www/Zope28/lib/python/persistent/cPersistence.so #21 0x28524a21 in P_getattr () from /usr/local/www/Zope28/lib/python/Persistence/_Persistence.so #22 0x284d1c29 in Wrapper_findattr () from /usr/local/www/Zope28/lib/python/Acquisition/_Acquisition.so #23 0x284d2367 in Wrapper_getattro () from /usr/local/www/Zope28/lib/python/Acquisition/_Acquisition.so #24 0x08091372 in _PyUnicodeUCS4_IsAlpha () #25 0x0809b310 in _PyEval_SliceIndex () #26 0x0809c420 in PyEval_EvalCodeEx () #27 0x080cd2d2 in PyFunction_SetClosure () #28 0x0805973c in PyObject_Call () #29 0x08096de7 in PyEval_CallObjectWithKeywords () #30 0x0805971b in PyObject_CallObject () #31 0x284cb59a in callfunction4 () from /usr/local/www/Zope28/lib/python/AccessControl/cAccessControl.so #32 0x284cbc85 in ZopeSecurityPolicy_validate () from /usr/local/www/Zope28/lib/python/AccessControl/cAccessControl.so #33 0x0805973c in PyObject_Call () #34 0x08096de7 in PyEval_CallObjectWithKeywords () #35 0x080ca342 in PyComplex_AsCComplex () #36 0x0805973c in PyObject_Call () #37 0x0805f192 in PyMethod_New () #38 0x0805973c in PyObject_Call () #39 0x08096de7 in PyEval_CallObjectWithKeywords () #40 0x0805971b in PyObject_CallObject () #41 0x284cb62d in callfunction5 () from /usr/local/www/Zope28/lib/python/AccessControl/cAccessControl.so #42 0x284cc6f9 in SecurityManager_validate () from /usr/local/www/Zope28/lib/python/AccessControl/cAccessControl.so #43 0x0809b310 in _PyEval_SliceIndex () #44 0x0809c420 in PyEval_EvalCodeEx () #45 0x0809a7ab in _PyEval_SliceIndex () #46 0x0809c420 in PyEval_EvalCodeEx () #47 0x0809a7ab in _PyEval_SliceIndex () #48 0x0809c420 in PyEval_EvalCodeEx () #49 0x0809a7ab in _PyEval_SliceIndex () #50 0x0809c420 in PyEval_EvalCodeEx () #51 0x080cd2d2 in PyFunction_SetClosure () #52 0x0805973c in PyObject_Call () #53 0x0809a3be in _PyEval_SliceIndex () #54 0x0809ba98 in _PyEval_SliceIndex () #55 0x0809c420 in PyEval_EvalCodeEx () #56 0x0809a7ab in _PyEval_SliceIndex () #57 0x0809c420 in PyEval_EvalCodeEx () #58 0x080cd2d2 in PyFunction_SetClosure () #59 0x0805973c in PyObject_Call () #60 0x0805f192 in PyMethod_New () #61 0x0805973c in PyObject_Call () #62 0x0809a3be in _PyEval_SliceIndex () ---Type <return> to continue, or q <return> to quit--- #63 0x0809c420 in PyEval_EvalCodeEx () #64 0x080cd2d2 in PyFunction_SetClosure () #65 0x0805973c in PyObject_Call () #66 0x0805f192 in PyMethod_New () #67 0x0805973c in PyObject_Call () #68 0x08096de7 in PyEval_CallObjectWithKeywords () #69 0x284d14e6 in CallMethodO () from /usr/local/www/Zope28/lib/python/Acquisition/_Acquisition.so #70 0x0805973c in PyObject_Call () #71 0x08098b36 in _PyEval_SliceIndex () #72 0x0809c420 in PyEval_EvalCodeEx () #73 0x080cd2d2 in PyFunction_SetClosure () #74 0x0805973c in PyObject_Call () #75 0x0809a3be in _PyEval_SliceIndex () #76 0x0809ba98 in _PyEval_SliceIndex () #77 0x0809c420 in PyEval_EvalCodeEx () #78 0x0809a7ab in _PyEval_SliceIndex () #79 0x0809c420 in PyEval_EvalCodeEx () #80 0x080cd2d2 in PyFunction_SetClosure () #81 0x0805973c in PyObject_Call () #82 0x0805f192 in PyMethod_New () #83 0x0805973c in PyObject_Call () #84 0x0809a3be in _PyEval_SliceIndex () #85 0x0809c420 in PyEval_EvalCodeEx () #86 0x080cd2d2 in PyFunction_SetClosure () #87 0x0805973c in PyObject_Call () #88 0x0805f192 in PyMethod_New () #89 0x0805973c in PyObject_Call () #90 0x08096de7 in PyEval_CallObjectWithKeywords () #91 0x284d14e6 in CallMethodO () from /usr/local/www/Zope28/lib/python/Acquisition/_Acquisition.so #92 0x0805973c in PyObject_Call () #93 0x08098b36 in _PyEval_SliceIndex () #94 0x0809c420 in PyEval_EvalCodeEx () #95 0x0809c5d2 in PyEval_EvalCode () #96 0x0809108d in _PyUnicodeUCS4_IsAlpha () #97 0x0809b310 in _PyEval_SliceIndex () #98 0x0809c420 in PyEval_EvalCodeEx () #99 0x080cd2d2 in PyFunction_SetClosure () #100 0x0805973c in PyObject_Call () #101 0x0805f192 in PyMethod_New () #102 0x0805973c in PyObject_Call () #103 0x0805e995 in _PyInstance_Lookup () #104 0x0805973c in PyObject_Call () #105 0x08098b36 in _PyEval_SliceIndex () #106 0x0809c420 in PyEval_EvalCodeEx () #107 0x0809a7ab in _PyEval_SliceIndex () #108 0x0809ba98 in _PyEval_SliceIndex () #109 0x0809ba98 in _PyEval_SliceIndex () #110 0x0809ba98 in _PyEval_SliceIndex () #111 0x0809c420 in PyEval_EvalCodeEx () #112 0x0809a7ab in _PyEval_SliceIndex () #113 0x0809ba98 in _PyEval_SliceIndex () #114 0x0809ba98 in _PyEval_SliceIndex () #115 0x0809ba98 in _PyEval_SliceIndex () #116 0x0809ba98 in _PyEval_SliceIndex () #117 0x0809ba98 in _PyEval_SliceIndex () #118 0x0809ba98 in _PyEval_SliceIndex () #119 0x0809ba98 in _PyEval_SliceIndex () #120 0x0809ba98 in _PyEval_SliceIndex () #121 0x0809ba98 in _PyEval_SliceIndex () #122 0x0809c420 in PyEval_EvalCodeEx () #123 0x0809a7ab in _PyEval_SliceIndex () #124 0x0809ba98 in _PyEval_SliceIndex () #125 0x0809ba98 in _PyEval_SliceIndex () ---Type <return> to continue, or q <return> to quit--- #126 0x0809ba98 in _PyEval_SliceIndex () #127 0x0809ba98 in _PyEval_SliceIndex () #128 0x0809ba98 in _PyEval_SliceIndex () #129 0x0809c420 in PyEval_EvalCodeEx () #130 0x0809a7ab in _PyEval_SliceIndex () #131 0x0809ba98 in _PyEval_SliceIndex () #132 0x0809ba98 in _PyEval_SliceIndex () #133 0x0809ba98 in _PyEval_SliceIndex () #134 0x0809ba98 in _PyEval_SliceIndex () #135 0x0809c420 in PyEval_EvalCodeEx () #136 0x080cd2d2 in PyFunction_SetClosure () #137 0x0805973c in PyObject_Call () #138 0x0805f192 in PyMethod_New () #139 0x0805973c in PyObject_Call () #140 0x0805e995 in _PyInstance_Lookup () #141 0x0805973c in PyObject_Call () #142 0x08098b36 in _PyEval_SliceIndex () #143 0x0809c420 in PyEval_EvalCodeEx () #144 0x0809a7ab in _PyEval_SliceIndex () #145 0x0809ba98 in _PyEval_SliceIndex () #146 0x0809c420 in PyEval_EvalCodeEx () #147 0x0809a7ab in _PyEval_SliceIndex () #148 0x0809c420 in PyEval_EvalCodeEx () #149 0x080cd2d2 in PyFunction_SetClosure () #150 0x0805973c in PyObject_Call () #151 0x0805f192 in PyMethod_New () #152 0x0805973c in PyObject_Call () #153 0x08096de7 in PyEval_CallObjectWithKeywords () #154 0x284d14e6 in CallMethodO () from /usr/local/www/Zope28/lib/python/Acquisition/_Acquisition.so #155 0x0805973c in PyObject_Call () #156 0x08096de7 in PyEval_CallObjectWithKeywords () #157 0x080909e3 in _PyUnicodeUCS4_IsAlpha () #158 0x0809b310 in _PyEval_SliceIndex () #159 0x0809ba98 in _PyEval_SliceIndex () #160 0x0809c420 in PyEval_EvalCodeEx () #161 0x0809a7ab in _PyEval_SliceIndex () #162 0x0809c420 in PyEval_EvalCodeEx () #163 0x0809a7ab in _PyEval_SliceIndex () #164 0x0809c420 in PyEval_EvalCodeEx () #165 0x0809a7ab in _PyEval_SliceIndex () #166 0x0809c420 in PyEval_EvalCodeEx () #167 0x0809a7ab in _PyEval_SliceIndex () #168 0x0809c420 in PyEval_EvalCodeEx () #169 0x0809a7ab in _PyEval_SliceIndex () #170 0x0809c420 in PyEval_EvalCodeEx () #171 0x080cd2d2 in PyFunction_SetClosure () #172 0x0805973c in PyObject_Call () #173 0x0805f192 in PyMethod_New () #174 0x0805973c in PyObject_Call () #175 0x08096de7 in PyEval_CallObjectWithKeywords () #176 0x0805b784 in PyInstance_New () #177 0x0805973c in PyObject_Call () #178 0x08096de7 in PyEval_CallObjectWithKeywords () #179 0x080b8a96 in _PyObject_GC_Del () #180 0x28206319 in pthread_create () from /usr/lib/libpthread.so.2 #181 0x282ba637 in _ctx_start () from /lib/libc.so.6
Matthew X. Economou wrote at 2006-6-18 11:53 -0400:
I tried the whole gdb threads debugging thing, with the following (nil) results. Any other ideas? A lot of the instructions online seem to be Linux-centric, so I'm at a loss as how to proceed:
(gdb) info threads ... 2 Thread 0x9064e00 (runnable) 0x0807b36f in PyType_IsSubtype () 1 Thread 0x9a32200 (runnable) 0x08b53622 in ?? () (gdb) thread 2 [Switching to thread 2 (Thread 0x9064e00 (runnable))]#0 0x0807b36f in PyType_IsSubtype () (gdb) call PyRun_SimpleString("import sys, traceback; sys.stderr=open('/tmp/tb','w',0); traceback.print_stack()")
Program received signal SIGSEGV, Segmentation fault.
I fear that you can call Python functions in general only on the current thread (the other's probably lack the GIL, an essential requirement to execute Python code). Moreover, it seems that your Python is build without debugging symbols. Debugging symbols are almost a must when you try to analyse problems at this level. I would rebuild Python with debugging symbols (and, if possible, without optimizations). -- Dieter
From: Dieter Maurer [mailto:dieter@handshake.de]
I fear that you can call Python functions in general only on the current thread (the other's probably lack the GIL, an essential requirement to execute Python code).
Well, I upgraded to Python 2.4, re-compiled Zope and friends, and updated a few of the third-party products, and my problems with Zope deadlocking have stopped, or at least my Zope has been running non-stop for the last 5 days. We'll see if it eventually crashes (hopefully not). Thanks for the help. Best wishes, Matthew -- jsoffron: I'm generally pretty high on national defense... Mr. Bad Example: Careful...it's a gateway policy. Before you know it, you'll be mainlining the hard stuff like trade agreements. jsoffron: Too late...I've been freebasing Nafta all day... Sweet, sweet NAFTA. - As seen on Slashdot
participants (2)
-
Dieter Maurer -
Matthew X. Economou