Core Dump (Zope 2.7, Python 2.3.3, FreeBSD 4.5)
I've been experiencing the most peculiar core dump under Zope 2.7. It's on a web application we only decided to revive late last week, so it was never tested under earlier beta/RC conditions. The core dump occurs when visiting manage_main on a particular folder. The contents of this folder are basically just Python Scripts, Page Templates, and a few DTML Methods (including standard_error_message). I can get manage_main to render in zopectl debug (using Testing.make_request to make enough of a REQUEST object for it to render with). And I can read everything in that particular folder with zopectl debug. I used this to experiment with moving items around between folders - including moving all of the templates/scripts into their own folder outside of the web application. It still core dumps. At one point, I thought I had narrowed it down to a particular template (although the reasons for why it would be this template elude me) by doing some interactive Python work [I had written some helper 'move_object' and 'copy_object' functions a while back, and used them to move items between a pair of folders and then seeing which one Zope would crash on]. I got the contents of the template out using ``zopectl debug``, and recreated the template by hand, now that I could actually get into the target folder. It worked for a brief moment, but so did the source folder, which now had this one particular template remaining. I tried various imports and exports of the web application, including using one that was known to still be working off of a production server (Zope 2.6.1, Python 2.1). I even tried copying the contents (individually) of the folder - completely removing them from original context. It still core dumps. (Before I tried imports and exports, it was running in a Data.fs that had been copied from a Zope 2.6.1 instance to 2.7. Python Scripts were recompiled) Finally, I thought it might be a Python 2.3.3/FreeBSD issue (like there were issues with default stack size on Python 2.1.x and FreeBSD). This Python 2.3.3 had been built from source just last week, on the day I installed Zope 2.7 final on our development server. So I downloaded Zope 2.6.4 and ran that with Python 2.3.3, importing the problematic export. I could list the contents of this particular folder just fine. The import had Python 2.1 compiled scripts, so I recompiled them and exported it again just to see if that was causing strange issues - but when brought into Zope 2.7, the core dump still happens. The folder in question is not terribly large - when I extracted the scripts/templates manually into a new folder, I think it has about 41 items in it. What should I do next? Should I familiarize myself with gdb and inspect the core? What are some things I could look for if that's the next step?
Jeffrey P Shell wrote:
What should I do next? Should I familiarize myself with gdb and inspect the core? What are some things I could look for if that's the next step?
I suspect that this is your next step. Try and figure out whether it's Python, or the C extensions that ship with Zope, that are at fault. Of course, if it's memory corruption, this could be tricky to figure out... -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Tue, 2004-02-17 at 11:45, Jeffrey P Shell wrote:
What should I do next? Should I familiarize myself with gdb and inspect the core? What are some things I could look for if that's the next step?
The code dump is an important clue, so it's definitely worth looking at. It's also pretty simple. Run "gdb [python] [core]" where python is the binary you use with Zope and core is the core file. A few commands will give the most benefit. "bt" will produce a backtrace, much like a Python traceback. It will tell us where the program actually failed and what was on the call stack. If you only do this, we'll probably be in good shape. If you put the following code in your ~/.gdbinit file, then you can also inspect the Python stack frames: # If you are in an eval_frame() function, calling pyframe with no # arguments will print the filename, function name, and line number. # It assumes that f is the name of the current frame. define pyframe x/s ((PyStringObject*)f->f_code->co_filename)->ob_sval x/s ((PyStringObject*)f->f_code->co_name)->ob_sval p f->f_lineno end That helps match the C stack and the Python stack and pinpoints what Python code was executing. Jeremy
We have a report for Zope 2.7 about an object magically turning into "None" (1 or 2 weeks ago). A similar problem was discovered recently by a colleague. I started analysing it today. It turned out that objects are non-deterministically converted into "None". I have seen such behaviour in the past (with an old Python version). Later, it turned out to be a memory corruption problem in this Python version. Memory corruption can also cause death with a core dump. It is well possible that our problem is caused by my Zope modifications or other (third party) C level extensions. But, there might also be some lingering problem in Python2.3.3/Zope2.7. I will thouroughly investigate our problem within the next week and report back. -- Dieter
Dieter, Have you seen this behavior with a debug build of Python? Under a debug build, you're more likely to get complaints about dodgy C code and the garbage collection will complain about refcount problems it can detect. Jeffrey, Any luck with you're core dump? Jeremy
On Feb 19, 2004, at 1:09 PM, Jeremy Hylton wrote:
Jeffrey,
Any luck with your core dump?
I recompiled Python 2.3 with a bigger stack size, and that seems to have worked. We had similar problems with Python 2.1.3 (it's a pretty infamous Python+FreeBSD problem, AFAICT), and I thought that it was patched in the Python core by now as I thought I heard something along those lines a while back, but I may have been hearing about the patch being applied to the BSD 'ports' collection. -- Jeffrey P Shell jeff@bottlerocket.net
[Jeffrey P Shell]
We had similar problems with Python 2.1.3 (it's a pretty infamous Python+FreeBSD problem, AFAICT), and I thought that it was patched in the Python core by now as I thought I heard something along those lines a while back, but I may have been hearing about the patch being applied to the BSD 'ports' collection.
I don't think anything relevant changed in the Python core, apart from special-casing regexp tests on FreeBSD to have ever-lower expectations. Googling suggests that the "BSD 'ports' collection" (I have no understanding of what that means, though) added a Makefile option WANT_HUGE_STACK_SIZE which, if specified, causes FreeBSD to give threads the same stack size most other OSes give by default. It's an interesting meaning for "huge", anyway <wink>.
On Friday 20 February 2004 05:52, Dieter Maurer wrote:
We have a report for Zope 2.7 about an object magically turning into "None" (1 or 2 weeks ago).
Sorry, I haven't been following this thread, but I will note that I've seen (repeatably) in my test Zope environment this behaviour. This is probably unrelated though... For me, the situation comes about when I am loading up my fresh (ie. clean ZODB) test Zope with a thousand users. While the load script is running (it runs on the command-line and goes TTW using urllib to register the users) if I access a page which summarises all users, I *sometimes* get None for some of the users. The page uses a Catalog to find basic user info, and sometimes uses getObject() to find the real user object. The getObject() appears to return None sometimes. Once the load is complete, I don't see any errors. So I blame some mechanism failing under load. I also often see ReadConflictErrors, but I understand they're going away soon :) Richard
On Fri, 20 Feb 2004 09:47:47 +1100 Richard Jones <richardjones@optushome.com.au> wrote:
On Friday 20 February 2004 05:52, Dieter Maurer wrote:
We have a report for Zope 2.7 about an object magically turning into "None" (1 or 2 weeks ago).
Sorry, I haven't been following this thread, but I will note that I've seen (repeatably) in my test Zope environment this behaviour.
This is probably unrelated though...
For me, the situation comes about when I am loading up my fresh (ie. clean ZODB) test Zope with a thousand users. While the load script is running (it runs on the command-line and goes TTW using urllib to register the users) if I access a page which summarises all users, I *sometimes* get None for some of the users. The page uses a Catalog to find basic user info, and sometimes uses getObject() to find the real user object. The getObject() appears to return None sometimes.
This is a guess, but perhaps the Catalog is masking some other low-level error and returning None instead? I'll see if that seems possible from the code. -Casey
On Fri, 20 Feb 2004 10:53:30 -0500 Casey Duncan <casey@zope.com> wrote:
On Fri, 20 Feb 2004 09:47:47 +1100 Richard Jones <richardjones@optushome.com.au> wrote:
On Friday 20 February 2004 05:52, Dieter Maurer wrote:
We have a report for Zope 2.7 about an object magically turning into "None" (1 or 2 weeks ago).
Sorry, I haven't been following this thread, but I will note that I've seen (repeatably) in my test Zope environment this behaviour.
This is probably unrelated though...
For me, the situation comes about when I am loading up my fresh (ie. clean ZODB) test Zope with a thousand users. While the load script is running (it runs on the command-line and goes TTW using urllib to register the users) if I access a page which summarises all users, I *sometimes* get None for some of the users. The page uses a Catalog to find basic user info, and sometimes uses getObject() to find the real user object. The getObject() appears to return None sometimes.
This is a guess, but perhaps the Catalog is masking some other low-level error and returning None instead? I'll see if that seems possible from the code.
This is indeed possible. getObject swallows all errors and returns None when one occurs. This would make it return None on busy systems if the traversal during getObject raised a read ConflictError. I consider this a bug and I will look into what the *real* desired semantics are. I've been bitten myself by this in applications which naively assumed getObject would always return a Zope object. It might be sane to return None in the case where a cataloged object has been deleted and not uncataloged. For Zope 2.7 at least we should probably preserve these semantics. I propose however that if an unusual exception (ConflictError, MemoryError, SystemExit ;^) occurs, it should be propagated to the client, since it indicates a true exceptional condition. ConflictError propagation especially would help not miss results since it would force a retry of the transaction. This is crucial to applications which rely on catalog results to access objects which require action (I'm thinking specifically of the reindexObjectSecurity method in CMFCatalogAware). For Zope 2.8 it might be reasonable to consider also making the case where accessing objects that have been removed without uncataloging is also an exception. What do people think? -Casey
On Saturday 21 February 2004 06:04, Casey Duncan wrote:
This is indeed possible. getObject swallows all errors and returns None when one occurs. This would make it return None on busy systems if the traversal during getObject raised a read ConflictError.
This would indeed result in the behaviour I was seeing.
I consider this a bug and I will look into what the *real* desired semantics are. I've been bitten myself by this in applications which naively assumed getObject would always return a Zope object.
You say "naively", I say "rightly". I could be wrong though :)
For Zope 2.8 it might be reasonable to consider also making the case where accessing objects that have been removed without uncataloging is also an exception.
I think this is a good policy. Richard
Casey Duncan wrote at 2004-2-20 10:53 -0500:
... This is a guess, but perhaps the Catalog is masking some other low-level error and returning None instead? I'll see if that seems possible from the code.
Some time ago, I found out that "OFS.Traversable.Traversable.unrestrictedTraverse" catched "ConflictError" and returned "default" if a "default" has been specified. I filed a bug report but I am not sure whether this was fixed (it is fixed in our Zope version, of course). -- Dieter
On Fri, 20 Feb 2004 22:18:33 +0100 Dieter Maurer <dieter@handshake.de> wrote:
Casey Duncan wrote at 2004-2-20 10:53 -0500:
... This is a guess, but perhaps the Catalog is masking some other low-level error and returning None instead? I'll see if that seems possible from the code.
Some time ago, I found out that "OFS.Traversable.Traversable.unrestrictedTraverse" catched "ConflictError" and returned "default" if a "default" has been specified.
I filed a bug report but I am not sure whether this was fixed (it is fixed in our Zope version, of course).
I think so, I was just checking this code the other day working on similar ConflictError swallowing happening in the Catalog. -Casey
This is a guess, but perhaps the Catalog is masking some other low-level error and returning None instead? I'll see if that seems possible from the code.
Some time ago, I found out that "OFS.Traversable.Traversable.unrestrictedTraverse" catched "ConflictError" and returned "default" if a "default" has been specified.
I filed a bug report but I am not sure whether this was fixed (it is fixed in our Zope version, of course).
It was fixed by Andreas at the end of last September. Florent -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:fg@nuxeo.com
This is what I found in the event.log today: There was a python.core around 19:33. 2004-02-23T19:33:34 INFO(0) ZServer HTTP server started at Mon Feb 23 19:33:34 2004 Hostname: localhost Port: 8001 ------ 2004-02-23T19:33:35 INFO(0) ZCS:21613 ClientStorage (pid=21613) created RW/normal for storage: 'elms' ------ 2004-02-23T19:33:35 INFO(0) ZEC:None ClientCache: storage='elms', size=20000000; file[0]=None ------ 2004-02-23T19:33:35 INFO(0) ZCS:21613 Testing connection <ManagedConnection ('127.0.0.1', 8888)> ------ 2004-02-23T19:33:35 INFO(0) zrpc-conn:127.0.0.1:8888 received handshake 'Z201' ------ 2004-02-23T19:33:35 INFO(0) ZCS:21613 Server authentication protocol None ------ 2004-02-23T19:33:35 INFO(0) ZCS:21613 Connected to storage: ('localhost', 8888) ------ 2004-02-23T19:33:35 INFO(0) ZCS:21613 Verifying cache ------ 2004-02-23T19:33:35 INFO(0) ZCS:21613 Waiting for cache verification to finish ------ 2004-02-23T19:33:35 INFO(0) ZCS:21613 Waiting for cache verification to finish ------ 2004-02-23T19:33:35 INFO(0) ZCS:21613 endVerify finishing ------ 2004-02-23T19:33:35 INFO(0) ZCS:21613 endVerify finished ------ 2004-02-23T19:33:36 PROBLEM(100) Zope No such file or directory: /zope/instances/elms/Products/RFFolder/help ------ 2004-02-23T19:33:36 PROBLEM(100) Zope No such file or directory: /zope/instances/elms/Products/RemoteUserFolder/help ------ 2004-02-23T19:33:37 INFO(0) Zope Ready to handle requests ------ 2004-02-23T19:34:04 INFO(0) ZServer HTTP server started at Mon Feb 23 19:34:04 2004 Hostname: localhost Port: 8001 ------ 2004-02-23T19:34:05 INFO(0) ZCS:4368 ClientStorage (pid=4368) created RW/normal for storage: 'elms' ------ 2004-02-23T19:34:05 INFO(0) ZEC:None ClientCache: storage='elms', size=20000000; file[0]=None ------ 2004-02-23T19:34:05 INFO(0) ZCS:4368 Testing connection <ManagedConnection ('127.0.0.1', 8888)> ------ 2004-02-23T19:34:05 INFO(0) zrpc-conn:127.0.0.1:8888 received handshake 'Z201' ------ 2004-02-23T19:34:05 INFO(0) ZCS:4368 Server authentication protocol None ------ 2004-02-23T19:34:05 INFO(0) ZCS:4368 Connected to storage: ('localhost', 8888) ------ 2004-02-23T19:34:05 INFO(0) ZCS:4368 Verifying cache ------ 2004-02-23T19:34:05 INFO(0) ZCS:4368 Waiting for cache verification to finish ------ 2004-02-23T19:34:05 INFO(0) ZCS:4368 Waiting for cache verification to finish ------ 2004-02-23T19:34:05 INFO(0) ZCS:4368 endVerify finishing ------ 2004-02-23T19:34:05 INFO(0) ZCS:4368 endVerify finished ------ 2004-02-23T19:34:06 PROBLEM(100) Zope No such file or directory: /zope/instances/elms/Products/RFFolder/help ------ 2004-02-23T19:34:06 PROBLEM(100) Zope No such file or directory: /zope/instances/elms/Products/RemoteUserFolder/help ------ 2004-02-23T19:34:07 INFO(0) Zope Ready to handle requests ------ 2004-02-23T19:34:39 INFO(0) ZODB conflict error at /VirtualHostBase/https/elms.vanth.org:443/Instances/a/VirtualHostRoot/pa gesFolder/cwmgrFolder/getMyDeliverer (1 conflicts since startup at 2004-02-23T19:34:07) ------ 2004-02-23T19:34:42 ERROR(200) ZODB A storage error occured during object abort. This shouldn't happen. Traceback (most recent call last): File "/zope/zopestable/lib/python/ZODB/Transaction.py", line 401, in _commit_error j.tpc_abort(self) # This should never fail File "/zope/zopestable/lib/python/ZODB/Connection.py", line 675, in tpc_abort self._storage.tpc_abort(transaction) AttributeError: 'NoneType' object has no attribute 'tpc_abort' ------ 2004-02-23T19:34:42 ERROR(200) eLMS Site ===User=sliferdw_vu; URL=https://elms.vanth.org/CourseWare/ncw_rosellr1_vuMuscle_Force/Delive ries/sliferdw_vu_BME101_ncw_rosellr1_vuMuscle_Force/renderQTI; form={'hi': '0', 'eLMSaction': 'render'}; Traceback (most recent call last): File "/zope/zopestable/lib/python/ZPublisher/Publish.py", line 104, in publish if transactions_manager: transactions_manager.commit() File "/zope/zopestable/lib/python/Zope/App/startup.py", line 221, in commit get_transaction().commit() File "/zope/zopestable/lib/python/ZODB/Transaction.py", line 233, in commit ncommitted += self._commit_objects(objects) File "/zope/zopestable/lib/python/ZODB/Transaction.py", line 348, in _commit_objects jar.commit(o, self) File "/zope/zopestable/lib/python/ZODB/Connection.py", line 370, in commit dbstore=self._storage.store AttributeError: 'NoneType' object has no attribute 'store' ------ 2004-02-24T06:23:12 INFO(0) Z2 Caught signal SIGTERM ------ 2004-02-24T06:23:12 INFO(0) Z2 Shutting down fast ------ 2004-02-24T06:23:12 INFO(0) ZServer closing HTTP to new connections ------ 2004-02-24T06:23:13 INFO(0) Zope Shutting down with exit code 0
I followed Jeremy's instructions to get the trace from python.core. Here's the script: Script started on Tue Feb 24 09:39:26 2004 $ ls Products bin etc log python.core var $ ls -lag total 53116 drwxrwx--- 7 zope zopeadm 512 Feb 23 19:04 . drwxrwx--- 6 zope zopeadm 512 Jan 26 17:38 .. drwxrwx--- 7 zope zopeadm 512 Jan 26 17:34 Products drwxrwx--- 2 zope zopeadm 512 Jan 28 15:08 bin drwxrwx--- 2 zope zopeadm 512 Jan 26 17:31 etc drwxrwx--- 2 zope zopeadm 512 Feb 17 17:19 log -rw------- 1 zope zopeadm 27158492 Feb 24 09:08 python.core drwxrwx--- 2 zope zopeadm 512 Feb 24 09:11 var $ gdb /zope/python23/bin/python ./python.core GNU gdb 4.16.1 Copyright 1996 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-unknown-openbsd3.4"... Core was generated by `python'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/lib/libpthread.so.2.1...done. Reading symbols from /usr/lib/libutil.so.9.0...done. Reading symbols from /usr/lib/libm.so.1.0...done. Reading symbols from /usr/lib/libc.so.30.1...done. Reading symbols from /usr/libexec/ld.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/_socket.so... done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/_ssl.so...done. Reading symbols from /usr/lib/libssl.so.8.0...done. Reading symbols from /usr/lib/libcrypto.so.10.0...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/strop.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/time.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/binascii.so... done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/math.so...done. ---Type <return> to continue, or q <return> to quit--- Reading symbols from /zope/python23/lib/python2.3/lib-dynload/_random.so... done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/fcntl.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/cStringIO.so... done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/md5.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/sha.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/pyexpat.so... done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/_weakref.so... done. Reading symbols from /zope/zopestable/lib/python/ZODB/cPersistence.so...done. Reading symbols from /zope/zopestable/lib/python/ZODB/TimeStamp.so...done. Reading symbols from /zope/zopestable/lib/python/ExtensionClass.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/cPickle.so... done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/struct.so...done. Reading symbols from /zope/zopestable/lib/python/ZODB/cPickleCache.so...done. Reading symbols from /zope/zopestable/lib/python/ZODB/coptimizations.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/select.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/zlib.so...done. Reading symbols from /usr/lib/libz.so.2.0...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/operator.so... done. ---Type <return> to continue, or q <return> to quit--- Reading symbols from /zope/python23/lib/python2.3/lib-dynload/datetime.so... done. Reading symbols from /zope/zopestable/lib/python/Acquisition.so...done. Reading symbols from /zope/zopestable/lib/python/ComputedAttribute.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/parser.so...done. Reading symbols from /zope/zopestable/lib/python/DocumentTemplate/cDocumentTemplate.so...done . Reading symbols from /zope/zopestable/lib/python/AccessControl/cAccessControl.so...done. Reading symbols from /zope/zopestable/lib/python/Record.so...done. Reading symbols from /zope/zopestable/lib/python/intSet.so...done. Reading symbols from /zope/zopestable/lib/python/BTrees/_OOBTree.so...done. Reading symbols from /zope/zopestable/lib/python/BTrees/_OIBTree.so...done. Reading symbols from /zope/zopestable/lib/python/BTrees/_IOBTree.so...done. Reading symbols from /zope/zopestable/lib/python/BTrees/_IIBTree.so...done. Reading symbols from /zope/zopestable/lib/python/MethodObject.so...done. Reading symbols from /zope/zopestable/lib/python/ThreadLock.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/_locale.so... done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/crypt.so...done. Reading symbols from /zope/zopestable/lib/python/MultiMapping.so...done. Reading symbols from /zope/zopestable/lib/python/Missing.so...done. Reading symbols from /zope/zopestable/lib/python/Products/ZCTextIndex/stopper.so...done. ---Type <return> to continue, or q <return> to quit--- Reading symbols from /zope/zopestable/lib/python/Products/ZCTextIndex/okascore.so...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/array.so...done. Reading symbols from /zope/zopestable/lib/python/IOBTree.so...done. Reading symbols from /zope/python23/lib/python2.3/site-packages/libxml2mod.so...done. Reading symbols from /zope/deps/lib/libxml2.so.8.6...done. Reading symbols from /usr/local/lib/libiconv.so.3.0...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/dl.so...done. Reading symbols from /zope/python23/lib/python2.3/site-packages/libxsltmod.so...done. Reading symbols from /zope/deps/lib/libxslt.so.2.3...done. Reading symbols from /zope/deps/lib/libexslt.so.8.3...done. Reading symbols from /zope/python23/lib/python2.3/lib-dynload/itertools.so... done. #0 0x6257255 in select_select (self=0x0, args=0x3d641f7c) at /usr/local/src/python_release23-maint/Modules/selectmodule.c:171 171 } (gdb) bt #0 0x6257255 in select_select (self=0x0, args=0x3d641f7c) at /usr/local/src/python_release23-maint/Modules/selectmodule.c:171 #1 0x1c0990ff in PyCFunction_Call (func=0x3c2bf38c, arg=0x3d641f7c, kw=0x0) at ../python_release23-maint/Objects/methodobject.c:73 #2 0x1c05b62b in call_function (pp_stack=0x3cf2d2fc, oparg=4) at ../python_release23-maint/Python/ceval.c:3439 #3 0x1c059e46 in eval_frame (f=0x3c53780c) at ../python_release23-maint/Python/ceval.c:2116 #4 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3c2fcee0, globals=0x3c302c64, locals=0x0, args=0x3c537770, argcount=2, kws=0x3c537778, kwcount=0, defs=0x3c3092f8, defcount=2, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #5 0x1c05d01b in fast_function (func=0x3c3049cc, pp_stack=0x3cf2d49c, n=2, na=2, nk=0) at ../python_release23-maint/Python/ceval.c:3529 #6 0x1c05b6b1 in call_function (pp_stack=0x3cf2d49c, oparg=2) at ../python_release23-maint/Python/ceval.c:3458 #7 0x1c059e46 in eval_frame (f=0x3c53760c) at ../python_release23-maint/Python/ceval.c:2116 #8 0x1c05cfbb in fast_function (func=0x3cdb756c, pp_stack=0x3cf2d5bc, n=2, na=2, nk=0) at ../python_release23-maint/Python/ceval.c:3518 #9 0x1c05b6b1 in call_function (pp_stack=0x3cf2d5bc, oparg=1) at ../python_release23-maint/Python/ceval.c:3458 #10 0x1c059e46 in eval_frame (f=0x3c53740c) at ../python_release23-maint/Python/ceval.c:2116 ---Type <return> to continue, or q <return> to quit--- #11 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cdaae20, globals=0x3cdb213c, locals=0x0, args=0x3c537360, argcount=3, kws=0x3c53736c, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #12 0x1c05d01b in fast_function (func=0x3cdb7374, pp_stack=0x3cf2d75c, n=3, na=3, nk=0) at ../python_release23-maint/Python/ceval.c:3529 #13 0x1c05b6b1 in call_function (pp_stack=0x3cf2d75c, oparg=2) at ../python_release23-maint/Python/ceval.c:3458 #14 0x1c059e46 in eval_frame (f=0x3c53720c) at ../python_release23-maint/Python/ceval.c:2116 #15 0x1c05cfbb in fast_function (func=0x3cd9bed4, pp_stack=0x3cf2d87c, n=2, na=2, nk=0) at ../python_release23-maint/Python/ceval.c:3518 #16 0x1c05b6b1 in call_function (pp_stack=0x3cf2d87c, oparg=1) at ../python_release23-maint/Python/ceval.c:3458 #17 0x1c059e46 in eval_frame (f=0x3c53700c) at ../python_release23-maint/Python/ceval.c:2116 #18 0x1c05cfbb in fast_function (func=0x3cdb856c, pp_stack=0x3cf2d99c, n=3, na=3, nk=0) at ../python_release23-maint/Python/ceval.c:3518 #19 0x1c05b6b1 in call_function (pp_stack=0x3cf2d99c, oparg=2) at ../python_release23-maint/Python/ceval.c:3458 #20 0x1c059e46 in eval_frame (f=0x3c536e0c) at ../python_release23-maint/Python/ceval.c:2116 #21 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3c2af7e0, globals=0x3c2b913c, locals=0x0, args=0x3d417178, argcount=2, kws=0x0, kwcount=0, defs=0x0, ---Type <return> to continue, or q <return> to quit--- defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #22 0x1c098854 in function_call (func=0x3c2f3bc4, arg=0x3d41716c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #23 0x1c011f7c in PyObject_Call (func=0x3c2f3bc4, arg=0x3d41716c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #24 0x1c016f7e in instancemethod_call (func=0x3c2f3bc4, arg=0x3c2bf38c, kw=0x0) at ../python_release23-maint/Objects/classobject.c:2433 #25 0x1c011f7c in PyObject_Call (func=0x3c22abbc, arg=0x3d588e6c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #26 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3c22abbc, arg=0x3d588e6c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #27 0x1c011fda in PyObject_CallObject (o=0x3c22abbc, a=0x3d588e6c) at ../python_release23-maint/Objects/abstract.c:1746 #28 0x75c62a0 in unghostify (self=0x3d4cf2f8) at ZODB/cPersistence.c:107 #29 0x75c795c in Per_setstate (self=0x3d4cf2f8) at ZODB/cPersistence.c:316 #30 0xa769913 in _BTree_get (self=0x3d4cf2f8, keyarg=0x3ceac600, has_key=0) at BTrees/BTreeTemplate.c:189 #31 0xa76e303 in BTree_get (self=0x3d4cf2f8, key=0x3ceac600) at BTrees/BTreeTemplate.c:225 #32 0x1c00f178 in PyObject_GetItem (o=0x3d4cf2f8, key=0x3ceac600) at ../python_release23-maint/Objects/abstract.c:94 #33 0x1c0582bd in eval_frame (f=0x3c536c0c) at ../python_release23-maint/Python/ceval.c:1162 #34 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cec4e60, globals=0x3cdbf1c4, ---Type <return> to continue, or q <return> to quit--- locals=0x0, args=0x3d790378, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #35 0x1c098854 in function_call (func=0x3cecbc34, arg=0x3d79036c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #36 0x1c011f7c in PyObject_Call (func=0x3cecbc34, arg=0x3d79036c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #37 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3cecbc34, arg=0x3d79036c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #38 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d452c50, meth=0x3cecbc34, args=0x3d79036c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #39 0x93d5089 in call_PMethod (self=0x3d810f80, inst=0x3d452c50, args=0x3c04702c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #40 0x93d50e9 in PMethod_call (self=0x3d810f80, args=0x3c04702c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #41 0x1c011f7c in PyObject_Call (func=0x3d810f80, arg=0x3c04702c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #42 0x1c05d09d in do_call (func=0x3d810f80, pp_stack=0x3cf2e20c, na=0, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #43 0x1c05b6c7 in call_function (pp_stack=0x3cf2e20c, oparg=0) at ../python_release23-maint/Python/ceval.c:3460 #44 0x1c059e46 in eval_frame (f=0x3c536a0c) at ../python_release23-maint/Python/ceval.c:2116 ---Type <return> to continue, or q <return> to quit--- #45 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d83e038, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #46 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d83e02c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #47 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d83e02c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #48 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d83e02c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #49 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d83e02c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #50 0x93d5089 in call_PMethod (self=0x3d810f08, inst=0x3d52cb70, args=0x3d7462ac, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #51 0x93d50e9 in PMethod_call (self=0x3d810f08, args=0x3d7462ac, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #52 0x1c011f7c in PyObject_Call (func=0x3d810f08, arg=0x3d7462ac, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #53 0x1c05d09d in do_call (func=0x3d810f08, pp_stack=0x3cf2e52c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #54 0x1c05b6c7 in call_function (pp_stack=0x3cf2e52c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #55 0x1c059e46 in eval_frame (f=0x3c53680c) ---Type <return> to continue, or q <return> to quit--- at ../python_release23-maint/Python/ceval.c:2116 #56 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf446a0, globals=0x3ce1e68c, locals=0x0, args=0x3d54f6d8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #57 0x1c098854 in function_call (func=0x3d198c34, arg=0x3d54f6cc, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #58 0x1c011f7c in PyObject_Call (func=0x3d198c34, arg=0x3d54f6cc, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #59 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198c34, arg=0x3d54f6cc, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #60 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198c34, args=0x3d54f6cc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #61 0x93d5089 in call_PMethod (self=0x3d810ce0, inst=0x3d52cb70, args=0x3d7900cc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #62 0x93d50e9 in PMethod_call (self=0x3d810ce0, args=0x3d7900cc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #63 0x1c011f7c in PyObject_Call (func=0x3d810ce0, arg=0x3d7900cc, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #64 0x1c05d09d in do_call (func=0x3d810ce0, pp_stack=0x3cf2e84c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #65 0x1c05b6c7 in call_function (pp_stack=0x3cf2e84c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 ---Type <return> to continue, or q <return> to quit--- #66 0x1c059e46 in eval_frame (f=0x3c39140c) at ../python_release23-maint/Python/ceval.c:2116 #67 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d37af78, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #68 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d37af6c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #69 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d37af6c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #70 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d37af6c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #71 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d37af6c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #72 0x93d5089 in call_PMethod (self=0x3d810ed8, inst=0x3d52cb70, args=0x3d588dac, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #73 0x93d50e9 in PMethod_call (self=0x3d810ed8, args=0x3d588dac, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #74 0x1c011f7c in PyObject_Call (func=0x3d810ed8, arg=0x3d588dac, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #75 0x1c05d09d in do_call (func=0x3d810ed8, pp_stack=0x3cf2eb6c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #76 0x1c05b6c7 in call_function (pp_stack=0x3cf2eb6c, oparg=1) ---Type <return> to continue, or q <return> to quit--- at ../python_release23-maint/Python/ceval.c:3460 #77 0x1c059e46 in eval_frame (f=0x3d67f80c) at ../python_release23-maint/Python/ceval.c:2116 #78 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf447a0, globals=0x3ce1e68c, locals=0x0, args=0x3d839238, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #79 0x1c098854 in function_call (func=0x3d198d14, arg=0x3d83922c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #80 0x1c011f7c in PyObject_Call (func=0x3d198d14, arg=0x3d83922c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #81 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198d14, arg=0x3d83922c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #82 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198d14, args=0x3d83922c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #83 0x93d5089 in call_PMethod (self=0x3d810c20, inst=0x3d52cb70, args=0x3d658a0c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #84 0x93d50e9 in PMethod_call (self=0x3d810c20, args=0x3d658a0c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #85 0x1c011f7c in PyObject_Call (func=0x3d810c20, arg=0x3d658a0c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #86 0x1c05d09d in do_call (func=0x3d810c20, pp_stack=0x3cf2ee8c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 ---Type <return> to continue, or q <return> to quit--- #87 0x1c05b6c7 in call_function (pp_stack=0x3cf2ee8c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #88 0x1c059e46 in eval_frame (f=0x3d66cc0c) at ../python_release23-maint/Python/ceval.c:2116 #89 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d8390f8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #90 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d8390ec, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #91 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d8390ec, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #92 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d8390ec, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #93 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d8390ec, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #94 0x93d5089 in call_PMethod (self=0x3d7da308, inst=0x3d52cb70, args=0x3d74628c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #95 0x93d50e9 in PMethod_call (self=0x3d7da308, args=0x3d74628c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #96 0x1c011f7c in PyObject_Call (func=0x3d7da308, arg=0x3d74628c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #97 0x1c05d09d in do_call (func=0x3d7da308, pp_stack=0x3cf2f1ac, na=1, nk=0) ---Type <return> to continue, or q <return> to quit--- at ../python_release23-maint/Python/ceval.c:3644 #98 0x1c05b6c7 in call_function (pp_stack=0x3cf2f1ac, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #99 0x1c059e46 in eval_frame (f=0x3c38ee0c) at ../python_release23-maint/Python/ceval.c:2116 #100 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf446a0, globals=0x3ce1e68c, locals=0x0, args=0x3d37ee98, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #101 0x1c098854 in function_call (func=0x3d198c34, arg=0x3d37ee8c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #102 0x1c011f7c in PyObject_Call (func=0x3d198c34, arg=0x3d37ee8c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #103 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198c34, arg=0x3d37ee8c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #104 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198c34, args=0x3d37ee8c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #105 0x93d5089 in call_PMethod (self=0x3d7da020, inst=0x3d52cb70, args=0x3d7460ec, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #106 0x93d50e9 in PMethod_call (self=0x3d7da020, args=0x3d7460ec, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #107 0x1c011f7c in PyObject_Call (func=0x3d7da020, arg=0x3d7460ec, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 ---Type <return> to continue, or q <return> to quit--- #108 0x1c05d09d in do_call (func=0x3d7da020, pp_stack=0x3cf2f4cc, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #109 0x1c05b6c7 in call_function (pp_stack=0x3cf2f4cc, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #110 0x1c059e46 in eval_frame (f=0x3d67fc0c) at ../python_release23-maint/Python/ceval.c:2116 #111 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d37ae78, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #112 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d37ae6c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #113 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d37ae6c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #114 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d37ae6c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #115 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d37ae6c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #116 0x93d5089 in call_PMethod (self=0x3d7da230, inst=0x3d52cb70, args=0x3d74622c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #117 0x93d50e9 in PMethod_call (self=0x3d7da230, args=0x3d74622c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #118 0x1c011f7c in PyObject_Call (func=0x3d7da230, arg=0x3d74622c, kw=0x0) ---Type <return> to continue, or q <return> to quit--- at ../python_release23-maint/Objects/abstract.c:1755 #119 0x1c05d09d in do_call (func=0x3d7da230, pp_stack=0x3cf2f7ec, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #120 0x1c05b6c7 in call_function (pp_stack=0x3cf2f7ec, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #121 0x1c059e46 in eval_frame (f=0x3c38ea0c) at ../python_release23-maint/Python/ceval.c:2116 #122 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf447a0, globals=0x3ce1e68c, locals=0x0, args=0x3d398ad8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #123 0x1c098854 in function_call (func=0x3d198d14, arg=0x3d398acc, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #124 0x1c011f7c in PyObject_Call (func=0x3d198d14, arg=0x3d398acc, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #125 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198d14, arg=0x3d398acc, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #126 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198d14, args=0x3d398acc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #127 0x93d5089 in call_PMethod (self=0x3d7da2f0, inst=0x3d52cb70, args=0x3d74626c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #128 0x93d50e9 in PMethod_call (self=0x3d7da2f0, args=0x3d74626c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 ---Type <return> to continue, or q <return> to quit--- #129 0x1c011f7c in PyObject_Call (func=0x3d7da2f0, arg=0x3d74626c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #130 0x1c05d09d in do_call (func=0x3d7da2f0, pp_stack=0x3cf2fb0c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #131 0x1c05b6c7 in call_function (pp_stack=0x3cf2fb0c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #132 0x1c059e46 in eval_frame (f=0x3d66c80c) at ../python_release23-maint/Python/ceval.c:2116 #133 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d2e8778, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #134 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d2e876c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #135 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d2e876c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #136 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d2e876c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #137 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d2e876c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #138 0x93d5089 in call_PMethod (self=0x3d7da290, inst=0x3d52cb70, args=0x3d74620c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #139 0x93d50e9 in PMethod_call (self=0x3d7da290, args=0x3d74620c, kw=0x0) ---Type <return> to continue, or q <return> to quit--- at ../Components/ExtensionClass/src/ExtensionClass.c:778 #140 0x1c011f7c in PyObject_Call (func=0x3d7da290, arg=0x3d74620c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #141 0x1c05d09d in do_call (func=0x3d7da290, pp_stack=0x3cf2fe2c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #142 0x1c05b6c7 in call_function (pp_stack=0x3cf2fe2c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #143 0x1c059e46 in eval_frame (f=0x3d67f40c) at ../python_release23-maint/Python/ceval.c:2116 #144 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf446a0, globals=0x3ce1e68c, locals=0x0, args=0x3d417498, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #145 0x1c098854 in function_call (func=0x3d198c34, arg=0x3d41748c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #146 0x1c011f7c in PyObject_Call (func=0x3d198c34, arg=0x3d41748c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #147 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198c34, arg=0x3d41748c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #148 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198c34, args=0x3d41748c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #149 0x93d5089 in call_PMethod (self=0x3d7da1e8, inst=0x3d52cb70, args=0x3d74624c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 ---Type <return> to continue, or q <return> to quit--- #150 0x93d50e9 in PMethod_call (self=0x3d7da1e8, args=0x3d74624c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #151 0x1c011f7c in PyObject_Call (func=0x3d7da1e8, arg=0x3d74624c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #152 0x1c05d09d in do_call (func=0x3d7da1e8, pp_stack=0x3cf3014c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #153 0x1c05b6c7 in call_function (pp_stack=0x3cf3014c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #154 0x1c059e46 in eval_frame (f=0x3d66d40c) at ../python_release23-maint/Python/ceval.c:2116 #155 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d83e0d8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #156 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d83e0cc, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #157 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d83e0cc, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #158 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d83e0cc, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #159 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d83e0cc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #160 0x93d5089 in call_PMethod (self=0x3d7da200, inst=0x3d52cb70, args=0x3d742c6c, kw=0x0) ---Type <return> to continue, or q <return> to quit--- at ../Components/ExtensionClass/src/ExtensionClass.c:768 #161 0x93d50e9 in PMethod_call (self=0x3d7da200, args=0x3d742c6c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #162 0x1c011f7c in PyObject_Call (func=0x3d7da200, arg=0x3d742c6c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #163 0x1c05d09d in do_call (func=0x3d7da200, pp_stack=0x3cf3046c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #164 0x1c05b6c7 in call_function (pp_stack=0x3cf3046c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #165 0x1c059e46 in eval_frame (f=0x3d20a00c) at ../python_release23-maint/Python/ceval.c:2116 #166 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf447a0, globals=0x3ce1e68c, locals=0x0, args=0x3d417278, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #167 0x1c098854 in function_call (func=0x3d198d14, arg=0x3d41726c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #168 0x1c011f7c in PyObject_Call (func=0x3d198d14, arg=0x3d41726c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #169 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198d14, arg=0x3d41726c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #170 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198d14, args=0x3d41726c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #171 0x93d5089 in call_PMethod (self=0x3d7da1a0, inst=0x3d52cb70, ---Type <return> to continue, or q <return> to quit--- args=0x3d74616c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #172 0x93d50e9 in PMethod_call (self=0x3d7da1a0, args=0x3d74616c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #173 0x1c011f7c in PyObject_Call (func=0x3d7da1a0, arg=0x3d74616c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #174 0x1c05d09d in do_call (func=0x3d7da1a0, pp_stack=0x3cf3078c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #175 0x1c05b6c7 in call_function (pp_stack=0x3cf3078c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #176 0x1c059e46 in eval_frame (f=0x3c37400c) at ../python_release23-maint/Python/ceval.c:2116 #177 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d450df8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #178 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d450dec, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #179 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d450dec, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #180 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d450dec, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #181 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d450dec, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 ---Type <return> to continue, or q <return> to quit--- #182 0x93d5089 in call_PMethod (self=0x3d7da140, inst=0x3d52cb70, args=0x3d74618c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #183 0x93d50e9 in PMethod_call (self=0x3d7da140, args=0x3d74618c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #184 0x1c011f7c in PyObject_Call (func=0x3d7da140, arg=0x3d74618c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #185 0x1c05d09d in do_call (func=0x3d7da140, pp_stack=0x3cf30aac, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #186 0x1c05b6c7 in call_function (pp_stack=0x3cf30aac, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #187 0x1c059e46 in eval_frame (f=0x3c371c0c) at ../python_release23-maint/Python/ceval.c:2116 #188 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf446a0, globals=0x3ce1e68c, locals=0x0, args=0x3d742cd8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #189 0x1c098854 in function_call (func=0x3d198c34, arg=0x3d742ccc, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #190 0x1c011f7c in PyObject_Call (func=0x3d198c34, arg=0x3d742ccc, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #191 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198c34, arg=0x3d742ccc, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #192 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198c34, args=0x3d742ccc, kw=0x0) ---Type <return> to continue, or q <return> to quit--- at ../Components/ExtensionClass/src/ExtensionClass.c:741 #193 0x93d5089 in call_PMethod (self=0x3d7da0e0, inst=0x3d52cb70, args=0x3d742f2c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #194 0x93d50e9 in PMethod_call (self=0x3d7da0e0, args=0x3d742f2c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #195 0x1c011f7c in PyObject_Call (func=0x3d7da0e0, arg=0x3d742f2c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #196 0x1c05d09d in do_call (func=0x3d7da0e0, pp_stack=0x3cf30dcc, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #197 0x1c05b6c7 in call_function (pp_stack=0x3cf30dcc, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #198 0x1c059e46 in eval_frame (f=0x3d66c40c) at ../python_release23-maint/Python/ceval.c:2116 #199 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d8391f8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #200 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d8391ec, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #201 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d8391ec, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #202 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d8391ec, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #203 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, ---Type <return> to continue, or q <return> to quit--- meth=0x3d198cdc, args=0x3d8391ec, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #204 0x93d5089 in call_PMethod (self=0x3d7da0b0, inst=0x3d52cb70, args=0x3d74612c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #205 0x93d50e9 in PMethod_call (self=0x3d7da0b0, args=0x3d74612c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #206 0x1c011f7c in PyObject_Call (func=0x3d7da0b0, arg=0x3d74612c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #207 0x1c05d09d in do_call (func=0x3d7da0b0, pp_stack=0x3cf310ec, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #208 0x1c05b6c7 in call_function (pp_stack=0x3cf310ec, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #209 0x1c059e46 in eval_frame (f=0x3c666c0c) at ../python_release23-maint/Python/ceval.c:2116 #210 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf447a0, globals=0x3ce1e68c, locals=0x0, args=0x3d83e058, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #211 0x1c098854 in function_call (func=0x3d198d14, arg=0x3d83e04c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #212 0x1c011f7c in PyObject_Call (func=0x3d198d14, arg=0x3d83e04c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #213 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198d14, arg=0x3d83e04c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 ---Type <return> to continue, or q <return> to quit--- #214 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198d14, args=0x3d83e04c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #215 0x93d5089 in call_PMethod (self=0x3d7da080, inst=0x3d52cb70, args=0x3d73cd8c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #216 0x93d50e9 in PMethod_call (self=0x3d7da080, args=0x3d73cd8c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #217 0x1c011f7c in PyObject_Call (func=0x3d7da080, arg=0x3d73cd8c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #218 0x1c05d09d in do_call (func=0x3d7da080, pp_stack=0x3cf3140c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #219 0x1c05b6c7 in call_function (pp_stack=0x3cf3140c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #220 0x1c059e46 in eval_frame (f=0x3c46900c) at ../python_release23-maint/Python/ceval.c:2116 #221 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d8390b8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #222 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d8390ac, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #223 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d8390ac, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #224 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, ---Type <return> to continue, or q <return> to quit--- arg=0x3d8390ac, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #225 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d8390ac, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #226 0x93d5089 in call_PMethod (self=0x3d63aef0, inst=0x3d52cb70, args=0x3d742ecc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #227 0x93d50e9 in PMethod_call (self=0x3d63aef0, args=0x3d742ecc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #228 0x1c011f7c in PyObject_Call (func=0x3d63aef0, arg=0x3d742ecc, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #229 0x1c05d09d in do_call (func=0x3d63aef0, pp_stack=0x3cf3172c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #230 0x1c05b6c7 in call_function (pp_stack=0x3cf3172c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #231 0x1c059e46 in eval_frame (f=0x3d66c00c) at ../python_release23-maint/Python/ceval.c:2116 #232 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf446a0, globals=0x3ce1e68c, locals=0x0, args=0x3d2d4ed8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #233 0x1c098854 in function_call (func=0x3d198c34, arg=0x3d2d4ecc, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #234 0x1c011f7c in PyObject_Call (func=0x3d198c34, arg=0x3d2d4ecc, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 ---Type <return> to continue, or q <return> to quit--- #235 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198c34, arg=0x3d2d4ecc, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #236 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198c34, args=0x3d2d4ecc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #237 0x93d5089 in call_PMethod (self=0x3d63af08, inst=0x3d52cb70, args=0x3d74610c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #238 0x93d50e9 in PMethod_call (self=0x3d63af08, args=0x3d74610c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #239 0x1c011f7c in PyObject_Call (func=0x3d63af08, arg=0x3d74610c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #240 0x1c05d09d in do_call (func=0x3d63af08, pp_stack=0x3cf31a4c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #241 0x1c05b6c7 in call_function (pp_stack=0x3cf31a4c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #242 0x1c059e46 in eval_frame (f=0x3c83a80c) at ../python_release23-maint/Python/ceval.c:2116 #243 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d37edd8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #244 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d37edcc, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #245 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d37edcc, kw=0x0) ---Type <return> to continue, or q <return> to quit--- at ../python_release23-maint/Objects/abstract.c:1755 #246 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d37edcc, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #247 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d37edcc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #248 0x93d5089 in call_PMethod (self=0x3d63afe0, inst=0x3d52cb70, args=0x3d7461ec, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #249 0x93d50e9 in PMethod_call (self=0x3d63afe0, args=0x3d7461ec, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #250 0x1c011f7c in PyObject_Call (func=0x3d63afe0, arg=0x3d7461ec, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #251 0x1c05d09d in do_call (func=0x3d63afe0, pp_stack=0x3cf31d6c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #252 0x1c05b6c7 in call_function (pp_stack=0x3cf31d6c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #253 0x1c059e46 in eval_frame (f=0x3d618c0c) at ../python_release23-maint/Python/ceval.c:2116 #254 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44620, globals=0x3ce1e68c, locals=0x0, args=0x3cbc9240, argcount=3, kws=0x0, kwcount=0, defs=0x3d187778, defcount=1, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #255 0x1c098854 in function_call (func=0x3d198bfc, arg=0x3cbc9234, kw=0x0) ---Type <return> to continue, or q <return> to quit--- at ../python_release23-maint/Objects/funcobject.c:504 #256 0x1c011f7c in PyObject_Call (func=0x3d198bfc, arg=0x3cbc9234, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #257 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198bfc, arg=0x3cbc9234, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #258 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198bfc, args=0x3cbc9234, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #259 0x93d5089 in call_PMethod (self=0x3d63ae90, inst=0x3d52cb70, args=0x3d74258c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #260 0x93d50e9 in PMethod_call (self=0x3d63ae90, args=0x3d74258c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #261 0x1c011f7c in PyObject_Call (func=0x3d63ae90, arg=0x3d74258c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #262 0x1c05d09d in do_call (func=0x3d63ae90, pp_stack=0x3cf3208c, na=2, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #263 0x1c05b6c7 in call_function (pp_stack=0x3cf3208c, oparg=2) at ../python_release23-maint/Python/ceval.c:3460 #264 0x1c059e46 in eval_frame (f=0x3c1a480c) at ../python_release23-maint/Python/ceval.c:2116 #265 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf447a0, globals=0x3ce1e68c, locals=0x0, args=0x3d83e0f8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 ---Type <return> to continue, or q <return> to quit--- #266 0x1c098854 in function_call (func=0x3d198d14, arg=0x3d83e0ec, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #267 0x1c011f7c in PyObject_Call (func=0x3d198d14, arg=0x3d83e0ec, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #268 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198d14, arg=0x3d83e0ec, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #269 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198d14, args=0x3d83e0ec, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #270 0x93d5089 in call_PMethod (self=0x3d63aed8, inst=0x3d52cb70, args=0x3d74614c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #271 0x93d50e9 in PMethod_call (self=0x3d63aed8, args=0x3d74614c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #272 0x1c011f7c in PyObject_Call (func=0x3d63aed8, arg=0x3d74614c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #273 0x1c05d09d in do_call (func=0x3d63aed8, pp_stack=0x3cf323ac, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #274 0x1c05b6c7 in call_function (pp_stack=0x3cf323ac, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #275 0x1c059e46 in eval_frame (f=0x3c13800c) at ../python_release23-maint/Python/ceval.c:2116 #276 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d37af38, argcount=2, kws=0x0, kwcount=0, defs=0x0, ---Type <return> to continue, or q <return> to quit--- defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #277 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d37af2c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #278 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d37af2c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #279 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d37af2c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #280 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d37af2c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #281 0x93d5089 in call_PMethod (self=0x3d63af98, inst=0x3d52cb70, args=0x3d7461cc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #282 0x93d50e9 in PMethod_call (self=0x3d63af98, args=0x3d7461cc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #283 0x1c011f7c in PyObject_Call (func=0x3d63af98, arg=0x3d7461cc, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #284 0x1c05d09d in do_call (func=0x3d63af98, pp_stack=0x3cf326cc, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #285 0x1c05b6c7 in call_function (pp_stack=0x3cf326cc, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #286 0x1c059e46 in eval_frame (f=0x3ceec40c) at ../python_release23-maint/Python/ceval.c:2116 #287 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf446a0, globals=0x3ce1e68c, ---Type <return> to continue, or q <return> to quit--- locals=0x0, args=0x3d839f38, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #288 0x1c098854 in function_call (func=0x3d198c34, arg=0x3d839f2c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #289 0x1c011f7c in PyObject_Call (func=0x3d198c34, arg=0x3d839f2c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #290 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198c34, arg=0x3d839f2c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #291 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198c34, args=0x3d839f2c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #292 0x93d5089 in call_PMethod (self=0x3d63a638, inst=0x3d52cb70, args=0x3d7460cc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #293 0x93d50e9 in PMethod_call (self=0x3d63a638, args=0x3d7460cc, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #294 0x1c011f7c in PyObject_Call (func=0x3d63a638, arg=0x3d7460cc, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #295 0x1c05d09d in do_call (func=0x3d63a638, pp_stack=0x3cf329ec, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #296 0x1c05b6c7 in call_function (pp_stack=0x3cf329ec, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #297 0x1c059e46 in eval_frame (f=0x3d26ec0c) at ../python_release23-maint/Python/ceval.c:2116 ---Type <return> to continue, or q <return> to quit--- #298 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf44760, globals=0x3ce1e68c, locals=0x0, args=0x3d3878f8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #299 0x1c098854 in function_call (func=0x3d198cdc, arg=0x3d3878ec, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #300 0x1c011f7c in PyObject_Call (func=0x3d198cdc, arg=0x3d3878ec, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #301 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198cdc, arg=0x3d3878ec, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #302 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198cdc, args=0x3d3878ec, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #303 0x93d5089 in call_PMethod (self=0x3d63a218, inst=0x3d52cb70, args=0x3d7461ac, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #304 0x93d50e9 in PMethod_call (self=0x3d63a218, args=0x3d7461ac, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #305 0x1c011f7c in PyObject_Call (func=0x3d63a218, arg=0x3d7461ac, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #306 0x1c05d09d in do_call (func=0x3d63a218, pp_stack=0x3cf32d0c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #307 0x1c05b6c7 in call_function (pp_stack=0x3cf32d0c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 #308 0x1c059e46 in eval_frame (f=0x3d20a80c) ---Type <return> to continue, or q <return> to quit--- at ../python_release23-maint/Python/ceval.c:2116 #309 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cf447a0, globals=0x3ce1e68c, locals=0x0, args=0x3d1c5358, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #310 0x1c098854 in function_call (func=0x3d198d14, arg=0x3d1c534c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #311 0x1c011f7c in PyObject_Call (func=0x3d198d14, arg=0x3d1c534c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #312 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d198d14, arg=0x3d1c534c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #313 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d198d14, args=0x3d1c534c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #314 0x93d5089 in call_PMethod (self=0x3d63af50, inst=0x3d52cb70, args=0x3d7460ac, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #315 0x93d50e9 in PMethod_call (self=0x3d63af50, args=0x3d7460ac, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #316 0x1c011f7c in PyObject_Call (func=0x3d63af50, arg=0x3d7460ac, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #317 0x1c05d09d in do_call (func=0x3d63af50, pp_stack=0x3cf3302c, na=1, nk=0) at ../python_release23-maint/Python/ceval.c:3644 #318 0x1c05b6c7 in call_function (pp_stack=0x3cf3302c, oparg=1) at ../python_release23-maint/Python/ceval.c:3460 ---Type <return> to continue, or q <return> to quit--- #319 0x1c059e46 in eval_frame (f=0x3d66d00c) at ../python_release23-maint/Python/ceval.c:2116 #320 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3cec98a0, globals=0x3ce1e68c, locals=0x0, args=0x3d742d98, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #321 0x1c098854 in function_call (func=0x3d1986bc, arg=0x3d742d8c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #322 0x1c011f7c in PyObject_Call (func=0x3d1986bc, arg=0x3d742d8c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #323 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d1986bc, arg=0x3d742d8c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #324 0x93d4f83 in callMethodWithPossibleHook (inst=0x3d52cb70, meth=0x3d1986bc, args=0x3d742d8c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:741 #325 0x93d5089 in call_PMethod (self=0x3d63a590, inst=0x3d52cb70, args=0x3c04702c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:768 #326 0x93d50e9 in PMethod_call (self=0x3d63a590, args=0x3c04702c, kw=0x0) at ../Components/ExtensionClass/src/ExtensionClass.c:778 #327 0x1c011f7c in PyObject_Call (func=0x3d63a590, arg=0x3c04702c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #328 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3d63a590, arg=0x3c04702c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #329 0x1c051481 in builtin_apply (self=0x0, args=0x3d6480cc) ---Type <return> to continue, or q <return> to quit--- at ../python_release23-maint/Python/bltinmodule.c:100 #330 0x1c0990ff in PyCFunction_Call (func=0x3c04fc2c, arg=0x3d6480cc, kw=0x0) at ../python_release23-maint/Objects/methodobject.c:73 #331 0x1c05b62b in call_function (pp_stack=0x3cf333bc, oparg=2) at ../python_release23-maint/Python/ceval.c:3439 #332 0x1c059e46 in eval_frame (f=0x3c374c0c) at ../python_release23-maint/Python/ceval.c:2116 #333 0x1c05cfbb in fast_function (func=0x3c41ff0c, pp_stack=0x3cf334dc, n=3, na=3, nk=0) at ../python_release23-maint/Python/ceval.c:3518 #334 0x1c05b6b1 in call_function (pp_stack=0x3cf334dc, oparg=3) at ../python_release23-maint/Python/ceval.c:3458 #335 0x1c059e46 in eval_frame (f=0x3c3fc80c) at ../python_release23-maint/Python/ceval.c:2116 #336 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3c3b2ea0, globals=0x3c41e824, locals=0x0, args=0x3c1b05c8, argcount=8, kws=0x3c1b05e8, kwcount=1, defs=0x3c41bc40, defcount=8, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #337 0x1c05d01b in fast_function (func=0x3c41fdbc, pp_stack=0x3cf3367c, n=10, na=8, nk=1) at ../python_release23-maint/Python/ceval.c:3529 #338 0x1c05b6b1 in call_function (pp_stack=0x3cf3367c, oparg=264) at ../python_release23-maint/Python/ceval.c:3458 #339 0x1c059e46 in eval_frame (f=0x3c1b040c) at ../python_release23-maint/Python/ceval.c:2116 #340 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3c326160, globals=0x3c31ef0c, ---Type <return> to continue, or q <return> to quit--- locals=0x0, args=0x3c668594, argcount=3, kws=0x3c6685a0, kwcount=1, defs=0x3c41d698, defcount=5, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #341 0x1c05d01b in fast_function (func=0x3c421064, pp_stack=0x3cf3381c, n=5, na=3, nk=1) at ../python_release23-maint/Python/ceval.c:3529 #342 0x1c05b6b1 in call_function (pp_stack=0x3cf3381c, oparg=259) at ../python_release23-maint/Python/ceval.c:3458 #343 0x1c059e46 in eval_frame (f=0x3c66840c) at ../python_release23-maint/Python/ceval.c:2116 #344 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3c326120, globals=0x3c31ef0c, locals=0x0, args=0x3c1abd7c, argcount=8, kws=0x3c1abd9c, kwcount=0, defs=0x3c3ba508, defcount=7, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #345 0x1c05d01b in fast_function (func=0x3c42109c, pp_stack=0x3cf339bc, n=8, na=8, nk=0) at ../python_release23-maint/Python/ceval.c:3529 #346 0x1c05b6b1 in call_function (pp_stack=0x3cf339bc, oparg=8) at ../python_release23-maint/Python/ceval.c:3458 #347 0x1c059e46 in eval_frame (f=0x3c1abc0c) at ../python_release23-maint/Python/ceval.c:2116 #348 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3c326460, globals=0x3c31ef0c, locals=0x0, args=0x3c1abb74, argcount=1, kws=0x3c1abb78, kwcount=2, defs=0x3c4212d8, defcount=7, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #349 0x1c05d01b in fast_function (func=0x3c421294, pp_stack=0x3cf33b5c, n=5, ---Type <return> to continue, or q <return> to quit--- na=1, nk=2) at ../python_release23-maint/Python/ceval.c:3529 #350 0x1c05b6b1 in call_function (pp_stack=0x3cf33b5c, oparg=513) at ../python_release23-maint/Python/ceval.c:3458 #351 0x1c059e46 in eval_frame (f=0x3c1aba0c) at ../python_release23-maint/Python/ceval.c:2116 #352 0x1c05ad32 in PyEval_EvalCodeEx (co=0x3c31a760, globals=0x3c31e68c, locals=0x0, args=0x3ceba558, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at ../python_release23-maint/Python/ceval.c:2663 #353 0x1c098854 in function_call (func=0x3c31d80c, arg=0x3ceba54c, kw=0x0) at ../python_release23-maint/Objects/funcobject.c:504 #354 0x1c011f7c in PyObject_Call (func=0x3c31d80c, arg=0x3ceba54c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #355 0x1c016f7e in instancemethod_call (func=0x3c31d80c, arg=0x3c2bf38c, kw=0x0) at ../python_release23-maint/Objects/classobject.c:2433 #356 0x1c011f7c in PyObject_Call (func=0x3c217edc, arg=0x3ceba46c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #357 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3c217edc, arg=0x3ceba46c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 #358 0x1c01459e in PyInstance_New (klass=0x3c3213ec, arg=0x3ceba46c, kw=0x0) at ../python_release23-maint/Objects/classobject.c:575 #359 0x1c011f7c in PyObject_Call (func=0x3c3213ec, arg=0x3ceba46c, kw=0x0) at ../python_release23-maint/Objects/abstract.c:1755 #360 0x1c05b4ff in PyEval_CallObjectWithKeywords (func=0x3c3213ec, arg=0x3ceba46c, kw=0x0) at ../python_release23-maint/Python/ceval.c:3346 ---Type <return> to continue, or q <return> to quit--- #361 0x1c0801d1 in t_bootstrap (boot_raw=0x3cccfa60) at ../python_release23-maint/Modules/threadmodule.c:180 #362 0x7c19c5d in _thread_start () #363 0x1f in ?? () Cannot access memory at address 0xffffffff. (gdb) pyframe No symbol "f" in current context. (gdb) q $ Script done on Tue Feb 24 09:41:45 2004
Sandor Palfy wrote at 2004-2-24 09:48 -0600:
I followed Jeremy's instructions to get the trace from python.core. Here's the script: ... $ gdb /zope/python23/bin/python ./python.core GNU gdb 4.16.1
Almost surely, your "gdb" is far too old to analyse the core of a multi-threaded program (such as Zope). The indicator for this is that "gdb" reports just the main thread (which usually does not have any problem). You need a gdb version 5.x (with "x" as high as possible).
Copyright 1996 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are
Please do not send a complete script the next time! Remove unintesting parts. E.g. we are not interested which shared object were loaded and we are only interested in the first 20 to 40 stack frames and only of the thread that got the signal.
#0 0x6257255 in select_select (self=0x0, args=0x3d641f7c) at /usr/local/src/python_release23-maint/Modules/selectmodule.c:171 171 } (gdb) bt #0 0x6257255 in select_select (self=0x0, args=0x3d641f7c) at /usr/local/src/python_release23-maint/Modules/selectmodule.c:171
It is almost surely not this thread that got the SIGSEGV (unless it is a C runtime stack overflow, quite common on BSD systems). -- Dieter
#0 0x6257255 in select_select (self=0x0, args=0x3d641f7c) at /usr/local/src/python_release23-maint/Modules/selectmodule.c:171 171 } (gdb) bt #0 0x6257255 in select_select (self=0x0, args=0x3d641f7c) at /usr/local/src/python_release23-maint/Modules/selectmodule.c:171
[Dieter Maurer]
It is almost surely not this thread that got the SIGSEGV (unless it is a C runtime stack overflow, quite common on BSD systems).
I've got serious wonders about that: 1. The full stack trace had more than 350 frames, which is excessive even for Zope <0.9 wink>. 2. Line 171 of selectmodule.c in Python branch release23-maint is the closing curly brace of internal function set2list(). When a debugger dies pointing to a function's closing curly, it can mean that the generated return-from-function C code is blowing up due to stack corruption. But not enough info to say for sure ...
Thanks for the hints. It's an OpenBSD3.4 system, shipped with gcc 2.95, that's why I have that old gdb. I recompiled python according to the intructions (Step 2) found here: http://zope.org/Members/ensane/HowTo.2003-09-24.2929 I also bumped up the stack limit from 4M to 8M for the zope user (I have no idea if this has any effect to the problem): $ ulimit -a ... stack(kbytes) 8192 Now I'm praying ;-) There have been no core dumps since the recompilation, but this issue somehow relates to the load, so I'm not sure if this helped. Sandor
Sandor Palfy wrote at 2004-2-24 09:11 -0600:
... ------ 2004-02-23T19:34:42 ERROR(200) ZODB A storage error occured during object abort. This shouldn't happen. Traceback (most recent call last): File "/zope/zopestable/lib/python/ZODB/Transaction.py", line 401, in _commit_error j.tpc_abort(self) # This should never fail File "/zope/zopestable/lib/python/ZODB/Connection.py", line 675, in tpc_abort self._storage.tpc_abort(transaction) AttributeError: 'NoneType' object has no attribute 'tpc_abort'
This means that the current transaction references a connection that is meanwhile closed. Some earlier request forgot to commit or abort the transaction; therefore, the transaction contains still references to the former connection. I had to fix product installation code because Zope forgot an abort when the installation of a product did not succeed. Apparently, this is not your problem. -- Dieter
[Dieter Maurer]
This means that the current transaction references a connection that is meanwhile closed.
Some earlier request forgot to commit or abort the transaction; therefore, the transaction contains still references to the former connection.
I had to fix product installation code because Zope forgot an abort when the installation of a product did not succeed. Apparently, this is not your problem.
When a connection is closed while modifications are pending for that connection, what *should* happen? Jeremy & I were just talking about that, and agree the current "benign neglect" sucks -- a subsequent attempt to commit the transaction can't succeed, and will fail in one of these obscure ways. It certainly wouldn't be right to commit the current transaction "by magic" when a connection is closed. It doesn't seem right to abort it by magic either, since it could, e.g., be that the connection being closed has no changes pending, but that other connections do have changes pending in the transaction, and silently aborting them could be a real surprise. I think I'd like to raise an exception when a connection is closed but changes from that connection are pending in the current transaction. Would anyone object to that? I doubt it was ever *intended* that transactions could be left in unusable states (or, if it was intended, the design has proven flawed in this respect). ZODB 4 kept track of enough stuff so that it was easy to do this there; ZODB 3 would require more work.
Dieter Maurer wrote at 2004-2-19 19:52 +0100:
We have a report for Zope 2.7 about an object magically turning into "None" (1 or 2 weeks ago).
A similar problem was discovered recently by a colleague. I started analysing it today. It turned out that objects are non-deterministically converted into "None".
I have seen such behaviour in the past (with an old Python version). Later, it turned out to be a memory corruption problem in this Python version. Memory corruption can also cause death with a core dump.
It is well possible that our problem is caused by my Zope modifications or other (third party) C level extensions. But, there might also be some lingering problem in Python2.3.3/Zope2.7.
After having looked at the Python code, I recognize the extreme low likelyhood that a memory corruption manifests itself in objects becoming "None". I was able to explain most (if not all) of our problems from harmless causes: * I had a bare "return" in a function that should return a value -- protected by a condition that is true only under very special (history dependent) circumstances. This explains the apparently non-deterministic apparent "transition" to "None". In fact, no object transitioned to "None" and the behaviour was completely deterministic. * Another family of incidents where apparently constant objects became "None" was explained by the well known "refresh" property to bind the variables in a flushed module to "None". * A third report is not yet explained but I am confident that it, too, will find an explanation without a memory corruption assumption. -- Dieter
It does sound suspiciously like the freebsd python issue. What do you get in the event log when python crashes ? Is it SIGSEGV(11)/SIGBUS(10) or whatever ? This tends not to get triggered with a zeo debug prompt, for whatever reason. To find out, you could: - rebuild python with a very large stack - 1Mb *should* be enough - rebuild python with debugging symbols, then run python/zope with gdb and see what the stack trace says when it crashes. - sprinkle the zope code with log statements until you narrow down where the crash is happening. If it's the stack thing, it's probably in a call to the re module.
participants (10)
-
Anthony Baxter -
Casey Duncan -
Dieter Maurer -
Florent Guillaume -
Jeffrey P Shell -
Jeremy Hylton -
Richard Jones -
Sandor Palfy -
Simon Michael -
Tim Peters