googling revealed the same error previously... http://mail.zope.org/pipermail/zope/2004-August/152343.html logging the class per the directions output the following. it seems to confirm the IOBTree class as the problem. ------ 2004-09-08T18:14:25 ERROR(200) ZODB Connection (!) <extension class BTrees.IOBTree.IOBTree at 407877e0> ------ 2004-09-08T18:14:25 ERROR(200) SiteError http://www.example.com/index_html Traceback (most recent call last): File "/opt/Zope/SoftwareHome272/lib/python/ZPublisher/Publish.py", line 107, in publish transactions_manager.commit() File "/opt/Zope/SoftwareHome272/lib/python/Zope/App/startup.py", line 222, in commit get_transaction().commit() File "/opt/Zope/SoftwareHome272/lib/python/ZODB/Transaction.py", line 236, in commit ncommitted += self._commit_objects(objects) File "/opt/Zope/SoftwareHome272/lib/python/ZODB/Transaction.py", line 351, in _commit_objects jar.commit(o, self) File "/opt/Zope/SoftwareHome/lib/python/ZODB/Connection.py", line 418, in commit state=object.__getstate__() SystemError: NULL object passed to Py_BuildValue thanks, travis On Sep 8, 2004, at 11:04 AM, Travis Miller wrote:

hi,

traceback from zmi error_log (previous traceback was from event.log) ..something to do with BTrees?

Traceback (innermost last): Module ZPublisher.Publish, line 107, in publish Module Zope.App.startup, line 222, in commit Module ZODB.Transaction, line 236, in commit Module ZODB.Transaction, line 351, in _commit_objects Module ZODB.Connection, line 411, in commit - __traceback_info__: (('BTrees.IOBTree', 'IOBTree'), '\x00\x00\x00\x00\x00\x00\x00\n', '') SystemError: NULL object passed to Py_BuildValue

thanks for any help

thanks, travis

On Sep 8, 2004, at 3:11 PM, Dieter Maurer wrote:

Travis Miller wrote at 2004-9-8 07:29 -0500:

...

... File "/opt/Zope/SoftwareHome272/lib/python/ZODB/Connection.py", line 411, in commit state=object.__getstate__() SystemError: NULL object passed to Py_BuildValue

Apparently, some C level extensions is buggy.

You must find out the class of "object" above.

The easiest way to do this is to catch the exception and print or (better) log "str(object.__class__)" in the "except" clause. Do not forget to reraise the exception afterwards.

you'll see from my later posts in the thread that the object seems to be of class BTrees.IOBTree.IOBTree i determined this from logging (following your suggestion in a previous thread) and the different traceback provided by the zmi error_log. our zeo setup is running on 3 boxes, 1 box for each ZCS and 1 box for the ZSS. restarting the ZCSs and the ZSS seemed to have no effect. the errors immediately started occuring after the zopes restarted. however, rebooting the systems this morning seemed to make the errors go away for now. they've been running for ~1hr with no Py_BuildValue errors. thanks, travis

[Dieter Maurer]

I checked the "Py_BuildValue" occurrences in the "BTrees" package.

I am quite convinced that they cannot show this behaviour unless your computer executes different code than coded there.

That's good to hear. I went thru the same exercise the first time I saw one of these reported, and came to the same conclusion. It doesn't, however, rule out compiler optimization bugs. Seems unlikely, but when the reported symptom is impossible, "unlikely" is attractive by comparison <wink>.

I would try to analyse the problem as follows:

* catch the exception and use "import pdb; pdb.set_trace()" in the except clause

* run Zope in the foreground until the error occurs and Zope stops in "set_trace()".

* attach the Zope process with GDB (or another debugger able to attach a running process

* put a GDB breakpoint on "Py_BuildValue"

* continue execution

* use "pdb" to reexecute "object.__getstate()__"

When your problem is deterministic, it will occur again.

You will stop in "Py_BuildValue". Continue until "Py_BuildValue" is called with a "NULL" value. Analyse the backtrace.

Calling object._check() could also be informative. For example, if we have a multi-bucket BTree, the ASSIGN(r, Py_BuildValue("OO", r, self->firstbucket)); line assumes (but does not verify) that self->firstbucket is non-NULL. If it is in fact NULL, Py_BuildValue would complain about that, and so would ._check().

[Travis Miller]

thanks guys for all of your help. i will study these tips and attempt to implement them in hopes of catching what's going wrong. hopefully it was a fluke, but i'd surely feel more comfortable if it was reproducible.

I don't believe (much <wink>) in flukes: something went wrong. But it's "impossible" for this specific thing to go wrong, so there's little hope of determining the cause unless/until someone *can* reproduce it. There are known problems now with apps (including at least Zope 2.7.2's Publisher) that do a subtransaction commit, get some exception later, and close the database connection without first abort()'ing or commit()'ing the transaction containing the subtransaction commit. That can lead to all sorts of bizarre symptoms, and this *may* be one of them (I'm not certain either way). The Collector report I pointed you at was interesting in this respect because that poster did see other exceptions in the log near the time of "the impossible" PyBuild_Value complaint. Current 2.7 branch ZODB code does not allow apps to do that anymore (it was always "forbidden", but never enforced (or even detected) before), and Zope's Publisher has been changed to stop doing it, So if you do find yourself able to reproduce it, you *may* get cheap relief by trying the current 2.7 branch code.

On Sep 9, 2004, at 10:09 AM, Tim Peters wrote:

[Travis Miller]

...

SystemError: NULL object passed to Py_BuildValue

Please open a Collector report. Looks like a similar thing was reported earlier, but with no followup:

http://zope.org/Collectors/Zope/1398

ok, done: http://zope.org/Collectors/Zope/1495 thanks, travis