Segfault in zope.configuration
Hi, I'm not sure if this is a Python issue or a zope issue. We're getting a segfault on 64-bit SuSE Linux (SLES 11), originating from z3c.autoinclude, which in turn called zope.configuration's <include /> implementation. This calls expat, which then crashes (no error, log message, or core file, but it has all the markings of a segfault) during parsing of the file. The weird thing is that it's not parsing of any file: it happens during a standard configure.zcml (for the collective.xdv package), but z3c.autoinclude itself is invoked from another ZCML file, so it must've been able to read that. Any tips on how to debug or similar experiences would be appreciated! Martin
On Thu, Apr 1, 2010 at 6:07 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Any tips on how to debug or similar experiences would be appreciated!
If you're on some Unix flavor, you should be able to deconstruct the return code from the runzope process to determine if the app was killed by a signal and, if so, which. If it's a segfault, you should be looking for signal.SIGSEGV. I'd be very interested in hearing if Expat is implicated. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Fred Drake wrote:
On Thu, Apr 1, 2010 at 6:07 AM, Martin Aspeli<optilude+lists@gmail.com> wrote:
Any tips on how to debug or similar experiences would be appreciated!
If you're on some Unix flavor, you should be able to deconstruct the return code from the runzope process to determine if the app was killed by a signal and, if so, which. If it's a segfault, you should be looking for signal.SIGSEGV.
I can't check until after the long weekend now, but I'm pretty sure it is.
I'd be very interested in hearing if Expat is implicated.
I'm pretty sure it is. The pdb rabbit hole ended at pyexpat.c. I can't see what's going on there, but when I did 'r' it blew up. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
On Thu, Apr 1, 2010 at 7:29 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
I'm pretty sure it is. The pdb rabbit hole ended at pyexpat.c. I can't see what's going on there, but when I did 'r' it blew up.
If you can point me at the ZCML file you were trying to parse (or email it to me privately if it's not in a public repository), I'd be glad to see what I can turn up. I'll also need to know what version of Python you're using. Thanks! -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Fred Drake wrote:
On Thu, Apr 1, 2010 at 7:29 AM, Martin Aspeli<optilude+lists@gmail.com> wrote:
I'm pretty sure it is. The pdb rabbit hole ended at pyexpat.c. I can't see what's going on there, but when I did 'r' it blew up.
If you can point me at the ZCML file you were trying to parse (or email it to me privately if it's not in a public repository), I'd be glad to see what I can turn up.
The one that blew up is http://svn.plone.org/svn/collective/collective.xdv/trunk/collective/xdv/conf..., at least that's what z3c.autoinclude was calling the include directive handler on.
I'll also need to know what version of Python you're using.
It's a fully patched SLES11. I think it's 2.6.4 (it's definitely 2.6), though unfortunately I don't have access here on the weekend. I've used virtualenv --no-site-packages to get a clean Python (SLES installs all kinds of crap in the global site), and then used that to run a buildout. Not sure how SuSE compiles their Python. One thing to note is that SLES has /usr/lib and /usr/lib64. Everything used in our build is (or should be to my knowledge) 64-bit, so linked against the 64-bit versions. Then again, I've had a lot of trouble getting things to build properly on SLES. The buildout's got client-specific stuff in it, but a minimal one would basically just install Plone + collective.xdv in a plone.recipe.zope2instance with the KGS at http://good-py.appspot.com/release/dexterity/2.0-next?plone=4.0b1-1 Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin Aspeli wrote:
Fred Drake wrote:
On Thu, Apr 1, 2010 at 7:29 AM, Martin Aspeli<optilude+lists@gmail.com> wrote:
I'm pretty sure it is. The pdb rabbit hole ended at pyexpat.c. I can't see what's going on there, but when I did 'r' it blew up. If you can point me at the ZCML file you were trying to parse (or email it to me privately if it's not in a public repository), I'd be glad to see what I can turn up.
The one that blew up is http://svn.plone.org/svn/collective/collective.xdv/trunk/collective/xdv/conf..., at least that's what z3c.autoinclude was calling the include directive handler on.
I'll also need to know what version of Python you're using.
It's a fully patched SLES11. I think it's 2.6.4 (it's definitely 2.6), though unfortunately I don't have access here on the weekend. I've used virtualenv --no-site-packages to get a clean Python (SLES installs all kinds of crap in the global site), and then used that to run a buildout.
Not sure how SuSE compiles their Python. One thing to note is that SLES has /usr/lib and /usr/lib64. Everything used in our build is (or should be to my knowledge) 64-bit, so linked against the 64-bit versions. Then again, I've had a lot of trouble getting things to build properly on SLES.
The buildout's got client-specific stuff in it, but a minimal one would basically just install Plone + collective.xdv in a plone.recipe.zope2instance with the KGS at http://good-py.appspot.com/release/dexterity/2.0-next?plone=4.0b1-1
One fault-isolation strategy would be to try building Python from source into someplace like /opt/Python-2.6.5, and see if the segfault recurs under it. /me is deeply suspicious of *any* distro-provided python, ever. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAku04yYACgkQ+gerLs4ltQ4x0gCg2Ak5GIrBGggpXYpnCzogWSlD tugAoNWewaihiPqG06QjnOlKkUFjWO8v =Umi6 -----END PGP SIGNATURE-----
On Thu, Apr 1, 2010 at 2:17 PM, Tres Seaver <tseaver@palladion.com> wrote:
/me is deeply suspicious of *any* distro-provided python, ever.
I'm also suspicious of 64-bit builds, given that I'm not using one on my dev machine. I've picked up the OpenSuSE installation ISOs; hopefully that's close enough to the SLES distributions that I'll be able to construct useful VMs. Martin: If I can't manage a useful 64-bit VM, I'll probably send you a script to try out. It'll likely be this weekend before I can really dive in. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Fred Drake wrote:
On Thu, Apr 1, 2010 at 2:17 PM, Tres Seaver<tseaver@palladion.com> wrote:
/me is deeply suspicious of *any* distro-provided python, ever.
I'm also suspicious of 64-bit builds, given that I'm not using one on my dev machine.
I've picked up the OpenSuSE installation ISOs; hopefully that's close enough to the SLES distributions that I'll be able to construct useful VMs.
Martin: If I can't manage a useful 64-bit VM, I'll probably send you a script to try out. It'll likely be this weekend before I can really dive in.
I really appreciate the help! As I said, I can't actually ssh into the machine from home, but once I'm back in the office on Tuesday (Monday evening your time, I guess) I'll be try out anything you suggest. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Martin Aspeli wrote:
Fred Drake wrote:
On Thu, Apr 1, 2010 at 7:29 AM, Martin Aspeli<optilude+lists@gmail.com> wrote:
I'm pretty sure it is. The pdb rabbit hole ended at pyexpat.c. I can't see what's going on there, but when I did 'r' it blew up. If you can point me at the ZCML file you were trying to parse (or email it to me privately if it's not in a public repository), I'd be glad to see what I can turn up. The one that blew up is http://svn.plone.org/svn/collective/collective.xdv/trunk/collective/xdv/conf..., at least that's what z3c.autoinclude was calling the include directive handler on.
I'll also need to know what version of Python you're using. It's a fully patched SLES11. I think it's 2.6.4 (it's definitely 2.6), though unfortunately I don't have access here on the weekend. I've used virtualenv --no-site-packages to get a clean Python (SLES installs all kinds of crap in the global site), and then used that to run a buildout.
Not sure how SuSE compiles their Python. One thing to note is that SLES has /usr/lib and /usr/lib64. Everything used in our build is (or should be to my knowledge) 64-bit, so linked against the 64-bit versions. Then again, I've had a lot of trouble getting things to build properly on SLES.
The buildout's got client-specific stuff in it, but a minimal one would basically just install Plone + collective.xdv in a plone.recipe.zope2instance with the KGS at http://good-py.appspot.com/release/dexterity/2.0-next?plone=4.0b1-1
One fault-isolation strategy would be to try building Python from source into someplace like /opt/Python-2.6.5, and see if the segfault recurs under it.
I actually tried this before the virtualenv, and couldn't get Python to build cleanly. I forget the details right now (it's been a frustrating week, server wise), but trying to build it again is definitely on my list.
/me is deeply suspicious of *any* distro-provided python, ever.
Me too. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
On Thu, Apr 01, 2010 at 06:07:26PM +0800, Martin Aspeli wrote:
I'm not sure if this is a Python issue or a zope issue. We're getting a segfault on 64-bit SuSE Linux (SLES 11), originating from z3c.autoinclude, which in turn called zope.configuration's <include /> implementation. This calls expat, which then crashes (no error, log message, or core file, but it has all the markings of a segfault) during parsing of the file.
On recent Ubuntu versions segfaults get recorded in dmesg and end up in /var/log/kern.log like this: Mar 22 02:10:03 fridge kernel: [7804719.781375] python[10447]: segfault at 7fff84a7c000 ip 00007fff83aae7cb sp 00007fff84a7bd98 error 7 in libc-2.9.so[7fff83a2b000+168000] I assume that's a standard kernel feature now (we've 2.6.28, x86_64).
The weird thing is that it's not parsing of any file: it happens during a standard configure.zcml (for the collective.xdv package), but z3c.autoinclude itself is invoked from another ZCML file, so it must've been able to read that.
Any tips on how to debug or similar experiences would be appreciated!
Can you reproduce this, reliably? I'd be reaching for python/expat debug symbols and gdb, probably. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
Marius Gedminas wrote:
On Thu, Apr 01, 2010 at 06:07:26PM +0800, Martin Aspeli wrote:
I'm not sure if this is a Python issue or a zope issue. We're getting a segfault on 64-bit SuSE Linux (SLES 11), originating from z3c.autoinclude, which in turn called zope.configuration's<include /> implementation. This calls expat, which then crashes (no error, log message, or core file, but it has all the markings of a segfault) during parsing of the file.
On recent Ubuntu versions segfaults get recorded in dmesg and end up in /var/log/kern.log like this:
Mar 22 02:10:03 fridge kernel: [7804719.781375] python[10447]: segfault at 7fff84a7c000 ip 00007fff83aae7cb sp 00007fff84a7bd98 error 7 in libc-2.9.so[7fff83a2b000+168000]
I assume that's a standard kernel feature now (we've 2.6.28, x86_64).
Interesting. I'll check.
The weird thing is that it's not parsing of any file: it happens during a standard configure.zcml (for the collective.xdv package), but z3c.autoinclude itself is invoked from another ZCML file, so it must've been able to read that.
Any tips on how to debug or similar experiences would be appreciated!
Can you reproduce this, reliably?
Yes.
I'd be reaching for python/expat debug symbols and gdb, probably.
I'm not sure how to do that. My gdb fu is *very* limited. I also assume I'd need to re-build python and/or expat for this? Martin
Hi Martin Given that its failing on a call to libc I would do a quick run with strace and ltrace. (In your case ltrace will be probably be more useful.) You will be able to see the calls into the libraries and you should see immediately if your segfault is inside expat or not. (ltrace output will be very long ;-) T On Fri, Apr 2, 2010 at 11:29 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Marius Gedminas wrote:
On Thu, Apr 01, 2010 at 06:07:26PM +0800, Martin Aspeli wrote:
I'm not sure if this is a Python issue or a zope issue. We're getting a segfault on 64-bit SuSE Linux (SLES 11), originating from z3c.autoinclude, which in turn called zope.configuration's<include /> implementation. This calls expat, which then crashes (no error, log message, or core file, but it has all the markings of a segfault) during parsing of the file.
On recent Ubuntu versions segfaults get recorded in dmesg and end up in /var/log/kern.log like this:
Mar 22 02:10:03 fridge kernel: [7804719.781375] python[10447]: segfault at 7fff84a7c000 ip 00007fff83aae7cb sp 00007fff84a7bd98 error 7 in libc-2.9.so[7fff83a2b000+168000]
I assume that's a standard kernel feature now (we've 2.6.28, x86_64).
Interesting. I'll check.
The weird thing is that it's not parsing of any file: it happens during a standard configure.zcml (for the collective.xdv package), but z3c.autoinclude itself is invoked from another ZCML file, so it must've been able to read that.
Any tips on how to debug or similar experiences would be appreciated!
Can you reproduce this, reliably?
Yes.
I'd be reaching for python/expat debug symbols and gdb, probably.
I'm not sure how to do that. My gdb fu is *very* limited. I also assume I'd need to re-build python and/or expat for this?
Martin
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
A: Because it destroys the flow of the conversation Q: Why is it bad? A: No, it's bad. Q: Should I top post in replies to mailing lists? On Fri, Apr 02, 2010 at 12:26:59PM +0800, Tim Hoffman wrote:
Hi Martin
Given that its failing on a call to libc I would do a quick run with strace and ltrace.
That was my segfault, not Martin's. I suspect hardware, as it's very non-deterministic (a small pure-Python script that runs every 5 minutes segfaults once in a few months).
(In your case ltrace will be probably be more useful.) You will be able to see the calls into the libraries and you should see immediately if your segfault is inside expat or not. (ltrace output will be very long ;-)
On Fri, Apr 2, 2010 at 11:29 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Marius Gedminas wrote:
On Thu, Apr 01, 2010 at 06:07:26PM +0800, Martin Aspeli wrote:
I'm not sure if this is a Python issue or a zope issue. We're getting a segfault on 64-bit SuSE Linux (SLES 11), originating from z3c.autoinclude, which in turn called zope.configuration's<include /> implementation. This calls expat, which then crashes (no error, log message, or core file, but it has all the markings of a segfault) during parsing of the file.
On recent Ubuntu versions segfaults get recorded in dmesg and end up in /var/log/kern.log like this:
Mar 22 02:10:03 fridge kernel: [7804719.781375] python[10447]: segfault at 7fff84a7c000 ip 00007fff83aae7cb sp 00007fff84a7bd98 error 7 in libc-2.9.so[7fff83a2b000+168000]
I assume that's a standard kernel feature now (we've 2.6.28, x86_64).
Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
sorry On Fri, Apr 2, 2010 at 9:07 PM, Marius Gedminas <marius@gedmin.as> wrote:
A: Because it destroys the flow of the conversation Q: Why is it bad? A: No, it's bad. Q: Should I top post in replies to mailing lists?
On Fri, Apr 02, 2010 at 12:26:59PM +0800, Tim Hoffman wrote:
Hi Martin
Given that its failing on a call to libc I would do a quick run with strace and ltrace.
That was my segfault, not Martin's. I suspect hardware, as it's very non-deterministic (a small pure-Python script that runs every 5 minutes segfaults once in a few months).
(In your case ltrace will be probably be more useful.) You will be able to see the calls into the libraries and you should see immediately if your segfault is inside expat or not. (ltrace output will be very long ;-)
On Fri, Apr 2, 2010 at 11:29 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Marius Gedminas wrote:
On Thu, Apr 01, 2010 at 06:07:26PM +0800, Martin Aspeli wrote:
I'm not sure if this is a Python issue or a zope issue. We're getting a segfault on 64-bit SuSE Linux (SLES 11), originating from z3c.autoinclude, which in turn called zope.configuration's<include /> implementation. This calls expat, which then crashes (no error, log message, or core file, but it has all the markings of a segfault) during parsing of the file.
On recent Ubuntu versions segfaults get recorded in dmesg and end up in /var/log/kern.log like this:
Mar 22 02:10:03 fridge kernel: [7804719.781375] python[10447]: segfault at 7fff84a7c000 ip 00007fff83aae7cb sp 00007fff84a7bd98 error 7 in libc-2.9.so[7fff83a2b000+168000]
I assume that's a standard kernel feature now (we've 2.6.28, x86_64).
Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux)
iEYEARECAAYFAku17BQACgkQkVdEXeem149bGwCfYKlubUF9+xxJ7oOM0hhRc6o9 BOMAniwIKQNDr+h224tdW7nonPoYtctr =7p1x -----END PGP SIGNATURE-----
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
On Fri, Apr 02, 2010 at 11:29:06AM +0800, Martin Aspeli wrote:
I'd be reaching for python/expat debug symbols and gdb, probably.
I'm not sure how to do that. My gdb fu is *very* limited.
Have you used pdb? gdb is mostly the same. gdb python .... and when it crashes, type 'bt' at the (gdb) prompt. The C stack trace reads backwards when compared to Python tracebacks: the main function is at the bottom, the innermost one at the top.
I also assume I'd need to re-build python and/or expat for this?
Not necessarily. Even bare gdb will tell you which library was it that got the segfault. If you want more, you can usually install debug symbols for Python and libexpat into /usr/lib/debug and re-run gdb, without having to rebuild your own python/libexpat. I don't know about SLES. I've tried Googling and only found http://en.opensuse.org/Packaging/Debuginfo which appears to be a page for packagers, and links to http://fedoraproject.org/wiki/StackTraces for end-user "how to get a useful stack trace" tutorial. AFAICs if you have a 'debuginfo-install' command (part of yum-utils) on your server, you should be golden. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
On Fri, 02 Apr 2010 11:29:06 +0800 Martin Aspeli <optilude+lists@gmail.com> wrote: Hello,
The weird thing is that it's not parsing of any file: it happens during a standard configure.zcml (for the collective.xdv package), but z3c.autoinclude itself is invoked from another ZCML file, so it must've been able to read that.
Any tips on how to debug or similar experiences would be appreciated!
Can you reproduce this, reliably?
Yes.
I already got segfaults myself with zope.component in Zope 2.11 when doing race conditions: two threads registering the same adapter at the same time. It corrupted Python memory leading to segfault few seconds after. Maybe somehow you have the same thing. I don't image writing a test for it, so I just adding a lock around all registration steps in the ZCA, and it worked fine. Best regards, Sylvain, -- Sylvain Viollon -- Infrae t +31 10 243 7051 -- http://infrae.com Hoevestraat 10 3033GC Rotterdam -- The Netherlands
Martin Aspeli wrote:
Hi,
I'm not sure if this is a Python issue or a zope issue. We're getting a segfault on 64-bit SuSE Linux (SLES 11), originating from z3c.autoinclude, which in turn called zope.configuration's<include /> implementation. This calls expat, which then crashes (no error, log message, or core file, but it has all the markings of a segfault) during parsing of the file.
The weird thing is that it's not parsing of any file: it happens during a standard configure.zcml (for the collective.xdv package), but z3c.autoinclude itself is invoked from another ZCML file, so it must've been able to read that.
I tried running it through gdb, and got: (gdb) run Starting program: /home/osc/python-env/bin/python2.6 ./bin/instance1 fg (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) [Thread debugging using libthread_db enabled] (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) (no debugging symbols found) 2010-04-07 09:38:22 INFO ZServer HTTP server started at Wed Apr 7 09:38:22 2010 Hostname: 0.0.0.0 Port: 8801 /home/osc/osc/eggs/collective.wtf-1.0b9-py2.6.egg/collective/wtf/exportimport.py:8: DeprecationWarning: InitializeClass is deprecated. import from App.class_init instead from Globals import InitializeClass Program exited normally. .. which is not very helpful. The status code is indeed zero. (python-env)osc@lwpn-osb-webback-2:~/osc> ./bin/instance1 fg2010-04-07 09:39:21 INFO ZServer HTTP server started at Wed Apr 7 09:39:21 2010 Hostname: 0.0.0.0 Port: 8801 /home/osc/osc/eggs/collective.wtf-1.0b9-py2.6.egg/collective/wtf/exportimport.py:8: DeprecationWarning: InitializeClass is deprecated. import from App.class_init instead from Globals import InitializeClass (python-env)osc@lwpn-osb-webback-2:~/osc> echo $? 0 Next, I tried to pdb from roughly where I know it dies. Sorry about the long text here. It's a bit weird, but I'll explain what's happening. (python-env)osc@lwpn-osb-webback-2:~/osc> ./bin/instance1 fg2010-04-07 09:47:37 INFO ZServer HTTP server started at Wed Apr 7 09:47:37 2010 Hostname: 0.0.0.0 Port: 8801 This is where the brekapoint was set, in z3c.autoinclude's <includePlugins /> handler. This looks for packages with a particular entry point and loads their meta.zcml, then configure.zcml. The problem happens during configure.zcml loading.
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(104)includePluginsDirective() -> for filename in zcml_to_look_for: <snip /> We've now stepped into the function that does the real work: -> def includeZCMLGroup(_context, info, filename, override=False): (Pdb) l 10 from z3c.autoinclude.plugin import PluginFinder 11 12 import logging 13 log = logging.getLogger("z3c.autoinclude") 14 15 -> def includeZCMLGroup(_context, info, filename, override=False): 16 includable_zcml = info[filename] 17 # ^^ a key error would mean that we are trying to include a group of ZCML 18 # with a filename that wasn't ever searched for. that *should* be an error 19 20 zcml_context = repr(_context.info) <snip /> This quickly gets to here, where it's loading a package and calling zope.configuration's <include /> handler. (Pdb) l 22 for dotted_name in includable_zcml: 23 log.debug('including file %s from package %s at %s', filename, dotted_name, zcml_context) 24 25 for dotted_name in includable_zcml: 26 includable_package = resolve(dotted_name) 27 -> if override: 28 includeOverrides(_context, filename, includable_package) 29 else: 30 include(_context, filename, includable_package) 31 32 (Pdb) n
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(30)includeZCMLGroup() -> include(_context, filename, includable_package) It's not na override, so we end up in zope.configuration now. --Call--
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(487)include() -> def include(_context, file=None, package=None, files=None): (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(493)include() -> if files: (Pdb) l 488 """Include a zcml file 489 490 See examples in tests/text_xmlconfig.py 491 """ 492 493 -> if files: 494 if file: 495 raise ValueError("Must specify only one of file or files") 496 elif not file: 497 file = 'configure.zcml' 498 (Pdb) l 499 # BBB 2006/12/19 -- to be removed after 12 months 500 # This is a backward-compatibility support for old site.conf 501 502 if package and (package.__name__ == 'zope.app'): 503 try: 504 import zope.app.zcmlfiles 505 except ImportError: 506 pass # maybe this is an old zope without zope.app.zcmlfiles 507 else: 508 dirpath, filename = os.path.split(file) 509 # be careful, because zope.app is a namespace package (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(496)include() -> elif not file: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(502)include() -> if package and (package.__name__ == 'zope.app'): (Pdb) l 497 file = 'configure.zcml' 498 499 # BBB 2006/12/19 -- to be removed after 12 months 500 # This is a backward-compatibility support for old site.conf 501 502 -> if package and (package.__name__ == 'zope.app'): 503 try: 504 import zope.app.zcmlfiles 505 except ImportError: 506 pass # maybe this is an old zope without zope.app.zcmlfiles 507 else: (Pdb) pp package.__name__ 'lw.portal' This is "our" package. You'll see the ZCML it's loading in a moment. Note that this is the first package it's trying to load. (Pdb) pp file 'configure.zcml' (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(524)include() -> context = config.GroupingContextDecorator(_context) (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(525)include() -> if package is not None: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(526)include() -> context.package = package (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(527)include() -> context.basepath = None (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(529)include() -> if files: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(535)include() -> paths = [context.path(file)] (Pdb) l 530 paths = glob(context.path(files)) 531 paths = zip([path.lower() for path in paths], paths) 532 paths.sort() 533 paths = [path for (l, path) in paths] 534 else: 535 -> paths = [context.path(file)] 536 537 for path in paths: 538 if context.processFile(path): 539 f = openInOrPlain(path) 540 logger.debug("include %s" % f.name) (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(537)include() -> for path in paths: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(538)include() -> if context.processFile(path): (Pdb) l 533 paths = [path for (l, path) in paths] 534 else: 535 paths = [context.path(file)] 536 537 for path in paths: 538 -> if context.processFile(path): 539 f = openInOrPlain(path) 540 logger.debug("include %s" % f.name) 541 542 context.basepath = os.path.dirname(path) 543 context.includepath = _context.includepath + (f.name, ) (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(539)include() -> f = openInOrPlain(path) (Pdb) l 534 else: 535 paths = [context.path(file)] 536 537 for path in paths: 538 if context.processFile(path): 539 -> f = openInOrPlain(path) 540 logger.debug("include %s" % f.name) 541 542 context.basepath = os.path.dirname(path) 543 context.includepath = _context.includepath + (f.name, ) 544 _context.stack.append(config.GroupingStackItem(context)) (Pdb) s --Call--
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(383)openInOrPlain() -> def openInOrPlain(filename): (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(426)openInOrPlain() -> try: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(427)openInOrPlain() -> fp = open(filename) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(437)openInOrPlain() -> return fp (Pdb) --Return--
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(437)openInOrPlain()-><open fi...x5dc1608> -> return fp (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(540)include() -> logger.debug("include %s" % f.name) (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(542)include() -> context.basepath = os.path.dirname(path) (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(543)include() -> context.includepath = _context.includepath + (f.name, ) (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(544)include() -> _context.stack.append(config.GroupingStackItem(context)) (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(546)include() -> processxmlfile(f, context) (Pdb) l 541 542 context.basepath = os.path.dirname(path) 543 context.includepath = _context.includepath + (f.name, ) 544 _context.stack.append(config.GroupingStackItem(context)) 545 546 -> processxmlfile(f, context) 547 f.close() 548 assert _context.stack[-1].context is context 549 _context.stack.pop() 550 551 def exclude(_context, file=None, package=None, files=None): (Pdb) s --Call--
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(367)processxmlfile() -> def processxmlfile(file, context, testing=False): (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(372)processxmlfile() -> src = InputSource(getattr(file, 'name', '<string>')) (Pdb) l 367 def processxmlfile(file, context, testing=False): 368 """Process a configuration file 369 370 See examples in tests/text_xmlconfig.py 371 """ 372 -> src = InputSource(getattr(file, 'name', '<string>')) 373 src.setByteStream(file) 374 parser = make_parser() 375 parser.setContentHandler(ConfigurationHandler(context, testing=testing)) 376 parser.setFeature(feature_namespaces, True) 377 try: (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(373)processxmlfile() -> src.setByteStream(file) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(374)processxmlfile() -> parser = make_parser() (Pdb) pp src <xml.sax.xmlreader.InputSource instance at 0x5f6d950> (Pdb) l 369 370 See examples in tests/text_xmlconfig.py 371 """ 372 src = InputSource(getattr(file, 'name', '<string>')) 373 src.setByteStream(file) 374 -> parser = make_parser() 375 parser.setContentHandler(ConfigurationHandler(context, testing=testing)) 376 parser.setFeature(feature_namespaces, True) 377 try: 378 parser.parse(src) 379 except SAXParseException: (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(375)processxmlfile() -> parser.setContentHandler(ConfigurationHandler(context, testing=testing)) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(376)processxmlfile() -> parser.setFeature(feature_namespaces, True) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(377)processxmlfile() -> try: (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(378)processxmlfile() -> parser.parse(src) (Pdb) s --Call--
/usr/lib64/python2.6/xml/sax/expatreader.py(100)parse() -> def parse(self, source): (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(102)parse() -> source = saxutils.prepare_input_source(source) (Pdb) l 97 98 # XMLReader methods 99 100 def parse(self, source): 101 "Parse an XML document from a URL or an InputSource." 102 -> source = saxutils.prepare_input_source(source) 103 104 self._source = source 105 self.reset() 106 self._cont_handler.setDocumentLocator(ExpatLocator(self)) 107 xmlreader.IncrementalParser.parse(self, source) (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(104)parse() -> self._source = source (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(105)parse() -> self.reset() (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(106)parse() -> self._cont_handler.setDocumentLocator(ExpatLocator(self)) (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(107)parse() -> xmlreader.IncrementalParser.parse(self, source) (Pdb) l 102 source = saxutils.prepare_input_source(source) 103 104 self._source = source 105 self.reset() 106 self._cont_handler.setDocumentLocator(ExpatLocator(self)) 107 -> xmlreader.IncrementalParser.parse(self, source) 108 109 def prepareParser(self, source): 110 if source.getSystemId() is not None: 111 self._parser.SetBase(source.getSystemId()) 112 (Pdb) s --Call-- /usr/lib64/python2.6/xml/sax/xmlreader.py(115)parse() -> def parse(self, source): (Pdb) n /usr/lib64/python2.6/xml/sax/xmlreader.py(116)parse() -> import saxutils (Pdb) /usr/lib64/python2.6/xml/sax/xmlreader.py(117)parse() -> source = saxutils.prepare_input_source(source) (Pdb) l 112 self._bufsize = bufsize 113 XMLReader.__init__(self) 114 115 def parse(self, source): 116 import saxutils 117 -> source = saxutils.prepare_input_source(source) 118 119 self.prepareParser(source) 120 file = source.getByteStream() 121 buffer = file.read(self._bufsize) 122 while buffer != "": (Pdb) n /usr/lib64/python2.6/xml/sax/xmlreader.py(119)parse() -> self.prepareParser(source) (Pdb) /usr/lib64/python2.6/xml/sax/xmlreader.py(120)parse() -> file = source.getByteStream() (Pdb) /usr/lib64/python2.6/xml/sax/xmlreader.py(121)parse() -> buffer = file.read(self._bufsize) (Pdb) /usr/lib64/python2.6/xml/sax/xmlreader.py(122)parse() -> while buffer != "":
Now we know what it's read, and is about to parse: the configure.zcml file from the package lw.portal. (Pdb) pp buffer '<configure\n xmlns="http://namespaces.zope.org/zope"\n xmlns:genericsetup="http://namespaces.zope.org/genericsetup"\n xmlns:monkey="http://namespaces.plone.org/monkey"\n i18n_domain="lw.portal">\n\n <includeDependencies package="." />\n \n <genericsetup:registerProfile\n name="default"\n title="Lotterywest portal policy"\n description="Configures Plone to become the Lotterywest portal"\n for="Products.CMFPlone.interfaces.IPloneSiteRoot"\n provides="Products.GenericSetup.interfaces.EXTENSION"\n directory="profiles/default"\n />\n \n <genericsetup:importStep\n name="lw.portal"\n title="Lotterywest portal policy"\n description="Imperative configuration for lw.portal"\n handler="lw.portal.setuphandlers.setupVarious"\n />\n\n</configure>\n' (Pdb) n
/usr/lib64/python2.6/xml/sax/xmlreader.py(123)parse() -> self.feed(buffer) (Pdb) l 118 119 self.prepareParser(source) 120 file = source.getByteStream() 121 buffer = file.read(self._bufsize) 122 while buffer != "": 123 -> self.feed(buffer) 124 buffer = file.read(self._bufsize) 125 self.close() 126 127 def feed(self, data): 128 """This method gives the raw XML data in the data parameter to (Pdb) s --Call-- /usr/lib64/python2.6/xml/sax/expatreader.py(196)feed() -> def feed(self, data, isFinal = 0): (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(197)feed() -> if not self._parsing: (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(198)feed() -> self.reset() (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(199)feed() -> self._parsing = 1 (Pdb) l 194 # IncrementalParser methods 195 196 def feed(self, data, isFinal = 0): 197 if not self._parsing: 198 self.reset() 199 -> self._parsing = 1 200 self._cont_handler.startDocument() 201 202 try: 203 # The isFinal parameter is internal to the expat reader. 204 # If it is set to true, expat will check validity of the entire (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(200)feed() -> self._cont_handler.startDocument() (Pdb) s --Call-- /usr/lib64/python2.6/xml/sax/handler.py(80)startDocument() -> def startDocument(self): (Pdb) n /usr/lib64/python2.6/xml/sax/handler.py(80)startDocument() -> def startDocument(self): (Pdb) l 75 Note that the locator will return correct information only 76 during the invocation of the events in this interface. The 77 application should not attempt to use it at any other time.""" 78 self._locator = locator 79 80 -> def startDocument(self): 81 """Receive notification of the beginning of a document. 82 83 The SAX parser will invoke this method only once, before any 84 other methods in this interface or in DTDHandler (except for 85 setDocumentLocator).""" (Pdb) n --Return-- /usr/lib64/python2.6/xml/sax/handler.py(80)startDocument()->None -> def startDocument(self): (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(202)feed() -> try: (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(207)feed() -> self._parser.Parse(data, isFinal) (Pdb) l 202 try: 203 # The isFinal parameter is internal to the expat reader. 204 # If it is set to true, expat will check validity of the entire 205 # document. When feeding chunks, they are not normally final - 206 # except when invoked from close. 207 -> self._parser.Parse(data, isFinal) 208 except expat.error, e: 209 exc = SAXParseException(expat.ErrorString(e.code), e, self) 210 # FIXME: when to invoke error()? 211 self._err_handler.fatalError(exc) 212 (Pdb) s
Now we're in C land. --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl() (Pdb) n --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) s --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) s --Call--
/usr/lib64/python2.6/xml/sax/expatreader.py(359)start_namespace_decl() -> def start_namespace_decl(self, prefix, uri): (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(360)start_namespace_decl() -> self._cont_handler.startPrefixMapping(prefix, uri) (Pdb) n --Return--
/usr/lib64/python2.6/xml/sax/expatreader.py(360)start_namespace_decl()->None -> self._cont_handler.startPrefixMapping(prefix, uri) (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) At this point, something is printed to the console. collective.wtf is a dependency of lw.portal, and its ZCML is being included from lw.portal. /home/osc/osc/eggs/collective.wtf-1.0b9-py2.6.egg/collective/wtf/exportimport.py:8: DeprecationWarning: InitializeClass is deprecated. import from App.class_init instead from Globals import InitializeClass At this point, it quits with status 0. If this makes sense to anyone, please do speak up. ;-) Martin
Martin Aspeli wrote:
At this point, something is printed to the console. collective.wtf is a dependency of lw.portal, and its ZCML is being included from lw.portal.
/home/osc/osc/eggs/collective.wtf-1.0b9-py2.6.egg/collective/wtf/exportimport.py:8: DeprecationWarning: InitializeClass is deprecated. import from App.class_init instead from Globals import InitializeClass
Okay, I found out what it does next: It now processes z3c.autoinclude's <includeDependencies />, which basically just goes through the dependencies in setup.py's install_requires and tries to load a configure.zcml for each. It successfully does that for a number of dependencies, up until it hits "collective.xdv" (same package I saw the problem with when I set up a minimal buildout that just had Plone and collective.xdv in it). The pdb log is: (Pdb) l 43 44 def includeDependenciesDirective(_context, package): 45 46 import pdb; pdb.set_trace() 47 48 -> if api.dependencies_disabled(): 49 log.warn('z3c.autoinclude.dependency is disabled but is being invoked by %s' % _context.info) 50 return 51 52 dist = distributionForPackage(package) 53 info = DependencyFinder(dist).includableInfo(['configure.zcml', 'meta.zcml']) (Pdb) n
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(52)includeDependenciesDirective() -> dist = distributionForPackage(package) (Pdb) n
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(53)includeDependenciesDirective() -> info = DependencyFinder(dist).includableInfo(['configure.zcml', 'meta.zcml']) (Pdb) n
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(55)includeDependenciesDirective() -> includeZCMLGroup(_context, info, 'meta.zcml') (Pdb) pp info {'configure.zcml': ['collective.xdv', 'five.grok', 'z3c.jbot'], 'meta.zcml': ['five.grok', 'z3c.jbot']} (Pdb) l 50 return 51 52 dist = distributionForPackage(package) 53 info = DependencyFinder(dist).includableInfo(['configure.zcml', 'meta.zcml']) 54 55 -> includeZCMLGroup(_context, info, 'meta.zcml') 56 includeZCMLGroup(_context, info, 'configure.zcml') 57 58 def includeDependenciesOverridesDirective(_context, package): 59 60 if api.dependencies_disabled(): (Pdb) n
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(56)includeDependenciesDirective() -> includeZCMLGroup(_context, info, 'configure.zcml') (Pdb) s --Call--
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(15)includeZCMLGroup() -> def includeZCMLGroup(_context, info, filename, override=False): (Pdb) n
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(16)includeZCMLGroup() -> includable_zcml = info[filename] (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(20)includeZCMLGroup() -> zcml_context = repr(_context.info) (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(22)includeZCMLGroup() -> for dotted_name in includable_zcml: (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(23)includeZCMLGroup() -> log.debug('including file %s from package %s at %s', filename, dotted_name, zcml_context) (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(22)includeZCMLGroup() -> for dotted_name in includable_zcml: (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(23)includeZCMLGroup() -> log.debug('including file %s from package %s at %s', filename, dotted_name, zcml_context) (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(22)includeZCMLGroup() -> for dotted_name in includable_zcml: (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(23)includeZCMLGroup() -> log.debug('including file %s from package %s at %s', filename, dotted_name, zcml_context) (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(22)includeZCMLGroup() -> for dotted_name in includable_zcml: (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(25)includeZCMLGroup() -> for dotted_name in includable_zcml: (Pdb)
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(26)includeZCMLGroup() -> includable_package = resolve(dotted_name) (Pdb) l 21 22 for dotted_name in includable_zcml: 23 log.debug('including file %s from package %s at %s', filename, dotted_name, zcml_context) 24 25 for dotted_name in includable_zcml: 26 -> includable_package = resolve(dotted_name) 27 if override: 28 includeOverrides(_context, filename, includable_package) 29 else: 30 include(_context, filename, includable_package) 31 (Pdb) n
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(27)includeZCMLGroup() -> if override: (Pdb) pp includable_package <module 'collective.xdv' from '/home/osc/osc/src/collective.xdv/collective/xdv/__init__.pyc'> (Pdb) n
/home/osc/osc/eggs/z3c.autoinclude-0.3.2-py2.6.egg/z3c/autoinclude/zcml.py(30)includeZCMLGroup() -> include(_context, filename, includable_package) (Pdb) s --Call-- Now we're back into zope.configuration:
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(487)include() -> def include(_context, file=None, package=None, files=None): (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(493)include() -> if files: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(496)include() -> elif not file: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(502)include() -> if package and (package.__name__ == 'zope.app'): (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(524)include() -> context = config.GroupingContextDecorator(_context) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(525)include() -> if package is not None: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(526)include() -> context.package = package (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(527)include() -> context.basepath = None (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(529)include() -> if files: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(535)include() -> paths = [context.path(file)] (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(537)include() -> for path in paths: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(538)include() -> if context.processFile(path): (Pdb) l 533 paths = [path for (l, path) in paths] 534 else: 535 paths = [context.path(file)] 536 537 for path in paths: 538 -> if context.processFile(path): 539 f = openInOrPlain(path) 540 logger.debug("include %s" % f.name) 541 542 context.basepath = os.path.dirname(path) 543 context.includepath = _context.includepath + (f.name, ) (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(539)include() -> f = openInOrPlain(path) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(540)include() -> logger.debug("include %s" % f.name) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(542)include() -> context.basepath = os.path.dirname(path) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(543)include() -> context.includepath = _context.includepath + (f.name, ) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(544)include() -> _context.stack.append(config.GroupingStackItem(context)) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(546)include() -> processxmlfile(f, context) (Pdb) s --Call--
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(367)processxmlfile() -> def processxmlfile(file, context, testing=False): (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(372)processxmlfile() -> src = InputSource(getattr(file, 'name', '<string>')) (Pdb) l 367 def processxmlfile(file, context, testing=False): 368 """Process a configuration file 369 370 See examples in tests/text_xmlconfig.py 371 """ 372 -> src = InputSource(getattr(file, 'name', '<string>')) 373 src.setByteStream(file) 374 parser = make_parser() 375 parser.setContentHandler(ConfigurationHandler(context, testing=testing)) 376 parser.setFeature(feature_namespaces, True) 377 try: (Pdb) n
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(373)processxmlfile() -> src.setByteStream(file) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(374)processxmlfile() -> parser = make_parser() (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(375)processxmlfile() -> parser.setContentHandler(ConfigurationHandler(context, testing=testing)) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(376)processxmlfile() -> parser.setFeature(feature_namespaces, True) (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(377)processxmlfile() -> try: (Pdb)
/home/osc/osc/eggs/zope.configuration-3.6.0-py2.6.egg/zope/configuration/xmlconfig.py(378)processxmlfile() -> parser.parse(src) (Pdb) l 373 src.setByteStream(file) 374 parser = make_parser() 375 parser.setContentHandler(ConfigurationHandler(context, testing=testing)) 376 parser.setFeature(feature_namespaces, True) 377 try: 378 -> parser.parse(src) 379 except SAXParseException: 380 raise ZopeSAXParseException(sys.exc_info()[1]), None, sys.exc_info()[2] 381 382 383 def openInOrPlain(filename): (Pdb) s --Call--
/usr/lib64/python2.6/xml/sax/expatreader.py(100)parse() -> def parse(self, source): (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(102)parse() -> source = saxutils.prepare_input_source(source) (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(104)parse() -> self._source = source (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(105)parse() -> self.reset() (Pdb) l 100 def parse(self, source): 101 "Parse an XML document from a URL or an InputSource." 102 source = saxutils.prepare_input_source(source) 103 104 self._source = source 105 -> self.reset() 106 self._cont_handler.setDocumentLocator(ExpatLocator(self)) 107 xmlreader.IncrementalParser.parse(self, source) 108 109 def prepareParser(self, source): 110 if source.getSystemId() is not None: (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(106)parse() -> self._cont_handler.setDocumentLocator(ExpatLocator(self)) (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(107)parse() -> xmlreader.IncrementalParser.parse(self, source) (Pdb) s --Call-- /usr/lib64/python2.6/xml/sax/xmlreader.py(115)parse() -> def parse(self, source): (Pdb) n /usr/lib64/python2.6/xml/sax/xmlreader.py(116)parse() -> import saxutils (Pdb) n /usr/lib64/python2.6/xml/sax/xmlreader.py(117)parse() -> source = saxutils.prepare_input_source(source) (Pdb) n /usr/lib64/python2.6/xml/sax/xmlreader.py(119)parse() -> self.prepareParser(source) (Pdb) /usr/lib64/python2.6/xml/sax/xmlreader.py(120)parse() -> file = source.getByteStream() (Pdb) l 115 def parse(self, source): 116 import saxutils 117 source = saxutils.prepare_input_source(source) 118 119 self.prepareParser(source) 120 -> file = source.getByteStream() 121 buffer = file.read(self._bufsize) 122 while buffer != "": 123 self.feed(buffer) 124 buffer = file.read(self._bufsize) 125 self.close() (Pdb) n /usr/lib64/python2.6/xml/sax/xmlreader.py(121)parse() -> buffer = file.read(self._bufsize) (Pdb) /usr/lib64/python2.6/xml/sax/xmlreader.py(122)parse() -> while buffer != "":
This is the full ZCML that it's loading: (Pdb) print buffer <configure xmlns="http://namespaces.zope.org/zope" xmlns:gs="http://namespaces.zope.org/genericsetup" xmlns:browser="http://namespaces.zope.org/browser" xmlns:zcml="http://namespaces.zope.org/zcml" xmlns:i18n="http://namespaces.zope.org/i18n" i18n_domain="collective.xdv"> <i18n:registerTranslations directory="locales"/> <include package="plone.app.registry" /> <include package="plone.transformchain" /> <gs:registerProfile name="default" title="XDV theme support" description="Installs a control panel to allow on-the-fly theming with xdv" directory="profiles/default" for="Products.CMFPlone.interfaces.IPloneSiteRoot" provides="Products.GenericSetup.interfaces.EXTENSION" /> <!-- Transform order 8850 - apply XDV transform --> <adapter name="collective.xdv.transform" factory=".transform.XDVTransform" /> <subscriber for=".interfaces.ITransformSettings plone.registry.interfaces.IRecordModifiedEvent" handler=".transform.invalidateCache" /> <browser:page name="xdv-settings" for="Products.CMFPlone.interfaces.IPloneSiteRoot" class=".browser.TransformSettingsControlPanel" permission="cmf.ManagePortal" /> <!-- Set X-XDV header in the request if XDV is enabled --> <subscriber for="ZPublisher.interfaces.IPubAfterTraversal" handler=".header.setHeader" /> <!-- Utility view - use in portal_css or similar as portal/@@xdv-check/enabled" --> <browser:page name="xdv-check" for="Products.CMFPlone.interfaces.IPloneSiteRoot" class=".browser.Utility" permission="zope.Public" allowed_attributes="enabled domain_enabled" /> <browser:resource name="collective.xdv.gif" image="icon.gif" /> </configure> (Pdb) n
/usr/lib64/python2.6/xml/sax/xmlreader.py(123)parse() -> self.feed(buffer) (Pdb) l 118 119 self.prepareParser(source) 120 file = source.getByteStream() 121 buffer = file.read(self._bufsize) 122 while buffer != "": 123 -> self.feed(buffer) 124 buffer = file.read(self._bufsize) 125 self.close() 126 127 def feed(self, data): 128 """This method gives the raw XML data in the data parameter to (Pdb) s --Call-- /usr/lib64/python2.6/xml/sax/expatreader.py(196)feed() -> def feed(self, data, isFinal = 0): (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(197)feed() -> if not self._parsing: (Pdb) l 192 name) 193 194 # IncrementalParser methods 195 196 def feed(self, data, isFinal = 0): 197 -> if not self._parsing: 198 self.reset() 199 self._parsing = 1 200 self._cont_handler.startDocument() 201 202 try: (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(198)feed() -> self.reset() (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(199)feed() -> self._parsing = 1 (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(200)feed() -> self._cont_handler.startDocument() (Pdb) /usr/lib64/python2.6/xml/sax/expatreader.py(202)feed() -> try: (Pdb) l 197 if not self._parsing: 198 self.reset() 199 self._parsing = 1 200 self._cont_handler.startDocument() 201 202 -> try: 203 # The isFinal parameter is internal to the expat reader. 204 # If it is set to true, expat will check validity of the entire 205 # document. When feeding chunks, they are not normally final - 206 # except when invoked from close. 207 self._parser.Parse(data, isFinal) (Pdb) n /usr/lib64/python2.6/xml/sax/expatreader.py(207)feed() -> self._parser.Parse(data, isFinal) (Pdb) l 202 try: 203 # The isFinal parameter is internal to the expat reader. 204 # If it is set to true, expat will check validity of the entire 205 # document. When feeding chunks, they are not normally final - 206 # except when invoked from close. 207 -> self._parser.Parse(data, isFinal) 208 except expat.error, e: 209 exc = SAXParseException(expat.ErrorString(e.code), e, self) 210 # FIXME: when to invoke error()? 211 self._err_handler.fatalError(exc) 212 (Pdb) n
Boom! (python-env)osc@lwpn-osb-webback-2:~/osc> echo $? 0 I stepped into the self.parser.Parse() line to get some more context (this is C code):
/usr/lib64/python2.6/xml/sax/expatreader.py(207)feed() -> self._parser.Parse(data, isFinal) (Pdb) s --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl() (Pdb) n --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) n --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) n --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) n --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) n --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) n --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) n --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) n --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(839)StartNamespaceDecl()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) n --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(479)CharacterData()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Return--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(604)StartElement()->None (Pdb) --Call--
/usr/src/packages/BUILD/Python-2.6/Modules/pyexpat.c(656)EndElement()->None (Pdb) BOOM. So this is still in pyexpat C code as far as I can tell. :-( Martin
On Tue, Apr 6, 2010 at 10:25 PM, Martin Aspeli <optilude+lists@gmail.com> wrote:
So this is still in pyexpat C code as far as I can tell. :-(
This is saddening. I've not managed a 64-bit sandbox, which I suspect is what I really need to debug that. Will shoot for this weekend, since last didn't pan out. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Fred Drake wrote:
On Tue, Apr 6, 2010 at 10:25 PM, Martin Aspeli<optilude+lists@gmail.com> wrote:
So this is still in pyexpat C code as far as I can tell. :-(
This is saddening.
But on the other hand, your dedication in helping me find a fix is heartening. ;)
I've not managed a 64-bit sandbox, which I suspect is what I really need to debug that. Will shoot for this weekend, since last didn't pan out.
I have a slight suspicion that lxml is involved somewhere. I've managed to make it crash reliably simply by doing a parse of an XML file, and I think such a parse may be happening as a side effect of a module import caused by that very ZCML file. That doesn't really explain why pdb dies when it dies, but I don't know how the C/Python boundary is working here (especially since I can't really see the C code). I'm re-building for the n-th time, so I'll check again tomorrow when that's done. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
On Wed, Apr 7, 2010 at 9:29 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
I have a slight suspicion that lxml is involved somewhere. I've managed to make it crash reliably simply by doing a parse of an XML file, and I think such a parse may be happening as a side effect of a module import caused by that very ZCML file.
If it can be nailed on lxml, I'll be happy. :-) Not because I want lxml to be at fault so much as I want Expat not to be. Invoking lxml as a side effect of import seems like it should be fairly easy to identify and comment out for testing purposes; if the ZCML configuration can then complete, you've implicated your suspect. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
participants (6)
-
Fred Drake -
Marius Gedminas -
Martin Aspeli -
Sylvain Viollon -
Tim Hoffman -
Tres Seaver