--On Montag, 8. November 2004 8:38 Uhr +0100 Jonas Meurer <jonas@freesources.org> wrote:

On 08/11/2004 Andreas Jung wrote:

...

The recommended Python version for Zope 2.6 is Python 2.1.X. as documented in various places. There was never an official support for Python 2.2 or higher.

yes, i already read this in the Zope Book and on other places, but some guy in #zope on freenode.net told me that python2.2 support was added in zope 2.6.3, and finally debian zope packages run zope 2.6.4 with python2.2.

Things are as they are. There is nothing in Zope 2.6 built-in to support Python 2.2. Although some people were running Zope 2.6 with any other version than Python 2.1.3 then they run it at their own risk. System recommendations are there to define a supported environment. if Debian installs Zope packages with the wrong versions then Debian should be blamed for not reading-the-fine-documentation and for wrong packaging. -aj

On 08/11/2004 Tres Seaver wrote:

While Andreas is correct about the supported version, I have successfully installed Zope 2.6.x for clients using Python 2.2.3, and saw some modest performance benefits; those instances still run as high-traffic, mission-critical applications for those clients.

our zope instances also ran quite well until *something* happened. we use zope 2.6.4 and python 2.2.3 from debian/testing.

First, if you are going to run Zope 2.6.x with Python 2.2.x, please ensure that it runs with 2.2.3; there are known crash bugs in earlier 2.2.x versions which make Zope particularly hard to run (likewise with 2.1.3; 2.1.2 and earlier had crash bugs discovered by Zope).

i ensured this, all zope 2.6.4 instances running with python2.2, use python 2.2.3, as no other python2.2 version is installed.

Second, I would strongly recommend compiling and running Zope with a separate, "dedicated" Python (installed into /usr/local/ZopePython, or some such). The rationale here is that the system Python may be packaged inappropriately for Zope (e,g,, RedHat builds with 4-byte unicode; some older systems did not have large-file support), and that the system python may be upgraded / munged by sysadmin operations which don't realize that they may impact Zope.

that sounds interessting. so you recomment to not use the default installed python 2.2.3, but compile an own one for running only zope? i'm not quite sure whether i understand the intension of this ... debian has several python packages for every major version (2.1, 2.2, 2.3), and these are only updated inside this develpment tree (2.2.2 to 2.2.3 for example). as zope packages depend on python2.2 packages, i'm sure that pyton2.2 is packaged appropriately for zope. or is there a general problem with other problems using the same python binary as zope?

If you can reproduce the issue with a Python you built yourself, and a Zope you built and installed from source, then this list can help you. Otherwise, you need to report a packaging bug to the Debian maintainers, as Andreas suggests.

as already mentioned in the initial post, i can reproduce the bug with zope built and installed from sources, using the debian python binary. i can reproduce it with python2.2 from debian, what is used by the zope packages as well, and i can also reproduce it with python2.1 from debian. the python2.1 package was installed recently, and python2.1 is _only_ used by zope on this machine. anyway the problem also occurs with a zope 2.6.4 built and installed from sources with python2.1 from debian. anyway i'm not able to reproduce the bug on _any_ other machine, regardless which python/zope versions i use. if you're still convinced that using a self-compiled python-version on the buggy machine will give further insight, i'll happily try that version as well, but currently i don't see any need for that. bye jonas

On 09/11/2004 Tres Seaver wrote:

...

as zope packages depend on python2.2 packages, i'm sure that pyton2.2 is packaged appropriately for zope.

Not necessarily; the Python package maintainer may not actively know about Zope.

at least zope maintainers should verify this.

Try the following:

$ for x in 1, 2; do \ /usr/bin/python2.$x /usr/lib/python2.$x/tests/test_largefile \ done

debian doesn't shup test_largefile with the packages, so i grepped it from python2.2.3 sources and copied it to /usr/lib/python2.*/test/test_largefile both runs gave only ...yes ouptuts, so i assume that largefile support is compiled into python packages.

If either of those tests fail, that would be one explanation, especially if your Data.fs is larger than 2 Gb. I *do* recommend compiling your own Python, at least to test whether it helps.

ok, i finally compiled python2.2 sources, after configuring them with: h:/opt/Python-2.2.3# ./configure --prefix=/opt --with-fpectl --enable-ipv6 # make # make install then i installed zope 2.6.4 from sources, using this python: h:/opt/Zope-2.6.4-src# /opt/bin/python2.2 wo_pcgi.py # chown root.zope var # mkdir var/log # chown zope.zope var/* # chmod o+t var # gzip -dc ../TinyTablePlus-0.9.tgz | tar -xf - # export EVENT_LOG_FILE="$basedir/var/log/Z2-event.log" # export EVENT_LOG_SEVERITY="-100" # export Z_REALM="KNOW-IT CMS" # export ZOPE_SECURITY_POLICY="PYTHON" # /opt/bin/python2.2 ./z2.py -a 127.0.0.1 -d 127.0.0.1 \ -u zope -w 9680 -W - -f 9621 -p - -F - -m - -l log/Z2.log \ -M log/Z2-detailed.log # after doing that, i contact to http://localhost:9680/, import some products into Control_Panel/Products and our application into the Root Folder, run the process that causes zope to crash, and _.oO_it_crashs_0o._ so now i'm absolutely sure that it's not a matter of python version. i tried python2.1 and python2.2 from debian, and python2.2 compiled from sources, and all fail. bye jonas

On 09/11/2004 Tim Peters wrote:

I wonder whether this is just a rediscovery of the interaction between non-standard (according to POSIX) LinuxThreads signal handling and revision 2.33 of Python's thread_pthread.h:

http://mail.zope.org/pipermail/zope-dev/2004-May/022813.html

This was always discussed before in the Zope world as "a difference" between Pythons in the 2.1 and 2.3 lines, since Zope skipped over the Python 2.2 line. But rev 2.33 was also part of Python 2.2, so the same problems on LinuxThreads systems would exist under Pythons in the 2.2 line too.

yes, i guess that this is about the same topic. sorry for rediscovering. bye jonas

On 09/11/2004 Tim Peters wrote:

[Jonas Meurer]

...

yes, i guess that this is about the same topic. sorry for rediscovering.

Hmm. I'm not at all irked that you rediscovered a problem <wink>. My purpose in posting was to point you at previous discussion of the same problem, so you can make quicker progress in finding a workaround that suits you.

hehe, i still would prefer solving the bug itself rather than building more and more workarounds. there is something that crashes zope. Event.log gives the information, that zope received a SIGHUP, and thus restarts. this problem can't be reproduced on any other machine, even if i use the same zope _and_ python versions, exactly similar configured and installed. anyway it occurs on _any_ zope installation/instance on this one particular server, and i still have no real glue what is causing it. bye jonas

On 10/11/2004 Tres Seaver wrote:

causing the problem. You can "fix" the problem by one of two means:

- Run with a newer, NPTL-enabled glibc. To stay on Debian, I belive this would entail upgrading from Woody to Sarge. I am only lightly familiar with Debian, so perhaps somebody can correct me?

thanks for that suggestion, anyway i'm quite sure that this isn't the problem, as all machines where i try to reproduce this bug run the same libc6 version (2.3.2.ds1). if it could be a kernel matter, the failing box runs 2.4.27, but i already tested this on another machine with 2.4.27 kernel as well, and it wasn't even reproducable there. could it be a matter of kernel configuration? other things that came into my mind are: -> the failing box has 3 virtual ips on virtual devices, maybe this causes problems? nevertheless i started zope with -a 127.0.0.1 and -d 127.0.0.1, so could this be an issue? -> maybe some python modules cause zope to fail? this would mean, that self-compiled python uses the installed debian python modules as well. the only modules installed on the failing box, but not on the other test machines are python2.2-egenix-mxdatetime, python2.2-egenix-mxtools, python2.2-optik and python2.2-photo, as these are required by Zope products we use.

- Install an alternate threading package, and compile Python to use it instead of the default one in libc. Note that I have *no* idea how to do this even on systems where I am more comfortable than with Debian.

so you mean replacing my libc6? i'm quite confident that my libc6 has threating support. are you discussing the issue, that python2.2 doesn't restart zope after SIGSEGV(11)? in this case ignore my response *g* this thread is about the problem itself, why zope SIGSEGVs at all. bye jonas

zope 2.6.4 compiled with python2.2 doesn't do this, in other words it fails. event.log tells me something like other processes are already accessing and thus locking the Data.fs.

is this a known issue, are workarounds available?

In python 2.2 and 2.3, signals are blocked on all threads except the main thread. This causes problems with synchronous signals like SIGSEGV which are always sent to the faulting thread. See <https://sourceforge.net/tracker/index.php? func=detail&aid=756924&group_id=5470&atid=105470> or <https://sourceforge.net/tracker/index.php? func=detail&aid=468347&group_id=5470&atid=305470> for more details. Since Python 2.4 hasn't been well tested for Zope yet, the suggestions that I can think of are: 1. Backport the threading/signal changes to 2.3. 2. Continue using 2.1. 3. Spearhead a movement to test Zope with Python 2.4. (someone has to be the first over the hill.) 4. Find what is causing the seg fault, and avoid the issue entirely.

[Andrew Langmead] ....

Since Python 2.4 hasn't been well tested for Zope yet,

I'll note that Zopes 3 and 2.8 are routinely tested with the 2.4 pre-releases (2.4 doesn't exist yet, of course), with little trouble.

the suggestions that I can think of are:

1. Backport the threading/signal changes to 2.3. 2. Continue using 2.1. 3. Spearhead a movement to test Zope with Python 2.4. (someone has to be the first over the hill.) 4. Find what is causing the seg fault, and avoid the issue entirely.

I like #3 best <wink>. Volunteers to work on Python 2.2 vanished long ago, and chances are good 2.3.5 will be the last release in the 2.3 line (2.3.5 will come out after 2.4, probably before this year ends). I don't know whether anyone did #1, but if not, there isn't much time remaining to do so before 2.3.5. Wasn't this specific problem also unique to LinuxThreads? That is, weren't Linux systems with POSIX-conforming NPTL threads immune to it? If so, add 5. Switch to NPTL threads.

[Tim Peters]

...

I like #3 best <wink>. Volunteers to work on Python 2.2 vanished long ago, and chances are good 2.3.5 will be the last release in the 2.3 line (2.3.5 will come out after 2.4, probably before this year ends). I don't know whether anyone did #1, but if not, there isn't much time remaining to do so before 2.3.5.

[Andrew Langmead]

The realinesigs7.patch in <https://sourceforge.net/tracker/? func=detail&aid=960406&group_id=5470&atid=305470> is pretty close to what Michael Hudson committed to 2.4, and applies pretty cleanly to 2.3.4. Someone well connected to the python development community <wink> might want to lobby for that change to be backported to the 2.3 branch. If someone wants a patch made that consists precisely of the changes mwh put into 2.4, I might be able to work that up.

The good news is that backporting bugfixes isn't a matter of lobbying. Changes didn't get made for 2.3.4, despite lobbying, because there wasn't yet agreement at that time about what *should* be done. It was trying to ram in 2.3.4 changes in the absence of agreement that required lobbing then, and that lobbying failed (and it should have failed). The bad news is that backporting bugfixes is almost purely a "scratch your own itch" thing -- backports happen if, and only if, someone volunteers to do all the work. If you volunteer to produce an appropriate patch for 2.3.5, upload it and set the priority to 9. I'll run interference then <wink> to assure it doesn't get dropped on the floor. It sounds like mwh should be assigned as the reviewer.

...

Wasn't this specific problem also unique to LinuxThreads? That is, weren't Linux systems with POSIX-conforming NPTL threads immune to it?

At first it seemed that way, or at least I came up with a Linux specific work around.

By "this specific problem", I mean only the business about Zope not being able to restart after some kinds of crashes. A Zope installation shouldn't care, e.g., whether Python's insane interface to GNU readline's insane implementation works in every endcase.

After I was knee deep in things, I discovered that the root cause came down to two things. 1) That Modules/readline.c was working in a non-thread-safe way (setting a signal handler that would longjmp() back out of the readline library, ignoring the fact that the thread that handled the signal may not have been the one that entered readline) and 2) That 2.2 tried to fix the readline problem by blocking all but the main thread, which would be the only one to use readline.

What 2.4 chances do do is back out of the 2.2/2.3 behavior of blocking signals on threads, resuming the 2.1 behavior. (The C signal handler runs on any thread, sets a flag, and the main thread runs the python handler at the next available moment.) and then changed Modules/readline.c to be thread-safe (and then thankfully Michael Hudson fixed my fix.)

Since all the problems were pthread/signal related, I guess another possibility would be to use a python with threading other than pthreads (the GNU pth library. Win32, Mac OSX's mach threads, etc.) but the suggestion to change deployment platforms seems pretty severe.

By most accounts I've seen, current NPTL is favored over LinuxThreads on all counts anyway. I don't feel bad about recommending that people fix their problems as a side effect of improving their lives <wink>.