hi list i try to execute a shell command (namely pdftk) from python/zope to fill out fdf forms in a pdf document this is my command, which is generated: pdftk /tmp/tmpRy0O_T.pdf fill_form /tmp/tmpz4mEG1.fdf output /tmp/tmpx1TxTD.pdf flatten the pdf and fdf files are temporary files i have a testcase which runs the surrounding method successfully (python only no zope), but wenn i access the method from zope it hangs when the pdftk command tries to write to the output files and i have to kill -9 the pdftk processes i've already tried with popen3, os.system etc. this problem does NOT occur on macos x, only on linux, for windows i don't know. my linux: Linux bd-pc 2.6.10-gentoo-r5 #2 Tue Jan 18 18:20:46 CET 2005 i686 Intel(R) Pentium(R) 4 CPU 2.00GHz GenuineIntel GNU/Linux is it possible that this is a gcj problem, because pdftk is java compiled with gcc? output of gcc -v Reading specs from /usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/specs Configured with: /var/tmp/portage/gcc-3.3.5-r1/work/gcc-3.3.5/configure --enable-version-specific-runtime-libs --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/3.3.5 --includedir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5/info --with-gxx-include-dir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/include/g++-v3 --host=i686-pc-linux-gnu --disable-altivec --enable-nls --without-included-gettext --enable-__cxa_atexit --enable-clocale=gnu --with-system-zlib --disable-checking --disable-werror --disable-libunwind-exceptions --enable-shared --enable-threads=posix --disable-multilib --enable-java-awt=gtk --enable-languages=c,c++,java Thread model: posix gcc version 3.3.5 (Gentoo Linux 3.3.5-r1, ssp-3.3.2-3, pie-8.7.7.1) thx in advance, bernd
I don't know about your specific situation, but we do lots of file writes from external methods. The only trick is to make sure that you are opening files and writing to a directory that zope has access to (eg. for us: /apps/zope/var). HTH Jonathan ----- Original Message ----- From: "Bernd Dorn" <zope-mailinglist@mopa.at> To: <zope@zope.org> Sent: February 12, 2005 4:09 AM Subject: [Zope] shell command in zope on linux
hi list
i try to execute a shell command (namely pdftk) from python/zope to fill out fdf forms in a pdf document
this is my command, which is generated: pdftk /tmp/tmpRy0O_T.pdf fill_form /tmp/tmpz4mEG1.fdf output /tmp/tmpx1TxTD.pdf flatten
the pdf and fdf files are temporary files
i have a testcase which runs the surrounding method successfully (python only no zope), but wenn i access the method from zope it hangs when the pdftk command tries to write to the output files and i have to kill -9 the pdftk processes
i've already tried with popen3, os.system etc.
this problem does NOT occur on macos x, only on linux, for windows i don't know.
my linux: Linux bd-pc 2.6.10-gentoo-r5 #2 Tue Jan 18 18:20:46 CET 2005 i686 Intel(R) Pentium(R) 4 CPU 2.00GHz GenuineIntel GNU/Linux
is it possible that this is a gcj problem, because pdftk is java compiled with gcc?
output of gcc -v Reading specs from /usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/specs Configured with: /var/tmp/portage/gcc-3.3.5-r1/work/gcc-3.3.5/configure --enable-version-specific-runtime-libs --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/3.3.5 --includedir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5/info --with-gxx-include-dir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/include/g+ +-v3 --host=i686-pc-linux-gnu --disable-altivec --enable-nls --without-included-gettext --enable-__cxa_atexit --enable-clocale=gnu --with-system-zlib --disable-checking --disable-werror --disable-libunwind-exceptions --enable-shared --enable-threads=posix --disable-multilib --enable-java-awt=gtk --enable-languages=c,c++,java Thread model: posix gcc version 3.3.5 (Gentoo Linux 3.3.5-r1, ssp-3.3.2-3, pie-8.7.7.1)
thx in advance, bernd _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
hm, i don' t think that this is the problem, because the test runs with the same user as the zope process On 12.02.2005, at 13:52, Jonathan Hobbs wrote:
I don't know about your specific situation, but we do lots of file writes from external methods. The only trick is to make sure that you are opening files and writing to a directory that zope has access to (eg. for us: /apps/zope/var).
HTH
Jonathan
----- Original Message ----- From: "Bernd Dorn" <zope-mailinglist@mopa.at> To: <zope@zope.org> Sent: February 12, 2005 4:09 AM Subject: [Zope] shell command in zope on linux
hi list
i try to execute a shell command (namely pdftk) from python/zope to fill out fdf forms in a pdf document
this is my command, which is generated: pdftk /tmp/tmpRy0O_T.pdf fill_form /tmp/tmpz4mEG1.fdf output /tmp/tmpx1TxTD.pdf flatten
the pdf and fdf files are temporary files
i have a testcase which runs the surrounding method successfully (python only no zope), but wenn i access the method from zope it hangs when the pdftk command tries to write to the output files and i have to kill -9 the pdftk processes
i've already tried with popen3, os.system etc.
this problem does NOT occur on macos x, only on linux, for windows i don't know.
my linux: Linux bd-pc 2.6.10-gentoo-r5 #2 Tue Jan 18 18:20:46 CET 2005 i686 Intel(R) Pentium(R) 4 CPU 2.00GHz GenuineIntel GNU/Linux
is it possible that this is a gcj problem, because pdftk is java compiled with gcc?
output of gcc -v Reading specs from /usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/specs Configured with: /var/tmp/portage/gcc-3.3.5-r1/work/gcc-3.3.5/configure --enable-version-specific-runtime-libs --prefix=/usr --bindir=/usr/i686-pc-linux-gnu/gcc-bin/3.3.5 --includedir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/include --datadir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5 --mandir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5/man --infodir=/usr/share/gcc-data/i686-pc-linux-gnu/3.3.5/info --with-gxx-include-dir=/usr/lib/gcc-lib/i686-pc-linux-gnu/3.3.5/ include/g+ +-v3 --host=i686-pc-linux-gnu --disable-altivec --enable-nls --without-included-gettext --enable-__cxa_atexit --enable-clocale=gnu --with-system-zlib --disable-checking --disable-werror --disable-libunwind-exceptions --enable-shared --enable-threads=posix --disable-multilib --enable-java-awt=gtk --enable-languages=c,c++,java Thread model: posix gcc version 3.3.5 (Gentoo Linux 3.3.5-r1, ssp-3.3.2-3, pie-8.7.7.1)
thx in advance, bernd _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Bernd Dorn wrote at 2005-2-12 10:09 +0100:
... i try to execute a shell command (namely pdftk) from python/zope to fill out fdf forms in a pdf document
this is my command, which is generated: pdftk /tmp/tmpRy0O_T.pdf fill_form /tmp/tmpz4mEG1.fdf output /tmp/tmpx1TxTD.pdf flatten
the pdf and fdf files are temporary files
i have a testcase which runs the surrounding method successfully (python only no zope), but wenn i access the method from zope it hangs when the pdftk command tries to write to the output files and i have to kill -9 the pdftk processes
i've already tried with popen3, os.system etc.
this problem does NOT occur on macos x, only on linux, for windows i don't know.
This question is more a "pdftk" one than a Zope one... Depending on how you start Zope, the process environment may well be quite different from that of a normal Python script: * the process may not have a control terminal * the process may have "stdin", "stdout" and "stderr" redirected to "/dev/null" * the working directory may be different Try to start Zope in the foreground ("bin/zopectl fg"). In this case, the first two points above do not apply. See whether "pdftk" then runs. You can try to attach the "pdftk" process with GDB to check where it hangs. The program should not be stripped when you try this. -- Dieter
On Feb 12, 2005, at 4:09 AM, Bernd Dorn wrote:
i have a testcase which runs the surrounding method successfully (python only no zope), but wenn i access the method from zope it hangs when the pdftk command tries to write to the output files and i have to kill -9 the pdftk processes
This may be another case where Python's signal handling problems are screwing up the way the child process executes. The fully blocked signal mask in inherited by the process you spawn, and many programs do not expect to be starting up with all their signals blocked. One way of proving this theory is to alter your test case to run in a thread rather than be executed by the main thread. Your requests to Zope will always be running in non-main threads, and so if your test is in a thread you will get a better apples to apples comparison. The signal problem is definitely the reason why you have to "kill -9" rather than a simple "kill -INT" If the test case stops working when run from a thread, you might want to try the following patch: <http://sourceforge.net/tracker/? func=detail&aid=1088832&group_id=5470&atid=305470> Or at the very least, arrange some other way for pdftk to get access to its signals. The following C program is a rough sketch towards what you would need. If you compile it with a name like "unblock", you would run it like this: unblock pdftk /tmp/tmpRy0O_T.pdf fill_form /tmp/tmpz4mEG1.fdf output /tmp/tmpx1TxTD.pdf flatten /* unblock.c unblock signals before running the specified process */ #include <stdio.h> #include <signal.h> int main(int argc, char **argv) { sigset_t blank; char *prog; int err; sigemptyset(&blank); sigprocmask(SIG_SETMASK, &blank, NULL); argv++; prog = *argv; err = execvp(prog, argv); if(err) { fprintf(stderr, "Error running %s: %d", prog, err); } return err; }
Andrew Langmead wrote:
On Feb 12, 2005, at 4:09 AM, Bernd Dorn wrote:
i have a testcase which runs the surrounding method successfully (python only no zope), but wenn i access the method from zope it hangs when the pdftk command tries to write to the output files and i have to kill -9 the pdftk processes
This may be another case where Python's signal handling problems are screwing up the way the child process executes. The fully blocked signal mask in inherited by the process you spawn, and many programs do not expect to be starting up with all their signals blocked.
One way of proving this theory is to alter your test case to run in a thread rather than be executed by the main thread. Your requests to Zope will always be running in non-main threads, and so if your test is in a thread you will get a better apples to apples comparison.
The signal problem is definitely the reason why you have to "kill -9" rather than a simple "kill -INT"
If the test case stops working when run from a thread, you might want to try the following patch:
<http://sourceforge.net/tracker/? func=detail&aid=1088832&group_id=5470&atid=305470>
Or at the very least, arrange some other way for pdftk to get access to its signals. The following C program is a rough sketch towards what you would need. If you compile it with a name like "unblock", you would run it like this:
unblock pdftk /tmp/tmpRy0O_T.pdf fill_form /tmp/tmpz4mEG1.fdf output /tmp/tmpx1TxTD.pdf flatten
/* unblock.c unblock signals before running the specified process */ #include <stdio.h> #include <signal.h>
int main(int argc, char **argv) { sigset_t blank; char *prog; int err;
sigemptyset(&blank); sigprocmask(SIG_SETMASK, &blank, NULL); argv++; prog = *argv; err = execvp(prog, argv); if(err) { fprintf(stderr, "Error running %s: %d", prog, err); } return err; }
thanks for your detailed answer ... it is exactly the problem you described and this is also the reason why it works on my mac machine where i've installed python 2.4, and the signal stuff is 2.3 specific i did not try the python patch, but used your unclock.c, and now it works, great!! thx a lot, bernd
i did not try the python patch, but used your unclock.c, and now it works, great!!
If possible, try the patch at some point. Apparently there was some fear in adding this to 2.3.5 (I say apparently because no one brought up their objections with me directly. I was just ignored entirely.) The more evidence that this very simple change (essentially removing the botch made to threads right before 2.2 went beta) has no side effects, the better chance we have of a supported python version and a supported zope version that work correctly together. At the very least, post your test case to the python bug collector as a bug. Make sure that the Python developers realize that this is a real problem effecting real people.
On Tue, Feb 15, 2005 at 12:18:21PM -0500, Andrew Langmead wrote: | >i did not try the python patch, but used your unclock.c, and now it | > works, great!! | | If possible, try the patch at some point. Apparently there was some fear | in adding this to 2.3.5 (I say apparently because no one brought up | their objections with me directly. I was just ignored entirely.) The patch does resolve the problem I had. I haven't run it extensively enough to know if there are any side-effects from it, though. -D --
Linux is not user-friendly. It -is- user-friendly. It is not ignorant-friendly and idiot-friendly. (Seen somewhere on the net.)
www: http://dman13.dyndns.org/~dman/ jabber: dman@dman13.dyndns.org
Hello Andrew. On Sat, Feb 12, 2005 at 08:20:48PM -0500, Andrew Langmead wrote: | On Feb 12, 2005, at 4:09 AM, Bernd Dorn wrote: | | >i have a testcase which runs the surrounding method successfully | >(python only no zope), but wenn i access the method from zope it hangs | >when the pdftk command tries to write to the output files and i have | >to kill -9 the pdftk processes | > | | This may be another case where Python's signal handling problems are | screwing up the way the child process executes. The fully blocked | signal mask in inherited by the process you spawn, and many programs do | not expect to be starting up with all their signals blocked. [...] | Or at the very least, arrange some other way for pdftk to get access to | its signals. The following C program is a rough sketch towards what you | would need. If you compile it with a name like "unblock", you would run | it like this: [...] Your message (partially quoted above) was quite timely. I found it via google while trying to resolve a problem I was having, and your "unblock.c" worked perfectly. My situation : Zope 2.7.4 (and 2.7.1) on python 2.3 on linux 2.6.7 on a PowerMac G4. From zope I need to run a java program passing it some data from the zodb and then returning the java program's output to the browser. (I am using IBM's JDK 1.4.2 because it is really the only one available) My zope/python code runs a shell script that runs java with the necessary environment (namely the classpath). It works fine with Sun's JDK 1.5.0 on my x86 development system. On the ppc system 'java -version' worked, but the real program would consume all of the cpu (in system, not user space) and never finish. I had to send it SIGKILL to stop it (SIGTERM had no effect). When I use your "unblock.c" program I have no problems at all. I will now try the SF patch you referenced and verify that that resolves the issue as well. Thanks, -D -- \begin{humor} Disclaimer: If I receive a message from you, you are agreeing that: 1. I am by definition, "the intended recipient" 2. All information in the email is mine to do with as I see fit and make such financial profit, political mileage, or good joke as it lends itself to. In particular, I may quote it on USENET or the WWW. 3. I may take the contents as representing the views of your company. 4. This overrides any disclaimer or statement of confidentiality that may be included on your message \end{humor} www: http://dman13.dyndns.org/~dman/ jabber: dman@dman13.dyndns.org
On Feb 24, 2005, at 10:54 AM, Derrick Hudson wrote:
Your message (partially quoted above) was quite timely. I found it via google while trying to resolve a problem I was having, and your "unblock.c" worked perfectly.
Quite timely? Don't worry, if you missed this one, you could have caught the next one. The issue comes up semi-frequently and I wind up sounding like a broken record. I'm happy that the patch and the workaround both resolve your issue.
On Thu, Feb 24, 2005 at 12:42:48PM -0500, Andrew Langmead wrote: | On Feb 24, 2005, at 10:54 AM, Derrick Hudson wrote: | >Your message (partially quoted above) was quite timely. I found it | >via google while trying to resolve a problem I was having, and your | >"unblock.c" worked perfectly. | | Quite timely? Well, it was long enough ago that google had indexed at least one archive containing the message. | Don't worry, if you missed this one, you could have | caught the next one. Perhaps ... I wasn't finding much that was helpful as I searched. | The issue comes up semi-frequently and I wind up | sounding like a broken record. :-(. I guess the bug report needs more references to these issues. | I'm happy that the patch and the workaround both resolve your issue. -D -- Bugs come in through open windows. Keep Windows shut! www: http://dman13.dyndns.org/~dman/ jabber: dman@dman13.dyndns.org
participants (5)
-
Andrew Langmead -
Bernd Dorn -
Derrick Hudson -
Dieter Maurer -
Jonathan Hobbs