Zope crashes on request of certain page
Dear Zope list, I have a serious problem with Zope. Every time a certain page is requested by a web browser for the second or third time (first time always works) the whole Zope server crashes :-( The only thing special about this page are calls of external Python methods from a custom extension. These methods don't do any special magic. They just open a MySQL database (using the MySQLdb module version 0.9.1) and return some HTML output. I was able to catch a somewhat informative error message when Zope was running in debug mode: 2006-03-27T12:18:57 PANIC(300) z2 Startup exception Traceback (innermost last): File /opt/zope/z2.py, line 495, in ? (Object: ZLogger) File <string>, line 1, in ? File /opt/zope/lib/python/Zope/__init__.py, line 37, in ? File /opt/zope/lib/python/ZODB/FileStorage.py, line 202, in __init__ File /opt/zope/lib/python/ZODB/lock_file.py, line 29, in lock_file StorageSystemError: Could not lock the database file. There must be another process that has opened the file. Operating system is SuSE Linux 8.0, Zope version is 2.5.0, Python 2.2, MySQL 3.23.48. I know, pretty old but upgrading any of these software components should be avoided due to emerging costs unless absolutely necessary! So if you have any idea how to fix this problem besides "upgrade to the newest versions" I would be glad to here from you! Thank you very much! -- Sincerely yours Sven Jacobs
--On 7. April 2006 10:34:08 +0200 Sven Jacobs <jacobs@tedsoft.de> wrote:
The only thing special about this page are calls of external Python methods from a custom extension. These methods don't do any special magic. They just open a MySQL database (using the MySQLdb module version 0.9.1) and return some HTML output.
A buggy extension can crash Zope..but this is not a Zope problem but a problem of the extension. Check for core dumps, enable core dumps using (ulimit from the shell), use gdb to debug core dumps if you can find some. ...that's the way to hunt down the problem.
I was able to catch a somewhat informative error message when Zope was running in debug mode:
2006-03-27T12:18:57 PANIC(300) z2 Startup exception Traceback (innermost last): File /opt/zope/z2.py, line 495, in ? (Object: ZLogger) File <string>, line 1, in ? File /opt/zope/lib/python/Zope/__init__.py, line 37, in ? File /opt/zope/lib/python/ZODB/FileStorage.py, line 202, in __init__ File /opt/zope/lib/python/ZODB/lock_file.py, line 29, in lock_file StorageSystemError: Could not lock the database file. There must be another process that has opened the file.
Possibly a followup error. Check if there is some other process locking the file (using lsof). If not, remove the Data.fs.lock file.
Operating system is SuSE Linux 8.0, Zope version is 2.5.0,
2.5.0? To be honest: upgrade _at_least_ to the latest 2.5.X version. Don't expect that anyone seriously cares about a) very old releases and b) .0 releases. -aj ----------------------------------------------------------------------- - Andreas Jung ZOPYX Ltd. & Co KG - - E-mail: info@zopyx.com Web: www.zopyx.com, www.zopyx.de - -----------------------------------------------------------------------
----- Original Message ----- From: "Sven Jacobs" <jacobs@tedsoft.de>
I have a serious problem with Zope. Every time a certain page is requested by a web browser for the second or third time (first time always works) the whole Zope server crashes :-(
The only thing special about this page are calls of external Python methods from a custom extension. These methods don't do any special magic. They just open a MySQL database (using the MySQLdb module version 0.9.1) and return some HTML output.
I was able to catch a somewhat informative error message when Zope was running in debug mode:
2006-03-27T12:18:57 PANIC(300) z2 Startup exception Traceback (innermost last): File /opt/zope/z2.py, line 495, in ? (Object: ZLogger) File <string>, line 1, in ? File /opt/zope/lib/python/Zope/__init__.py, line 37, in ? File /opt/zope/lib/python/ZODB/FileStorage.py, line 202, in __init__ File /opt/zope/lib/python/ZODB/lock_file.py, line 29, in lock_file StorageSystemError: Could not lock the database file. There must be another process that has opened the file.
It sounds like your external method is causing the problem, and an easy way to test this is to remove all of the code from the external method (ie. just turn it into a test stub routine) and see if the error goes away. If this works, then add code back into the external method, one logical 'chunk' at a time and test. Continue this until you add in the chunk of code that causes the error. It can often be useful to add debug info into a python list object, as your external method progresses, and then return this list to the calling routine (which then displays the contents of the list within your normal html page). HTH Jonathan
Sven Jacobs wrote:
Dear Zope list,
I have a serious problem with Zope. Every time a certain page is requested by a web browser for the second or third time (first time always works) the whole Zope server crashes :-(
The only thing special about this page are calls of external Python methods from a custom extension. These methods don't do any special magic. They just open a MySQL database (using the MySQLdb module version 0.9.1) and return some HTML output.
I was able to catch a somewhat informative error message when Zope was running in debug mode:
2006-03-27T12:18:57 PANIC(300) z2 Startup exception Traceback (innermost last): File /opt/zope/z2.py, line 495, in ? (Object: ZLogger) File <string>, line 1, in ? File /opt/zope/lib/python/Zope/__init__.py, line 37, in ? File /opt/zope/lib/python/ZODB/FileStorage.py, line 202, in __init__ File /opt/zope/lib/python/ZODB/lock_file.py, line 29, in lock_file StorageSystemError: Could not lock the database file. There must be another process that has opened the file.
Operating system is SuSE Linux 8.0, Zope version is 2.5.0, Python 2.2, MySQL 3.23.48. I know, pretty old but upgrading any of these software components should be avoided due to emerging costs unless absolutely necessary!
So if you have any idea how to fix this problem besides "upgrade to the newest versions" I would be glad to here from you!
Thank you very much!
Use a MySQL database adapter in Zope, and use ZSQL Methods to retrieve data from the database. See here: http://www.plope.com/Books/2_7Edition/RelationalDatabases.stx HTH! -- Floyd May Senior Systems Analyst CTLN - CareerTech Learning Network fmay@okcareertech.org
Sven Jacobs wrote at 2006-4-7 10:34 +0200:
... 2006-03-27T12:18:57 PANIC(300) z2 Startup exception Traceback (innermost last): File /opt/zope/z2.py, line 495, in ? (Object: ZLogger) File <string>, line 1, in ? File /opt/zope/lib/python/Zope/__init__.py, line 37, in ? File /opt/zope/lib/python/ZODB/FileStorage.py, line 202, in __init__ File /opt/zope/lib/python/ZODB/lock_file.py, line 29, in lock_file StorageSystemError: Could not lock the database file. There must be another process that has opened the file.
This does not look like a crash. The message tells you that a start failed because another running (!) process still has the ZODB storage file locked. Almost surely, Zope is not dead (at least not completely). I have seen similar behaviour in the past: caused by a Python threading bug triggered by a Linux 2.4 signal handling peculiarity: A fatal signal kills the main Zope thread (handling all IO with the clients) but let the remaining threads in an insane state (could only be removed with "kill -9"). Current Python versions should have fixed this problem.
Operating system is SuSE Linux 8.0, Zope version is 2.5.0, Python 2.2,
Almost surely, one of the configurations with the bug mentioned above... -- Dieter
Dieter Maurer wrote:
This does not look like a crash.
The message tells you that a start failed because another running (!) process still has the ZODB storage file locked.
Almost surely, Zope is not dead (at least not completely).
I have seen similar behaviour in the past: caused by a Python threading bug triggered by a Linux 2.4 signal handling peculiarity:
A fatal signal kills the main Zope thread (handling all IO with the clients) but let the remaining threads in an insane state (could only be removed with "kill -9").
This is exactly what happens! The process still exists but it doesn't respond.
Current Python versions should have fixed this problem.
Operating system is SuSE Linux 8.0, Zope version is 2.5.0, Python 2.2,
Almost surely, one of the configurations with the bug mentioned above...
Well, for testing I tried to copy the Zope DB (Data.fs.*) into a fresh OpenSuSE 10.0 installation. Zope version there is 2.7.7. It works but now I get an authorization request dialog on every page. Any idea what could have gone wrong during the upgrade? -- Sincerely yours Sven Jacobs
Sven Jacobs wrote at 2006-4-11 09:34 +0200:
... Well, for testing I tried to copy the Zope DB (Data.fs.*) into a fresh OpenSuSE 10.0 installation. Zope version there is 2.7.7. It works but now I get an authorization request dialog on every page. Any idea what could have gone wrong during the upgrade?
Install the product "VerboseSecurity" and look (tell us) what it reports. -- Dieter
Dieter Maurer schrieb:
Well, for testing I tried to copy the Zope DB (Data.fs.*) into a fresh OpenSuSE 10.0 installation. Zope version there is 2.7.7. It works but now I get an authorization request dialog on every page. Any idea what could have gone wrong during the upgrade?
Install the product "VerboseSecurity" and look (tell us) what it reports.
Did that! But the authorization dialog did not change. There is no extra information about why an authorization is required. Or where do I find this information? PS: What's up with the Zope Administrator's Guide Project?! There has been no progress for four years! I was looking for a guide on how to upgrade a Zope version including the Zope database. Come on guys, this front really needs improvement!
On 24.04.06 08:26:02, Sven Jacobs wrote:
Dieter Maurer schrieb:
Well, for testing I tried to copy the Zope DB (Data.fs.*) into a fresh OpenSuSE 10.0 installation. Zope version there is 2.7.7. It works but now I get an authorization request dialog on every page. Any idea what could have gone wrong during the upgrade? Install the product "VerboseSecurity" and look (tell us) what it reports.
Did that! But the authorization dialog did not change. There is no extra information about why an authorization is required. Or where do I find this information?
Have a look at the error_log in your Zope root. Andreas -- Increased knowledge will help you now. Have mate's phone bugged.
Sven Jacobs wrote at 2006-4-24 08:26 +0200:
Dieter Maurer schrieb:
Well, for testing I tried to copy the Zope DB (Data.fs.*) into a fresh OpenSuSE 10.0 installation. Zope version there is 2.7.7. It works but now I get an authorization request dialog on every page. Any idea what could have gone wrong during the upgrade?
Install the product "VerboseSecurity" and look (tell us) what it reports.
Did that! But the authorization dialog did not change. There is no extra information about why an authorization is required. Or where do I find this information?
You read the installation instructions for "VerboseSecurity". It (hopefully) tells you that you have to activate Python implemented Zope Security, to reconfigure the "error_log" object (to not ignore "Unauthorized" exceptions) and then to look in the "error_log" object for information about the resulting "Unauthorized" exception. -- Dieter
You read the installation instructions for "VerboseSecurity".
It (hopefully) tells you that you have to activate Python implemented Zope Security,
I extracted the archive into the Products directory of the instance home. The product now is listed under installed products.
to reconfigure the "error_log" object (to not ignore "Unauthorized" exceptions) and then to look in the "error_log" object for information about the resulting "Unauthorized" exception.
The readme just tells me
Just remove "Unauthorized" from the list of ignored exceptions.
I have no idea where to find this list. Is it a config file or somehwere hidden within the management interface of Zope? Also there is no error_log file. Not in var nor in log.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 25 Apr 2006, at 10:16, Sven Jacobs wrote:
Just remove "Unauthorized" from the list of ignored exceptions.
I have no idea where to find this list. Is it a config file or somehwere hidden within the management interface of Zope?
Also there is no error_log file. Not in var nor in log.
error_log is an object in the root of your ZODB. It is not on the filesystem. It has the list of ignored exceptions. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFETepcRAx5nvEhZLIRAv05AJ4mztUaRt6JWonSHCWkesE0La7guACeOBwT PbuskaqDRigEy9qC3WTixoo= =6V5R -----END PGP SIGNATURE-----
The readme just tells me
Just remove "Unauthorized" from the list of ignored exceptions.
I have no idea where to find this list. Is it a config file or somehwere hidden within the management interface of Zope?
Ok, sorry, found it myself. I was looking on the file system not in the Zope DB (file structure) itself. Found the error_log object and removed "Unauthorized".
participants (7)
-
Andreas Jung -
Andreas Pakulat -
Dieter Maurer -
Floyd May -
Jens Vagelpohl -
Jonathan -
Sven Jacobs