Can logreopen restarts Zope ?
Hi there, I observed a strange behavior yesterday night on one of our ZEO cluster. We got 4 ZEO servers on the cluster. We are rotating the ZEO logs once every night using logreopen. Until now, we didn't observe any misbehavior. Meaning, logs were rotated and then reopened ok on the four of them every night since couple of weeks now. Yesterday night, we were having a long nightly running process (several hours) on one of the ZEO node. (Around 500 Mo logs on this one). The reopen occurred while the Zope server was still working. The reopen went ok but it restarted the Zope server at the same time which of course stopped the actual running operation of ours. Did someone experienced this ? Any feedbacks appreciated. Thanks. J. P.S : Python-2.4.2, Zope-2.9.0 -- Julien Anguenot | Nuxeo R&D (Paris, France) Open Source ECM - www.nuxeo.com CPS Platform - http://www.cps-project.org Mobile: +33 (0) 6 72 57 57 66
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30 Jun 2006, at 13:40, Julien Anguenot wrote:
Yesterday night, we were having a long nightly running process (several hours) on one of the ZEO node. (Around 500 Mo logs on this one). The reopen occurred while the Zope server was still working. The reopen went ok but it restarted the Zope server at the same time which of course stopped the actual running operation of ours.
Did someone experienced this ?
Any feedbacks appreciated.
For maximum reliability I tend to forego the built-in log reopening in favor of using the logrotate utility in "copytruncate" mode. This will not require any log file reopening, logrotate will simply make a copy of the existing log and then truncate the logfile that continues to be in use to zero length. Zope or the ZEO server won't have to know or care at all at that point. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEpRWdRAx5nvEhZLIRAqZ+AKCVGGWI8suqWyxPw707WMIlFiScEACeO0Va KO7qZIC0AaBV+T5EudaNNxY= =fGMe -----END PGP SIGNATURE-----
Jens Vagelpohl wrote:
On 30 Jun 2006, at 13:40, Julien Anguenot wrote:
Yesterday night, we were having a long nightly running process (several hours) on one of the ZEO node. (Around 500 Mo logs on this one). The reopen occurred while the Zope server was still working. The reopen went ok but it restarted the Zope server at the same time which of course stopped the actual running operation of ours.
Did someone experienced this ?
Any feedbacks appreciated.
For maximum reliability I tend to forego the built-in log reopening in favor of using the logrotate utility in "copytruncate" mode. This will not require any log file reopening, logrotate will simply make a copy of the existing log and then truncate the logfile that continues to be in use to zero length. Zope or the ZEO server won't have to know or care at all at that point.
Thank you Jens for the quick reply. I think I'm going to send these instructions, to be on the safe side, to the administrators taking care of the cluster. Do you have any idea about the "zopectl logreopen" misbehavior as someone who's avoiding using it explicitly ;) ? Cheers, J. -- Julien Anguenot | Nuxeo R&D (Paris, France) Open Source ECM - www.nuxeo.com CPS Platform - http://www.cps-project.org Mobile: +33 (0) 6 72 57 57 66
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30 Jun 2006, at 14:30, Julien Anguenot wrote:
For maximum reliability I tend to forego the built-in log reopening in favor of using the logrotate utility in "copytruncate" mode. This will not require any log file reopening, logrotate will simply make a copy of the existing log and then truncate the logfile that continues to be in use to zero length. Zope or the ZEO server won't have to know or care at all at that point.
Thank you Jens for the quick reply.
I think I'm going to send these instructions, to be on the safe side, to the administrators taking care of the cluster.
Do you have any idea about the "zopectl logreopen" misbehavior as someone who's avoiding using it explicitly ;) ?
I'm afraid I don't. I have seen issues myself where the server would just die and not come up again, but never investigated it. The logreopen solution offers a safe and lasting fix and it even simplifies log rotation configuration. Here's a very simple one that rotates ZEO server and client logs for a simple one-client ZEO setup, we use it for www.zope.de: [jens@barney ~]$ cat /etc/logrotate.d/zope.de /home/zope/zope.de/log/*.log { weekly rotate 10 missingok compress copytruncate } jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEpRxuRAx5nvEhZLIRAu2QAKCMo1tttOQGULCRcWlYcVRn4psWrwCfXoAk 1ycxYnJQNbCz/Ab+KYJUh+A= =rYwK -----END PGP SIGNATURE-----
Jens Vagelpohl wrote:
On 30 Jun 2006, at 14:30, Julien Anguenot wrote:
For maximum reliability I tend to forego the built-in log reopening in favor of using the logrotate utility in "copytruncate" mode. This will not require any log file reopening, logrotate will simply make a copy of the existing log and then truncate the logfile that continues to be in use to zero length. Zope or the ZEO server won't have to know or care at all at that point.
Thank you Jens for the quick reply.
I think I'm going to send these instructions, to be on the safe side, to the administrators taking care of the cluster.
Do you have any idea about the "zopectl logreopen" misbehavior as someone who's avoiding using it explicitly ;) ?
I'm afraid I don't. I have seen issues myself where the server would just die and not come up again, but never investigated it. The logreopen solution offers a safe and lasting fix and it even simplifies log rotation configuration.
Here's a very simple one that rotates ZEO server and client logs for a simple one-client ZEO setup, we use it for www.zope.de:
[jens@barney ~]$ cat /etc/logrotate.d/zope.de /home/zope/zope.de/log/*.log { weekly rotate 10 missingok compress copytruncate }
Alright. Thank you for the configuration script. J. -- Julien Anguenot | Nuxeo R&D (Paris, France) Open Source ECM - www.nuxeo.com CPS Platform - http://www.cps-project.org Mobile: +33 (0) 6 72 57 57 66
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Julien Anguenot wrote:
Jens Vagelpohl wrote:
On 30 Jun 2006, at 13:40, Julien Anguenot wrote:
Yesterday night, we were having a long nightly running process (several hours) on one of the ZEO node. (Around 500 Mo logs on this one). The reopen occurred while the Zope server was still working. The reopen went ok but it restarted the Zope server at the same time which of course stopped the actual running operation of ours.
Did someone experienced this ?
Any feedbacks appreciated. For maximum reliability I tend to forego the built-in log reopening in favor of using the logrotate utility in "copytruncate" mode. This will not require any log file reopening, logrotate will simply make a copy of the existing log and then truncate the logfile that continues to be in use to zero length. Zope or the ZEO server won't have to know or care at all at that point.
Thank you Jens for the quick reply.
I think I'm going to send these instructions, to be on the safe side, to the administrators taking care of the cluster.
Do you have any idea about the "zopectl logreopen" misbehavior as someone who's avoiding using it explicitly ;) ?
I think the shift to the oh-so-much-better Python logging module is not complete. See this bit in ZEO/runzeo.py: def handle_sigusr2(self): # TODO: this used to reinitialize zLOG. How do I achieve # the same effect with Python's logging package? # Should we restart as with SIGHUP? log("received SIGUSR2, but it was not handled!", \ level=logging.WARNING) svn blames Tim for the comment, and Gintautas for the actual logging call change. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEpURZ+gerLs4ltQ4RArORAJ46/o7lrmH99qDR3O51qWyu7wCdXgCeO3Lc KajVH3VK8oGsawX0xA7aoPg= =gVUG -----END PGP SIGNATURE-----
On 6/30/06, Tres Seaver <tseaver@palladion.com> wrote:
I think the shift to the oh-so-much-better Python logging module is not complete. See this bit in ZEO/runzeo.py:
Aha. Indeed, we are running ZEO, and my failed attempts of reproducing this did not use ZEO. :) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
participants (4)
-
Jens Vagelpohl -
Julien Anguenot -
Lennart Regebro -
Tres Seaver