Hi all,
A number of Enfold's customers have requested a reasonable logfile rotation scheme for Zope on Windows. Enfold would like to work on this and contribute it back to Zope. The intention of this mail is to find a consensus on the general solution we should adopt, so we can supply patches with the greatest chance of getting into the core.
Since version 2.7, Zope has moved to a logging model similar to that used by most Linux servers. This means Zope can rely on external tools for all rotation, making the Zope implementation very simple - all it need to do it re-open the logfile upon request. Unfortunately, this model relies on the operating system being able to rename a file while it is in use, which is not true for Windows.
There are a number of solutions to this problem. In the interests of brevity, I have briefly listed a few alternatives and why they suck, then my preferred solution is covered in more detail.
* Move to a Zope 2.6 model, where Zope itself manages all rotation logic. This has the obvious drawback of requiring all supporting code exist in Zope just for Windows, and all the reasons Zope dropped it in the first place.
* Use a default log name based on the date and rotation schedule. For example, assuming a rotation schedule of "weekly", the logfile name would include the date of the start of the current week. This is similar to the scheme used by Microsoft IIS. Drawbacks include the need to wake at midnight to check if the name of the logfile has changed, that the scheme leaves lots of old files around, and that logging on Windows is significantly different than for other operating systems.
* Do nothing. The drawback of this is that it leaves Zope on Windows a second-class citizen, and with administrators unable to control their logs.
* Point users at the win32 "event log" logger. This is not pratical when the number of events being logged is high, so is probably only suitable as a special handler for only ERROR events, for example.
My preferred model is the following alternative:
* Have the Zope logfile handler make a "snapshot" of the log, and like Zope on Linux, rely on external tools for all logfile rotation semantics.
When making a "snapshot", Zope simply closes the existing logfile, renames it to a well-known name (eg, "{logname}.snapshot", and then re-opens the original logfile name (which should no longer exist as it was just renamed). If the rename fails (ie, the .snapshot file already exists), Zope simply ignores the error and continues to append to the existing log-file.
This now means that "{logname}.snapshot" can be rotated by an external tool. For demonstration purposes, the following script should offer a simple rotation tool for Windows:
% kill --signal=USR2 zope_pid # whatever the Windows equiv. is % sleep 5 # wait for the file to appear % mv -f event.log.1 event.log.2 % mv -f event.log.snapshot event.log.1
You will note that this is quite similar to a script that would work on Linux. The key differences are that on Windows we send the signal *before* the process, and that we make no attempt to touch the file currently being logged to, only the snapshot just made. Existing tools (eg logrotate) should be capable of rotating Zope (unfortunately logrotate doesn't seem to have a reasonable windows port available.)
Note that for simplicity, the above assumes that the signal module will work for us on Windows. This is probably not true, but it should be possible to implement a win32 specific simulation. I've avoided this so we can focus on the log semantics.
I welcome all comments on these schemes, or any others I have not listed.
Thanks,
Mark.
On Jan 24, 2005, at 8:12 PM, Mark Hammond wrote:
A number of Enfold's customers have requested a reasonable logfile rotation scheme for Zope on Windows. Enfold would like to work on this and contribute it back to Zope. The intention of this mail is to find a consensus on the general solution we should adopt, so we can supply patches with the greatest chance of getting into the core.
Since ZConfig and zLOG are both designed to be extensible. They are built so that new loggers can be created without any changes to the Zope core. It seems to me that most of this could be handled by an add-on package. The ZConfig documentation http://svn.zope.org/ZConfig/trunk/doc/zconfig.pdf?rev=440&view=log has as its extension example how to add a new logger (a "log to a pager" logger) to the system. (Of course once this is built, this add-on package could migrate to the core if there is a compelling need, either packaging convenience (Windows users don't need to grab an extra package to manage their systems well.) or so that enhancements to zLOG's loghandlers can keep the win32 loggers in mind.)
As a rough cut, this will create a logger with a logrotate behavior similar to what you are looking for.
win32logger/__init__.py: # empty file to signify a package.
win32logger/component.xml: <component prefix="win32logger.LogHandlers"> <!-- extend the logging subsystem with a new file logger --> <import package="zLOG"/>
<sectiontype name="win32-logfile" datatype=".Win32FileHandlerFactory" implements="zLOG.loghandler" extends="logfile"> <key name="rotate-path" required="yes"/> </sectiontype> </component>
win32logger/LogHandlers.py: import zLOG import os
class Win32FileHandler(zLOG.LogHandlers.FileHandler): """" A file based loghandler that renames on rotate """ def __init__(self, path, rotate_path): zLOG.LogHandlers.FileHandler.__init__(self,path) self.rotateFilename = rotate_path
def reopen(self): self.close() error = None try: os.rename(self.baseFilename, self.rotateFilename) except OSError, err: error = err self.stream = open(self.baseFilename, self.mode) if error: zLOG.LOG("Win32Logger", zLOG.ERROR, "Rotate Error", error)
class Win32FileHandlerFactory(zLOG.datatypes.FileHandlerFactory): def create_loghandler(self): return Win32FileHandler(self.section.path,self.section.rotate_path)
and then adjust the zope.conf with:
%import win32logger
<eventlog> level all <win32-logfile> level DEBUG path $INSTANCE/log/-event.log rotate-path $INSTANCE/log/event.log.closed </win32-logfile> </eventlog>
<logger access> level WARN <win32-logfile> path $INSTANCE/log/Z2.log rotate-path $INSTANCE/log/Z2.log.closed </win32-logfile>
</logger>
Of course, it needs to be fleshed out with better error handling, etc.
Although time consuming, maybe the better behavior would be to have the reopen method copy the current log to the backup file and then seek() and truncate() the original file. That would take longer to rotate, but would protect against log files whose close() succeed but open() fails.
-
Andrew Langmead -
Mark Hammond