Anybody know of any software that will aggregate logs in a variety of formats and allow me to correlate events in one log with another? I'm getting really REALLY tired of eyeballing and grepping a multitude of logs on different servers, mentally translating incorrect timestamps, and trying to work out what the heck from one log corresponds to something in another. Anytime something goes wrong, i waste a huge amount of time on this. Well, duh, this should be automated, right? I googled a bit and came up with nothing useful; then I started hacking up a python script, which I'm sure I could do in an hour or three, but - blah, it would be a huge timesaver if somebody had written something like this already. I need to handle: * Different log formats. I'm mainly interested in the zope access log format (aka apache format) and the zope event log format. Must tolerate malformed lines like the #!@)%* ftp lines in the zope access log. * System clocks being incorrect. On some of the servers I work with, I am not the primary admin and can't do much about the %@W#% clock being wrong. (*cough* firewall blocking ntp for no good reason *cough*) So I'd like the software to let me specify how far wrong the various logs are, and compensate. * Servers in different time zones. Rare for me, but it does come up. * Filtering based on arbitrary strings, or (better) regular expressions, on a per-log basis. * Multi-megabyte logs. Sometimes i go to check a log and sysadmin has forgotten to rotate ever... -PW
When you get a solution let me know. I am in the middle of log hell at the moment (including Zopes logs) trying to get things to sync up properly across multiple machines. When my stuff works, I'll share. -d On Fri, 23 Sep 2005, Paul Winkler wrote:
Anybody know of any software that will aggregate logs in a variety of formats and allow me to correlate events in one log with another?
I'm getting really REALLY tired of eyeballing and grepping a multitude of logs on different servers, mentally translating incorrect timestamps, and trying to work out what the heck from one log corresponds to something in another. Anytime something goes wrong, i waste a huge amount of time on this.
Well, duh, this should be automated, right?
I googled a bit and came up with nothing useful; then I started hacking up a python script, which I'm sure I could do in an hour or three, but - blah, it would be a huge timesaver if somebody had written something like this already.
I need to handle:
* Different log formats. I'm mainly interested in the zope access log format (aka apache format) and the zope event log format. Must tolerate malformed lines like the #!@)%* ftp lines in the zope access log.
* System clocks being incorrect. On some of the servers I work with, I am not the primary admin and can't do much about the %@W#% clock being wrong. (*cough* firewall blocking ntp for no good reason *cough*) So I'd like the software to let me specify how far wrong the various logs are, and compensate.
* Servers in different time zones. Rare for me, but it does come up.
* Filtering based on arbitrary strings, or (better) regular expressions, on a per-log basis.
* Multi-megabyte logs. Sometimes i go to check a log and sysadmin has forgotten to rotate ever...
-PW
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
--
Paul Winkler wrote at 2005-9-23 16:27 -0500:
Anybody know of any software that will aggregate logs in a variety of formats and allow me to correlate events in one log with another?
I'm getting really REALLY tired of eyeballing and grepping a multitude of logs on different servers, mentally translating incorrect timestamps, and trying to work out what the heck from one log corresponds to something in another. Anytime something goes wrong, i waste a huge amount of time on this.
Maybe, you log via "syslog" and send all syslog messages to a central server (then you get a single logfile (at least when you want) with all your log entries). -- Dieter
Paul Winkler wrote at 2005-9-23 16:27 -0500:
Anybody know of any software that will aggregate logs in a variety of formats and allow me to correlate events in one log with another?
I'm getting really REALLY tired of eyeballing and grepping a multitude of logs on different servers, mentally translating incorrect timestamps, and trying to work out what the heck from one log corresponds to something in another. Anytime something goes wrong, i waste a huge amount of time on this.
Maybe, you log via "syslog" and send all syslog messages to a central server (then you get a single logfile (at least when you want) with all your log entries).
That's a fine idea for the future, but I'm looking more for something to help with already existing logs where this isn't the case. thanks, PW
participants (3)
-
Dennis Allison -
Dieter Maurer -
Paul Winkler