[Zope] Log aggregator?

Paul Winkler pw_lists at slinkp.com
Fri Sep 23 17:27:03 EDT 2005


Anybody know of any software that will aggregate logs in a variety
of formats and allow me to correlate events in one log with another?

I'm getting really REALLY tired of eyeballing and grepping a multitude
of logs on different servers, mentally translating incorrect timestamps,
and trying to work out what the heck from one log corresponds to something
in another.  Anytime something goes wrong, i waste a huge amount of time
on this.

Well, duh, this should be automated, right?

I googled a bit and came up with nothing useful;
then I started hacking up a python script, which I'm sure I could
do in an hour or three, but - blah, it would be a huge timesaver if
somebody had written something like this already.

I need to handle:

* Different log formats.  I'm mainly interested in the zope access log
format (aka apache format) and the zope event log format.
Must tolerate malformed lines like the #!@)%* ftp lines in the zope access
log.

* System clocks being incorrect.
On some of the servers I work with, I am not the primary admin and can't do
much about the %@W#% clock being wrong. (*cough* firewall blocking
ntp for no good reason *cough*)
So I'd like the software to let me specify how far wrong
the various logs are, and compensate.

* Servers in different time zones.  Rare for me, but it does come up.

* Filtering based on arbitrary strings, or (better) regular expressions,
on a per-log basis.

* Multi-megabyte logs. Sometimes i go to check a log and sysadmin has
forgotten to rotate ever...

-PW





More information about the Zope mailing list