RE: [Zope] Squid vs Apache+mod_proxy+mod_gzip
Another logging option for Squid: we log with log_mime_headers on, which dumps a rather verbose log, but we filter this to combined format for later use and archival. Only catch is that Squid does not at current support large-file support for logs, and does die if log file exceeds 2gb, so you have to rotate frequently. Sean
-----Original Message----- From: Toby Dickenson [mailto:tdickenson@geminidataloggers.com] Sent: Monday, June 09, 2003 8:34 AM To: george donnelly; zope@zope.org Subject: Re: [Zope] Squid vs Apache+mod_proxy+mod_gzip
On Saturday 07 June 2003 19:30, george donnelly wrote:
My question is, in your experience, how does Squid match up against Apache 1.3 + mod_proxy + mod_gzip in the areas of speed and robustness?
I am happy with squid so far. Ive never felt its speed or robustness lacking.
Its SSL handling is a little less mature.... I have had some problems in the past, but havent been able determine whether the blame lies with openssl or squid.
No problems big enough to drive me back to apache though.
also, with squid, how do people get useful logs for analysis by awstats etc out of it? based on what I've looked at, the logs are not as detailed as those given by apache.
The squid native log has extra fields relating to proxying that apache logs do not (for example, which back-end zope server handled the request, and why that one was chosen). I am using Analog which parses this log nicely, ignoring the fields that it doesnt know about.
The squid native log is missing referrer and user agent information, but those fields are stored in seperate log files.
Is there anything else you thought was missing?
-- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Mon, Jun 09, 2003 at 08:55:56AM -0700, sean.upton@uniontrib.com wrote:
Another logging option for Squid: we log with log_mime_headers on, which dumps a rather verbose log, but we filter this to combined format for later use and archival.
Interesting idea. I just tried this briefly and I see how it would do the job. In my test case it logged about 8 times more data per GET than without 'log_mime_headers'. As someone else asked, could you share your script for converting that data into combined log format as expected by log analyzers? It looks like a simple Perl script would do the job, extracting referrer and user-agent data when available from the corresponding headers. Are there any tricks to that? -- Fred Yankowski fred@ontosys.com tel: +1.630.879.1312 OntoSys, Inc PGP keyID: 7B449345 fax: +1.630.879.1370 www.ontosys.com 38W242 Deerpath Rd, Batavia, IL 60510-9461, USA
Fred wrote:
As someone else asked, could you share your script for converting that data into combined log format as expected by log analyzers? It looks like a simple Perl script would do the job, extracting referrer and user-agent data when available from the corresponding headers. Are there any tricks to that? No tricks, because each line in access/referer/agent file corresponds to the same hit. You can just use concatenate the fields together.
I noticed I had a newer version of the perl-script I sent to awstats contrib. It's available at http://gewis.nl/~pieterb/zope/access_referer_agent2_combined.v0.96.pl.txt I used it for Lotus Notes, but I guess it can easily be used for Squid as well... Pieter
On Tue, Jun 10, 2003 at 12:33:28AM +0200, PieterB wrote:
No tricks, because each line in access/referer/agent file corresponds to the same hit. You can just use concatenate the fields together.
Indeed, looks trivial. I've set things up to use the simple perl script mentioned earlier today to convert the 'log_mime_hdrs' log entries to common format. I think Squid also needs to be configured with 'emulate_httpd_log on' for that particular script to work.
I noticed I had a newer version of the perl-script I sent to awstats contrib. It's available at http://gewis.nl/~pieterb/zope/access_referer_agent2_combined.v0.96.pl.txt
I used it for Lotus Notes, but I guess it can easily be used for Squid as well...
That script seems to assume that every entry in access.log has a corresponding entry in referer.log and useragent.log, so that merging can be done simply line by line. In my experience, squid only adds a line to referer.log when there is a Referer header in the HTTP request, so many access.log lines have no corresponding referer.log line. -- Fred Yankowski fred@ontosys.com tel: +1.630.879.1312 OntoSys, Inc PGP keyID: 7B449345 fax: +1.630.879.1370 www.ontosys.com 38W242 Deerpath Rd, Batavia, IL 60510-9461, USA
On Monday 09 June 2003 16:55, sean.upton@uniontrib.com wrote:
Another logging option for Squid: we log with log_mime_headers on, which dumps a rather verbose log, but we filter this to combined format for later use and archival.
Is the tool to convert into combined format publicly available? -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson
Another logging option for Squid: we log with log_mime_headers on, which dumps a rather verbose log, but we filter this to combined format for later use and archival. Is the tool to convert into combined format publicly available? Yep, its access_referer_agent2combined.pl and it's available at http://awstats.sourceforge.net/awstats_contrib.html#CONTRIB It doesn't do any comparing of the dates, but just merges the fields of the referer and agent files into the combined file.
I found it easier to write the script in Perl than in Python. (I shouldn't say that outloud I guess ;) I wrote it for Awstats which is a great loganalyzer (although written in Perl) Pieter
participants (4)
-
Fred Yankowski -
PieterB -
sean.upton＠uniontrib.com -
Toby Dickenson