Zope and Log File Analysis
Hello, So I've been struggling for years now to try and provide thorough log analysis for my Zope based web site. The big problem I have is determining "file types" without having "file extensions", since many objects are name with a simple id that doesn't contain a typical file extension. This makes it supremely difficult to determine viewing statistics based on content type for example ... Or determining what should be concidered a "page view", and so on. The best solution I can think of is to actually come up with standard file extensions for the various content types I use ... It's slightly annoying because of the "dot" in the ID, but otherwise, it should work ok. Has anybody confronted this issue before? Any other alternatives I haven't concidered? Any insight would be much appreciated! Thanks, Jean-François Doyon Internet Service Development and Systems Support / Spécialiste de dèveloppements internet et soutien technique Canada Centre for Remote Sensing/Centre Canadien de télédétection Natural Resources Canada/Ressources Naturelles Canada http://atlas.gc.ca Tel./Tél.: (613) 992-4902 Fax: (613) 947-2410
Am Mittwoch, 18. Oktober 2006 22:03 schrieb Doyon, Jean-Francois:
The big problem I have is determining "file types" without having "file extensions", since many objects are name with a simple id that doesn't contain a typical file extension.
This makes it supremely difficult to determine viewing statistics based on content type for example ... Or determining what should be concidered a "page view", and so on.
one not so elegant solution to log real page views only: create your own logger, and call it somewhere in you master template (probably close to the end of it so logging doesn't happen if there was an exception before that point). Cheers, Sascha -- Lalisio GmbH www.lalisio.com Puschkinstraße 1 fon +49-(0)361/541 43 80 99084 Erfurt fax +49-(0)361/541 43 79 kontakt@lalisio.com +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ AKTUELLER HINWEIS (Oktober 2006) Wussten Sie schon, dass Lalisio ab sofort den Dokumentenlieferdienst CISTI in seine Literatursuchmaschine einbindet? Sobald Sie über unsere Literatursuchmaschine in den Datenbanken Ingenta oder Projekt MUSE relevante Artikel finden, können Sie die bibliographischen Daten per Mausklick an CISTI übermitteln und Kopien der Artikel bestellen. Testen Sie unser Angebot unter www.lalisio.com! +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--On 19. Oktober 2006 12:34:28 +0200 Sascha Ottolski <sascha.ottolski@lalisio.com> wrote:
Am Mittwoch, 18. Oktober 2006 22:03 schrieb Doyon, Jean-Francois:
The big problem I have is determining "file types" without having "file extensions", since many objects are name with a simple id that doesn't contain a typical file extension.
This makes it supremely difficult to determine viewing statistics based on content type for example ... Or determining what should be concidered a "page view", and so on.
one not so elegant solution to log real page views only: create your own logger, and call it somewhere in you master template (probably close to the end of it so logging doesn't happen if there was an exception before that point).
The elegant solution is to run Apache in front of Zope as reverse proxy and using Apaches CustomLog/Logformat option. The %...o option allows you to log any HTTP header. -aj
Hmmmm, but said headers wouldn't be sufficient, unless I set custom ones for each (CMF, in my case) content type. Otherwise it'd mostly be text/html, which wouldn't be much help. I already use the whole Apache/ReverseProxy setup, so when I talk about this, I do mean Apache log files BTW. I'll look at that idea though, I'd have to see if there are tools that provide analysis of http headers somehow. Thanks, J.F. -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Andreas Jung Sent: October 19, 2006 6:38 AM To: Sascha Ottolski; zope@zope.org Subject: Re: [Zope] Zope and Log File Analysis --On 19. Oktober 2006 12:34:28 +0200 Sascha Ottolski <sascha.ottolski@lalisio.com> wrote:
Am Mittwoch, 18. Oktober 2006 22:03 schrieb Doyon, Jean-Francois:
The big problem I have is determining "file types" without having "file extensions", since many objects are name with a simple id that doesn't contain a typical file extension.
This makes it supremely difficult to determine viewing statistics based on content type for example ... Or determining what should be concidered a "page view", and so on.
one not so elegant solution to log real page views only: create your own logger, and call it somewhere in you master template (probably close to the end of it so logging doesn't happen if there was an exception before that point).
The elegant solution is to run Apache in front of Zope as reverse proxy and using Apaches CustomLog/Logformat option. The %...o option allows you to log any HTTP header. -aj
--On 19. Oktober 2006 08:07:37 -0400 "Doyon, Jean-Francois" <jdoyon@NRCan.gc.ca> wrote:
Hmmmm, but said headers wouldn't be sufficient, unless I set custom ones for each (CMF, in my case) content type.
Why wouldn't that help? A logged header "image/gif" or "image/jpeg" is clearly an indicator for an image? Isn't it? -aj
Indeed, but it wouldn't help if I wanted to tell a "Document" from a "Map Text" from a "Topic Text" for example in my case. Those are various content types that would all show up as text/html. -----Original Message----- From: Andreas Jung [mailto:lists@zopyx.com] Sent: October 19, 2006 8:15 AM To: Doyon, Jean-Francois; zope@zope.org Subject: RE: [Zope] Zope and Log File Analysis --On 19. Oktober 2006 08:07:37 -0400 "Doyon, Jean-Francois" <jdoyon@NRCan.gc.ca> wrote:
Hmmmm, but said headers wouldn't be sufficient, unless I set custom ones for each (CMF, in my case) content type.
Why wouldn't that help? A logged header "image/gif" or "image/jpeg" is clearly an indicator for an image? Isn't it? -aj
Ah yes, indeed, I could do custom logging ... But then I'd have to worry about the performance impact, the fact that because these logs would be custom, I'd need custom analysis tools as well (unless I can make it compatible somehow?) ... And so on ... We'd much rather stick with more standard solutions! Thanks for the idea though ... J.F. -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Sascha Ottolski Sent: October 19, 2006 6:34 AM To: zope@zope.org Subject: Re: [Zope] Zope and Log File Analysis Am Mittwoch, 18. Oktober 2006 22:03 schrieb Doyon, Jean-Francois:
The big problem I have is determining "file types" without having "file extensions", since many objects are name with a simple id that doesn't contain a typical file extension.
This makes it supremely difficult to determine viewing statistics based on content type for example ... Or determining what should be concidered a "page view", and so on.
one not so elegant solution to log real page views only: create your own logger, and call it somewhere in you master template (probably close to the end of it so logging doesn't happen if there was an exception before that point). Cheers, Sascha -- Lalisio GmbH www.lalisio.com Puschkinstraße 1 fon +49-(0)361/541 43 80 99084 Erfurt fax +49-(0)361/541 43 79 kontakt@lalisio.com +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ AKTUELLER HINWEIS (Oktober 2006) Wussten Sie schon, dass Lalisio ab sofort den Dokumentenlieferdienst CISTI in seine Literatursuchmaschine einbindet? Sobald Sie über unsere Literatursuchmaschine in den Datenbanken Ingenta oder Projekt MUSE relevante Artikel finden, können Sie die bibliographischen Daten per Mausklick an CISTI übermitteln und Kopien der Artikel bestellen. Testen Sie unser Angebot unter www.lalisio.com! +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Hi, Doyon, Jean-Francois wrote:
Ah yes, indeed, I could do custom logging ... But then I'd have to worry about the performance impact, the fact that because these logs would be custom, I'd need custom analysis tools as well (unless I can make it compatible somehow?) ... And so on ...
We'd much rather stick with more standard solutions!
Have you seen: - http://www.zope.org/Members/sbrauer/Webalizer-CT/README -- Regards, PhilK "Say what you mean. Bear witness. Iterate." —John M. Ford
participants (4)
-
Andreas Jung -
Doyon, Jean-Francois -
Philip Kilner -
Sascha Ottolski