Hello, I was just wondering if Zope logs who visits what on a site ??? Usually everything that happens on your webpages (when you e.g. use apache) ends up in you webservers access_log. But does ZServer logs the same ? If so - has anyone created a product to get this data out of the log files and display them on a webpage ?? Before I started on Zope, I was using something called webalizer. Or - is it hard to "analyze" a log file and display the data from it ??? Regards, Gitte Wange
Hello Gitte, I am using webalizer to access my log files for zope, the Z2.log file, I have a cronjob that runs every evening 55 0 * * * /usr/local/bin/virtual /usr/local/bin/webalizer -n myzope -o /www/htdocs/zope/stats/ /usr/local/Zope/var/Z2.log this gives me some of the info I need. Although, being in the publishing industry, we require additional information, from our registered site users. I would like to be able to analyse for each user where they went, which page and be able to break this by their demographic data. Would any one has good pointers on where I can find info for analysing ZODB data -- for example: how would you request zodb to tell you how many records within a certain geo area there are? I did post a question on the ZODB mail list http://lists.zope.org/pipermail/zodb-dev/2001-June/001034.html and got a reply, that suggested that I needed to look OO databases differently -- as it seems I understand better from examples, could somebody be kind enough to perhaps send me couple of examples that analyse the data from the ZODB Thanks Norman -----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Gitte Wange Sent: 08 July 2001 13:49 To: zope@zope.org Subject: [Zope] Site statistics Hello, I was just wondering if Zope logs who visits what on a site ??? Usually everything that happens on your webpages (when you e.g. use apache) ends up in you webservers access_log. But does ZServer logs the same ? If so - has anyone created a product to get this data out of the log files and display them on a webpage ?? Before I started on Zope, I was using something called webalizer. Or - is it hard to "analyze" a log file and display the data from it ??? Regards, Gitte Wange _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Hi! On Sun, 8 Jul 2001, Norman Khine wrote:
55 0 * * * /usr/local/bin/virtual /usr/local/bin/webalizer -n myzope -o /www/htdocs/zope/stats/ /usr/local/Zope/var/Z2.log
What is /usr/local/bin/virtual? Oleg. ---- Oleg Broytmann http://www.zope.org/Members/phd/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
Hi Norman, --On Sonntag, 8. Juli 2001 17:33 +0100 Norman Khine <khine@btinternet.com> wrote:
Hello Gitte, I am using webalizer to access my log files for zope, the Z2.log file, I have a cronjob that runs every evening
this should be efficient.
Although, being in the publishing industry, we require additional information, from our registered site users. I would like to be able to analyse for each user where they went, which page and be able to break this by their demographic data.
Hm. May be you mix up publishing industry with intelligence service? ;) The only solution to get all the data you want is to bother the user untill she bleeds. But this isn't probably what you want here. All the magic things analyzers do to get "demographical data" is to lie. Probably their creators and for sure their users dont know, but they simple lie :-) A web request may be bound to an ip-address. Since none of the users carry their own personal IP-address with them around, you get the adresses of some silly grey boxes. You will probably never know where this box really sits. Even more not what user in front of it. (See also the meaning of proxies and IP pools) You might only get the address of the netblock owner, but this is seldom the one who uses your site :) Sure, you can get some imformation with setting cookies and/or using sessions. But this means bothering the user and you dont get all of your users if they start to avoid your pages because they dont want cookies, personal data or simply see a slow server because of all the statistic stuff it does.
Would any one has good pointers on where I can find info for analysing ZODB data -- for example: how would you request zodb to tell you how many records within a certain geo area there are? I did post a question on the ZODB mail list http://lists.zope.org/pipermail/zodb-dev/2001-June/001034.html
Only the ZPublisher sees the IP address. It has absolutely nothing to do with the ZODB here.
and got a reply, that suggested that I needed to look OO databases differently -- as it seems I understand better from examples, could somebody be kind enough to perhaps send me couple of examples that analyse the data from the ZODB
Thanks
Norman
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Gitte Wange Sent: 08 July 2001 13:49 To: zope@zope.org Subject: [Zope] Site statistics
Hello,
I was just wondering if Zope logs who visits what on a site ???
Usually everything that happens on your webpages (when you e.g. use apache) ends up in you webservers access_log. But does ZServer logs the same ?
If so - has anyone created a product to get this data out of the log files and display them on a webpage ?? Before I started on Zope, I was using something called webalizer.
Or - is it hard to "analyze" a log file and display the data from it ???
Regards, Gitte Wange
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Hello Tino, I am sure the Intelligence Services have a more advanced way of tracking users, or perhaps not, they have laws and credit cards ;^( To be a bit more clearer, I really wanted to know how to manipulate the data from the zodb to produce reports as specified, mainly because I am building a reader circulation application, which is currently running on a SQL Server, and I want to transfer this to zope. I can have a mix between the two, but from pure intellectual use, I wanted to find out what and how this works. Also note that all our members/prospective members have the option not to give out their details, as you say on the net you could hide your details as you wish, nobody really cares unless they want something from you. Cheers Norman -----Original Message----- From: Tino Wildenhain [mailto:tino@wildenhain.de] Sent: 08 July 2001 20:18 To: Norman Khine; Gitte Wange Cc: Zope Subject: RE: [Zope] Site statistics Hi Norman, --On Sonntag, 8. Juli 2001 17:33 +0100 Norman Khine <khine@btinternet.com> wrote:
Hello Gitte, I am using webalizer to access my log files for zope, the Z2.log file, I have a cronjob that runs every evening
this should be efficient.
Although, being in the publishing industry, we require additional information, from our registered site users. I would like to be able to analyse for each user where they went, which page and be able to break this by their demographic data.
Hm. May be you mix up publishing industry with intelligence service? ;) The only solution to get all the data you want is to bother the user untill she bleeds. But this isn't probably what you want here. All the magic things analyzers do to get "demographical data" is to lie. Probably their creators and for sure their users dont know, but they simple lie :-) A web request may be bound to an ip-address. Since none of the users carry their own personal IP-address with them around, you get the adresses of some silly grey boxes. You will probably never know where this box really sits. Even more not what user in front of it. (See also the meaning of proxies and IP pools) You might only get the address of the netblock owner, but this is seldom the one who uses your site :) Sure, you can get some imformation with setting cookies and/or using sessions. But this means bothering the user and you dont get all of your users if they start to avoid your pages because they dont want cookies, personal data or simply see a slow server because of all the statistic stuff it does.
Would any one has good pointers on where I can find info for analysing ZODB data -- for example: how would you request zodb to tell you how many records within a certain geo area there are? I did post a question on the ZODB mail list http://lists.zope.org/pipermail/zodb-dev/2001-June/001034.html
Only the ZPublisher sees the IP address. It has absolutely nothing to do with the ZODB here.
and got a reply, that suggested that I needed to look OO databases differently -- as it seems I understand better from examples, could somebody be kind enough to perhaps send me couple of examples that analyse the data from the ZODB
Thanks
Norman
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Gitte Wange Sent: 08 July 2001 13:49 To: zope@zope.org Subject: [Zope] Site statistics
Hello,
I was just wondering if Zope logs who visits what on a site ???
Usually everything that happens on your webpages (when you e.g. use apache) ends up in you webservers access_log. But does ZServer logs the same ?
If so - has anyone created a product to get this data out of the log files and display them on a webpage ?? Before I started on Zope, I was using something called webalizer.
Or - is it hard to "analyze" a log file and display the data from it ???
Regards, Gitte Wange
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
I see that someone has already pointed out the Z2.log, but in case anyone is interested in more complex solutions, here's one we're pursuing. We're going to use multiple Apache (HTTP/HTTPS) servers as the front ends to a line of ZEO clients. Using Z2.log with multiple machines is very clumsy and I want to be able to make powerful realtime queries, so we are going to use PostgreSQL (or Oracle, if forced - ZODB is *not* appropriate for this). http://techdocs.postgresql.org/techdocs/apachepglogd.php This will collect all of the hits to all of the HTTP(S) servers and will keep track of things I did not see in Z2.log such as time spent to answer the request. ("Who gets to pay for the next ZEO client?") If I want a report, I'll whip up something using psycopg and ZGDChart. I'll also let my users do this (as we do with Analog on Apache now) instead of resorting to the tacky counter images. I won't have disks on the ZEO clients (nor on the Apache servers), so I will probably eliminate Z2.log. --kyler
If you're running Apache in front of Zope (with mod_rewrite/mod_proxy) then you can still use Apache's logfiles. I'm currently doing this and it works fine. But I have to filter out the /manage* urls from reports. -Kevin --- "Kyler B. Laird" <laird@ecn.purdue.edu> wrote:
I see that someone has already pointed out the Z2.log, but in case anyone is interested in more complex solutions, here's one we're pursuing.
We're going to use multiple Apache (HTTP/HTTPS) servers as the front ends to a line of ZEO clients. Using Z2.log with multiple machines is very clumsy and I want to be able to make powerful realtime queries, so we are going to use PostgreSQL (or Oracle, if forced - ZODB is *not* appropriate for this). http://techdocs.postgresql.org/techdocs/apachepglogd.php
This will collect all of the hits to all of the HTTP(S) servers and will keep track of things I did not see in Z2.log such as time spent to answer the request. ("Who gets to pay for the next ZEO client?") If I want a report, I'll whip up something using psycopg and ZGDChart. I'll also let my users do this (as we do with Analog on Apache now) instead of resorting to the tacky counter images.
I won't have disks on the ZEO clients (nor on the Apache servers), so I will probably eliminate Z2.log.
--kyler
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
__________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail http://personal.mail.yahoo.com/
On Sun, 8 Jul 2001 13:33:46 -0700 (PDT) you wrote:
If you're running Apache in front of Zope (with mod_rewrite/mod_proxy)
I recently came to the conclusion that it's got to be FastCGI. Zope is way too proxy- hostile. (I might start wielding my kludge stick at this problem again sometime soon, but I lost in my bout with it last night.)
then you can still use Apache's logfiles.
With several servers, it's not something I want to do. We have over 70 Apache servers running that way now and trying to manage years of logs is enough of a pain. (I just submitted a request for a restore of logs that were blown away in a Solaris upgrade.) Attempting to scan through more than a few days of logs for a realtime query is just ridiculous. I'm hoping PostgreSQL will be my savior. --kyler
The most flexible program for this task is Analog. It allows you e.g. to exclude URLs based on regular expressions.. Andreas ----- Original Message ----- From: "Kevin L" <kevinsl@yahoo.com> To: "Kyler B. Laird" <laird@ecn.purdue.edu>; "Gitte Wange" <gitte@mmmanager.org> Cc: <zope@zope.org> Sent: Sonntag, 8. Juli 2001 15:33 Subject: Re: [Zope] Site statistics
If you're running Apache in front of Zope (with mod_rewrite/mod_proxy) then you can still use Apache's logfiles. I'm currently doing this and it works fine. But I have to filter out the /manage* urls from reports.
-Kevin
--- "Kyler B. Laird" <laird@ecn.purdue.edu> wrote:
I see that someone has already pointed out the Z2.log, but in case anyone is interested in more complex solutions, here's one we're pursuing.
We're going to use multiple Apache (HTTP/HTTPS) servers as the front ends to a line of ZEO clients. Using Z2.log with multiple machines is very clumsy and I want to be able to make powerful realtime queries, so we are going to use PostgreSQL (or Oracle, if forced - ZODB is *not* appropriate for this). http://techdocs.postgresql.org/techdocs/apachepglogd.php
This will collect all of the hits to all of the HTTP(S) servers and will keep track of things I did not see in Z2.log such as time spent to answer the request. ("Who gets to pay for the next ZEO client?") If I want a report, I'll whip up something using psycopg and ZGDChart. I'll also let my users do this (as we do with Analog on Apache now) instead of resorting to the tacky counter images.
I won't have disks on the ZEO clients (nor on the Apache servers), so I will probably eliminate Z2.log.
--kyler
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
__________________________________________________ Do You Yahoo!? Get personalized email addresses from Yahoo! Mail http://personal.mail.yahoo.com/
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Hi Gitte, did you ever wonder why this var/Z2.log file grows and grows until it fills the disk? ;) Regards Tino ;) --On Sonntag, 8. Juli 2001 14:48 +0200 Gitte Wange <gitte@mmmanager.org> wrote:
Hello,
I was just wondering if Zope logs who visits what on a site ???
Usually everything that happens on your webpages (when you e.g. use apache) ends up in you webservers access_log. But does ZServer logs the same ?
If so - has anyone created a product to get this data out of the log files and display them on a webpage ?? Before I started on Zope, I was using something called webalizer.
Or - is it hard to "analyze" a log file and display the data from it ???
Regards, Gitte Wange
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
The Zope logfile is compliant with the common logfile format for webservers. So you can use any program to analyze the Zope logs (my favourite is ANALOG (www.analog.cx)). The newest versions of Zope also log the user informations so you can create more detailed statistics when you need it. Andreas ----- Original Message ----- From: "Gitte Wange" <gitte@mmmanager.org> To: <zope@zope.org> Sent: Sonntag, 8. Juli 2001 07:48 Subject: [Zope] Site statistics
Hello,
I was just wondering if Zope logs who visits what on a site ???
Usually everything that happens on your webpages (when you e.g. use apache) ends up in you webservers access_log. But does ZServer logs the same ?
If so - has anyone created a product to get this data out of the log files and display them on a webpage ?? Before I started on Zope, I was using something called webalizer.
Or - is it hard to "analyze" a log file and display the data from it ???
Regards, Gitte Wange
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
From: Gitte Wange <gitte@mmmanager.org>
Before I started on Zope, I was using something called webalizer.
Webalyzer works on the zope log.... I've actually been mucking about trying to figure out how to run it from zope, or every nn minutes, and then have the output html available (probably the last part, using localFS or something).... haven't gotten too far on the project yet tho.
participants (8)
-
Andreas Jung -
Gitte Wange -
Kevin L -
Kyler B. Laird -
marc lindahl -
Norman Khine -
Oleg Broytmann -
Tino Wildenhain