Non compliant robots
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 What do people do when their sites get traversed by a robot that keeps generating Zope errors ( cookies not enabled I guess ) that doesn't observe the robots.txt conventions? ( I get the errors emailed me which is annoying :-( ) The one currently going thru my site comes from 171.69.236.238 and 171.69.236.239 - -- Regards, Graham Chiu gchiu<at>compkarori.co.nz http://www.compkarori.co.nz/index.php Powered by Interbase and Zope -----BEGIN PGP SIGNATURE----- Version: PGPsdk version 1.7.1 iQA/AwUBOVpAZbTRdIWzaLpMEQLmGwCcDRIgm2pZo0XLAGTd+5iM0EcYlzIAoLQj r1yx7QiLPzpVABkExHGi8UM6 =dJ5z -----END PGP SIGNATURE-----
What do people do when their sites get traversed by a robot that keeps generating Zope errors ( cookies not enabled I guess ) that doesn't observe the robots.txt conventions? ( I get the errors emailed me which is annoying :-( )
Hey Graham, I have exactly the same problem (robot hits my site, I get 400 emails of errors even with a robots.txt file). I am not sure what the answer is except to add something to your error page that would filter out robot errors. J
In article <B580AF7D.8DB8%jatwood@bwanazulia.com>, J. Atwood <jatwood@bwanazulia.com> writes
I have exactly the same problem (robot hits my site, I get 400 emails of errors even with a robots.txt file). I am not sure what the answer is except to add something to your error page that would filter out robot errors.
Since mine happens only on certain pages (forms), perhaps I should just build up a list of bad remote IP addresses ( criteria - generates more than 5 errors on the same page ), and then redirect those clients to another non form page with indexing information for my site... sounds like the basis of a product <g> -- Regards, Graham Chiu gchiu<at>compkarori.co.nz http://www.compkarori.co.nz/index.php Powered by Interbase and Zope
participants (2)
-
Graham Chiu -
J. Atwood