Shane Hathaway <shane@zope.com> writes:
Looking at the zope.org logs, I once saw GoogleBot generate URLs like this to zope.org of 1000 characters or more.
I had this too, leading to heavy traffic what's more. Cf http://zwiki.org/TheRobotProblem. Since then, as a temporary measure I set the "no robots" meta header in standard_wiki_header. Also, I cleaned up various link "holes" that would lead to infinite urls, using page_url() and wiki_url(). Finally, all wiki links were changed to use absolute urls (now optional). These things should help, and in fact it should be safe to allow google into a modern zwiki. Back to the original poster: yes, to avoid getting hammered by search engines it is necessary to make sure you expose no links leading to infinite urls. This may not be as hard as you think. Hackers can make up urls, but search engines don't (yet). -Simon