[ZO-Coll] [ZOC] 81/ 1 Request "robots.txt has invalid syntax"

[Collector: NEW Zope.org (the ...]

Issue #81 Update (Request) "robots.txt has invalid syntax"
 Status Pending, content/bug medium
To followup, visit:
  http://collector.zope.org/ZopeOrg/81

==============================================================
= Request - Entry #1 by pieterb on Jun 21, 2003 5:58 am

The current /robots.txt has the following contents:

> User-agent: wget
> Disallow: /
>
> User-agent: Wget
> Disallow: /
>
> # Ask Google to skip search queries and the like.
> User-agent: Googlebot
> Disallow: /*?

You simply can't ask's robots like Google to skip certain queries. You need to have a <meta name="robots" value="noindex,nofollow"> in the head of those pages. See http://www.robotstxt.org/wc/exclusion.html (or other pages on that site), for more information.

PieterB, 

--
http://zwiki.org/PieterB
==============================================================