[ZO-Coll] [ZOC] 81/ 2 Resolve "robots.txt has invalid syntax"

[Collector: NEW Zope.org (the ...]

Issue #81 Update (Resolve) "robots.txt has invalid syntax"
 Status Resolved, content/bug medium
To followup, visit:
  http://collector.zope.org/ZopeOrg/81

==============================================================
= Resolve - Entry #2 by mj on Aug 6, 2003 3:31 pm

 Status: Pending => Resolved

Google specifically supports this syntax; it was taken from their FAQ.
________________________________________
= Request - Entry #1 by pieterb on Jun 21, 2003 5:58 am

The current /robots.txt has the following contents:

> User-agent: wget
> Disallow: /
>
> User-agent: Wget
> Disallow: /
>
> # Ask Google to skip search queries and the like.
> User-agent: Googlebot
> Disallow: /*?

You simply can't ask's robots like Google to skip certain queries. You need to have a <meta name="robots" value="noindex,nofollow"> in the head of those pages. See http://www.robotstxt.org/wc/exclusion.html (or other pages on that site), for more information.

PieterB, 

--
http://zwiki.org/PieterB
==============================================================