[ZWeb] Zope.org currently unusable

[Jens Vagelpohl]

On Mar 10, 2005, at 15:27, Andrew Sawyers wrote:

> I need to read up on the robots.txt spec.  Excellent Mark, thanks.
> Andrew

That piece is not part of the spec. Just like the wildcards that Google 
claims they use (and I still don't believe that works as advertised). 
This is the spec:

http://www.robotstxt.org/wc/norobots.html

Here is a robots.txt validator:

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

Here's a funny one: Some collected all the reckless/useless user agents 
for exclusion:

http://www.searchenginegenie.com/Dangerous-user-agents.htm

This one explains Slurp-specific extensions:

http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html

jens