[ZWeb] Zope.org currently unusable

[Mark Pratt]

Jens,

You are correct that the crawl-delay parameter is not part of the spec.
Their are plenty of examples where specs don't keep up with the times 
and their is no harm done using that tag.
Worrying about a robots.txt file is a bit over the top :-)

Thanks for the link to the reckless/useless user agents page.

Cheers,

Mark

On Mar 10, 2005, at 3:39 PM, Jens Vagelpohl wrote:

>
> On Mar 10, 2005, at 15:27, Andrew Sawyers wrote:
>
>> I need to read up on the robots.txt spec.  Excellent Mark, thanks.
>> Andrew
>
> That piece is not part of the spec. Just like the wildcards that 
> Google claims they use (and I still don't believe that works as 
> advertised). This is the spec:
>
> http://www.robotstxt.org/wc/norobots.html
>
> Here is a robots.txt validator:
>
> http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
>
> Here's a funny one: Some collected all the reckless/useless user 
> agents for exclusion:
>
> http://www.searchenginegenie.com/Dangerous-user-agents.htm
>
> This one explains Slurp-specific extensions:
>
> http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
>
> jens
>
> _______________________________________________
> Zope-web maillist  -  Zope-web at zope.org
> http://mail.zope.org/mailman/listinfo/zope-web
>
>