[ZWeb] Zope.org currently unusable

[Mark Pratt]

one brief correction

Meant to say:

"worrying about validating a robots.txt file is a bit over the top" :-)

Cheers,

Mark

On Mar 10, 2005, at 4:38 PM, Mark Pratt wrote:

> Jens,
>
> You are correct that the crawl-delay parameter is not part of the spec.
> Their are plenty of examples where specs don't keep up with the times 
> and their is no harm done using that tag.
> Worrying about a robots.txt file is a bit over the top :-)
>
> Thanks for the link to the reckless/useless user agents page.
>
> Cheers,
>
> Mark
>
> On Mar 10, 2005, at 3:39 PM, Jens Vagelpohl wrote:
>
>>
>> On Mar 10, 2005, at 15:27, Andrew Sawyers wrote:
>>
>>> I need to read up on the robots.txt spec.  Excellent Mark, thanks.
>>> Andrew
>>
>> That piece is not part of the spec. Just like the wildcards that 
>> Google claims they use (and I still don't believe that works as 
>> advertised). This is the spec:
>>
>> http://www.robotstxt.org/wc/norobots.html
>>
>> Here is a robots.txt validator:
>>
>> http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
>>
>> Here's a funny one: Some collected all the reckless/useless user 
>> agents for exclusion:
>>
>> http://www.searchenginegenie.com/Dangerous-user-agents.htm
>>
>> This one explains Slurp-specific extensions:
>>
>> http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
>>
>> jens
>>
>> _______________________________________________
>> Zope-web maillist  -  Zope-web at zope.org
>> http://mail.zope.org/mailman/listinfo/zope-web
>>
>>
>
> _______________________________________________
> Zope-web maillist  -  Zope-web at zope.org
> http://mail.zope.org/mailman/listinfo/zope-web
>
>