[Zope] TIP: Robots.txt

[BZ]

It is always the trade off when  you tell robots what to and what not 
to index. I will let Zope security keep up the bad robots, but I rather 
good robots not even try.

BZ


On Saturday, November 9, 2002, at 02:03  PM, Dylan Reinhardt wrote:

> I'm not sure how smart it is to put your manage and acl_users 
> directories in robots.txt, frankly.
>
> For benevolent robots that observe the restrictions you've placed on 
> them, this does little aside from telling them not to index your 
> images... which most of them don't anyway.  They won't index what they 
> can't get to and whatever is password protected isn't likely to be 
> indexed.
>
> For less benevolent visitors, however, you've essentially told them 
> exactly where to look for your vulnerabilities.  Not cool.
>
> I don't know what you're seeing, but I get visitation from malicious 
> robots at least daily.  I'd hate to think what I'd get if I actually 
> gave away any more implementation details.
>
> I also do virtual hosting and I've been hard-pressed to improve on 
> having a single robots.txt in the root that allows everything for 
> everyone.  Pages that need to be kept private should be password 
> protected... everything that's not should be considered fully public.  
> And that single file in root is trivial to override on a site-by-site 
> basis, as desired.
>
> FWIW,
>
> Dylan
>
>
>
> At 07:59 AM 11/9/2002 -0500, you wrote:
>> I have many websites running off of one Zope installation and so it 
>> was not too bright to put a robots.txt fill in the Zope root and let 
>> all the sites inherit it down. Today, though, I wanted to customize 
>> one of them so I did this.
>>
>> # Robots file for <dtml-var absolute_url>
>> User-agent: *
>> Disallow: /Manage
>> Disallow: /acl_users
>> Disallow: /images
>> <dtml-if robot_local>
>> <dtml-var robot_local>
>> </dtml-if>
>>
>> Again, nothing hugely brilliant, but I thought it was interesting 
>> enough to publish.
>>
>> I will put it up on Zope.org.
>>
>> Cheers,
>> BZ
>>
>>
>> _______________________________________________
>> Zope maillist  -  Zope@zope.org
>> http://lists.zope.org/mailman/listinfo/zope
>> **   No cross posts or HTML encoding!  **
>> (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce
>> http://lists.zope.org/mailman/listinfo/zope-dev )
>
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce
> http://lists.zope.org/mailman/listinfo/zope-dev )
>