[Zope] TIP: Robots.txt

[Dylan Reinhardt]

I'm not sure how smart it is to put your manage and acl_users directories 
in robots.txt, frankly.

For benevolent robots that observe the restrictions you've placed on them, 
this does little aside from telling them not to index your images... which 
most of them don't anyway.  They won't index what they can't get to and 
whatever is password protected isn't likely to be indexed.

For less benevolent visitors, however, you've essentially told them exactly 
where to look for your vulnerabilities.  Not cool.

I don't know what you're seeing, but I get visitation from malicious robots 
at least daily.  I'd hate to think what I'd get if I actually gave away any 
more implementation details.

I also do virtual hosting and I've been hard-pressed to improve on having a 
single robots.txt in the root that allows everything for everyone.  Pages 
that need to be kept private should be password protected... everything 
that's not should be considered fully public.  And that single file in root 
is trivial to override on a site-by-site basis, as desired.

FWIW,

Dylan



At 07:59 AM 11/9/2002 -0500, you wrote:
>I have many websites running off of one Zope installation and so it was 
>not too bright to put a robots.txt fill in the Zope root and let all the 
>sites inherit it down. Today, though, I wanted to customize one of them so 
>I did this.
>
># Robots file for <dtml-var absolute_url>
>User-agent: *
>Disallow: /Manage
>Disallow: /acl_users
>Disallow: /images
><dtml-if robot_local>
><dtml-var robot_local>
></dtml-if>
>
>Again, nothing hugely brilliant, but I thought it was interesting enough 
>to publish.
>
>I will put it up on Zope.org.
>
>Cheers,
>BZ
>
>
>_______________________________________________
>Zope maillist  -  Zope@zope.org
>http://lists.zope.org/mailman/listinfo/zope
>**   No cross posts or HTML encoding!  **
>(Related lists - http://lists.zope.org/mailman/listinfo/zope-announce
>http://lists.zope.org/mailman/listinfo/zope-dev )