It is always the trade off when you tell robots what to and what not to index. I will let Zope security keep up the bad robots, but I rather good robots not even try. BZ On Saturday, November 9, 2002, at 02:03 PM, Dylan Reinhardt wrote:

I'm not sure how smart it is to put your manage and acl_users directories in robots.txt, frankly.

For benevolent robots that observe the restrictions you've placed on them, this does little aside from telling them not to index your images... which most of them don't anyway. They won't index what they can't get to and whatever is password protected isn't likely to be indexed.

For less benevolent visitors, however, you've essentially told them exactly where to look for your vulnerabilities. Not cool.

I don't know what you're seeing, but I get visitation from malicious robots at least daily. I'd hate to think what I'd get if I actually gave away any more implementation details.

I also do virtual hosting and I've been hard-pressed to improve on having a single robots.txt in the root that allows everything for everyone. Pages that need to be kept private should be password protected... everything that's not should be considered fully public. And that single file in root is trivial to override on a site-by-site basis, as desired.

FWIW,

Dylan

At 07:59 AM 11/9/2002 -0500, you wrote:

...

I have many websites running off of one Zope installation and so it was not too bright to put a robots.txt fill in the Zope root and let all the sites inherit it down. Today, though, I wanted to customize one of them so I did this.

# Robots file for <dtml-var absolute_url> User-agent: * Disallow: /Manage Disallow: /acl_users Disallow: /images <dtml-if robot_local> <dtml-var robot_local> </dtml-if>

Again, nothing hugely brilliant, but I thought it was interesting enough to publish.

I will put it up on Zope.org.

Cheers, BZ

_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

Yes, it's a tradeoff. But you're balancing unknown (possibly substantial) risks against little perceivable benefit. Robots.txt is something of a relic from a simpler time... like finger plans or unsecured rlogin. If you want to make something secure, it's a poor strategy to provide more information than you have to and depend on the visitor to act responsibly with it. Far better you should take appropriate measures to secure what you want to protect. Robots.txt would have been far more useful had it been written as an *inclusion* protocol rather than an *exclusion* protocol... but then again, 1994 was a simpler time for the web. It seemed simpler to do it this way, and that was that. YMMV, of course... security measures should scale with your actual requirements. Perhaps I'm just paranoid. Dylan At 02:42 PM 11/9/2002 -0500, you wrote:

It is always the trade off when you tell robots what to and what not to index. I will let Zope security keep up the bad robots, but I rather good robots not even try.

BZ

On Saturday, November 9, 2002, at 02:03 PM, Dylan Reinhardt wrote:

...

I'm not sure how smart it is to put your manage and acl_users directories in robots.txt, frankly.

For benevolent robots that observe the restrictions you've placed on them, this does little aside from telling them not to index your images... which most of them don't anyway. They won't index what they can't get to and whatever is password protected isn't likely to be indexed.

For less benevolent visitors, however, you've essentially told them exactly where to look for your vulnerabilities. Not cool.

I don't know what you're seeing, but I get visitation from malicious robots at least daily. I'd hate to think what I'd get if I actually gave away any more implementation details.

I also do virtual hosting and I've been hard-pressed to improve on having a single robots.txt in the root that allows everything for everyone. Pages that need to be kept private should be password protected... everything that's not should be considered fully public. And that single file in root is trivial to override on a site-by-site basis, as desired.

FWIW,

Dylan

At 07:59 AM 11/9/2002 -0500, you wrote:

...

I have many websites running off of one Zope installation and so it was not too bright to put a robots.txt fill in the Zope root and let all the sites inherit it down. Today, though, I wanted to customize one of them so I did this.

# Robots file for <dtml-var absolute_url> User-agent: * Disallow: /Manage Disallow: /acl_users Disallow: /images <dtml-if robot_local> <dtml-var robot_local> </dtml-if>

Again, nothing hugely brilliant, but I thought it was interesting enough to publish.

I will put it up on Zope.org.

Cheers, BZ

_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )