[Zope] Ever seen this problem?

['Eric W. Sink']

Ian Blenke writes:

> Something looks like it is attempting to walk your Zope site. The fact
> that it checks for /robots.txt first shows that it is trying to play
> nice. When it walks your site, it is clobbering your Zope server.
> 
> Here is a sample robots.txt that you should use:
> 
> 	User-agent: *
> 	Disallow: /
> 
> - Ian C. Blenke <icblenke@2c2.com>

If your hypothesis is correct, then I concede that using such a
robots.txt file would be a plausible workaround.  However:

1.  WHY would a Zope server go down simply because something is
walking it?

2.  I have run wget -r on this server quite a bit, simulating a robot,
walking everything on the site.  There has been no correlation between
this activity and problems with the site.  I therefore assert that our
Zope server does NOT go down simply because something is walking it.

3.  What if I want the crawlers to be able to walk my site?  Perhaps I
want the search engines to be indexing our content.

I appreciate the suggestion, but using a robots.txt file to prevent
walking seems like fixing the symptom rather than the problem.
Furthermore, I'm not even convinced that this would fix the symptom,
because of my experiences mentioned in point 2, above.

-- 
Eric W. Sink, Software Craftsman
SourceGear Corporation
eric@sourcegear.com