Ian Blenke writes:
Something looks like it is attempting to walk your Zope site. The fact that it checks for /robots.txt first shows that it is trying to play nice. When it walks your site, it is clobbering your Zope server.
Here is a sample robots.txt that you should use:
User-agent: * Disallow: /
- Ian C. Blenke <icblenke@2c2.com>
If your hypothesis is correct, then I concede that using such a robots.txt file would be a plausible workaround. However: 1. WHY would a Zope server go down simply because something is walking it? 2. I have run wget -r on this server quite a bit, simulating a robot, walking everything on the site. There has been no correlation between this activity and problems with the site. I therefore assert that our Zope server does NOT go down simply because something is walking it. 3. What if I want the crawlers to be able to walk my site? Perhaps I want the search engines to be indexing our content. I appreciate the suggestion, but using a robots.txt file to prevent walking seems like fixing the symptom rather than the problem. Furthermore, I'm not even convinced that this would fix the symptom, because of my experiences mentioned in point 2, above. -- Eric W. Sink, Software Craftsman SourceGear Corporation eric@sourcegear.com