[Zope] Performance related to content loops/depth

Granzow, Doug (NCI) granzowd@mail.nih.gov
Thu, 6 Jun 2002 13:04:41 -0400


Bear with me, I'm a sysadmin, not a developer.  

I'm maintaining a Zope 2.5.1 server with Python 2.1.3 on Solaris 7.  We
recently upgraded from Zope 2.3.1.  I'm seeing a problem where,
occasionally, a bot will come along to index our site and suddenly Zope
stops responding to all requests.  It looks like what is happening is that
the bot finds a loop in the content where each time you click on a link, you
are returned to the same page, but the URL has one more level of depth.  For
example, I see requests in the log file for things like this:

siteroot/info/catalog/catalog/catalog/catalog/catalog/catalog/catalog/catalo
g/catalog/catalog/catalog/catalog/catalog/catalog/catalog

When a bot hits this it keeps going deeper and deeper until we deny access
to the bot.  When I request a URL like that in my browser, even just one
time, Python's CPU usage shoots to near 100%, and stays there for several
seconds.  If I keep clicking on the link to go deeper, it doesn't take long
for it to effectively become a denial of service attack, because the
processing time required to process each request seems to grow more and more
as you go deeper.

Is there any way to stop this from happening?  Obviously we want to remove
loops like this from the content, but as I said, I'm the sysadmin, not the
developer.  

Any input is appreciated!