[Zope] robots.txt

[Farrell, Troy]

I don't know where your robot.txt is coming from.  To the best of my
knowledge, zope doesn't create one for you, and you can add your own.  You
might want to make sure it has something like:
User-Agent: *
Disallow: /misc_/
Disallow: /p_/
Disallow: /manage

Troy

-----Original Message-----
From: Paz [mailto:paz@chello.nl]
Sent: Friday, June 01, 2001 1:34 AM
To: Zope
Subject: [Zope] robots.txt


Hi,

Of course we all know about robots.txt, but I must say I havent created one
in my zopeshop, yet, I see that crawlers are coming in and checking them
out, with the 200 status. I don't have one in apache either which is my
front-end, so where is it coming from? Does zope create on automatically?
Will one that I create myself (DTML Document) in the root of zope be picked
up?


Thanks,
Paz


_______________________________________________
Zope maillist  -  Zope@zope.org
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )