[Zope] Zope URLs and spiders' behavior

Sean Lee sean at erexi.com.tw
Wed Dec 3 09:03:06 EST 2003


I wonder if anyone has experienced this:

a) Google displays links to Zope site with a space in them 
For example, the search result has the right Title (and click on it
works), but underneath the page summary the URL (some, not all) may be
displayed like this
http://domain.com/dir/ subdir/index_html

b) Spiders download non-existing directories
For example, the site has two directories, site.com/dir1, site.com/dir2
The weird thing is that dir1 can be accessed in these (and more) ways:
http://site.com/dir2/dir1
http://site.com/dir1/dir2/dir1
Unnecessary to mention, the spider keeps downloading all the time.
The worst of all is, the directories really can be accessed that way.
Why is this possible?
I noticed some spiders (Altavista, for example) are more likely to do
that, some less. I blocked some that I can, the others are too important
to block. 

Thanks
Zoper 2.5.x





More information about the Zope mailing list