[ZWeb] google results duplication
Shane Hathaway
shane@digicool.com
Fri, 11 May 2001 22:17:42 -0400
Karl Anderson wrote:
>
> Searching zope.org with google gets lots of duplicate results due to
> multiple views of many objects that contain the normal view content as
> well. This makes google less useful for searching.
>
> Wiki pages are especially bad for this - the edit, history, backlinks,
> and advanced action views for each page end up in the google listing.
> Most of the other types have a prettyprinting page that shows up.
>
> Is there an easy way to exclude these? Will robots.txt work for page
> suffixes, or just subdirectories?
ZWiki just needs to be fixed to not generate infinite URLs based on
acquisition. If you browse around a ZWiki long enough, you'll find that
path elements in the URL get repeated. Or you can just browse the
Zope.org logs to find googlebot and its excessive URLs.
Shane