Ahh, Harvest. Brings back old memories. I started using it in 1992/93 and we actually fielded some consulting opportunities using it before the group disbanded (most going to Netscape to start the Catalog Server).
I thought Harvest was mostly dead?
Anyway, having a web crawler index your site has strengths and limitations as have been described here. We at one time had extended Harvest to make it "play nice" with an object system we had created (for employment info), but in the end the two architectures were too disimilar.
With the advent of the Zope Catalog an intersting derivative idea might be to allow Zope to cough up RDF descriptions of definable (i.e., /Zope/Marketing/*, /Zope/US/Sales*, etc.) portions of the object space. I know that InfoSeek's Ultraseek and Netscape's Compass servers will gather RDF (to use a Harvest word). This lets the local server add most of the value and the spider's can come along for a relatively low-impact RDF snarfing visit. Thoughts? --Rob