[Zope-dev] Search Features and Zope Directions Road Map

[Brian Lloyd]

> after reading the Directions Roadmap from DC, I was surprised that
> a substantial improvement of the searching features of Zope, wasn't
> mentioned as a major concern.
>
> <snip>
>
> Moreover,
> everybody uses more Google to look for everything, bypassing
> windows, doors, and
> portals!. Why? Because it's terribly smart (not mentioning its 6,000 Linux
> boxes, by the way), and because there's no need to follow the
> highly-engineered
> information architecture of a web site, if there's a trustful
> shorcut to the
> relevant content!. So, if I'd have to mention one big feature
> improvement to
> Zope, I wouldn't doubt: "search engine".
> <snip>
>
> On the other hand, Fast, from
> Norway, already have a nice multimedia search engine, from regular,
> non-structured, spidered web pages. Can we do that "structuring the
> unstructured" thing within Zope?

You have posed an important question (and, probably some answers),
that hopefully I can clarify.

One of the all-important points of the Zope directions document
is that our number one goal is to make it wildly easier for
_developers_ to create and deploy quality components. Why is this
so important? Your questions in this email is why that is so
important.

You are very interested in high-quality search capabilities, and
others certainly are as well. Some other folks care more about
E-Commerce, or Corba integration, or communication with Java
components.

The problem, of course, is that even if DC devoted every single
person here to creating the "best search engine" (which we
couldn't do for very long - we'd soon be gone), we would still
be hard pressed to even come close to making everybody happy
or being competitive with every other search engine vendor
out there. And the reality is that it is not our goal in life
to be a better Google than Google.

Multiply that by the number of things people want (ECommerce, Corba,
et. al.), and the problem is quite clear - *DC cannot possibly
provide the best, most featureful and competitive component for
every problem*.

The *solution* to this problem is what is outlined in the Zope
directions document - dramatically lowering the bar of development
to allow a thriving marketplace of robust components (that are *not*
written by DC), allows interested parties to write (or better yet,
reuse) "the best x component" for their purposes.

In the future, Zope may come with "some batteries included", in that
a Zope distribution may include the latest versions of the most popular
and widely used components. But we hope that the idea of "The ZCatalog"
(for instance) will fall by the wayside. Zope may still come with a
search component such as ZCatalog that is useful for certain tasks and
perhaps as a learningtool, but it will not be an infinitely-scalable
infinitely-featureful thing that everyone uses for every problem.

The hope is that when you outgrow ZCatalog you can move on to other
search components particularly suited to your problem domain. If
you scale beyond what ZC can handle, maybe you move up to some
VeritySearch component that makes use of existing software. Even
now, with the current pain level of component development, building
a VeritySearch component would probably take considerably less time
than building and maintaining equivalent features into "the ZCatalog".

This is the future - the way that Zope will succeed is by being the
best framework and component integration platform for the Web, not
by trying to compete with verticals like search engine vendors on
feature points. "Use the right tool for the job" is something we
have always believed in, and providing a platform that will allow
you to use and integrate the most appropriate tools will be our
focus going forward. That is why "substantial improvement of
searching features" is not on the futures roadmap - we do not want
to provide the best search engine for every task. We want to make
it easy for you to build or integrate the "right" search solution
for your task.


Brian Lloyd        brian@digicool.com
Software Engineer  540.371.6909
Digital Creations  http://www.digicool.com