[Zope] ZCatalog Queries...

[Chris McDonough]

 > > > It's be nice if ZCatalog had a good general purpose
> > > interface, and was a
> > > bit more robust.
> > > (the BTree implementation which has been mentioned a few 
> times springs
> > > to mind here ;-)
> > 
> > Can you be more specific?
> 
> Andy can fill you in on the specifics.

OK...

> 
> >  What's insufficient about the current
> > implementation?
> 
> It doesn't scale well, especially for things where you have 
> lots of new
> data arriving (this is the BTree problem, I think...)

Yes, we're still working on a "broadtree" implementation that may allay
some of these problems, although I don't have an ETA.

> 
> It has no published and well defined query syntax (there's 
> patches here,
> bits there, but no definitive document on how to use it, how to batch
> with it, how to perform complex and structured queries, particularly
> with TextIndex'es)

Hopefully, the Zope book will make it more clear from a user
perspective, and sometime in the very distant future I will be writing a
chapter in the developer's guide about the catalog.  I agree that the
ZCatalog wrapper should probably wrap more of the underlying catalog's
methods, but these need to be rationalized, defined and then documented
in the API docs.  This is something for dev.zope.org, probably.

> 
> Don't get me wrong, it is very cool, but only kindof 70% there :S
> (and I get the impression that doing the remaining 30% properly would
> require a rewrite...)
> 
> As an example, we've been trying to do Zope-based versions of the
> mailing list archives for a coupla months now and the Catalog keeps
> exploding in different ways (huge resource consumption, even for only
> 30K messages or so, no matter what storage is used)

Yes.  Tweaking and the broadtree stuff should make this a little better.

> 
> Then there's the ubiquitous 'KeyError's and other associated 
> weirdness,
> all of which leaves me feeling a lot less than totally 
> confident in the
> Catalog ;-)

These are independent of the coupling problem in the btree, and I'm
trying to vanquish them now.  This is why the annoying logging code was
added to the catalog.  It seems to be related to the TextIndex
implementation, but I'm still trying to pin it down.