[Zope-dev] Re: ZCatalog algorithms

[Damian Morton]

This is a note to the Zope dev team.

Perhaps I'm out of place suggesting this, but theres an extremely good indexing package, which is open source (i think), and is available, called MG. Theres a book that goes with it called "Managing Gigabytes" co-authored by an old university lecturer of mine, Allistair Moffat. the book and package can be found at http://www.cs.mu.oz.au/mg/

The focus of the package, and the book, is in creating highly efficient compressed indexes and data. They use Perfect Hashing for their indexes. Their indexing technique is geared to creating static indexes, and they rebuild the Perfect Hash tables and the compression dictionary once in a while.

I note from reading through the ZCatalog code that the authors have some performance concerns, and were wondering where to go next. You could do worse than taking a look at the MG code and/or book.