[Zope] Questions on ZODB BTrees versus bsddb BTrees

[Terry Jones]

[Following up on my original posting, with a summary of resolution.]

Thanks for the help & pointers to ZCatalog.

I actually did find a standalone version of ZCatalog, created by Kevin
Dangoor, at http://www.blazingthings.com/dev/zcatalog.html

His pulls code out of Zope3, and rewrites some of it. Kevin's code and one
example each requires one trivial change to work on the current Zope3 SVN
version (email me if you need these).

I read the plope chapter on ZCatalog, but its emphasis is on getting at
ZCatalog from zope, whereas I want to do everything from python, with no
zope instance.

Meanwhile, I had decided that my option #3

|  3. Redesign what I'm doing so as to only need ZODB BTrees. This is quite
|     far from optimal (in terms of space and time), but might be the
|     quickest to implement.
| 
|     However, this may not solve the problem. If a fundamental need is to
|     (e.g.) find all values >= X, where duplicates are allowed but you're
|     using trees that do not allow duplicate keys, this is difficult.  Hence
|     the question about byValue().

actually wasn't too hard, and that I should simply store values as keys and
let values be a list of objects that had that value. (I hope that makes
sense)

Then, finally reading the Zope3 src/zope/app/catalog/README.txt I saw that
that is basically what ZCatalog already does (except with a
BTrees.IFBTree.IFTreeSet not a list), as well as adding indexing.

That solution wastes space, as values are stored twice. How much of a waste
depends on the multiplicity of value values, but it could about a 2x waste
(i.e., when all values are distinct and their size is large compared to the
size of an docid). And there's more pickling, and more traffic to the
storage.  But.... that's life I guess. I'd rather the simpler, easier
solution that lies wholly within ZODB/ZEO and which lets me have just a
single transaction and recovery type to worry about than to hack a hybrid
ZODB/bsddb just to save space and have to deal with those issues in a
robust way.

So that's it, problem solved. Thanks again for the pointer to ZCatalog.

Terry