[ZODB-Dev] BTrees and id mapping (was Re: Data.fs size grows non-stop)

[Pedro Ferreira]

Hello all,

So, I've been experimenting with a new structure for our date indexes, 
and I was considering using an IOBTree that would map integer timestamps 
to BTree-based sets. As for the sets, I considered two options:

 1. An IITreeSet based on the OID and an additional IOBTree that maps 
back the OIDs to objects;
 2. An OOTreeSet;

As expected 1 was light years faster when I tried to query for large 
timespans (doing normal union operations using OOTreeSets seems to be a 
very slow operation). Still, for single days (and generally timespans 
smaller than 7 days), the OOTreeSet seems to be much faster (apparently, 
iterating through the all the ids in the resulting set and fetching the 
corresponding object takes some time, even for sets with only one 
member, so, IITreeSet seems to be in disadvantage here).
Does anyone have any advice on optimizing the first case? I believe the 
IITreeSet is the way to go, but if only there was some way to make the 
int<->object conversion operation less expensive...

Thanks in advance,

Pedro