[Zope-dev] Re: 64-bit BTrees

[Philipp von Weitershausen]

Fred Drake wrote:
> I have a need for 64-bit BTrees (at least for IOBTree and OIBTree),
> and I'm not the first.  I've created a feature development branch for
> this, and checked in my initial implementation.
> 
> I've modified the existing code to use PY_LONG_LONG instead of int for
> the key and/or value type; there's no longer a 32-bit version in the
> modified code.  Any Python int or long that can fit in 64 bits is
> accepted; ValueError is raised for values that require 65 bits (or
> more).  Keys and values that can be reported as Python ints are, and
> longs are only returned when the value cannot be converted to a Python
> int.
> 
> This can have a substantial effect on memory consumption, since keys
> and/or values now take twice the space.  There may be performance
> issues as well, but those have not been tested.
> 
> There are new unit tests, but more are likely needed.
> 
> If you're interested in getting the code from Subversion, it's available at:
> 
>     svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/
> 
> Ideally, this or some variation on this could be folded back into the
> main development for ZODB.  If this is objectionable, making 64-bit
> btrees available would require introducing new versions of the btrees
> (possibly named LLBTree, LOBTree, and OLBTree).

-1 to the L*BTree flavours. The 'long' type is disappearing from Python
anyways, why should the implementation detail worry us regarding BTrees?

+1 to making the I*BTree flavours 64bit-aware.

If Jim is right, existing pickles will be read back just fine and as
long as the integers stay under 32 bit, they won't even be bigger. The
only real implication therefore is the pushing around twice as many bits
in memory. Dieter estimates 20% to 35% slowdown for the C algorithms
(whatever that means), Tim seems to think it won't have such a big
effect. I guess we'll only know after some benchmarks.

Philipp