[ZODB-Dev] FileStorage index size optimizaion (Re: FileStorage patch)

[Jim Fulton]

Thanks Chris,

Important note:

To get the benefit of this, you have to remove your old index.
FileStorage uses an index object that it pickles to it's
index file when you save the index (normal shutdown or pack).
If FileStorage reads the old index from the index file, the index
will have the old type (aka type({}) ;). To get the FileStorage
to use the new BTree-based index implementation, you need to
get it to build a new index by starting without an index file
or packing.

Jim

Chris Withers wrote:
> 
> emf wrote:
> >
> > As I currently run 30-60 storage servers on a machine, I would be very
> > interested in testing out such a patch, if you'd be willing to send it
> > along.
> 
> It's in CVS, just check out the appropriate branch:
> 
> Jim Fulton wrote:
> >
> > OK, I made a CVS branch, BTreeFSIndex-branch (made from the Zope-2_4-branch),
> > for just the BTrees and ZODB directories.  If you update to that branch
> > you should get my experimental changes.  The BTrees package has a new
> > extension, _fsBTrees that has 2-char to 6-char BTree types.
> >
> > The ZODB fsIndex.py provides a FileStorage index based on this BTree.
> > You should get a memory consumption of only a little more than 8 bytes
> > per object. Note that the file size is limited to about 256 terabytes.
> > Nothing is free. :)
> 
> cheers,
> 
> Chris
> 
> PS: Still haven't managed to get the machine resurrected :-(

--
Jim Fulton           mailto:jim@zope.com       Python Powered!        
CTO                  (888) 344-4332            http://www.python.org  
Zope Corporation     http://www.zope.com       http://www.zope.org