Thanks Chris, Important note: To get the benefit of this, you have to remove your old index. FileStorage uses an index object that it pickles to it's index file when you save the index (normal shutdown or pack). If FileStorage reads the old index from the index file, the index will have the old type (aka type({}) ;). To get the FileStorage to use the new BTree-based index implementation, you need to get it to build a new index by starting without an index file or packing. Jim Chris Withers wrote:
emf wrote:
As I currently run 30-60 storage servers on a machine, I would be very interested in testing out such a patch, if you'd be willing to send it along.
It's in CVS, just check out the appropriate branch:
Jim Fulton wrote:
OK, I made a CVS branch, BTreeFSIndex-branch (made from the Zope-2_4-branch), for just the BTrees and ZODB directories. If you update to that branch you should get my experimental changes. The BTrees package has a new extension, _fsBTrees that has 2-char to 6-char BTree types.
The ZODB fsIndex.py provides a FileStorage index based on this BTree. You should get a memory consumption of only a little more than 8 bytes per object. Note that the file size is limited to about 256 terabytes. Nothing is free. :)
cheers,
Chris
PS: Still haven't managed to get the machine resurrected :-(
-- Jim Fulton mailto:jim@zope.com Python Powered! CTO (888) 344-4332 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org