BTreeFolder2.objectIds() - accessing _tree.keys() slow
I have very large BTreeFolder2 (CMFMember via BaseBTreeFolder in Archetypes) - has about 260k items in _tree - objectIds() is painfully slow, as is self._tree.keys() - I've casually observed using the meta type index to get the object ids is many orders of magnitude faster. Hacking objectIds() as follows (diff against trunk pasted inline) - gettting ids off of the meta type index for all used meta types - seems to make things much quicker. Two questions: 1. Can this be considered safe in all cases (I'll code this behavior on a Subclass or in a monkey patch)? 2. If yes to #1, is there an appropriate place to lobby for this to be changed, perhaps in BTreeFolder2 product itself? Index: BTreeFolder2.py =================================================================== --- BTreeFolder2.py (revision 41285) +++ BTreeFolder2.py (working copy) @@ -341,10 +341,14 @@ # Returns a list of subobject ids of the current object. # If 'spec' is specified, returns objects whose meta_type # matches 'spec'. + + mti = self._mt_index + if spec is None: + spec = mti.keys() + if spec is not None: if isinstance(spec, StringType): spec = [spec] - mti = self._mt_index set = None for meta_type in spec: ids = mti.get(meta_type, None) Thanks, Sean +----------------------------------------------------------+ Sean Upton SignOnSanDiego.com Site Technology Supervisor The San Diego Union-Tribune 619.718.5241 sean.upton@signonsandiego.com 350 Camino De La Reina San Diego, CA 92108 Plone Powered! plone.org ++ python.org ++ zope.org +----------------------------------------------------------+
sean.upton@uniontrib.com wrote:
I have very large BTreeFolder2 (CMFMember via BaseBTreeFolder in Archetypes) - has about 260k items in _tree - objectIds() is painfully slow, as is self._tree.keys() - I've casually observed using the meta type index to get the object ids is many orders of magnitude faster.
Hacking objectIds() as follows (diff against trunk pasted inline) - gettting ids off of the meta type index for all used meta types - seems to make things much quicker. Two questions:
Are you sure this actually works? _mt_index.keys() is supposed to provide a list of all meta_types used in the folder. To get the object ids from it, you'd need something like this: ids = [] for d in self._mt_index.values(): ids.extend(d.keys()) The structure of _mt_index is documented in a comment: _mt_index = None # OOBTree: { meta_type -> OIBTree: { id -> 1 } } (It's strange that I chose to use an OIBTree instead of an OOTreeSet. Maybe I didn't know about the set support in the BTree module at the time.) Shane
sean.upton@uniontrib.com wrote:
I have very large BTreeFolder2 (CMFMember via BaseBTreeFolder in Archetypes) - has about 260k items in _tree - objectIds() is painfully slow, as is self._tree.keys()
Independently of the fastest way, I'm curious what use case you have to have a need to get all the keys of a big BTree. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
participants (3)
-
Florent Guillaume -
sean.upton＠uniontrib.com -
Shane Hathaway