[Zope-dev] BTreeFolder2.objectIds() - accessing _tree.keys() slow

[sean.upton at uniontrib.com]

> > Hacking objectIds() as follows (diff against trunk pasted inline) - 
> > gettting ids off of the meta type index for all used meta types - 
> > seems to make things much quicker.  Two questions:
> 
> Are you sure this actually works?  _mt_index.keys() is 
> supposed to provide a list of all meta_types used in the 
> folder.  To get the object ids from it, you'd need something 
> like this:
> 
> ids = []
> for d in self._mt_index.values():
>    ids.extend(d.keys())
> 
> The structure of _mt_index is documented in a comment:
> 
>      _mt_index = None  # OOBTree: { meta_type -> OIBTree: { 
> id -> 1 } }

I'm pretty sure this works.  The in each OIBTree in _mt_index, the keys are
ids of contained objects for all respective meta types in the folder - I use
self._mt_index.keys() to list all meta_types and leverage your original code
that runs when spec parameter has been passed gets run every time - loops
through all meta_types does a union() of set and ids for each meta_type.

This returns the equivalent of running
self.objectIds(spec=self._mt_index.keys()) on the current trunk/release
code, which should be identical to self._tree.keys(), but much, much faster.
I'm still somewhat ignorant as to why self._tree.keys() is so slow with
100k-plus objects (waking up too many persistent objects?), but using the
ids stored a few layers deep in the _mt_index seems a viable alternative
with the same expected return result.

With a bit more effort put in, the diff pasted below should be more complete
and illustrate better:


Index: BTreeFolder2.py
===================================================================
--- BTreeFolder2.py     (revision 41285)
+++ BTreeFolder2.py     (working copy)
@@ -341,21 +341,22 @@
         # Returns a list of subobject ids of the current object.
         # If 'spec' is specified, returns objects whose meta_type
         # matches 'spec'.
-        if spec is not None:
-            if isinstance(spec, StringType):
-                spec = [spec]
-            mti = self._mt_index
-            set = None
-            for meta_type in spec:
-                ids = mti.get(meta_type, None)
-                if ids is not None:
-                    set = union(set, ids)
-            if set is None:
-                return ()
-            else:
-                return set.keys()
+
+        mti = self._mt_index
+        if spec is None:
+            spec = mti.keys() #all meta types
+
+        if isinstance(spec, StringType):
+            spec = [spec]
+        set = None
+        for meta_type in spec:
+            ids = mti.get(meta_type, None)
+            if ids is not None:
+                set = union(set, ids)
+        if set is None:
+            return ()
         else:
-            return self._tree.keys()
+            return set.keys()


     security.declareProtected(access_contents_information,