BTreeFolder2 add speed with 10k objects
I'm using a CMFBTreeFolder to hold objects for a query later. Each object in the folder has a few string members. The folder now has about 10,200 objects. When I add a new instance of the object to the folder, it takes around 18 seconds. I've tracked the hotspot down to BTreeFolder2._setOb()'s: tree[id] = object which takes around 17 seconds. By comparison, the rest of the request including cataloging the object takes less than a second. I'm using Zope 2.6.1, CMF 1.3.1. I updated by BTreeFolder2 code to the latest from CVS and ran manage_cleanup() which reported no errors. Any ideas how I can improve this? Thanks, Jeff
Jeff Youel wrote at 2003-10-20 12:02 -0700:
I'm using a CMFBTreeFolder to hold objects for a query later. Each object in the folder has a few string members. The folder now has about 10,200 objects. When I add a new instance of the object to the folder, it takes around 18 seconds. I've tracked the hotspot down to BTreeFolder2._setOb()'s:
tree[id] = object
which takes around 17 seconds. By comparison, the rest of the request including cataloging the object takes less than a second.
That is really astonishing: As always for trees, insertion takes time in the order of the tree depth, which is logarithmic in the number of tree entries, provided the tree is quite balanced. The "BTrees.check" module has a function "display". You can use it to display your tree. What be interesting how it looks like. Dieter
From: Dieter Maurer, Sent: Tuesday, October 21, 2003 1:08 PM
Thanks for the response.
That is really astonishing:
It is, but I'm not sure my test rigging was correct. Today, it looks like the BTreeFolder2._setOb() is taking the time but the tree[id]=object line is not responsible. Instead the object.manage_afterAdd(object, self) call seems to be the slowdown. I was able to work around the problem by replacing BTreeFolder2._setObject() with my own version for this special case.
The "BTrees.check" module has a function "display". You can use it to display your tree. What be interesting how it looks like.
Thanks, I'll try that. Jeff
Jeff Youel wrote at 2003-10-21 14:44 -0700:
... It is, but I'm not sure my test rigging was correct. Today, it looks like the BTreeFolder2._setOb() is taking the time but the tree[id]=object line is not responsible. Instead the object.manage_afterAdd(object, self) call seems to be the slowdown.
I like to suggest my "ZopeProfiler" product to analyse such things <http://www.dieter.handshake.de/pyprojects/zope> It is unable to safely measure time spent in C functions (such as "tree[id]= object") and will associate their time with the enclosing Python function. Nevertheless, you can clearly notice when the time is spent in another Python function (such as "manage_afterAdd"). Dieter
participants (2)
-
Dieter Maurer -
Jeff Youel