Re: [Zope-dev] SVN: Zope/trunk/ Optimized the `OFS.Traversable.getPhysicalPath` method to avoid excessive amounts of method calls. Thx to Nikolay Kim from Enfold
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/14/2011 04:16 AM, Hanno Schlichting wrote:
Log message for revision 122213: Optimized the `OFS.Traversable.getPhysicalPath` method to avoid excessive amounts of method calls. Thx to Nikolay Kim from Enfold
Changed: U Zope/trunk/doc/CHANGES.rst U Zope/trunk/src/OFS/Traversable.py
-=- Modified: Zope/trunk/doc/CHANGES.rst =================================================================== --- Zope/trunk/doc/CHANGES.rst 2011-07-14 07:16:04 UTC (rev 122212) +++ Zope/trunk/doc/CHANGES.rst 2011-07-14 08:16:08 UTC (rev 122213) @@ -19,6 +19,9 @@ Features Added ++++++++++++++
+- Optimized the `OFS.Traversable.getPhysicalPath` method to avoid excessive + amounts of method calls. + - During startup open a connection to every configured database, to ensure all of them can indeed be accessed. This avoids surprises during runtime when traversal to some database mountpoint could fail as the underlying storage
Modified: Zope/trunk/src/OFS/Traversable.py =================================================================== --- Zope/trunk/src/OFS/Traversable.py 2011-07-14 07:16:04 UTC (rev 122212) +++ Zope/trunk/src/OFS/Traversable.py 2011-07-14 08:16:08 UTC (rev 122213) @@ -114,14 +114,47 @@ access this object again later, for example in a copy/paste operation. getPhysicalRoot() and getPhysicalPath() are designed to operate together. + + This implementation is optimized to avoid excessive amounts of function + calls while walking up from an object on a deep level. """ - path = (self.getId(),) + try: + id = self.id + except AttributeError: + id = self.getId() + else: + if id is None: + id = self.getId()
p = aq_parent(aq_inner(self)) + if p is None: + return (id, )
- if p is not None: - path = p.getPhysicalPath() + path + path = [id] + func = self.getPhysicalPath.im_func + while p is not None: + if func is p.getPhysicalPath.im_func: + try: + pid = p.id + except AttributeError: + pid = p.getId() + else: + if pid is None: + pid = p.getId()
+ path.insert(0, pid) + try: + p = p.__parent__ + except AttributeError: + p = None + else: + if IApplication.providedBy(p): + path.insert(0, '') + path = tuple(path) + else: + path = p.getPhysicalPath() + tuple(path) + break + return path
security.declarePrivate('unrestrictedTraverse')
While we're at it, 'list.insert(0,foo)' is known to be slower in a tight loop than 'list.append(foo)', with a 'reversed' at the end of the loop:: $ python -m timeit -s "from string import letters" "path = []" \ "for letter in letters: path.append(letter)" \ "result = tuple(reversed(path))" 100000 loops, best of 3: 15.9 usec per loop $ python -m timeit -s "from string import letters" "path = []" \ "for letter in letters: path.insert(0, letter)" \ "result = tuple(path)" 10000 loops, best of 3: 25.3 usec per loop For the sake of comparison, the original tuple addition is actually between the two: $ python -m timeit -s "from string import letters" "result = ()" \ "for letter in letters: result += (letter,)" 10000 loops, best of 3: 21.6 usec per loop Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4fD4gACgkQ+gerLs4ltQ7b3QCg1z+oZKKDZ5IhuMvAockJ9mvx SO4AoIr3IEMnq78m70DZAOc0yV9+++y4 =8vOY -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/14/2011 11:47 AM, Tres Seaver wrote:
While we're at it, 'list.insert(0,foo)' is known to be slower in a tight loop than 'list.append(foo)', with a 'reversed' at the end of the loop::
$ python -m timeit -s "from string import letters" "path = []" \ "for letter in letters: path.append(letter)" \ "result = tuple(reversed(path))" 100000 loops, best of 3: 15.9 usec per loop
$ python -m timeit -s "from string import letters" "path = []" \ "for letter in letters: path.insert(0, letter)" \ "result = tuple(path)" 10000 loops, best of 3: 25.3 usec per loop
For the sake of comparison, the original tuple addition is actually between the two:
$ python -m timeit -s "from string import letters" "result = ()" \ "for letter in letters: result += (letter,)" 10000 loops, best of 3: 21.6 usec per loop
A further micro-optimization is to pre-allocate a big list in a thread local. Running the attached script produces:: $ python perftest.py Via insert_0: 24.8401839733 Via append: 15.6596238613 speedup: 37.0% Via tuple_add: 21.9555268288 speedup: 11.6% Via prealloc: 10.5278339386 speedup: 57.6% Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4fIVAACgkQ+gerLs4ltQ6BlACguSHcHLWGjVIVJlQYIpL42RNc mqMAn1Y16tyVmUkxd2ipNrZSeZpSFVEA =oJJY -----END PGP SIGNATURE-----
On Thu, Jul 14, 2011 at 7:03 PM, Tres Seaver <tseaver@palladion.com> wrote:
A further micro-optimization is to pre-allocate a big list in a thread local. Running the attached script produces::
$ python perftest.py Via insert_0: 24.8401839733 Via append: 15.6596238613 speedup: 37.0% Via tuple_add: 21.9555268288 speedup: 11.6% Via prealloc: 10.5278339386 speedup: 57.6%
Thanks for that test! I've run it with a much more reasonable test data set from one of our live databases, for example: letters = ['', 'nordic', 'en', 'nordic-council', 'organisation-and-structure', 'committees', 'welfare-committee', 'meetings-and-meeting-documents', 'meeting-of-the-welfare-committee-31-march-2011-stockholm'] which produces quite a different result: Via insert_0: 4.35216093063 Via append: 4.80827713013 speedup: -10.5% Via tuple_add: 1.73557782173 speedup: 60.1% Via prealloc: 2.17882084846 speedup: 49.9% If I run it with a smaller segment, like: letters = ['', 'nordic', 'en', 'nordic-council'] the effect is even clearer: Via insert_0: 1.88715004921 Via append: 2.91810202599 speedup: -54.6% Via tuple_add: 0.809303998947 speedup: 57.1% Via prealloc: 1.56584882736 speedup: 17.0% Looks like the tuple add is faster until you hit very long sequences. But even if you had such a nested ZODB structure, the majority of lookups would still happen for the much shorter sequences. So I think the tuple add is the winner for this case. Hanno
participants (2)
-
Hanno Schlichting -
Tres Seaver