Speed win in Python's urllib.quote
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 While profiling a page full of dynamically-generated links, I was interested to see Python's 'urllib._fast_quote' show up quite high in the total time list. After looking at it, I have found a pretty huge speed win available, posted to Python's SF tracker as bug #1285086: https://sourceforge.net/tracker/index.php?func=detail&aid=1285086&group_id=5... Zope makes *heavy* use of urllib.quote (quoting each element of the path in 'absolute_url', for instance), which makes it a particularly interesting speedup for us. I'm attaching the speed test and the patch, for consideration by the Zope development community: if this is enough of a win, we might consider including a patched urllib in Zope2/Zope3 (or monkey-patching urllib.quote). Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDIG9X+gerLs4ltQ4RArBaAJwIWac5xvHjmQg2S3oXXIFWzrecjACghGuw 5NnQQGNw55JDWjrzRyCEitA= =iIpp -----END PGP SIGNATURE-----
Tres Seaver wrote:
Zope makes *heavy* use of urllib.quote (quoting each element of the path in 'absolute_url', for instance), which makes it a particularly interesting speedup for us.
Indeed, what would be the reasons for NOT including it? How should we go about measuring the speed difference this makes?
I'm attaching the speed test and the patch, for consideration by the Zope development community: if this is enough of a win, we might consider including a patched urllib in Zope2/Zope3 (or monkey-patching urllib.quote).
FWIW, I'd favour a monkey patch of ulrlib.quote, pending inclusion of the patch into the Python source tree, which is something we should all be fighting for ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On 9/12/05, Chris Withers <chris@simplistix.co.uk> wrote:
Indeed, what would be the reasons for NOT including it? How should we go about measuring the speed difference this makes?
How about: http://public-bertha.in.nuxeo.com/~ben/funkload/ I've seen it demoed, but never used it. It seems very cool. It can do functional load testing and should be able to make a reasonable test of the speed improvement.
Lennart Regebro wrote:
Is this site ment to be public? The name doesn't resolve for me. Is the head from http://svn.nuxeo.org/trac/pub/browser/funkload/ sufficient? Martijn Pieters
Martijn Pieters wrote:
Is this site ment to be public? The name doesn't resolve for me. Is the head from http://svn.nuxeo.org/trac/pub/browser/funkload/ sufficient?
Reply to self: in the zope3 list the answer to the above question has been found; it was "Yes". Martijn Pieters
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Actually, this is public : http://public.dev.nuxeo.com/~ben/funkload/ :) J. Martijn Pieters wrote:
Martijn Pieters wrote:
Is this site ment to be public? The name doesn't resolve for me. Is the head from http://svn.nuxeo.org/trac/pub/browser/funkload/ sufficient?
Reply to self: in the zope3 list the answer to the above question has been found; it was "Yes".
Martijn Pieters
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
- -- Julien Anguenot | Nuxeo R&D (Paris, France) CPS Platform : http://www.cps-project.org Zope3 / ECM : http://www.z3lab.org mail: anguenot at nuxeo.com; tel: +33 (0) 6 72 57 57 66 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFDJqEDGhoG8MxZ/pIRAgCfAJ9yNblBWJ9s9iTYHWGqsObcXZHP1gCfR2Xh /nLeoaYhJazqtKi2ELStO04= =Wod1 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tres Seaver wrote:
While profiling a page full of dynamically-generated links, I was interested to see Python's 'urllib._fast_quote' show up quite high in the total time list. After looking at it, I have found a pretty huge speed win available, posted to Python's SF tracker as bug #1285086:
https://sourceforge.net/tracker/index.php?func=detail&aid=1285086&group_id=5...
Zope makes *heavy* use of urllib.quote (quoting each element of the path in 'absolute_url', for instance), which makes it a particularly interesting speedup for us.
On further consideration (and after some more profiling), I would actually prefer inlining the speedup code directly into OFS.Traversable.Traversable, to remove the extra function call as well as to avoid the gnarliness of monkey-patching Python. An alternative would be to call 'quote' only once (for the completed string), rather than repeatedly (for each path element). The SF traffic for the bug is interesing: the fix checked in is an improvement, but not nearly as big a win for the case (very common in English-language Zope sites, anyway) where the path elements are already URL safe. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDJXob+gerLs4ltQ4RAlgbAKCEP2yIxCetpiY9SW2+BDr5PJFfTACgoS9H c8X+2m06eCpNhZTcKpsunv0= =u0g7 -----END PGP SIGNATURE-----
(zope3-dev trimmed) Hi Tres, Tres Seaver wrote:
On further consideration (and after some more profiling), I would actually prefer inlining the speedup code directly into OFS.Traversable.Traversable, to remove the extra function call as well as to avoid the gnarliness of monkey-patching Python.
Fair enough :-)
An alternative would be to call 'quote' only once (for the completed string), rather than repeatedly (for each path element).
Wouldn't that then encode all the /'es? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Withers wrote:
Tres Seaver wrote:
On further consideration (and after some more profiling), I would actually prefer inlining the speedup code directly into OFS.Traversable.Traversable, to remove the extra function call as well as to avoid the gnarliness of monkey-patching Python.
Fair enough :-)
An alternative would be to call 'quote' only once (for the completed string), rather than repeatedly (for each path element).
Wouldn't that then encode all the /'es?
No -- urllib.quote takes a 'safe' argument, which defaults to '/', as an extension to the 'always_safe' set:: $ python2.3 Python 2.3.5 (#2, Aug 30 2005, 13:43:24) [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
from urllib import quote quote('/abc/def/ghi') '/abc/def/ghi'
Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDJqPx+gerLs4ltQ4RAqI4AKDEP6eiq/IzrYISuYM1AbJYAWQvWwCfXCYE 7AXqhMubzcfedZwO0Gr77x8= =o9gI -----END PGP SIGNATURE-----
participants (5)
-
Chris Withers -
Julien Anguenot -
Lennart Regebro -
Martijn Pieters -
Tres Seaver