[Zope-dev] ZPT Optimization Opportunity

Sun Sep 26 04:01:31 EDT 2004

Tres Seaver wrote:

> After seeing Chris McDonough's excellent paper at the Plone conference 
> on optimizing content delivery using the new IStreamIterator iterface, I 
> began experimenting with implementing such an optimization in 
> ZopePageTemplates.
> 
> I played around with having 'ZopePageTemplate._exec' request that it 
> receive an iterator, instead of the usual big string.  I hoped that such 
> a change might enable greater concurrency and memory footprint, by 
> avoiding creation of the big string at all;  instead,  medusa to push 
> out the "chunk stream" represented by the StringIO buflist, while the 
> appserver would be free to handle a new request without needing to 
> malloc / copy the data.
> 
> Here are the timings I have seen so far, using 'zopectl debug', with the 
> following template:
> 
> --------------- Template source --------------------------------
> <html>
> <body>
> <div tal:repeat="item python:[x for x in range(1000)]"
>      tal:content="item">ITEM</div>
> </body>
> </html>
> ----------------------------------------------------------------
> 
> --------------- Before the patch -------------------------------
> 
>>>> Zope.debug('/test_iter', t=1)
> 
> 250.2 milliseconds
> 
>>>> Zope.debug('/test_iter', t=1)
> 
> 106.7 milliseconds
> 
>>>> Zope.debug('/test_iter', t=1)
> 
> 106.5 milliseconds
> 
>>>> Zope.debug('/test_iter', t=1)
> 
> 124.6 milliseconds
> ----------------------------------------------------------------
> 
> --------------- After the patch --------------------------------
>  >>> Zope.debug('/test_iter', t=1)
> 249.2 milliseconds
>  >>> Zope.debug('/test_iter', t=1)
> 107.2 milliseconds
>  >>> Zope.debug('/test_iter', t=1)
> 125.0 milliseconds
>  >>> Zope.debug('/test_iter', t=1)
> 162.1 milliseconds
> ----------------------------------------------------------------
> 
> Given that the performance looks similar in this context, which doesn't 
> benefit from the medusa / concurrency intent of the patch, it seems as 
> though it might be a win (of *course* there aren't any tests for it!)
> 
> I am attaching the patch I have so far for review and comment.
> 
> Tres.
> 
> 
> ------------------------------------------------------------------------
> 
>     This patch causes ZopePageTemplates to return an IStreamIterator
>     when called (published), potentially allowing medusa to return the
>     same response payload (via the iterator) without needing to have the
>     appserver thread join it into a big string.
> 
> --- PageTemplate.py	2004-09-25 21:39:50.595938847 -0400
> +++ PageTemplate.py.new	2004-09-25 21:39:31.983326567 -0400
> @@ -83,7 +84,7 @@
>              c['root'] = self
>          return c
>  
> -    def pt_render(self, source=0, extra_context={}):
> +    def pt_render(self, source=0, extra_context={}, iter_handler=None):
>          """Render this Page Template"""
>          if not self._v_cooked:
>              self._cook()
> @@ -100,7 +101,10 @@
>                         getEngine().getContext(c),
>                         output,
>                         tal=not source, strictinsert=0)()
> -        return output.getvalue()
> +        if iter_handler is not None:
> +            return iter_handler(output)
> +        else:
> +            return output.getvalue()
>  
>      def __call__(self, *args, **kwargs):
>          if not kwargs.has_key('args'):
> --- ZopePageTemplate.py	2004-09-25 21:37:07.956803011 -0400
> +++ ZopePageTemplate.py.new	2004-09-25 21:37:23.024870003 -0400
> @@ -34,6 +34,8 @@
>  from OFS.Cache import Cacheable
>  from OFS.Traversable import Traversable
>  from OFS.PropertyManager import PropertyManager
> +from ZPublisher.Iterators import IStreamIterator
> +
>  from PageTemplate import PageTemplate
>  from Expressions import SecureModuleImporter
>  from PageTemplateFile import PageTemplateFile
> @@ -253,7 +255,8 @@
>          # Execute the template in a new security context.
>          security.addContext(self)
>          try:
> -            result = self.pt_render(extra_context=bound_names)
> +            result = self.pt_render(extra_context=bound_names,
> +                                    iter_handler=StringIOIterator)
>              if keyset is not None:
>                  # Store the result in the cache.
>                  self.ZCacheable_set(result, keywords=keyset)
> @@ -331,6 +334,26 @@
>  setattr(ZopePageTemplate, 'source.xml',  ZopePageTemplate.source_dot_xml)
>  setattr(ZopePageTemplate, 'source.html', ZopePageTemplate.source_dot_xml)
>  
> +
> +class StringIOIterator:
> +    """ Adapt a StringIO object to IStreamIterator.
> +    """
> +
> +    __implements__ = (IStreamIterator,)
> +
> +    def __init__(self, stringio):
> +        self._buflist = stringio.buflist
> +        self._index = 0
> +
> +    def next(self):
> +
> +        if self._index >= len(self._buflist):
> +            raise StopIteration
> +
> +        data, self._index = self._buflist[self._index], self._index + 1
> +
> +        return data
> +
>  # Product registration and Add support
>  manage_addPageTemplateForm = PageTemplateFile(
>      'www/ptAdd', globals(), __name__='manage_addPageTemplateForm')
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope )