[Zope-dev] (ZPatterns) Speeding up Skinscripts

[Phillip J. Eby]

At 01:57 PM 3/29/01 -0800, John Eikenberry wrote:
>
>We have a fairly large and complex app framework built on ZPatterns. It
>uses MySQL for storage and the standard Specialist/Rack/DataSkin setup with
>skinscripts for attributes and triggers.
>
>We've found that the speed of getItem is a bit slower than we need. For
>instance retrieving 200 dataskins takes about 8 seconds on a P2-300. After
>profiling and digging around I think I've found the primary bottleneck.
Its the
>running of eval() on the skinscript's python expression (stored in the
Compute
>class as _fromex and Triggers as callexpr).

Congratulations, you're the first person (that I know of, anyway) to hit a
wall here.  However, I think you may be barking up the wrong tree on your
profiling.  See below.  

>The optimization I've been looking at is changing the code from storing a
>string and eval()ing the string to using compile() at save time and exec()
when
>evaluated.
>
>Profiling these 2 ways in little test programs seems to indicate about a 2.5x
>speedup. Not huge, but combined with better hardware should be enough.
>
>But I'm curious why this avenue wasn't taken to begin with. Seems like the
way
>to do it to me. Am I missing something? 

Yes, as your later message mentions, ZPatterns already does that caching.
I think what's more likely to be taking up time is either:

1) Zope security checks on the expressions, or
2) DTML execution of your SQL queries

Or perhaps something else altogether.  First a few comments on the above,
then I'll ask some more probing questions regarding your setup.

SkinScript expressions execute using Zope security checking.  If you're
using a "WITH QUERY SQLmethod() COMPUTE attributelist" statement, security
checks are applied to both the query expression and to every name
referenced in 'attributelist'.  So the longer the attribute list, the
slower the execution.  Although, now that I think of it, eval() is not used
if you use a pure attribute name list like "name1,name2,name3", so the
problem probably isn't on the attribute list side.  This makes it much less
likely that the security machinery is to blame (unless you have a very slow
user folder which also doesn't cache well, and you haven't given your
SkinScript proxy roles to shortcut the security checks.)


Now, for your SQL query...  is it by any chance dynamically generated in a
complex way?  I mean, more complex than an sqlvar or two?  Keep in mind
that when Zope caches SQL queries it does it by generating the text of the
query and using it as a cache key.  That means any DTML in your query will
be executed each time.  It'd have to be pretty complex, or else there'd
need to be lots of slow security lookups, to get the kind of poor
performance you're seeing, though.

Okay.  Here are a few things to look at in your app:

1. If you're not using LoginManager as your user folder, and maybe even if
you are, your SkinScripts should have proxy roles.  Proxy roles can slash
the overhead of performing security checks, and are usually appropriate for
SkinScripts because SkinScript is "internal" to the object and Zope
security will apply to the results of the SkinScripts as well.

2. Why are you retrieving 200 dataskins?  Is this all in one transaction?
If so, you may have a design issue.  Multi-object operations should
generally be implemented as domain-specific methods on the Specialist, so
they can be implementation-specific (and thereby take advantage of speedups
like a ZCatalog index or specialized SQL queries).  In other words, you
probably want to iterate directly over a 200-row query result from an
SQLMethod, rather than retrieiving 200 DataSkins.  (You're accessing over
10 times as many DataSkins in one transaction as our most complex ZPatterns
application touches in its most complex type of transaction, which involves
about 4 or 5 Specialists at once!)  Pretty much, if you're displaying a
list or report or doing some kind of summary analysis or mass update, it
belongs in an implementation-specific method in the Specialist.

3. If you *really* need 200 dataskins in one transaction, and the overhead
is *really* in SkinScript, then you can bypass the overhead by writing an
AttributeProvider of your own in Python.  I'm guessing, however, that
SkinScript per se is not the source of the slowdown, and you need to look
more closely at the things the SkinScript is calling.