[Zope-CVS] CVS: Packages/cZPT - README.txt:1.1

[Shane Hathaway]

Update of /cvs-repository/Packages/cZPT
In directory cvs.zope.org:/tmp/cvs-serv23309

Added Files:
	README.txt 
Log Message:
Optimized TALInterpreter and restrictedTraverse() in C.

This code is just educational for now.  Before doing much more, we should
write performance tests and try out Pyrex.


=== Added File Packages/cZPT/README.txt ===

Shane Hathaway
shane@zope.com

This code (TAL/cTALInterpreter.c and OFS/cTraversable.c) is an
experiment in Python to C optimization.  What I wanted to learn:

- how to convert Python code to C while handling memory management and
exceptions correctly.  It's very hard to match the safety of the Python
interpreter.

- whether there is a benefit in optimizing TALInterpreter and
restrictedTraverse().

- whether there are unforeseen problems.

- whether there is a better, more maintainable way to do this.


What I learned:

- To write maintainable and safe C code from Python, you have to
follow some conventions.  These are the conventions I discovered to be
the most beneficial:

  - Every variable of type "PyObject*" must be either owned or
  borrowed at all times.  A variable should not be owned in one part
  of the function and borrowed in other parts.  Violating this rule
  often leads to memory management problems.  A reasonable exception
  is that a function argument may be turned into an owned reference
  early in the function.

  - Functions with any complexity should use "goto" to handle
  exceptions.  Combined with the two macros CHECKPTR and CHECKINT,
  this makes exception handling much simpler.

  - Python variables usually need to be owned references.  When
  converting a Python function, start by declaring an owned
  "PyObject*" for every local Python variable (except for variables
  that will be represented as basic C types).

Even after I followed all these rules, I still made mistakes that led
to segfaults.  I fixed all the bugs I found, but it's still
unsatisfying to know that there may be other memory management bugs
lurking.


- There is indeed a benefit to optimizing this code.  In fact, even
though cTALInterpreter is not fully optimized (it doesn't use the most
efficient Python calls everywhere), the Python profiler seems to show
a significant speedup.  It would be beneficial for someone to write a
real performance test.  There is a simple speed test in OFS/ for
cTraversable.restrictedTraverse() that shows a 2x-3x speed improvement
over the standard restrictedTraverse().


- An unforeseen issue is that error tracebacks involving
TALInterpreter don't show as much information.


A better way?

Pyrex might be a better option for doing this.  I haven't tried it,
but it allows you to write code that's almost Python while integrating
C variables into the source.  I understand it generates the C code,
which you can then compile into an extension.  The resulting extension
has a good chance of being just as safe as the original Python code
and about as fast as an extension written by hand.

Psyco is another option, but it seems likely to me that Psyco would
not generate code that is as fully optimized as what Pyrex generates.
Psyco should be better at general optimization, however.  There is no
reason one should not be able to combine both techniques into the same
program.

So this code, cTALInterpreter and cTraversable, should be considered
merely educational until someone comes on board who can write the
speed tests and try out Pyrex.