[Zope] keeping a python object alive across pageviews

[Kirk Lowery]

We are integrating a postgresql linguistic database (text+metadata)
with Zope. The database uses a search engine (http://emdros.org) which
has a data model optimized to this task. emdros is written in C++ and
we used SWIG (http://www.swig.org) to create python bindings.

We're using page templates and external methods that call the emdros
library to (1) establish a database connection, (2) submit the query
to postgresql, (3) display the results by calling a response template.
And we have run into an implementation question:

Whenever an external method is used to call the search engine, the
search engine then makes a database connection. Zope database adapters
cannot be used here, since it is emdros -- not Zope -- that is
managing the database. When that external method completes, the
connection is lost. The more frequently that method is used, the
greater the performance hit because of the overhead of restablishing
the connection.

Zope database adapters could solve this problem, but that would mean
reimplementing the entire search engine in python as a Zope product,
which doesn't seem to be a practical solution.

When the external method is called, it creates a python object which
houses the C++ connection object. The problem is that this Python
object -- and hence the connection -- dies when the external method
returns. So the question becomes: How do we keep a python object
alive within the Zope process across pageviews?  How do we associate
such a python object with a particular user session?

Our strategy at this time is to call an external method which does not
return until it receives some sort of signal; rather it "busy-waits,"
thereby keeping the emdros connection object alive.

Our question: Do we understand the issues correctly? Are there any
other alternatives for keeping that external Python object alive that
we should consider?

All comments or suggestions are appreciated.

TIA!

Kirk Lowery
<klowery at wts.edu>

Ulrik Petersen
<http://emdros.org/contact.html>

--
Theorie ist, wenn man alles weiss und nichts klappt.
Praxis ist, wenn alles klappt und keiner weiss warum.
Bei uns sind Theorie und Praxis vereint:
nichts klappt und keiner weiss warum!