[Zope] Zope and Threads

[alan runyan]

Hi.

Just wanted to post this for whoever searches for Zope,
Threads and ZSQLMethods or relational databases in the future.  It seems 
that very simplistic things will work
if you spawn a thread inside of Zope.  But in the thread
if you start doing anything really related to ZODB
you could have all sorts of issues.

This is my interpretation of events.  I could be wrong.
I have started smoking again since the beginning of this
fiasco.

The situation:

  - need to run a thread that does lots of ZSQLMethod calls
  and then generates a Report

  - we would launch a Report thread in Zope

  - but how it interacted with the database connection was
  very strange.  Connections that were launched with the
  thread would be reused and would get really bizarre problems.
  like inserts not showing up, etc.

  - I spoke with a few gurus and they say that spawning threads
  in zope works fine (albeit a PITA).  but none were doing any ZSQL 
acrobatics.

  - postgres connections seemed to leak as would ZODB connections.
  Eventually I would run out of ZODB threads (which ZServer uses)
  and ZServer would stop responding because a thread would be left
  holding the last zodb connection which would have been leaked.

  - app would lock up.

The solution:

  - use ZEO! (I was already using it) and both kapil and stevea
  suggested it.  But I was DETERMINED to get the threading solution
  to work (no where did it say it WOULD NOT WORK, now I am writing that
  it appears NOT TO WORK)

  - now the ZEO client that takes the report requests simply generate
  a ReportRequest class, pickles it, put it in a filesystem folder and
  goes on its marry way.

  - the report zeo client [separte process] watches the filesystem and 
picks up pickles, unpickles, generates the report and removes the pickle.

  so far it works beautifully!  I plan to post on zopelabs the threading
  solution with a huge warning not to use it in conjunction with 
ZSQLMethods.
  but I think it should be (actively) discouraged spawning threads inside 
of
  ZEO.

~runyaga