Hi. Just wanted to post this for whoever searches for Zope, Threads and ZSQLMethods or relational databases in the future. It seems that very simplistic things will work if you spawn a thread inside of Zope. But in the thread if you start doing anything really related to ZODB you could have all sorts of issues. This is my interpretation of events. I could be wrong. I have started smoking again since the beginning of this fiasco. The situation: - need to run a thread that does lots of ZSQLMethod calls and then generates a Report - we would launch a Report thread in Zope - but how it interacted with the database connection was very strange. Connections that were launched with the thread would be reused and would get really bizarre problems. like inserts not showing up, etc. - I spoke with a few gurus and they say that spawning threads in zope works fine (albeit a PITA). but none were doing any ZSQL acrobatics. - postgres connections seemed to leak as would ZODB connections. Eventually I would run out of ZODB threads (which ZServer uses) and ZServer would stop responding because a thread would be left holding the last zodb connection which would have been leaked. - app would lock up. The solution: - use ZEO! (I was already using it) and both kapil and stevea suggested it. But I was DETERMINED to get the threading solution to work (no where did it say it WOULD NOT WORK, now I am writing that it appears NOT TO WORK) - now the ZEO client that takes the report requests simply generate a ReportRequest class, pickles it, put it in a filesystem folder and goes on its marry way. - the report zeo client [separte process] watches the filesystem and picks up pickles, unpickles, generates the report and removes the pickle. so far it works beautifully! I plan to post on zopelabs the threading solution with a huge warning not to use it in conjunction with ZSQLMethods. but I think it should be (actively) discouraged spawning threads inside of ZEO. ~runyaga
alan runyan wrote:
Hi.
Just wanted to post this for whoever searches for Zope, Threads and ZSQLMethods or relational databases in the future. It seems that very simplistic things will work if you spawn a thread inside of Zope. But in the thread if you start doing anything really related to ZODB you could have all sorts of issues.
This is my interpretation of events. I could be wrong. I have started smoking again since the beginning of this fiasco.
The situation:
- need to run a thread that does lots of ZSQLMethod calls and then generates a Report
- we would launch a Report thread in Zope
- but how it interacted with the database connection was very strange. Connections that were launched with the thread would be reused and would get really bizarre problems. like inserts not showing up, etc.
Just out of curiosity. How did you handle commits in the Report thread? Was this problems always related to using ZSQLMethod or did the problem also appear when only the ZODB was used, or have you not tried pure ZODB?
so far it works beautifully! I plan to post on zopelabs the threading solution with a huge warning not to use it in conjunction with ZSQLMethods. but I think it should be (actively) discouraged spawning threads inside of ZEO.
Does these problems only appear in ZEO or would they appear in a single mounted database Zope as well? Johan -- Johan Carlsson Tel: + 46 8 31 24 94 Colliberty Mob: + 46 70 558 25 24 Torsgatan 72 Email: johanc@easypublisher.com SE-113 37 STOCKHOLM
<snip />
Just out of curiosity. How did you handle commits in the Report thread?
the Report Thread was only a read-only thread it did not attempt to commit.
Was this problems always related to using ZSQLMethod or did the problem also appear when only the ZODB was used, or have you not tried pure ZODB?
did not try pure ZODB - but I have a feeling that would work. life is good as long as you dont try to WRITE in the thread because the transaction_manager blocks writes to the ZODB unless the thread is under its control.
so far it works beautifully! I plan to post on zopelabs the threading solution with a huge warning not to use it in conjunction with ZSQLMethods. but I think it should be (actively) discouraged spawning threads inside of ZEO.
Does these problems only appear in ZEO or would they appear in a single mounted database Zope as well?
I would imagine it would be problems in any Zope. I dont see how it would be specific to ClientStorage. ~runyaga
alan runyan wrote at 2003-7-9 13:06 -0500:
Just wanted to post this for whoever searches for Zope, Threads and ZSQLMethods or relational databases in the future. It seems that very simplistic things will work if you spawn a thread inside of Zope. But in the thread if you start doing anything really related to ZODB you could have all sorts of issues.
The mailing list archives have some threads about this... There are a few issues you must observe: In general you must not share a ZODB connection between threads. This means, you must not pass persistent objects between threads (as they contain a hidden reference to a ZODB connection (_p_jar) and may use it non-deterministically). Open a new ZODB connection in the new thread. Its "root()" method returns the ZODB root object. Pass an object's path between threads rather than the object itself. Use "[un]restrictedTraverse" to locate the object from the root given the path. Do not forget to commit/abort the (implicitly) started transaction. Do not forget to close the opened ZODB connection Dieter
The mailing list archives have some threads about this...
There are a few issues you must observe:
In general you must not share a ZODB connection between threads. This means, you must not pass persistent objects between threads (as they contain a hidden reference to a ZODB connection (_p_jar) and may use it non-deterministically).
Open a new ZODB connection in the new thread. Its "root()" method returns the ZODB root object.
Pass an object's path between threads rather than the object itself. Use "[un]restrictedTraverse" to locate the object from the root given the path.
Do not forget to commit/abort the (implicitly) started transaction.
did not do this (as I was selecting only from ZSQLMethods) - this could have very well given us the odditiy of people adding data and it appearing to show up and then not committing.
Do not forget to close the opened ZODB connection
very good summary dieter. thanks ~runyaga
participants (3)
-
alan runyan -
Dieter Maurer -
Johan Carlsson