still segfaults, this time no ZMySQLDA
Hi, It's me and my instability problems again :-) The system: Zope-2.4.4b1 (binary linux-x86 from zope.org) Python-2.1.2 (see above) Products (excluding Zope defaults and ZClass based ones): LocalFS Pagina0_Base (Hiperlógica product) Pagina1_Base (Hiperlógica product) Renderable exUserFolder It has an 800+MiB Data.fs (when packed) that is crammed full of python scripts. I'm still looking for a way to recompile them all without sending the machine (1GiB physical RAM) into a swap storm which would ultimately crash Zope or the machine and keep it from commiting the transaction that caused all the trouble in the first place (one of these days I'll refactor the subtransaction code out of the ZCatalog and into a ZopeFind that will accept an action parameter, along with a subtransaction one, to perform these kinds of global maintenance in a reasonable way). None of the products mentioned above use C compiled modules. I disabled all the use of mysql on this instance so that I could remove ZMySQLDA and faked the authentication system so that it's usage pattern remained the same even without mysql. After that I reenabled the cycle-gc. The system started crashing about every 5 minutes. As much as I'd have liked pointing a finger at ZMySQLDA, start the process of migration to PostgreSQL and consider this problem solved once and for all, I have finally come to the conclusion that there is no one else to blame but Zope and Python themselves. I have since reenabled ZMySQLDA use by the site and disabled the cycle-gc. The options I think I have left: * Try the new ExtensionClass.h * Run Zope under a memory analiser such as Valgrind (which might not be feasible if the runtimes are too slow): http://devel-home.kde.org/~sewardj/ * Try migrating to Zope-2.5.1b1 The situation is starting to look desperate. We now have to restart the server twice a day to keep the machine from swapping. This can't go on any longer. Any suggestion is apreciated. -- Ideas don't stay in some minds very long because they don't like solitary confinement.
On Tue, 2002-02-26 at 20:11, Leonardo Rochael Almeida wrote:
[...] The options I think I have left: * Try the new ExtensionClass.h
Which one, by the way? the one in HEAD, the one in 2.5 branch or a different one altogether? Does it have any impact in the objects in the ZODB? Can I run a ZEO Client with it against a ZEO Server with a different ExtensionClass.h?
* Run Zope under a memory analiser such as Valgrind (which might not be feasible if the runtimes are too slow): http://devel-home.kde.org/~sewardj/
Scratch that. Valgrind doesn't do threads. Anyone knows any alternatives? -- Ideas don't stay in some minds very long because they don't like solitary confinement.
in lib/python/Products/PythonScripts/PythonScripts.py, try changing Script_magic to be 4, and see if that helps. FYI, some of this is written up in http://www.zope.org/Members/matt/StabilityHOWTO On Tuesday, February 26, 2002, at 07:15 PM, Leonardo Rochael Almeida wrote:
On Tue, 2002-02-26 at 20:11, Leonardo Rochael Almeida wrote:
[...] The options I think I have left: * Try the new ExtensionClass.h
Which one, by the way? the one in HEAD, the one in 2.5 branch or a different one altogether? Does it have any impact in the objects in the ZODB? Can I run a ZEO Client with it against a ZEO Server with a different ExtensionClass.h?
* Run Zope under a memory analiser such as Valgrind (which might not be feasible if the runtimes are too slow): http://devel-home.kde.org/~sewardj/
Scratch that. Valgrind doesn't do threads. Anyone knows any alternatives?
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
On Tue, 2002-02-26 at 21:55, Matthew T. Kromer wrote:
in lib/python/Products/PythonScripts/PythonScripts.py, try changing Script_magic to be 4, and see if that helps.
Nope, time between restart after the change and 1st crash was aprox. 3m15s.
FYI, some of this is written up in
hmm, I didn't know that page. It mentions trying with the -t 1. In this case, what is the real total number of threads? is it one for Medusa and one for handling Zope requests or is it just one overall? In this case I might be able to run it under Valgrind after all. Should I bother trying a different ExtensionClass.h? Cheers, Leo -- Ideas don't stay in some minds very long because they don't like solitary confinement.
Leonardo Rochael Almeida wrote:
On Tue, 2002-02-26 at 21:55, Matthew T. Kromer wrote:
in lib/python/Products/PythonScripts/PythonScripts.py, try changing Script_magic to be 4, and see if that helps.
Nope, time between restart after the change and 1st crash was aprox. 3m15s.
Rats! I was hoping that would flush things out, but wasn't expecting it to work.
FYI, some of this is written up in
hmm, I didn't know that page. It mentions trying with the -t 1. In this case, what is the real total number of threads? is it one for Medusa and one for handling Zope requests or is it just one overall? In this case I might be able to run it under Valgrind after all.
Should I bother trying a different ExtensionClass.h?
Cheers, Leo
With -t 1, you're still running with more than one real thread, but only 1 worker thread is processing Zope requests. Most of the time when there is a threading issue, the worker Zope threads are the ones that do the potentially thread-unsafe work, so running with -t 1 will prevent a thread switch from occurring while an exposure could take place. You don't need to bother with the different ExtensionClass.h; the magic in my special version is NOT necessary to apply, and should not make your system any more stable. -- Matt Kromer Zope Corporation http://www.zope.com/
So, what should I try now? On Wed, 2002-02-27 at 16:18, Matthew T. Kromer wrote:
Leonardo Rochael Almeida wrote:
On Tue, 2002-02-26 at 21:55, Matthew T. Kromer wrote:
in lib/python/Products/PythonScripts/PythonScripts.py, try changing Script_magic to be 4, and see if that helps.
Nope, time between restart after the change and 1st crash was aprox. 3m15s.
Rats! I was hoping that would flush things out, but wasn't expecting it to work.
FYI, some of this is written up in
hmm, I didn't know that page. It mentions trying with the -t 1. In this case, what is the real total number of threads? is it one for Medusa and one for handling Zope requests or is it just one overall? In this case I might be able to run it under Valgrind after all.
Should I bother trying a different ExtensionClass.h?
Cheers, Leo
With -t 1, you're still running with more than one real thread, but only 1 worker thread is processing Zope requests. Most of the time when there is a threading issue, the worker Zope threads are the ones that do the potentially thread-unsafe work, so running with -t 1 will prevent a thread switch from occurring while an exposure could take place.
You don't need to bother with the different ExtensionClass.h; the magic in my special version is NOT necessary to apply, and should not make your system any more stable.
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
On Thu, 2002-02-28 at 10:55, Matthew T. Kromer wrote:
On Wednesday, February 27, 2002, at 06:49 PM, Leonardo Rochael Almeida wrote:
So, what should I try now?
Did -t 1 help? If not, do you feel comfortable hooking up gdb and getting a traceback?
Funny enough, running with -t 1 solved it. /me starts looking sideways to ZMySQLDA again, since it's not the new beta version... The performance, understandably, is not the greatest, but it's tolerable (where tolerable means the client hasn't complained yet). I'll start looking at ways to recompile my python scripts using another ZEO Client. I think I'll catalog all of them (using a very small subtransaction threshould) and use a "REQUEST.write()" in a PyScript to list them. -- Ideas don't stay in some minds very long because they don't like solitary confinement.
On Thursday, February 28, 2002, at 02:09 PM, Leonardo Rochael Almeida wrote:
Funny enough, running with -t 1 solved it.
/me starts looking sideways to ZMySQLDA again, since it's not the new beta version...
Good! Based on the fact that I'm not being innundated with "I'm still crashing!" complaints, I'm going to *guess* that -t 1 isolated the bad module, which was probably being activated even though you didn't think it was. Maybe something traversed to it, or activated it in some other way. I don't think there is a thread-sensitive gremlin in Zope 2.5, but I'm asking all parties to keep an eye open for a vanilla 2.5 (or 2.4.4) crash that -t 1 solves.
On Thu, 2002-02-28 at 16:09, Leonardo Rochael Almeida wrote:
On Thu, 2002-02-28 at 10:55, Matthew T. Kromer wrote:
On Wednesday, February 27, 2002, at 06:49 PM, Leonardo Rochael Almeida wrote:
So, what should I try now?
Did -t 1 help? If not, do you feel comfortable hooking up gdb and getting a traceback?
Funny enough, running with -t 1 solved it.
/me starts looking sideways to ZMySQLDA again, since it's not the new beta version...
The performance, understandably, is not the greatest, but it's tolerable (where tolerable means the client hasn't complained yet).
I'll start looking at ways to recompile my python scripts using another ZEO Client. I think I'll catalog all of them (using a very small subtransaction threshould) and use a "REQUEST.write()" in a PyScript to list them.
COOL!! I found a kickass way to recompile all PythonScripts without bringing down the machine and writing the least amount of code ever possible! Short version: Create a python script in Zope root that recompiles "context" and catalog all python scripts using that as a metadata. Long version: put the following python script in your root: ============= ## Script (Python) "recompile" ##bind container=container ##bind context=context ##bind namespace= ##bind script=script ##bind subpath=traverse_subpath ##parameters= ##title= ## # recompile the thing context.write(context.read()) # return something to waste catalog space :-) from string import join return "recompiled %s\n" % join(context.getPhysicalPath(),"/") ============= Now put a ZCatalog in your root, call it anything, say, pyScripts. Go to the "Indexex" tab and remove all indexes but the "path" index. Go back and go to the "Metadata" tab; remove all metadata and insert "recompile" and getPhysicalPath (this one is completely optional). Go to the Advanced tab and set the subtransaction threshould to a number that looks reasonable to you (low enough so as not to crash your machine :-). Go to the "Find Objects" tab, make sure that only "Script (Python)" is selected (verify, especially, that "All types" isn't selected) and click "Find and Catalog". wait... wait... wait some more... go get a coke or a coffee or something... wait a little more... When you see the result page, you know that your pythonScripts are recompiled (of course it won't take that long unless you have an 800+MiB Data.fs crammed full of python scripts :-) Cheers, Leo -- Ideas don't stay in some minds very long because they don't like solitary confinement.
On Thursday 28 February 2002 02:51 pm, Leonardo Rochael Almeida allegedly wrote: [snip]
When you see the result page, you know that your pythonScripts are recompiled (of course it won't take that long unless you have an 800+MiB Data.fs crammed full of python scripts :-)
Cheers, Leo
I hope those weren't all coded in a textarea 8*). Ouch. /---------------------------------------------------\ Casey Duncan, Sr. Web Developer National Legal Aid and Defender Association c.duncan@nlada.org \---------------------------------------------------/
Leonardo Rochael Almeida writes:
... I'm still looking for a way to recompile them all without sending the machine (1GiB physical RAM) into a swap storm which would ultimately crash Zope or the machine and keep it from commiting the transaction that caused all the trouble in the first place (one of these days I'll refactor the subtransaction code out of the ZCatalog and into a ZopeFind that will accept an action parameter, along with a subtransaction one, to perform these kinds of global maintenance in a reasonable way). You may commit the transaction after several recompilations (as you suggest). You can see how to do this in e.g. "emulateRedirect" at
participants (4)
-
Casey Duncan -
Dieter Maurer -
Leonardo Rochael Almeida -
Matthew T. Kromer