Re: [Zope-dev] disabling gc does not necessarily hide memory corruption
Chris McDonough wrote This behavior is with the patches to the compiler package that were the genesis of this thread?
No, prior to that. I posted something about it to zope-dev an age or six ago - it seemed like the RestrictedPython compiler had mungo mungo numbers of cycles, and we found we couldn't run a zeo client on a nogc python for more than about an hour. ah - here's the message I sent: Date: Wed, 17 Oct 2001 18:10:27 +1000 To: zope-dev@zope.org From: Anthony Baxter <anthony@interlink.com.au> Subject: RestrictedPython under no-GC 2.1 seems to leak a hell of a lot. Running current Zope 2.4 CVS under python 2.1 built --without-cycle-gc, I'm seeing an absolutely _enormous_ amount of leaked objects in RestrictedPython. This is on one of our ZEO clients, after 45 minutes of runtime. The deltas are after 5 minutes. RestrictedPython.compiler.transformer.Transformer 146527 163546 +17019 RestrictedPython.compiler.visitor.ASTVisitor 55842 62333 +6491 RestrictedPython.compiler.misc.Set 34827 38843 +4016 RestrictedPython.compiler.pyassem.FlowGraph 19173 21413 +2240 RestrictedPython.compiler.ast.Stmt 19174 21390 +2216 RestrictedPython.compiler.pycodegen.CodeGenerator 18333 20486 +2153 RestrictedPython.compiler.symbols.SymbolVisitor 15303 17096 +1793 RestrictedPython.compiler.ast.Name 13917 15542 +1625 RestrictedPython.compiler.ast.Const 14083 15691 +1608 RestrictedPython.compiler.ast.CallFunc 13798 15405 +1607 RestrictedPython.MutatingWalker.MutatingWalker 11968 13360 +1392 RestrictedPython.compiler.ast.Return 11986 13371 +1385 RestrictedPython.compiler.ast.Function 11986 13371 +1385 RestrictedPython.RestrictionMutator.RestrictionMutator 11800 13166 +1366 RestrictedPython.compiler.pycodegen.LocalNameFinder 11028 12322 +1294 RestrictedPython.compiler.pyassem.Block 10226 11401 +1175 RestrictedPython.compiler.pyassem.PyFlowGraph 9587 10707 +1120 RestrictedPython.compiler.misc.Stack 9587 10703 +1116 RestrictedPython.compiler.ast.Module 9590 10698 +1108 RestrictedPython.compiler.ast.Global 5761 6433 +672 RestrictedPython.compiler.ast.Assign 5761 6433 +672 RestrictedPython.compiler.ast.AssName 5761 6433 +672 [....]
Funny thing. My instalation isn't leaking noticeably after I disabled cycle-gc, and I'm running a Zeo client/server combo (in a single machine) On Tue, 2001-12-18 at 01:58, Anthony Baxter wrote:
Chris McDonough wrote This behavior is with the patches to the compiler package that were the genesis of this thread?
No, prior to that. I posted something about it to zope-dev an age or six ago - it seemed like the RestrictedPython compiler had mungo mungo numbers of cycles, and we found we couldn't run a zeo client on a nogc python for more than about an hour.
ah - here's the message I sent:
Date: Wed, 17 Oct 2001 18:10:27 +1000 To: zope-dev@zope.org From: Anthony Baxter <anthony@interlink.com.au> Subject: RestrictedPython under no-GC 2.1 seems to leak a hell of a lot.
Running current Zope 2.4 CVS under python 2.1 built --without-cycle-gc, I'm seeing an absolutely _enormous_ amount of leaked objects in RestrictedPython. This is on one of our ZEO clients, after 45 minutes of runtime. The deltas are after 5 minutes.
RestrictedPython.compiler.transformer.Transformer 146527 163546 +17019 RestrictedPython.compiler.visitor.ASTVisitor 55842 62333 +6491 RestrictedPython.compiler.misc.Set 34827 38843 +4016 RestrictedPython.compiler.pyassem.FlowGraph 19173 21413 +2240 RestrictedPython.compiler.ast.Stmt 19174 21390 +2216 RestrictedPython.compiler.pycodegen.CodeGenerator 18333 20486 +2153 RestrictedPython.compiler.symbols.SymbolVisitor 15303 17096 +1793 RestrictedPython.compiler.ast.Name 13917 15542 +1625 RestrictedPython.compiler.ast.Const 14083 15691 +1608 RestrictedPython.compiler.ast.CallFunc 13798 15405 +1607 RestrictedPython.MutatingWalker.MutatingWalker 11968 13360 +1392 RestrictedPython.compiler.ast.Return 11986 13371 +1385 RestrictedPython.compiler.ast.Function 11986 13371 +1385 RestrictedPython.RestrictionMutator.RestrictionMutator 11800 13166 +1366 RestrictedPython.compiler.pycodegen.LocalNameFinder 11028 12322 +1294 RestrictedPython.compiler.pyassem.Block 10226 11401 +1175 RestrictedPython.compiler.pyassem.PyFlowGraph 9587 10707 +1120 RestrictedPython.compiler.misc.Stack 9587 10703 +1116 RestrictedPython.compiler.ast.Module 9590 10698 +1108 RestrictedPython.compiler.ast.Global 5761 6433 +672 RestrictedPython.compiler.ast.Assign 5761 6433 +672 RestrictedPython.compiler.ast.AssName 5761 6433 +672
[....]
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Ah ok... yeah, there were reportedly several leaks in the compiler stuff that have been fixed in CVS. I imagine that's what this is. Anthony Baxter wrote:
Chris McDonough wrote
This behavior is with the patches to the compiler package that were the genesis of this thread?
No, prior to that. I posted something about it to zope-dev an age or six ago - it seemed like the RestrictedPython compiler had mungo mungo numbers of cycles, and we found we couldn't run a zeo client on a nogc python for more than about an hour.
ah - here's the message I sent:
Date: Wed, 17 Oct 2001 18:10:27 +1000 To: zope-dev@zope.org From: Anthony Baxter <anthony@interlink.com.au> Subject: RestrictedPython under no-GC 2.1 seems to leak a hell of a lot.
Running current Zope 2.4 CVS under python 2.1 built --without-cycle-gc, I'm seeing an absolutely _enormous_ amount of leaked objects in RestrictedPython. This is on one of our ZEO clients, after 45 minutes of runtime. The deltas are after 5 minutes.
RestrictedPython.compiler.transformer.Transformer 146527 163546 +17019 RestrictedPython.compiler.visitor.ASTVisitor 55842 62333 +6491 RestrictedPython.compiler.misc.Set 34827 38843 +4016 RestrictedPython.compiler.pyassem.FlowGraph 19173 21413 +2240 RestrictedPython.compiler.ast.Stmt 19174 21390 +2216 RestrictedPython.compiler.pycodegen.CodeGenerator 18333 20486 +2153 RestrictedPython.compiler.symbols.SymbolVisitor 15303 17096 +1793 RestrictedPython.compiler.ast.Name 13917 15542 +1625 RestrictedPython.compiler.ast.Const 14083 15691 +1608 RestrictedPython.compiler.ast.CallFunc 13798 15405 +1607 RestrictedPython.MutatingWalker.MutatingWalker 11968 13360 +1392 RestrictedPython.compiler.ast.Return 11986 13371 +1385 RestrictedPython.compiler.ast.Function 11986 13371 +1385 RestrictedPython.RestrictionMutator.RestrictionMutator 11800 13166 +1366 RestrictedPython.compiler.pycodegen.LocalNameFinder 11028 12322 +1294 RestrictedPython.compiler.pyassem.Block 10226 11401 +1175 RestrictedPython.compiler.pyassem.PyFlowGraph 9587 10707 +1120 RestrictedPython.compiler.misc.Stack 9587 10703 +1116 RestrictedPython.compiler.ast.Module 9590 10698 +1108 RestrictedPython.compiler.ast.Global 5761 6433 +672 RestrictedPython.compiler.ast.Assign 5761 6433 +672 RestrictedPython.compiler.ast.AssName 5761 6433 +672
[....]
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
On Mon, 17 Dec 2001 23:13:54 -0500, Chris McDonough <chrism@zope.com> wrote:
Ah ok... yeah, there were reportedly several leaks in the compiler stuff that have been fixed in CVS. I imagine that's what this is.
Is the compiler still based on bytecodehacks? If so, I dont think this is a bug As I understand it, that package is designed to leave cyclic garbage around under the assumption that python's garbage collector will break the cycles. Early versions of bytecodehacks broke the cycles manually, at a great performance cost. Toby Dickenson tdickenson@geminidataloggers.com
It is no longer based on bytecodehacks, it uses Jeremy's compiler module. It did produce some (unintended) circrefs, but I think the version in CVS as well as the version that will ship with Python 2.2 has these taken out. Toby Dickenson wrote:
On Mon, 17 Dec 2001 23:13:54 -0500, Chris McDonough <chrism@zope.com> wrote:
Ah ok... yeah, there were reportedly several leaks in the compiler stuff that have been fixed in CVS. I imagine that's what this is.
Is the compiler still based on bytecodehacks? If so, I dont think this is a bug
As I understand it, that package is designed to leave cyclic garbage around under the assumption that python's garbage collector will break the cycles. Early versions of bytecodehacks broke the cycles manually, at a great performance cost.
Toby Dickenson tdickenson@geminidataloggers.com
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
"CM" == Chris McDonough <chrism@zope.com> writes:
CM> Ah ok... yeah, there were reportedly several leaks in the CM> compiler stuff that have been fixed in CVS. I imagine that's CM> what this is. Actually, I don't think the leaks haven't been fixed in CVS. It would be at best painful to try and get the compiler code to work without cycle GC. I really don't want to have to do it. Jeremy
Jeremy Hylton wrote Actually, I don't think the leaks haven't been fixed in CVS. It would be at best painful to try and get the compiler code to work without cycle GC. I really don't want to have to do it.
In that case, there should probably be a readme note or similar saying "if you're using pythonscripts, don't bother without cyclic GC". For me, I run the ZEO server without GC (because that stops the stomped stack bug killing the ZEO server), and the ZEO clients with GC (and just wear the restarts. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
On Wed, 19 Dec 2001 06:07:00 +1100, Anthony Baxter <anthony@interlink.com.au> wrote:
For me, I run the ZEO server without GC (because that stops the stomped stack bug killing the ZEO server)
Do you have reason to believe that this bug can affect the ZEO server? that would be very bad. Toby Dickenson tdickenson@geminidataloggers.com
Oh, ok, sorry for the misreport. It sounds then as if we need to fix whatever is causing the memory corruption that GC trips over. I dont know that we have a real good handle on what this is. Note that in the meantime, folks who turn off gc in order to work around the issue who are having leakage problems should probably consider the use of AutoLance, a product that will restart your Zope if memory consumption is too high: http://www.zope.org/Members/mcdonc/Products/AutoLance . It currently only works on Linux.
Actually, I don't think the leaks haven't been fixed in CVS. It would be at best painful to try and get the compiler code to work without cycle GC. I really don't want to have to do it.
Oh, ok, sorry for the misreport. It sounds then as if we need to fix whatever is causing the memory corruption that GC trips over. I dont know that we have a real good handle on what this is.
I'm pretty sure we do. The stack estimation problems cause memory stomping, which happens to tend to be found much more quickly with GC enabled. Zope 2.5 b3 will contain a fixed compiler package. Brian Lloyd brian@zope.com Software Engineer 540.361.1716 Zope Corporation http://www.zope.com
It would be good if someone who is experiencing random crashes could confirm that the the new compiler package fixes their problem. ----- Original Message ----- From: "Brian Lloyd" <brian.lloyd@zope.com> To: "Chris McDonough" <chrism@zope.com>; <jeremy@zope.com> Cc: "Anthony Baxter" <anthony@interlink.com.au>; "Zope Developers list" <zope-dev@zope.org> Sent: Wednesday, December 19, 2001 9:35 AM Subject: RE: [Zope-dev] disabling gc does not necessarily hide memory corruption
Oh, ok, sorry for the misreport. It sounds then as if we need to fix whatever is causing the memory corruption that GC trips over. I dont know that we have a real good handle on what this is.
I'm pretty sure we do. The stack estimation problems cause memory stomping, which happens to tend to be found much more quickly with GC enabled.
Zope 2.5 b3 will contain a fixed compiler package.
Brian Lloyd brian@zope.com Software Engineer 540.361.1716 Zope Corporation http://www.zope.com
On Wed, 2001-12-19 at 13:34, Chris McDonough wrote:
It would be good if someone who is experiencing random crashes could confirm that the the new compiler package fixes their problem.
I might risk losing a few more points with my sysadmin to test it (Hi daniduc :-) if there was an easily downloadable package with clear instructions on how to apply it to Zope 2.4.3 (I don't feel like messing with CVS for this), and if you tell me that it won't corrupt my Data.fs or anything :-) Cheers, Leo -- Ideas don't stay in some minds very long because they don't like solitary confinement.
On Wed, Dec 19, 2001 at 05:24:58PM -0200, Leonardo Rochael Almeida wrote:
On Wed, 2001-12-19 at 13:34, Chris McDonough wrote:
It would be good if someone who is experiencing random crashes could confirm that the the new compiler package fixes their problem.
I might risk losing a few more points with my sysadmin to test it (Hi daniduc :-) if there was an easily downloadable package with clear instructions on how to apply it to Zope 2.4.3 (I don't feel like messing with CVS for this), and if you tell me that it won't corrupt my Data.fs or anything :-)
You can download files from CVS as a tarball. Just go to http://cvs.zope.org/Zope/lib/python/RestrictedPython and use the link at the bottom. -- Martijn Pieters | Software Engineer mailto:mj@zope.com | Zope Corporation http://www.zope.com/ | Creators of Zope http://www.zope.org/ ---------------------------------------------
On Wed, 2001-12-19 at 18:02, Martijn Pieters wrote:
[...] You can download files from CVS as a tarball. Just go to http://cvs.zope.org/Zope/lib/python/RestrictedPython and use the link at the bottom.
Should I apply the ExtensionClass.h fix from CVS mentioned here in the list too? -- Ideas don't stay in some minds very long because they don't like solitary confinement.
Oh, and which branch should I get the tarball from? On Wed, 2001-12-19 at 18:20, Leonardo Rochael Almeida wrote:
On Wed, 2001-12-19 at 18:02, Martijn Pieters wrote:
[...] You can download files from CVS as a tarball. Just go to http://cvs.zope.org/Zope/lib/python/RestrictedPython and use the link at the bottom.
Should I apply the ExtensionClass.h fix from CVS mentioned here in the list too?
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
I'm not sure the ExtensionClass fix is baked yet, and there's not yet an overwhelming consensus that it helps stability, not to mention that I'm not sure how to install it, so no. ;-) I think you can use the trunk, no branch required. Hopefully the fixes will be merged soon. ----- Original Message ----- From: "Leonardo Rochael Almeida" <leo@hiper.com.br> To: "Zope Developers list" <zope-dev@zope.org> Sent: Wednesday, December 19, 2001 3:22 PM Subject: Re: [Zope-dev] disabling gc does not necessarily hide memorycorruption
Oh, and which branch should I get the tarball from?
On Wed, 2001-12-19 at 18:20, Leonardo Rochael Almeida wrote:
On Wed, 2001-12-19 at 18:02, Martijn Pieters wrote:
[...] You can download files from CVS as a tarball. Just go to http://cvs.zope.org/Zope/lib/python/RestrictedPython and use the link
at the
bottom.
Should I apply the ExtensionClass.h fix from CVS mentioned here in the list too?
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Well the ExtensionClass.h fix I think is completely harmless (assuming you recompile every ExtensionClass based C module) but I'm not convinced it's necessary, either. It's more of an insurance policy -- if every C module plays by the rules and checks the type flags field of the type object first, then the ExtensionClass.h file fix isn't necessary. Of course, anytime someone says well "if every..." I get suspicious. Those people who have reported that it helps their system stability are probably benefiting because stack stomps (from the two known stack-clobbering bugs in Python 2.1) end up leaving pointers to extensionclass objects in places they don't belong (ie places which have already CHECKED the flags, and thus don't check them again). Those C modules then use a NULL pointer check, which properly halts chasing of chains with the ExtensionClass.h patch, where it would have led to another wild pointer without. Its the same reason disabling GC also helps some people; it reduces the activities that follow bad pointers. Actually, I was kind of hoping Martijn Faassen would pipe up and say "I applied the restricted python patches you've already put up on the Zope-2_4-branch, and my problems with ParsedXML went away!" since he's one of the folks that did NOT benefit from applying the ExtensionClass.h patch. On Wednesday, December 19, 2001, at 03:44 PM, Chris McDonough wrote:
I'm not sure the ExtensionClass fix is baked yet, and there's not yet an overwhelming consensus that it helps stability, not to mention that I'm not sure how to install it, so no. ;-)
I think you can use the trunk, no branch required. Hopefully the fixes will be merged soon.
----- Original Message ----- From: "Leonardo Rochael Almeida" <leo@hiper.com.br> To: "Zope Developers list" <zope-dev@zope.org> Sent: Wednesday, December 19, 2001 3:22 PM Subject: Re: [Zope-dev] disabling gc does not necessarily hide memorycorruption
Oh, and which branch should I get the tarball from?
On Wed, 2001-12-19 at 18:20, Leonardo Rochael Almeida wrote:
On Wed, 2001-12-19 at 18:02, Martijn Pieters wrote:
[...] You can download files from CVS as a tarball. Just go to http://cvs.zope.org/Zope/lib/python/RestrictedPython and use the link
at the
bottom.
Should I apply the ExtensionClass.h fix from CVS mentioned here in the list too?
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Matthew T. Kromer wrote: [snip]
Actually, I was kind of hoping Martijn Faassen would pipe up and say "I applied the restricted python patches you've already put up on the Zope-2_4-branch, and my problems with ParsedXML went away!" since he's one of the folks that did NOT benefit from applying the ExtensionClass.h patch.
Here I'm piping up, finally; I didn't catch this thread until before. I found a pyassem.py 6 days old in the trunk, so I'm putting that into a Zope2.5.0b2 now.. restarting Zope.. Nope. Would be odd if it had, as this code shouldn't be exercising restricted Python at all anyway (it's just a Python product in the filesystem). Setting ZOPE_SECURITY_POLICY to PYTHON makes the problem go away, so I suspect the problem is still with cAccessControl.. (or in ParsedXML, but the code isn't complicated securitywise as far as I am aware, and doesn't crash when I run it in plain python) Regards, Martijn
I just found out that the RestrictedPython package isn't actually yet fixed! Shane is working on it. He's sweating and I think I almost heard him curse once. ;-) So please hang tight and we'll let you know when it's safe to go in the water. Note that at the time that it's fixed, all that needs to be done is to download the tarball as per Martijn's instructions, move the current "RestrictedPython" package aside, and put the new RestrictedPython package in its place, restart Zope. ----- Original Message ----- From: "Leonardo Rochael Almeida" <leo@hiper.com.br> To: "Zope Developers list" <zope-dev@zope.org> Sent: Wednesday, December 19, 2001 2:24 PM Subject: Re: [Zope-dev] disabling gc does not necessarily hide memorycorruption
On Wed, 2001-12-19 at 13:34, Chris McDonough wrote:
It would be good if someone who is experiencing random crashes could
confirm
that the the new compiler package fixes their problem.
I might risk losing a few more points with my sysadmin to test it (Hi daniduc :-) if there was an easily downloadable package with clear instructions on how to apply it to Zope 2.4.3 (I don't feel like messing with CVS for this), and if you tell me that it won't corrupt my Data.fs or anything :-)
Cheers, Leo
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
After reading this thread, my sysadmin (hi daniduc :-) just pointed a gun at my head and said: "When those guys say it's ready, you'll wait 24h and THEN you will try the fix" :-) Cheers, Leo On Wed, 2001-12-19 at 18:34, Chris McDonough wrote:
I just found out that the RestrictedPython package isn't actually yet fixed! Shane is working on it. He's sweating and I think I almost heard him curse once. ;-) So please hang tight and we'll let you know when it's safe to go in the water.
Note that at the time that it's fixed, all that needs to be done is to download the tarball as per Martijn's instructions, move the current "RestrictedPython" package aside, and put the new RestrictedPython package in its place, restart Zope.
-- Ideas don't stay in some minds very long because they don't like solitary confinement.
Em Quarta 19 Dezembro 2001 18:38, Leonardo Rochael Almeida escreveu:
After reading this thread, my sysadmin (hi daniduc :-) just pointed a
Hi Leo! =)
gun at my head and said: "When those guys say it's ready, you'll wait 24h and THEN you will try the fix" :-)
Aw, c'm'om, If I weren't paranoic I could not be a sysadmin, could I? ;) But we will try this fix, hope that contributes with the Zope Community! Regards -- daniel lobato duclos -- daniduc@hiper.com.br -- http://www.hiperlogica.com.br -----------------------------------------------------------------------------
Chris McDonough wrote:
I just found out that the RestrictedPython package isn't actually yet fixed! Shane is working on it. He's sweating and I think I almost heard him curse once. ;-) So please hang tight and we'll let you know when it's safe to go in the water.
Okay, if there's no fix in the trunk then diregard the message I just sent; I just applied the only change I could find that seemed recent enough. Still, it sounds odd if the ParsedXML failures in Zope 2.5 were caused by this problem, for reasons described in my other post. :) Regards, Martijn
participants (10)
-
Anthony Baxter -
Brian Lloyd -
Chris McDonough -
Daniel Duclos -
jeremy＠zope.com -
Leonardo Rochael Almeida -
Martijn Faassen -
Martijn Pieters -
Matthew T. Kromer -
Toby Dickenson