Hey all, A client of mine seems to be plagued with memory leaks. I have a CMF / Plone site on We fixed some security issues that caused a major leak this is the same I believe as http://collector.zope.org/Zope/421, by jiggling settings this no longer an issue. Anyway we are still getting some minor leaks that severly limit the amount of the traffic the box is going to be able take. Lots of different objects are being leaked, including good old DateTime, however I dont believe its this, DateTime just happens to be there. Possible culprits are (trimmed down debug page) ZPT: Products.PageTemplates.Expressions.PathExpr, Products.PageTemplates.Expressions.SubPathExpr, Products.PageTemplates.TALES.TALESError etc... or AccessControl AccessControl.unauthorized.Unauthorized, AccessControl.SecurityManager.SecurityManager etc... A quick rummage through LeakFinder hasn't helped too much so Im not sure what too recommend. Good old install ZEO and restart Zope every few hours? Is anyone else having issues? Configuration: Plone 0.99 "Almost There" CMF Beta 1.3 beta 1 Zope 2.5.1 Python 2.1.3 RedHat Linux 7.1 Using Apache and pcgi to serve... -- Andy McKay
I do also see the ZPT expression stuff leaking. In my case in this order: Products.PageTemplates.Expressions.SubPathExpr Products.PageTemplates.Expressions.PathExpr Products.PageTemplates.ZRPythonExpr.PythonExpr This one does *not* go away when setting ZOPE_SECURITY_POLICY=PYTHON. I am also leaking refs to persistent User objects; may be a side effect though - like your DateTimes. The site in question serves large ZSQL result sets via ZPT to authenticated users. No CMF/Plone involved. Stefan --On Montag, 22. Juli 2002 13:45 -0700 Andy McKay <andy@agmweb.ca> wrote:
Hey all,
A client of mine seems to be plagued with memory leaks. I have a CMF / Plone site on We fixed some security issues that caused a major leak this is the same I believe as http://collector.zope.org/Zope/421, by jiggling settings this no longer an issue. Anyway we are still getting some minor leaks that severly limit the amount of the traffic the box is going to be able take. Lots of different objects are being leaked, including good old DateTime, however I dont believe its this, DateTime just happens to be there.
Possible culprits are (trimmed down debug page) ZPT: Products.PageTemplates.Expressions.PathExpr, Products.PageTemplates.Expressions.SubPathExpr, Products.PageTemplates.TALES.TALESError etc... or AccessControl AccessControl.unauthorized.Unauthorized, AccessControl.SecurityManager.SecurityManager etc...
A quick rummage through LeakFinder hasn't helped too much so Im not sure what too recommend. Good old install ZEO and restart Zope every few hours? Is anyone else having issues?
Configuration:
Plone 0.99 "Almost There" CMF Beta 1.3 beta 1 Zope 2.5.1 Python 2.1.3 RedHat Linux 7.1 Using Apache and pcgi to serve... -- Andy McKay
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- Those who write software only for pay should go hurt some other field. /Erik Naggum/
Hmm of course it doesnt mean ZPT expressions are causing the leak, but are a side effect. Of course my DTML ZopeZen is watertight and isnt leaking at all :) ----- Original Message ----- From: "Stefan H. Holek" <stefan@epy.co.at> To: "Andy McKay" <andy@agmweb.ca> Cc: <zope-dev@zope.org> Sent: Tuesday, July 23, 2002 3:37 AM Subject: Re: [Zope-dev] Memory Leaks
I do also see the ZPT expression stuff leaking. In my case in this order:
Products.PageTemplates.Expressions.SubPathExpr Products.PageTemplates.Expressions.PathExpr Products.PageTemplates.ZRPythonExpr.PythonExpr
This one does *not* go away when setting ZOPE_SECURITY_POLICY=PYTHON. I am also leaking refs to persistent User objects; may be a side effect though - like your DateTimes.
The site in question serves large ZSQL result sets via ZPT to authenticated users. No CMF/Plone involved.
Stefan
--On Montag, 22. Juli 2002 13:45 -0700 Andy McKay <andy@agmweb.ca> wrote:
Hey all,
A client of mine seems to be plagued with memory leaks. I have a CMF / Plone site on We fixed some security issues that caused a major leak this is the same I believe as http://collector.zope.org/Zope/421, by jiggling settings this no longer an issue. Anyway we are still getting some minor leaks that severly limit the amount of the traffic the box is going to be able take. Lots of different objects are being leaked, including good old DateTime, however I dont believe its this, DateTime just happens to be there.
Possible culprits are (trimmed down debug page) ZPT: Products.PageTemplates.Expressions.PathExpr, Products.PageTemplates.Expressions.SubPathExpr, Products.PageTemplates.TALES.TALESError etc... or AccessControl AccessControl.unauthorized.Unauthorized, AccessControl.SecurityManager.SecurityManager etc...
A quick rummage through LeakFinder hasn't helped too much so Im not sure what too recommend. Good old install ZEO and restart Zope every few hours? Is anyone else having issues?
Configuration:
Plone 0.99 "Almost There" CMF Beta 1.3 beta 1 Zope 2.5.1 Python 2.1.3 RedHat Linux 7.1 Using Apache and pcgi to serve... -- Andy McKay
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- Those who write software only for pay should go hurt some other field. /Erik Naggum/
On Tue, 23 Jul 2002 12:37:39 +0200 "Stefan H. Holek" <stefan@epy.co.at> wrote:
I do also see the ZPT expression stuff leaking. In my case in this order:
Products.PageTemplates.Expressions.SubPathExpr Products.PageTemplates.Expressions.PathExpr Products.PageTemplates.ZRPythonExpr.PythonExpr
This one does *not* go away when setting ZOPE_SECURITY_POLICY=PYTHON. I am also leaking refs to persistent User objects; may be a side effect though - like your DateTimes.
The site in question serves large ZSQL result sets via ZPT to authenticated users. No CMF/Plone involved.
I also have a leaking Zope, i'am also using large ZSQL result sets but no ZPT. I work with a result set of about 500 rows. I do a lot of stuff for each row of the result set, e.g. setup new objects in zope. All my work is done in a own product, if i work the first times with it, i could see my zope grows up about 500MB.... After that a found out, to do a get_transaction.commit(1) after i have done all my work for each row and the zope was not growing up with such big steps. I posted this problem some times ago, but no one could give a answer to this problem. All zope sites a have where we use ZSQL are growing up... Hope anyone could solve this..... Regards andre
Stefan
--On Montag, 22. Juli 2002 13:45 -0700 Andy McKay <andy@agmweb.ca> wrote:
Hey all,
A client of mine seems to be plagued with memory leaks. I have a CMF / Plone site on We fixed some security issues that caused a major leak this is the same I believe as http://collector.zope.org/Zope/421, by jiggling settings this no longer an issue. Anyway we are still getting some minor leaks that severly limit the amount of the traffic the box is going to be able take. Lots of different objects are being leaked, including good old DateTime, however I dont believe its this, DateTime just happens to be there.
Possible culprits are (trimmed down debug page) ZPT: Products.PageTemplates.Expressions.PathExpr, Products.PageTemplates.Expressions.SubPathExpr, Products.PageTemplates.TALES.TALESError etc... or AccessControl AccessControl.unauthorized.Unauthorized, AccessControl.SecurityManager.SecurityManager etc...
A quick rummage through LeakFinder hasn't helped too much so Im not sure what too recommend. Good old install ZEO and restart Zope every few hours? Is anyone else having issues?
Configuration:
Plone 0.99 "Almost There" CMF Beta 1.3 beta 1 Zope 2.5.1 Python 2.1.3 RedHat Linux 7.1 Using Apache and pcgi to serve... -- Andy McKay
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- Those who write software only for pay should go hurt some other field. /Erik Naggum/
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
On Tue, 23 Jul 2002 12:37:39 +0200 "Stefan H. Holek" <stefan@epy.co.at> wrote:
I do also see the ZPT expression stuff leaking. In my case in this order:
Products.PageTemplates.Expressions.SubPathExpr Products.PageTemplates.Expressions.PathExpr Products.PageTemplates.ZRPythonExpr.PythonExpr
This one does *not* go away when setting ZOPE_SECURITY_POLICY=PYTHON. I am also leaking refs to persistent User objects; may be a side effect though - like your DateTimes.
The site in question serves large ZSQL result sets via ZPT to authenticated users. No CMF/Plone involved.
Stefan
--On Montag, 22. Juli 2002 13:45 -0700 Andy McKay <andy@agmweb.ca> wrote:
Hey all,
A client of mine seems to be plagued with memory leaks. I have a CMF / Plone site on We fixed some security issues that caused a major leak this is the same I believe as http://collector.zope.org/Zope/421, by jiggling settings this no longer an issue. Anyway we are still getting some minor leaks that severly limit the amount of the traffic the box is going to be able take. Lots of different objects are being leaked, including good old DateTime, however I dont believe its this, DateTime just happens to be there.
Possible culprits are (trimmed down debug page) ZPT: Products.PageTemplates.Expressions.PathExpr, Products.PageTemplates.Expressions.SubPathExpr, Products.PageTemplates.TALES.TALESError etc... or AccessControl AccessControl.unauthorized.Unauthorized, AccessControl.SecurityManager.SecurityManager etc...
A quick rummage through LeakFinder hasn't helped too much so Im not sure what too recommend. Good old install ZEO and restart Zope every few hours? Is anyone else having issues?
Configuration:
Plone 0.99 "Almost There" CMF Beta 1.3 beta 1 Zope 2.5.1 Python 2.1.3 RedHat Linux 7.1 Using Apache and pcgi to serve... -- Andy McKay
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- Those who write software only for pay should go hurt some other field. /Erik Naggum/
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
On Tuesday July 23 2002 12:37, Stefan H. Holek wrote:
I do also see the ZPT expression stuff leaking. In my case in this order:
Products.PageTemplates.Expressions.SubPathExpr Products.PageTemplates.Expressions.PathExpr Products.PageTemplates.ZRPythonExpr.PythonExpr
This is usually what I see but today I noticed this in debug info after the site had been running for a couple of hours - it doesn't look good :-(: RestrictedPython.compiler_2_1.misc.Set: 37488 RestrictedPython.compiler_2_1.pycodegen.CodeGenerator: 18123 RestrictedPython.compiler_2_1.pyassem.Block: 14876 RestrictedPython.compiler_2_1.ast.Const: 12063 RestrictedPython.compiler_2_1.ast.CallFunc: 11864 RestrictedPython.compiler_2_1.ast.Name: 11314 RestrictedPython.compiler_2_1.pyassem.FlowGraph: 10329 Products.PageTemplates.Expressions.SubPathExpr: 10269 RestrictedPython.compiler_2_1.ast.Stmt: 8768 RestrictedPython.compiler_2_1.visitor.ASTVisitor: 7746 RestrictedPython.compiler_2_1.pyassem.PyFlowGraph: 7740 RestrictedPython.compiler_2_1.misc.Stack: 5165 RestrictedPython.compiler_2_1.pyassem.LineAddrTable: 5159 Products.PageTemplates.Expressions.PathExpr: 4635 RestrictedPython.compiler_2_1.ast.Return: 4584 RestrictedPython.compiler_2_1.ast.Assign: 3571 RestrictedPython.compiler_2_1.ast.AssName: 3546 RestrictedPython.compiler_2_1.ast.Function: 2587 RestrictedPython.compiler_2_1.pycodegen.NestedFunctionCodeGenerator: 2586 RestrictedPython.compiler_2_1.symbols.FunctionScope: 2582 RestrictedPython.compiler_2_1.ast.Module: 2581 RestrictedPython.compiler_2_1.symbols.ModuleScope: 2578 RestrictedPython.compiler_2_1.ast.Global: 1970 Products.PageTemplates.ZRPythonExpr.PythonExpr: 1949 RestrictedPython.compiler_2_1.ast.Compare: 1559 Products.CMFLocalizer.GettextExpr.GettextExpr: 1204 RestrictedPython.compiler_2_1.ast.Keyword: 1136 RestrictedPython.compiler_2_1.ast.Add: 858 RestrictedPython.compiler_2_1.ast.And: 853 Persistence.PersistentMapping: 828 RestrictedPython.compiler_2_1.ast.Discard: 815 Shared.DC.Scripts.Bindings.NameAssignments: 792 RestrictedPython.compiler_2_1.ast.Or: 723 RestrictedPython.compiler_2_1.ast.If: 653 Products.PageTemplates.Expressions.StringExpr: 577 ExtensionClass: 537 RestrictedPython.compiler_2_1.ast.Mod: 444 AccessControl.cAccessControl.PermissionRole: 430 RestrictedPython.compiler_2_1.ast.Pass: 424 App.special_dtml.DTMLFile: 372 Products.CMFCore.FSPageTemplate.FSPageTemplate: 291 Shared.DC.Scripts.Signature.FuncCode: 288 RestrictedPython.compiler_2_1.ast.Tuple: 276 RestrictedPython.compiler_2_1.ast.Not: 238 Products.CMFCore.FSPythonScript.FSPythonScript: 203 Interface.Method.Method: 183 Products.ZGadflyDA.gadfly.kjParser.ParseRule: 161 RestrictedPython.compiler_2_1.ast.Subscript: 152 RestrictedPython.compiler_2_1.ast.List: 151 App.PersistentExtra.Persistent: 146 RestrictedPython.compiler_2_1.ast.For: 142 AccessControl.SecurityInfo.ClassSecurityInfo: 137 App.ImageFile.ImageFile: 135DocumentTemplate.DT_Util.Eval: 128 RestrictedPython.compiler_2_1.ast.Dict: 127 RestrictedPython.compiler_2_1.ast.Sliceobj: 114O FS.SimpleItem.SimpleItem: 113 Products.CMFCore.PortalFolder.PortalFolder: 102 Products.CMFCore.FSDTMLMethod.FSDTMLMethod: 100 Acquisition.Acquirer: 100 Products.PageTemplates.Expressions.NotExpr: 96 DateTime.DateTime.DateTime: 96O FS.PropertySheets.PropertySheets: 95 Products.DRM.PurseTool.TransactionObject: 90 Products.CMFDefault.DublinCore
This one does *not* go away when setting ZOPE_SECURITY_POLICY=PYTHON. I am also leaking refs to persistent User objects; may be a side effect though - like your DateTimes.
The site in question serves large ZSQL result sets via ZPT to authenticated users. No CMF/Plone involved.
Stefan
--On Montag, 22. Juli 2002 13:45 -0700 Andy McKay <andy@agmweb.ca> wrote:
Hey all,
A client of mine seems to be plagued with memory leaks. I have a CMF / Plone site on We fixed some security issues that caused a major leak this is the same I believe as http://collector.zope.org/Zope/421, by jiggling settings this no longer an issue. Anyway we are still getting some minor leaks that severly limit the amount of the traffic the box is going to be able take. Lots of different objects are being leaked, including good old DateTime, however I dont believe its this, DateTime just happens to be there.
Possible culprits are (trimmed down debug page) ZPT: Products.PageTemplates.Expressions.PathExpr, Products.PageTemplates.Expressions.SubPathExpr, Products.PageTemplates.TALES.TALESError etc... or AccessControl AccessControl.unauthorized.Unauthorized, AccessControl.SecurityManager.SecurityManager etc...
A quick rummage through LeakFinder hasn't helped too much so Im not sure what too recommend. Good old install ZEO and restart Zope every few hours? Is anyone else having issues?
Configuration:
Plone 0.99 "Almost There" CMF Beta 1.3 beta 1 Zope 2.5.1 Python 2.1.3 RedHat Linux 7.1 Using Apache and pcgi to serve... -- Andy McKay
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
-- Regards, Thomas Olsen
jeremy@alum.mit.eduThomas Olsen wrote:
On Tuesday July 23 2002 12:37, Stefan H. Holek wrote:
I do also see the ZPT expression stuff leaking. In my case in this order:
Products.PageTemplates.Expressions.SubPathExpr Products.PageTemplates.Expressions.PathExpr Products.PageTemplates.ZRPythonExpr.PythonExpr
This is usually what I see but today I noticed this in debug info after the site had been running for a couple of hours - it doesn't look good :-(:
RestrictedPython.compiler_2_1.misc.Set: 37488 RestrictedPython.compiler_2_1.pycodegen.CodeGenerator: 18123 RestrictedPython.compiler_2_1.pyassem.Block: 14876 RestrictedPython.compiler_2_1.ast.Const: 12063 (...)
Do you have Python's cyclic garbage collector enabled? It's required for Zope 2.4 and above, primarily because the compiler package used by Zope makes lots of cyclic references and never cleans them up. You know, I never thought of this before, but we could perhaps manually clean up the cyclic references made by the compiler package. It would probably be easy. Goodness, why did this never occur to me? :-) Shane
On Tuesday August 6 2002 16:51, Shane Hathaway wrote:
Do you have Python's cyclic garbage collector enabled? It's required for Zope 2.4 and above, primarily because the compiler package used by Zope makes lots of cyclic references and never cleans them up.
Isn't that enabled per default? To be honest I don't event know where to do it.
You know, I never thought of this before, but we could perhaps manually clean up the cyclic references made by the compiler package. It would probably be easy. Goodness, why did this never occur to me? :-)
:-)
Shane
-- Regards, Thomas Olsen
participants (5)
-
Andre Schubert -
Andy McKay -
Shane Hathaway -
Stefan H. Holek -
Thomas Olsen