Zope 2.4 on a Debian 2.2 linux server. After 3 days uptime, Zope is taking lot and lot of memory on my server although this is a very small site with very few hits. socrate:~# memstat | sort -rn | more 1958684k 157416k: PID 14967 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14966 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14965 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14964 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14963 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14692 (/opt/Zope-2.4.0-linux2-x86/bin/python) What can cause Zope to use that much memory ? How Zope manage its memory ? Regards, Jerome. ______________________________________________________________________________ ifrance.com, l'email gratuit le plus complet de l'Internet ! vos emails depuis un navigateur, en POP3, sur Minitel, sur le WAP... http://www.ifrance.com/_reloc/email.emailif
Jerome hebert writes:
Zope 2.4 on a Debian 2.2 linux server. ... socrate:~# memstat | sort -rn | more 1958684k 157416k: PID 14967 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14966 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14965 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14964 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14963 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14692 (/opt/Zope-2.4.0-linux2-x86/bin/python) Your Zope uses 157 MB. This is not yet something, you need to worry about.
Under Linux, threads are implemented by processes. However, they share the memory. All Zope processes/threads together use 157 MB, not each one of them. Dieter
At 21:40 11/06/2002, Dieter Maurer wrote:
Jerome hebert writes:
Zope 2.4 on a Debian 2.2 linux server. ... socrate:~# memstat | sort -rn | more 1958684k 157416k: PID 14967 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14966 (/opt/Zope-2.4.0-linux2-x86/bin/python) Your Zope uses 157 MB. This is not yet something, you need to worry about.
Under Linux, threads are implemented by processes. However, they share the memory. All Zope processes/threads together use 157 MB, not each one of them.
Still 157 MB is too much for a Zope server with so few hits. There must be something wrong. When I start Zope, it only takes a few MB of memory, going to more than one hundred is not acceptable. If Zope is not able to use less memory, I will have to stay with Apache and Perl. But more likely the problem is coming from my setup because I'm new to Zope and Python so perhaps I certainly haven't done the right thing. One possibility I see is that the problem is coming from my external methods. I have an external method doing SOAP request (using the SOAP.py module) and I use a dump python class around the returned SOAP objects to avoid the problem of security exception when accessing the SOAP objects in DTML. Possibly the SOAP objects are never destroyed by Zope and this would explain why Zope continuously grows in memory. Can someone explain how the object returned from external scripts are destroyed by Zope after the HTTP request has been completed ? Below is an extract of my external method: # get_objects() IS THE EXTERNAL METHOD CALLED FROM DTML class somp: """ A class to access SOMP Agent""" __allow_access_to_unprotected_subobjects__ = 1 def get_objects(self, type='', id='', status='', options={'attributes':''}): """ get objects""" rt = [] self.somp = SOAP.SOAPProxy(self.proxy, unwrap_results = 0, namespace = self.namespace ) objects = self.somp.get_objects(type=type, id=id, status=status, options=options) for object in objects: sop = sompobject(object) sop.somp = self.somp rt.append(sop) return rt # DUMP PYTHON CLASS AROUND SOAP OBJECTS class sompobject: __allow_access_to_unprotected_subobjects__ = 1 def __init__(self, object): self.__allow_access_to_unprotected_subobjects__ = 1 self.object = object def __getattr__(self,name): try: return self.object[name] except AttributeError: try: return self.object['attributes'][name]['value'] except AttributeError: raise AttributeError, "Was Asking for attribut " + name Regards, Jerome. At 21:40 11/06/2002, Dieter Maurer wrote:
Jerome hebert writes:
Zope 2.4 on a Debian 2.2 linux server. ... socrate:~# memstat | sort -rn | more 1958684k 157416k: PID 14967 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14966 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14965 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14964 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14963 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14692 (/opt/Zope-2.4.0-linux2-x86/bin/python) Your Zope uses 157 MB. This is not yet something, you need to worry about.
Under Linux, threads are implemented by processes. However, they share the memory. All Zope processes/threads together use 157 MB, not each one of them.
Dieter
______________________________________________________________________________ ifrance.com, l'email gratuit le plus complet de l'Internet ! vos emails depuis un navigateur, en POP3, sur Minitel, sur le WAP... http://www.ifrance.com/_reloc/email.emailif
Jerome hebert writes:
... Zope consumes too much memory ... One possibility I see is that the problem is coming from my external methods.
I have an external method doing SOAP request (using the SOAP.py module) and I use a dump python class around the returned SOAP objects to avoid the problem of security exception when accessing the SOAP objects in DTML.
Possibly the SOAP objects are never destroyed by Zope and this would explain why Zope continuously grows in memory. Usually, Python uses reference counting to release objects which are no longer referenced. That works quite reliably, even for objects created in External Methods (no difference wherever the object is created).
However, reference counting is unable to release circular structures. Since Python 2.0, there is a cyclic garbage collector to detect and free circular garbage. For security reasons, this collector does not release cycles where an object has a "__del__" method (see c.l.p for the reasons). Folklore tells that the cyclic garbage collector is also unable to release "ExtensionClass" instances. If this were the case, then you would need to be very careful with almost all special Zope classes (as they probably derive from ExtensionClass). This would mean, if you happen to create circular structures containing an ExtensionClass instance, the cycle would not be freed. There are tools to locate for circular structures. You may also get a hint, when you look at the "Debug management" page in "Control_Panel". Dieter
I have to agree with...uh... dammit, too many quote levels. It's taking too much memory. If you restart the server and it takes substantially less memory, then a leak is the only reasonable conclusion. I myself have found a leak in 2.5.1 (Solaris) so I know they are there. I suggest you attempt to isolate the issue using the Debug control panel screen, then submit a collector issue: http://collector.zope.org. In the meantime, you may need to resort to periodic server restarts via cron. However, given the magnitude of your heaps it's possible that you're leaking so fast you won't be able to keep the servers up for more than a few hours in production. There is also a leak detector product on zope.org that might give you some help although I have not had time to try it out. http://www.zope.org/Members/hathawsh/LeakFinder
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Jerome hebert Sent: Wednesday, June 12, 2002 2:02 AM To: zope@zope.org Cc: Dieter Maurer Subject: [Zope] External scripts objects memory not released ?
At 21:40 11/06/2002, Dieter Maurer wrote:
Jerome hebert writes:
Zope 2.4 on a Debian 2.2 linux server. ... socrate:~# memstat | sort -rn | more 1958684k 157416k: PID 14967 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14966 (/opt/Zope-2.4.0-linux2-x86/bin/python) Your Zope uses 157 MB. This is not yet something, you need to worry about.
Under Linux, threads are implemented by processes. However, they share the memory. All Zope processes/threads together use 157 MB, not each one of them.
Still 157 MB is too much for a Zope server with so few hits.
There must be something wrong. When I start Zope, it only takes a few MB of memory, going to more than one hundred is not acceptable.
If Zope is not able to use less memory, I will have to stay with Apache and Perl. But more likely the problem is coming from my setup because I'm new to Zope and Python so perhaps I certainly haven't done the right thing.
One possibility I see is that the problem is coming from my external methods.
I have an external method doing SOAP request (using the SOAP.py module) and I use a dump python class around the returned SOAP objects to avoid the problem of security exception when accessing the SOAP objects in DTML.
Possibly the SOAP objects are never destroyed by Zope and this would explain why Zope continuously grows in memory.
Can someone explain how the object returned from external scripts are destroyed by Zope after the HTTP request has been completed ?
Below is an extract of my external method:
# get_objects() IS THE EXTERNAL METHOD CALLED FROM DTML class somp: """ A class to access SOMP Agent""" __allow_access_to_unprotected_subobjects__ = 1
def get_objects(self, type='', id='', status='', options={'attributes':''}): """ get objects""" rt = [] self.somp = SOAP.SOAPProxy(self.proxy, unwrap_results = 0, namespace = self.namespace ) objects = self.somp.get_objects(type=type, id=id, status=status, options=options) for object in objects: sop = sompobject(object) sop.somp = self.somp rt.append(sop) return rt
# DUMP PYTHON CLASS AROUND SOAP OBJECTS class sompobject: __allow_access_to_unprotected_subobjects__ = 1
def __init__(self, object): self.__allow_access_to_unprotected_subobjects__ = 1 self.object = object
def __getattr__(self,name): try: return self.object[name] except AttributeError: try: return self.object['attributes'][name]['value'] except AttributeError: raise AttributeError, "Was Asking for attribut " + name
Regards, Jerome.
At 21:40 11/06/2002, Dieter Maurer wrote:
Jerome hebert writes:
Zope 2.4 on a Debian 2.2 linux server. ... socrate:~# memstat | sort -rn | more 1958684k 157416k: PID 14967 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14966 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14965 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14964 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14963 (/opt/Zope-2.4.0-linux2-x86/bin/python) 157416k: PID 14692 (/opt/Zope-2.4.0-linux2-x86/bin/python) Your Zope uses 157 MB. This is not yet something, you need to worry about.
Under Linux, threads are implemented by processes. However, they share the memory. All Zope processes/threads together use 157 MB, not each one of them.
Dieter
__________________________________________________________________ ____________ ifrance.com, l'email gratuit le plus complet de l'Internet ! vos emails depuis un navigateur, en POP3, sur Minitel, sur le WAP... http://www.ifrance.com/_reloc/email.emailif
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
participants (3)
-
Charlie Reiman -
Dieter Maurer -
Jerome hebert