Python use more and more memory
Hi, The memory used by python getting more and more. Is that a normal behavoir. In beginning I start Zope. the python use about 26M memory. After several hours get the result by "top" Sort by memory usage PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 4931 www-data 13 0 94324 92M 2504 S 3.5 10.4 7:43 python 5022 www-data 9 0 94324 92M 2504 S 0.0 10.4 0:00 python 5023 www-data 9 0 94324 92M 2504 S 0.0 10.4 15:55 python 5024 www-data 14 0 94324 92M 2504 S 14.5 10.4 16:55 python 5025 www-data 10 0 94324 92M 2504 S 0.3 10.4 15:35 python 5026 www-data 9 0 94324 92M 2504 S 0.0 10.4 16:32 python Have any idea why? TerryH
Terry writes:
The memory used by python getting more and more. Is that a normal behavoir.
In beginning I start Zope. the python use about 26M memory.
After several hours get the result by "top" Sort by memory usage
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 4931 www-data 13 0 94324 92M 2504 S 3.5 10.4 7:43 python 5022 www-data 9 0 94324 92M 2504 S 0.0 10.4 0:00 python 5023 www-data 9 0 94324 92M 2504 S 0.0 10.4 15:55 python 5024 www-data 14 0 94324 92M 2504 S 14.5 10.4 16:55 python 5025 www-data 10 0 94324 92M 2504 S 0.3 10.4 15:35 python 5026 www-data 9 0 94324 92M 2504 S 0.0 10.4 16:32 python
Have any idea why? That is not unusual.
Zope makes extensive use of caches. They get filled over time and consume memory. Only, when memory consumption grows unbounded, you want to start looking for leaks. There is a "LeakFinder" product to help you in this case. Dieter
Dieter Maurer wrote:
Terry writes:
The memory used by python getting more and more. Is that a normal behavoir.
In beginning I start Zope. the python use about 26M memory.
After several hours get the result by "top" Sort by memory usage
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 4931 www-data 13 0 94324 92M 2504 S 3.5 10.4 7:43 python 5022 www-data 9 0 94324 92M 2504 S 0.0 10.4 0:00 python 5023 www-data 9 0 94324 92M 2504 S 0.0 10.4 15:55 python 5024 www-data 14 0 94324 92M 2504 S 14.5 10.4 16:55 python 5025 www-data 10 0 94324 92M 2504 S 0.3 10.4 15:35 python 5026 www-data 9 0 94324 92M 2504 S 0.0 10.4 16:32 python
Have any idea why? That is not unusual.
Zope makes extensive use of caches. They get filled over time and consume memory.
Only, when memory consumption grows unbounded, you want to start looking for leaks. There is a "LeakFinder" product to help you in this case.
In addition, search for ChrisD's AutoLance product on zope.org, which restarts zope if it's memory usage goes over a configurable threshold. It's a band-aid solution, but it works quite well. cheers, oliver
Dieter Maurer at 20:18 (UTC+0100) on Tue, 10 Dec 2002 said: = Terry writes: = > The memory used by python getting more and more. = > Is that a normal behavoir. = > = > In beginning I start Zope. = > the python use about 26M memory. = > = > After several hours = > get the result by "top" = > Sort by memory usage = > = > PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND = > 4931 www-data 13 0 94324 92M 2504 S 3.5 10.4 7:43 python = > 5022 www-data 9 0 94324 92M 2504 S 0.0 10.4 0:00 python = > 5023 www-data 9 0 94324 92M 2504 S 0.0 10.4 15:55 python = > 5024 www-data 14 0 94324 92M 2504 S 14.5 10.4 16:55 python = > 5025 www-data 10 0 94324 92M 2504 S 0.3 10.4 15:35 python = > 5026 www-data 9 0 94324 92M 2504 S 0.0 10.4 16:32 python = > = > Have any idea why? For easier reading, I inserted spaces in the above top output. Hope this didn't mess it up for someone else. Maybe I'm missing something, but I see the %MEM staying at 10.4; SHARE and SIZE are likewise staying the same. Seems that it's behaving itself quite well. Or...? ken
On Wed, Dec 11, 2002 at 09:27:07AM -0500, ken wrote:
Dieter Maurer at 20:18 (UTC+0100) on Tue, 10 Dec 2002 said: = Terry writes: = > In beginning I start Zope. = > the python use about 26M memory. = > = > After several hours = > get the result by "top" = > Sort by memory usage = > = > PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND = > 4931 www-data 13 0 94324 92M 2504 S 3.5 10.4 7:43 python = > 5022 www-data 9 0 94324 92M 2504 S 0.0 10.4 0:00 python = > 5023 www-data 9 0 94324 92M 2504 S 0.0 10.4 15:55 python = > 5024 www-data 14 0 94324 92M 2504 S 14.5 10.4 16:55 python = > 5025 www-data 10 0 94324 92M 2504 S 0.3 10.4 15:35 python = > 5026 www-data 9 0 94324 92M 2504 S 0.0 10.4 16:32 python = > = > Have any idea why?
For easier reading, I inserted spaces in the above top output. Hope this didn't mess it up for someone else.
That's much prettier.
Maybe I'm missing something, but I see the %MEM staying at 10.4; SHARE and SIZE are likewise staying the same. Seems that it's behaving itself quite well. Or...?
I think the listing is a snapshot at one time - the lines are the different threads and the memory figures are in fact shared memory. The percentage is 10.4, but the actual size is 94M, which compared to 26M at the beginning is a sizeable increase. Felix.
Felix Ulrich-Oltean at 14:36 (UTC-0000) on Wed, 11 Dec 2002 said: = On Wed, Dec 11, 2002 at 09:27:07AM -0500, ken wrote: = > Dieter Maurer at 20:18 (UTC+0100) on Tue, 10 Dec 2002 said: = > = Terry writes: = > = > In beginning I start Zope. = > = > the python use about 26M memory. = > = > = > = > After several hours = > = > get the result by "top" = > = > Sort by memory usage = > = > = > = > PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND = > = > 4931 www-data 13 0 94324 92M 2504 S 3.5 10.4 7:43 python = > = > 5022 www-data 9 0 94324 92M 2504 S 0.0 10.4 0:00 python = > = > 5023 www-data 9 0 94324 92M 2504 S 0.0 10.4 15:55 python = > = > 5024 www-data 14 0 94324 92M 2504 S 14.5 10.4 16:55 python = > = > 5025 www-data 10 0 94324 92M 2504 S 0.3 10.4 15:35 python = > = > 5026 www-data 9 0 94324 92M 2504 S 0.0 10.4 16:32 python = > = > = > = > Have any idea why? = > ... = > Maybe I'm missing something, but I see the %MEM staying at 10.4; SHARE = > and SIZE are likewise staying the same. Seems that it's behaving itself = > quite well. Or...? = = I think the listing is a snapshot at one time - the lines are the = different threads and the memory figures are in fact shared memory. = The percentage is 10.4, but the actual size is 94M, which compared to = 26M at the beginning is a sizeable increase. = ... Working from that and looking at the second instance, we can see that process 5022 has been running for 0.00 (just started, presumably), but still has claimed 92M, the same as other processes which have been running much longer. On the face of it, this doesn't indicate a memory leak-- which would be something to be concerned about-- but that, for some reason, python thinks it requires more memory upon initiating a process. By itself, this doesn't tell us what is causing the increase in demand for memory. It would be a situation calling for diagnostics collected over time, as with sar or vmstat, and these correlated with demands made on python. Cheers, ken
ken writes:
Felix Ulrich-Oltean at 14:36 (UTC-0000) on Wed, 11 Dec 2002 said:
= On Wed, Dec 11, 2002 at 09:27:07AM -0500, ken wrote: = > Dieter Maurer at 20:18 (UTC+0100) on Tue, 10 Dec 2002 said: = > = Terry writes: = > = > In beginning I start Zope. = > = > the python use about 26M memory. = > = > = > = > After several hours = > = > get the result by "top" = > = > Sort by memory usage = > = > = > = > PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND = > = > 4931 www-data 13 0 94324 92M 2504 S 3.5 10.4 7:43 python = > = > 5022 www-data 9 0 94324 92M 2504 S 0.0 10.4 0:00 python = > = > 5023 www-data 9 0 94324 92M 2504 S 0.0 10.4 15:55 python = > = > 5024 www-data 14 0 94324 92M 2504 S 14.5 10.4 16:55 python = > = > 5025 www-data 10 0 94324 92M 2504 S 0.3 10.4 15:35 python = > = > 5026 www-data 9 0 94324 92M 2504 S 0.0 10.4 16:32 python = > = > = > = > Have any idea why? = > ... = > Maybe I'm missing something, but I see the %MEM staying at 10.4; SHARE = > and SIZE are likewise staying the same. Seems that it's behaving itself = > quite well. Or...? = = I think the listing is a snapshot at one time - the lines are the = different threads and the memory figures are in fact shared memory. = The percentage is 10.4, but the actual size is 94M, which compared to = 26M at the beginning is a sizeable increase. = ...
Working from that and looking at the second instance, we can see that process 5022 has been running for 0.00 (just started, presumably), but still has claimed 92M, the same as other processes which have been running much longer. The various processes you see represent the different threads comprising your Zope.
By definition, threads share most of the process resources, this includes the address space. Therefore, all threads have the same memory, they share it, whether or not they still did any work by themselves. Dieter
Thanks to the list, I think I found the stupid thing I did. Because I replace the "Maximum rows to retrieve" of a ZSQL Method by "0", sometimes there are 1??,??? rows return from the Database. Then even hang my Database. Thanks to everyone. Felix Ulrich-Oltean wrote:
On Wed, Dec 11, 2002 at 09:27:07AM -0500, ken wrote:
Dieter Maurer at 20:18 (UTC+0100) on Tue, 10 Dec 2002 said: = Terry writes: = > In beginning I start Zope. = > the python use about 26M memory. = > = > After several hours = > get the result by "top" = > Sort by memory usage = > = > PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND = > 4931 www-data 13 0 94324 92M 2504 S 3.5 10.4 7:43 python = > 5022 www-data 9 0 94324 92M 2504 S 0.0 10.4 0:00 python = > 5023 www-data 9 0 94324 92M 2504 S 0.0 10.4 15:55 python = > 5024 www-data 14 0 94324 92M 2504 S 14.5 10.4 16:55 python = > 5025 www-data 10 0 94324 92M 2504 S 0.3 10.4 15:35 python = > 5026 www-data 9 0 94324 92M 2504 S 0.0 10.4 16:32 python = > = > Have any idea why?
For easier reading, I inserted spaces in the above top output. Hope this didn't mess it up for someone else.
That's much prettier.
Maybe I'm missing something, but I see the %MEM staying at 10.4; SHARE and SIZE are likewise staying the same. Seems that it's behaving itself quite well. Or...?
I think the listing is a snapshot at one time - the lines are the different threads and the memory figures are in fact shared memory. The percentage is 10.4, but the actual size is 94M, which compared to 26M at the beginning is a sizeable increase.
Felix.
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
participants (5)
-
Dieter Maurer -
Felix Ulrich-Oltean -
ken -
Oliver Bleutgen -
Terry