Hi All, I'm suffering with an app server that keeps blowing through all its memory. I have a suspicion this is due to a dodgy python script or page template pulling too many objects into memory. However, by the time I know to look, the app server throws MemoryErrors for each and every request. What're the best ways to go about debugging this? Would DeadlockDebugger help or will it suffer from the same problem? Of course, it'd be lovely if you could actually specify an upper memory bound for a particular request to use, and/or Zope as a whole, but these kinds of things always seem to stall at the development stage :-( cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
----- Original Message ----- From: "Chris Withers" <chris@simplistix.co.uk> To: "zope list user" <zope@zope.org> Sent: Thursday, August 09, 2007 12:09 PM Subject: [Zope] debugging memory exhaustion
Hi All,
I'm suffering with an app server that keeps blowing through all its memory.
I have a suspicion this is due to a dodgy python script or page template pulling too many objects into memory. However, by the time I know to look, the app server throws MemoryErrors for each and every request.
What're the best ways to go about debugging this? Would DeadlockDebugger help or will it suffer from the same problem?
Of course, it'd be lovely if you could actually specify an upper memory bound for a particular request to use, and/or Zope as a whole, but these kinds of things always seem to stall at the development stage :-(
If you're running on a linux machine you could look into the ulimit command to limit the process memory utilization, which may allow you to see what it happening. Jonathan
Jonathan wrote:
Of course, it'd be lovely if you could actually specify an upper memory bound for a particular request to use, and/or Zope as a whole, but these kinds of things always seem to stall at the development stage :-(
If you're running on a linux machine you could look into the ulimit command to limit the process memory utilization, which may allow you to see what it happening.
Er no, that would just exacerbate the problem. The machine itself is functional, just with low memory. Zope is throwing MemoryErrors because the OS won't give it anymore. ulimit'ing zope would just make those MemoryErrors come sooner... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
----- Original Message ----- From: "Chris Withers" <chris@simplistix.co.uk> To: "Jonathan" <dev101@magma.ca> Cc: "zope list user" <zope@zope.org> Sent: Thursday, August 09, 2007 1:24 PM Subject: Re: [Zope] debugging memory exhaustion
Jonathan wrote:
Of course, it'd be lovely if you could actually specify an upper memory bound for a particular request to use, and/or Zope as a whole, but these kinds of things always seem to stall at the development stage :-(
If you're running on a linux machine you could look into the ulimit command to limit the process memory utilization, which may allow you to see what it happening.
Er no, that would just exacerbate the problem. The machine itself is functional, just with low memory. Zope is throwing MemoryErrors because the OS won't give it anymore. ulimit'ing zope would just make those MemoryErrors come sooner...
In your original post you said: "Of course, it'd be lovely if you could actually specify an upper memory bound for a particular request to use, and/or Zope as a whole" If your server is still alive and Zope is throwing memory errors, then Zope has hit its specified upper memory limit. What do you want Zope to do at this point? Jonathan
Jonathan wrote:
"Of course, it'd be lovely if you could actually specify an upper memory bound for a particular request to use, and/or Zope as a whole"
If your server is still alive and Zope is throwing memory errors, then Zope has hit its specified upper memory limit.
No, Zope does little if anything to control the amount of memory it uses. If it did, we wouldn't see MemoryErrors ;-)
What do you want Zope to do at this point?
Start dumping objects out of the ZODB object cache when it reaches a certain memory limit, preferably configurable in zope.conf. cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
----- Original Message ----- From: "Chris Withers" <chris@simplistix.co.uk> To: "Jonathan" <dev101@magma.ca> Cc: "zope list user" <zope@zope.org> Sent: Thursday, August 09, 2007 5:47 PM Subject: Re: [Zope] debugging memory exhaustion
Jonathan wrote:
"Of course, it'd be lovely if you could actually specify an upper memory bound for a particular request to use, and/or Zope as a whole"
If your server is still alive and Zope is throwing memory errors, then Zope has hit its specified upper memory limit.
No, Zope does little if anything to control the amount of memory it uses. If it did, we wouldn't see MemoryErrors ;-)
By "Zope" i meant the process running zope (i agree that Zope does not track process memory utilization!).
What do you want Zope to do at this point?
Start dumping objects out of the ZODB object cache when it reaches a certain memory limit, preferably configurable in zope.conf.
There is already a way to control (within limits) the number of objects in the ZODB cache, so you must be referring to something else? (ie. a way to control memory eating 'dodgy' scripts?). Jonathan
Jonathan wrote:
No, Zope does little if anything to control the amount of memory it uses. If it did, we wouldn't see MemoryErrors ;-)
By "Zope" i meant the process running zope (i agree that Zope does not track process memory utilization!).
Answering a question I haven't asked isn't helpful...
What do you want Zope to do at this point?
Start dumping objects out of the ZODB object cache when it reaches a certain memory limit, preferably configurable in zope.conf.
There is already a way to control (within limits) the number of objects in the ZODB cache, so you must be referring to something else? (ie. a way to control memory eating 'dodgy' scripts?).
Number and size have little to do with each other... I've always wanted there to be a memory-size based limit on object cache which is maintained throughout a transaction, not just at transaction boundaries. However, that is hard to do... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote at 2007-8-9 17:09 +0100:
... I'm suffering with an app server that keeps blowing through all its memory.
I have a suspicion this is due to a dodgy python script or page template pulling too many objects into memory. However, by the time I know to look, the app server throws MemoryErrors for each and every request.
"ZODB.DB.DB" provides methods for detailed analysis of the cached objects. It is used for the "Extreme cache detail" ZMI page but can also be used differently (e.g. from an "External Methode" or via the "ZServer Monitor Server" interactively). Thus, provided the memory blowup comes from objects cached in ZODB caches, you have a good chance to find out at least what type of objects are causing this behaviour.
What're the best ways to go about debugging this? Would DeadlockDebugger help or will it suffer from the same problem?
"DeadlockDebugger" would help you only when the activity eating your memory would also considerably eat CPU resources (which may indeed be the case).
Of course, it'd be lovely if you could actually specify an upper memory bound for a particular request to use, and/or Zope as a whole, but these kinds of things always seem to stall at the development stage :-(
You should find the culprit. While a better cache management might avoid the MemoryError, the observed bad behaviour would trash your cache -- and following activities would be by several orders slower than usual.... -- Dieter
Dieter Maurer wrote:
"ZODB.DB.DB" provides methods for detailed analysis of the cached objects.
Cool, I'll bear that in mind...
It is used for the "Extreme cache detail" ZMI page but can also be used differently (e.g. from an "External Methode" or via the "ZServer Monitor Server" interactively).
I've never known how to use the monitor server. Are there examples or documentation anywhere?
Of course, it'd be lovely if you could actually specify an upper memory bound for a particular request to use, and/or Zope as a whole, but these kinds of things always seem to stall at the development stage :-(
You should find the culprit. While a better cache management might avoid the MemoryError, the observed bad behaviour would trash your cache -- and following activities would be by several orders slower than usual....
Yes, but it's a bad failure mode. Slow-but-functional is better than throwing MemoryErrors in my book... (especially as a trashed cache resolves itself, whereas MemoryErrors always seem to require a restart...) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote at 2007-8-13 10:02 +0100:
...
It is used for the "Extreme cache detail" ZMI page but can also be used differently (e.g. from an "External Methode" or via the "ZServer Monitor Server" interactively).
I've never known how to use the monitor server. Are there examples or documentation anywhere?
Yes, I have seen monitor server documentation -- but I forgot where. But it is very easy: You need to enable the monitor server. This entails its configuration and the creation of an emergency user (with plain text password). Then, you use the "ZServer.medusa.monitor_client[_win32]" to connect to your monitor server. You will be asked for authentication and use the emercency user and its password. After successful authentication, you have a Python interpreter prompt inside your Zope process. You use "from Zope2 import app; app=app()" and you have an "app" object -- similar to "bin/zopectl debug".
... Yes, but it's a bad failure mode. Slow-but-functional is better than throwing MemoryErrors in my book... (especially as a trashed cache resolves itself, whereas MemoryErrors always seem to require a restart...)
How preferences differ.... But after you have found the culprit (and removed it), there will be no more "MemoryError"s :-) -- Dieter
Dieter Maurer wrote:
Then, you use the "ZServer.medusa.monitor_client[_win32]" to connect to your monitor server.
Where/what is this?
You will be asked for authentication and use the emercency user and its password.
After successful authentication, you have a Python interpreter prompt inside your Zope process.
You use
"from Zope2 import app; app=app()"
and you have an "app" object -- similar to "bin/zopectl debug".
Very cool as I'm guessing you can use the usual python threadish stuff to haev a look at what all the other processes are doing?
How preferences differ....
I'm guessing you don't have customers who do scripting? ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote at 2007-8-14 08:22 +0100:
Dieter Maurer wrote:
Then, you use the "ZServer.medusa.monitor_client[_win32]" to connect to your monitor server.
Where/what is this?
You do not understand Python's package notation? Unbelievable... Think about it, you will get it ;-)
...
You use
"from Zope2 import app; app=app()"
and you have an "app" object -- similar to "bin/zopectl debug".
Very cool as I'm guessing you can use the usual python threadish stuff to haev a look at what all the other processes are doing?
It depends what you mean by "threadish stuff". You can use "threadframe" (the extension behind "DeadlockDebugger") to see the tracebacks of all Python threads. However, you cannot do much more with the other threads (not to speak about "processes", at all). -- Dieter
participants (3)
-
Chris Withers -
Dieter Maurer -
Jonathan