Hi all, I had a nightmare with my ZEO setup last night. Had to drive to the data center to clean up the mess... in wee hours. (3AM) The ZEO server has 4G RAM and running on RedHat Linux 8 with 2.4.0 custom kernel. Thought it was plenty. I have one ZEO client with 2G RAM ZEO server: 4G RAM, P4 Xeon Dual (RH Linux with 2.4.0 kernel) ZEO client: 2G RAM, P3 tualatin Dual (RH Linux with 2.4.0 kernel) It seems that the server never stops caching the objects it serves into memory until kswapd chimes in and starts paging away. kswapd was hogging all resources left and I had to reboot. I wasn't able to type a single keystroke remotely. argh.. a remote reboot attempt failed miserably... I did some research and found many reported problems with kswapd and kernel 2.4.0. What I'm worried about though is why ZEO let this happen. I did some experimentation after a clean reboot: adding and reading objects to the ZEO storage with the top utility running on a console. After an hour, ZEO was using more than 2G of RAM. The objects are just simple CMF/Plone stuff: documents and small sized images with no heavy-weight files. The traffic? Minimal. As I was the only one adding and reading stuff. And this might be a real stupid question but, when I flush cached objects using a ZEO client's management screen, what am I flushing? Objects from the ZEO server's memory or client's? Flushing objects didn't free up any memory on the ZEO server. Anybody ever had similar problems with ZEO (or Zope per se) hogging resources? How can I limit the number of objects the ZEO server itself is caching? Just hope it's not leaking memory. --------------------------------------------------------------- Wankyu Choi CEO/President NeoQuest Communications, Inc. http://www.zoper.net http://www.neoboard.net ---------------------------------------------------------------
I just stopped the ZEO server and found all that 2G memory it was using didn't free up at all. I'm running ZEO 2.01b1 Leaking? -----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org] On Behalf Of Wankyu Choi Sent: Wednesday, April 16, 2003 6:18 AM To: zope@zope.org Subject: [Zope] ZEO hogging too much memory Hi all, I had a nightmare with my ZEO setup last night. Had to drive to the data center to clean up the mess... in wee hours. (3AM) The ZEO server has 4G RAM and running on RedHat Linux 8 with 2.4.0 custom kernel. Thought it was plenty. I have one ZEO client with 2G RAM ZEO server: 4G RAM, P4 Xeon Dual (RH Linux with 2.4.0 kernel) ZEO client: 2G RAM, P3 tualatin Dual (RH Linux with 2.4.0 kernel) It seems that the server never stops caching the objects it serves into memory until kswapd chimes in and starts paging away. kswapd was hogging all resources left and I had to reboot. I wasn't able to type a single keystroke remotely. argh.. a remote reboot attempt failed miserably... I did some research and found many reported problems with kswapd and kernel 2.4.0. What I'm worried about though is why ZEO let this happen. I did some experimentation after a clean reboot: adding and reading objects to the ZEO storage with the top utility running on a console. After an hour, ZEO was using more than 2G of RAM. The objects are just simple CMF/Plone stuff: documents and small sized images with no heavy-weight files. The traffic? Minimal. As I was the only one adding and reading stuff. And this might be a real stupid question but, when I flush cached objects using a ZEO client's management screen, what am I flushing? Objects from the ZEO server's memory or client's? Flushing objects didn't free up any memory on the ZEO server. Anybody ever had similar problems with ZEO (or Zope per se) hogging resources? How can I limit the number of objects the ZEO server itself is caching? Just hope it's not leaking memory. --------------------------------------------------------------- Wankyu Choi CEO/President NeoQuest Communications, Inc. http://www.zoper.net http://www.neoboard.net --------------------------------------------------------------- _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Wankyu Choi wrote:
I just stopped the ZEO server and found all that 2G memory it was using didn't free up at all.
I'm running ZEO 2.01b1
Leaking?
I'm not an expert at linux kernel internals, but AFAIK even if the server is leaking, the kernel should reclaim the space when the process is stopped. You wrote you use kernel 2.4.0, that's very brave I'd say. I know it's easier said than done, but you better first install an newer linux kernel before looking at the zope stuff. You get at least the promise of a saner out-of-memory behavior. Other things you can do are setting process limits, so that the ZEO/zope servers cannot bring down the whole machine. There's also a zope specific product from chrism, autolance (search zope.org) which makes zope monitoring and restarting itself, when some conditions (memory etc.) are met. HTH, oliver
You wrote you use kernel 2.4.0, that's very brave I'd say. I know it's easier said than done, but you better first install an newer linux kernel before looking at the zope stuff. You get at least the promise of a saner out-of-memory behavior.
Stupid me... I was up 24 hours and out of my mind. I'm using kernel 2.4.20, not 2.4.0. Sorry for the confusion.
Other things you can do are setting process limits, so that the ZEO/zope servers cannot bring down the whole machine. There's also a zope specific product from chrism, autolance (search zope.org) which makes zope monitoring and restarting itself, when some conditions (memory etc.) are met.
Thanks for the pointer. Guess I'll also give djb's softlimit a try. But I am still worried about lost memory. I need to reboot the system again in an hour or so. 3G out of 4G is being used by no apparent process now even when I stop the ZEO server. Wankyu Choi --------------------------------------------------------------- Wankyu Choi CEO/President NeoQuest Communications, Inc. http://www.zoper.net http://www.neoboard.net ---------------------------------------------------------------
Wankyu Choi wrote:
You wrote you use kernel 2.4.0, that's very brave I'd say. I know it's easier said than done, but you better first install an newer linux kernel before looking at the zope stuff. You get at least the promise of a saner out-of-memory behavior.
Stupid me... I was up 24 hours and out of my mind. I'm using kernel 2.4.20, not 2.4.0. Sorry for the confusion.
No problem, but a pity cause this rules out a simple explanation.
Other things you can do are setting process limits, so that the ZEO/zope servers cannot bring down the whole machine. There's also a zope specific product from chrism, autolance (search zope.org) which makes zope monitoring and restarting itself, when some conditions (memory etc.) are met.
Thanks for the pointer. Guess I'll also give djb's softlimit a try.
But I am still worried about lost memory. I need to reboot the system again in an hour or so. 3G out of 4G is being used by no apparent process now even when I stop the ZEO server.
Just another remark, linux doesn't release all memory an app uses, even if the app exited. Don't ask me about the hows and whys. What does "free" tell you? cheers, oliver
Argh.. Toby Dickenson enlightened me on a grave mistake on my part. I was reading the output of top all wrong. The only problem now is what to do with that buggy kswapd kicking in when resources are scarce. Toby suggests rmap patches while ReiserFS guys suggests aa1. I'll give what Toby suggests a try first. I'm a user of DirectoryStorage after all :-) Thanks for your help. Cheers, Wankyu Choi --------------------------------------------------------------- Wankyu Choi CEO/President NeoQuest Communications, Inc. http://www.zoper.net http://www.neoboard.net --------------------------------------------------------------- -----Original Message----- From: Oliver Bleutgen [mailto:myzope@gmx.net] Sent: Wednesday, April 16, 2003 10:23 PM To: Wankyu Choi; zope@zope.org Subject: Re: [Zope] ZEO hogging too much memory Wankyu Choi wrote:
You wrote you use kernel 2.4.0, that's very brave I'd say. I know it's easier said than done, but you better first install an newer linux kernel before looking at the zope stuff. You get at least the promise of a saner out-of-memory behavior.
Stupid me... I was up 24 hours and out of my mind. I'm using kernel 2.4.20, not 2.4.0. Sorry for the confusion.
No problem, but a pity cause this rules out a simple explanation.
Other things you can do are setting process limits, so that the ZEO/zope servers cannot bring down the whole machine. There's also a zope specific product from chrism, autolance (search zope.org) which makes zope monitoring and restarting itself, when some conditions (memory etc.) are met.
Thanks for the pointer. Guess I'll also give djb's softlimit a try.
But I am still worried about lost memory. I need to reboot the system again in an hour or so. 3G out of 4G is being used by no apparent process now even when I stop the ZEO server.
Just another remark, linux doesn't release all memory an app uses, even if the app exited. Don't ask me about the hows and whys. What does "free" tell you? cheers, oliver
I forgot to post a followup to this thread. ( I did post a followup to the same thread on the DirectoryStroage maillinglist ;-) I solved this problem by applying the aa1 patch to the 2.4.20 kernel. If any of you experience a similar problem ( kswapd hogging the resources and never letting it go, etc ) try the aa1 patch. I also tried gentoo patches, but they didn't fix the problem. The problem was: when your low memory is scarce, kswapd kicks in and hogs all the remaining resources. A simple packing attempt of more than 1G storage would reproduce this madness with a vanilla 2.4.20 kernel. Even when things get cool, kswapd wouldn't let go the resources it's hogging. The cached memory, for example, is there for you to see but none of your processes can't retrieve any of it. The faster your IO devices (I've got a hardware raid, for example), the worse this problem gets. Try the following patch: http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.20aa1. bz2 My problems gone with this patch. Cheers, Wankyu Choi --------------------------------------------------------------- Wankyu Choi CEO/President NeoQuest Communications, Inc. http://www.zoper.net http://www.neoboard.net --------------------------------------------------------------- -----Original Message----- From: Oliver Bleutgen [mailto:myzope@gmx.net] Sent: Wednesday, April 16, 2003 9:02 PM To: Wankyu Choi Cc: zope@zope.org Subject: Re: [Zope] ZEO hogging too much memory Wankyu Choi wrote:
I just stopped the ZEO server and found all that 2G memory it was using didn't free up at all.
I'm running ZEO 2.01b1
Leaking?
I'm not an expert at linux kernel internals, but AFAIK even if the server is leaking, the kernel should reclaim the space when the process is stopped. You wrote you use kernel 2.4.0, that's very brave I'd say. I know it's easier said than done, but you better first install an newer linux kernel before looking at the zope stuff. You get at least the promise of a saner out-of-memory behavior. Other things you can do are setting process limits, so that the ZEO/zope servers cannot bring down the whole machine. There's also a zope specific product from chrism, autolance (search zope.org) which makes zope monitoring and restarting itself, when some conditions (memory etc.) are met. HTH, oliver
participants (2)
-
Oliver Bleutgen -
Wankyu Choi