Better Memeory Management Feature Request
I just posted this in the Collector but I wanted to get some feedback from the list as well. That way those greedy bastards at DC won't get all the fun of fixing and extending zope. We Need some sort of method to keep memory usage in check. Currently a brand new instance with an empty database will consume 4 MB of memory. If you upload 1 or 2 6 MB files into this database the running process swells up to 18 MB in memory and no matter how many times you flush the cache the process still remains the same size. For that matter we're unable to decrease the number of objects stored in the cache as well. Maybe we are using it wrong? It would be nice to be able to set a quota of 10 MB to the total amount of memory each process uses. Or to have more aggressive garbage collection. If we can get a fix on this issue then running multiple instances of Zope from the same box should be a breeze. (Although maybe expensive, but I've always wanted to have a box with 1GB of RAM) --------------------------------------------------- - Scott Robertson Phone: 714.972.2299 - - CodeIt Computing Fax: 714.972.2399 - - http://codeit.com - ---------------------------------------------------
Scott Robertson wrote:
I just posted this in the Collector but I wanted to get some feedback from the list as well. That way those greedy bastards at DC won't get all the fun of fixing and extending zope.
Oh, we're far too tired to be greedy. :^) Funny, getting jacked by your ISP (as we discussed on the phone today) and having to spend four weeks dialing our LAN into the net over a modem gives you a new...perspective. All I can say is, thank goodness for CodeIt getting the zope.org machine online so fast. (Does anybody else notice how much faster the Zope site connectivity is now? Pretty sweet...)
We Need some sort of method to keep memory usage in check. Currently a brand new instance with an empty database will consume 4 MB of
Hey, that's less than Netscape running on Linux! :^)
memory. If you upload 1 or 2 6 MB files into this database the running process swells up to 18 MB in memory and no matter how many times you
Hmm, though as you know I'm a rube, I beg to differ. Since I believe everything I hear, I can repeat the following with a straight face. In fact, I'm almost ashamed to send this email, thus publicly demonstrating my complete stupidity on the subject. But hey, it's only email. Python apps like Zope allocate memory then release memory. What the operating system does with the released memory is its problem. Most Unices seem to keep the virtual size at the highest point allocated by a process. That is, the number will won't automatically go down (though the real size will). If the operating system decides that something else needs the memory, it will take the memory back. NT, however, will do so. Or perhaps I've got it all wrong. Anyways, we had a major, major app -- the classified ad system that spawned Principia and then Zope -- where the customer insisted on running each newspaper in its own process. Then complained when their memory footprint swelled to gigs. We *easily* spent 300 manhours trying to do something here (actually, this is the genesis of the tunable object cache).
flush the cache the process still remains the same size. For that matter we're unable to decrease the number of objects stored in the cache as well. Maybe we are using it wrong?
It would be nice to be able to set a quota of 10 MB to the total amount of memory each process uses. Or to have more aggressive garbage collection.
Some requirements questions -- how do you want the policy to be implemented? a. Never let the process size go above the number. b. Check before each request to see if the process size is above the number. c. Check after. When the process size is exceeded by the first byte, what would you like to do?
If we can get a fix on this issue then running multiple instances of Zope from the same box should be a breeze. (Although maybe expensive, but I've always wanted to have a box with 1GB of RAM)
Are you running each of them with different Pythons? If you run them with the same Python, will they share the code segment thingy for the interpreter? What is the RSS in the above example, or in the case where you have four Zopes running? --Paul
On Wed, 10 Feb 1999, Paul Everitt wrote:
Scott Robertson wrote:
We Need some sort of method to keep memory usage in check. Currently a brand new instance with an empty database will consume 4 MB of
Hey, that's less than Netscape running on Linux! :^)
But as you've surely noticed, netscape warns if one user starts more than one instance. ;) OTOH, an ISP supporting Zope might expect to run dozens or hundreds of instances on one box.
Hmm, though as you know I'm a rube, I beg to differ. Since I believe everything I hear, I can repeat the following with a straight face. In fact, I'm almost ashamed to send this email, thus publicly demonstrating my complete stupidity on the subject. But hey, it's only email.
Python apps like Zope allocate memory then release memory. What the operating system does with the released memory is its problem. Most Unices seem to keep the virtual size at the highest point allocated by a process. That is, the number will won't automatically go down (though That's not really true for any current Linux system. Linux basically offers two ways to get memory: -) "Traditional" Unix way with brk(), which adjusts the size of the data segment. Because it's basically sets the length of the segment, it's seldom the case that the process could release memory, so most C libraries don't bother the overhead of checking if some memory is freeable this way. -) anonymous mmaps. This is done for bigger allocs on Linux (at least), and there exists a munmap syscall :)
the real size will). If the operating system decides that something else needs the memory, it will take the memory back. NT, however, will do so. Linux will do so too :)
Ok, on a RH5.x system, with python 1.5.2b1: Python 1.5.2b1 (#1, Feb 3 1999, 20:02:48) [GCC egcs-2.90.27 980315 (egc on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
import os def showmem(): os.system("ps axm | head -1 ; ps axm | grep py | grep -v grep") ... showmem() PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP RSS SHRD LIB DT COMMAND 12918 p3 262 103 276 956 1232 0 1232 836 0 99 python a="A"*64*1024 showmem() PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP RSS SHRD LIB DT COMMAND 12918 p3 262 153 276 1020 1296 0 1296 836 0 115 python l=[] for i in range(100): l.append(a+str(i)) ... showmem() PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP RSS SHRD LIB DT COMMAND 12918 p3 262 1797 276 7424 7700 0 7700 836 0 1716 python del l showmem() PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP RSS SHRD LIB DT COMMAND 12918 p3 262 1933 276 1020 1296 0 1296 836 0 115 python
Or perhaps I've got it all wrong. Not completely. Small allocations are recycled in-process under Unix (usually), to save upon ``expensive'' user-to-kernel-mode switches.
Anyways, we had a major, major app -- the classified ad system that spawned Principia and then Zope -- where the customer insisted on running each newspaper in its own process. Then complained when their memory footprint swelled to gigs. We *easily* spent 300 manhours trying to do something here (actually, this is the genesis of the tunable object cache). The problem here are numbers. Question: How many different newspapers did your customer have? Probably a smaller number than the number of virtual hosts an average ISPs puts on one box.
Even 4MB per virtual host must be considered ``expensive'', when compared to the typical unmeasurable memory usage of a virtual host in Apache. I know that Zope provides MUCH more than a just a HTTP server, but some ISP might have considered Zope as an User interface even for static serving.
flush the cache the process still remains the same size. For that matter we're unable to decrease the number of objects stored in the cache as well. Maybe we are using it wrong?
It would be nice to be able to set a quota of 10 MB to the total amount of memory each process uses. Or to have more aggressive garbage collection.
Some requirements questions -- how do you want the policy to be implemented?
a. Never let the process size go above the number.
b. Check before each request to see if the process size is above the number.
c. Check after.
d.) set a soft ulimit.
When the process size is exceeded by the first byte, what would you like to do?
When the signal comes in, start to aggressivly to garbage collect, etc.
If we can get a fix on this issue then running multiple instances of Zope from the same box should be a breeze. (Although maybe expensive, but I've always wanted to have a box with 1GB of RAM)
Are you running each of them with different Pythons? If you run them with the same Python, will they share the code segment thingy for the interpreter?
See above, as python is an interpreter, the code segment is not that big: After starting an interactive python: TRS 276 DRS 956 And the 276Kb are shared mostly so or so on a Linux system, which tend to share pure code pages. Andreas -- Win95: n., A huge annoying boot virus that causes random spontaneous system crashes, usually just before saving a massive project. Easily cured by UNIX. See also MS-DOS, IBM-DOS, DR-DOS, Win 3.x, Win98.
On Wed, 10 Feb 1999, Paul Everitt wrote:
Oh, we're far too tired to be greedy. :^) Funny, getting jacked by your ISP (as we discussed on the phone today) and having to spend four weeks dialing our LAN into the net over a modem gives you a new...perspective. All I can say is, thank goodness for CodeIt getting the zope.org machine online so fast.
(Does anybody else notice how much faster the Zope site connectivity is now? Pretty sweet...)
Like that huh? =)
We Need some sort of method to keep memory usage in check. Currently a brand new instance with an empty database will consume 4 MB of
Hey, that's less than Netscape running on Linux! :^)
True, and as soon as I can figure out away to charge people to run multiple instances of netscape navigator, I will probally post on the Mozilla list that their needs to be better memory management.
memory. If you upload 1 or 2 6 MB files into this database the running process swells up to 18 MB in memory and no matter how many times you
Hmm, though as you know I'm a rube, I beg to differ. Since I believe everything I hear, I can repeat the following with a straight face. In fact, I'm almost ashamed to send this email, thus publicly demonstrating my complete stupidity on the subject. But hey, it's only email.
Python apps like Zope allocate memory then release memory. What the operating system does with the released memory is its problem. Most Unices seem to keep the virtual size at the highest point allocated by a process. That is, the number will won't automatically go down (though the real size will). If the operating system decides that something else needs the memory, it will take the memory back. NT, however, will do so.
Or perhaps I've got it all wrong.
You might be correct. It sounds good in theory any way, guess I will have to test it. If you're wrong I might have to come back and kill you for lying to me, but I'm sure you will understand.
Anyways, we had a major, major app -- the classified ad system that spawned Principia and then Zope -- where the customer insisted on running each newspaper in its own process. Then complained when their memory footprint swelled to gigs. We *easily* spent 300 manhours trying to do something here (actually, this is the genesis of the tunable object cache).
flush the cache the process still remains the same size. For that matter we're unable to decrease the number of objects stored in the cache as well. Maybe we are using it wrong?
It would be nice to be able to set a quota of 10 MB to the total amount of memory each process uses. Or to have more aggressive garbage collection.
Some requirements questions -- how do you want the policy to be implemented?
This is one of those hard questions to anwser. I am a firm beliver in letting the user use as much as much resources as they want (and charge them of course) as long as it dosen't affect others. I need to be able to say that your zope site well run at the minimum this fast, regardless of what anybody else is currently doing.
a. Never let the process size go above the number.
We will probally have to put a hard limit in their somewhere.
b. Check before each request to see if the process size is above the number.
Sounds to slow (or maybe preceived slow) .
c. Check after.
This sounds better. Temporarily going over the limits should be ok. I plan to have a bit of swap space around for those occasions when someone's site is not behaving.
When the process size is exceeded by the first byte, what would you like to do?
Either stop adding things to the cache in other words make it go back to the DB each time. and/or flush the cache if possible. Is it possible to look at the cache as seperate from the process? In other words put the limit on just the cache not the process as a whole? This might make things easier.
If we can get a fix on this issue then running multiple instances of Zope from the same box should be a breeze. (Although maybe expensive, but I've always wanted to have a box with 1GB of RAM)
Are you running each of them with different Pythons? If you run them with the same Python, will they share the code segment thingy for the interpreter?
Your guess is as good as mine. I think that it needs a brand new instance of python for each instance of zope (?) but I'm not sure.
What is the RSS in the above example, or in the case where you have four Zopes running?
An empty DB is 1736. I'll get back to you on one of the bloated ones. --------------------------------------------------- - Scott Robertson Phone: 714.972.2299 - - CodeIt Computing Fax: 714.972.2399 - - http://codeit.com - ---------------------------------------------------
On Wed, Feb 10, 1999 at 05:29:22PM -0800, Scott Robertson wrote: ,----- | On Wed, 10 Feb 1999, Paul Everitt wrote: | | > What is the RSS in the above example, or in the case where you have four | > Zopes running? | > | | An empty DB is 1736. I'll get back to you on one of the bloated ones. | `----- Unless it's considered proprietary, I'd be curious to know what the resident size of the Zope process for Zope.org is (or is that the one that was shooting up to 18MB?) This is just kind of a reference point, because I have seen the kind and quantity of things on that site. Also, pardon my naivete on this one, but: RAM is relatively cheap these days. Even PCs can be considered relatively cheap these days. If, to cover the cost of having a 10-12MB process resident in memory, I had to pay a $100 startup fee or some such, I would probably do so. After paying that fee, though, I would expect to pay the same as any other hosting customer. Feel free to let me know if I'm way off base :) Kevin -- Kevin Dangoor kid@ans.net / 734-214-7349
On Wed, 10 Feb 1999, Kevin Dangoor wrote:
Also, pardon my naivete on this one, but: RAM is relatively cheap these But: Often the hardware has a limit, and Linux as such does have one too at this moment (I believe it's 2GB on x86). Most Super7 motherboards do have an cacherange of 128MB. etc.
18/128: 14% 18/2048: 0.8%
days. Even PCs can be considered relatively cheap these days. If, to cover the cost of having a 10-12MB process resident in memory, I had to pay a $100 startup fee or some such, I would probably do so. After paying that fee, though, I would expect to pay the same as any other hosting customer. The problem here is, that you are using a measurable part of the hosting server (memorywise) compared to the Apache VirtualHost, which doesn't seem to use measurable amounts of memory per VirtualHost :(
Andreas -- Win95: n., A huge annoying boot virus that causes random spontaneous system crashes, usually just before saving a massive project. Easily cured by UNIX. See also MS-DOS, IBM-DOS, DR-DOS, Win 3.x, Win98.
Kevin Dangoor wrote:
Unless it's considered proprietary, I'd be curious to know what the resident size of the Zope process for Zope.org is (or is that the one that was shooting up to 18MB?) This is just kind of a reference point, because I have seen the kind and quantity of things on that site.
Proprietary...not sure I remember that term :^) The zope.org RSS is 21.5 Mb. I'm not sure whether to think this is good or bad...we upload all the distributions and PDF files into the database. Anyway, this is a *very* good conversation to be having. Our newspaper folks had 80+ newspapers, each running two processes. It was on a 64-bit machine (Digital Unix), meaning everything took more memory. We had to work hard to keep the processes at an average of 15 Mb. Thus, doing the math, we were talking about 2.4 Gb of total memory consumed. Needless to say, we tried to encourage them to run several papers in one process. So Zope needs to be able to host 100+ "virtual hosts". If the direction is doing them as separate processes/installations, then every little bit helps. This is a perfect job for open source. There are folks on this list with a lot of talent at looking at these kinds of things and running tools and tests to see what happens. --Paul
On 10 Feb 99, at 23:22, Kevin Dangoor wrote:
Also, pardon my naivete on this one, but: RAM is relatively cheap these days. Even PCs can be considered relatively cheap these days. If, to cover the cost of having a 10-12MB process resident in memory, I had to pay a $100 startup fee or some such, I would probably do so. After paying that fee, though, I would expect to pay the same as any other hosting customer.
Feel free to let me know if I'm way off base :)
Well... I've been thinking about porting Python (and Zope) to NetWare. You know, that's the operating system where everything runs at ring 0. That's also the OS that doesn't have any virtual memory capability. Naturally this issue is of interest to me. ;-) Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com ICQ: 14856937
On Wed, 10 Feb 1999, Kevin Dangoor wrote:
Unless it's considered proprietary, I'd be curious to know what the resident size of the Zope process for Zope.org is (or is that the one that was shooting up to 18MB?) This is just kind of a reference point, because I have seen the kind and quantity of things on that site.
Also, pardon my naivete on this one, but: RAM is relatively cheap these days. Even PCs can be considered relatively cheap these days. If, to cover the cost of having a 10-12MB process resident in memory, I had to pay a $100 startup fee or some such, I would probably do so. After paying that fee, though, I would expect to pay the same as any other hosting customer.
Feel free to let me know if I'm way off base :)
You are very correct. That's why are first Zope Virtual Hosting Machine will have between .5GB-1GB of memory to start with. But it only took me 5 minutes to get the process to swell up to 18MB. That means some industrious/mallicious user could conceivably swell theirs up to 100MB even the full GB and then nobody else would get to play. --------------------------------------------------- - Scott Robertson Phone: 714.972.2299 - - CodeIt Computing Fax: 714.972.2399 - - http://codeit.com - ---------------------------------------------------
On Thu, Feb 11, 1999 at 08:13:40AM -0800, Scott Robertson wrote:
You are very correct. That's why are first Zope Virtual Hosting Machine will have between .5GB-1GB of memory to start with. But it only took me 5 minutes to get the process to swell up to 18MB. That means some industrious/mallicious user could conceivably swell theirs up to 100MB even the full GB and then nobody else would get to play.
Any know what happens when Pyhton/Zope hits a resource limit on the machine? :-) If it degrades "ok" then you could at least instigate limits :-) You could even set Zope to crank down its own limit. Chris -- | Christopher Petrilli | petrilli@amber.org
participants (6)
-
Andreas Kostyrka -
Brad Clements -
Christopher G. Petrilli -
Kevin Dangoor -
Paul Everitt -
Scott Robertson