On Wed, 10 Feb 1999, Paul Everitt wrote:
Scott Robertson wrote:
We Need some sort of method to keep memory usage in check. Currently a brand new instance with an empty database will consume 4 MB of
Hey, that's less than Netscape running on Linux! :^)
But as you've surely noticed, netscape warns if one user starts more than one instance. ;) OTOH, an ISP supporting Zope might expect to run dozens or hundreds of instances on one box.
Hmm, though as you know I'm a rube, I beg to differ. Since I believe everything I hear, I can repeat the following with a straight face. In fact, I'm almost ashamed to send this email, thus publicly demonstrating my complete stupidity on the subject. But hey, it's only email.
Python apps like Zope allocate memory then release memory. What the operating system does with the released memory is its problem. Most Unices seem to keep the virtual size at the highest point allocated by a process. That is, the number will won't automatically go down (though That's not really true for any current Linux system. Linux basically offers two ways to get memory: -) "Traditional" Unix way with brk(), which adjusts the size of the data segment. Because it's basically sets the length of the segment, it's seldom the case that the process could release memory, so most C libraries don't bother the overhead of checking if some memory is freeable this way. -) anonymous mmaps. This is done for bigger allocs on Linux (at least), and there exists a munmap syscall :)
the real size will). If the operating system decides that something else needs the memory, it will take the memory back. NT, however, will do so. Linux will do so too :)
Ok, on a RH5.x system, with python 1.5.2b1: Python 1.5.2b1 (#1, Feb 3 1999, 20:02:48) [GCC egcs-2.90.27 980315 (egc on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
import os def showmem(): os.system("ps axm | head -1 ; ps axm | grep py | grep -v grep") ... showmem() PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP RSS SHRD LIB DT COMMAND 12918 p3 262 103 276 956 1232 0 1232 836 0 99 python a="A"*64*1024 showmem() PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP RSS SHRD LIB DT COMMAND 12918 p3 262 153 276 1020 1296 0 1296 836 0 115 python l=[] for i in range(100): l.append(a+str(i)) ... showmem() PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP RSS SHRD LIB DT COMMAND 12918 p3 262 1797 276 7424 7700 0 7700 836 0 1716 python del l showmem() PID TTY MAJFLT MINFLT TRS DRS SIZE SWAP RSS SHRD LIB DT COMMAND 12918 p3 262 1933 276 1020 1296 0 1296 836 0 115 python
Or perhaps I've got it all wrong. Not completely. Small allocations are recycled in-process under Unix (usually), to save upon ``expensive'' user-to-kernel-mode switches.
Anyways, we had a major, major app -- the classified ad system that spawned Principia and then Zope -- where the customer insisted on running each newspaper in its own process. Then complained when their memory footprint swelled to gigs. We *easily* spent 300 manhours trying to do something here (actually, this is the genesis of the tunable object cache). The problem here are numbers. Question: How many different newspapers did your customer have? Probably a smaller number than the number of virtual hosts an average ISPs puts on one box.
Even 4MB per virtual host must be considered ``expensive'', when compared to the typical unmeasurable memory usage of a virtual host in Apache. I know that Zope provides MUCH more than a just a HTTP server, but some ISP might have considered Zope as an User interface even for static serving.
flush the cache the process still remains the same size. For that matter we're unable to decrease the number of objects stored in the cache as well. Maybe we are using it wrong?
It would be nice to be able to set a quota of 10 MB to the total amount of memory each process uses. Or to have more aggressive garbage collection.
Some requirements questions -- how do you want the policy to be implemented?
a. Never let the process size go above the number.
b. Check before each request to see if the process size is above the number.
c. Check after.
d.) set a soft ulimit.
When the process size is exceeded by the first byte, what would you like to do?
When the signal comes in, start to aggressivly to garbage collect, etc.
If we can get a fix on this issue then running multiple instances of Zope from the same box should be a breeze. (Although maybe expensive, but I've always wanted to have a box with 1GB of RAM)
Are you running each of them with different Pythons? If you run them with the same Python, will they share the code segment thingy for the interpreter?
See above, as python is an interpreter, the code segment is not that big: After starting an interactive python: TRS 276 DRS 956 And the 276Kb are shared mostly so or so on a Linux system, which tend to share pure code pages. Andreas -- Win95: n., A huge annoying boot virus that causes random spontaneous system crashes, usually just before saving a massive project. Easily cured by UNIX. See also MS-DOS, IBM-DOS, DR-DOS, Win 3.x, Win98.