[Zope] Better Memeory Management Feature Request

[Andreas Kostyrka]

On Wed, 10 Feb 1999, Paul Everitt wrote:

> 
> Scott Robertson wrote:
> > We Need some sort of method to keep memory usage in check.  Currently a
> > brand new instance with an empty database will consume 4 MB of
> 
> Hey, that's less than Netscape running on Linux! :^)
But as you've surely noticed, netscape warns if one user starts more than
one instance. ;) OTOH, an ISP supporting Zope might expect to run dozens
or hundreds of instances on one box.

> Hmm, though as you know I'm a rube, I beg to differ.  Since I believe
> everything I hear, I can repeat the following with a straight face.  In
> fact, I'm almost ashamed to send this email, thus publicly demonstrating
> my complete stupidity on the subject.  But hey, it's only email.
> 
> Python apps like Zope allocate memory then release memory.  What the
> operating system does with the released memory is its problem.  Most
> Unices seem to keep the virtual size at the highest point allocated by a
> process.  That is, the number will won't automatically go down (though
That's not really true for any current Linux system.
Linux basically offers two ways to get memory:
-) "Traditional" Unix way with brk(), which adjusts the size of the data
   segment. Because it's basically sets the length of the segment, it's
   seldom the case that the process could release memory, so most C
   libraries don't bother the overhead of checking if some memory is
   freeable this way.
-) anonymous mmaps. This is done for bigger allocs on Linux (at least),
   and there exists a munmap syscall :)

> the real size will).  If the operating system decides that something
> else needs the memory, it will take the memory back.  NT, however, will
> do so.
Linux will do so too :)

Ok, on a RH5.x system, with python 1.5.2b1:
Python 1.5.2b1 (#1, Feb  3 1999, 20:02:48)  [GCC egcs-2.90.27 980315 (egc
on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import os
>>> def showmem(): os.system("ps axm | head -1 ; ps axm | grep py | grep
-v grep")
... 
>>> showmem()
  PID TTY MAJFLT MINFLT   TRS   DRS  SIZE  SWAP   RSS  SHRD   LIB  DT COMMAND
12918  p3    262    103   276   956  1232     0  1232   836     0  99 python 
>>> a="A"*64*1024
>>> showmem()
  PID TTY MAJFLT MINFLT   TRS   DRS  SIZE  SWAP   RSS  SHRD   LIB  DT COMMAND
12918  p3    262    153   276  1020  1296     0  1296   836     0 115 python 
>>> l=[]
>>> for i in range(100): l.append(a+str(i))
... 
>>> showmem()
  PID TTY MAJFLT MINFLT   TRS   DRS  SIZE  SWAP   RSS  SHRD   LIB  DT COMMAND
12918  p3    262   1797   276  7424  7700     0  7700   836     0 1716 python 
>>> del l
>>> showmem()
  PID TTY MAJFLT MINFLT   TRS   DRS  SIZE  SWAP   RSS  SHRD   LIB  DT COMMAND
12918  p3    262   1933   276  1020  1296     0  1296   836     0 115 python 


> Or perhaps I've got it all wrong.
Not completely. Small allocations are recycled in-process under Unix
(usually), to save upon ``expensive'' user-to-kernel-mode switches.

> Anyways, we had a major, major app -- the classified ad system that
> spawned Principia and then Zope -- where the customer insisted on
> running each newspaper in its own process.  Then complained when their
> memory footprint swelled to gigs.  We *easily* spent 300 manhours trying
> to do something here (actually, this is the genesis of the tunable
> object cache).
The problem here are numbers. Question: How many different newspapers did
your customer have? Probably a smaller number than the number of virtual
hosts an average ISPs puts on one box. 

Even 4MB per virtual host must be considered ``expensive'', when compared
to the typical unmeasurable memory usage of a virtual host in Apache.

I know that Zope provides MUCH more than a just a HTTP server, but some
ISP might have considered Zope as an User interface even for static
serving. 

 > 
> > flush the cache the process still remains the same  size. For that matter
> > we're unable to decrease the number of objects stored in the cache as
> > well. Maybe we are using it wrong?
> > 
> > It would be nice to be able to set a quota of 10 MB to the total amount
> > of memory each process uses. Or to have more aggressive garbage
> > collection.
> 
> Some requirements questions -- how do you want the policy to be
> implemented?
> 
> a. Never let the process size go above the number.
> 
> b. Check before each request to see if the process size is above the
> number.
> 
> c. Check after.

d.) set a soft ulimit.

> 
> When the process size is exceeded by the first byte, what would you like
> to do?
When the signal comes in, start to aggressivly to garbage collect, etc.

>  
> > If we can get a fix on this issue then running multiple instances of Zope
> > from the same box should be a breeze. (Although maybe expensive, but I've
> > always wanted to have a box with 1GB of RAM)
> 
> Are you running each of them with different Pythons?  If you run them
> with the same Python, will they share the code segment thingy for the
> interpreter?
See above, as python is an interpreter, the code segment is not that big:
After starting an interactive python:
TRS      276
DRS      956

And the 276Kb are shared mostly so or so on a Linux system, which tend to
share pure code pages.

Andreas
-- 
Win95: n., A huge annoying boot virus that causes random spontaneous system
     crashes, usually just before saving a massive project.  Easily cured by
     UNIX.  See also MS-DOS, IBM-DOS, DR-DOS, Win 3.x, Win98.