[Zope] Counting Memory Usage aka. Reading Top output.

[Bill Anderson]

"J. Atwood" wrote:
> 
> You read correctly, just not the entire post. I said that the machine was
> broken and was not seeing the 128 MB of RAM I have installed, instead it was
> only using the 12 on the system board (DELL).

Been there. :)

append "mem=128M" to lilo (no quotes) for the reboot (as root: 'lilo -R
<imagename> append mem=128M' where <imagename> is the kenrel image you
boot to). Should work fine ... unless of course the RAM (or motherboard)
is dead :<


> As for Python sucking up memory (and one of the reasons I posted) it is
> true. I have it on another machine with 256 MB or memory and here is what it
> is doing..
> 
> TSIZE  SIZE SWAP  RSS SHARE STAT  LIB %CPU %MEM  CTIME COMMAND
> 403 44376    0  43M  1424 S       0  0.0 17.2   6:08 python
> 403 44376    0  43M  1424 S       0  0.0 17.2   0:00 python
> 403 44376    0  43M  1424 S       0  0.0 17.2   2:53 python
> 403 44376    0  43M  1424 S       0  0.0 17.2   1:33 python
> 403 44376    0  43M  1424 S       0  0.0 17.2   7:57 python
> 403 44376    0  43M  1424 S       0  0.0 17.2   6:11 python
> 
> That is 43MB per!

NO, IT ISN'T! 

Sorry if that sounds snappy, but I have grown quite weary of all these
repeated claims. That is not how it is, and this has been posted many,
many times. For those who really need a low level explanations, a trip
down memory lane, combined with a visit to the Kernel list is in order.

Think about it, do the math.
Is this machine swapping for every action it takes? It would have to be
_IF_ you were correct. Just waht you pasted above would be consuming
258MB ram. Nearly _anything_ you did with your machine would require
swapping. Ask youself this: What are the chances that _every_ thread is
consuming the *exact* same amount of all resources reported, *all* the
time, regardless of what any individual thread's task?
 
I realize that not everyone here is versed in the particulars, but I
would hope that we (collectively) could realize (after repeated posts by
more than one or two people) that on Linux (at least), when you look at
the processes' data, threads report the totals, not individual usage.
One of my machines has anywhere between 5 and 10 Zservers running at any
given time. many have thread counts of around 6. I also have other
processes spawning _lots_ of threads. If I added up the 'totals' on a
'per-thread basis', It would amount to a lot more memory than is in the
machine. Specifically, the above 'formula' would claim that my Roxen
server on this machine is using (22 threads * 10MB/thread) 220MB of
physical RAM! Roxen is certainly _not_ using 220MB.


Yet I can compile, start up an additional Xserver, etc. with _no_
swapping (until I _actually_ exceed the physical RAM).

It would certainly help us track down the alleged memory leak if we
could get to the actual data, and not the assumptions.

BTW, RSS _includes_ library pages for ELF processes.

> Zope uses what it can take. On the machine with 12MB it uses this..
> 
> SIZE SWAP  RSS SHARE STAT  LIB %CPU %MEM  CTIME COMMAND
> 6592  816   332 S       0  0.0  5.9   3:06 python
> 6592  816   332 S       0  0.0  5.9   0:00 python
> 6592  816   332 S       0  0.0  5.9   2:04 python
> 6592  816   332 S       0  0.0  5.9   1:54 python
> 6592  816   332 S       0  0.0  5.9   1:39 python
> 6592  816   332 S       0  0.0  5.9   2:46 python
> 
> I am sure on a machine with 5 GB it would use most of it. Such is the nature
> of Linux (I think). It goes up and down with usage and with packing the
> database.

AIUI, Python limits the memory in that if it isn't available, it doesn't
try to force the issue. If it is available, _and_ it needs it, it will
take it. In this case, it is only taking what it can get; that doesn't
mean it doesn't _need_ more. :)

I hope that all didn't come off wrong (mean or something). As I said, it
is really hard to verify and track memory leak claims with incorrect
data. When you are serving/proposing Zope with/to Fortune 500 companies,
this becomes a very important issue, believe me. I am not saying there
is not a memory leak, just that all these incorrec tposts are making the
process of verifying and identifying one very difficult. That and
those-who-do't-want-Zope-to-be-used see these posts(yes, they monitor
the list solely to see problems) and my customers/prospects come
questioning me about it. Then I have to explain this whole thing to
them. The we, as a community, look bad.

Bill


-- 
In flying I have learned that carelessness and overconfidence are 
usually far more dangerous than deliberately accepted risks. 
          -- Wilbur Wright in a letter to his father, September 1900