[Zope] Better Memeory Management Feature Request

[Scott Robertson]

On Wed, 10 Feb 1999, Paul Everitt wrote:

> 
> Oh, we're far too tired to be greedy.  :^)  Funny, getting jacked by
> your ISP (as we discussed on the phone today) and having to spend four
> weeks dialing our LAN into the net over a modem gives you a
> new...perspective.  All I can say is, thank goodness for CodeIt getting
> the zope.org machine online so fast.
> 
> (Does anybody else notice how much faster the Zope site connectivity is
> now?  Pretty sweet...)

Like that huh? =)

>  
> > We Need some sort of method to keep memory usage in check.  Currently a
> > brand new instance with an empty database will consume 4 MB of
> 
> Hey, that's less than Netscape running on Linux! :^)

True, and as soon as I can figure out away to charge people to run
multiple instances of netscape navigator, I will probally post on the
Mozilla list that their needs to be better memory management.


 > 
> > memory. If you upload 1 or 2 6 MB files into this database the running
> > process swells up to 18 MB in memory and no matter how many times you
> 
> Hmm, though as you know I'm a rube, I beg to differ.  Since I believe
> everything I hear, I can repeat the following with a straight face.  In
> fact, I'm almost ashamed to send this email, thus publicly demonstrating
> my complete stupidity on the subject.  But hey, it's only email.
> 
> Python apps like Zope allocate memory then release memory.  What the
> operating system does with the released memory is its problem.  Most
> Unices seem to keep the virtual size at the highest point allocated by a
> process.  That is, the number will won't automatically go down (though
> the real size will).  If the operating system decides that something
> else needs the memory, it will take the memory back.  NT, however, will
> do so.
> 
> Or perhaps I've got it all wrong.

You might be correct. It sounds good in theory any way, guess I will have
to test it. If you're wrong I might have to come back and kill you for
lying to me, but I'm sure you will understand.  


> 
> Anyways, we had a major, major app -- the classified ad system that
> spawned Principia and then Zope -- where the customer insisted on
> running each newspaper in its own process.  Then complained when their
> memory footprint swelled to gigs.  We *easily* spent 300 manhours trying
> to do something here (actually, this is the genesis of the tunable
> object cache).
> 
> > flush the cache the process still remains the same  size. For that matter
> > we're unable to decrease the number of objects stored in the cache as
> > well. Maybe we are using it wrong?
> > 
> > It would be nice to be able to set a quota of 10 MB to the total amount
> > of memory each process uses. Or to have more aggressive garbage
> > collection.
> 
> Some requirements questions -- how do you want the policy to be
> implemented?

This is one of those hard questions to anwser. I am a firm beliver in
letting the user use as much as much resources as they want (and charge
them of course) as long as it dosen't affect others. I need to be able to
say that your zope site well run at the minimum this fast, regardless  
of what anybody else is currently doing.

> 
> a. Never let the process size go above the number.

We will probally have to put a hard limit in their somewhere. 

> 
> b. Check before each request to see if the process size is above the
> number.

Sounds to slow (or maybe preceived slow) . 

> 
> c. Check after.

This sounds better. Temporarily going over the limits should be ok. I plan
to have a bit of swap space around for those occasions when someone's site
is not behaving.

> 
> When the process size is exceeded by the first byte, what would you like
> to do?

Either stop adding things to the cache in other words make it go back to
the DB each time. and/or flush the cache if possible.

Is it possible to look at the cache as seperate from the process? In other
words put the limit on just the cache not the process as a whole? This
might make things easier.


>  
> > If we can get a fix on this issue then running multiple instances of Zope
> > from the same box should be a breeze. (Although maybe expensive, but I've
> > always wanted to have a box with 1GB of RAM)
> 
> Are you running each of them with different Pythons?  If you run them
> with the same Python, will they share the code segment thingy for the
> interpreter?

Your guess is as good as mine. I think that it needs a brand new instance
of python for each instance of zope (?) but I'm not sure.

> 
> What is the RSS in the above example, or in the case where you have four
> Zopes running?
> 

An empty DB is 1736. I'll get back to you on one of the bloated ones. 

---------------------------------------------------
- Scott Robertson             Phone: 714.972.2299 -
- CodeIt Computing            Fax:   714.972.2399 -
-                http://codeit.com                -
---------------------------------------------------