[Zope] Better Memeory Management Feature Request

[Paul Everitt]

Scott Robertson wrote:
> I just posted this in the Collector but I wanted to get some feedback from
> the list as well. That way those greedy bastards at DC won't get all the
> fun of fixing and extending zope.

Oh, we're far too tired to be greedy.  :^)  Funny, getting jacked by
your ISP (as we discussed on the phone today) and having to spend four
weeks dialing our LAN into the net over a modem gives you a
new...perspective.  All I can say is, thank goodness for CodeIt getting
the zope.org machine online so fast.

(Does anybody else notice how much faster the Zope site connectivity is
now?  Pretty sweet...)
 
> We Need some sort of method to keep memory usage in check.  Currently a
> brand new instance with an empty database will consume 4 MB of

Hey, that's less than Netscape running on Linux! :^)

> memory. If you upload 1 or 2 6 MB files into this database the running
> process swells up to 18 MB in memory and no matter how many times you

Hmm, though as you know I'm a rube, I beg to differ.  Since I believe
everything I hear, I can repeat the following with a straight face.  In
fact, I'm almost ashamed to send this email, thus publicly demonstrating
my complete stupidity on the subject.  But hey, it's only email.

Python apps like Zope allocate memory then release memory.  What the
operating system does with the released memory is its problem.  Most
Unices seem to keep the virtual size at the highest point allocated by a
process.  That is, the number will won't automatically go down (though
the real size will).  If the operating system decides that something
else needs the memory, it will take the memory back.  NT, however, will
do so.

Or perhaps I've got it all wrong.

Anyways, we had a major, major app -- the classified ad system that
spawned Principia and then Zope -- where the customer insisted on
running each newspaper in its own process.  Then complained when their
memory footprint swelled to gigs.  We *easily* spent 300 manhours trying
to do something here (actually, this is the genesis of the tunable
object cache).

> flush the cache the process still remains the same  size. For that matter
> we're unable to decrease the number of objects stored in the cache as
> well. Maybe we are using it wrong?
> 
> It would be nice to be able to set a quota of 10 MB to the total amount
> of memory each process uses. Or to have more aggressive garbage
> collection.

Some requirements questions -- how do you want the policy to be
implemented?

a. Never let the process size go above the number.

b. Check before each request to see if the process size is above the
number.

c. Check after.

When the process size is exceeded by the first byte, what would you like
to do?
 
> If we can get a fix on this issue then running multiple instances of Zope
> from the same box should be a breeze. (Although maybe expensive, but I've
> always wanted to have a box with 1GB of RAM)

Are you running each of them with different Pythons?  If you run them
with the same Python, will they share the code segment thingy for the
interpreter?

What is the RSS in the above example, or in the case where you have four
Zopes running?

--Paul