[Zope-dev] Cache growing during single REQUEST

[Leonardo Rochael Almeida]

On Fri, 2003-08-29 at 06:28, Chris Withers wrote:
> Shane Hathaway wrote:
> > as a lower bound.  (Note that the cache is still allowed to grow 
> > indefinitely within the scope of a request, however.)
> 
> Which is still pretty broken IMHO :-(
> 
> This is the single biggest cause of Zope becoming unresponsive for me:
> people doing silly things that drag hordes of objects into memory i na single 
> request,

Now that would be an interesting feature: An upper bound on the number
of objects a request is allowed touch, period. If a request requires
more than that it's rolled back.

>  made worse by the fact that Python is extraordinarily bad at giving 
> memory back to the OS after that request has been completed ;-)

On Linux (and probably other nixes), this is a libc (mis)feature. You
see, malloc() is a libc abstraction. The memory the OS makes avaliable
to each process is actually a contiguous block with a variable upper (or
is it lower?) bound. This bound is usually changed thru the sbrk() call,
which is a wrapper to the brk() call. The latter tells to the OS the
amount of heap space the process think it needs. The former is an
increment or decrement wrapper around the latter.

When the process runs out of heap (malloc()) space, it (The libc in the
python interpreter, in our case) calls sbrk() with a positive number
(eg. it asks for more memory).

Now, no matter how much memory is later free()d (or in our case, python
objects deleted), the system never calls sbrk() with a negative value.
If it were to do that, it (the libc) would first have to make sure that
there are no pointers to allocated space still in the area. I'm not
sure, but I think that the glibc in Linux just doesn't bother.

Defragmentation of the area is not an option because, in C, due to
pointer arithmetic, you can never be sure where all the pointers are, to
change them to the new locations.

Now this is all from memory (and a cursory read of the sbrk(2) man page)
so the above could be full of, erm..., incorrections :-)

Corrections to the above explanation are welcome.

Cheers, Leo

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.