[Zope] Large jumps of memory use increase, and seeking overall understanding of memory use

[Jean Lagarde]

On 2/25/07, Jens Vagelpohl <jens at dataflake.org> wrote:
>
> On 25 Feb 2007, at 00:05, Jean Lagarde wrote:
> > I fully understand that Zope has to be restarted regularly
>
> This may be needed in a situation where you have a memory-hungry
> application (like Plone), but otherwise there is no general need to
> restart Zope regularly. My websites run months without restart, and
> they don't balloon in memory footprint.
>

Thanks Roché and Jens. Yes, of course I meant this in the context of a
public Plone application.

> That setting is a little bit misleading. These values represent
> targets, not hard limits. If a request pulls in a much larger number
> of objects into memory then that chunk of memory is now held by
> Python and won't be released, small target value or not.

I also realize that.  Although here the part that is not clear to me:
Python holds that memory and won't release it to the OS, but can't it
reuse it internally for future object loads? If it just keeps piling
up objects in memory then even talking of a target is meaningless.

> > You would need to study all the inefficiencies inherent in the
> > content types you use, or all the nasty stuff that gets pulled in
> > for viewing the site furniture shown for every page etc. Very few
> > product authors out there put memory efficiency onto their priority
> > list.
>
> P.S.: Memory is cheap. Cheap enough even for a non-profit. Go get
> some, or at least compare the cost of buying meory with the cost of
> countless hours you already have and may continue to spend on this.
> I'm not saying it's not a worthwhile task to try and get to the
> bottom of code inefficiencies, but keep the practical side in mind.

Yep, that's what I keep on telling my co-administrators. As I said we
actually have a lot more physical memory available, but I need their
buy-in to increase the process limits on FreeBSD (I'm not succeeding
by the way -- got an email this morning suggesting that Zope/Plone was
a bad idea instead). And yes, I do allow crawlers and they are poking
at all kinds of things. However, I was still hoping that someone would
address the specific examples I provided to tell me if they are in the
ballpark of what is commonly observed. I understand all the
generalities, but I want to find out if something is broken in my
particular implementation (possibly because of a bad practice on my
part, etc.). I realize that looking at requests only does not give the
whole picture (although I think it should be the largest source of
Zope activity by far, am I wrong?), and that the descriptions I have
provided are far from sufficient to do a complete determination, but
let's just take the first one:

Memory size was at 417640 (Kb) from 03:01:40 to 03:04:41 (for three
minutes). During these three minutes, there were requests for our
front page and apparently everything that accompanies it in Plone,
i.e. the Plone scripts.jss, the style sheets, bunch of icon gifs,
image thumbs the logo.jpg, etc. No memory increase during that time.
At 03:05:41 I logged a memory increase to 422704. Between 03:04:41 and
03:05:41, there was a single request at 03:05:02 for an
image_view_fullscreen. For those unfamiliar with Plone, that is one of
the barest page served: Just an image and a link to go back to Plone
(none of the Plone portlets, etc.). The access log shows that the
number of bytes returned was 1137. There was no GET for the actual
image. There were no memory increase whatsoever for at least the next
30 minutes, and there were a number of other requests during that
time. Now I'm looking for hypotheses to explain the 5Mb memory
increase. Here are some:

1) It's impossible. I must have made a mistake when I created that
merged log (i.e. I must be looking at bad data).

2) The 5 Mb necessarily had to be used by the object cache to load one
or more objects. Note that the image was 2 Mb at most, and the object
itself can't be much larger, so even if it was loaded in the cache
then, my question is what else was loaded? Or, does Zope/Python grab
more memory for the cache in chunks? (I know that when appending to a
Python list for example, the list will grow in chunks, but that's only
space for pointers and not the actual objects pointed to.)

3) Part of the 5 Mb was probably used loading into the object cache,
but a significant part could have been used elsewhere (but the
increase was still due to the request). So my question then is: Where
else would previously unallocated megabytes of memory be needed to
serve a page? I am using the default CacheFu behavior for images, so I
know it wasn't also RAM cached.

4) Could have been a coincidence, unrelated to the request. My
question then is: What else could Zope be doing in the background,
other than serve requests, that consumes megabytes of new memory. (We
don't use Webdav, or ftp; I'm quite positive in any case that no
imports were being made. I do use external editor, but that would show
up in the HTTP access log).

Cheers,

Jean