[Zope] Large jumps of memory use increase, and seeking overall understanding of memory use

[Jens Vagelpohl]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On 25 Feb 2007, at 00:05, Jean Lagarde wrote:
> I fully understand that Zope has to be restarted regularly

This may be needed in a situation where you have a memory-hungry  
application (like Plone), but otherwise there is no general need to  
restart Zope regularly. My websites run months without restart, and  
they don't balloon in memory footprint.


> My object cache size was initially at the default of 5000 objects per
> threads (4 threads default). I reduced this to 2000 just to see, but
> as I expected, the overall memory use behavior was the same.

That setting is a little bit misleading. These values represent  
targets, not hard limits. If a request pulls in a much larger number  
of objects into memory then that chunk of memory is now held by  
Python and won't be released, small target value or not.


> Overall, my custom memory-use/access-log shows long periods with no
> increase in memory at all, but there are some big jumps at times
> (during a one minute tick) by 5, 10, even 25 Mb! As pretty much
> expected, Zope never gives much memory back to the OS; that part I
> understand, however I would like to get it to a state where it mostly
> plateaus and then only slowly creeps up.

This is likely to be impossible, especially with large amounts of  
content. I don't know your application, but I have two sites I know  
pretty well where content is public, and web crawlers/spiders become  
the enemy because they cause large chunks of content to be loaded  
that is not within the normal "window" of content used by normal site  
users. That's just one specific example where a quick jump in memory  
footprint can be observed.


> Alex and I are volunteers doing this work for a non-profit, so any
> ideas or "yep, that's normal, live with it" would be greatly
> appreciated! (If the latter answer, I would still like to better
> understand where that memory is being used; I still don't understand
> how it could be the object cache, which is often cited as the main
> cause of memory creep).

You would need to study all the inefficiencies inherent in the  
content types you use, or all the nasty stuff that gets pulled in for  
viewing the site furniture shown for every page etc. Very few product  
authors out there put memory efficiency onto their priority list.

jens


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFF4W4ZRAx5nvEhZLIRAsFvAKC189MpGOfP+fI5w0ckR7W9ceMIZgCgovW/
2cMz670YuvmUi7dWQ3GufUY=
=kdBX
-----END PGP SIGNATURE-----