[Zope-dev] Very severe memory leak

[kosh]

On Friday 22 August 2003 03:38 pm, Leonardo Rochael Almeida wrote:
> Hi,
>
> In trying to narrow down the causes of the leak, we detected what looks
> like a minor one. If you insert a very simple dtml method in Zope root
> and bang on it at a rate of, say, 100 requests per second, Zope will
> increase it's memory footprint very slowly, say, a couple hundred Ks
> every 10 minutes (both RSS and virtual size). I figure this has
> something to do with memory fragmentation or some other low level stuff.
> But this is definetly not our main concern, which is to find out how is
> it that Zope is leaking DateTimes so heavily in our site, and if it is
> really all those DateTimes that are hogging our memory. We need help,
> and we need it desperately. If anyone wants any other information we'd
> be happy to provide.

Okay here are a few things to try based on this information. 

1) Take the zclasses and burn them and do it as a python product instead. From 
items I have seen/done in the past this makes a fairly large difference for 
speed and a little in memory. However with it being faster speed you can have 
smaller caches.

2) Convert dtml objects to python scripts where feasible. Python scripts often 
run 10-40 times faster then dtml does and they seem to have a much better 
memory footprint. With them running a lot faster you can also cut down on 
your caching. You can write write scripts that are fairly complex that can 
generate ~ 200 req/s but I have not gotten even close to that with dtml.

3)If you still have problems use psyco on your python product that you use to 
replace the zclasses and bind it to the methods that make the most speed 
difference. Caching is good but having it run fast enough to not need caching 
is often better.

I have a fair number of sites now that range in size from 100M - 1GB that get 
a fair bit of usage with tens of thousands of objects and they have a memory 
usage of about 20-45M of ram under zope 2.6.1. They used to have a memory 
usage in the 80-400M range but that was before I fixed a number of things.