[Zope] Memory creep
John Hile
johnh@hilgraeve.com
Sat, 20 May 2000 15:09:27 -0400
This is a multi-part message in MIME format.
------=_NextPart_000_0000_01BFC26D.63E43FF0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
We have been having the problem with memory usage creeping slowly upward
that I've seen reported by several others in this list. I read several of
the threads on this topic in the archives but, though there were a lot of
theories floated, no firm conclusions seemed to be drawn. Like the others,
memory usage as reported by 'free' on the '-/+ buffers/cache' line creeps
steadily upward until the site crashes. Restarting Zope frees most of the
memory.
We're running RedHat Linux 6.1, on a dual- 550M processor box with 256Mb of
memory, Zope 2.1.4 using Apache, mod-ssl and PCGI, ZMySqlDA 1.1.3. The ZODB
holds only non-dynamic data, all user data is kept in the MySQL data base.
I tried testing on our development server while no one else was accessing
the site with the 'free' utility displaying memory use every 5 seconds in an
attempt to find out what activity caused memory usage to creep upward
without coming back down.
To cut to the chase, I seem to have narrowed it down to access to the MySQL
database from multiple threads. As it happens, we have a very thin client
piece that can access the server through http outside the browser. Most use
of the site is through a browser, but this small client checks with the
server occasionally for scheduled tasks when the browser isn't up. In some
cases, access through the browser will use a plug-in to trigger the client
to check immediately rather than waiting for its infrequent poll interval.
After an afternoon of testing, I discovered that, with no one else accessing
the site, I could access pages as fast as I was able through the browser
only (no client) or from the client only (sped up to poll every few seconds)
with no memory creep. But when both were running at once (as they normally
do), the memory usage would slowly creep up.
Other details:
Letting the client poll and query the MySQL database frequently while using
the browser to access only pages that did NOT access the DB resulted in no
creep.
Having two people simultaneously access pages that DID access the database
without the client running on either machine DID cause memory creep.
Having two people access pages that did NOT access the database without the
client running did NOT cause memory creep.
Looks to me like there is a memory leak when ZMySQLDA is accessed from
multiple threads.
Anyone have any ideas on where I could go to pin this down further?
------=_NextPart_000_0000_01BFC26D.63E43FF0
Content-Type: text/html;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META content=3D"text/html; charset=3Diso-8859-1" =
http-equiv=3DContent-Type>
<META content=3D"MSHTML 5.00.3017.1000" name=3DGENERATOR></HEAD>
<BODY>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>We =
have been having=20
the problem with memory usage creeping slowly upward that I've seen =
reported by=20
several others in this list. I read several of the threads on this topic =
in the=20
archives but, though there were a lot of theories floated, no firm =
conclusions=20
seemed to be drawn. Like the others, memory usage as reported by 'free' =
on the=20
'-/+ buffers/cache' line creeps steadily upward until the site crashes.=20
Restarting Zope frees most of the memory.</SPAN></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><SPAN=20
class=3D546054018-20052000></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>We're =
running RedHat=20
Linux 6.1, on a dual- 550M processor box with 256Mb of memory, Zope =
2.1.4 using=20
Apache, mod-ssl and PCGI, ZMySqlDA 1.1.3. The ZODB holds only =
non-dynamic data,=20
all user data is kept in the MySQL data base.</SPAN></FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>I =
tried testing on=20
our development server while no one else was accessing the site with the =
'free'=20
utility displaying memory use every 5 seconds in an attempt to find out =
what=20
activity caused memory usage to creep upward without coming back=20
down.</SPAN></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><SPAN=20
class=3D546054018-20052000></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>To cut =
to the chase,=20
I seem to have narrowed it down to access to the MySQL database from =
multiple=20
threads. As it happens, we have a very thin client piece that can access =
the=20
server through http outside the browser. Most use of the site is through =
a=20
browser, but this small client checks with the server occasionally for =
scheduled=20
tasks when the browser isn't up. In some cases, access through the =
browser will=20
use a plug-in to trigger the client to check immediately rather than =
waiting for=20
its infrequent poll interval. </SPAN></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><SPAN=20
class=3D546054018-20052000></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>After =
an afternoon=20
of testing, I discovered that, with no one else accessing the site, I =
could=20
access pages as fast as I was able through the browser only (no =
client) or=20
from the client only (sped up to poll every few seconds) with no memory =
creep.=20
But when both were running at once (as they normally do), the memory =
usage would=20
slowly creep up. </SPAN></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><SPAN=20
class=3D546054018-20052000></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>Other=20
details:</SPAN></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><SPAN=20
class=3D546054018-20052000></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN =
class=3D546054018-20052000>Letting the client=20
poll and query the MySQL database frequently while using the browser to =
access=20
only pages that did NOT access the DB resulted in no =
creep.</SPAN></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><SPAN=20
class=3D546054018-20052000></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>Having =
two people=20
simultaneously access pages that DID access the database without the =
client=20
running on either machine DID cause memory =
creep.</SPAN></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><SPAN=20
class=3D546054018-20052000></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>Having =
two people=20
access pages that did NOT access the database without the client running =
did NOT=20
cause memory creep.</SPAN></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><SPAN=20
class=3D546054018-20052000></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>Looks =
to me like=20
there is a memory leak when ZMySQLDA is accessed from multiple=20
threads.</SPAN></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><SPAN=20
class=3D546054018-20052000></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial size=3D2><SPAN class=3D546054018-20052000>Anyone =
have any=20
ideas on where I could go to pin this down=20
further?</SPAN></FONT></DIV></BODY></HTML>
------=_NextPart_000_0000_01BFC26D.63E43FF0--