Re: [Zope] Zope Eating Memory for Breakfast
--- In zope@egroups.com, Tony McDonald <tony.mcdonald@n...> wrote:
This is on Solaris 2.7 with Zope 2.1.4. I can provide any other info you like, but I'm not sure what would be useful to know. PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND 5866 zope 8 10 0 2262M 2034M cpu2 149:09 24.70% python
Wow wow wow, sometimes my Zope eats up to 160 MB of RAM memory, and I thought that was bad. :) There's gotta be a memory leak somewhere in Zope. And we'd better figure out where it is. I have already seen several messages complaining about Zope's memory usage. ------------------------------------------------------------- OK, let us try to narrow down a bit. (1) Version: I also use Zope 2.1.4. System is Linux. (2) Webserver: ZServer with multiple threads. (3) Products: I use SybaseDAv2 for database adapter. I also use a bit of Gadfly (through SQLSession), this of course is a temporary feature because eventually I want to get rid of Gadfly. (4) ZODB: I don't have anything that dynamically changes Zope's ZODB, that is, I don't upload files into ZODB (Do you?), I don't create more Zope users on the fly. In fact, I believe my site can run from a CD-Rom, if necessary. (5) Size of ZODB: my Data.fs is about 9 MB. (6) Uptime pattern: the maximum time I have run Zope is less than a week. Usually I restart it a few times a week. (7) Memory growth pattern: my Zope starts out with 7 MB in each of the 4 threads. It starts to climb up slowly, by the end of the day it reaches 18 MB per thread. After 3 days it climbs up to 38 MB. That's about 160 MB total. (No, I don't think the threads are sharing memory, because the Linux "top" command also tells me the percentage of memory usage.) (8) Performance: I have severe problem with SQLSession and Gadfly, and also some problem with SybaseDAv2. I think the performance problem is related to the RAM memory consumption, because when I clean up Gadfly (deleting non-used table records), the Gadfly performance increased substantially. (9) Database Management cache parameters: (from Zope's control panel) Total number of objects in the database 4285 Total number of objects in all of the caches combined 1533 Target size 400 Target maximum time between accesses 60 Flush Cache: Full Sweep and Minimize don't help to reduce the memory consumption. ------------------------------------------------------------ Please post back your site's info. And let us see if we can narrow it down a bit. Anyone else that has memory problem please also help us to narrow down the problem. Why do we need to narrow it down? Because at this moment I am not sure whether the memory leak comes from a product or from Zope, and I am not sure whether it happens in Unix systems or in Windows as well, and I am not sure whether threads have problems, etc. Let's narrow down the problem first. thanks, Hung Jung ______________________________________________________ Get Your Private, Free Email at http://www.hotmail.com
Hung Jung Lu wrote:
--- In zope@egroups.com, Tony McDonald <tony.mcdonald@n...> wrote:
This is on Solaris 2.7 with Zope 2.1.4. I can provide any other info you like, but I'm not sure what would be useful to know. PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND 5866 zope 8 10 0 2262M 2034M cpu2 149:09 24.70% python
Wow wow wow, sometimes my Zope eats up to 160 MB of RAM memory, and I thought that was bad. :)
There's gotta be a memory leak somewhere in Zope. And we'd better figure out where it is. I have already seen several messages complaining about Zope's memory usage.
-------------------------------------------------------------
OK, let us try to narrow down a bit.
(1) Version: I also use Zope 2.1.4. System is Linux.
Zope version: Zope 2.1.4 (source release, python 1.5.2, linux2) Python version: 1.5.2 (#1, Feb 14 2000, 18:27:27) [GCC 2.95.1 19990816 (release)] System Platform: sunos5 Process ID: 2977 (5) Running for: 7 hours 26 min 7 sec That is Solaris, not linux. It was the source dist. Strange. PS. It's up to 28Meg now.
(2) Webserver: ZServer with multiple threads.
Apache. I can't remember which version...
(3) Products: I use SybaseDAv2 for database adapter. I also use a bit of Gadfly (through SQLSession), this of course is a temporary feature because eventually I want to get rid of Gadfly.
ZMySQLDA-1-1-3 ZOracleDA PTK Confera ZPhotoAlbum But really I'm not using the DA's or PTK right now. They're installed, but the code that uses them is still in test.
(4) ZODB: I don't have anything that dynamically changes Zope's ZODB, that is, I don't upload files into ZODB (Do you?), I don't create more Zope users on the fly. In fact, I believe my site can run from a CD-Rom, if necessary.
My site changes a bit more than that, but it's still fairly static. My web-content manager may post a new org-chart or internal news story, but we don't even have that many people connecting to read yet.
(5) Size of ZODB: my Data.fs is about 9 MB.
That would be 525Meg. My god, that's large. But I think that's mainly pictures.
(6) Uptime pattern: the maximum time I have run Zope is less than a week. Usually I restart it a few times a week.
No clue, as I restart it every couple of days to get the size of that db down.
(7) Memory growth pattern: my Zope starts out with 7 MB in each of the 4 threads. It starts to climb up slowly, by the end of the day it reaches 18 MB per thread. After 3 days it climbs up to 38 MB. That's about 160 MB total. (No, I don't think the threads are sharing memory, because the Linux "top" command also tells me the percentage of memory usage.)
When I restart it, it starts out at about 18Meg. Then it gradually grows, and I eventually notice that it's at 258M. I'll keep a better eye on that.
(8) Performance: I have severe problem with SQLSession and Gadfly, and also some problem with SybaseDAv2. I think the performance problem is related to the RAM memory consumption, because when I clean up Gadfly (deleting non-used table records), the Gadfly performance increased substantially.
The only performance problems I've noticed are once the memory consuption starts putting the box into swapping.
(9) Database Management cache parameters: (from Zope's control panel)
Total number of objects in the database 4285 Total number of objects in all of the caches combined 1533 Target size 400 Target maximum time between accesses 60
Flush Cache: Full Sweep and Minimize don't help to reduce the memory consumption.
Total number of objects in the database 11057 Total number of objects in all of the caches combined 1502 Target size 400 Target maximum time between accesses 20 I also have Zope on another Solaris Box (which is considerably smaller) and on a Linux machine. Neither present this problem. The other Solaris box is our test machine, and no code goes on the Production Box that hasn't gone on the test machine, so I'm even more confused. One of our first thoughts concerned the compiler on the big boy, as it was put there before we got here (old admins) but the same one would go on the test as the prod. Anyone else? Monty
Monty Taylor wrote:
Zope version: Zope 2.1.4 (source release, python 1.5.2, linux2) Python version: 1.5.2 (#1, Feb 14 2000, 18:27:27) [GCC 2.95.1 19990816 (release)] System Platform: sunos5 Process ID: 2977 (5) Running for: 7 hours 26 min 7 sec
What about for your test server? What are the loads/restart frequency of the test server?
(7) Memory growth pattern: my Zope starts out with 7 MB in each of the 4 threads. It starts to climb up slowly, by the end of the day it reaches 18 MB per thread. After 3 days it climbs up to 38 MB. That's about 160 MB total. (No, I don't think the threads are sharing memory, because the Linux "top" command also tells me the percentage of memory usage.)
When I restart it, it starts out at about 18Meg. Then it gradually grows, and I eventually notice that it's at 258M. I'll keep a better eye on that.
So I make sure I understand: o Your Data.fs is ~525MB o Zope's memory usage starts at 1t ~18MB o Zope's memory usage grows for there o Uptime: Zope is restarted (or is the machine restarted?) a few times a week Right?
Total number of objects in the database 11057 Total number of objects in all of the caches combined 1502 Target size 400 Target maximum time between accesses 20
I also have Zope on another Solaris Box (which is considerably smaller)
What os is it running?
and on a Linux machine. Neither present this problem. The other Solaris box is our test machine, and no code goes on the Production Box that hasn't gone on the test machine, so I'm even more confused. One of our first thoughts concerned the compiler on the big boy, as it was put there before we got here (old admins) but the same one would go on the test as the prod.
Anyone else?
Just trying to narrow by process of elimination.... When you put Zope on the big boy, was it a clean install, or was it copied over from the test machine? If the latter, I am presuming it was rebuilt, correct? What about options to the compiler (if any), such as optimizations, architecture-dependent flags, etc.? -- In flying I have learned that carelessness and overconfidence are usually far more dangerous than deliberately accepted risks. -- Wilbur Wright in a letter to his father, September 1900
Bill Anderson wrote:
Monty Taylor wrote:
Zope version: Zope 2.1.4 (source release, python 1.5.2, linux2) Python version: 1.5.2 (#1, Feb 14 2000, 18:27:27) [GCC 2.95.1 19990816 (release)] System Platform: sunos5 Process ID: 2977 (5) Running for: 7 hours 26 min 7 sec
What about for your test server? What are the loads/restart frequency of the test server?
Well, there's virtually no load on the test server, because I use the Zope installation on it to test new products, but not the day-to-day DTML-type development. ( I figure that if a piece of DTML can crash a server, I'm not good enough yet to write it. ) But it stays up indefinitely. -- Zope version: Zope 2.1.4 (source release, python 1.5.2, linux2) Python version: 1.5.2 (#1, Feb 14 2000, 18:27:27) [GCC 2.95.1 19990816 (release)] System Platform: sunos5 Process ID: 9004 (5) Running for: 14 days 13 hours 31 sec
(7) Memory growth pattern: my Zope starts out with 7 MB in each of the 4 threads. It starts to climb up slowly, by the end of the day it reaches 18 MB per thread. After 3 days it climbs up to 38 MB. That's about 160 MB total. (No, I don't think the threads are sharing memory, because the Linux "top" command also tells me the percentage of memory usage.)
When I restart it, it starts out at about 18Meg. Then it gradually grows, and I eventually notice that it's at 258M. I'll keep a better eye on that.
So I make sure I understand: o Your Data.fs is ~525MB o Zope's memory usage starts at 1t ~18MB o Zope's memory usage grows for there o Uptime: Zope is restarted (or is the machine restarted?) a few times a week
Right?
Right, Zope is restarted a few times a week currently, but that's because I notice it having grown huge. There really isn't anything else that would ever cause it to come down.
Total number of objects in the database 11057 Total number of objects in all of the caches combined 1502 Target size 400 Target maximum time between accesses 20
I also have Zope on another Solaris Box (which is considerably smaller)
What os is it running?
bash-2.03$ uname -a SunOS canis 5.7 Generic_106541-08 sun4u sparc SUNW,Ultra-5_10
and on a Linux machine. Neither present this problem. The other Solaris box is our test machine, and no code goes on the Production Box that hasn't gone on the test machine, so I'm even more confused. One of our first thoughts concerned the compiler on the big boy, as it was put there before we got here (old admins) but the same one would go on the test as the prod.
Anyone else?
Just trying to narrow by process of elimination.... When you put Zope on the big boy, was it a clean install, or was it copied over from the test machine? If the latter, I am presuming it was rebuilt, correct? What about options to the compiler (if any), such as optimizations, architecture-dependent flags, etc.?
Now this is the one thing my SysAdmin is looking into -- our C compiler situation. We've got GCC but no native C compiler, making us wonder about the validity of the GCC on the box. (Previous sysadmins) I do know that the installation on the Production box was built on the smaller test box, but architecture wise they are the same -- just size is different.
Hung Jung Lu wrote: Well, I am *not* having memory problems, but since my setup may be useful in that regard:
--- In zope@egroups.com, Tony McDonald <tony.mcdonald@n...> wrote:
This is on Solaris 2.7 with Zope 2.1.4. I can provide any other info you like, but I'm not sure what would be useful to know. PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND 5866 zope 8 10 0 2262M 2034M cpu2 149:09 24.70% python
Wow wow wow, sometimes my Zope eats up to 160 MB of RAM memory, and I thought that was bad. :)
There's gotta be a memory leak somewhere in Zope. And we'd better figure out where it is. I have already seen several messages complaining about Zope's memory usage.
-------------------------------------------------------------
OK, let us try to narrow down a bit.
(1) Version: I also use Zope 2.1.4. System is Linux.
About 7 currently running Zopes at the moment. Ranging from 2.1.3 to 2.1.6, with the majority being 2.1.4
(2) Webserver: ZServer with multiple threads.
ZServer, 4-6 threads, running behind Roxen (-1 on it's own)
(3) Products: I use SybaseDAv2 for database adapter. I also use a bit of Gadfly (through SQLSession), this of course is a temporary feature because eventually I want to get rid of Gadfly.
(4) ZODB: I don't have anything that dynamically changes Zope's ZODB, that is, I don't upload files into ZODB (Do you?), I don't create more Zope users on the fly. In fact, I believe my site can run from a CD-Rom, if necessary.
I am using Sybase DA, PTK on four, and BannerClasses which modifies the Data.fs _every_ page view.
(5) Size of ZODB: my Data.fs is about 9 MB.
Sizes: from <1M to 50M at a given time.
(6) Uptime pattern: the maximum time I have run Zope is less than a week. Usually I restart it a few times a week.
Mang up for weeks at atime, obvious exception would be the two test servers.
(7) Memory growth pattern: my Zope starts out with 7 MB in each of the 4 threads. It starts to climb up slowly, by the end of the day it reaches 18 MB per thread. After 3 days it climbs up to 38 MB. That's about 160 MB total. (No, I don't think the threads are sharing memory, because the Linux "top" command also tells me the percentage of memory usage.)
The Linux 'Top' command is a tricky beast. How much memory is _in_use_? Can't tell that with Top (without a callculator anyway ;). Use 'free', it will report the amount currently in use, the amount in buffer, cache, etc. ...and do the math for you. If I were to count the memory for _each_ thread, it would allegedly be on the order of 200-300MB easy. In reality, it isn't using anywhere near that. The memory statistics per thread in Linux _is_ a 'shared' memory amongst the threads (though not in the programmatic definition).
(9) Database Management cache parameters: (from Zope's control panel)
Total number of objects in the database 4285 Total number of objects in all of the caches combined 1533 Target size 400 Target maximum time between accesses 60
Flush Cache: Full Sweep and Minimize don't help to reduce the memory consumption.
Average object cache target: 5000 Number in cache (average): 4000 Average max time: 180 seconds. Hope it helps in some way. Bill -- In flying I have learned that carelessness and overconfidence are usually far more dangerous than deliberately accepted risks. -- Wilbur Wright in a letter to his father, September 1900
Bill Anderson wrote:
Hung Jung Lu wrote:
Actually, no that was me. :) Dunno how that got in there.
Well, I am *not* having memory problems, but since my setup may be useful in that regard:
--- In zope@egroups.com, Tony McDonald <tony.mcdonald@n...> wrote:
This is on Solaris 2.7 with Zope 2.1.4. I can provide any other info you like, but I'm not sure what would be useful to know. PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND 5866 zope 8 10 0 2262M 2034M cpu2 149:09 24.70% python
Wow wow wow, sometimes my Zope eats up to 160 MB of RAM memory, and I thought that was bad. :)
There's gotta be a memory leak somewhere in Zope. And we'd better figure out where it is. I have already seen several messages complaining about Zope's memory usage.
-------------------------------------------------------------
OK, let us try to narrow down a bit.
(1) Version: I also use Zope 2.1.4. System is Linux.
About 7 currently running Zopes at the moment. Ranging from 2.1.3 to 2.1.6, with the majority being 2.1.4
(2) Webserver: ZServer with multiple threads.
ZServer, 4-6 threads, running behind Roxen (-1 on it's own)
(3) Products: I use SybaseDAv2 for database adapter. I also use a bit of Gadfly (through SQLSession), this of course is a temporary feature because eventually I want to get rid of Gadfly.
(4) ZODB: I don't have anything that dynamically changes Zope's ZODB, that is, I don't upload files into ZODB (Do you?), I don't create more Zope users on the fly. In fact, I believe my site can run from a CD-Rom, if necessary.
I am using Sybase DA, PTK on four, and BannerClasses which modifies the Data.fs _every_ page view.
(5) Size of ZODB: my Data.fs is about 9 MB.
Sizes: from <1M to 50M at a given time.
(6) Uptime pattern: the maximum time I have run Zope is less than a week. Usually I restart it a few times a week.
Mang up for weeks at atime, obvious exception would be the two test servers.
(7) Memory growth pattern: my Zope starts out with 7 MB in each of the 4 threads. It starts to climb up slowly, by the end of the day it reaches 18 MB per thread. After 3 days it climbs up to 38 MB. That's about 160 MB total. (No, I don't think the threads are sharing memory, because the Linux "top" command also tells me the percentage of memory usage.)
The Linux 'Top' command is a tricky beast. How much memory is _in_use_? Can't tell that with Top (without a callculator anyway ;). Use 'free', it will report the amount currently in use, the amount in buffer, cache, etc. ...and do the math for you.
If I were to count the memory for _each_ thread, it would allegedly be on the order of 200-300MB easy. In reality, it isn't using anywhere near that. The memory statistics per thread in Linux _is_ a 'shared' memory amongst the threads (though not in the programmatic definition).
(9) Database Management cache parameters: (from Zope's control panel)
Total number of objects in the database 4285 Total number of objects in all of the caches combined 1533 Target size 400 Target maximum time between accesses 60
Flush Cache: Full Sweep and Minimize don't help to reduce the memory consumption.
Average object cache target: 5000 Number in cache (average): 4000 Average max time: 180 seconds.
Hope it helps in some way. Bill
-- In flying I have learned that carelessness and overconfidence are usually far more dangerous than deliberately accepted risks. -- Wilbur Wright in a letter to his father, September 1900
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
-- In flying I have learned that carelessness and overconfidence are usually far more dangerous than deliberately accepted risks. -- Wilbur Wright in a letter to his father, September 1900
participants (3)
-
Bill Anderson -
Hung Jung Lu -
Monty Taylor