Zope memory usage out of control
Hi! First of all let me just say what a wonderful system Zope is. We've been running dot.kde.org on a Squishdot/Zope setup very successfully for a quite a while now. Unfortunately, the admin of the server is getting a bit edgy about the increasing memory usage of Zope and is pressing for action. From the very start we've had to restart Zope daily (with a cron job) because Zope would be using large amounts of memory after several hours. After 15 hours of uptime this is the current situation as reported by top: PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND 12093 zope 0 0 94988 90M 796 S 0 0.7 11.9 9:01 python 12122 zope 0 0 94988 90M 796 S 0 0.0 11.9 0:03 python 12123 zope 0 0 94988 90M 796 S 0 0.3 11.9 32:56 python 12124 zope 4 0 94988 90M 796 S 0 15.4 11.9 33:06 python 12125 zope 0 0 94988 90M 796 S 0 0.0 11.9 33:10 python 12126 zope 0 0 94988 90M 796 S 0 0.0 11.9 36:52 python Please tell me that's really a single process with several threads all sharing memory? :-) dot.kde.org is certainly a very busy site, is this the kind of memory usage to be expected from Zope? Has anyone seen this kind of problem before? We're using Zope 2.2.1 with the hotfixes on a Red Hat Linux box. Any hints short of switching to another news system appreciated (the SlashCode guys have been on our case). ;) Thanks, Navin. PS I'm not sure if my confirmation of subscription worked or not, please Cc: navindra@cs.mcgill.ca meanwhile, otherwise I'll just check the archives.
Hi!
First of all let me just say what a wonderful system Zope is. We've been running dot.kde.org on a Squishdot/Zope setup very successfully for a quite a while now.
Great...
Unfortunately, the admin of the server is getting a bit edgy about the increasing memory usage of Zope and is pressing for action. From the very start we've had to restart Zope daily (with a cron job) because Zope would be using large amounts of memory after several hours.
After 15 hours of uptime this is the current situation as reported by top:
PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND 12093 zope 0 0 94988 90M 796 S 0 0.7 11.9 9:01 python 12122 zope 0 0 94988 90M 796 S 0 0.0 11.9 0:03 python 12123 zope 0 0 94988 90M 796 S 0 0.3 11.9 32:56 python 12124 zope 4 0 94988 90M 796 S 0 15.4 11.9 33:06 python 12125 zope 0 0 94988 90M 796 S 0 0.0 11.9 33:10 python 12126 zope 0 0 94988 90M 796 S 0 0.0 11.9 36:52 python
Please tell me that's really a single process with several threads all sharing memory? :-)
Yes. ;-)
dot.kde.org is certainly a very busy site, is this the kind of memory usage to be expected from Zope? Has anyone seen this kind of problem before?
Does the memory usage grow without bound? Or does it stay steady at a large amount? (Or have you not had the opportunity to find out?)
We're using Zope 2.2.1 with the hotfixes on a Red Hat Linux box.
Any hints short of switching to another news system appreciated (the SlashCode guys have been on our case). ;)
Don't even think about it!! :-)
Please tell me that's really a single process with several threads all sharing memory? :-)
Yes. ;-)
Phew. ;)
dot.kde.org is certainly a very busy site, is this the kind of memory usage to be expected from Zope? Has anyone seen this kind of problem before?
Does the memory usage grow without bound? Or does it stay steady at a large amount? (Or have you not had the opportunity to find out?)
Well, my belief is that it's growing without bounds. It starts at 13M, jumps to 18M and then gets progressively higher. It used to grow up to 30M-50M or so. I'm currently experimenting with reducing cache size/time (target=50 objects / time=10s currently) but I'm not sure that helps yet. It seems to have improved the situation slightly, but we'll only be able to tell in a few hours.
We're using Zope 2.2.1 with the hotfixes on a Red Hat Linux box.
Any hints short of switching to another news system appreciated (the SlashCode guys have been on our case). ;)
Don't even think about it!! :-)
Don't worry, I'll fight it. ;-) Thanks for the amazingly quick response! Cheers, Navin.
It sounds like a potential memory leak. These can be hard to track down... any 3rd party product (or anything in the Zope core) can potentially leak. Using the debug screen of the control panel can show you reference counts, which can help a lot in determining what kinds of objects have the highest reference count. Using the auto-refresh (or periodic visits to the "Refresh" link )on the debug screen when the site is in normal operation can show you which kinds of objects are having a high "referred to" delta between snapshots, which can maybe help us track down any leaks that may be occurring. This is probably the place to start. Listing all of your Products might help determine where the leak might be too. ----- Original Message ----- From: "Navindra Umanee" <navindra@cs.mcgill.ca> To: <chrism@digicool.com>; <zope@zope.org> Sent: Monday, February 05, 2001 8:16 PM Subject: Re: [Zope] Zope memory usage out of control
Please tell me that's really a single process with several threads all sharing memory? :-)
Yes. ;-)
Phew. ;)
dot.kde.org is certainly a very busy site, is this the kind of memory usage to be expected from Zope? Has anyone seen this kind of problem before?
Does the memory usage grow without bound? Or does it stay steady at a
large
amount? (Or have you not had the opportunity to find out?)
Well, my belief is that it's growing without bounds. It starts at 13M, jumps to 18M and then gets progressively higher. It used to grow up to 30M-50M or so.
I'm currently experimenting with reducing cache size/time (target=50 objects / time=10s currently) but I'm not sure that helps yet. It seems to have improved the situation slightly, but we'll only be able to tell in a few hours.
We're using Zope 2.2.1 with the hotfixes on a Red Hat Linux box.
Any hints short of switching to another news system appreciated (the SlashCode guys have been on our case). ;)
Don't even think about it!! :-)
Don't worry, I'll fight it. ;-)
Thanks for the amazingly quick response!
Cheers, Navin.
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Navindra Umanee Sent: Monday, February 05, 2001 7:37 PM To: zope@zope.org Subject: [Zope] Zope memory usage out of control
Please tell me that's really a single process with several threads all sharing memory? :-)
Linux gives each thread a process ID, so ps shows each thread as a process. There's no way (that I know of) to tell from ps if they're threads or processes, but since we know Zope is threaded, we know those are threads. If you happen to run MySQL, you'll see the same thing.
dot.kde.org is certainly a very busy site, is this the kind of memory usage to be expected from Zope? Has anyone seen this kind of problem before?
I just recently started paying attention to Zope memory use when I noticed my process using over 100MB for a virtually empty site. I have since discovered that a couple dozen 300k images that I have are causing it. Apparently, Zope loads objects in memory in order to send them. This means that several large objects can eat up memory fast. When I ran a test with the ExtImage product, I don't get the Zope bloat I do when using Zope's builtin Image object. This is a huge problem for me since I have dozens of images in Zope that are viewed *maybe* once every few days. After the first viewer, Zope shoots up in memory use and never goes back down.
We're using Zope 2.2.1 with the hotfixes on a Red Hat Linux box.
I'm using Zope 2.3.0, but I don't think anything has changed in this respect.
Any hints short of switching to another news system appreciated (the SlashCode guys have been on our case). ;)
As Chris pointed out, it could be a memory leak. However, if it goes up to a large amount and hangs around there for a while, it sounds more to me like there are some large objects being served (or many objects being cached???). Others know far more about this than I, but this has been my experience. I'm very interested in having Zope serve Image/File objects without having to read them into memory first. Running ExtFile/Image is a nice option since it also stores the large files in the file system, but I don't mind a large Data.fs as much as I mind having all of my memory eaten up for no good reason. Perhaps the future Image/File objects can optionally be stored externally and served without the memory bloat. Should I add a feature request to the collector, or does DC already have a solution in mind for this issue? _______________________ Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
--On Monday, February 05, 2001 20:58:13 -0500 Ron Bickers <rbickers@logicetc.com> wrote:
Linux gives each thread a process ID, so ps shows each thread as a process. There's no way (that I know of) to tell from ps if they're threads or processes, but since we know Zope is threaded, we know those are threads. If you happen to run MySQL, you'll see the same thing.
ps ax -H will indent child threads. ps axm will supposedly show threads but I had problems with that on RedHat. -- -mindlace- zopatista community liason
-----Original Message----- From: ethan mindlace fremen [mailto:mindlace@digicool.com] Sent: Monday, February 05, 2001 9:11 PM To: Ron Bickers; Navindra Umanee; zope@zope.org Subject: RE: [Zope] Zope memory usage out of control
ps ax -H will indent child threads. ps axm will supposedly show threads but I had problems with that on RedHat.
-H will indent child processes as well, so although helpful, it still doesn't distinguish between threads and processes. 'm' is documented to show threads, but at least RedHat says "ERROR: Thread display not implemented." I do wish there was a way to get a BSDish display of only "real" processes and make showing threads optional. _______________________ Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
On Mon, 5 Feb 2001, ethan mindlace fremen wrote:
--On Monday, February 05, 2001 20:58:13 -0500 Ron Bickers <rbickers@logicetc.com> wrote:
Linux gives each thread a process ID, so ps shows each thread as a process. There's no way (that I know of) to tell from ps if they're threads or processes, but since we know Zope is threaded, we know those are threads. If you happen to run MySQL, you'll see the same thing.
ps ax -H will indent child threads. ps axm will supposedly show threads but I had problems with that on RedHat.
There's also ps auxfw, but I like pstree on RedHat: ... |-python---python---python---4*[python] |-rhnsd |-safe_mysqld---mysqld---mysqld---mysqld ... The only way I know to distinquish threads from ordinary children is to look at the memory profile. For threads, they will always be exactly the same, because they share the same memory map. -- Andy Dustman PGP: 0xC72F3F1D @ .net http://dustman.net/andy "Normally with carbonara you use eggs, but I used lobster brains instead." -- Masahiko Kobe (Iron Chef Italian): 30-year-old Giant Lobster Battle
Ron, it would be good if you could add a feature request to the Collector about large file object streaming. I'm not superfamiliar with the code there, but I *think* the Image stuff is supposed to break the Image object into smaller subobjects and load each one serially out of the ZODB when it needs to serve them. If it's not doing this properly, it needs to get fixed. ----- Original Message ----- From: "Ron Bickers" <rbickers@logicetc.com> To: "Navindra Umanee" <navindra@cs.mcgill.ca>; <zope@zope.org> Sent: Monday, February 05, 2001 8:58 PM Subject: RE: [Zope] Zope memory usage out of control
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Navindra Umanee Sent: Monday, February 05, 2001 7:37 PM To: zope@zope.org Subject: [Zope] Zope memory usage out of control
Please tell me that's really a single process with several threads all sharing memory? :-)
Linux gives each thread a process ID, so ps shows each thread as a process. There's no way (that I know of) to tell from ps if they're threads or processes, but since we know Zope is threaded, we know those are threads. If you happen to run MySQL, you'll see the same thing.
dot.kde.org is certainly a very busy site, is this the kind of memory usage to be expected from Zope? Has anyone seen this kind of problem before?
I just recently started paying attention to Zope memory use when I noticed my process using over 100MB for a virtually empty site. I have since discovered that a couple dozen 300k images that I have are causing it.
Apparently, Zope loads objects in memory in order to send them. This means that several large objects can eat up memory fast. When I ran a test with the ExtImage product, I don't get the Zope bloat I do when using Zope's builtin Image object.
This is a huge problem for me since I have dozens of images in Zope that are viewed *maybe* once every few days. After the first viewer, Zope shoots up in memory use and never goes back down.
We're using Zope 2.2.1 with the hotfixes on a Red Hat Linux box.
I'm using Zope 2.3.0, but I don't think anything has changed in this respect.
Any hints short of switching to another news system appreciated (the SlashCode guys have been on our case). ;)
As Chris pointed out, it could be a memory leak. However, if it goes up to a large amount and hangs around there for a while, it sounds more to me like there are some large objects being served (or many objects being cached???). Others know far more about this than I, but this has been my experience.
I'm very interested in having Zope serve Image/File objects without having to read them into memory first. Running ExtFile/Image is a nice option since it also stores the large files in the file system, but I don't mind a large Data.fs as much as I mind having all of my memory eaten up for no good reason. Perhaps the future Image/File objects can optionally be stored externally and served without the memory bloat. Should I add a feature request to the collector, or does DC already have a solution in mind for this issue?
_______________________
Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
You know what Chris... I'm glad you mentioned this. I just put 2 and 2 together and may have finally gotten 4. I have been testing with 3 different objects: Image, ExtImage, and Photo (http://www.zope.org/Members/Drew/Photo). Here is what I observed: * Zope serving 50 different ExtImage objects of 340k each resulted in no significant increase in memory use (<1MB). * Zope serving 50 different Image objects of 340k each resulted in ~15MB memory increase. Continuing to serve more Image objects that have not already been served did not result in a further increase. * Zope serving the preview display of 50 different Photo objects (340k original size) resulted in ~35MB memory increase. Continuing to serve more Photo objects made Zope's memory use continue climbing. It remained the same for days as long as no unviewed Photo objects were viewed. Viewing different display sizes of Photos already viewed did not increase memory use. After finally studying the Photo code, I discovered that the different sized images are each stored as object attributes (strings using StringIO), not as separate Image objects. So each object has about 6 different strings, each the length of the image of that display size. It seems that this type of object would cause Zope to eat my memory. Can someone verify that this would do it? I'll be a happy camper if this is the problem. I can change the Photo product to store them as Image objects and hopefully get my memory back. _______________________ Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
-----Original Message----- From: Chris McDonough [mailto:chrism@digicool.com] Sent: Monday, February 05, 2001 10:42 PM To: Ron Bickers; Navindra Umanee; zope@zope.org Subject: Re: [Zope] Zope memory usage out of control
Ron, it would be good if you could add a feature request to the Collector about large file object streaming. I'm not superfamiliar with the code there, but I *think* the Image stuff is supposed to break the Image object into smaller subobjects and load each one serially out of the ZODB when it needs to serve them. If it's not doing this properly, it needs to get fixed.
More easily than I imagined, I modified the Photo product to use Image objects instead of string attributes to store the various display sizes. The result... my Zope process is now using ~15MB of memory instead of the 100+MB it used before. I guess I've answered my own question by example. If anyone uses the Photo product and would like my "fix", let me know. The fix will convert existing Photo objects so you don't have to redo anything. In case anyone is interested in a plain Photo product example, I'm building a house and have over a hundred photos of the process at http://www.bickersfamily.org/Photos/House. The Photo product (and a digital camera) made it very easy to throw something half usable together, and my modifications make it far less hungry. Thanks again Chris for that final nugget of info I needed. _______________________ Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Ron Bickers Sent: Tuesday, February 06, 2001 1:03 AM To: Chris McDonough; Navindra Umanee; zope@zope.org Subject: RE: [Zope] Zope memory usage out of control
After finally studying the Photo code, I discovered that the different sized images are each stored as object attributes (strings using StringIO), not as separate Image objects. So each object has about 6 different strings, each the length of the image of that display size.
It seems that this type of object would cause Zope to eat my memory. Can someone verify that this would do it? I'll be a happy camper if this is the problem. I can change the Photo product to store them as Image objects and hopefully get my memory back.
Does a similar memory usage issue occur with file objects? We're interested in using Zope for classroom support in which students will upload Powerpoint and PDF files into Zope for general distribution before using them in classroom presentations. These files can range from 50K to ~ 3M and over the course of a semester we might accumulate more than 20 of them per class. I don't have a problem with the size of data.fs, but caching large numbers of these in RAM isn't necessary. It would seem that the desired functionality for a file would be to serve it from the disk without caching. Jim Harrison Univ. of Pittsburgh
On Tue, 6 Feb 2001 03:01:14 -0500, "Ron Bickers" <rbickers@logicetc.com> wrote:
More easily than I imagined, I modified the Photo product to use Image objects instead of string attributes to store the various display sizes. The result... my Zope process is now using ~15MB of memory instead of the 100+MB it used before. I guess I've answered my own question by example.
FYI, this change will give you an additional memory saving. Normally ZServer stores the whole response in memory for the duration of it being downloaded by the client. For a slow client this may be some time. However, the Image and File objects ensure that this is buffered on disk, by using RESPONSE.write and setting a Content-Length header. (the disk buffering only happens if the length is large) Toby Dickenson tdickenson@geminidataloggers.com
Ron Bickers wrote:
After finally studying the Photo code, I discovered that the different sized images are each stored as object attributes (strings using StringIO), not as separate Image objects. So each object has about 6 different strings, each the length of the image of that display size.
It seems that this type of object would cause Zope to eat my memory. Can someone verify that this would do it? I'll be a happy camper if this is the problem. I can change the Photo product to store them as Image objects and hopefully get my memory back.
Ron, The change you've proposed would have precisely the effect you want. I've been contemplating writing a new product using ZPatterns to replace the Photo Product for exactly this reason. Here are my thoughts so far: ArchiveImageFile ZClass - subclasses Image and DataSkin - stores a single image size - (receives original image data and resizing settings, generates specific size with PIL. Or receives resized image data?) ArchiveImageFiles Specialist - instantiates and stores ArchiveImageFile objects - I *think* that this should receive a single copy of the original file data, and a list of settings, and return the appropriate object references after instantiation of the different ArchiveImageFile objects ArchiveImage ZClass - subclasses DataSkin - contains metadata about the ArchiveImage, and pointers to the appropriate ArchiveImageFile objects - has *specific* sizing settings for ArchiveImageFile object instantiation (can be restored to the default settings from ArchiveImages) ArchiveImages Specialist - instantiates, stores, indexes and searches ArchiveImage objects - maintains the *default* sizing settings for ArchiveImageFile object instantiation The question right now is, where does the PIL code for creating the differently sized versions go. Does the ArchiveImages specialist iterate over the list of settings, generating a different version to hand off to ArchiveImageFiles, or does it hand off the original file data and the settings to each ArchiveImageFile and then get references back for the individual versions to attach to the ArchiveImage? Ponder. Michael Bernstein.
On Monday 05 February 2001 19:41, Chris McDonough wrote:
Ron, it would be good if you could add a feature request to the Collector about large file object streaming. I'm not superfamiliar with the code there, but I *think* the Image stuff is supposed to break the Image object into smaller subobjects and load each one serially out of the ZODB when it needs to serve them. If it's not doing this properly, it needs to get fixed.
looking quickly through the image code from cvs, its apparent that images are broken up into separate persistent chunks when uploaded into a primitive linked list structure thats stored in the zodb. however in serving them zope loads all the chunked objects anyways, (without invalidating the object (thread zodb) cache on any), so there is no benefit on image serving in terms of memory performance, since the entire image needs to be loaded up to serve to the client anyway and the all of the chunks will be resident in the PicklerCache at that point. which brings up an interesting question of how do you remove objects from the PicklerCache. it looks like a call to self._p_jar.cache.gc_item(key,self) will do it?, but it also looks like self._p_changed=None will also work? two notes, 1. this is a very specific condition when you don't want to serve an image very frequently 2. it won't decrease memory usage as reported by ps unless multiple images are requested on a page. (each image = request = transaction?) kapil
participants (9)
-
Andy Dustman -
Chris McDonough -
ender -
ethan mindlace fremen -
Jim Harrison -
Michael Bernstein -
Navindra Umanee -
Ron Bickers -
Toby Dickenson