I have a Python Script that looks like this: <id>latestphotos</id> <params></params> objects = [] for object in context.objectValues(['Folder','Photo']): if object.meta_type == 'Folder': objects = objects + object.latestphotos() else: objects.append(object) return objects This I run from a DTML Method like this: <dtml-in "_.sequence.sort(latestphotos(), (('bobobase_modification_time',),))" reverse size=4 orphan=0> XXX ... XXX </dtml-in> The script crawls through a folder structure which contains Photo's and subfolders. It returns a healthy list of about 400 Photo's (from the Photo product) which are all object referenses. It works like it's supposed to, but the server leaks like hell when I run this. From about 5-10% of the memory to up to 50-70%!! Having accidently triggered this script I have to restart the server. Any ideas anyone why? Other solutions?
Peter Bengtsson writes:
The script crawls through a folder structure which contains Photo's and subfolders. It returns a healthy list of about 400 Photo's (from the Photo product) which are all object referenses. It works like it's supposed to, but the server leaks like hell when I run this. From about 5-10% of the memory to up to 50-70%!! Having accidently triggered this script I have to restart the server. This is not a leak.
Your script forces Zope to load 400 photos from ZODB into memory. This requires space... Depending on OS (Linux may do it), memory consumption may go down again when the objects are flushed from cache. Dieter
tisdagen den 18 december 2001 23:43 skrev Dieter Maurer:
Peter Bengtsson writes:
The script crawls through a folder structure which contains Photo's and subfolders. It returns a healthy list of about 400 Photo's (from the Photo product) which are all object referenses. It works like it's supposed to, but the server leaks like hell when I run this. From about 5-10% of the memory to up to 50-70%!! Having accidently triggered this script I have to restart the server.
This is not a leak.
Your script forces Zope to load 400 photos from ZODB into memory. This requires space...
But they are just references, right? References to the objects. I can't just use objectIds because the returning list I want to sort by a property on the photoobjects. I'll try ZopeFind or whatever that thing is called. Any other suggestions?
On Wed, 19 Dec 2001 10:00:16 +0100, Peter Bengtsson <mail@peterbe.com> wrote:
But they are just references, right? References to the objects. I can't just use objectIds because the returning list I want to sort by a property on the photoobjects.
Does your script use any attributes from the photo object? any at all? if so, all of its attributes are loaded into memory. Im not sure how the photo product works, but if the photo data is stored as an attribute of the *same* object then this explains the memory growth. The best way to avoid this problem is to store the photo metadata in a separate persistent object from the bulk data, allowing the metadata to be loaded into memory separately from the bulk data. Ive not looked at the photo product, but it possibly could (and probably should) be doing this itself anyway. If thats not how the Photo product works, you could create your own folderish object for storing your photo metadata. Lets call it a PhotoFrame. Each PhotoFrame then contains a single Photo instance. I hope that helps Toby Dickenson tdickenson@geminidataloggers.com
I missed the beginning of this thread, but the memory / attributes problem you describe was a problem in the original photo product, which has been fixed in Ron Bickers' newer product. seb On Wed, 2001-12-19 at 11:11, Toby Dickenson wrote:
On Wed, 19 Dec 2001 10:00:16 +0100, Peter Bengtsson <mail@peterbe.com> wrote:
But they are just references, right? References to the objects. I can't just use objectIds because the returning list I want to sort by a property on the photoobjects.
Does your script use any attributes from the photo object? any at all? if so, all of its attributes are loaded into memory.
Im not sure how the photo product works, but if the photo data is stored as an attribute of the *same* object then this explains the memory growth.
The best way to avoid this problem is to store the photo metadata in a separate persistent object from the bulk data, allowing the metadata to be loaded into memory separately from the bulk data.
Ive not looked at the photo product, but it possibly could (and probably should) be doing this itself anyway.
If thats not how the Photo product works, you could create your own folderish object for storing your photo metadata. Lets call it a PhotoFrame. Each PhotoFrame then contains a single Photo instance.
I hope that helps
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of seb bacon
I missed the beginning of this thread, but the memory / attributes problem you describe was a problem in the original photo product, which has been fixed in Ron Bickers' newer product.
Indeed. If you're using Andrew's Photo product, this is most likely your problem. The old product stored photo data as string attributes. The new one uses Zope Image objects (or optionally ExtImage objects to store the data in the file system). See http://www.zope.org/Members/rbickers/Photo As per what Dieter said, if all the attributes load on accessing the object, the entire image data would load from the old Photo product. This was the most significant reason I wrote the new product. I just ran your script on my set of 453 Photos, and there was zero increase in memory use. Note that if you're going to use ExtImage, I recommend getting the latest CVS version from SourceForge which includes several ExtImage fixes. _______________________ Ron Bickers Logic Etc, Inc.
I missed the beginning of this thread, but the memory / attributes problem you describe was a problem in the original photo product, which has been fixed in Ron Bickers' newer product.
Indeed. If you're using Andrew's Photo product, this is most likely your problem. The old product stored photo data as string attributes. The new one uses Zope Image objects (or optionally ExtImage objects to store the data in the file system). See http://www.zope.org/Members/rbickers/Photo
I'll do that then! Definitly; just a matter of time :)
As per what Dieter said, if all the attributes load on accessing the object, the entire image data would load from the old Photo product. This was the most significant reason I wrote the new product.
I just ran your script on my set of 453 Photos, and there was zero increase in memory use.
Like Toby Dickenson pointed out, if you use any attribute of the object it then takes up all that memory and in fact I do. I do not only use objectValues(). I actually do something like this: <firstly in one script> for photo in context.tested_objectValues('Photo'): <secondly in another> filtered_list=[] for photo in context.objectValues('Photos'): if not hasattr(photo,'avoid'): filtered_list.append(photo) return filtered_list
Note that if you're going to use ExtImage, I recommend getting the latest CVS version from SourceForge which includes several ExtImage fixes.
The renaming problem solved?
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Peter Bengtsson
Like Toby Dickenson pointed out, if you use any attribute of the object it then takes up all that memory and in fact I do.
Before I created new Photo, my site went from using 12MB to 150MB or more of memory use after just viewing some photos. There's no question that the old Photo has that problem and that the new one fixes it.
Note that if you're going to use ExtImage, I recommend getting the latest CVS version from SourceForge which includes several ExtImage fixes.
The renaming problem solved?
If I recall, that problem was solved in the ExtImage product and in Zope itself. The CVS CHANGES.txt lists the recent fixes. _______________________ Ron Bickers Logic Etc, Inc.
-> > > Note that if you're going to use ExtImage, I recommend getting -> > the latest -> > > CVS version from SourceForge which includes several ExtImage fixes. For some of us, using CVS is not convenient. Is there anyway to make an automatic "build" (i.e., tarball) available daily via HTTP? I'm thinking a simple shell script cron job here... By the way, If I'm not using ExtFile/ExtImage, and my file is not stored in the filesystem, then how is the data stored within Zope? I'm planning on using Zope for large (hundreds of megs) video files. Thanks, Derek
-----Original Message----- From: Derek Simkowiak [mailto:dereks@realloc.net]
For some of us, using CVS is not convenient. Is there anyway to make an automatic "build" (i.e., tarball) available daily via HTTP? I'm thinking a simple shell script cron job here...
I believe SourceForge has some features for that. I'll look into it.
By the way, If I'm not using ExtFile/ExtImage, and my file is not stored in the filesystem, then how is the data stored within Zope? I'm planning on using Zope for large (hundreds of megs) video files.
It's stored like every other object, including Zope's Image and File objects... in the ZODB database. If you have 10 video files that are hundreds of MB, you're going to have a Data.fs that's tens of hundreds of MB in size. _______________________ Ron Bickers Logic Etc, Inc.
-----Original Message----- From: Derek Simkowiak [mailto:dereks@realloc.net]
-> > > Note that if you're going to use ExtImage, I recommend getting -> > the latest -> > > CVS version from SourceForge which includes several ExtImage fixes.
For some of us, using CVS is not convenient. Is there anyway to make an automatic "build" (i.e., tarball) available daily via HTTP? I'm thinking a simple shell script cron job here...
Ok. It doesn't appear that SourceForge has any simple way to do this, so I'm going to skip it for now. I did, however, go ahead and release Photo 1.2.3, which is only a bug fix release (see CHANGES.txt). There is no memory leak in prior releases (in case anyone is misled by the subject), but the discussion of memory use problems in the old Photo product nudged me to go ahead and release this. I highly recommend ExtImage users upgrade (and let me know if you have any problems). http://www.zope.org/Members/rbickers/Photo Enjoy! _______________________ Ron Bickers Logic Etc, Inc.
participants (9)
-
Derek Simkowiak -
Dieter Maurer -
Peter Bengtsson -
Ron Bickers -
Ron Bickers -
Ron Bickers -
Ron Bickers -
seb bacon -
Toby Dickenson