We want to build a image database driven website. Visitors will trigger searches resulting in pages that roughly resemble Goggle's Image search results page: <http://images.google.com/images?q=cars> As in Goggle a click on a thumbnail will lead to a page with a <single view> of the picture (max 600 px wide). If the original still is bigger, there might be a link to a full-size view. The database will hold 8,000 - 10,000 pictures for a start, most of them small gifs. Image types will mostly be GIF and JPEG. Every image needs a <thumbnail>, a <single view> and a <full view>. Also there should be all kinds of metadata (jobs we used it for, creator, date of creation, keywords, description, notes, etc) Now the ?'s: If we chose to use the ZODB, how should the images with all their thumbnails and metadata stored for fast access? What would be the best strategy? Create some kind of object (folder) containing the thumbnails and metadata for each single image? Or better separate the thumb's from the original image? Or just dump everything into the same BTreeFolder? And would ZODB be fast enough? Or are there better solutions if speed matters? Lalo described they use PostgreSQL but if possible I'd like to keep everything inside Zope (maybe place the bare image files in an external file system). Might it make sense to generate the thumbnails on the fly when requested? So we only have the "original" files in the database (I guess not if speed we need). Is the property of a picture the right place to store the metadata? If so I'm not really sure what to do if I create a new property called "client" but the values could be many at the same time (Nike + Adidas + Puma) for a single image ... how will this be handled? thanks for any answers! 'K:?
Hi, There are a number of photo products that let you do stuff like this for Zope and which might make an excellent start for this project. I have tried: http://www.zope.org/Members/rbickers/Photo/ with some success.
And would ZODB be fast enough? Or are there better solutions if speed matters? Lalo described they use PostgreSQL but if possible I'd like to keep everything inside Zope (maybe place the bare image files in an external file system). I would store the images external, but on the other hand, if you find a product that works for you, go with the product first.
Douwe Osinga http://douweosinga.com
Hi, I am experiencing problems with the amount of memory Zope is using. I'm using Zope 2.6.2 and Windows 2000 server. Now, things work fine for days and then suddenly something goes wrong and the amount of memory in use by Zope increases until 2Gb or so and the process stops responding completely. My current theory is that this happens when some requests take very long to answer (ie. 1 minute or so). Something else must be waiting for the requests to finish, but before it does, something goes wrong. In my current setup, I have an update process that takes some time and that is hidden in a transparent pixel. User goes to a page, page loads fine & relatively fast, except for one pixel, which only loads when a database cleanup is ready. Cron would be more elegant, but it should work, right? Is there anybody having the same problems, or even better, a solution to this? Douwe Osinga http://douweosinga.com
Douwe Osinga wrote at 2003-10-29 10:52 +0100:
I am experiencing problems with the amount of memory Zope is using. I'm using Zope 2.6.2 and Windows 2000 server. Now, things work fine for days and then suddenly something goes wrong and the amount of memory in use by Zope increases until 2Gb or so and the process stops responding completely.
My current theory is that this happens when some requests take very long to answer (ie. 1 minute or so).
This may be so but only as a side effect: Something that takes long can grab lots of memory.... Otherwise, there is no strict connection between long running requests and high memory usage. I see 3 potential causes (there may be more): * you have a piece of broken code that consumes memory in an infinite loop. You could use the "-M" option of "z2.py" to log all requests and find out the requests that bring your Zope down. From these requests, you could dig deeper to locate the broken code. * you have memory leaks -- pieces of code that leak (usually) small amounts of memory. Over time, you loose more and more memory. You can use Shane's "LeakFinder" product to track this down. * you really need that much memory This may be the case, e.g., for large "ZCatalog" updates or mass imports. In some cases, you can use "subtransaction"s to reduce the memory burden. I do not expect that you have this case. -- Dieter
Hi Dieter, Thanks for the tips.
Douwe Osinga wrote at 2003-10-29 10:52 +0100:
I am experiencing problems with the amount of memory Zope is using. I'm using Zope 2.6.2 and Windows 2000 server. Now, things work fine for days and then suddenly something goes wrong and the amount of memory in use by Zope increases until 2Gb or so and the process stops responding completely.
My current theory is that this happens when some requests take very long to answer (ie. 1 minute or so).
I see 3 potential causes (there may be more): [...] * you have memory leaks -- pieces of code that leak (usually) small amounts of memory. Over time, you loose more and more memory. You can use Shane's "LeakFinder" product to track this down. I installed the product and found out that when the memory consumption grows, I also get a *lot* of instances of IOBucket and OOBucket and similar named objects, but mostly DateTime objects. Since I had never heard of the first kind of objects and they sounded rather like basic building blocks, I put a patch on the DateTime. It gave me references to pieces of code that without a miss were doing lookups from the ZCatalogue, i.e. a ZClass that was instantiated with something like:
zobj = context.Catalog.getobject( catitem.data_record_id_ ) where zobj has properties of DateTime type, so that might be the problem. The logs actually suggest that this statement is then executed for 300 times a second, though the surrounding code does not seem to do that. Now, the ZClass is question is also derived from a class I created myself in Python, so that is probably where the memory leak is to be found. The thing is that I don't know what I'm looking for: how do I create a memory leak in Python? Doesn't the garbage collector clear these things? Any help is much appreciated. Douwe Osinga
Douwe Osinga wrote at 2003-11-18 18:01 +0100:
... I installed the product and found out that when the memory consumption grows, I also get a *lot* of instances of IOBucket and OOBucket and similar named objects,
These are data types from the "BTrees" package. They are extensively used for indexes and the ZCatalog (among others).
but mostly DateTime objects.
They, too, probably come from the catalog.
... zobj = context.Catalog.getobject( catitem.data_record_id_ )
where zobj has properties of DateTime type, so that might be the problem. The logs actually suggest that this statement is then executed for 300 times a second, though the surrounding code does not seem to do that.
Wow! Quite fast....
Now, the ZClass is question is also derived from a class I created myself in Python, so that is probably where the memory leak is to be found. The thing is that I don't know what I'm looking for: how do I create a memory leak in Python? Doesn't the garbage collector clear these things?
Any help is much appreciated.
It is quite difficult to track down memory leaks. First of all: Python's main garbage collector uses reference counting (and therefore is not a standard garbage collector). It can collect only non-cyclic garbage. Cyclic garbage contains an unreachable object with a reference (usually indirect) to itself. Modern Python version have a collecting garbage collector as well which is able to recycle also cyclic garbage. However, external C extensions must play with it and "ExtensionClass" (base class of almost all Zope classes) does not yet do that. Thus, Zope objects containing cyclic references are (usually) not collected. Furthermore, the cyclic garbage collector will not release and recycle garbage that contains objects with destructors ("__del__" methods). This is because reclaiming such cycles may lead to non-deterministic behaviour depending on the order of destructor calls. That's for background information... Now to some techniques how to find the leak: * Go to "Control_Panel" -> "Database Management" --> "Flush cache" and press "Minimize". This will clear the ZODB caches (which contribute to your reference counts) * Reexamine the reference counts. Almost all of the "DateTime" and "*Bucket", "*Set" objects should have gone. If you still have lots of objects of some class, then such objects may be part of cycles. Examine there code. If these steps do not reveal something strange -- let your system run for quite a while and then redo the steps above. Memory leaks are persistent -- they never go away, this means, when you wait long enough, it gets easier to recognize the leaking object class. -- Dieter
On 28 Oct 2003 at 21:10, you wrote: Date sent: Tue, 28 Oct 2003 21:10:48 +0100 From: Kai Vermehr <k@eboy.com> To: ZOPE mailingliste <zope@zope.org> Subject: [Zope] image meta
We want to build a image database driven website.
I used the ZODB and CMFPhoto and CMFPhotoAlbum to create an archive site that has a few thousand images within Plone. I like the additional metadata that Plone provides, including multiple keywords (subjects) that could be defined and searched on. The database is around 250-300Mb. Platform is W2K. Have a look at http://fishability.biz/Doig Cliff ============================================ fishAbility Management and Communications Cliff Quinn and Sam Quinn Victoria, BC, Canada (250) 727-7879 http://fishAbility.biz/
Do you have GIFs with transparency or only JPEGs? I have Photo installed but it seems to convert everything ... where are the original files? And I'm to stupid to find a decent manual ... I'll check CMF Photo ... 'K:) on 29.10.2003 19:24 Uhr, Cliff Quinn at cquinn@fishAbility.biz wrote:
On 28 Oct 2003 at 21:10, you wrote:
Date sent: Tue, 28 Oct 2003 21:10:48 +0100 From: Kai Vermehr <k@eboy.com> To: ZOPE mailingliste <zope@zope.org> Subject: [Zope] image meta
We want to build a image database driven website.
I used the ZODB and CMFPhoto and CMFPhotoAlbum to create an archive site that has a few thousand images within Plone. I like the additional metadata that Plone provides, including multiple keywords (subjects) that could be defined and searched on. The database is around 250-300Mb. Platform is W2K.
Have a look at http://fishability.biz/Doig
Cliff
============================================ fishAbility Management and Communications Cliff Quinn and Sam Quinn Victoria, BC, Canada (250) 727-7879 http://fishAbility.biz/
-- Kai Vermehr | eBoy http://www.eboy.com
participants (4)
-
Cliff Quinn -
Dieter Maurer -
Douwe Osinga -
Kai Vermehr