[Zope-dev] Re: Catalog performance

[Seb Bacon]

Nguyen Quan Son wrote:
> Hi,
> I have a problem with performance and memory consumption when trying to do some statistics, using following code:
> ...
> docs = container.portal_catalog(meta_type='Document', ...)
> for doc in docs:
>     obj = doc.getObject()
>     value = obj.attr
>     ...
> 
> With about 10.000 documents this Python script takes 10 minutes and more than 500MB of memory, after that I had to restart Zope. I
> am running Zope 2.6.1 + Plone 1.0 on Windows 2000, Xeon P4 with 1GB RAM.
> What's wrong with this code? Any suggestion is appreciated.

With getObject(), you're loading entire objects into memory in order to 
grab a single attribute.  This is very wasteful.  Try putting the 
attribute into the metadata for the catalog and grabbing it from there. 
  Then you can do:

  for doc in docs:
      value = doc.attr

seb