[Zope] ZCatalog Strategy

[Dieter Maurer]

Mark Gibson wrote at 2006-3-22 19:28 -0700:
>I'm struggling to weigh the cost of getObject() vs. the cost of adding 
>more metadata to the catalog.  I'll explain my situation.
>
>I have 10,000 widgets cataloged.  I do a path and date query that 
>returns me maybe 12 of these.  Then I have a choice of calling 
>getObject().getData() on each of these, or I could add getData to the 
>catalog metadata.
>
>Does the cost of calling getObject() for a dozen objects justify 
>creating a new metadata field?
>
>More generally how does a large amount of metadata in the catalog affect 
>performance of queries?

This is very difficult to say.


  We have used the standard metadata set for a long time.
  It contains "description". However, in our cases
  "description" was usually several kb large.

  We found that all catalog operations (lookup, indexing)
  were very slow and modifying catalog operations caused
  huge transaktions.

  The "IOBBuckets" containing the metadata records have
  been the culprit. Usually, they contain 45 metadata records.
  If you need one record, you in fact handle usually 45 of them.
  In our case, the buckets usually have been several 100 kB big -- much
  larger then usual container objects...


On the other hand, if you have small metadata records (say, containing
a few integers), then loading them may be much faster
than loading all intermediate objects to your final object -- especially
for applications such as determining statistics data (when you
are processing a lot of objects).


-- 
Dieter