Has the physical path of the object changed? If not, the newer (2.3.0 + ) catalog stuff should be smart enough to figure out whether anything inside the object has changed during catalog_object. If nothing has changed, none of the indexes or metadata columns should be updated. We're scrambling right now to write tests for this kind of thing. :-( Note that the algoritm is simple - for each index, compare the what exists in the index to what is to be put in. If they're the same, do nothing. If they're different, reindex. I wasn't able to understand completely from your description whether the object method your're attempting to index via a TextIndex actually returns different data or not when you recatalog it. Does it? There's a huge possibility that the merge stuff isn't working completely because until now there have been no deterministic tests. ----- Original Message ----- From: "R. David Murray " <bitz@bitdance.com> To: <zope-dev@zope.org> Sent: Saturday, March 03, 2001 2:29 PM Subject: Re: [Zope-dev] ZCatalog hackery
I've got an analogous but different problem with ZCatalog udpates. I'd like to ask for ideas about the best way to handle this.
I've got a ZPatterns based ap that is pulling data from a postgresql database. But associated with the object created out of the database are signficant chunks of html (Author bios, book descriptions). I've stored these html chunks in the database as well, though I don't think that matters to the problem or possible solution strategies.
The issue is that parts of the database get updated periodically from an external source. That is, the author and book tables get replaced wholesale. But the Book and Author objects are cataloged using ZCatalog, doing a full text index on a combination of fields from the external database and the html chunks. As far as I can see at the moment, this means that every object has to get uncataloged and recataloged, meaning every time the database update happens the zodb inflates considerably, even though very little data has actually changed.
So far I can think of two simple solutions to this problem: (1) pack often (2) put the Catalog into a mounted storage backed by a non-undoable storage. I'd rather not do either one of these (the first for obvious reasons, the second simply because I don't want to take the time to learn how to set up a non-undoable storage). Am I missing some other obvious options? It seems like there *ought* to be a way to avoid the overhead of updating the catalog for objects that haven't really changed.
--RDM
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )