A solution might be a kind of "lazy catalog awareness": Instead of mangling a new object through one or more catalogs when it is created, this object could be added to a list of objects to be cataloged later. This way, the transaction to insert a new object would become much "cheaper". I'm working on this, but right now it is quite messy. (I'm new to Python and Zope, and hence I'm stumbling over a few, hmmm, trip-wires...)
This purpose aligns well with those of the ArmoredCatalog proposal as well.. see http://dev.zope.org/Wikis/DevSite/Proposals/ArmoredCatalog .
But even using such a "lazy catalog awareness", you might get into trouble. Using the ZCatalog's "find objects" function, I hit the limits of my Linux box: 640 MB RAM were not enough...
This should not happen. :-( I'm really disappointed that the bloat and memory consumption issues are still plaguing the ZCatalog. At one point, I really thought we had it pretty much licked. I suppose this was naive.
A few weeks ago, I've posted this (admittedly not fully cooked) patch to this list, but did not get yet any response.
I apologize for this. We have a fairly formalized process for handling feature-ish collector issues, and this hasn't come round on the guitar. I'm beyond disappointed that people are still having unacceptable bloat, enough that something like this patch needed to be submitted. It's disheartening. :-( - C