[Zope-dev] Zcatalog bloat problem (berkeleydb is a solution?)

[Chris McDonough]

> A solution might be a kind of "lazy catalog awareness": Instead of
> mangling a new object through one or more catalogs when it is created,
> this object could be added to a list of objects to be cataloged later.
> This way, the transaction to insert a new object would become much
> "cheaper". I'm working on this, but right now it is quite messy. (I'm
> new to Python and Zope, and hence I'm stumbling over a few, hmmm,
> trip-wires...)

This purpose aligns well with those of the ArmoredCatalog proposal as well..
see http://dev.zope.org/Wikis/DevSite/Proposals/ArmoredCatalog .

> But even using such a "lazy catalog awareness", you might get into
> trouble. Using the ZCatalog's "find objects" function, I hit the limits
> of my Linux box: 640 MB RAM were not enough...

This should not happen.  :-(

I'm really disappointed that the bloat and memory consumption issues are
still plaguing the ZCatalog.  At one point, I really thought we had it
pretty much licked.  I suppose this was naive.

> A few weeks ago, I've posted this (admittedly not fully cooked) patch to
> this list, but did not get yet any response.

I apologize for this.  We have a fairly formalized process for handling
feature-ish collector issues, and this hasn't come round on the guitar.  I'm
beyond disappointed that people are still having unacceptable bloat, enough
that something like this patch needed to be submitted.  It's disheartening.
:-(

- C