[Zope-dev] Zcatalog bloat problem (berkeleydb is a solution?)

[abel deuring]

Chris McDonough wrote:
> 
> > A solution might be a kind of "lazy catalog awareness": Instead of
> > mangling a new object through one or more catalogs when it is created,
> > this object could be added to a list of objects to be cataloged later.
> > This way, the transaction to insert a new object would become much
> > "cheaper". I'm working on this, but right now it is quite messy. (I'm
> > new to Python and Zope, and hence I'm stumbling over a few, hmmm,
> > trip-wires...)
> 
> This purpose aligns well with those of the ArmoredCatalog proposal as well..
> see http://dev.zope.org/Wikis/DevSite/Proposals/ArmoredCatalog .

I'm afraid that I did not understand much of the discussion on this
page. (But I don't intend to become a ZODB developer, so I'll simply
ignore it...) But if I'm right, this "lazy catalog awareness" would
mainly mean that ArmoredCatalog gets "official API calls" (1) to add
object to an update or delete list and (2) to index/unindex the objects
in these list. I think that this would be really useful.

> 
> > But even using such a "lazy catalog awareness", you might get into
> > trouble. Using the ZCatalog's "find objects" function, I hit the limits
> > of my Linux box: 640 MB RAM were not enough...
> 
> This should not happen.  :-(
> 
> I'm really disappointed that the bloat and memory consumption issues are
> still plaguing the ZCatalog.  At one point, I really thought we had it
> pretty much licked.  I suppose this was naive.
> 
> > A few weeks ago, I've posted this (admittedly not fully cooked) patch to
> > this list, but did not get yet any response.
> 
> I apologize for this.  We have a fairly formalized process for handling
> feature-ish collector issues, and this hasn't come round on the guitar.  I'm
> beyond disappointed that people are still having unacceptable bloat, enough
> that something like this patch needed to be submitted.  It's disheartening.

Chris,

never mind :) It's also my fault: I'm contended with reading a mailing
list (and sometimes making more or less clever comments), and I know
that Zope has far more elaborate discussion systems, like Wikis and fish
bowls. Only that I'm too lazy to scan through all this stuff to find a
better place for comments... And I know that it's fairly easy to miss a
mail :) 

Abel