[Zope-dev] Re: [Repoze-dev] repoze.bfg

[Chris McDonough]

Ian Bicking wrote:
> Chris McDonough wrote:
>>> There's been a tension in the opencore stuff with the catalog, mostly 
>>> that it's easy to setup and use for things, but it doesn't really 
>>> work for things outside of the ZODB.  Or, I guess theoretically you 
>>> could catalog things not in the ZODB, but it's never happened.
>>
>> IMO, mostly it's a matter of deciding what "index" and "catalog" means 
>> for searching.  If you only need full-text search, some indexing 
>> systems are totally inappropriate.  Likewise, if you only need "field" 
>> indexes that match just one particular value, it's sometimes vastly 
>> simpler just to come up with your own system based on, e.g. a 
>> relational table, than it is to try to use somebody else's "indexer" 
>> or "catalog".
> 
> They are similar in that they both need to get information about 
> updates, and a way to reindex.  Full text search can be a little lazier, 
> as being 100% up-to-date isn't such a big issue.

I'd typically handle the "info about updates" bit by sending an event (which I 
suspect cabochon is all about).

>>> That said, there's a real need for cross-system indexing of different 
>>> kinds.  There's text search indexes, but other kinds of more strict 
>>> indexes also make sense.  It would be very handy to have an index 
>>> that wasn't tied to the ZODB, a database, or anything else.  (It 
>>> could be implemented using the ZODB or a database, of course.)
>>
>> Xapian seems like a good choice too.  It has its own persistence 
>> mechanism and reasonable Python bindings (although from experience 
>> maybe slightly memory-leaky).
> 
> Yes, once you get the documents into it.  Actually, it's really a 
> many-to-many notification system that's needed.  We have one that needs 
> documenting and review 
> (http://www.openplans.org/projects/cabochon/project-home) but while it 
> handles notifications and events (as do several other systems), that 
> doesn't cover reindexing the site.  And then you also need a set of 
> useful endpoints, but those can grow over time.

Stuff that I'm not sure about above:

   "many-to-many notification system"

   "reindexing the site"

That said...

I guess what you're saying is that it's hard to send an event that makes it 
clear what needs to be done as a result of the event being sent.

In the case where you have a content node that holds data, that node would be 
attched to the event.  it's pretty clear that you need to do.. that thing 
changed, and the indexing code needs to inspect it and reindex whatever indexes 
you've got sitting around that are willing to index data about those types of nodes.

In the case where you need to deal with the simultaneous case that some row in 
an RDB was changed, it's not quite as clear.  What does it mean and who should 
do what with that event?  It's all policy at that point.  I'm not sure any 
framework helps.  It's just block-and-tackle event reception and reaction, "do 
the needful", isn't it?

- C