[Grok-dev] Re: declaring catalog indexes

[Philipp von Weitershausen]

On 16 Apr 2007, at 15:30 , Martijn Faassen wrote:
> Philipp von Weitershausen wrote:
>> Martijn Faassen wrote:
>>> The main drawback to this approach is that all indexes are set up  
>>> centrally. Of course you can write code to adjust the catalog  
>>> later (and have no setup_catalog), but are there patterns we can  
>>> come up with to make the creation of indexes more easy?
>>>
>>> Let's translate the above in another proposal:
>>>
>>> class App(grok.Application):
>>>     grok.local_utility(IntIds, provides=IIntIds)
>>>     grok.local_utility(Catalog, provides=ICatalog,  
>>> name='my_catalog')
>> Why do you make the catalog a named utility? I've seen you like  
>> doing that (hurry.query's API always expects a name for the  
>> catalog, which makes for a bit of a confusing API if the catalog's  
>> an unnamed utility).
>
> I think it's mostly because we started that way with the Document  
> Library, which has multiple catalogs, where that makes sense. We do  
> need to provide for the use case where the catalog is unnamed.

Yes. All I'm saying is that we should avoid *having* to pass in the  
name of the catalog if it's empty. In hurry.query, an unnamed catalog  
causes a lot of method calls to query objects with an empty string;  
it's not always obvious that this empty string actually stands for  
the utility name of the catalog.

>>> class AIndex(grok.FieldIndex):
>>>    grok.catalog(App, 'my_catalog')
>>>    grok.attribute('a_attribute', ISomeInterface)
>> As somebody who knows the AttributeIndex API I would've expected  
>> something like this:
>>   class AIndex(grok.FieldIndex):
>>       grok.catalog(App, 'my_catalog')
>>       field_name = 'a_attribute'
>>       field_callable = False
>>       interface = ISomeInterface
>
> Right, I was looking exactly for people familiar with other Index  
> APIs. :) Then again, this API looks equivalent to that of  
> FieldIndex, which I tried to cover (with a hidden default for the  
> 'callable' argument :).
>
> We could also make this:
>
> grok.index_parameters('a_attribute', ISomeInterface, False)
>
> Thinking about this some more, it's a bit confusing that  
> grok.FieldIndex in fact does not in any way subclass from the  
> *real* FieldIndex.

It doesn't? I thought it would, otherwise I wouldn't have suggested  
spelling out those class attributes.

>> To reduce typing, the "grok.catalog(...)" line could be guessed if  
>> everything is in the same module and there's only one catalog (an  
>> unnamed utility) in the site.
>> Still, I wonder how many people (coming from Archetypes or other  
>> libraries that mangle a bit more information together) would  
>> expect to set the indexing flag directly in the model or in the  
>> schema. While this would mangle up things badly, I could think of  
>> a more compact spelling: write a data schema and then write an  
>> "index schema" next to it:
>>   class IMySchema(Interface):
>>       a_attribute = schema.Int()
>>       b_attribute = schema.Datetime()
>>       def c_method():
>>           pass
>>   class MySchemaIndexes(grok.Indexes):
>>       grok.schema(IMySchema) # this is actually the default
>>       a_attribute = grok.FieldIndex()
>>       b_attribute = grok.TextIndex(lexicon)  # additional parameters
>>       c_method = grok.SetIndex()   # field_callable is automatically
>>                                    # determined from the schema
>> What do you think?
>
> It's an interesting approach. I think I like it. :) It wouldn't  
> work for a flexible run-time configuration of a catalog, but  
> neither would mine and this is more concise. It also allows one to  
> define indexes all over the place.
>
> When the grok.Indexes subclasses are grokked, they would set up a  
> subscriber for Application. This subscriber would install the  
> intids/ catalog if needed (either nameless or being informed by  
> grok.name() on MySchemaIndexes). It would then install the  
> appropriate indexes.
>
> Calling these 'grok.SetIndex' and such would be a bit dangerous  
> however, as people that set up a catalog manually would be tempted  
> to instantiate that and plug it into a catalog, and get very confused.

Good point.

> Perhaps we need to come up with something like grok.indexschema' to  
> import these things from... Perhaps grok.indexschema.Text,  
> grokindexschema.Field, grok.indexschema.Set? It needs to be somehow  
> clear that these things are *descriptions* and not the thing  
> themselves, just like schema fields are descriptions of attributes  
> and not used to implement the attributes themselves.

I like grok.index.{Set|Field|Text|...} or grok.indexschema.{Set|Field| 
Text|...}.

> Also interesting would be something like an 'AutoIndex'. This would  
> figure out from the schema what kind of index to install. If the  
> schema  defines a sequence field, we automatically create a  
> SetIndex. Too far for the first phase though.

Yeah, but a very nice idea.

>>> The indexes are grokked automatically and get created in the  
>>> catalog when 'App' is created using a subscriber. This allows an  
>>> extension modules to create new indexes in the core. Of course  
>>> the app would still need to be reinstalled to enable those  
>>> indexes, but with some clever event registration  
>>> (IApplicationSetup event?) we might be able to handle that quite  
>>> nicely as well.
>> I think after setting up the catalog, we want to send an event  
>> (CatalogAddedEvent or something like that). Only that way we can  
>> ensure that indexes are set up *after* the catalog (Zope doesn't  
>> guarantee ordered execution of event handlers).
>
> Right. Your approach sounds good, except that I don't know how to  
> do named catalogs with that. There would be only a single catalog.  
> My approach though less nice in some ways does allow this.

We could put another directive into the index schema:

   class MySchemaIndexes(grok.Indexes):
       grok.schema(IMySchema) # this is actually the default
       grok.catalog('')       # this is actually the default

       ...

>> The TextIndex from zope.app.catalog.text also takes a lexicon as a  
>> fourth argument, though that too can be set as an attribute like  
>> the others.
>
> Just pass it along to the index definition constructor, indeed.
>
> I like your approach! I might just go ahead and start working on  
> this in a branch.

Cool!