[Zope-dev] Sorting for zope.index

[Chris McDonough]

Dan Korostelev wrote:
> 2008/12/28 Chris McDonough <chrism at plope.com>:
>> BTW, one optimization for fieldindex sorting that I'm fooling around with now
>> (based on  profiling data from a real-world application) is to:
>>
>> - Keep around a list of docids in forward value-sorted order.
>>
>> - Keey around a list of docids in reverse value-sorted order.
>>
>> - Iterate over these lists within a generator when it's an ascending sort with a
>>  limit or when it's a reverse sort; yield back the docids we're passed in the
>>  order implied by the list.
>>
>> When I say "list" above, I actually mean some ZODB-friendly data structure that
>> isn't just a plain Python list (that's actually what I'm fooling around with
>> creating now).
>>
>> FTR, doing this would let us get the KeyError-when-docid-isnt-there behavior
>> back, and it would make the reverse-sort case (where we use sorted(docids,
>> key=rev_index.get, reverse=reverse) far less sucky.
>>
>> I'll let you know how this goes.
> 
> I also thought about that. But I'm not sure if we should add it to
> FieldIndex. It could be a subclass like SortingFieldIndex or
> something. Because fieldindexes are used wide and zope.index can't
> depend on something like zope.app.generations for nice migrating. It
> might be a bit overload when sorting is not needed.

I agree.  I'll add it to repoze.catalog for my customer and we'll let it simmer
there a while.

> 
> Also, the KeyError thing is matter of interface. Even if we create a
> fieldindex that can efficiently check and raise KeyError, others may
> not, so I'd leave the interface as it is now.

Right.

> 
> BTW, you might be interested in zc.blist package for the ZODB-friendly
> balanced lists structure.
> 

Thanks!  I knew I remembered something like this this around somewhere... holy
__setitem__ method Batman! ;-)

- C