[Zope-dev] Sorting for zope.index

[Dan Korostelev]

2008/12/28 Chris McDonough <chrism at plope.com>:
> BTW, one optimization for fieldindex sorting that I'm fooling around with now
> (based on  profiling data from a real-world application) is to:
>
> - Keep around a list of docids in forward value-sorted order.
>
> - Keey around a list of docids in reverse value-sorted order.
>
> - Iterate over these lists within a generator when it's an ascending sort with a
>  limit or when it's a reverse sort; yield back the docids we're passed in the
>  order implied by the list.
>
> When I say "list" above, I actually mean some ZODB-friendly data structure that
> isn't just a plain Python list (that's actually what I'm fooling around with
> creating now).
>
> FTR, doing this would let us get the KeyError-when-docid-isnt-there behavior
> back, and it would make the reverse-sort case (where we use sorted(docids,
> key=rev_index.get, reverse=reverse) far less sucky.
>
> I'll let you know how this goes.

I also thought about that. But I'm not sure if we should add it to
FieldIndex. It could be a subclass like SortingFieldIndex or
something. Because fieldindexes are used wide and zope.index can't
depend on something like zope.app.generations for nice migrating. It
might be a bit overload when sorting is not needed.

Also, the KeyError thing is matter of interface. Even if we create a
fieldindex that can efficiently check and raise KeyError, others may
not, so I'd leave the interface as it is now.

BTW, you might be interested in zc.blist package for the ZODB-friendly
balanced lists structure.

-- 
WBR, Dan Korostelev