Dan Korostelev wrote:

2008/12/28 Chris McDonough <chrism@plope.com>:

...

BTW, one optimization for fieldindex sorting that I'm fooling around with now (based on profiling data from a real-world application) is to:

- Keep around a list of docids in forward value-sorted order.

- Keey around a list of docids in reverse value-sorted order.

- Iterate over these lists within a generator when it's an ascending sort with a limit or when it's a reverse sort; yield back the docids we're passed in the order implied by the list.

When I say "list" above, I actually mean some ZODB-friendly data structure that isn't just a plain Python list (that's actually what I'm fooling around with creating now).

FTR, doing this would let us get the KeyError-when-docid-isnt-there behavior back, and it would make the reverse-sort case (where we use sorted(docids, key=rev_index.get, reverse=reverse) far less sucky.

I'll let you know how this goes.

I also thought about that. But I'm not sure if we should add it to FieldIndex. It could be a subclass like SortingFieldIndex or something. Because fieldindexes are used wide and zope.index can't depend on something like zope.app.generations for nice migrating. It might be a bit overload when sorting is not needed.

I agree. I'll add it to repoze.catalog for my customer and we'll let it simmer there a while.

Also, the KeyError thing is matter of interface. Even if we create a fieldindex that can efficiently check and raise KeyError, others may not, so I'd leave the interface as it is now.

Right.

BTW, you might be interested in zc.blist package for the ZODB-friendly balanced lists structure.

Thanks! I knew I remembered something like this this around somewhere... holy __setitem__ method Batman! ;-) - C