[Zope3-Users] Efficiently retrieving objects from an index ordered after an object attribute.

[Hermann Himmelbauer]

Hi,
I have a class looking similar to this:

class person
  birthdate = None   # Some date
  name = None # First name
  gender = None # Male/Femail
  
Objects of this class are then added to a catalog, for which I defined an 
index (zc.catalog -> ValueIndex) for each attribute.

What I want to do now is to retrieve e.g. all male persons but ordered by age 
(birthday), while the order should be choosable between ascending/descending. 
Moreover, I have to take in mind that my database may hold a massive amount 
of person objects, so the solution should be somehow efficient.

I thought about two solutions:

1) Simply do a catalog.searchResults(gender = {'any_of' : ('male, )}, then put 
the results in some list of tuples, where the tuple consists of (birthdate, 
personObject) and do a list.sort().
This solution is, however not very efficient

2) In the birthdate-index, persons are already correctly sorted into a BTree. 
So I thought about using this index somehow, e.g. by adding the index to my 
query, e.g.:

catalog.searchResults(birthdate={'any' : None}, gender={'any_of':('male,)}

However, this seems not to work due to the code in zope.app.catalog.catalog, 
class Catalog, function apply() (~ line 135):

---------------- snip -------------
        results.sort() # order from smallest to largest

        _, result = results.pop(0)
        for _, r in results:
            _, result = self.family.IF.weightedIntersection(result, r)

        return result
------------- snip -------------

First of all, I don't know what exactly weightedIntersection() does, moreover, 
it seems, that the smaller set is regarded first, which means that 
the "birthdate"-index is more or less not regarded.

So maybe, I should create an own catalog class that inherits from Catalog, 
which overrides the apply() function and maybe implements a custom 
searchResults() function which implements ordering?

3) Just use the "birthdate" index for ordering and do the rest of the 
filtering manually. This may be even faster than (1) as sorting the results 
will be quite costly.

Another issue is that I want to batch the results in my HTML-page, so I'd 
favour a solution where I can select e.g. the first 50 results, or, even 
better, a result set (e.g. 150-200).

Any ideas on an effective solution to this problem?

Best Regards,
Hermann

-- 
hermann at qwer.tk
GPG key ID: 299893C7 (on keyservers)
FP: 0124 2584 8809 EF2A DBF9  4902 64B4 D16B 2998 93C7