[Zope-dev] Re: [Zope-CMF] Better MemberData search for CMF

[Florent Guillaume]

[The main discussion is focused on CMF, please keep zope-cmf at zope.org in
the Cc.]

Basically we need a search API.

I'm Cc-ing zope-dev, exuserfolder-devel and plone-developers (for GRUF)
because we're talking about user folders. The last part is especially of
interest to you guys.

Jens wrote:
> Florent Wrote:
> > Currently CMF's MemberDataTool has
> >   searchMemberData(search_param, search_term, attributes=())
> > but it can only do a search on one attribute. It returns a list of 
> > dictionaries.
> >
> > CMFLDAP's MemberDataTool inherits the one from CMF, and so actually 
> > does the search only in cached members. This is a big problem for me.
> 
> The way it is done in CMFLDAP stems from how the LDAPUserFolder reacts 
> to calls such as getUsers, which will retrieve all users. Calls like 
> that are inappropriate and even dangerous for any user folder that has 
> a storage with large numbers of records. Zope (and possibly the storage 
> backend used by the user folder) will come to a standstill if all of a 
> sudden thousands of records come flooding back and they all need real 
> Zope user objects created for them.
> 
> The LDAPUserFolder has a notion of caching users to avoid the speed 
> penalty caused by having to go to LDAP. I made the conscious decision 
> to implement getUsers in a way that will only look at the 
> currently-cached users. Anything else is potentially disruptive for the 
> running Zope or the LDAP backend.

Yes I understand the reasons for this behavior. Still it's painful, and
is a proof we need a search API.

> > Plone (in MembershipTool) has
> >   searchForMembers(**kw)
> > that could search on arbitrary attributes but actually only does it on 
> > name, email, roles, and last_login_time. It returns a list of user 
> > objects.
> >
> > CMFLDAP's PloneLDAPMembershipTool seems to be more intelligent than 
> > its MemberDataTool and do the full LDAP search, but I'm not sure.
> 
> The PloneLDAPMembershipTool's searchForMembers is different in response 
> to specific Plone needs. In my opinion it still sucks really bad. If 
> you look at the code in searchForMembers in the standard Plone 
> membership tool you will see that it does the "dangerous" thing of just 
> grabbing all users from the user folder and then looking at each user 
> to see if the search criteria fit.
> 
> There's another item that makes searching members in a CMF-based portal 
> more complicated: Users that you find in the user folder are not 
> necessarily members, even if the user seems to have the Member role. 
> You always end up having to introspect the MemberData tool to see if it 
> has a member data wrapper with the same ID.

Yeah maybe we also need a MembershipTool.hasMember() method ? It's
better than doing 'id in mtool.listMemberIds()' ...

> > I'd like to propose a standardized API on which we can agree for the 
> > future:
> >   MemberDataTool:
> >     searchForMembers(mapping, maybe_some_options=None, **kw)
> > It would search on all the keys of mapping+kw.
> > The options would be there to ask for a substring or exact match. To 
> > be decided later.
> > I'd like it to return a list of users, but this may be problematic 
> > from a security point of view. What do you think? Otherwise we could 
> > make it take an attributes=('email', 'fullname') args, and return a 
> > dict with at least the specified attributes.
> 
> If you want to go the "sequence of dictionaries" route then the 
> properties as defined on the MemberData Tool should be looked up to 
> define a set of standard attributes to return, meaning attributes above 
> and beyond the user name.

Yes.

> > A side problem is that this method will have to have knowledge of how 
> > to do a search in the user folder. And there are lots of those, with 
> > nowhere an standardized API in sight to do search either :-( (actually 
> > I don't know any that can do a search appart from LDAPUserFolder). 
> > Maybe we could take the oportunity to define one ? Searching for 
> > userids, roles, groups (for me), is a prerequisite for me.
> 
> A standard API for such searches on the user folders would be a 
> prerequisite for any of this so that the user folder, which is the only 
> link in this chain that really knows how to make the search efficient, 
> can do so. That would completely eliminate that nasty pattern of "I 
> will grab all users myself and then look at each user object".

I agree, so I'd like to propose an API for this too. I haven't thought
about it much. LDAPUserFolder has
  findUser(search_param, search_term, attrs=[])
but it only searches one one term. So I propose:
  searchUsers(mapping, attributes=(), other_options=..., **kw)
it would work in the same manner as searchForMembers above.
As above, do we make it return a sequence of user objects or a sequence
of dictionaries ? In any case we'll need a way to only return user ids.

Florent

-- 
Florent Guillaume, Nuxeo (Paris, France)
+33 1 40 33 79 87  http://nuxeo.com  mailto:fg at nuxeo.com