[Zope] Large user populations and local roles

[Sam Gendler]

Stuart 'Zen' Bishop wrote:
> 
> On Sat, 1 Apr 2000, Dennis Nichols wrote:
> 
> > 3. I can't imagine what will happen when I want to add a local role and
> > Zope tries to build a SELECT list of 100,000 names and sends it to my browser!
> 
> What will happen is you will patch your Zope installation, and then
> submit it to the collector for DC to make standard :-)
> 
> I have around 3000 at the moment, and the default install still works
> provided I only do security management on my LAN rather than via modem.
> 
> > So am I crazy to use Zope for something this size, or just too dense to see
> > the obvious solution?
> 
> I think you are on the right track. Just avoid calling the
> getUserNames or getUsers methods in whatever acl_users folder you
> end up using. Also trace the source code in getUser to ensure that
> getUserNames is not called at any stage - I can confirm GUF handles
> this nicely provided you implement the userExists hook, but I don't
> know about the others.
> 
> For > 10,000 users, I would actually suggest patching or subclassing
> whatever userfolder you go for and making the getUserNames and getUsers
> methods throw an exception or at least log a message, so you can
> eliminate all the calls. These methods need to remain in the API as plenty
> of people need them, but are just unworkable on large sites.

There are some simple solutions to these problems.  The only place in
the UserDB in which a query retrieves the entire list of records during
the couse of normal authentication is when you are using domain level
authentication.  When I built a UserDB installation that had 10's of
thousands of users, I just copmletely eradicated that code from the
product (although the field is still there in the table, for
compatibility with zope).  Rewriting the page that lists the users so
that it displays them in batches is trivial in zope.  Additionally, I
created a page that allows me to type in a username in order to pull up
their record.

When updating large numbers of records, it is usually easier to do it
with SQL rather than with the Zope interface.  If you are using MySQL
(or many other databases, for that matter), it is easy to assign a new
role to every user with a single query.  I don't have the code in front
of me, but MySQL provides lots of string manipulation functions,
including an append function (althoughb the syntax escapes me right now)

Something like

Update users, set roles = append(roles, ", new_role") where not
instr(roles, "new_role")

will add a new role to every user that doesn not already have that role
in their list.  It probably executes on a table of 20,000 users is well
less than a second.

--sam


-- 
--------------------------
Sam Gendler
CTO, Impossible, Inc.
1222 State St. Suite 250
Santa Barbara, CA. 93101
p: 805-560-0508
f: 805-560-0608
c: 805-689-1191
e: sgendler@impossible.com