Large user populations and local roles
Here's my situation. I'm starting to develop a Zope-based site that could have 100,000 or more (a lot more!) self-registered users and maybe 20 to 30 content creators for about 15 different content areas with only a few content creators per area. 1. I think the large number of users should be no problem for one of the DB-type user folders (GUF, UserDB, MySQLUF, ... yet to be determined) 2. I think the number of content creators and areas is within reason for using local roles. They won't change often. 3. I can't imagine what will happen when I want to add a local role and Zope tries to build a SELECT list of 100,000 names and sends it to my browser! So am I crazy to use Zope for something this size, or just too dense to see the obvious solution? Regards, -Dennis -- Dennis Nichols nichols@tradingconnections.com
At 11:41 AM 4/1/00 -0600, you wrote:
Here's my situation. I'm starting to develop a Zope-based site that could have 100,000 or more (a lot more!) self-registered users and maybe 20 to 30 content creators for about 15 different content areas with only a few content creators per area.
1. I think the large number of users should be no problem for one of the DB-type user folders (GUF, UserDB, MySQLUF, ... yet to be determined) 2. I think the number of content creators and areas is within reason for using local roles. They won't change often. 3. I can't imagine what will happen when I want to add a local role and Zope tries to build a SELECT list of 100,000 names and sends it to my browser!
So am I crazy to use Zope for something this size, or just too dense to see the obvious solution?
I know that UserDB lets you modify all SQL methods and DTML methods it is using. So just write more intelligent methods, and make use of the great batching features of DTML-IN. Also, in your cases I would develop additional admin methods that allow you to do things faster. For example you want to add a certain role to all your users, just have a special SQL method for that. You may also want to alter your user table. For example give your user table additional fields, like a admin:boolean or whatever you need. UserDB gives you a lot of flexibility. Regards, Stephan -- Stephan Richter - (901) 573-3308 - srichter@cbu.edu CBU - Physics & Chemistry; Framework Web - Web Design & Development PGP Key: 735E C61E 5C64 F430 4F9C 798E DCA2 07E3 E42B 5391
On Sat, 1 Apr 2000, Dennis Nichols wrote:
3. I can't imagine what will happen when I want to add a local role and Zope tries to build a SELECT list of 100,000 names and sends it to my browser!
What will happen is you will patch your Zope installation, and then submit it to the collector for DC to make standard :-) I have around 3000 at the moment, and the default install still works provided I only do security management on my LAN rather than via modem.
So am I crazy to use Zope for something this size, or just too dense to see the obvious solution?
I think you are on the right track. Just avoid calling the getUserNames or getUsers methods in whatever acl_users folder you end up using. Also trace the source code in getUser to ensure that getUserNames is not called at any stage - I can confirm GUF handles this nicely provided you implement the userExists hook, but I don't know about the others. For > 10,000 users, I would actually suggest patching or subclassing whatever userfolder you go for and making the getUserNames and getUsers methods throw an exception or at least log a message, so you can eliminate all the calls. These methods need to remain in the API as plenty of people need them, but are just unworkable on large sites. -- ___ // Zen (alias Stuart Bishop) Work: zen@cs.rmit.edu.au // E N Senior Systems Alchemist Play: zen@shangri-la.dropbear.id.au //__ Computer Science, RMIT WWW: http://www.cs.rmit.edu.au/~zen
On Sun, 2 Apr 2000, Stuart 'Zen' Bishop wrote:
On Sat, 1 Apr 2000, Dennis Nichols wrote:
3. I can't imagine what will happen when I want to add a local role and Zope tries to build a SELECT list of 100,000 names and sends it to my browser!
What will happen is you will patch your Zope installation, and then submit it to the collector for DC to make standard :-)
I have around 3000 at the moment, and the default install still works provided I only do security management on my LAN rather than via modem.
Acutally, if you use an ZSQL method to get the user list, it will probably be cut off at 1000 (I believe that is the default). But a cut-off list is only marginally useful anyway. What is really needed is a small search form that returns a partial user list.
So am I crazy to use Zope for something this size, or just too dense to see the obvious solution?
Hopefully not, or we'll end up in the same cell. -- andy dustman | programmer/analyst | comstar.net, inc. telephone: 770.485.6025 / 706.549.7689 | icq: 32922760 | pgp: 0xc72f3f1d "Therefore, sweet knights, if you may doubt your strength or courage, come no further, for death awaits you all, with nasty, big, pointy teeth!"
Stuart 'Zen' Bishop wrote:
On Sat, 1 Apr 2000, Dennis Nichols wrote:
3. I can't imagine what will happen when I want to add a local role and Zope tries to build a SELECT list of 100,000 names and sends it to my browser!
What will happen is you will patch your Zope installation, and then submit it to the collector for DC to make standard :-)
I have around 3000 at the moment, and the default install still works provided I only do security management on my LAN rather than via modem.
So am I crazy to use Zope for something this size, or just too dense to see the obvious solution?
I think you are on the right track. Just avoid calling the getUserNames or getUsers methods in whatever acl_users folder you end up using. Also trace the source code in getUser to ensure that getUserNames is not called at any stage - I can confirm GUF handles this nicely provided you implement the userExists hook, but I don't know about the others.
For > 10,000 users, I would actually suggest patching or subclassing whatever userfolder you go for and making the getUserNames and getUsers methods throw an exception or at least log a message, so you can eliminate all the calls. These methods need to remain in the API as plenty of people need them, but are just unworkable on large sites.
There are some simple solutions to these problems. The only place in the UserDB in which a query retrieves the entire list of records during the couse of normal authentication is when you are using domain level authentication. When I built a UserDB installation that had 10's of thousands of users, I just copmletely eradicated that code from the product (although the field is still there in the table, for compatibility with zope). Rewriting the page that lists the users so that it displays them in batches is trivial in zope. Additionally, I created a page that allows me to type in a username in order to pull up their record. When updating large numbers of records, it is usually easier to do it with SQL rather than with the Zope interface. If you are using MySQL (or many other databases, for that matter), it is easy to assign a new role to every user with a single query. I don't have the code in front of me, but MySQL provides lots of string manipulation functions, including an append function (althoughb the syntax escapes me right now) Something like Update users, set roles = append(roles, ", new_role") where not instr(roles, "new_role") will add a new role to every user that doesn not already have that role in their list. It probably executes on a table of 20,000 users is well less than a second. --sam -- -------------------------- Sam Gendler CTO, Impossible, Inc. 1222 State St. Suite 250 Santa Barbara, CA. 93101 p: 805-560-0508 f: 805-560-0608 c: 805-689-1191 e: sgendler@impossible.com
participants (5)
-
Andy Dustman -
Dennis Nichols -
Sam Gendler -
Stephan Richter -
Stuart 'Zen' Bishop