[Zope-CMF] Large User Database...

[Joachim Werner]

Hi!

> I have an existing client site with 40,000+ registered users.  I want to
> convert them into the CMF for the Portal capabilities.  The CMF, as is,
will
> handle 90% of their needs.  I can code the rest.  Their site is focused on
> product technical support.  My testing thus far has proven the existing
> system code will port easily with very little rewriting.
>
> The big question is...  Can the standard CMF handle a 40k+ user base, with
> over 100k+ page hits per day?  Average data transferred is about 5 gig per
> day.

a) Handling the user base

Having 40.000 users in one acl_users folder is rather impossible. But with
LDAP user folders or an SQL-based solution everything should be fine. It's
mainly a question of how the user management frontend looks like ...

c) Handling 40.000 data folders for the users ...

... could be a problem. You certainly should not put them all in one
hierarchy level. Do you actually need data folders for every user? If yes,
the user folders should be put into a B-Tree-Folder. You might also hit some
limitations of the data.fs file size, but there are well-known ways to get
around that (as Mark describes in the other posting). You might also have a
look at the various products that store part of the data in the filesystem.
The filesystem-based image and file objects even exist in a CMF-version
already. If all that doesn't help, try alternative data storages like the
BerkeleyDB storage ...

c) Handling the load

Zope certainly can do this, but maybe not out-of-the-box on a "small"
machine. With highly dynamic pages, you get something like 10-15 pages per
second from a single Pentium III 800 MHz server with "enough RAM". This is
not a real benchmark test, just a rule-of-thumb I've come up with after
doing some ab (Apache Benchmark) test runs. That's probably a bit too little
for you.

But you can tune a lot:
- use RAM-based caching
- use Proxy caching
- use ZEO and add more servers
- use larger servers (Zope scales mainly with the processor power and RAM;
I'm not sure about how dual-processor machines will scale ...)

With caching, you can make sure that the Zope core will not have to handle
requests for static parts of your site, like images, downloads etc. too
often. In your specific case, you'll probably have a lot of data that
doesn't change too often. Or are the 40.000+ users all editors? ;-)

BTW: Uptime in Austria run Zope on Alpha servers from Compaq. They seem to
be quite fast ...

Cheers

Joachim