[Zope] Some basic questions on Zope performance/scalability
Jens Vagelpohl
jens@zope.com
Mon, 9 Sep 2002 22:28:16 -0400
first of all, please do not crosspost. this admonishment is part of
every message footer sent out on these lists. i restricted the answer
to zope@zope.org, where this question belongs.
> 1. I made a custom user folder using MySQL as a storage backend. But I
> always feel wary about that User Folder method getUsers(). I have
> about 300,000 users in a MySQL db table. Is this method really meant
> to return that many users? Or the fundamental design of a Zope User
> Folder is intended for a small group of users?
no one forces you to return all users. as a data point, the
LDAPUserFolder product will only return those user objects marked as
"active" in the built-in user object cache at that moment. it will not
try to retrieve all records, which would possibly bring the site to a
complete halt if the LDAP database had thousands of users.
> 2. Again, I have a 300,000-strong user base. Would it be wise to let
> them create a home folder and start adding things of their own? Could
> ZODB's FileStorage model take that much data without any problems or
> without severe perfomance penalty? I'm trying to pull out huge data
> objects as external files (as I did with NeoBoard's attachments), but
> I guess there's a limit to that approach too (and it's seemingly
> stupid not to use ZODB for all the benefits it offers).
FileStorage is more robust than most people think. my personal opinion
is that if you stay away from the current versioning implementation it
will be able to take most abuse you can throw at it.
> Simply put, say you already have 300,000 users with tons of data for
> each of them and you want to put them and their data somewhere in
> and/or outside of Zope. What would you do?
if you want large folders you need to look at BTree Folders, which are
better suited to hold many objects. the BTreeFolder2 product (->
cvs.zope.org underneath "Products) is probably the best at this point.
also, since boundless objects in a folder will make your zope
management interface access to those folders slow (and possibly
impossibele) you should devise a more intelligent path scheme for the
member folder locations. just putting everyone under "Members" is a bad
idea. a suitable scheme (just an example, there are countless others)
would be like this:
- user "joeblow" is under /members/j/jo/joeblow
- user "fredjones" is under /Members/f/fr/janedoe
jens