Some basic questions on Zope performance/scalability
Hi all, First, I apologize for posting this. I'm sure the topic has been discussed time and again, but I couldn't find a definite answer to my queries: 1. I made a custom user folder using MySQL as a storage backend. But I always feel wary about that User Folder method getUsers(). I have about 300,000 users in a MySQL db table. Is this method really meant to return that many users? Or the fundamental design of a Zope User Folder is intended for a small group of users? 2. Again, I have a 300,000-strong user base. Would it be wise to let them create a home folder and start adding things of their own? Could ZODB's FileStorage model take that much data without any problems or without severe perfomance penalty? I'm trying to pull out huge data objects as external files (as I did with NeoBoard's attachments), but I guess there's a limit to that approach too (and it's seemingly stupid not to use ZODB for all the benefits it offers). Simply put, say you already have 300,000 users with tons of data for each of them and you want to put them and their data somewhere in and/or outside of Zope. What would you do? Thanks in advance. Best Regards, Wankyu Choi
first of all, please do not crosspost. this admonishment is part of every message footer sent out on these lists. i restricted the answer to zope@zope.org, where this question belongs.
1. I made a custom user folder using MySQL as a storage backend. But I always feel wary about that User Folder method getUsers(). I have about 300,000 users in a MySQL db table. Is this method really meant to return that many users? Or the fundamental design of a Zope User Folder is intended for a small group of users?
no one forces you to return all users. as a data point, the LDAPUserFolder product will only return those user objects marked as "active" in the built-in user object cache at that moment. it will not try to retrieve all records, which would possibly bring the site to a complete halt if the LDAP database had thousands of users.
2. Again, I have a 300,000-strong user base. Would it be wise to let them create a home folder and start adding things of their own? Could ZODB's FileStorage model take that much data without any problems or without severe perfomance penalty? I'm trying to pull out huge data objects as external files (as I did with NeoBoard's attachments), but I guess there's a limit to that approach too (and it's seemingly stupid not to use ZODB for all the benefits it offers).
FileStorage is more robust than most people think. my personal opinion is that if you stay away from the current versioning implementation it will be able to take most abuse you can throw at it.
Simply put, say you already have 300,000 users with tons of data for each of them and you want to put them and their data somewhere in and/or outside of Zope. What would you do?
if you want large folders you need to look at BTree Folders, which are better suited to hold many objects. the BTreeFolder2 product (-> cvs.zope.org underneath "Products) is probably the best at this point. also, since boundless objects in a folder will make your zope management interface access to those folders slow (and possibly impossibele) you should devise a more intelligent path scheme for the member folder locations. just putting everyone under "Members" is a bad idea. a suitable scheme (just an example, there are countless others) would be like this: - user "joeblow" is under /members/j/jo/joeblow - user "fredjones" is under /Members/f/fr/janedoe jens
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Jens Vagelpohl Sent: Monday, September 09, 2002 7:32 PM To: Wankyu Choi; zope@zope.org Subject: Re: [Zope] Some basic questions on Zope performance/scalability
- user "joeblow" is under /members/j/jo/joeblow - user "fredjones" is under /Members/f/fr/janedoe
err, that would be /Members/f/fr/fredjones of course
jens
It's the Internet. Fred can be Jane if she wants to.
Please do not cross-post! I don't know that getUsers is used anywhere outside of the standard User folder managment interface. If you therefore design a custom managment interface for your user folder, you can simply have it always return an empty list. There is also a parallel method "getUserNames" maybe you can just implement this one, or even have it always return an empty list. Then define custom methods for managing your users, or have no user management interface in Zope at all. That would be easier yet ;^) As for where to store them, I'm no LDAP expert but that might be an avenue to explore. The LDAPUserFolder for Zope is quite nice and highly scalable so I hear. hth, -Casey On Monday 09 September 2002 09:22 pm, Wankyu Choi wrote:
Hi all,
First, I apologize for posting this. I'm sure the topic has been discussed time and again, but I couldn't find a definite answer to my queries:
1. I made a custom user folder using MySQL as a storage backend. But I always feel wary about that User Folder method getUsers(). I have about 300,000 users in a MySQL db table. Is this method really meant to return that many users? Or the fundamental design of a Zope User Folder is intended for a small group of users?
2. Again, I have a 300,000-strong user base. Would it be wise to let them create a home folder and start adding things of their own? Could ZODB's FileStorage model take that much data without any problems or without severe perfomance penalty? I'm trying to pull out huge data objects as external files (as I did with NeoBoard's attachments), but I guess there's a limit to that approach too (and it's seemingly stupid not to use ZODB for all the benefits it offers).
Simply put, say you already have 300,000 users with tons of data for each of them and you want to put them and their data somewhere in and/or outside of Zope. What would you do?
Thanks in advance.
Best Regards,
Wankyu Choi
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
participants (4)
-
Casey Duncan -
Charlie Reiman -
Jens Vagelpohl -
Wankyu Choi