ZEO clusters on a single box
Hi there We have a server running many ZEO clusters (many more than the number of CPUs, of course). Each cluster consists of a master and two clients. Would it make sense to run both clients and spread requests across them? Or would it be better to just increase the number of threads of a single client, and only use the second client for './bin/zopectl debug', and for failover if the first client needs to be restarted? Would it ever make sense to have *more* than 2 clients per master, in this scenario (many more processes than CPUs)? So far, I've had the following response to my questions: """ [...] at every zope/plone conference there are people who claim that adding more threads makes no difference and that you should rather add more zeo clients, but my CS background says more threads can share memory and perform better. But for zope it seems that each threads maintains it's own cache anyway, so there is little memory sharing going on. You may as well run 10 clients with one thread each rather than one client with 10 threads, it would make little difference other than the overhead for the extra process, which is less than 100MB. I'm a little disappointed in zope's memory usage patterns. It doesn't really matter how much memory you have or what sorts of limits you impose, it seems zope always uses more and more until it consumes about 1GB of swap and starts killing the box. So you need to monitor RAM usage with nagios or something and restart the instance every so often. On some of our managed machines we restart zope about once an hour because of this. That just doesn't make good sense, you kill your cache every time. """ RTFM with pointers most appreciated. -- jean . .. .... //\\\oo///\\
--On 13. August 2008 12:26:28 +0700 Jean Jordaan <jean.jordaan@gmail.com> wrote:
Hi there
We have a server running many ZEO clusters (many more than the number of CPUs, of course). Each cluster consists of a master and two clients.
Would it make sense to run both clients and spread requests across them?
Or would it be better to just increase the number of threads of a single client, and only use the second client for './bin/zopectl debug', and for failover if the first client needs to be restarted?
Would it ever make sense to have *more* than 2 clients per master, in this scenario (many more processes than CPUs)?
I think there is no general rule. E.g. we are running a production cluster with 10 ZEO clients where two or three nodes care about long-running requests (like document checkin/checkouts). All other clients deal with the "standard" request. With this approach we can have different configurations for the number of threads and cache-size. The balancing is done based on patterns where we identify the long-running requests through a pattern with the URL. -aj
Jean Jordaan wrote at 2008-8-13 12:26 +0700:
We have a server running many ZEO clusters (many more than the number of CPUs, of course). Each cluster consists of a master and two clients.
Would it make sense to run both clients and spread requests across them?
Usually, this would make sense when your clusters have very different request rates (such that the probability is high, that requests to a single cluster run in parallel while other clusters are lazy). -- Dieter
participants (3)
-
Andreas Jung -
Dieter Maurer -
Jean Jordaan