I am seeing a large amounts of these 2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.12:50602 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.23:49161 Blocked transaction restarted. 2005-08-31T09:05:21 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T09:05:22 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:18:26 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T10:18:28 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:59:08 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T10:59:17 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:59:31 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T11:00:13 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T11:00:15 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T11:00:16 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 2. -----Original Message----- From: Paul Winkler [mailto:pw_lists@slinkp.com] Sent: Wednesday, August 31, 2005 1:33 PM To: Kennamore, Matthew G [NTK] Cc: zope@zope.org Subject: Re: [Zope] Zope scalabilty and problems
Here is our situation, we originally launched zope with a small user base in mind and like most IT implementations it got away from us quickly and we have over 1000 users now. We are experiencing huge slowdowns and lots of complaints of performance.
Let me outline our system:
I have a load balancer that round robins incoming request to 3 zeo servers
Just to get the terminology correct: These are Zope servers. They are ZEO *clients*.
I have 1 zope server on the back en that severs the data, this is a large single processor machine with 4 gigs of ram
I think you mean ZEO server, not Zope server.
Each zeo server is 2 gigs of ram, single processor 3.2 gighz
That is plenty of hardware for most sites...
Short term I plan to change all zeo's to 4 gigs ram and add 3 additional servers.
That might help if your users' typical usage patterns are unavoidably dynamic: e.g. lots of unique catalog queries, lots of personalization, and/or lots of writes. But it's more typical to have quite a lot of reads of rarely-changing data. In which case, the usual low-cost answer to performance issues is: cache, cache, and more cache. There are several levels of caching, which vary in difficulty of setup and potential performance impact. See below...
The Technologies we are using are Zope, Zeo, plone and so forth. Why are we having these large issues? I see sites with hundreds of thousands of users and they don't seem to have these issues? We plan to recode our site to get away from the Data.fs as soon as we can but that is a ways off yet. What am I doing wrong can anyone help?
Plone is very highly configurable, and every one of those runtime options adds a performance cost. You certainly can run high-performance plone sites on your hardware, you just need to tune it properly. The general steps are: 1) Test your performance. See http://plope.com/Members/chrism/loadtesting-methodology/view?searchterm=perf... 2) Profile to see what's slow. 3) Increase caching and/or optimize slow code identified by profiling. Caching is generally a bigger win than optimizing, for less effort. (I have been doing full-time zope work for about 4 years now and I have yet to ever consciously optimize any code ... I try to have efficiency in mind while writing it but that's it... so far there's always been a simple solution via caching. But I don't work on mega-heavy-load sites.) 4) Repeat until satisfied. The best docs I've found on this are: "Scaling Zope", Chris McDonough's presentation from 2003: http://plope.com/misc/szweb Read that one from start to finish at least once. There is also a good chapter on maintaining & optimizing plone in Andy McKay's "Plone book". Another good one is http://web.archive.org/web/20030723200058/http://matthardy.us/mywiki/HighPer... but the original has gone missing so the charts aren't there. -PW