[Zope] Zope scalabilty and problems

[Kennamore, Matthew G [NTK]]

I am seeing a large amounts of these
2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.12:50602 Transaction blocked waiting for storage. Clients waiting: 1.
2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.23:49161 Blocked transaction restarted.
2005-08-31T09:05:21 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1.
2005-08-31T09:05:22 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted.
2005-08-31T10:18:26 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1.
2005-08-31T10:18:28 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted.
2005-08-31T10:59:08 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1.
2005-08-31T10:59:17 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted.
2005-08-31T10:59:31 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 1.
2005-08-31T11:00:13 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted.
2005-08-31T11:00:15 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1.
2005-08-31T11:00:16 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 2.



 -----Original Message-----
From: 	Paul Winkler [mailto:pw_lists at slinkp.com] 
Sent:	Wednesday, August 31, 2005 1:33 PM
To:	Kennamore, Matthew G [NTK]
Cc:	zope at zope.org
Subject:	Re: [Zope] Zope scalabilty and problems

> Here is our situation, we originally launched zope with a small user
> base in mind and like most IT implementations it got away from us
> quickly and we have over 1000 users now.  We are experiencing huge
> slowdowns and lots of complaints of performance.
>
> Let me outline our system:
>
> I have a load balancer that round robins incoming request to 3 zeo
> servers

Just to get the terminology correct: These are Zope servers. They are ZEO
*clients*.

> I have 1 zope server on the back en that severs the data, this is a
> large single processor machine with 4 gigs of ram

I think you mean ZEO server, not Zope server.

> Each zeo server is 2 gigs of ram, single processor 3.2 gighz

That is plenty of hardware for most sites...

> Short term I plan to change all zeo's to 4 gigs ram and add 3 additional
> servers.

That might help if your users' typical usage patterns are unavoidably
dynamic: e.g. lots of unique catalog queries, lots of personalization,
and/or lots of writes.

But it's more typical to have quite a lot of reads of rarely-changing data.
In which case, the usual low-cost answer to performance issues is:
cache, cache, and more cache. There are several levels of caching, which
vary in difficulty of setup and potential performance impact. See below...

> The Technologies we are using are Zope, Zeo, plone and so forth.  Why
> are we having these large issues?  I see sites with hundreds of
> thousands of users and they don't seem to have these issues?  We plan to
> recode our site to get away from the Data.fs as soon as we can but that
> is a ways off yet.  What am I doing wrong can anyone help?

Plone is very highly configurable, and every one of those runtime options
adds a performance cost. You certainly can run high-performance plone
sites on your hardware, you just need to tune it properly.


The general steps are:

1) Test your performance. See
http://plope.com/Members/chrism/loadtesting-methodology/view?searchterm=performance

2) Profile to see what's slow.

3) Increase caching and/or optimize slow code identified by profiling.
Caching is generally a bigger win than optimizing, for less effort.
(I have been doing full-time zope work for about 4 years now and I have
yet to ever consciously optimize any code ... I try to have efficiency in
mind while writing it but that's it... so far there's always been a simple
 solution via caching.  But I don't work on mega-heavy-load sites.)

4) Repeat until satisfied.

The best docs I've found on this are:
"Scaling Zope", Chris McDonough's presentation from 2003:
http://plope.com/misc/szweb
Read that one from start to finish at least once.


There is also a good chapter on maintaining & optimizing plone in Andy
McKay's "Plone book".

Another good one is
http://web.archive.org/web/20030723200058/http://matthardy.us/mywiki/HighPerformanceZope
but the original has gone missing so the charts aren't there.


-PW