[Zope] Scaling problems, or something else?

[Gerhard Schmidt]

On Wed, May 16, 2007 at 05:52:16PM -0400, Paul Winkler wrote:
> On Wed, May 16, 2007 at 04:24:56PM -0500, Jens Vagelpohl wrote:
> > There's a difference between scaling and making something faster. ZEO  
> > makes a single instance slower, right. But you can deal with more  
> > requests concurrently using ZEO. That's what I consider "scaling".
> 
> Agreed.  But at the same time, I don't think it makes sense to keep
> deploying more and more badly tuned instances. That's what I consider
> "blind shotgun scaling" :) You need to scale, but you also need to
> tune - and you need to be pragmatic about which is the appropriate
> approach at any given point in time.

I would love to tune our system, but not using ZEO isn't an option. I don't
see how a single Server ever will be able to do what 13 Server do right now.

Our system is a single Instance. With up to 70k requests per hour (20
requests per second). I don't see a way to get rid of the ZEO server. 

We have expirienced some problem with ZEO when requests take to long. 

Sometimes we have Problems like this 
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.29:52569) Transaction blocked waiting for storage. Clients waiting: 1.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.36:56041) Transaction blocked waiting for storage. Clients waiting: 2.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.33:63884) Transaction blocked waiting for storage. Clients waiting: 3.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.34:64355) Transaction blocked waiting for storage. Clients waiting: 4.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.25:63215) Transaction blocked waiting for storage. Clients waiting: 5.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.28:58213) Transaction blocked waiting for storage. Clients waiting: 6.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.30:59149) Transaction blocked waiting for storage. Clients waiting: 7.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.31:58930) Transaction blocked waiting for storage. Clients waiting: 8.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.35:64097) Transaction blocked waiting for storage. Clients waiting: 9.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.27:63627) Transaction blocked waiting for storage. Clients waiting: 10.
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.26:63146) Blocked transaction restarted.  Clients waiting: 9
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.29:52569) Blocked transaction restarted.  Clients waiting: 8
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.36:56041) Blocked transaction restarted.  Clients waiting: 7
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.33:63884) Blocked transaction restarted.  Clients waiting: 6
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.34:64355) Blocked transaction restarted.  Clients waiting: 5
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.25:63215) Blocked transaction restarted.  Clients waiting: 4
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.28:58213) Blocked transaction restarted.  Clients waiting: 3
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.30:59149) Blocked transaction restarted.  Clients waiting: 2
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.31:58930) Blocked transaction restarted.  Clients waiting: 1
------
2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.35:64097) Blocked transaction restarted.

This incident wasn't a Problem because it was resolved within on second.
But sometimes situations like this take up tum 30 seconds to resolve. 
The site is completly unresponsiv in this time and take up to 10 minutes
to resume normal opration (Responsetimes < 1 sec per dynamic page) 

I haven't been able to track down the Problem that causes this. But the 
frequency has droped quite dramatic since we updated our fontend Servers
to more recent CPUs. 

It seams there is a posibility for an deadlock when requests take to much
time to process. 

But the main Problem what we have is the memory growth of the Zope server 
processes. They grow to 500 MB of Memory bevor serving the first request. 
an wile running the constantly growing until the hit the limit of the
physical memory. When they do, they slow down very dramaticaly
(responstimes 800% higher than usual) we have done some debugging and it 
seams that die python garbage collection kicks in and kills the whole
performance. The only solution we have com up with is to restart the zope
Server before the hit the physical memory limit. 

Bye
	Estartu
  
-- 
----------------------------------------------------------------------------
Gerhard Schmidt    | Nick : estartu      IRC : Estartu  |
Fischbachweg 3     |                                    |  PGP Public Key
86856 Hiltenfingen | EMail: estartu at augusta.de          |  on request 
Germany            | 					|  

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 305 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope/attachments/20070521/39ad0d71/attachment.bin