Scaling problems, or something else?
Just a quick call for ideas on this problem we have... Setup: Zope 2.7.5 (~180 sites) -> apache (-> varnish for 1 high profile site) Most noticeable symtoms: Takes 30 sec. or more to serve a page, or times out. Sporadic problem, but allways during general high load. Lasts less than 1 hour. Restarting zope does not help. Lots of apache processes in '..reading..' state Apache accesses and volume is down. Server load is unchanged, and < 2.0 Apache processes is way up (~250 aganinst <40) Netstat "established" connections is WAY up (~650 aganist < 50) Is this zope hitting some sort of limit and just letting Apache hang? Would setting up ZEO on the same box make a difference, or would it be better to extend varnish coverage? What would you do to pinpoint the problem? Any ideas are welcome! Regards Gaute Amundsen
On Wed, May 09, 2007 at 12:07:54PM +0200, Gaute Amundsen wrote:
Just a quick call for ideas on this problem we have...
Setup: Zope 2.7.5 (~180 sites) -> apache (-> varnish for 1 high profile site)
Most noticeable symtoms: Takes 30 sec. or more to serve a page, or times out. Sporadic problem, but allways during general high load. Lasts less than 1 hour. Restarting zope does not help. Lots of apache processes in '..reading..' state Apache accesses and volume is down. Server load is unchanged, and < 2.0 Apache processes is way up (~250 aganinst <40) Netstat "established" connections is WAY up (~650 aganist < 50)
The increase in netstat connections and apache processes indicates lots of simultaneous traffic, but it's interesting that Apache accesses is down. Since hits are logged only on completion, it may be that many of the requests are hung.
Is this zope hitting some sort of limit and just letting Apache hang? Would setting up ZEO on the same box make a difference,
ZEO doesn't buy you any performance unless you have multiple Zope clients reading from it, and a load balancer in front. This will help IF your application is CPU-bound, which yours is not (I assume by server load you mean CPU). ZEO can actually *hurt* if you're IO-bound, because it adds network overhead to ZODB reads and writes. It's very bad if you have large Image or File objects (which you probably shouldn't have in the ZODB anyway).
or would it be better to extend varnish coverage?
Probably a good idea anyway... but you want to find out what the problem really is.
What would you do to pinpoint the problem?
I'd first try hitting Zope directly during the problem to see if the slowdown is there. If so, I'd then try either: - DeadlockDebugger may be informative. http://www.zope.org/Members/nuxeo/Products/DeadlockDebugger - Enable Zope's trace log and use requestprofiler.py to see if there is a pattern to the requests that trigger the problem. Eg. maybe all your zope worker threads are waiting on some slow IO task. See the logger section of zope.conf. -- Paul Winkler http://www.slinkp.com
On Wednesday 09 May 2007 16:41, Paul Winkler wrote:
On Wed, May 09, 2007 at 12:07:54PM +0200, Gaute Amundsen wrote:
Just a quick call for ideas on this problem we have...
Setup: Zope 2.7.5 (~180 sites) -> apache (-> varnish for 1 high profile site)
Most noticeable symtoms: Takes 30 sec. or more to serve a page, or times out. Sporadic problem, but allways during general high load. Lasts less than 1 hour. Restarting zope does not help. Lots of apache processes in '..reading..' state Apache accesses and volume is down. Server load is unchanged, and < 2.0 Apache processes is way up (~250 aganinst <40) Netstat "established" connections is WAY up (~650 aganist < 50)
The increase in netstat connections and apache processes indicates lots of simultaneous traffic, but it's interesting that Apache accesses is down. Since hits are logged only on completion, it may be that many of the requests are hung.
That was my reasoning too.
Is this zope hitting some sort of limit and just letting Apache hang? Would setting up ZEO on the same box make a difference,
ZEO doesn't buy you any performance unless you have multiple Zope clients reading from it, and a load balancer in front. This will help IF your application is CPU-bound, which yours is not (I assume by server load you mean CPU).
So there is no other possible limit in a zope instance than IO or CPU? If cpu was the limiting factor I would see the 2 python processes running 90% and dozens of httpd's taking up the rest? Can You think of any good parameters I can get at with a small script that would be good to graph with all the rest to shed som light on this? (we are using munin) Something out of Control_Panel/Database/main/manage_activity perhaps? Is there a way to get that data out without going through port 8080? How about something out of /proc/`cat /home/zope/sites/site1/var/Z2.pid`/XXX? Need to read up on procfs I guess.
ZEO can actually *hurt* if you're IO-bound, because it adds network overhead to ZODB reads and writes. It's very bad if you have large Image or File objects (which you probably shouldn't have in the ZODB anyway).
Good to hear. I ws not particularly relishing the thought of the nescesary load balancing on that single box either :-/
or would it be better to extend varnish coverage?
Probably a good idea anyway... but you want to find out what the problem really is.
What would you do to pinpoint the problem?
I'd first try hitting Zope directly during the problem to see if the slowdown is there. If so, I'd then try either:
Should be possible with lynx on localhost. Have done that before for other purpose, Should have thought of that. Maybe I wil start logging the responstime directly like that! Hm.. good idea :)
- DeadlockDebugger may be informative. http://www.zope.org/Members/nuxeo/Products/DeadlockDebugger
Sounds a little drastic on a production server, but it may stil come to that.. Ought to test it out on another server I guess.
- Enable Zope's trace log and use requestprofiler.py to see if there is a pattern to the requests that trigger the problem. Eg. maybe all your zope worker threads are waiting on some slow IO task. See the logger section of zope.conf.
That looks interesting, except that it can take 15 minutes or more to restart zope when load is at the worst. I could try it outside of peak ours I guess. Thanks for the innput. Really helped get me unstuck, as you can see :) Regards Gaute
On Thu, May 10, 2007 at 02:10:40AM +0200, Gaute Amundsen wrote:
On Wednesday 09 May 2007 16:41, Paul Winkler wrote:
On Wed, May 09, 2007 at 12:07:54PM +0200, Gaute Amundsen wrote: So there is no other possible limit in a zope instance than IO or CPU?
Well, there's RAM of course. If you run out and start swapping a lot, that's a big problem :) btw, if you haven't come across this yet, I recommend reading Chris McDonough's "Scaling Zope" presentation. It's a couple years old but still a very good place to start: http://www.plope.org/misc/szweb/img0.html
Something out of Control_Panel/Database/main/manage_activity perhaps?
Very helpful if the trouble is ZODB I/O. Have a look at the activity graph and see if you're getting lots of loads... might mean your ZODB cache is too small and is getting thrashed. Also, I don't think you've mentioned what sort of app this is. Is it mostly reads or are there lots of writes? "Mostly reads" is a lot easier to optimize :)
Is there a way to get that data out without going through port 8080?
Don't think so. But I've never looked.
Maybe I wil start logging the responstime directly like that! Hm.. good idea :)
If you need that, the trace log can also tell you exactly what the response time per request is.
- DeadlockDebugger may be informative. http://www.zope.org/Members/nuxeo/Products/DeadlockDebugger
Sounds a little drastic on a production server, but it may stil come to that.. Ought to test it out on another server I guess.
Definitely test-drive it on a scratch server, but production is where you really need it :-) re. trace log:
That looks interesting, except that it can take 15 minutes or more to restart zope when load is at the worst. I could try it outside of peak ours I guess.
Yeah, restart during off hours and leave it enabled until you have one episode of peak load slowdown. -- Paul Winkler http://www.slinkp.com
On Thursday 10 May 2007 17:57, Paul Winkler wrote:
On Thu, May 10, 2007 at 02:10:40AM +0200, Gaute Amundsen wrote:
On Wednesday 09 May 2007 16:41, Paul Winkler wrote:
On Wed, May 09, 2007 at 12:07:54PM +0200, Gaute Amundsen wrote:
So there is no other possible limit in a zope instance than IO or CPU?
Well, there's RAM of course. If you run out and start swapping a lot, that's a big problem :)
About 12G in that box, so that's unlikely.
btw, if you haven't come across this yet, I recommend reading Chris McDonough's "Scaling Zope" presentation. It's a couple years old but still a very good place to start: http://www.plope.org/misc/szweb/img0.html
Looks good. Thanks :-)
Something out of Control_Panel/Database/main/manage_activity perhaps?
Very helpful if the trouble is ZODB I/O. Have a look at the activity graph and see if you're getting lots of loads... might mean your ZODB cache is too small and is getting thrashed.
I set cache-size to 10000 last night, up from the default. I felt I had to try that before I had a grap in place, so I don't have good numbers to estimate tha change, but with about 20 hits a sec on apache, there was close to 500000 loads the last hour, and just 3000 writes. Does that look reasonable? The built in help seems to indicate I should increase the cache until reads approaches zero..?
Also, I don't think you've mentioned what sort of app this is. Is it mostly reads or are there lots of writes? "Mostly reads" is a lot easier to optimize :)
Big CMS system with about 70 virtual domains.
Is there a way to get that data out without going through port 8080?
Don't think so. But I've never looked.
Hm. I guess if I at least use a url on the management port, it ought to stay available when everything else bogs down.
Maybe I wil start logging the responstime directly like that! Hm.. good idea :)
If you need that, the trace log can also tell you exactly what the response time per request is.
Good for debugging, but for permanent use, I wil use this: http://muninexchange.projects.linpro.no/?search&cid=10&pid=61&phid=81
- DeadlockDebugger may be informative. http://www.zope.org/Members/nuxeo/Products/DeadlockDebugger
Sounds a little drastic on a production server, but it may stil come to that.. Ought to test it out on another server I guess.
Definitely test-drive it on a scratch server, but production is where you really need it :-)
re. trace log:
That looks interesting, except that it can take 15 minutes or more to restart zope when load is at the worst. I could try it outside of peak ours I guess.
Yeah, restart during off hours and leave it enabled until you have one episode of peak load slowdown.
Gaute.
Gaute Amundsen wrote:
On Thursday 10 May 2007 17:57, Paul Winkler wrote:
On Thu, May 10, 2007 at 02:10:40AM +0200, Gaute Amundsen wrote:
On Wednesday 09 May 2007 16:41, Paul Winkler wrote:
On Wed, May 09, 2007 at 12:07:54PM +0200, Gaute Amundsen wrote: So there is no other possible limit in a zope instance than IO or CPU? Well, there's RAM of course. If you run out and start swapping a lot, that's a big problem :)
About 12G in that box, so that's unlikely.
How big's your Data.fs? If it's bigger than 12GB, then careless code can easilly chew through all your ram...
Very helpful if the trouble is ZODB I/O. Have a look at the activity graph and see if you're getting lots of loads... might mean your ZODB cache is too small and is getting thrashed.
I set cache-size to 10000 last night, up from the default.
If you're using a ZCatalog, you might consider splitting it into it's own zodb in order to improve the performance of this cache...
Big CMS system with about 70 virtual domains.
How often does the content change? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Tuesday 15 May 2007 15:20, Chris Withers wrote:
Gaute Amundsen wrote:
On Thursday 10 May 2007 17:57, Paul Winkler wrote:
On Thu, May 10, 2007 at 02:10:40AM +0200, Gaute Amundsen wrote:
On Wednesday 09 May 2007 16:41, Paul Winkler wrote:
On Wed, May 09, 2007 at 12:07:54PM +0200, Gaute Amundsen wrote:
So there is no other possible limit in a zope instance than IO or CPU?
Well, there's RAM of course. If you run out and start swapping a lot, that's a big problem :)
About 12G in that box, so that's unlikely.
How big's your Data.fs? If it's bigger than 12GB, then careless code can easilly chew through all your ram...
As it happens its' very close to 12 G!
Very helpful if the trouble is ZODB I/O. Have a look at the activity graph and see if you're getting lots of loads... might mean your ZODB cache is too small and is getting thrashed.
I set cache-size to 10000 last night, up from the default.
If you're using a ZCatalog, you might consider splitting it into it's own zodb in order to improve the performance of this cache...
Using it in small bits all over the place. That would make it rather hard to mount separately, would it not?
Big CMS system with about 70 virtual domains.
Make that ~180 hosts in apache and ~85 in zope.
How often does the content change?
Not often at all. I've got a munin plugin for it now, and that indicates the average is less than 2500 stores over 5 min. Rather hard to see actually, against the reads which fluctuates between 30 K and 100 K. (will plublish that plugin, once I add some comments and such.)
cheers,
Chris
On Tue, May 15, 2007 at 12:00:52PM +0200, Gaute Amundsen wrote:
I set cache-size to 10000 last night, up from the default. I felt I had to try that before I had a grap in place, so I don't have good numbers to estimate tha change, but with about 20 hits a sec on apache, there was close to 500000 loads the last hour, and just 3000 writes. Does that look reasonable?
Not really. Too many loads; you're getting a lot of cache misses and blowing out the cache a lot. Keep on doubling the cache size until your whole working set is in RAM all the time.
The built in help seems to indicate I should increase the cache until reads approaches zero..?
Yeah, for some definition of "approaches". I think a more realistic minimum would be number of threads * number of ZODB writes (since each write potentially invalidates one cached object per thread). It's hard to quantify the point at which you're caching enough, but you're not even close.
Also, I don't think you've mentioned what sort of app this is. Is it mostly reads or are there lots of writes? "Mostly reads" is a lot easier to optimize :)
Big CMS system with about 70 virtual domains.
Based on your activity graph, you do indeed have a lot more reads than writes. What's the CMS based on? Plone? Silva? Chris W. had a good point about the catalog. Do you have blobby data in the ZODB? (large images or files)? Those tend to play havoc with zodb cache activity, since one OFS.Image is stored as an arbitrarily long chain of small persistent objects. So whereas a Plone Document or a Page Template needs only one entry in the cache, an Image might need hundreds. -- Paul Winkler http://www.slinkp.com
On Wednesday 16 May 2007 00:13, Paul Winkler wrote:
On Tue, May 15, 2007 at 12:00:52PM +0200, Gaute Amundsen wrote:
I set cache-size to 10000 last night, up from the default. I felt I had to try that before I had a grap in place, so I don't have good numbers to estimate tha change, but with about 20 hits a sec on apache, there was close to 500000 loads the last hour, and just 3000 writes. Does that look reasonable?
Not really. Too many loads; you're getting a lot of cache misses and blowing out the cache a lot. Keep on doubling the cache size until your whole working set is in RAM all the time.
Will do!
The built in help seems to indicate I should increase the cache until reads approaches zero..?
Yeah, for some definition of "approaches". I think a more realistic minimum would be number of threads * number of ZODB writes (since each write potentially invalidates one cached object per thread).
That would give me 2500*6 = 15000. Looks like a reasonable goal.
It's hard to quantify the point at which you're caching enough, but you're not even close.
I can see from the config that we tried cache-size 20000 once, but probably set it back since something else was probably the problem that time :-(
Also, I don't think you've mentioned what sort of app this is. Is it mostly reads or are there lots of writes? "Mostly reads" is a lot easier to optimize :)
Big CMS system with about 70 virtual domains.
I keep mixing up the numbers here I notice, but make that ~180 hosts in apache and ~85 in zope.
Based on your activity graph, you do indeed have a lot more reads than writes.
What's the CMS based on? Plone? Silva?
Nothing of the sort. I'ts a "innhouse" thing... Could be lots of gremlins burried there I know :-/
Chris W. had a good point about the catalog.
Do you have blobby data in the ZODB? (large images or files)? Those tend to play havoc with zodb cache activity, since one OFS.Image is stored as an arbitrarily long chain of small persistent objects. So whereas a Plone Document or a Page Template needs only one entry in the cache, an Image might need hundreds.
Not _that_ much. There is some from olden times, but we moved all our image stuff over into something based on phpgallery 2 years ago. thanks :) Gaute
On Thu, May 10, 2007 at 02:10:40AM +0200, Gaute Amundsen wrote:
So there is no other possible limit in a zope instance than IO or CPU? If cpu was the limiting factor I would see the 2 python processes running 90% and dozens of httpd's taking up the rest?
I'm running a zope site with up 70k requests per hour (40k on avarage). I've noticed that the main reason vor performance problem is not enough memory. We are running 13 Frontend Zope servers and a ZEO Backend. All our frontendservers have 8gig ram. Zope gets major performance Problems when it reaches the limit of physical memory. Check your system if the. So having 2 zope Processes on the same system increases the Problem. Zope 2.7 doesn't scale very well with the ZEO. The more Frontend server you get the more Read Conflicts you have. Migration to Zope2.8 reduced this problem. I have build a squid proxy in reverse mode take the Request and spead them per round robin to the ZopeFrontends. This takes quite a lot load from the systems as the squid caches most of the static content (images, PDF files etc). Regard Estartu -- ---------------------------------------------------------------------------- Gerhard Schmidt | Nick : estartu IRC : Estartu | Fischbachweg 3 | | PGP Public Key 86856 Hiltenfingen | EMail: estartu@augusta.de | on request Germany | |
On Wednesday 16 May 2007 07:52, Gerhard Schmidt wrote:
On Thu, May 10, 2007 at 02:10:40AM +0200, Gaute Amundsen wrote:
So there is no other possible limit in a zope instance than IO or CPU? If cpu was the limiting factor I would see the 2 python processes running 90% and dozens of httpd's taking up the rest?
I'm running a zope site with up 70k requests per hour (40k on avarage). I've noticed that the main reason for performance problem is not enough memory.
By a rough estimate we run about 36K /hour into apache, and perhaps half of that from zope. The munin process_memory plugin I installed yesterday indicates the we only consume about 600M of that, so I guess it looks more and more like this might be hour bottleneck.
We are running 13 Frontend Zope servers and a ZEO Backend.
Woha! Either I am wrong about our load, or you have a completely differnet type of site..
All our frontendservers have 8gig ram. Zope gets major performance Problems when it reaches the limit of physical memory. Check your system if the. So having 2 zope Processes on the same system increases the Problem.
We do actually, have two. From a time we just "hadd to do something"! Consolidation time...
Zope 2.7 doesn't scale very well with the ZEO. The more Frontend server you get the more Read Conflicts you have. Migration to Zope2.8 reduced this problem.
I have build a squid proxy in reverse mode take the Request and spead them per round robin to the ZopeFrontends. This takes quite a lot load from the systems as the squid caches most of the static content (images, PDF files etc).
Hm.. yes, that's the way we have been inclined.. just skip the whole ZEO thing, and go straight for Varnish integration. Lotts of old sinns that have to be paid off for that to work for everything however.. At some point Varnish url rewriting wil be good enough, and then we can cut apache out of it too.. thanks :) Gaute
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 16 May 2007, at 10:13, Gaute Amundsen wrote:
Zope 2.7 doesn't scale very well with the ZEO. The more Frontend server you get the more Read Conflicts you have. Migration to Zope2.8 reduced this problem.
I have build a squid proxy in reverse mode take the Request and spead them per round robin to the ZopeFrontends. This takes quite a lot load from the systems as the squid caches most of the static content (images, PDF files etc).
Hm.. yes, that's the way we have been inclined.. just skip the whole ZEO thing, and go straight for Varnish integration. Lotts of old sinns that have to be paid off for that to work for everything however..
I don't think he advocates removing ZEO. That would be silly. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD4DBQFGSyd/RAx5nvEhZLIRAsDDAJiz279yOqezgUZ31H4WF22dSWthAKCSGllN FzA0ehOb1MlMWoIVOQxQ0g== =POCf -----END PGP SIGNATURE-----
On Wed, May 16, 2007 at 10:47:11AM -0500, Jens Vagelpohl wrote:
On 16 May 2007, at 10:13, Gaute Amundsen wrote:
Hm.. yes, that's the way we have been inclined.. just skip the whole ZEO thing, and go straight for Varnish integration. Lotts of old sinns that have to be paid off for that to work for everything however..
I don't think he advocates removing ZEO. That would be silly.
Why? I don't think ZEO is a panacea that magically makes apps faster in all cases. If you're not CPU-bound, the added network overhead and increased chance of write conflicts might be a net loss. Based on everything he's said so far, it sounds like Gaute still has plenty of headroom on his CPU. As for the other benefits of ZEO: - multiple ZEO clients does allow you better responsiveness with high concurrency, but you can also try increasing the number of threads to get the same effect without the added network overhead. - not using ZEO means that Zope is a single point of failure. Can't argue with that :) -- Paul Winkler http://www.slinkp.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 16 May 2007, at 11:04, Paul Winkler wrote:
On Wed, May 16, 2007 at 10:47:11AM -0500, Jens Vagelpohl wrote:
On 16 May 2007, at 10:13, Gaute Amundsen wrote:
Hm.. yes, that's the way we have been inclined.. just skip the whole ZEO thing, and go straight for Varnish integration. Lotts of old sinns that have to be paid off for that to work for everything however..
I don't think he advocates removing ZEO. That would be silly.
Why? I don't think ZEO is a panacea that magically makes apps faster in all cases. If you're not CPU-bound, the added network overhead and increased chance of write conflicts might be a net loss. Based on everything he's said so far, it sounds like Gaute still has plenty of headroom on his CPU.
There's a difference between scaling and making something faster. ZEO makes a single instance slower, right. But you can deal with more requests concurrently using ZEO. That's what I consider "scaling". jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFGS3aoRAx5nvEhZLIRAn6rAJ9eL6UpxP0KsFgBsnntiltxejKLLQCfTZ5D xYXbrs8cb0rzyndHeKGiSM8= =MFcw -----END PGP SIGNATURE-----
On Wed, May 16, 2007 at 04:24:56PM -0500, Jens Vagelpohl wrote:
There's a difference between scaling and making something faster. ZEO makes a single instance slower, right. But you can deal with more requests concurrently using ZEO. That's what I consider "scaling".
Agreed. But at the same time, I don't think it makes sense to keep deploying more and more badly tuned instances. That's what I consider "blind shotgun scaling" :) You need to scale, but you also need to tune - and you need to be pragmatic about which is the appropriate approach at any given point in time. -- Paul Winkler http://www.slinkp.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 16 May 2007, at 16:52, Paul Winkler wrote:
On Wed, May 16, 2007 at 04:24:56PM -0500, Jens Vagelpohl wrote:
There's a difference between scaling and making something faster. ZEO makes a single instance slower, right. But you can deal with more requests concurrently using ZEO. That's what I consider "scaling".
Agreed. But at the same time, I don't think it makes sense to keep deploying more and more badly tuned instances. That's what I consider "blind shotgun scaling" :) You need to scale, but you also need to tune - and you need to be pragmatic about which is the appropriate approach at any given point in time.
My perspective is always high-traffic systems. I never even consider small sites with few hits, for which ZEO won't do much. I have a blind spot for those ;) jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) iD8DBQFGS46TRAx5nvEhZLIRAtImAJkBSPFTiwPO4k3LeydaI0udXtv2xwCfcyHQ 0qnQuviEBZsbMD9Uin+/d+8= =kNMS -----END PGP SIGNATURE-----
On Wed, May 16, 2007 at 05:52:16PM -0400, Paul Winkler wrote:
On Wed, May 16, 2007 at 04:24:56PM -0500, Jens Vagelpohl wrote:
There's a difference between scaling and making something faster. ZEO makes a single instance slower, right. But you can deal with more requests concurrently using ZEO. That's what I consider "scaling".
Agreed. But at the same time, I don't think it makes sense to keep deploying more and more badly tuned instances. That's what I consider "blind shotgun scaling" :) You need to scale, but you also need to tune - and you need to be pragmatic about which is the appropriate approach at any given point in time.
I would love to tune our system, but not using ZEO isn't an option. I don't see how a single Server ever will be able to do what 13 Server do right now. Our system is a single Instance. With up to 70k requests per hour (20 requests per second). I don't see a way to get rid of the ZEO server. We have expirienced some problem with ZEO when requests take to long. Sometimes we have Problems like this 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.29:52569) Transaction blocked waiting for storage. Clients waiting: 1. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.36:56041) Transaction blocked waiting for storage. Clients waiting: 2. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.33:63884) Transaction blocked waiting for storage. Clients waiting: 3. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.34:64355) Transaction blocked waiting for storage. Clients waiting: 4. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.25:63215) Transaction blocked waiting for storage. Clients waiting: 5. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.28:58213) Transaction blocked waiting for storage. Clients waiting: 6. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.30:59149) Transaction blocked waiting for storage. Clients waiting: 7. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.31:58930) Transaction blocked waiting for storage. Clients waiting: 8. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.35:64097) Transaction blocked waiting for storage. Clients waiting: 9. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.27:63627) Transaction blocked waiting for storage. Clients waiting: 10. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.26:63146) Blocked transaction restarted. Clients waiting: 9 ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.29:52569) Blocked transaction restarted. Clients waiting: 8 ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.36:56041) Blocked transaction restarted. Clients waiting: 7 ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.33:63884) Blocked transaction restarted. Clients waiting: 6 ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.34:64355) Blocked transaction restarted. Clients waiting: 5 ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.25:63215) Blocked transaction restarted. Clients waiting: 4 ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.28:58213) Blocked transaction restarted. Clients waiting: 3 ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.30:59149) Blocked transaction restarted. Clients waiting: 2 ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.31:58930) Blocked transaction restarted. Clients waiting: 1 ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.35:64097) Blocked transaction restarted. This incident wasn't a Problem because it was resolved within on second. But sometimes situations like this take up tum 30 seconds to resolve. The site is completly unresponsiv in this time and take up to 10 minutes to resume normal opration (Responsetimes < 1 sec per dynamic page) I haven't been able to track down the Problem that causes this. But the frequency has droped quite dramatic since we updated our fontend Servers to more recent CPUs. It seams there is a posibility for an deadlock when requests take to much time to process. But the main Problem what we have is the memory growth of the Zope server processes. They grow to 500 MB of Memory bevor serving the first request. an wile running the constantly growing until the hit the limit of the physical memory. When they do, they slow down very dramaticaly (responstimes 800% higher than usual) we have done some debugging and it seams that die python garbage collection kicks in and kills the whole performance. The only solution we have com up with is to restart the zope Server before the hit the physical memory limit. Bye Estartu -- ---------------------------------------------------------------------------- Gerhard Schmidt | Nick : estartu IRC : Estartu | Fischbachweg 3 | | PGP Public Key 86856 Hiltenfingen | EMail: estartu@augusta.de | on request Germany | |
Gerhard Schmidt wrote at 2007-5-21 08:42 +0200:
... We have expirienced some problem with ZEO when requests take to long.
Sometimes we have Problems like this 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.29:52569) Transaction blocked waiting for storage. Clients waiting: 1. ------ 2007-05-21T03:36:22 INFO ZEO.StorageServer (56016/10.152.64.36:56041) Transaction blocked waiting for storage. Clients waiting: 2. ...
What you see here are informations over commit lock contention: Between the "vote" (start of) and the "finish" (end of) (second and third part of the commit protocol) a "FileStorage" must hold a lock to prevent modifications of the voted state. When another connection tries to commit during this time, ZEO will report a "transaction blocked ...". I had to enhance ZEO to report which transactions are blocking in order to quickly resolve problems.
... This incident wasn't a Problem because it was resolved within on second. But sometimes situations like this take up tum 30 seconds to resolve. The site is completly unresponsiv in this time and take up to 10 minutes to resume normal opration (Responsetimes < 1 sec per dynamic page)
That's strange. I have never seen this (though commit lock contention is not unusual in a site busy writing (as ours)). It should not have any lasting effects....
.... It seams there is a posibility for an deadlock when requests take to much time to process.
Never observed something like this -- but a colleague already had committed a monster transaction -- which committed for ages.... That prompted me to add the more detailed information to the "Transaction blocked" log entry.
But the main Problem what we have is the memory growth of the Zope server processes. They grow to 500 MB of Memory bevor serving the first request.
That's strange, too. You may take a look at my "analyseObjects" <http://www.dieter.handshake.de/pyprojects/zope#analyseObjects> It was developped to help in the determination of memory leaks but it can be useful to analyse unreasonably high memory loads after startup as well. It has several drawbacks, however: * It knows only about the garbage collector registered objects. A Python debug build is necessary to learn about all Python objects * It does not know how much size the objects use. An integration with "PySizer" may improve on this.
an wile running the constantly growing until the hit the limit of the physical memory. When they do, they slow down very dramaticaly (responstimes 800% higher than usual) we have done some debugging and it seams that die python garbage collection kicks in and kills the whole performance.
The garbage collector by itself does not know about the physical memory limit -- but when it kicks in and large parts of your objects have been swapped out, its traversal (to determine unreachable objects) will load them and this can drastically slow down the process. -- Dieter
Dieter Maurer wrote:
I had to enhance ZEO to report which transactions are blocking in order to quickly resolve problems.
It would be nice if you could feed those enhancements back into the source tree... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Paul Winkler wrote at 2007-5-16 12:04 -0400:
... - multiple ZEO clients does allow you better responsiveness with high concurrency, but you can also try increasing the number of threads to get the same effect without the added network overhead.
If you have several CPUs (and a CPU bound activity) then an increased number of threads is inferior to multiple processes as the GIL prevents more than a single thread running Python code. -- Dieter
On Wed, May 16, 2007 at 05:13:27PM +0200, Gaute Amundsen wrote:
On Wednesday 16 May 2007 07:52, Gerhard Schmidt wrote:
All our frontendservers have 8gig ram. Zope gets major performance Problems when it reaches the limit of physical memory. Check your system if the. So having 2 zope Processes on the same system increases the Problem.
We do actually, have two. From a time we just "hadd to do something"! Consolidation time...
Well, maybe. Is it an SMP box? You can happily run 1 Zope process per CPU. I'm told that recent versions of Linux should be pretty good at keeping CPU affinity without any special configuration.
At some point Varnish url rewriting wil be good enough, and then we can cut apache out of it too..
I highly doubt Apache will become your bottleneck anytime soon :) But of course it's one more thing to admin. -- Paul Winkler http://www.slinkp.com
participants (6)
-
Chris Withers -
Dieter Maurer -
Gaute Amundsen -
Gerhard Schmidt -
Jens Vagelpohl -
Paul Winkler