[Zope] Scaling problems, or something else?

[Gaute Amundsen]

On Thursday 10 May 2007 17:57, Paul Winkler wrote:
> On Thu, May 10, 2007 at 02:10:40AM +0200, Gaute Amundsen wrote:
> > On Wednesday 09 May 2007 16:41, Paul Winkler wrote:
> > > On Wed, May 09, 2007 at 12:07:54PM +0200, Gaute Amundsen wrote:
> >
> > So there is no other possible limit in a zope instance than IO or CPU?
>
> Well, there's RAM of course.  If you run out and start swapping a lot,
> that's a big problem :)
>
About 12G in that box, so  that's unlikely.

> btw, if you haven't come across this yet, I recommend reading
> Chris McDonough's "Scaling Zope" presentation. It's a couple years old
> but still a very good place to start:
> http://www.plope.org/misc/szweb/img0.html
>
Looks good. Thanks :-)

> > Something out of Control_Panel/Database/main/manage_activity
> >  perhaps?
>
> Very helpful if the trouble is ZODB I/O.  Have a look at the activity
> graph and see if you're getting lots of loads... might mean your ZODB
> cache is too small and is getting thrashed.

I set cache-size to 10000 last night, up from the default.
I felt I had to try that before I had a grap in place, so I don't have good 
numbers to estimate tha change, but with about 20 hits a sec on apache, there 
was close to 500000 loads the last hour, and just 3000 writes. 
Does that look reasonable?

The built in help seems to indicate I should increase the cache until reads 
approaches zero..? 

> Also, I don't think you've mentioned what sort of app this is.
> Is it mostly reads or are there lots of writes?
> "Mostly reads" is a lot easier to optimize :)
>
Big CMS system with about 70 virtual domains.

> > Is there a way to get that data out without going through port 8080?
>
> Don't think so. But I've never looked.
>
Hm. I guess if I at least use a url on the management port, it ought to stay 
available when everything else bogs down.

> > Maybe I wil start logging the responstime directly like that!
> > Hm.. good idea :)
>
> If you need that, the trace log can also tell you exactly what the
> response time per request is.
>
Good for debugging, but for permanent use, I wil use this:
http://muninexchange.projects.linpro.no/?search&cid=10&pid=61&phid=81

> > > - DeadlockDebugger may be informative.
> > >   http://www.zope.org/Members/nuxeo/Products/DeadlockDebugger
> >
> > Sounds a little drastic on a production server, but it may stil come
> > to that..  Ought to test it out on another server I guess.
>
> Definitely test-drive it on a scratch server, but production is where
> you really need it :-)
>
> re. trace log:
> > That looks interesting, except that it can take 15 minutes or more
> > to restart zope when load is at the worst. I could try it outside of
> > peak ours I guess.
>
> Yeah, restart during off hours and leave it enabled until you have one
> episode of peak load slowdown.

Gaute.