[Zope] Scaling problems, or something else?

[Gaute Amundsen]

On Wednesday 16 May 2007 07:52, Gerhard Schmidt wrote:
> On Thu, May 10, 2007 at 02:10:40AM +0200, Gaute Amundsen wrote:
> > So there is no other possible limit in a zope instance than IO or CPU?
> > If cpu was the limiting factor I would see the 2 python processes running
> > 90% and dozens of httpd's taking up the rest?
>
> I'm running a zope site with up 70k requests per hour (40k on avarage).
> I've noticed that the main reason for performance problem is not enough
> memory.

By a rough estimate we run about 36K /hour into apache, and perhaps half of 
that from zope.
The munin process_memory plugin I installed yesterday indicates the we only 
consume about 600M of that, so I guess it looks more and more like this might 
be hour bottleneck. 

> We are running 13 Frontend Zope servers and a ZEO Backend.
>
Woha! Either I am wrong about our load, or you have a completely differnet 
type of site..

> All our frontendservers have 8gig ram. Zope gets major performance Problems
> when it reaches the limit of physical memory. Check your system if the.
> So having 2 zope Processes on the same system increases the Problem.
>
We do actually, have two. From a time we just "hadd to do something"!
Consolidation time...


> Zope 2.7 doesn't scale very well with the ZEO. The more Frontend server
> you get the more Read Conflicts you have. Migration to Zope2.8 reduced this
> problem.
>
> I have build a squid proxy in reverse mode take the Request and spead them
> per round robin to the ZopeFrontends. This takes quite a lot load from the
> systems as the squid caches most of the static content (images, PDF files
> etc).
>
Hm.. yes, that's the way we have been inclined.. just skip the whole ZEO 
thing, and go straight for Varnish integration. Lotts of old sinns that have 
to be paid off for that to work for everything however..

At some point Varnish url rewriting wil be good enough, and then we can cut 
apache out of it too..

thanks :)

Gaute