[Zope] high loads, more threads, spinning wheels

[Stefan H. Holek]

I assume what you see are "hangs" i.e. Zope appearing to not accept new 
requests, as soon as all 4 worker threads are busy waiting for the 
RDBMS system to answer.

Now, using more threads will push back the point of unresponsiveness, 
but as soon as you have the 13th request hitting Zope, you are back at 
square one.

I see two ways of tackling the situation:

- Optimize the hell out of your RDBMS and your SQL queries. Make them 
return fast enough to handle the load.

- Write a front-end that does not rely on Zope but takes advantage of 
Apache's scalability features. This could be done in mod_python or 
mod_perl for example.

In that one project I am thinking about we ended up doing both. 
Sometimes you just have to accept that Zope is probably not the right 
tool for the job.

Stefan


On 2. Mai 2005, at 15:28, Jim Abramson wrote:

>>
>> Upping the number of threads is unlikely to give you better
>> performance. The only case where this could make sense is if
>> you had something like a highly saturated RDBMS backend,
>> tying up your worker threads.
>>
>
> Well this is *exactly* the issue I'm facing.  I need to soften
> the impact of this situation if possible, and it's going to be
> ZEO in the end, but rolling this out is probably not going to
> be possible in the very short term.
>
>> But - Zope threads don't operate like you probably expect
>> from knowing Apache and similar models. For one, they *never*
>> will run in parallel.
>> Python employs a global interpreter lock (GIL), so there will
>> only be a single thread "working" at all times. What you want
>> is to up the number of processes (interpreters) not the
>> number of threads. Hence ZEO.
>>
>
> So far I've done some very off-the-cuff tests upping the
> threads/connections to 12/16 and banging on it with jmeter.
> The results were favorable in comparison to the default config.
> Could this have been a fluke?
>
> Jim
>

--
Software Engineering is Programming when you can't. --E. W. Dijkstra