i went to each Zeo client control_panel, and see that there's a mzximum of four connections open at any given time.
OK, I'm not sure this is real meaningful. A lot can happen in between refreshes. This is only a rough sort of "realtime" guess of what's going on.
why wouldn't increasing the number of threads with -t25 affect the way zope behave? do i need to change it in z2.py?
No... -t[whatever] does the same thing as NUMBER_OF_THREADS in z2.py.
There is also a profiling facility inside Control_Panel/debug that does some sort of automated performance compilation for you, though I've never had a reason to use it. When it's running, it also purportedly slows Zope down quite a bit.
profiling is not enabled since i dont really know the heck it does.
I don't know what it does specifically, either. I've never needed to use it. Maybe somebody else can comment.
I suggest that you:
1) Monitor your Zope clients' thread usages by using their debug facilities. done. as mentioned above, the number of connections is four at most. what else should i look for in the debugInfo?
That's about it for your problem. Maybe the profiling stuff can give more info, I'm not sure.
most of the z2 threads are used, although only 3-4 are used heavily. what does this show?
It shows that four threads have database connections. I'm curious as to why you're only seeing four threads being served by database connections too. You may want to ramp your number of threads down to 5 (-t5). 25 doesn't do much for you because only 7 threads can be using database connections at any given time. The high number of threads you're servicing might actually be slowing things down as they contend for available database connections. Why are you frontending this with apache? how are you servicing two ZEO clients via a single apache? Does it do some sort of round-robining? Or does each box have its own apache? Is there any reason you're not just using straight ZServer? I'm wondering if apache processes contending to talk to available Zope python threads might be slowing stuff down? It doesn't seem as if bumping your number of database connections will do us any good here, because it doesn't seem you're using all 7. I'm not sure *why* you're not using all 7, because you've got at least 15 processes waiting around just on your top screen for DB connections. Maybe manage_debug isn't telling you the whole story, or maybe you caught it at a low-load period. I wonder if the profiling stuff would help here. Maybe someone else can jump in here, this is all I can think of at the moment.