RE: [Zope] zope2.2.0 and what is high load
i went to each Zeo client control_panel, and see that there's a mzximum of four connections open at any given time.
OK, I'm not sure this is real meaningful. A lot can happen in between refreshes. This is only a rough sort of "realtime" guess of what's going on.
why wouldn't increasing the number of threads with -t25 affect the way zope behave? do i need to change it in z2.py?
No... -t[whatever] does the same thing as NUMBER_OF_THREADS in z2.py.
There is also a profiling facility inside Control_Panel/debug that does some sort of automated performance compilation for you, though I've never had a reason to use it. When it's running, it also purportedly slows Zope down quite a bit.
profiling is not enabled since i dont really know the heck it does.
I don't know what it does specifically, either. I've never needed to use it. Maybe somebody else can comment.
I suggest that you:
1) Monitor your Zope clients' thread usages by using their debug facilities. done. as mentioned above, the number of connections is four at most. what else should i look for in the debugInfo?
That's about it for your problem. Maybe the profiling stuff can give more info, I'm not sure.
most of the z2 threads are used, although only 3-4 are used heavily. what does this show?
It shows that four threads have database connections. I'm curious as to why you're only seeing four threads being served by database connections too. You may want to ramp your number of threads down to 5 (-t5). 25 doesn't do much for you because only 7 threads can be using database connections at any given time. The high number of threads you're servicing might actually be slowing things down as they contend for available database connections. Why are you frontending this with apache? how are you servicing two ZEO clients via a single apache? Does it do some sort of round-robining? Or does each box have its own apache? Is there any reason you're not just using straight ZServer? I'm wondering if apache processes contending to talk to available Zope python threads might be slowing stuff down? It doesn't seem as if bumping your number of database connections will do us any good here, because it doesn't seem you're using all 7. I'm not sure *why* you're not using all 7, because you've got at least 15 processes waiting around just on your top screen for DB connections. Maybe manage_debug isn't telling you the whole story, or maybe you caught it at a low-load period. I wonder if the profiling stuff would help here. Maybe someone else can jump in here, this is all I can think of at the moment.
On Fri, 11 Aug 2000, Chris McDonough wrote:
i went to each Zeo client control_panel, and see that there's a mzximum of four connections open at any given time.
comments to the above. i saw only four threads because i started that particular box with the default. on the box with -t25, i see 7 open connections at any given time. sorry, my mistake.
That's about it for your problem. Maybe the profiling stuff can give more info, I'm not sure.
i'll try and read more about that profiling. but not knowing what it does really scares me :)
It shows that four threads have database connections. I'm curious as to why you're only seeing four threads being served by database connections too. as mentioned above, my bad. right after i sent the prev mail, i went to the other box with -t25, and it does have 7 open connections.
You may want to ramp your number of threads down to 5 (-t5). 25 doesn't do much for you because only 7 threads can be using database connections at any given time. The high number of threads you're servicing might actually be slowing things down as they contend for available database connections.
will do. and hope to see that it does something.
Why are you frontending this with apache? how are you servicing two ZEO clients via a single apache? Does it do some sort of round-robining? Or does each box have its own apache? each box has its own apache. i'm round robining (straight dns round robin). ZEO is also a concern here because i'm using it blindly, unfortunately. but it works great if i got less than 150 simultaneous connections. and this symptom i saw prevuiously, even without ZEO.
Is there any reason you're not
just using straight ZServer? I'm wondering if apache processes contending to talk to available Zope python threads might be slowing stuff down?
we need to serve some othe things from apache, that can't be done thru ZServer alone (a little php stuff)
It doesn't seem as if bumping your number of database connections will do us any good here, because it doesn't seem you're using all 7. I'm not sure *why* you're not using all 7, because you've got at least 15 processes waiting around just on your top screen for DB connections.
i did some grepping and saw the magical number 7 in ZODB/DB.py. is that the number to turn up if i wanted to try increasing the db pool connection? i do have postgresql with ZPygreSQL. will that have anything to do with what i'm facing?
Maybe manage_debug isn't telling you the whole story, or maybe you caught it at a low-load period. I wonder if the profiling stuff would help here.
Maybe someone else can jump in here, this is all I can think of at the moment.
thanks for sparing some time helping me out. i love zope, and open source in general, and i can't bear having to switch to using something i don't really like.
Hello! I need to find more info on how Zope handles threading with relation to database connections. I am currently considering and evaluating Zope as one of the options we have to build a really large, completely databasedriven "enterprise scale" web-platform. I am a bit worried about this "maximum of 7 threads per db connection" limitation mentioned previously, and I need to know a) what does this mean in terms of accesibility, b) how does it affect performace, and c) how does Zope work (in detail) with external database connections. We are going to be using Oracle as our DB backend, all our served data will be database *only* (some of it will be fairly large), and efficient database connections is *crucial* to what we intend to do. We are going to have between 10-20k users at most and I expect the maximun of simultaneous connections to be beween 400 - 900 under heavy load; "normal load" will probably oscillate around 25-200 simultaneous connections (these numbers are crude guesses, but take into account increased usage over time as more services are provided in the future). Bandwith problems are not an issue, nor is computing power. I would appreciate any pointers to documentation or people to ask. Sincerely, /dario - -------------------------------------------------------------------- Dario Lopez-Kästen Systems Developer Chalmers Univ. of Technology dario@ita.chalmers.se ICQ will yield no hits IT Systems & Services
"Dario Lopez-Kästen" wrote:
Hello!
Hello Dario!
I need to find more info on how Zope handles threading with relation to database connections.
How about: http://www.zope.org/Documentation/Developer/Models/ZODB The jargon of threads and connections (and then threads again) can be pretty confusing in Zope. First, ZServer is the thing in front listening for socket requests. It has a connection pool (select based) which maps into Python threads in the Zope application. (ZServer is part of the Zope process, I'm discussing logical partitioning.) Each concept of a "thread" in Zope is actually a "database connection" to the ZODB. Threads get a private copy of the transaction space, thus greatly simplifying (to the point of removal) the need for programmers to think about thread safety. If you're talking about the thread concurrency a connection to Oracle, then that's a bit different. I _think_ that ultimately this maps to the number of Zope database connections. This conversation is probably better in the zope-dev list.
I am currently considering and evaluating Zope as one of the options we have to build a really large, completely databasedriven "enterprise scale" web-platform. I am a bit worried about this "maximum of 7 threads per db
This isn't a maximum. It's just the default. It's easy to increase the number of connections.
connection" limitation mentioned previously, and I need to know a) what does this mean in terms of accesibility, b) how does it affect performace, and c) how does Zope work (in detail) with external database connections.
We are going to be using Oracle as our DB backend, all our served data will be database *only* (some of it will be fairly large), and efficient database connections is *crucial* to what we intend to do.
Since you're not using any persistent objects in Zope (unless you do caching of the SQL data, which you probably should), then you don't have to worry about process growth from having more database connections for Zope. Jack it up to 25. Since much of the time in a Zope hit is taken by rendering to HTML, a number like 25 probably means you can process a hundred simultaneous Oracle requests. That works out to be an average of around ten million hits per day. If you want more, increase the number.
We are going to have between 10-20k users at most and I expect the maximun of simultaneous connections to be beween 400 - 900 under heavy load; "normal load" will probably oscillate around 25-200 simultaneous connections (these numbers are crude guesses, but take into account increased usage over time as more services are provided in the future).
Bandwith problems are not an issue, nor is computing power.
I would appreciate any pointers to documentation or people to ask.
Is this an intranet or a public site? --Paul
participants (4)
-
Bak ＠ kedai -
Chris McDonough -
Dario Lopez-K�sten -
Paul Everitt