RE: [Zope] Options for handling concurrency?
From: Tony McDonald [mailto:tony.mcdonald@ncl.ac.uk]
At 9:43 am +0100 27/2/00, Martijn Pieters wrote:
- Have a look at /Control_Panel/manage_debug. It'll show you
reference
counts and currenlty open connections, which can be very helpful in targeting hotspots.
Aye carumba! - what a lot of information. I'm going to start delving around there and see what happens. Is there any additional documentation available on what the info. means? - I've looked in the ApplicationManager.py code is a wee bit over my head.
It shows you reference counts, and how they change over time. Reference counts are how many references there are to classes and methods. This generally lets you see how many instances there are of those classes. It allows you to see where the hotspots might be.
As an aside (to the list), looking around the ApplicationManager.py reveals something called manage_profile. It looks like the main Zope functions are being profiled here (ie not at the object level). It's interesting, but it does seem to be a resource hog...
But so is something in your app. You might want to run it and see what analysis you get. For more documentation of the data the profiler returns, see: http://www.python.org/doc/lib/profile.html
- Also, what DBMS are you using? If you are using a level 1 or 2 DA (see http://www.zope.org/Members/petrilli/DARoadmap) then you can do only one DB query at a time. It might be time to choose a different DB, or if you are using the old MySQL DA, to upgrade.
It's MySQL, but using the latest DA - I'm pretty sure I'm doing more than one query at a time as I set off a ZSQL query that takes 9 seconds from the MySQL command line, and then started to use aspects of the site that I *know* use MySQL lookups. BTW, the 9 second SQL query isn't used in the site! :)
This only shows that MySQL is multithreaded, not that the DA is. Run the expensive, 9 second query in a ZSQL Method. Then try and access another ZSQL Method on the same Zope server. If it is multithreaded, you should get results about as fast as when the heavy one isn't running. If it is the old MySQL DA, you'll have to wait until the 9 sec query is finished.
- If you are using ZSQL a lot, do you use caching? Even a cache expiry of 1 second would help a heavily hit site tremendously, noone will notice the data returned being at most 1 second out of date.But all hits with equal parameters on the same thread in Zope within that second need only one DB query.
I do use caching Martijn, but I must admit that it's been a hit and miss affair. I'm trying to do as many of my SQL queries in external methods (ironically for the performance), so I may well be losing a lot of the caching performance breakthrough there .... hmmm. I wonder if I could do something like...
This shouldn't make any difference. Only in doing excessive loops in loops and the like, would an External Method be any use. -- Martijn Pieters, Software Engineer | Digital Creations http://www.digicool.com | Creators of Zope http://www.zope.org | mailto:mj@digicool.com ICQ: 4532236 | PGP: http://wwwkeys.nl.pgp.net:11371/pks/lookup?op=get&search=0xA8A32149 -------------------------------------------
At 8:19 am -0500 29/2/00, Martijn Pieters wrote:
It shows you reference counts, and how they change over time. Reference counts are how many references there are to classes and methods. This generally lets you see how many instances there are of those classes. It allows you to see where the hotspots might be.
Thanks for the insights Martijn, I'll use the manage_debug with care...
As an aside (to the list), looking around the ApplicationManager.py reveals something called manage_profile. It looks like the main Zope functions are being profiled here (ie not at the object level). It's interesting, but it does seem to be a resource hog...
But so is something in your app.
I know :(
You might want to run it and see what analysis you get. For more documentation of the data the profiler returns, see:
Again, thanks for the pointer - I'll look into that.
This only shows that MySQL is multithreaded, not that the DA is. Run the expensive, 9 second query in a ZSQL Method. Then try and access another ZSQL Method on the same Zope server. If it is multithreaded, you should get results about as fast as when the heavy one isn't running. If it is the old MySQL DA, you'll have to wait until the 9 sec query is finished.
Yup. I've checked it out. That's exactly what's happening, ie the 9 second query doesn't seem to be slowing the other queries down at all.
I do use caching Martijn, but I must admit that it's been a hit and miss affair. I'm trying to do as many of my SQL queries in external methods (ironically for the performance), so I may well be losing a lot of the caching performance breakthrough there .... hmmm. I wonder if I could do something like...
This shouldn't make any difference. Only in doing excessive loops in loops and the like, would an External Method be any use.
I do have to do nested loops, and as I've written in another email, I've got most of it working in an External Method now. The performance increase is pretty impressive (9-10 times faster than a ZODB solution), but I know I'm missing the ZODB magic... Many thanks for the thoughts Martijn... cheers tone. ------ Dr Tony McDonald, FMCC, Networked Learning Environments Project http://nle.ncl.ac.uk/ The Medical School, Newcastle University Tel: +44 191 222 5888 Fingerprint: 3450 876D FA41 B926 D3DD F8C3 F2D0 C3B9 8B38 18A2
participants (2)
-
Martijn Pieters -
Tony McDonald