Re: [Zope] Major problems with slow catalog querie
I have also found that the catalog becomes very sluggish after a 'find and catalog' to catalog the 10,500 items. It takes 10 mins for the catalog to return with the 'catalog' tab listing the cataloged items. It takes a further 10 mins before the python process drops its usuage of the processor from 90-95% down to 0-10%. From this point on, the catalog takes 40-45 seconds to return found items using the site search form. Restarting Zope brings the catalog back to the 1.5-5 seconds I had achieved previously. Next step is to learn abit about the zope profiler and interpreting its results I think? __________________________________________________ Do You Yahoo!? Everything you'll ever need on one web page from News and Sport to Email and Music Charts http://uk.my.yahoo.com
=?iso-8859-1?Q?Lea_Smith?= writes:
I have also found that the catalog becomes very sluggish after a 'find and catalog' to catalog the 10,500 items. It takes 10 mins for the catalog to return with the 'catalog' tab listing the cataloged items. It takes a further 10 mins before the python process drops its usuage of the processor from 90-95% down to 0-10%. From this point on, the catalog takes 40-45 seconds to return found items using the site search form. Restarting Zope brings the catalog back to the 1.5-5 seconds I had achieved previously. Seems that you hit the main memory limit on your machine.
Try to flush the ZODB cache (--> Control_Panel --> Database management --> Flush Cache) and see whether it brings down the time without a need to restart Zope. Dieter
On Thursday 19 Sep 2002 1:34 pm, Lea Smith wrote:
I have also found that the catalog becomes very sluggish after a 'find and catalog' to catalog the 10,500 items. It takes 10 mins for the catalog to return with the 'catalog' tab listing the cataloged items. It takes a further 10 mins before the python process drops its usuage of the processor from 90-95% down to 0-10%. From this point on, the catalog takes 40-45 seconds to return found items using the site search form. Restarting Zope brings the catalog back to the 1.5-5 seconds I had achieved previously.
Using Zope 2.5.x ? This sounds like a typical symptom caused by excessive memory pressure on the ZODB object cache. You are probably seeing zope memory usage increase too? 2.6.x has a redesigned cache that shouldnt exhibit these problem.
fre, 2002-09-20 kl. 10:39 skrev Toby Dickenson:
On Thursday 19 Sep 2002 1:34 pm, Lea Smith wrote:
I have also found that the catalog becomes very sluggish after a 'find and catalog' to catalog the 10,500 items. It takes 10 mins for the catalog to return with the 'catalog' tab listing the cataloged items. It takes a further 10 mins before the python process drops its usuage of the processor from 90-95% down to 0-10%. From this point on, the catalog takes 40-45 seconds to return found items using the site search form. Restarting Zope brings the catalog back to the 1.5-5 seconds I had achieved previously.
Using Zope 2.5.x ? This sounds like a typical symptom caused by excessive memory pressure on the ZODB object cache. You are probably seeing zope memory usage increase too?
2.6.x has a redesigned cache that shouldnt exhibit these problem.
Hmmm ... thanks to Toby for posting this. We are running a CMF site under Zope-2.5 and it's a slooooooooooow site :/ I just thought that it was the portal_catalog queries that caused this - which your posting indicated - and accepted that for the moment. I will try and switch the stuff to Zope-2.6 and see if it makes any improvements. Our use of memory on our Zope instance homes are BIG ! I hope to bring it down ... Greetings, Gitte Wange
I will try and switch the stuff to Zope-2.6 and see if it makes any improvements. Our use of memory on our Zope instance homes are BIG ! I hope to bring it down ...
out of interest... How many publisher threads do you have? What are the number on the control panel/database/cache parameters page?
fre, 2002-09-20 kl. 13:36 skrev Toby Dickenson:
I will try and switch the stuff to Zope-2.6 and see if it makes any improvements. Our use of memory on our Zope instance homes are BIG ! I hope to bring it down ...
out of interest...
How many publisher threads do you have?
Do you mean how many threads the instances are started with ?? Our most slowest instance is started with 6 threads. It has been started with 8 but was still slow.
What are the number on the control panel/database/cache parameters page?
Total number of objects in the database 44617 Total number of objects in all caches 2574 Target Size: 400 Target max time between accesses: 60 Was this the info you wanted ? Greetings, Gitte Wange P.S. We are desperate to make the site run faster - we just don't know how. Our server has 512 MB RAM in it so ..
first thing you really need to do is increase the "target size". right now you are telling zope to keep the number of cached objects at 400, even though you obviously have the machine power to handle much more than that. this leads to aggressive wiping of objects out of the cache, which have to be read in again, etc, slowing down everything. i would set that to maybe 5000 and then watch for a couple days how the "Total number of objects in all caches" number reacts. you want to be at a point where the two numbers are not too far apart, with the "target size" being higher so that zope will not be aggressive about wiping the cache. jens On Friday, Sep 20, 2002, at 07:44 US/Eastern, Gitte Wange wrote:
fre, 2002-09-20 kl. 13:36 skrev Toby Dickenson:
I will try and switch the stuff to Zope-2.6 and see if it makes any improvements. Our use of memory on our Zope instance homes are BIG ! I hope to bring it down ...
out of interest...
How many publisher threads do you have?
Do you mean how many threads the instances are started with ?? Our most slowest instance is started with 6 threads. It has been started with 8 but was still slow.
What are the number on the control panel/database/cache parameters page?
Total number of objects in the database 44617 Total number of objects in all caches 2574 Target Size: 400 Target max time between accesses: 60
Was this the info you wanted ?
Greetings, Gitte Wange
P.S. We are desperate to make the site run faster - we just don't know how. Our server has 512 MB RAM in it so ..
On Friday 20 Sep 2002 12:44 pm, Gitte Wange wrote:
Do you mean how many threads the instances are started with ?? Our most slowest instance is started with 6 threads. It has been started with 8 but was still slow.
What are the number on the control panel/database/cache parameters page?
Total number of objects in the database 44617
Target Size: 400
So your target is 400 objects per thread. Are your published methods CPU bound? If they are not (for example if you are querying an oracle server on another machine) then you are probably making use of all 6 threads, and we can expect to see 400*6=2400 objects total. If your methods are all CPU bound then several of those threads will be idle, so we would expect the quiescent total size would be smaller. (2.6 gives size statistics per-thread, which takes away this guesswork)
Total number of objects in all caches 2574
This number is a little higher than your target of 2400, indicating that you have moderate (but not high) memory pressure. This is costing you in two ways: 1. IO to move objects from disk into memory. You possibly are not seeing disk activity if there is ram free for disk caching. 2. ZODB uses more CPU time than it should when working out which objects should get removed from cache. The new object cache in 2.6 will probably make a difference, but not necessarily a dramatic one. The next question is about what type of objects these are..... If they are mostly ZCatalog-derived BTree objects then a typical in-memory size might be 500 bytes. In total you are using 1M for object caching. If they are mostly large file downloads then they are typically 64k, and you are using 150M. I am assuming you would want to use more than that, so you probably want to increase the 'target size' to increase memory used for caching. Dont put it too high or ZODB (in 2.5, at least) will spend too much CPU managing this cache.
Target max time between accesses: 60
Once you have increased the target size, check whether the 'total number of object' is about right. If it never gets high enough, try increasing this. If it gets too high, reduce it. (If it gets really high and appears to grow out of control, you need 2.6)
P.S. We are desperate to make the site run faster - we just don't know how. Our server has 512 MB RAM in it so ..
If you methods are CPU bound you could experiment with fewer threads, perhaps 2, 3, or 4. Increase the 'target size' proportionally for a fair test to keep total memory usage constant.
On Friday 20 September 2002 07:44 am, Gitte Wange wrote: [snip]
P.S. We are desperate to make the site run faster - we just don't know how. Our server has 512 MB RAM in it so ..
Tweaking the ZODB cache is all well and good, but I'm not sure it will make a dramatic improvement (but it might, you never know). Are you doing any sort of page caching in front of Zope, either using Squid or Apache/mod_proxy? This can make a huge difference in performance. And just setting up the caching servers are not enough, you also need to setup caching policies using HTTP Cache Managers. Even if you don't have a caching server, setting up RAM Caches for frequently called scripts and page templates containing canned queries (like latest news, lists of objects, etc) can be a big win. HTTP Cache Managers can be still be used on images, static files, and even semi-static pages to tell the browsers to cache them and refresh them only at controlled intervals. Here is some add't info: http://www.zope.org/Members/rbeer/caching http://www.zope.org/Members/htrd/howto/squid hth, -Casey
----- Original Message ----- From: "Casey Duncan" <casey@zope.com> To: "Gitte Wange" <gitte@mmmanager.org>; <zope@zope.org> Sent: Friday, September 20, 2002 4:38 PM Subject: Re: [Zope] Major problems with slow catalog querie
On Friday 20 September 2002 07:44 am, Gitte Wange wrote: [snip]
P.S. We are desperate to make the site run faster - we just don't know how. Our server has 512 MB RAM in it so ..
Tweaking the ZODB cache is all well and good, but I'm not sure it will make a dramatic improvement (but it might, you never know).
I find it very difficult to "hit" the right number for 'Target size' but I'm working at :-) Right now tha numbers are: Total number of objects in the database: 49006 Total number of objects in all caches: 23404 Target size: 5000 Target max time between accesses: 60 And I was supposed to make the caches objects count and the target size numbers alike ? (Not to far from each other?) Wow :-) I'm at a hard job here ...
Are you doing any sort of page caching in front of Zope, either using Squid or Apache/mod_proxy? This can make a huge difference in performance. And just setting up the caching servers are not enough, you also need to setup caching policies using HTTP Cache Managers.
Even if you don't have a caching server, setting up RAM Caches for frequently called scripts and page templates containing canned queries (like latest news, lists of objects, etc) can be a big win. HTTP Cache Managers can be still be used on images, static files, and even semi-static pages to tell the browsers to cache them and refresh them only at controlled intervals.
Here is some add't info:
http://www.zope.org/Members/rbeer/caching http://www.zope.org/Members/htrd/howto/squid
Thanks Casey I'll look at the links ... We are using Apache/mod_proxy in front of Zope in order to use Virtual Hosting ... Would it be better to use Squid instead ?? Greetings, Gitte Wange
participants (6)
-
Casey Duncan -
Dieter Maurer -
Gitte Wange -
Jens Vagelpohl -
Lea Smith -
Toby Dickenson