[Zope] Major problems with slow catalog querie

[Toby Dickenson]

On Friday 20 Sep 2002 12:44 pm, Gitte Wange wrote:

> Do you mean how many threads the instances are started with ??
> Our most slowest instance is started with 6 threads. It has been starte=
d
> with 8 but was still slow.

> > What are the number on the control panel/database/cache parameters pa=
ge?
>
> Total number of objects in the database 44617

> Target Size: 400

So your target is 400 objects per thread.

Are your published methods CPU bound? If they are not (for example if you=
 are=20
querying an oracle server on another machine) then you are probably makin=
g=20
use of all 6 threads, and we can expect to see 400*6=3D2400 objects total=
=2E

If your methods are all CPU bound then several of those threads will be i=
dle,=20
so we would expect the quiescent total size would be smaller.

(2.6 gives size statistics per-thread, which takes away this guesswork)


> Total number of objects in all caches 2574

This number is a little higher than your target of 2400, indicating that =
you=20
have moderate (but not high) memory pressure.=20

This is costing you in two ways:
1. IO to move objects from disk into memory. You possibly are not seeing
   disk activity if there is ram free for disk caching.
2. ZODB uses more CPU time than it should when working out which objects
   should get removed from cache.

The new object cache in 2.6 will probably make a difference, but not=20
necessarily a dramatic one.

The next question is about what type of objects these are.....

If they are mostly ZCatalog-derived BTree objects then a typical in-memor=
y=20
size might be 500 bytes. In total you are using 1M for object caching.

If they are mostly large file downloads then they are typically 64k, and =
you=20
are using 150M.

I am assuming you would want to use more than that, so you probably want =
to=20
increase the 'target size' to increase memory used for caching. Dont put =
it=20
too high or ZODB (in 2.5, at least) will spend too much CPU managing this=
=20
cache.=20

> Target max time between accesses: 60

Once you have increased the target size, check whether the 'total number =
of=20
object' is about right. If it never gets high enough, try increasing this=
=2E If=20
it gets too high, reduce it. (If it gets really high and appears to grow =
out=20
of control, you need 2.6)

> P.S. We are desperate to make the site run faster - we just don't know
> how. Our server has 512 MB RAM in it so ..

If you methods are CPU bound you could experiment with fewer threads, per=
haps=20
2, 3, or 4. Increase the 'target size' proportionally for a fair test to =
keep=20
total memory usage constant.