[ZODB-Dev] Speeding up ZODB (was "redis cache for RelStorage")

[Pedro Ferreira]

Hello.

> Is your storage server CPU bound?  

load average: 1.47, 1.34, 1.20

mpstat:
"""
11:40:13 AM  CPU   %user   %nice    %sys %iowait    %irq   %soft %steal
  %idle    intr/s
11:40:13 AM  all    5.82    0.00    0.69    1.11    0.01    0.12    0.00
  92.25    548.52
"""

I guess it's not very high, for an 8 core machine.

> Starting with ZODB 3.10, ZEO
> storage servers are multi-threaded. They have a thread for each
> client.  We have a storage server that has run at 120% cpu on a 4-core
> box.  Also, if you use zc.FileStorage, packing is mostly done in a
> separate process.

I didn't know this package even existed. I will give it a try, but in
any case our problems don't seem to be related to packing.

> A major source of slow down *can* be disk access times. How's IO wait
> on your server?  

As you can see from the mpstat snapshot above, just around 1%. I have
checked iostat as well, and the number of transactions per second seems
to be very low considering the maximum allowed by the hardware.

> Look at how your application is using data. If you have requests that
> have to load a lot of data, maybe you can refactor your application to
> load fewer.

A quick check with nethogs shows values of network usage oscillating
between 100 and 200 KB/s. But I guess that if I were loading an
excessive amount of data, this value would be higher, no?

Thanks a lot for your answers.

Pedro

-- 
José Pedro Ferreira

Software Developer, Indico Project
http://indico-software.org

+-----------+
+  '``'--- `+  CERN - European Organization for Nuclear Research
+ |CERN|  / +  1211 Geneve 23, Switzerland
+ ..__. \.  +  IT-UDS-AVC
+  \\___.\  +  Office: 513-R-0042
+      /    +  Tel. +41227677159
+-----------+