[ZODB-Dev] LargeFS3

[Tamas Hegedus]

Hi,

I just would like to summarize my experience:
Two files:
1. 800M; approx 200,000 record
2. 6G; approx 2,400,000 record

-----------------------------------
POPULATION:
a, it was pretty linear (see details below);
- average values: 50,000 object in 2-4 minutes; 600M RAM; 120M swap
- after the 2050000 obj there was something: slowing down for the next
50,000 obj; RAM usage went up to 900M, swap up to 450M; finally RAM
stabilized around 800M, swap around 450M

b, file size:
- fs: 8G (not so bed)
- fs.index: 21.7M

-----------------------------------
RETRIEVAL (one object):

Somehow, the sturtup cost is very high (takes 3 minutes).
Can you tell me what happens at startup? Reading of the 21M index into
the memory?

---------------
A, zodb
Config:
14:44:08 2006
14:47:04 2006

Open:
14:47:04 2006
14:47:04 2006

Connection:
14:47:04 2006
14:47:04 2006

Retreive:
14:47:04 2006
<biohegedus.SProt.Record object at 0xb733efac>
14:47:04 2006
---------------
B, zeo:
- zeostartup: seems fast, but after zeoctl start finishes it seems that
the same things happens in the background;
- retrival using zeoclient very fast:
Config:
15:02:24 2006
15:02:25 2006

Open:
15:02:25 2006
15:02:25 2006

Connection:
15:02:25 2006
15:02:25 2006

Retreive:
15:02:25 2006
<biohegedus.SProt.Record object at 0xb6fe502c>
15:02:25 2006

=========================================================
OTHERS (with zeoclient):
- k = OOBTree.keys(): fast < 1.0s
- len(k) is very slow: > 10 min

- selecting 100,000 random keys in an ordered way (rndNumList.sort()): 
10 minutes
- Retreiving 100,000 random objects by primary keys (keys are not 
ordered; RAM usage went up slowly from 100->400M; HT processor(s) were 
used only 0-15%
Sat Feb 11 18:57:41 2006
Sat Feb 11 19:10:33 2006
approx 130 objects/s

Some RDBMS comparison can be found at
http://mbiosql.biohegedus.org/sbmarks.htm
!!! The time of that benchmark the database was much smaller.
!!! I just selected two attributes (two raws from one table) of an 
object from the database; now the whole object is instantiated.
!!! So not so bed. Although some real RDBMS comparison with SQLobject 
should be done.

--------------------------------
!!! The queries I used for testing are not typical for me: e.g. I never 
do a len(keys())... So I will see...

Regards,
Tamas

-------------------------------------------------------------------------------------------
MY POPULATION LOG
Started: 11 11:41:45 2006
50000   Sat Feb 11 11:45:27 2006
100000  Sat Feb 11 11:49:26 2006
150000  Sat Feb 11 11:54:05 2006
200000  Sat Feb 11 11:58:33 2006
Started: 11 11:58:59 2006
50000   Sat Feb 11 12:01:30 2006
100000  Sat Feb 11 12:04:12 2006
150000  Sat Feb 11 12:06:39 2006
200000  Sat Feb 11 12:09:05 2006
250000  Sat Feb 11 12:11:53 2006
300000  Sat Feb 11 12:14:31 2006
350000  Sat Feb 11 12:17:21 2006
400000  Sat Feb 11 12:19:37 2006
450000  Sat Feb 11 12:21:56 2006
500000  Sat Feb 11 12:24:20 2006
550000  Sat Feb 11 12:26:56 2006
600000  Sat Feb 11 12:30:13 2006
650000  Sat Feb 11 12:32:39 2006
700000  Sat Feb 11 12:35:03 2006
750000  Sat Feb 11 12:37:19 2006
800000  Sat Feb 11 12:39:35 2006
850000  Sat Feb 11 12:42:03 2006
900000  Sat Feb 11 12:44:39 2006
950000  Sat Feb 11 12:46:49 2006
1000000 Sat Feb 11 12:49:02 2006
1050000 Sat Feb 11 12:51:04 2006
1100000 Sat Feb 11 12:53:28 2006
1150000 Sat Feb 11 12:55:50 2006
1200000 Sat Feb 11 12:58:38 2006
1250000 Sat Feb 11 13:01:20 2006
1300000 Sat Feb 11 13:03:49 2006
1350000 Sat Feb 11 13:06:21 2006
1400000 Sat Feb 11 13:08:50 2006
1450000 Sat Feb 11 13:11:16 2006
1500000 Sat Feb 11 13:13:35 2006
1550000 Sat Feb 11 13:16:09 2006
1600000 Sat Feb 11 13:18:30 2006
1650000 Sat Feb 11 13:20:47 2006
1700000 Sat Feb 11 13:23:22 2006
1750000 Sat Feb 11 13:25:46 2006
1800000 Sat Feb 11 13:28:08 2006
1850000 Sat Feb 11 13:30:52 2006
1900000 Sat Feb 11 13:33:32 2006
1950000 Sat Feb 11 13:36:03 2006
2000000 Sat Feb 11 13:38:50 2006
2050000 Sat Feb 11 13:41:32 2006
2100000 Sat Feb 11 13:48:57 2006
2150000 Sat Feb 11 13:53:45 2006
2200000 Sat Feb 11 13:56:23 2006
2250000 Sat Feb 11 13:58:42 2006
2300000 Sat Feb 11 14:01:05 2006
2350000 Sat Feb 11 14:04:02 2006
2400000 Sat Feb 11 14:07:13 2006
2450000 Sat Feb 11 14:10:33 2006
2500000 Sat Feb 11 14:13:08 2006
2550000 Sat Feb 11 14:15:41 2006
2600000 Sat Feb 11 14:18:18 2006


-- 
Tamas Hegedus, PhD          | phone: (1) 919-966 0329
UNC - Biochem & Biophys     | fax:   (1) 919-966 5178
5007A Thurston-Bowles Bldg  | mailto:hegedus at med.unc.edu
Chapel Hill, NC, 27599-7248 | http://biohegedus.org