[Zope] Folder with one million Documents?

[Thomas Guettler]

  Hi!

I am developing a simple DMS. Up to now I use a python product with a 
BTreeFolder which
contains all the documents. Every document gets an ID with 
DateTime().millis(). There will
be up to 50 users working at the same time. And in the end I will have 
up to 3 million documents.

Is there a better class than BTreeFolder for such mass storage?

For the curious, here is one result of the benchmarks:

I benchmark it with httperf:
 +5 requests per second
 +10MBit connection between client and server
 +every request creates a document

Anzahl der Dokumente: 2159
httperf.exe: warning: open file limit > FD_SETSIZE; limiting max. # of 
open files to FD_SETSIZE
httperf.exe --timeout=5 --client=0/1 --server=prophet --port=8080 
--uri=/a/benchmarks/create_new_doc --rate=5 --send-buffer=4096 
--recv-buffer=16384 --add-header='Authorization: Basic em9wZTp6b3Bl\n' 
--num-conns=1000 --num-calls=1
Maximum connect burst length: 1

Total: connections 1000 requests 1000 replies 571 test-duration 204.814 s

Connection rate: 4.9 conn/s (204.8 ms/conn, <=26 concurrent connections)
Connection time [ms]: min 110.0 avg 977.2 max 5257.0 median 289.5 stddev 
1315.9
Connection time [ms]: connect 0.5
Connection length [replies/conn]: 1.000

Request rate: 4.9 req/s (204.8 ms/req)
Request size [B]: 120.0

Reply rate [replies/s]: min 0.0 avg 2.9 max 5.2 stddev 2.2 (40 samples)
Reply time [ms]: response 976.4 transfer 0.0
Reply size [B]: header 216.0 content 79.0 footer 0.0 (total 295.0)
Reply status: 1xx=0 2xx=562 3xx=0 4xx=0 5xx=9

CPU time [s]: user 83.96 system 117.15 (user 41.0% system 57.2% total 98.2%)
Net I/O: 1.4 KB/s (0.0*10^6 bps)

Errors: total 429 client-timo 429 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0
Anzahl der Dokumente: 3075
--end:  Fri Jan 25 14:25:34 2002