[ZODB-Dev] ZODB blob support.

[Christian Theune]

Hi all.

I packed up a Zope 2.8 with my ZODB branch for blobs and implemented a
"stupid" OFS.Blob.File object and made some simple "ab" tests. Here are
the results of the "high tuned" OFS.File.File (without blob) versus the
most simple OFS.Blob.File (but with blob support):

ab2 was called this way: ab2 -n 1000 -c 20 http://localhost:8080/asdf

Round 1, filesize ~200k
=======================

OFS.File.File
-------------

Concurrency Level:      20
Time taken for tests:   7.100714 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      215900603 bytes
HTML transferred:       215632268 bytes
Requests per second:    140.83 [#/sec] (mean)
Time per request:       142.014 [ms] (mean)
Time per request:       7.101 [ms] (mean, across all concurrent
requests)
Transfer rate:          29692.79 [Kbytes/sec] received

This one didn't get all connections through and dropped a couple
sockets. Therefore the amount of data transferred differs.

OFS.Blob.File
-------------

Concurrency Level:      20
Time taken for tests:   5.539628 seconds
Complete requests:      1000
Failed requests:        998
   (Connect: 0, Length: 998, Exceptions: 0)
Write errors:           0
Non-2xx responses:      2
Total transferred:      215248884 bytes
HTML transferred:       215084695 bytes
Requests per second:    180.52 [#/sec] (mean)
Time per request:       110.793 [ms] (mean)
Time per request:       5.540 [ms] (mean, across all concurrent
requests)
Transfer rate:          37945.33 [Kbytes/sec] received

Funnily this one also kicked _one_ (exactly the last) connection...

Round 2, filesize ~70MB
=======================

OFS.File.File
-------------

No stats. The server spilled my harddisk after about 100 Requests, I had
to kill ab.

OFS.Blob.File
-------------

Memory and disk usage is pretty low, the python producer spills out the
data constantly with about ~100 MB per second (loopback). This takes a
while, though. 

The python process loads the cpu pretty much, this could be better. :)

Concurrency Level:      20
Time taken for tests:   500.718270 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      -89217134 bytes
HTML transferred:       -89385104 bytes
Requests per second:    2.00 [#/sec] (mean)
Time per request:       10014.365 [ms] (mean)
Time per request:       500.718 [ms] (mean, across all concurrent
requests)
Transfer rate:          -174.00 [Kbytes/sec] received

Obviously, the second round was even for OFS.Blob.File bound by the
internal transfer rate of the loopback device and other overhead stuff.
I decided to go for a third round with less data again:

Round 3, filesize ~5MB
======================

OFS.File.File
-------------

Concurrency Level:      20
Time taken for tests:   297.781491 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      696487394 bytes
HTML transferred:       696216872 bytes
Requests per second:    3.36 [#/sec] (mean)
Time per request:       5955.630 [ms] (mean)
Time per request:       297.781 [ms] (mean, across all concurrent
requests)
Transfer rate:          2284.10 [Kbytes/sec] received

In this case python did not use more RAM. The process was very
disk-based IO bound. Python couldn't make use of the complete CPU due to
that and used it about 25%.

OFS.Blob.File
-------------

Concurrency Level:      20
Time taken for tests:   35.506402 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      682015648 bytes
HTML transferred:       681848696 bytes
Requests per second:    28.16 [#/sec] (mean)
Time per request:       710.128 [ms] (mean)
Time per request:       35.506 [ms] (mean, across all concurrent
requests)
Transfer rate:          18758.03 [Kbytes/sec] received

This didn't end in any unexpected ends. No connections dropped. The
process made full use of the CPU.

So, for a first shot, on a prototype. This looks pretty promising to me.
Especially as I didn't have to do anything special for an application
object to support this, code handling blobs is likely to get _much_ more
robust.

Cheers,
Christian

-- 
gocept gmbh & co. kg - schalaunische str. 6 - 06366 koethen - germany
www.gocept.com - ct at gocept.com - phone +49 3496 30 99 112 -
fax +49 3496 30 99 118 - zope and plone consulting and development
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20050322/4beddf30/attachment.bin