[ZODB-Dev] Relstorage Blob support and Oracle

[Martijn Pieters]

On Fri, Jun 10, 2011 at 07:17, Shane Hathaway <shane at hathawaymix.org> wrote:
> I see your thinking now.  RelStorage will download multiple chunks from
> Oracle but will now only upload one chunk per blob to Oracle.  If others
> want to do the same for PostgreSQL or MySQL, you've set the example of
> how to do it without disruption.
>
> Thank you for not changing the schema.  I get a lot of complaints these
> days anytime the schema changes.  I ask just one favor: please run some
> manual tests with multi-gigabyte blobs that don't fit in RAM.

I've committed my refactor now.

In testing I ran into some limitations of cx_Oracle; it uses the older
(pre 10.1) OCI APIs that only allow offsets and sizes of up to 4GB
(e.g. using 32-bit unsigned integers). Moreover, it's naive use of
signed integers instead of unsigned ones means that on 32-bit
platforms the largest *LOB write and read offsets it can handle is 2GB
- 1 byte (sys.maxint). I've seen that the author is already aware that
there is a newer API that upgrades the offset and size types to 64-bit
unsigned integers, and I've asked him what the status is of cx_Oracle
supporting that.

In the meantime, to answer your request for a favour: Uploading and
downloading blobs within these limits worked great and memory usage
wasn't impacted at all.

For us these limits are absolutely not a problem. Although our Oracle
setup is performing very nicely when it comes to BLOB reading and
writing (multiple MBs per second), pulling a BLOB of several GB to a
file in your local non-shared blob cache before serving is going to
take too long anyway, and you really want to use a shared blob storage
for that instead.

I've only tested the Oracle blob upload and download methods in the
mover, since that's what changed in this commit. I didn't run the full
test suite as I lacked the required SYS access on the oracle cluster
and didn't have time to set up a local Oracle database. If you could
run the test suite against your Oracle setup that'd be great. :-)

Next I'll look into adding PostgreSQL support too; it has a nicer API
still in that it let's you specify files directly to upload from or
download to.

-- 
Martijn Pieters