[ZODB-Dev] Relstorage Blob support and Oracle

[Martijn Pieters]

(Re-send since zope.org blocked my email because my ISP smarthost has
been blacklisted because someone else has left their mail server relay
wide open and forwarded spam to a honeypot)

On Thu, Jun 9, 2011 at 19:09, Shane Hathaway <shane at hathawaymix.org> wrote:
> Well, the ZODB blob interfaces make it impossible to actually stream
> blobs from the database.  The entire blob has to be read into cache
> before the client can get it, so I'm not sure what you will gain.

I'm not interested in streaming the blob info straight over Zope; just
streaming it to the local blob cache. Yes, we'll still need to read
the whole blob, but streaming from Oracle to the filesystem is going
to be faster if we follow best practices for how Oracle handles these.
This means less load on the Oracle cluster, and requests involving
ZODB blobs not in the cache completing faster.

>> I'll be refactoring the blob support currently found in RelStorage
>> 1.5b2 to just store one blob in one row, using the cx_Oracle
>> LOB.read() and .write() methods, which let you read and write to blobs
>> in chunks to avoid memory overload, and I'll reuse the blob chunk size
>> to determine how much we read / write per iteration.
>>
>> I am currently leaning towards dropping the chunk column in the Oracle
>> schema altogether; it certainly won't hold any other value than
>> integer "0" after my refactor. Any reason to keep it, other than that
>> others whom already are using 1.5b2 on an Oracle database will now
>> have to drop that column again (or set it to a default value of "0" on
>> insert)? Should the code support reading blob chunks still?
>
> No, please don't change the schema.  I know all of the supported
> databases support a form of streaming, but they are all wildly
> different.  I suggest that chunking is much simpler and has little
> overhead compared with streaming.

I'm retaining the schema; there is a chance people have updated to
1.5b2 already and are using blobs in production. My refactor maintains
compatibility with the chunked blob storage.

-- 
Martijn Pieters