[ZODB-Dev] ZODB blob support.

[Chris McDonough]

On Mon, 2005-03-21 at 08:16, Jeremy Hylton wrote:
> On Thu, 17 Mar 2005 14:59:32 -0500, Chris McDonough <chrism at plope.com> wrote:
> > In preparation for some work that will tentatively happen at the Pycon
> > sprints, I've put up a very basic proposal outlining "blob" support for
> > ZODB.  It would help if people with a stake in ZODB had the time to
> > comment on it before work starts (please comment here, as opposed to on
> > the proposal itself, just so we get the widest audience).
> > 
> > http://www.plope.com/Members/chrism/zodb_blob_proposal
> 
> Just a quick reply, sorry it's coming after the sprint started.  
> 
> How does this work with ZEO?  I assume that almost every site that
> cares about blob support will be running ZEO, where it's not obvious
> how to get direct file access.  Aren't you stuck using all the
> standard, slow ZEO mechanisms?  I suppose you could store the blobs in
> individual files outside the standard ZEO cache, but you ought to
> think about how they interact with things like the cache budget.

We had a bit of discussion at the sprint about this and there are two
camps:

- People who are willing to accept a weaker guarantee of data
  integrity and thus who are willing to both write and read data
  to/from a shared filesystem directly from ZEO clients.

- People are are unwilling to accept any weakening of data integrity
  guarantee and who want the ZEO server itself to be responsible
  for writing data (the ZEO clients must send the server the data
  over RPC).  OTOH, if configured properly, the data may be read
  from a shared filesystem.

I'm in the former, FWIW.  Jim is in the latter.  Christian Theune is
somewhere in between, I think.  In the latter strategy, blob data may
end up in the ZEO cache (if data is read over ZEO) so some changes may
be necessary.  In the former, it will never touch it, so none are.

> Also, I assume that reads and writes are transactional as for normal
> objects.  So writing a byte somewhere in a file causes the object to
> be modified and registered with the database connection / transaction
> manager.

I think we want to use something like what I'm told Toby does for
DirectoryStorage (using os.rename in the 2nd phase, keeping a journal to
be able to back out a failed 2nd-phase commit).

- C