[ZODB-Dev] Storing blob data out of zodb into amazon s3?

[Sean Upton]

On Thu, Jul 5, 2012 at 1:05 PM, Noe Nieto <nnieto at noenieto.com> wrote:
> I was wondering if it's possible to write some sort of "cloudstorage" plugin
> for ZODB that saves blobs in amazon S3 or something similar.
>
> The idea is cool because all things cloud are cool. But there might be some
> shortcomings like performance or something like that. What do you think?

A bit of a tangent: I think this conversation is more interesting in
the general case -- not specific to S3/EBS/AWS.  I have a different
idea bubbling based on eventual need:

  * I want to partition blobs to different named storage destinations
("local", "near", "far/cloud") based on policy (e.g. byte-length of
data, mime detected from magic number / file signature of data, or
some kind of MRU/LRU ruleset).

  * I would rather leave the actual location and implementation
details up to filesystem... e.g. local FS, NFS, FUSE.

  * My primary use case is shuffling big (video) files to NFS instead
of storing on my VPS/VM local storage, but I may want to cache MRU
files and certainly keep all small blobs on local storage.  But the
idea here is just as applicable to any IaaS case where vendor(s) (in
my case, my University's internal/campus cloud services) provide
multiple storage venues (VPS/VM local, EBS, S3, etc).

Sean