[ZODB-Dev] Storing blob data out of zodb into amazon s3?
Jim Fulton
jim at zope.com
Thu Jul 5 20:58:17 UTC 2012
On Thu, Jul 5, 2012 at 3:05 PM, Noe Nieto <nnieto at noenieto.com> wrote:
> I was wondering if it's possible to write some sort of "cloudstorage" plugin
> for ZODB that saves blobs in amazon S3 or something similar.
>
> The idea is cool because all things cloud are cool.
Um, well no. :)
But if you're in AWS, I think there are some advantages to S3 over
EBS.
> But there might be some
> shortcomings like performance or something like that. What do you think?
I took an initial stab, as Satchit pointed out. It was a bit naive.
S3 is too slow, especially when committing transactions.
I wanna make this work though. My current thinking is:
- Save to the blob directory.
- Have a separate process that moves files to S3 (copy to s3, then
delete from blob directory on success or later).
- On blob load, if the file can't be found on the local directory,
fetch from S3.
There are a number of possible variations:
- Don't delete from the blob directory immediately after upload to use
it as a cache for new content.
- Have the client look in S3 before contacting the ZEO server
(although that has the downside of having to distribute keys to the
clients). (Maybe only do this if the serial is more than some time
in the past.)
- Have the client fetch from S3 if the blob isn't in the ZEO server's
blob directory (as indicated by a response from the ZEO server).
- Have a separate process that serves the blobs deom the blob
directory or from S3 via HTTP.
I did some experiments that showed significant performance benefits
in a heavily loaded ZEO server in serving blobs from an separate
HTTP server.
If you're going to serve blob data directly, without modification, it
may make sense to store it in S3 at the application level, rather than
going through ZODB. Alternatively, it would be nice to have a way to
get the S3 location for a blob. Maybe this could be done through some
extension API. <shrug>
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm
More information about the ZODB-Dev
mailing list