[ZODB-Dev] Amazon SimpleDB Adapter
Laurence Rowe
l at lrowe.co.uk
Sat Oct 11 19:26:18 EDT 2008
Shane Hathaway wrote:
> Benjamin Liles wrote:
>> Currently at the Plone conference it seems that a large number of people
>> are beginning to host their Plone sites on the Amazon EC2 service. A
>> simpleDB adapter might be a good way to provide persistent storage for
>> an EC2 base Zope instance. Has there been any interest in this? If I
>> was to write one, should I add it to RelStorage or create my own package
>> along the lines of relstorage.adapters.simpledb?
>
> This sounds interesting! We should add an adapter to RelStorage. We
> might run into some trouble with MVCC, but I think we can solve that.
> We should also use Amazon S3 directly for blob storage.
>
> In general, Amazon's services seem a much better fit for ZODB apps than
> what Google is offering.
I'm not sure RelStorage is the best place for it - SimpleDB is very
different to relational databases.
A couple of years ago I experimented with s3storage [1]. This turned out
to be very slow due to the number of writes performed every transaction
- one per object, though this could be improved if the writes were
parallelized. It reached the point where zope2 would start up. This took
about 10 or 15 mintutes at the time (I did not have access to EC2 at the
time and this was over public wifi).
It worked by creating it's own indexes in S3. I don't think SimpleDB
will give any advantage unless it is shown to be faster to query than
S3. You cannot store pickles directly in SimpleDB because it is limited
to an attribute size of 1024 bytes.
The challenge in building such a system is in Amazon's eventual
consistency model means you cannot know how up to date your view of the
data is. I think it could make a great backend for storing pickles
(keyed by oid, tid) but it is probably much easier to have a separate
index to consult during loadSerial.
It may also be worth experimenting with DirectoryStorage over s3fs [2].
Laurence
[1] http://code.google.com/p/s3storage
[2] http://code.google.com/p/s3fs
More information about the ZODB-Dev
mailing list