[ZODB-Dev] Amazon SimpleDB Adapter

[Shane Hathaway]

Laurence Rowe wrote:
> I'm not sure RelStorage is the best place for it - SimpleDB is very 
> different to relational databases.

RelStorage doesn't use much of a relational database either (except
during packing).

> A couple of years ago I experimented with s3storage [1]. This turned out 
> to be very slow due to the number of writes performed every transaction 
> - one per object, though this could be improved if the writes were 
> parallelized. It reached the point where zope2 would start up. This took 
> about 10 or 15 mintutes at the time (I did not have access to EC2 at the 
> time and this was over public wifi).
> 
> It worked by creating it's own indexes in S3. I don't think SimpleDB 
> will give any advantage unless it is shown to be faster to query than 
> S3. You cannot store pickles directly in SimpleDB because it is limited 
> to an attribute size of 1024 bytes.
> 
> The challenge in building such a system is in Amazon's eventual 
> consistency model means you cannot know how up to date your view of the 
> data is. I think it could make a great backend for storing pickles 
> (keyed by oid, tid) but it is probably much easier to have a separate 
> index to consult during loadSerial.

Thanks for the background on S3 and SimpleDB.  Using Amazon's storage
services as a ZODB backend is sounding ever more like an interesting
challenge.

Shane