[ZODB-Dev] Persisting Python objects in MongoDB
Stephan Richter
srichter at cipherhealth.com
Tue Nov 8 13:55:02 UTC 2011
Hi everyone,
as you might have seen from my checkins on Friday, I released mongopersist. In
short, it implements a persistence backend in MongoDB (that, with a little bit
of effort, could be used as a ZODB replacement in Zope):
http://pypi.python.org/pypi/mongopersist
Some background and motivation:
Many years ago, Jim told me that the persistence code could be used to
implement any storage backend. But until now I never had the need to confirm
that statement. The ZODB and its various storages served me just fine. But then
I decided to use MongoDB as storage for my latest project. (I just recently
used storm with PostGreSQL for a project and was once again annoyed about the
impedance mismatch between the stiff relational and flexible object worlds.)
It was out of question that I just wanted to use pymongo straight up, since it
makes admin interfaces just very painful to write. I first tried p01.mongo, but
unfortunately it tries to re-implement persistence with some of the most
annoying ORM-style features (such as explicit attribute declarations).
So, I came to the conclusion I wanted to build an abstraction layer myself and
see how well it would work. An ORM-style solution was out of question. I
consider attribute-type declarations ORM-style, since it prohibits me to
extend objects in arbitrary ways (think annotations, sub-classing, etc). I
then looked into implementing a ZODB storage. Unfortunately, the ZODB really
wants to deal with pickles and I wanted to store objects in a way that the
data could be used without Python.
Thus, implementing a storage mechanism on top persistence seemed like the best
choice. It was not too hard, since ZODB's storage implementations provided
good examples. Here are some of the main features:
* Full persistent data manager implementation making storing objects seamless.
* Data manager can be used to load and dump objects directly.
* Automatic and explicit specification of the mongo collection to use.
* Store multiple object types in the same collection. This is particularly
useful, if you have sub-classes of an object type.
* Storage of non-persistent objects. Pretty much everything that can be
pickled can be stored. There is some work left to do to support all pickle
hooks and object constructors.
* Plugins to provide custom serializers and de-serializers for objects. This
is useful for objects whose reduced state is not suitable for Mongo storage,
such as datetime.date.
* Control over document granularity. In the ZODB, any persistent object causes
a separate pickle/document. In mongopersist you can choose whether a
persistent object is stored as part of its parent object or not. (The Mongo
guys make a big deal about choosing document granularity.)
* Cross database support, i.e. full DBRef object references are used
everywhere.
* Optional write-conflict detection.
* Simple Mapping implementation that represents a collection.
* Optional Zope Support: Multiple container implementations for various use
cases. The main container supports one collection for multiple instances by
keeping a parent reference. The container itself can be connected to a ZODB or
mongopersist. The container also provides full access to the MongoDB query
API, filtering by parent automatically and supporting raw or object results.
Before you jump on this right away, here are some features that are missing:
* BTreeContainer instances will probably store, but they would not provide
sensible output in Mongo. Custom hooks need to be written to handle
BTreeContainers sensibly.
* Since the pickle module does not cleanly separate reducing an object and
creating a pickle string, mongopersist must effectively re-implement the pickle
API. I have implemented the most common use cases, but some pickle hooks and
handling of classes with __new__ constructors are still missing.
* Cyclic references of non-persistent objects is not supported.
That said, we are about to head into production with mongopersist. If you need
any help or want those above two problems addressed, feel free to contact me.
So was Jim right? Absolutely!
Regards,
Stephan
--
Entrepreneur and Software Geek
Google me. "Zope Stephan Richter"
More information about the ZODB-Dev
mailing list