[ZODB-Dev] ZODB alternate serialization format patch
Shane Hathaway
shane at hathawaymix.org
Thu Jan 15 12:15:54 EST 2009
Hanno Schlichting wrote:
> Shane Hathaway wrote:
>> I have just created a patch for ZODB that makes the object serialization
>> format pluggable, meaning that it should now be possible to write Python
>> code that causes ZODB to store in formats other than Python pickles.
>
> Cool :)
>
>> In addition to making the data format pluggable, the patch puts most of
>> ZODB's dependencies on the cPickle module in one place, so now if we
>> decide to improve our usage of the cPickle module, we can make that
>> change in one place rather than several.
>
> I'm wondering if this would be a good opportunity to include a version
> marker in addition to the protocol format to the API's?
Maybe. Starting with pickle protocol version 2, pickles do start with a
version header.
> The ZODB still uses version one of the pickle protocol throughout and so
> far it has been complicated and cumbersome to change this in any way.
Is it cumbersome? I tried changing to pickle protocol v2 while making
this patch. Only 4 of the 3000+ ZODB tests failed. One of them failed
because we have some pickle introspection code that does not yet
understand protocol 2; that should be easy to fix. Another failed
because it seems to depend on the exact length of a generated pickle,
which is a bit silly. I didn't look at the rest of the test failures
because I assume they are all similarly superficial.
> While the documentation of Protocol Buffers mention that they try to be
> stable and avoid incompatible versions, I don't trust any standard to be
> so generic that it can avoid incompatible changes over a period of many
> years.
The patch I made might cover this. After this patch, ZODB will look for
the serializer format name in curly braces at the beginning of each
serialized object. That format name could include a version number if
necessary.
BTW, I've learned that Google's protocol buffers lack the concept of a
mapping, and I think that might be a major problem for my current
project. We might switch to a different language agnostic format like
Thrift or JSON.
Shane
More information about the ZODB-Dev
mailing list