[ZODB-Dev] Changing the pickle protocol?

[Hanno Schlichting]

On Wed, Apr 28, 2010 at 6:02 PM, Jim Fulton <jim at zope.com> wrote:
> On Wed, Apr 28, 2010 at 11:43 AM, Hanno Schlichting <hanno at hannosch.eu> wrote:
>> On Wed, Apr 28, 2010 at 5:11 PM, Jim Fulton <jim at zope.com> wrote:
>> - The pickle and cPickle modules now automatically intern the strings
>> used for attribute names, reducing memory usage of the objects
>> resulting from unpickling. (Contributed by Jake McGuire; issue 5084.)
>
> I can't see why this should be protocol specific.
>
>> - The cPickle module now special-cases dictionaries, nearly halving
>> the time required to pickle them. (Contributed by Collin Winter; issue
>> 5670.)
>
> That's odd. cPickle lready special-cased dictionaries.
>
>> Unless I've misread the code, these changes only apply to protocol
>> two.
>
> We should double check that. I'll take a closer look.

Sorry for the noise. I took those statements from memory and what I'd
seen in the unladen swallow codebase, where these changes where first
made.

I double checked the actual changes as done in Python and they are not
related to protocol two. The issue 5670 change applies to protocol one
and above. The two changesets are at
http://svn.python.org/view/python/trunk/Modules/cPickle.c?r1=72223&r2=72909
and http://svn.python.org/view/python/trunk/Modules/cPickle.c?r1=71938&r2=72223.

>> And then there's the old claims of pep 307 stating that pickling
>> new-style classes would be more efficient.
>
> Which doesn't apply to persistent object since they're handled differently.

Sure, not every object in a zodb derives from persistent, but the
impact of this is probably small.

>> Right. I'm not particular interested in the pickle class registry.
>> Having a hard dependency between code filling the registry and the
>> actual data has all sorts of implications. I don't really want to go
>> there myself.
>
> It has some positive implications if you get it right:
>
> - Smaller pickles
> - Easier class renaming
> - Potentially greater security
>
> Getting it right is almost certainly a bigger project than
> anyone wants to deal with right now.

Indeed. I see that as a separate project.

> Cool. BTW, you might want to (try :) to search the list archives. I
> think someone did some
> experiments a while back.

I think that was me :)

Hanno