[Zope-dev] Changing and migrating persistence structure

[Jim Fulton]

On Thu, Aug 5, 2010 at 2:36 AM, Martin Aspeli <optilude+lists at gmail.com> wrote:
...
> I have a package (plone.registry) that currently has a persistent
> structure like this:
>
> Registry(Persistent)
> |
> +--> Records(Persistent)
>       |
>       +--> BTree of Record(Persistent)
>              |
>              +--> PersistentField(Persistent)
>
> That is, a Registry is a persistent object containing a persistent
> Records object that in turn contains a BTree of persistent Record

Since BTrees are mapping, I assume that you mean the values are
records and that the keys are something boring like strings or
integers.

I like to use mathematical notation when talking about BTrees and
sets, as in:

  Registry
     BTree {? -> Record}


> objects that contain a persistent PersistentField and a primitive
> value.
>
> This is quite inefficient, though, because it results in a lot of
> object loads. On any given request, some of our projects load a dozen
> or more values from the registry. Each is just a simple primitive, but
> we need to load the full shebang to make it work.

Not sure what you mean by "full shebang".

>
> Now, I'd like to move to this structure:
>
>  Registry(Persistent)
>  |
>  +--> Records
>       |
>       +--> BTree of Field
>       |
>       +--> BTree of values

I'm foggy on what "field" and "value" are here or what your queries
are doing. Maybe this is just a distraction.

> Here, there's only one Persistent object, plus the two BTrees: one
> holding all the fields and one holding all the values. Records no
> longer needs to be persistent (its attributes become part of the
> parent Registry's _p_jar).

I wonder what role "Records" plays independent of the "Registry".

I also wonder why it matters whether it is persistent or not.

> Fields no longer need to be persistent
> either, since they are in effect immutable objects. Values are
> primitives anyway.
>
> I've done this (in a branch) and it works for new sites. However, I'm
> having a nightmare trying to migrate old sites. As soon as I access
> anything that uses the registry, I get ZODB errors, because the
> persistent structure is now different. In particular, it's trying to
> read a value into e.g. a Records object that used to derive from
> Persistent, but now no longer does.

What savings do you get by making Records non-persistent?

> What is the best way to manage this type of migration?

Today, it probably makes the most sense to make new classes for the
non-persistemnt objects.  You'll then need to write a script to
rebuild the data structures.

Jim

-- 
Jim Fulton