[Zope-dev] Changing and migrating persistence structure

[Martin Aspeli]

Hi,

[I posted this to zodb-dev, but it seems that list isn't working at
the moment(?) so I thought I'd try here too]

I have a package (plone.registry) that currently has a persistent
structure like this:

Registry(Persistent)
|
+--> Records(Persistent)
       |
       +--> BTree of Record(Persistent)
              |
              +--> PersistentField(Persistent)

That is, a Registry is a persistent object containing a persistent
Records object that in turn contains a BTree of persistent Record
objects that contain a persistent PersistentField and a primitive
value.

This is quite inefficient, though, because it results in a lot of
object loads. On any given request, some of our projects load a dozen
or more values from the registry. Each is just a simple primitive, but
we need to load the full shebang to make it work.

Now, I'd like to move to this structure:

 Registry(Persistent)
 |
 +--> Records
       |
       +--> BTree of Field
       |
       +--> BTree of values

Here, there's only one Persistent object, plus the two BTrees: one
holding all the fields and one holding all the values. Records no
longer needs to be persistent (its attributes become part of the
parent Registry's _p_jar). Fields no longer need to be persistent
either, since they are in effect immutable objects. Values are
primitives anyway.

I've done this (in a branch) and it works for new sites. However, I'm
having a nightmare trying to migrate old sites. As soon as I access
anything that uses the registry, I get ZODB errors, because the
persistent structure is now different. In particular, it's trying to
read a value into e.g. a Records object that used to derive from
Persistent, but now no longer does.

What is the best way to manage this type of migration?

In terms of API compatibility, I'd really like to keep
plone.registry.Record as the name and module path of the record, since
it is used by the API. The difference is that before it was persisted
and returned by an API on the Registry. Now, it's constructed as
needed on the fly from the internal data structure.

The same applies to the various field types that derive from
PersistentField, which are now Persistent, but won't be. There's code
and documentation out there that use these.

I'm less worried about the Records object, which was always an
implementation detail, and the BTree-of-records, which will never have
been accessed directly.

Cheers,
Martin