[Zope-CMF] GenericSetup proposal: upgrade profiles

[Maurits van Rees]

Hi Rob,

Thanks for the feedback!

Rob Miller, on 2009-06-24:
> i'm not actively working on GenericSetup these days, so i don't know that my 
> opinion counts for much, but i'm -1 on this.  to me it seems to introduce an 
> entirely new way of handling upgrades that is likely to confuse people. 
> GenericSetup is more than complex enough already, IMO.
>
> Maurits van Rees wrote:
>> Use case
>> --------
>> 
>> For our use case we assume that you have made an add-on product.  It
>> has an EXTENSION profile.  You have applied the profile in a website.
>> Now you want to add a catalog index.  Your options are:
>> 
>> A. Add this in the current extension profile.  This means that any
>>    time you apply this profile this index will be removed and
>>    recreated so you lose data; you need to reindex the index (manually
>>    or in python code - not GenericSetup) and that takes time.
>
> in my mind this is the core problem, that there is no way to
> non-destructively apply an index to a catalog.  this is a weakness
> in GenericSetup, but developing an entirely new alternative
> mechanism for handling upgrades seems to me the wrong solution.

In general, I think we could use an option like 'override' or
'only_once', next to the purge and remove options, like this:

  <property name="email_from_name" override="False"
    type="string">Site Administrator</property>

  <property name="email_charset" override="True"
    type="string">utf-8</property>

This would then mean:

- at first the properties do not exist yet

- apply this profile -> properties do exist, with the values given

- change email_from_name to 'Acme Inc' and email_charset to 'ascii'

- reapply this profile

- email_from_name is untouched at 'Acme Inc' and email_charset has
  been overridden to 'utf-8'

Problem may be that if you do this in one import handler (say for the
properties) users may be caught off guard when this option does
nothing in another (like for the catalog) and loose data/time that
way.

>> B. Add an upgrade step.  Works fine, but this needs python code along
>>    these lines:
>> 
>>      indexes = catalog.indexes()
>>      if 'new' not in indexes:
>>          catalog.addIndex('new', 'some_meta_type')
>>          catalog.manage_reindexIndex(ids=['new'])
>> 
>>    But why add python code when there is already a perfectly fine
>>    import handler that can read a catalog.xml?  We are back to the
>>    dark days before there was GenericSetup then.  (Well, okay, writing
>>    python code is not *that* bad. ;-))
>> 
>> C. Add it to the current extension profile.  Do not apply this profile
>>    ever again: only once at install time.
>
> this confuses me.  in my mind, extension profiles should be idempotent.  IOW, 
> it should _always_ be safe to re-apply an extension profile to a site, 
> assuming that extension profile does indeed represent the final state that 
> you'd like the site to have.

In my mind, an extension profile is just a starting point, at least
partly.  Like above, you set a property email_from_name so the code
that expects this property to exist actually works.  Then your client
comes along and changes this property to something he likes.  You do
not want to reset this to the default when reapplying the profile.

For another part, an extension profile can serve as a set of known
good settings and be reapplied to cleanup the mess that the client
made when clicking through the ZMI.

>>  Instead add an upgrade step
>>    that applies the catalog step from this profile.  But this has
>>    basically the same problem as option A: if you later add a second
>>    catalog index you would need to create a new upgrade step to apply
>>    the catalog step and then your first index would be removed and
>>    recreated empty again.
>
> again, this seems to me a weakness of the catalog index importing, and it 
> should be fixed there.
>
> it is option (C) which i would recommend, so much so that there is even a 
> dedicated ZCML tag specific for this purpose: genericsetup:upgradeDepends. 
> here's an example from the GenericSetup tests:
>
>        ...       <genericsetup:upgradeDepends
>        ...           title="Bar Upgrade dependency profile import steps"
>        ...           description="Re-imports steps from the profile"
>        ...           import_profile="profile-Products.CMFDefault:default"
>        ...           import_steps="baz bat"
>        ...           run_deps="True"
>        ...           purge="True"
>        ...           />
>
> upgradeDepends lets you specify any arbitrary import steps from any arbitrary 
> profile, so that you don't have to repeat the boilerplate code of invoking 
> those import steps by hand whenever you need to write such an upgrade step.

This is definitely an improvement, and I am certainly going to use
that when GenericSetup 1.5 becomes available.  But it does not help
against a catalog index importer that recreates an empty index; or
against a properties importer that resets properties to their factory
default.


>> D. Define a new EXTENSION profile that just has a catalog.xml.  Either
>>    go to the import tab of portal_setup to run all steps of that
>>    profile, or create an upgrade step that does this.  But this means
>>    that in several spots the list of available profiles gets longer,
>>    which threatens to make it hard to use.
>
> i think the use of profiles for upgrades was an understandable stop-gap when 
> GS did not yet provide any other mechanism for handling upgrades.  doing so 
> now is, IMO, a mistake, even if you designate a specific type of profile for 
> this purpose.

Like I said: Plone is starting to do that now.  If Plone would instead
add an upgrade step that reapplies, say, its properties step because
one property was added there, you will have a mess: among others the
email_from_address (from the example above) would be reset to 'Site
Administrator' in all Plone sites.  That is not what you want.

Of course something like an 'override' option would help here.

But that gets tricky with a snippet like this:

  <property name="default_page_types" type="lines">
   <element value="Topic"/>
  </property>

If you add 'override="False"' here, then what are you going to do?  If
the complete property itself is not there, it is easy: add it with
this element.  But if the property is already there, but not the
element, then what do you do?

1. If this is the first time the profile is being applied you should
   add the element.

2. If this profile has been applied earlier, but this setting was not
   in it at the time, you should add the element.

3. But the element may have also been added earlier by the profile and
   has been deliberately removed by the site admin: you should not add
   it again.  But you cannot know the difference between this and the
   second possibility.

If this snippet would be in an upgrade profile, it is clear that the
property and the element should be added.

>>    The migration machinery of CMFPlone is moving to the
>>    plone.app.upgrade package (not released yet), which already defines
>>    about 24 upgrade steps with 11 extension profiles and that will
>>    only increase, potentially one for each point release of Plone.
>>    See http://svn.plone.org/svn/plone/plone.app.upgrade/trunk
>> 
>>    (And the CMFQuickInstaller can get confused when a package has more
>>    than one profile; but that is not a problem for this mailing list,
>>    and I expect that in Plone 4 either the CMFQuickInstaller is gone
>>    or it ends up as just a small wrapper around GenericSetup.)
>> 
>> Similar arguments could be made about adding a property to a property
>> sheet: you only want to do that once and not override any changes made
>> to that property by a site admin when you reapply the profile.
>> 
>> So I think another option would help:
>> 
>> E. Define an UPGRADE profile that just has a catalog.xml.  Going to
>>    the import tab of portal_setup should not list this profile, but
>>    only EXTENSION and BASE profiles, so that list is not so cluttered
>>    anymore.  You now just create an upgrade step that applies the
>>    profile.
>
> so if your product requires this new index, so much so that you made an 
> upgrade profile for it, it's reasonable to assume that you'd want any new 
> installations of this product to also have this new index.  does this mean 
> that when you apply an extension profile, all upgrade profiles are 
> automatically applied?  or do you have to add the index to the extension 
> profile as well?

Oops, I overlooked that point...  For something like Plone this is
easier, as it has a base profile that is applied only once for new
installs.

I wonder how much sense it would make to create a base profile for an
add-on product and use that only for new installs; in upgrade steps
you could then again use extension/upgrade profiles.  But a base
profile always sounds pretty destructive, not caring about whatever
settings other products may have added.  Maybe the difference between
a base and extension profile is not as clear cut and I should
explore base profiles more.


In the end, what I want to avoid is having to write my own python code
when there already is an import handler that is working fine as long
as it is called only once but has unwanted effects (data loss,
preference loss, time loss) when called twice; and then having to copy
that code around for each project I work on.

Are there other smart ideas on how to solve this?

-- 
Maurits van Rees | http://maurits.vanrees.org/
            Work | http://zestsoftware.nl/
"This is your day, don't let them take it away." [Barlow Girl]