[Zope3-Users] Generations - a missed use case? (Evolving an application for the first time)

[Jim Fulton]

Jeff Shell wrote:
> I have an application where I'm trying to use 'zope.app.generations'
> for the first time. And after much pulling of hair and looking at the
> core code, I found what may be a missed scenario.
> 
> Basically, we deployed this application for a customer and now they
> want some changes. It changes the schema of one item to store values
> in tuples instead of strings. I wrote an evolve module, 'evolve1.py'
> in myapp.generations, made the schema manager, registered it as a
> utility, bla bla bla. But my code seemed like it would never run.
> 
> I'd run the debuzope script and look at the database
> root['zope.app.generations'], and there was 'myapp' with the value of
> 1. I'd keep deleting that key and committing the transaction, but my
> code would still never run. I tried running it manually using
> pdb.run() to step through it and make sure that it was finding the
> right objects and doing its job. My code was fine. So I looked at the
> code in zope.app.generations.generations and found this interesting
> tidbit:
> 
>         for key, manager in findManagers():
>             generation = generations.get(key)
> 
>             if generation == manager.generation:
>                 continue
> 
>             if generation is None:
>                 # This is a new database, so no old data
> 
>                 if IInstallableSchemaManager.providedBy(manager):
>                     try:
>                         manager.install(context)
>                     except:
>                         transaction.abort()
>                         logging.getLogger('zope.app.generations').exception(
>                             "Failed to install %s",
>                             key)
>                         raise
> 
>                 generations[key] = manager.generation
>                 transaction.commit()
>                 continue
> 
>  (the code continues from there
> 
> There's one problem here - in my situation, it's NOT A NEW DATABASE.
> There is old data that needs to be evolved, but there's no record of a
> generation for this application because there was no need for a schema
> manager until now.
> 
> I really like the concept and general implementation of the schema
> manager, but this scenario is driving me crazy. I could write an
> 'install' script, but that doesn't really cover this situation. After
> install is run, the database marks the application generation. This
> makes sense for new applications installing themselves - there's no
> old data to update, so if the application is at generation 5, for
> example, it doesn't need to be evolved to '5' if all of the data
> that's installed or used is already in generation 5 form. (ie - if I
> were deploying my application fresh today, my fields would already be
> tuples instead of strings).
> 
> But my situation, where I already have a deployed application, is not
> covered by this. I *could* write an 'install' script for the schema
> manager that did this first evolution that I need to do. But then that
> installer would have to be updated with all of the future evolutions
> as well - since in theory, I could update an application from before
> the schema manager and need to bring it up to generation 5 or 8 from
> essentially 0.

Note that a common strategy for install scripts is to run evolution
scripts.  This is fairly straightforward.

The assumption of the generations system was that you would use it
from the start.  You make a good point though that many people won't
pay attention to the generations system until they need it, which, as things
are, is too late.

> It seems like the Schema Manager needs an 'evolve from 0' option, with
> '0' being set by the evolution script of no previous evolution was
> found but (somehow) existing data could be detected.  The other
> solutions seem to be:
> 
> * Write an install script that then manually calls all of the evolvers
> to bring things up to the current generation.
> * Always put a schema manager in your application, with the starting
> generation of 0, so that you can upgrade in the future.
> 
> Neither option seems quite tenable - like a bad hack that goes against
> the Zope 3 concepts. You shouldn't need a schema manager utility until
> you need it,

Fron a practical perspective, I agree.

 > and having a script that manually does
> 'evolve1.evolve(context); evolve2.evolve(context)'... seems like it
> goes against the sort of problem that the generations system is trying
> to solve.

I don't really see why.  See below.

> Is there something about the schema manager/generations system that I missed?

Only this: I don't want to saddle developers with supporting all
old versions of their products.  In many cases, this could represent a
very significant burden.

There are really 2 related cases: install and catchup.  Install
is meant for situations in which an application hasn't been used
before, while catchup is for apps that need to get around not providing
generation support in the past.  Unfortunately, there's no way for the
generation system to know which situation applies.  Presumably, an install
script can have the logic to know this and perform some appropriate
action.  Note too that an application that needs to catch up from
an "unmanaged" generation to a current generation might prefer to
provide a direct conversion, rather than taking intermediate steps.
Assuming that the best way to catch up is by running all evolution scripts
from (before) the beggining is guessing too much.

In summary, I think that the installation script is the right way to
handle this situation.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org