[Grok-dev] Pinning Down Versions

[Jan-Wijbrand Kolman]

===============
Pinned versions
===============

During the Neanderthal Grok Sprint, Luciano and I have been working
on "pinning down versions" to protect Grok development and deployment
from broken, incompatible and conflicting egg releases of all the
dependencies.

The first thing we did is to come up with and test (on Linux, OS X
and Windows) a list of known working egg versions. You can see the
current list here:

   http://svn.zope.org/*checkout*/grok/trunk/versions.cfg

With this list of versions, we updated Grok's `buildout.cfg` to:

   1) use the `versions.cfg` by way of the `extends` directive.

   2) point the `versions` directive to the versions section that is
      defined in the `versions.cfg` file.

What this now means for the development *of* Grok and the development
*with* Grok is described below.

**NOTE:** Now that we have a set of versions that actually work, we
can evolve this to something with as much final releases in at as
possible.

For developers *of* Grok
=========================

The `buildout.cfg` of Grok now use the `versions` and `extends`
directive like so::

   [buildout]
   ....
   extends = versions.cfg
   versions = versions # being defined in the versions.cfg file
   ....

Now, when running `./bin/buildout` it will make sure exactly those
versions of the egg dependencies are downloaded and installed for
this Grok buildout.

During Grok's lifecycle newer versions of the egg dependencies might
be released. If you want to test such a newer version, you could
temporarily modify the `buildout.cfg` and override the `versions`
section like so::

   [buildout]
   extends = versions.cfg
   versions = versions # being defined in the versions.cfg file
   ....

   [versions]
   # whatever specific egg version is defined in this section,
   # will override the version in the versions.cfg
   zope.i18nmessageid = 3.5.0

One other way to do this is to create a temmporary config file,
say `dev.cfg` and experiment with newer eggs versions like so::

   [buildout]
   extends = buildout.cfg

   [versions]
   # whatever specific egg version is defined in this section,
   # will override the version in the versions.cfg
   zope.i18nmessageid = 3.5.0

Make sure you use this config file when running the buildout command
like so::

   $ ./bin/buildout -c dev.cfg

If the newer egg version proves to be compatible, the newer version
number can be put in the `versions.cfg` and checked in. We think it
is very important to make sure the newer egg version *and its
dependencies* are to be available on at least Windows too.

Whenever a Grok release tag is made, we obviously have a tagged fixed
`versions.cfg` file as well. Besides the regular steps needed to make
an actual Grok release, extra steps will be required to support
grokproject (the reason is explained in the following section):

   1) the tagged `versions.cfg` needs to be uploaded to
      http://grok.zope.org/releaseinfo/grok-[versionnumber].cfg

   2) the version of the Grok egg needs to be added to the
      `grok-[versionnumber].cfg` file, e.g. `grok = 0.11`.

   3) the contents of http://grok.zope.org/releaseinfo/current
      needs to be updated to contain the name of the current
      'released' Grok versions info file. E.g. `grok-0.11.cfg`.

For developers *with* Grok
==========================

We think it is reasonable to assume that users of Grok, in other
words application developers that use Grok as a basis, are using
grokproject for setting up the initial project structure they'll use.
At least they should be strongly encouraged to do so.

We should also support upgrading an already existing project that has
been created with grokproject, i.e. when a new Grok has been released.

Initial project creation
------------------------

For the grokproject user, nothing will change really. He will issue
the grokproject command and this will setup an project structure for
him.

What the new grokproject will do by default is this:

   1) fetch the name of the current versions file from
      http://grok.zope.org/releaseinfo/current

   2) add to the `buildout.cfg` an `extends` directive pointing to
      the URL of the named versions file. E.g.
      http://grok.zope.org/releaseinfo/grok-0.11.cfg

   3) add the `versions` directive pointing to the `[versions]`
      section in the named versions file

Now when the buildout command is run, exactly those versions of the
eggs will get installed.

When running grokproject this behaviour can be overridden. The
application developer can choose to specify the URL pointing to
custom versions file like so::

   $ grokproject \
     --version-info-url=http://hostname/some/path/versionsfile.cfg

The `extends` directive will then point to that URL.

**NOTE:** the implementation for the "new" grokproject lives on the
"jw-luciano-use-versions-file" branch until it is merged, probably
somewhere next week.

Overriding versions of particular eggs
--------------------------------------

It is likely that the application developer introduces more
dependencies on other software. He should of course add this
dependecy in his `setup.py file`. If he wants to make sure that a
specific version of the dependency is used when deploying the
project, he can add this specification to his `buildout.cfg` like so::

   [buildout]
   extends = http://grok.zope.org/releaseinfo/grok-0.11.cfg
   versions = versions
   ....

   [versions]
   spam = 1.2.3

Likewise, if the application developers wants to override the version
of one of the eggs grok depends on he can adjust his `buildout.cfg`
to look like this::

   [buildout]
   extends = http://grok.zope.org/releaseinfo/grok-0.11.cfg
   versions = versions
   ....

   [versions]
   zope.i18nmessageid = 3.5.0 # will override the original spec

By the way, users of grokproject should be encouraged to let the Grok
developers know about wanting to use a newer version of a current
Grok dependency. With this information, the Grok developers can
consider this newer version to be specified - "blessed" - for the
upcomming release of Grok.

Upgrading to a new Grok release
-------------------------------

Whenever an application developer is ready to upgrade his project to
a new version of Grok, he only needs to change the URL that points to
the versions file to the newest versions file::

   [buildout]
   extends = http://grok.zope.org/releaseinfo/grok-0.99.cfg
   versions = versions
   ....

Then, he needs to run his buildout command again to make it install
the eggs that are now specified for this latest release of Grok.

"Fixing up" current grokproject-made projects
---------------------------------------------

Projects that have been created with the current grokproject command
can be "fixed up" by:

   1) removing the `newest = false` line from the project's
      `buildout.cfg`.

   2) adding the following two lines to the `[buildout]` section::

        [buildout]
        ....
        extends = http://grok.zope.org/releaseinfo/grok-0.10.1.cfg
        versions = versions

and then running the buildout command again.

**NOTE**: this "fix up method" will work when Grok 0.10.1 has been
released. If you want to try anyway, you could try using this URL:
http://grok.zope.org/releaseinfo/grok-0.10.1b1.cfg


Next steps:

* Martijn and I have been working on a versions file with as much
   final releases (instead of alphas and betas) in it as possible.
   This needs to be finalized and tested on Windows. We can probably
   do this on monday.

* Release 0.10.1 and people using Grok-0.10 should then be able
   to use 0.10.1's versions list to "fix up" their projects.

* Review the jw-luciano-use-versions-file branch, and release
   grokproject.

* Tweak the versions file for the grok trunk and release
   Grok-0.11.


And of course, we're curious for you feedback!


kind regards,
jw

ps. I'd like to thank Luciano for pairing-up during the sprint!
     It has been a pleasure to work together on this!