[Zope-dev] December release post-mortem

[Jim Fulton]

Now that I've had a week or so to recover from making the Zope 3
releases, I'd like to look at how we did on our first timed releases.

Of course, the releases didn't happen in December.  In fact, the Zope 2
Windows release still hasn't happened.

That we were late isn't a great surprise, given that this was
our first timed release.  The beginning of the release cycle was delayed
because people didn't really realize/remember that we needed to freeze by
November 1.

The release was delayed past December in large part due to problems
found with the Zope 3 Twisted server.  In retrospect, I should have accepted
these, made ZServer the default and marked the twisted server experimental.
Instead, I spent 2 weeks thinking the resolution only needed a few more hours
of work.  I should have known better.  I'm disappointed that the people
pushing the Twisted server didn't provide more help during this period, but
it was a holiday time and one of the main drivers, Stephan, warned way
in advance that we shouldn't be finishing a release during a holiday period.
Of course, we should have been finishing the release in early December.

In the future, if someone introduces a major change, they *must* be
committed to be available to deal with issues that arise during the
release cycle.  Perhaps we need to pick different release dates to
avoid holidays.  Stephan has suggested moving the dates up to
November/May rather than December/June.

And then there are the Windows releases.  Making Zope 2 windows releases
is very painful and there don't seem to be many people willing to help.
We've avoided the pain for Zope 3 by being less ambitious.  We let distutils
do most of the work.  The result is that making a windows release takes minutes
and is highly automated, but the experience for the end user is less than
ideal,  Many would rightfully argue that it is inadequate.  What we need
is a release process that is as easy as the Zope 3 windows release process
and produces a result as usable as the Zope 2 windows release.  I'm not sure
exactly what the answer is, but I am sure we need to take a fresh approach.
Whatever approach we take needs to be highly automated and must not require
a lot of specialized Windows expertise.

I think that packaging is a significant challenge to our release process.
The Zope 2 release was complicated by the use of zpkg.  The zpkg system
was developed to allow us to decouple the Zope 3 repository and releases.
It was too painful to mold the repository to fleeting release decisions.
We wanted people to put experimental and add-on applications in the
repository so that they would be tested as we rapidly evolved things.  While
zpkg was crucial for the last 3 releases, I don't think it provides the best
long-term approach.  Rather than extracting a release from a larger repository
tree, I think it would be better, going forward, to assemble a release from
smaller individual repository trees or from releases.  This is feasible because:

- Python is finally growing a packaging system with eggs,

- buildbot provides a mechanism for getting things tested without
   putting everything in a single repository, and

- the new test runner's layer makes it feasible to define independent
   functional tests for packages.

We have begun to see Zope 3 decomposed into separate projects knit together
via svn:externals.  I'd like to see that trend continue and to perhaps
switch to making the knitting process use eggs,  I'd like to leverage
eggs to make the release process much simpler.  It should be easy for people
to make releases so that there could easily be multiple distributions
aimed at different needs and expectations.

As part of this decomposition, we need to make zope.app much smaller.
Early in Zope 3 development, I proposed a simple rule for organization of
the repository: Anything that depended on zope.app went in zope.app. Anything
else went in zope.  If there was doubt, put it in zope.app.  I think that
served us well at the time, but I think we've moved beyond that,  I'd like
to migrate most of zope.app elsewhere.  Zope 2.10 should not include zope.app.

Note that I'm banking heavily on eggs without personally having worked with
them much.  I'm very hopeful that we can make them work for us.

In the end, I consider the "December" release to be largely successful, given
the challenges.

These were some of my reactions to this first attempt at time-based releases.
What do other folks think?

Given that the Zope Foudation will be formed during this next release cycle,
I think this is a good time to take stock and think about how to move
forward,

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org