[Zope-dev] why external version indexes don't fulfill all use cases for development

[Martijn Faassen]

Hey,

On Nov 12, 2007 12:02 AM, Stephan Richter <srichter at cosmos.phy.tufts.edu> wrote:
[snip]
> Like Linux distributions, there will be a KGS for every Zope 3
> release. I have already requested a new directory called "zope-dev" where new
> feature releases can be tested.

Okay, I didn't understand that KGS is replacing the monolithic release
story for Zope 3. That's fine as far as it goes. I was focused on the
ability that eggs give us for packages to move at different speeds of
evolution, and the desire to pick those eggs that we prefer in our
applications. If you don't need that, then KGS is basically Zope 3
release + a few features.

[snip]
> > We intend to let packages move at different feature-release speeds,
> > and we can't have a KGS for each package.
>
> You do not need to have a single KGS for every package. But believing that we
> can just randomly make new feature releases that work with the rest of the
> world is naive at best. We have seen already what happens, if everyone uses
> their own set of versions and packages.

Clearly things didn't work in the past. You can't just throw random
versions of eggs together. We couldn't do anything better, as the
information about what worked together was missing. KGS adds that
information back, and that's great. But the information is external to
the actual packages. This has drawbacks. I'm saying that if we add
this information in the packages internally, we'll be better off, as
historical versions and future versions can work. The reuse story is
improved. You can make your own selection of versions and have a
decent chance it will work together.

> A development KGS will be used to test new feature releases.
>
> > What KGS doesn't have is history.
>
> Yes, it does. Why do you think I manage the "controlled-packages.cfg" file in
> SVN? And in SVN, I do not create branches and tags without a reason.

Okay, that's a history. It's a history external to the packages, while
the packages have their own history. It's also a global history, while
packages can evolve independently. Development decisions of a
package's development can change dependency information. It therefore
seems natural to me that this information is maintained next to the
package. If it's not, and zope.component starts to rely on a newer
version of zope.interface, I'd need to maintain this centrally with
KGS. We introduce a new monolithic structure where we just removed it.
We add back explicitly what was there implicitly: an SVN trunk of Zope
3 maintaining versions that all work together. That's fine to retain
the features Zope 3 development had, but I thought the point of
splitting Zope 3 up was to be able to forget about the SVN trunk of
Zope 3 and just worry about what's right for zope.component.

[snip]
> > With Grok, we use an external versions list. We can use this to solve
> > the above problem. We basically take snapshots of what is in KGS. This
> > allows us to maintain some history, though it isn't ideal either, as
> > it's quite a bit of overhead.
>
> How is this overhead?

Besides releasing Grok, we also need to maintain snapshots of what is
in KGS, make such changes as are needed, and publish them. Previously
we just released new versions of Grok. That's increased maintenance
and release overhead I'd like to get rid of again.

> > If I build an application or framework on top of Grok, I will need to
> > maintain yet another external list for the extra packages of this
> > application, fixing those versions.
>
> Why? I don't follow that?

Because these packages may be of different versions that in KGS, or
may not be managed by KGS altogether.

> > So, while annoying, that is somewhat manageable. Now imagine I want to
> > use a completely separate Python library with my Grok application. This
> > python library has dependencies itself again. This means I will need to
> > know about versions of those dependencies as well, and fix them into my
> > application's list.
>
> Yes. I see this as an advantage. Version specifications in `setup.py` usually
> contain ranges of allowed versions. What happens if one release in the range
> does not work? Then you make false promises. The only way to avoid this would
> be by specifying all allowed versions exactly, which makes no sense.

That's true. Where I'd like to specify this is as near as possible to
where I make the decision to fix these versions. In case of an
application, that may be the application. Often that's not the case
though: in the case of a library that uses these packages, I'd like it
to be the library, and in case of a framework, I'd like it to be the
framework. When I develop an application at most I'd like to get the
warning: these packages still don't have fixed versions. I'd prefer
that list to be empty.

> > There are some fundamental problems with external lists or indexes:
> >
> > * we need to know about the dependency of dependencies, even if we never
> > use them directly. Information hiding is broken.
>
> This is a requirement not a problem statement. I don't understand "Information
> hiding is broken." I cannot see how "information hiding" could be a good
> thing here.

Sure, it's my requirement. I think it's important if you want
beginners to be able to figure out what they're supposed to be doing.
I think it's important for agile development, too. Feel free to ignore
it entirely.

> > * a single list will never do it. We intend to have many different
> > applications that may depend on different versions of packages. Grok may
> > need a newer zope.publication than your application does. A Grok
> > extension may need an even newer version than Grok does. We'll be baking
> > endless amounts of lists this way.
>
> I never claimed that we will have just one list.
>
> Your assumption is that there will be an endless amount of working sets and
> that they are easy to find. While theoretically possible, this will not
> happen. I think the events of the past two months have shown that the
> opposite happens and it is actually hard to find working sets, once
> development digresses too much.

The events since july (which is when I started to have to deal with
this problem) show that you need this information *somewhere*. The
information was entirely missing! You can't use this as evidence that
the only solution is to maintain lists external to the packages.

[snip]
> > If this information is inside the packages itself, the history will be
> > automatically maintained with Subversion and existing releases. History
> > therefore works: if I install Grok 0.11, I would get all dependencies of
> > Grok 0.11 automatically without having to worry about external indexes.
>
> This is the overly simplistic world view that we had about two months ago.

Stephan, that's an overly simplistic description of history. In fact I
have been thinking about this problem for a while now. A month and a
half ago I wrote this:

http://faassen.n--tree.net/blog/view/weblog/2007/09/26/0

> Because we are all developing on top of different stacks and people expect to
> pick and choose, having a common foundation is the only possible way.

> Let me give you a dire scenario.
[snip dire scenario]

* eggs should state the minimum requirement (I'm C. I depend on B, or
I depend on B, newer than version 3) and a suggestion (I depend on
version 3.1. I know that, at least, works)

* eggs should be able to override suggestions down below in the
hierarchy. I depend on C, but I want B version 3.2. Forget what C
suggested.

* eggs shouldn't be able to say incompatible things. So you can't say,
I depend on C, but I want B earlier than version 3, if C already says
it needs version 3 or newer.

> Let's say you have a package A-1.0.0. You also have a package B-1.0.0 that
> depends on A. You suggest fixing versions, so you would write in the dep list
> of B: 'A => 1.0.0'.

> You now release a new feature version of A, A-1.1.0, that
> is incompatible with B-1.0.0. So package B-1.0.0 will be broken until you
> release B-1.0.1 that states 'A >= 1.0.0 and A <= 1.0.99'.

> This problem here is that you have to re-release B only for the sake of
> changing the version requirement.

It doesn't exist if you do what I suggested above. It requires some
changes to setuptools, which I have proposed a month and a half ago.

[snip]
> One promise Jim and others made to me about eggs was that once we had a stable
> set (which I worked on for the current KGS), releasing would become much
> simpler, because many packages would not need new releases for very long
> stretches at a time. With your suggestion, packages will need to be released
> all the time.

Luckily that was never my suggestion.

Regards,

Martijn