[Zope-dev] why external version indexes don't fulfill all use cases for development

[Jim Fulton]

On Nov 11, 2007, at 2:06 AM, Martijn Faassen wrote:

> Hi there,
>
> I've been doing some more thinking about external version indexes  
> (like Grok's versions.cfg on a URL, and like KGS) and why they  
> won't solve all our problems. I have a new way to express it, so  
> let me try it out on you all.
>
> What KGS solves is that it allows the ongoing development and  
> testing of an integrated Zope 3.

I see it addressing a more general problem of having a known good  
combination of components that work together. There's nothing Zope 3  
specific about this.

> That is, there's a Zope 3 'trunk' of versions that keeps being  
> updated as there are bugfix releases.

That's not how I see it. As I've said before, I would model this on  
linux distributions, where each feature release has a repository of  
packages for that release, including bug fixes.

> I'm not sure what happens as soon as someone wants to make a new  
> feature release of any package.

They make a new release.  At some point, someone will make a new KGS  
that incorporates this.

> Presumably they end up in KGS too. After all, we won't have a  
> single Zope 3.4 and then a single Zope 3.5 for which we can create  
> a new KGS.

Why not?  I would expect that there would be Zope 3.4 and Zope 3.5  
KGSs.  There might be additional KGSs that include some of the same  
components. Anyone can assemble a KGS if they think that in doing so,  
they can add value.


> We intend to let packages move at different feature-release speeds,  
> and we can't have a KGS for each package.

Of course not.

> Another problem KGS can solve is to add some release hygiene to the  
> cheeseshop: do not remove old releases or overwrite them.

I don't really understand this.  Maybe you mean that a KGS can be a  
better alternative to the cheeseshop. I can certainly see that.

> What KGS doesn't have is history. When I release an application or  
> framework and I used KGS to make sure that all my versions were  
> correct, it will work on the day of release. As soon as enough  
> bugfix (or feature) releases make it to KGS, something will  
> inevitably break. We've seen innocuous changes breaking code a lot  
> of times, so we can't pretend that never happens. It *will* happen.

Yup. Which is why you should record versions you use.

> This breaks a fundamental assumption for releases. When I release  
> something, I expect it to work tomorrow, next month, and next year.

If you want this, then you can't rely on the KGS.  When releasing our  
applications, we don't rely on a KGS. We fix all of the versions  
we're using.  IMO, the KGS shouldn't try to solve this problem.  A  
KGS should be helpful for developers and development frameworks.  A  
KGS will be more useful if the quality remains high.   A KGS is  
similar to a traditional monolithic release.  After all, bug fix Zope  
releases have been known to break applications too.

> With code, we know that history, and branches, and so on, are  
> important. We use Subversion. With KGS we only have an ongoing trunk.

I'm not sure why you keep saying "trunk".  I'm not sure if you are  
being imprecise, or if I'm missing something.
There's no reason a KGS couldn't be managed with a revision control  
system. That might be a very good idea.

> With Grok, we use an external versions list. We can use this to  
> solve the above problem. We basically take snapshots of what is in  
> KGS. This allows us to maintain some history, though it isn't ideal  
> either, as it's quite a bit of overhead.

Yup.  I think both KGSs and version lists are valid approaches.  Each  
has different strengths and weaknesses.

> If I build an application or framework on top of Grok, I will need  
> to maintain yet another external list for the extra packages of  
> this application, fixing those versions. We could probably even use  
> the extends feature of buildout to have this list point at Grok's  
> list so we have to repeat ourselves less should we want to build  
> something on top of *that* application or framework again.

Yup.

> So, while annoying, that is somewhat manageable. Now imagine I want  
> to use a completely separate Python library with my Grok  
> application. This python library has dependencies itself again.  
> This means I will need to know about versions of those dependencies  
> as well, and fix them into my application's list.

Yes

> There are some fundamental problems with external lists or indexes:
>
> * we need to know about the dependency of dependencies, even if we  
> never use them directly. Information hiding is broken.

I'm not sure how this is a problem with version lists (external or  
otherwise) or indexes.


>
> * a single list will never do it. We intend to have many different  
> applications that may depend on different versions of packages.  
> Grok may need a newer zope.publication than your application does.  
> A Grok extension may need an even newer version than Grok does.  
> We'll be baking endless amounts of lists this way.

I think each application will need to come up with a version list for  
each of it's releases.  In development, an application can use an  
index or external version list as a starting point.  For example, I  
see a KGS being useful as a (fairly) stable baseline for  
development.  When an application is ready for release, it should fix  
it's versions.  I've tried to make this easy to do with buildout.   
When you're preparing to make a release, run buildout in verbose mode  
(-v) It will print out the versions it picked in a format that is  
easily turned into a version list.


> If this information is inside the packages itself, the history will  
> be automatically maintained with Subversion and existing releases.  
> History therefore works: if I install Grok 0.11, I would get all  
> dependencies of Grok 0.11 automatically without having to worry  
> about external indexes.

Applications should maintain version lists.  Frameworks are another  
matter.  If you fix the versions in Grok, then it will harder for  
people to get bug fixes to packages you depend on.  There's a trade  
off here of stability and flexibility.

> Information hiding works: if I use foo 1.3 and foo 1.3 knows it  
> needs bar 1.7, it'll simply get that and I don't have to know about  
> it. I don't even need to worry about the *existence* of bar.

Yup.

> People have been saying that since Linux distributions use external  
> indexes, we should too, as we are dealing with the same problem as  
> Linux distributions. While the problem is similar, I think the  
> nature of development makes our problems, and therefore our  
> solutions, quite different from the way distributions do it.
>
> How are we different?
>
> We have many, many different small distributions (package +  
> dependencies) that can be combined. We have such a small  
> distribution for each application. We have such a small  
> distribution for each extension. Not just that. We have such a  
> small distribution for each *release* of an application. We have  
> such a small distribution for each *release* of an extension.

If this dependency information is in an easily override-able form,  
then I think this is a sane strategy.

No one is advocating a KGS for each application.  A KGS is merely an  
attempt to provide a stable ecosystem of components that work  
together.  Just like Linux is a platform for delivering applications,  
a KGS is a platform for building applications.

> I therefore still believe that version dependency information  
> should move out of external indexes and into packages.

IMO, I think a KGS can provider a useful baseline for developers.   
Applications should also fix versions used for specific releases.

Jim

--
Jim Fulton
Zope Corporation