Change in repository approach to software sharing
The Zope project includes a number of interrelated subprojects, such as: - Zope 2 - Zope 3 - ZODB - ZConfig Software from the ZODB and ZConfig projects are shared by Zope 2 and Zope 3. We want this sharing to be very convenient for people working on Zope 2 and Zope 3. We don't want users of the Zope 2 and Zope 3 repository to have to do separate checkouts for ZODB and ZConfig. CVS supports such software sharing through it's "module" system. The module system has some flaws, so we use symbolic links instead. In a response to: http://dev.zope.org/Zope3/MovingSCMToSubversion John Belmonte has suggested that Zope 2 and Zope 3 should depend on specific versions of shared packages, rather than on the head. I'm inclined to agree. People working on ZODB and ZConfig have to test their changes against both Zope 2 and Zope 3 to avoid breakage. This is very burdensome and causes much pain when they get it wrong. Fortunately, subversion provides a mechanism for sharing specific revisions. We'll be able to have the convenience of getting ZODB and ZConfig (and other shared software) when we do a checkout *and* we'll be able to control what parts we get. To see how this will work, we'll look at ZConfig as an example (because it has a single package) of reusable software that we will include in Zope 3. In the repository, we'll have a top-level Zope3 project directory, with the standard svn subdirectories "trunk", "branches", and "tags". We'll also have a top-level ZConfig project directory. The "trunk" of the ZConfig Python package will be in ZConfig/trunk/src/ZConfig. If we create a tag T1 of ZConfig, then the Python package for that tag will be in ZConfig/tags/T1/src/ZConfig. Now, when we set up the Zope 3 repository, we will create the ZConfig package in Zope 3 by copying a *tag* from the ZConfig project: svn copy svn+ssh://svn.zope.org/repos/ZConfig/tags/T1/src/ZConfig \ svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig -m 'Bring ZConfig T1 into main branch' Then, whenever someone checks out the Z3 trunk, they'll get the ZConfig from T1. If we need to, we can even make Zope3-local changes to ZConfig. Later, we may decide to upgrade the Zope 3 head to use ZConfig tag 3. At that point, we can recopy from the tag, or we can merge changes made between the two tags. A similar technique can be used for external software, as described in: http://svnbook.red-bean.com/svnbook/ch07s04.html Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
[Jim Fulton]
The Zope project includes a number of interrelated subprojects, such as:
- Zope 2
- Zope 3
- ZODB
- ZConfig
Software from the ZODB and ZConfig projects are shared by Zope 2 and Zope 3.
Note that ZODB also depends on ZConfig.
We want this sharing to be very convenient for people working on Zope 2 and Zope 3. We don't want users of the Zope 2 and Zope 3 repository to have to do separate checkouts for ZODB and ZConfig. CVS supports such software sharing through it's "module" system. The module system has some flaws, so we use symbolic links instead.
In a response to:
http://dev.zope.org/Zope3/MovingSCMToSubversion
John Belmonte has suggested that Zope 2 and Zope 3 should depend on specific versions of shared packages, rather than on the head. I'm inclined to agree.
Didn't we endure a lot of pain over the last year to get away from that model? At one point there were 7 lines of ZODB I was wrestling with every day. For example, there was the ZODB 3.1 branch, which "in theory" corresponded to the Zope 2.6 branch, but in practice never matched it. Life got much better (for both Zope 2.6 maintenance and ZODB 3.1 maintenance) when we abandoned the ZODB3-3_1-branch branch of ZODB3 and did all ZODB 3.1 maintenance directly on the Zope-2_6-branch branch of Zope. Likewise for ZODB3-3_2-branch versus Zope-2_7-branch: the former was abandoned, and life got better for everyone because of that. Currently the HEAD of the ZODB module is the same as the HEAD view of ZODB when viewed from the Zope and Zope3 modules, which isn't a pure win, but isn't a pure loss either.
People working on ZODB and ZConfig have to test their changes against both Zope 2 and Zope 3 to avoid breakage.
I don't see that this part can change, no matter how names are shuffled.
This is very burdensome
That part either -- all the tests in all contexts still need to be run.
and causes much pain when they get it wrong.
Ditto, although you're (re)introducing mechanism that alleviates this at the cost of increasing the number of ZODB snapshots in active use.
Fortunately, subversion provides a mechanism for sharing specific revisions. We'll be able to have the convenience of getting ZODB and ZConfig (and other shared software) when we do a checkout *and* we'll be able to control what parts we get.
To see how this will work, we'll look at ZConfig as an example (because it has a single package) of reusable software that we will include in Zope 3.
In the repository, we'll have a top-level Zope3 project directory, with the standard svn subdirectories "trunk", "branches", and "tags".
We'll also have a top-level ZConfig project directory. The "trunk" of the ZConfig Python package will be in ZConfig/trunk/src/ZConfig. If we create a tag T1 of ZConfig, then the Python package for that tag will be in ZConfig/tags/T1/src/ZConfig.
Now, when we set up the Zope 3 repository, we will create the ZConfig package in Zope 3 by copying a *tag* from the ZConfig project:
svn copy svn+ssh://svn.zope.org/repos/ZConfig/tags/T1/src/ZConfig \ svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig -m 'Bring ZConfig T1 into main branch'
Then, whenever someone checks out the Z3 trunk, they'll get the ZConfig from T1.
I don't yet know enough about svn to guess how the next step works: since ZODB all by itself depends on ZConfig, presumably similar stitching of ZConfig will take place in the top-level ZODB project. Now when we go on to stitch ZODB into Z3, how will ZODB's attempt to stitch in its own version of ZConfig play with Z3's attempt to stitch in T1 above? I'm not picturing how this works. Maybe it's simple!
If we need to, we can even make Zope3-local changes to ZConfig.
That would bloat ZConfig maintenance headaches exponentially. I'm confident of that, because of ZODB experience with (at least) 7 concurrently active ZODB snapshots -- it's simply intractable to keep straight what should and shouldn't get cross-ported across this set; mistakes and oversights are as much the norm as the exception then; the non-ZODB people doing stitching of ZODB into their projects won't be confident about exactly which tag they should stitch in; expediency will cause them to stitch in "whatever seems to work" at the time; and then the latter becomes an active snapshot that needs to be maintained too. Don't make Zope3-local changes to ZConfig or ZODB, though, and at least that nest of rats can be sidestepped.
Later, we may decide to upgrade the Zope 3 head to use ZConfig tag 3.
Ya, and the Zope head may decide to use ZConfig tag 44, while the Zope 2.8 maintenance branch decides to use ZConfig tag 37 despite that ZConfig 3.3.1 (which is supposed to go out with the next Zope 2.8 release) is actually released at tag 39. The dark side of "flexibility" is that it creates that many more ways to get out of synch.
At that point, we can recopy from the tag, or we can merge changes made between the two tags.
Right now, the burden for making ZConfig changes "work" is entirely on Fred, because the changes he makes show up everywhere at once. Similarly for ZODB versus me. The "we" in the above is muddy: whose responsibility will it be to recopy/merge? A clear answer to that may make it work better. As a counterexample, it was never clear whose responsibility(ies) it was to coordinate ZODB versions with Zope(2) releases, and before we eliminated the distinct brances for ZODB and Zope, the version of ZODB shipped with a given Zope release never corresponded with a ZODB release. It's only in the last year that we managed to create ZODB releases that truly matched Zope releases. Then again, there was a lot of hosage in those processes <0.5 wink>.
Tim Peters wrote:
[Jim Fulton]
The Zope project includes a number of interrelated subprojects, such as:
- Zope 2
- Zope 3
- ZODB
- ZConfig
Software from the ZODB and ZConfig projects are shared by Zope 2 and Zope 3.
Note that ZODB also depends on ZConfig.
Yup and zdaemon and zLOG
We want this sharing to be very convenient for people working on Zope 2 and Zope 3. We don't want users of the Zope 2 and Zope 3 repository to have to do separate checkouts for ZODB and ZConfig. CVS supports such software sharing through it's "module" system. The module system has some flaws, so we use symbolic links instead.
In a response to:
http://dev.zope.org/Zope3/MovingSCMToSubversion
John Belmonte has suggested that Zope 2 and Zope 3 should depend on specific versions of shared packages, rather than on the head. I'm inclined to agree.
Didn't we endure a lot of pain over the last year to get away from that model? At one point there were 7 lines of ZODB I was wrestling with every day. For example, there was the ZODB 3.1 branch, which "in theory" corresponded to the Zope 2.6 branch, but in practice never matched it. Life got much better (for both Zope 2.6 maintenance and ZODB 3.1 maintenance) when we abandoned the ZODB3-3_1-branch branch of ZODB3 and did all ZODB 3.1 maintenance directly on the Zope-2_6-branch branch of Zope. Likewise for ZODB3-3_2-branch versus Zope-2_7-branch: the former was abandoned, and life got better for everyone because of that. Currently the HEAD of the ZODB module is the same as the HEAD view of ZODB when viewed from the Zope and Zope3 modules, which isn't a pure win, but isn't a pure loss either.
I think that subversion will make it easier to manage the changes. Periodically, changes will be merged into Zope 2 and Zope 3. This should be straightforward.
People working on ZODB and ZConfig have to test their changes against both Zope 2 and Zope 3 to avoid breakage.
I don't see that this part can change, no matter how names are shuffled.
ZODB needs to be tested with Zope 2 and Zope 3 whenever updates are merged into those projects. Of course, you'll probably want to test more frequently, but you don't have to test every change you make.
This is very burdensome
That part either -- all the tests in all contexts still need to be run.
But not for every change.
and causes much pain when they get it wrong.
Ditto, although you're (re)introducing mechanism that alleviates this at the cost of increasing the number of ZODB snapshots in active use.
Fortunately, subversion provides a mechanism for sharing specific revisions. We'll be able to have the convenience of getting ZODB and ZConfig (and other shared software) when we do a checkout *and* we'll be able to control what parts we get.
To see how this will work, we'll look at ZConfig as an example (because it has a single package) of reusable software that we will include in Zope 3.
In the repository, we'll have a top-level Zope3 project directory, with the standard svn subdirectories "trunk", "branches", and "tags".
We'll also have a top-level ZConfig project directory. The "trunk" of the ZConfig Python package will be in ZConfig/trunk/src/ZConfig. If we create a tag T1 of ZConfig, then the Python package for that tag will be in ZConfig/tags/T1/src/ZConfig.
Now, when we set up the Zope 3 repository, we will create the ZConfig package in Zope 3 by copying a *tag* from the ZConfig project:
svn copy svn+ssh://svn.zope.org/repos/ZConfig/tags/T1/src/ZConfig \ svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig -m 'Bring ZConfig T1 into main branch'
Then, whenever someone checks out the Z3 trunk, they'll get the ZConfig from T1.
I don't yet know enough about svn to guess how the next step works: since ZODB all by itself depends on ZConfig, presumably similar stitching of ZConfig will take place in the top-level ZODB project.
Yup
Now when we go on to stitch ZODB into Z3, how will ZODB's attempt to stitch in its own version of ZConfig play with Z3's attempt to stitch in T1 above? I'm not picturing how this works. Maybe it's simple!
You won't stich in ZODB's copy of ZConfig. You'll need to work with the one used in Zope.
If we need to, we can even make Zope3-local changes to ZConfig.
That would bloat ZConfig maintenance headaches exponentially. I'm confident of that, because of ZODB experience with (at least) 7 concurrently active ZODB snapshots -- it's simply intractable to keep straight what should and shouldn't get cross-ported across this set; mistakes and oversights are as much the norm as the exception then; the non-ZODB people doing stitching of ZODB into their projects won't be confident about exactly which tag they should stitch in; expediency will cause them to stitch in "whatever seems to work" at the time; and then the latter becomes an active snapshot that needs to be maintained too.
Don't make Zope3-local changes to ZConfig or ZODB, though, and at least that nest of rats can be sidestepped.
We can decide to do that. We can even recopy (rather than merge) when we sync up. The important thing to realize is that we won't *maintain* ZODB in Zope 3 as such. Rather, we'll periodically sync up ZODB changes since the last sync.
Later, we may decide to upgrade the Zope 3 head to use ZConfig tag 3.
Ya, and the Zope head may decide to use ZConfig tag 44, while the Zope 2.8 maintenance branch decides to use ZConfig tag 37 despite that ZConfig 3.3.1 (which is supposed to go out with the next Zope 2.8 release) is actually released at tag 39. The dark side of "flexibility" is that it creates that many more ways to get out of synch.
At that point, we can recopy from the tag, or we can merge changes made between the two tags.
Right now, the burden for making ZConfig changes "work" is entirely on Fred, because the changes he makes show up everywhere at once. Similarly for ZODB versus me. The "we" in the above is muddy: whose responsibility will it be to recopy/merge?
The person using the shared code. For example, whoever merges ZODB updates into Zope 3 will need to make sure they work.
A clear answer to that may make it work better. As a counterexample, it was never clear whose responsibility(ies) it was to coordinate ZODB versions with Zope(2) releases, and before we eliminated the distinct brances for ZODB and Zope, the version of ZODB shipped with a given Zope release never corresponded with a ZODB release. It's only in the last year that we managed to create ZODB releases that truly matched Zope releases. Then again, there was a lot of hosage in those processes <0.5 wink>.
The goal is for Zope to depend on specific releases of ZODB. This should work a bit like vendor imports, but subversion should make it a good bit more convenient. If the new approach turns out not to work, we'll try something else. :) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
We'll also have a top-level ZConfig project directory. The "trunk" of the ZConfig Python package will be in ZConfig/trunk/src/ZConfig. If we create a tag T1 of ZConfig, then the Python package for that tag will be in ZConfig/tags/T1/src/ZConfig.
The /src/ in the above paths seems like a dead chicken to me, why's it there?
Now, when we set up the Zope 3 repository, we will create the ZConfig package in Zope 3 by copying a *tag* from the ZConfig project:
svn copy svn+ssh://svn.zope.org/repos/ZConfig/tags/T1/src/ZConfig \ svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig -m 'Bring ZConfig T1 into main branch'
I think you'll find this doesn't work when svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig already exists. It's the one thing that's disappointed me about SVN so far :-( (well, that and not beign able t odo inter-repository operations ;-)
Later, we may decide to upgrade the Zope 3 head to use ZConfig tag 3. At that point, we can recopy from the tag, or we can merge changes made between the two tags.
My experience has been that it's best just to have shared software "exist" in each location where it's used and maintained in those places with new versions being brought across via merges. Merges in SVN have ended up feeling a LOT nicer for me that with CVS, but the key is still to keep the number of maintained locations of a piece of shared software to an absolute minimum, as Tim points out. Another option, and one which I'd really strongly suggest here if it could be made to work is the "external definitions" stuff: http://svnbook.red-bean.com/svnbook/ch07s03.html The svn:externals way of doing things means you only develop the shared software it it's own tag/trunk/branches structure and just link to it from where you want to use it, which sounds like it might save a few grey hairs on Tim's part since there won't be any "local development" in the locations where the shared software is used... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Jim Fulton wrote:
...
Now, when we set up the Zope 3 repository, we will create the ZConfig package in Zope 3 by copying a *tag* from the ZConfig project:
svn copy svn+ssh://svn.zope.org/repos/ZConfig/tags/T1/src/ZConfig \ svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig -m 'Bring ZConfig T1 into main branch'
I think you'll find this doesn't work when svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig already exists.
That's not relevent to the example, as, in the example, we are creating it for the first time.
Later, we may decide to upgrade the Zope 3 head to use ZConfig tag 3. At that point, we can recopy from the tag,
Do do so, we may have to remove the old copy first.
or we can merge changes made between the two tags.
My experience has been that it's best just to have shared software "exist" in each location where it's used and maintained in those places with new versions being brought across via merges.
That's what I'm suggesting.
Merges in SVN have ended up feeling a LOT nicer for me that with CVS, but the key is still to keep the number of maintained locations of a piece of shared software to an absolute minimum, as Tim points out.
There should be only one place that shared software is maintained. Periodically, we'll need to resync copies.
Another option, and one which I'd really strongly suggest here if it could be made to work is the "external definitions" stuff: http://svnbook.red-bean.com/svnbook/ch07s03.html
That's close to what we have now, with repolinks. It's cleaner that repolinks, but it has the same basic flaw, the shared software is *truly* shared: - Users of the software have no control over when they get changes to it - A use of the software can make what they might think is a local change, check it in, and break other users. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
I think you'll find this doesn't work when svn+ssh://svn.zope.org/repos/Zope3/trunk/src/ZConfig already exists.
That's not relevent to the example, as, in the example, we are creating it for the first time.
Yep, I know, but I've been there, and you feel great after the first time ,and disappointed when it doesn't work again ;-)
Do do so, we may have to remove the old copy first.
Yeah, which means two operations for each copy :-S
My experience has been that it's best just to have shared software "exist" in each location where it's used and maintained in those places with new versions being brought across via merges.
That's what I'm suggesting.
...hence just doing the merge which is the desired effect anyway ;-)
Another option, and one which I'd really strongly suggest here if it could be made to work is the "external definitions" stuff: http://svnbook.red-bean.com/svnbook/ch07s03.html
That's close to what we have now, with repolinks. It's cleaner that repolinks, but it has the same basic flaw, the shared software is *truly* shared:
- Users of the software have no control over when they get changes to it
I may have misread the external definitions docs, but I'm pretty sure you specify a revision number in the property, so you only ever get the exact version you want, and hence changes when you want them. Or did I miss something?
- A use of the software can make what they might think is a local change, check it in, and break other users.
Indeed. But since "other users" will stick with their specific revision, this should be okay, and it's better that such breakages get found sooner, rather than having several gradually forking copies of the same code (I almost ran into this problem with some custom products used in multiple SVN'ed instance-home's before slapping myself very hard...) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
participants (3)
-
Chris Withers -
Jim Fulton -
Tim Peters