Now that I've had a week or so to recover from making the Zope 3 releases, I'd like to look at how we did on our first timed releases. Of course, the releases didn't happen in December. In fact, the Zope 2 Windows release still hasn't happened. That we were late isn't a great surprise, given that this was our first timed release. The beginning of the release cycle was delayed because people didn't really realize/remember that we needed to freeze by November 1. The release was delayed past December in large part due to problems found with the Zope 3 Twisted server. In retrospect, I should have accepted these, made ZServer the default and marked the twisted server experimental. Instead, I spent 2 weeks thinking the resolution only needed a few more hours of work. I should have known better. I'm disappointed that the people pushing the Twisted server didn't provide more help during this period, but it was a holiday time and one of the main drivers, Stephan, warned way in advance that we shouldn't be finishing a release during a holiday period. Of course, we should have been finishing the release in early December. In the future, if someone introduces a major change, they *must* be committed to be available to deal with issues that arise during the release cycle. Perhaps we need to pick different release dates to avoid holidays. Stephan has suggested moving the dates up to November/May rather than December/June. And then there are the Windows releases. Making Zope 2 windows releases is very painful and there don't seem to be many people willing to help. We've avoided the pain for Zope 3 by being less ambitious. We let distutils do most of the work. The result is that making a windows release takes minutes and is highly automated, but the experience for the end user is less than ideal, Many would rightfully argue that it is inadequate. What we need is a release process that is as easy as the Zope 3 windows release process and produces a result as usable as the Zope 2 windows release. I'm not sure exactly what the answer is, but I am sure we need to take a fresh approach. Whatever approach we take needs to be highly automated and must not require a lot of specialized Windows expertise. I think that packaging is a significant challenge to our release process. The Zope 2 release was complicated by the use of zpkg. The zpkg system was developed to allow us to decouple the Zope 3 repository and releases. It was too painful to mold the repository to fleeting release decisions. We wanted people to put experimental and add-on applications in the repository so that they would be tested as we rapidly evolved things. While zpkg was crucial for the last 3 releases, I don't think it provides the best long-term approach. Rather than extracting a release from a larger repository tree, I think it would be better, going forward, to assemble a release from smaller individual repository trees or from releases. This is feasible because: - Python is finally growing a packaging system with eggs, - buildbot provides a mechanism for getting things tested without putting everything in a single repository, and - the new test runner's layer makes it feasible to define independent functional tests for packages. We have begun to see Zope 3 decomposed into separate projects knit together via svn:externals. I'd like to see that trend continue and to perhaps switch to making the knitting process use eggs, I'd like to leverage eggs to make the release process much simpler. It should be easy for people to make releases so that there could easily be multiple distributions aimed at different needs and expectations. As part of this decomposition, we need to make zope.app much smaller. Early in Zope 3 development, I proposed a simple rule for organization of the repository: Anything that depended on zope.app went in zope.app. Anything else went in zope. If there was doubt, put it in zope.app. I think that served us well at the time, but I think we've moved beyond that, I'd like to migrate most of zope.app elsewhere. Zope 2.10 should not include zope.app. Note that I'm banking heavily on eggs without personally having worked with them much. I'm very hopeful that we can make them work for us. In the end, I consider the "December" release to be largely successful, given the challenges. These were some of my reactions to this first attempt at time-based releases. What do other folks think? Given that the Zope Foudation will be formed during this next release cycle, I think this is a good time to take stock and think about how to move forward, Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Wed, Jan 18, 2006 at 07:36:35AM -0500, Jim Fulton wrote: | And then there are the Windows releases. Making Zope 2 windows releases | is very painful and there don't seem to be many people willing to help. | We've avoided the pain for Zope 3 by being less ambitious. We let distutils | do most of the work. The result is that making a windows release takes | minutes | and is highly automated, but the experience for the end user is less than | ideal, Many would rightfully argue that it is inadequate. What we need | is a release process that is as easy as the Zope 3 windows release process | and produces a result as usable as the Zope 2 windows release. I'm not sure | exactly what the answer is, but I am sure we need to take a fresh approach. | Whatever approach we take needs to be highly automated and must not require | a lot of specialized Windows expertise. The installers do not require much Windows expertise. In fact, they require a lot of 'makefile' expertise right now, and some Inno Setup expertise, not much else. At Enfold Systems we are using our own home-grown python-based process to cobble together all the dependencies for building installers. We haven't switched to Zope 2.9 though. When the time comes around for a switch, our goal is to switch from Inno Setup to Wix [1], at which point we hope to contribute this work to Zope. That might take another 6 months though and sure we don't want to hold back the Zope Windows installers that long. [1] http://blogs.msdn.com/robmen/archive/2004/04/05/107709.aspx -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com
Sidnei da Silva wrote:
On Wed, Jan 18, 2006 at 07:36:35AM -0500, Jim Fulton wrote: | And then there are the Windows releases. Making Zope 2 windows releases | is very painful and there don't seem to be many people willing to help. | We've avoided the pain for Zope 3 by being less ambitious. We let distutils | do most of the work. The result is that making a windows release takes | minutes | and is highly automated, but the experience for the end user is less than | ideal, Many would rightfully argue that it is inadequate. What we need | is a release process that is as easy as the Zope 3 windows release process | and produces a result as usable as the Zope 2 windows release. I'm not sure | exactly what the answer is, but I am sure we need to take a fresh approach. | Whatever approach we take needs to be highly automated and must not require | a lot of specialized Windows expertise.
The installers do not require much Windows expertise. In fact, they require a lot of 'makefile' expertise right now, and some Inno Setup expertise, not much else.
Sorry, Inno Setup is a windows installation builder. I consider this windows expertise.
At Enfold Systems we are using our own home-grown python-based process to cobble together all the dependencies for building installers. We haven't switched to Zope 2.9 though.
I consider switching build languages from make to Python a definate step forward.
When the time comes around for a switch, our goal is to switch from Inno Setup to Wix [1], at which point we hope to contribute this work to Zope. That might take another 6 months though and sure we don't want to hold back the Zope Windows installers that long.
Unless the process is automated enough that *I* can do it, whoever suggests a new system needs to be prepared to operate it reliably as well. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Wed, Jan 18, 2006 at 10:27:25AM -0500, Jim Fulton wrote: | >The installers do not require much Windows expertise. In fact, they | >require a lot of 'makefile' expertise right now, and some Inno Setup | >expertise, not much else. | | Sorry, Inno Setup is a windows installation builder. I consider this | windows expertise. You don't have to know how to operate Inno Setup for building a installer though. It pretty much boils down to editing the makefiles to contain the proper version number and typing 'make'. I would consider that automated enough. What about adding a target that does this into the buildbot so we get nightly installers? -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com
Sidnei da Silva wrote:
On Wed, Jan 18, 2006 at 10:27:25AM -0500, Jim Fulton wrote: | >The installers do not require much Windows expertise. In fact, they | >require a lot of 'makefile' expertise right now, and some Inno Setup | >expertise, not much else. | | Sorry, Inno Setup is a windows installation builder. I consider this | windows expertise.
You don't have to know how to operate Inno Setup for building a installer though. It pretty much boils down to editing the makefiles to contain the proper version number and typing 'make'. I would consider that automated enough. What about adding a target that does this into the buildbot so we get nightly installers?
People up to now have come up with systems like this that they thought were automated enough. That's why we don't have a 2.9 release for windows. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Wed, Jan 18, 2006 at 10:45:20AM -0500, Jim Fulton wrote: | People up to now have come up with systems like this that they thought were | automated enough. That's why we don't have a 2.9 release for windows. What about we turn that around. How would you describe a 'automated enough' build environment? I suspect you consider: python setup.py bdist_wininst to be pretty close to that. How does it differ from: make installer once all dependencies are in place? I agree that the procedure for building the current Windows installer, though documented (yes, it is documented), has more steps than required. One place where it could be streamlined is that it expects you to download the Python 2.3 Windows Installer and tarball manually and put them into a specific directory. That could certainly be done by the makefile. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com
Sidnei da Silva wrote:
On Wed, Jan 18, 2006 at 10:45:20AM -0500, Jim Fulton wrote: | People up to now have come up with systems like this that they thought were | automated enough. That's why we don't have a 2.9 release for windows.
What about we turn that around. How would you describe a 'automated enough' build environment? I suspect you consider:
python setup.py bdist_wininst
to be pretty close to that.
I think http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/ZopeWindows... Is pretty close. Note that this has a number os steps, but there are few and they are well documented, so I don't have to think.
How does it differ from:
make installer
It uses a real language.
once all dependencies are in place?
The process has to include getting al of the dependencies in place.
I agree that the procedure for building the current Windows installer, though documented (yes, it is documented), has more steps than required. One place where it could be streamlined is that it expects you to download the Python 2.3 Windows Installer and tarball manually and put them into a specific directory. That could certainly be done by the makefile.
As I said before, the fact that we don't have a windows release is proof that the process isn't automated enough. I also know for a fact that Tim did a *lot* of work to get the installer that he asked people to review. This might be inevitable, given the changes in Python, but I don't think it needs to be as bad as it is. And, as I said before, we shouldn't be inventing this ourselves if we can possibly avoid it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Wed, Jan 18, 2006 at 11:24:20AM -0500, Jim Fulton wrote: | Sidnei da Silva wrote: | >On Wed, Jan 18, 2006 at 10:45:20AM -0500, Jim Fulton wrote: | >| People up to now have come up with systems like this that they thought | >were | >| automated enough. That's why we don't have a 2.9 release for windows. | > | >What about we turn that around. How would you describe a 'automated | >enough' build environment? I suspect you consider: | > | > python setup.py bdist_wininst | > | >to be pretty close to that. | | I think | | http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/ZopeWindows... | | Is pretty close. Note that this has a number os steps, but there are few | and they are well documented, so I don't have to think. Not that much different from what already exists for Zope 2: http://svn.zope.org/Zope/trunk/inst/WinBuilders/README.txt?rev=39675&view=au... | As I said before, the fact that we don't have a windows release | is proof that the process isn't automated enough. That's not a proof that the process is not automated enough. The transition from python2.3 to 2.4 *is* non-trivial because python changed from distutils to MSI. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com
Sidnei da Silva wrote: ...
| As I said before, the fact that we don't have a windows release | is proof that the process isn't automated enough.
That's not a proof that the process is not automated enough. The transition from python2.3 to 2.4 *is* non-trivial because python changed from distutils to MSI.
Except that the same sort of problems occurred with 2.8. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Wed, Jan 18, 2006 at 11:46:33AM -0500, Jim Fulton wrote: | Sidnei da Silva wrote: | ... | >| As I said before, the fact that we don't have a windows release | >| is proof that the process isn't automated enough. | > | >That's not a proof that the process is not automated enough. The | >transition from python2.3 to 2.4 *is* non-trivial because python | >changed from distutils to MSI. | | Except that the same sort of problems occurred with 2.8. I would describe the problems with the Windows installers as lack of interest by the developers, not problems with installer building automation. One way or another, the best I can propose right now is to integrate the building of the Windows installer into a buildbot, either ZC's or Enfold Systems'. We already build the nightly Windows Installer for Plone, so it wouldn't be a big deal to do the same for Zope. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com
Sidnei da Silva wrote:
On Wed, Jan 18, 2006 at 11:46:33AM -0500, Jim Fulton wrote: | Sidnei da Silva wrote: | ... | >| As I said before, the fact that we don't have a windows release | >| is proof that the process isn't automated enough. | > | >That's not a proof that the process is not automated enough. The | >transition from python2.3 to 2.4 *is* non-trivial because python | >changed from distutils to MSI. | | Except that the same sort of problems occurred with 2.8.
I would describe the problems with the Windows installers as lack of interest by the developers, not problems with installer building automation.
We don't have any more Windows interest for Zope 3, but we have no problem creating windows releases. That's because we made the barrier very low.
One way or another, the best I can propose right now is to integrate the building of the Windows installer into a buildbot, either ZC's or Enfold Systems'.
We already build the nightly Windows Installer for Plone, so it wouldn't be a big deal to do the same for Zope.
Cool. Then we can just pick up tomorrows output and ship that. ;) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Hi, On Wed, 2006-01-18 at 13:40 -0200, Sidnei da Silva wrote:
On Wed, Jan 18, 2006 at 10:27:25AM -0500, Jim Fulton wrote: | >The installers do not require much Windows expertise. In fact, they | >require a lot of 'makefile' expertise right now, and some Inno Setup | >expertise, not much else. | | Sorry, Inno Setup is a windows installation builder. I consider this | windows expertise.
You don't have to know how to operate Inno Setup for building a installer though. It pretty much boils down to editing the makefiles to contain the proper version number and typing 'make'. I would consider that automated enough. What about adding a target that does this into the buildbot so we get nightly installers?
You have to know Inno. I took care for the Windows builds for a while and it was a large pain to get it started. I'm not a natural windows hacker, although I was developing on windows some years ago and know my way around. Additionally the whole cludge (was/is) very fragile so that new releases pretty reliably required tweaking the build environment. As Tim said one day: You need to work/develop on windows on a daily basis to be able to cut releases for it. I'd second this for any platform actually. Christian -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - ct@gocept.com - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development
Christian Theune wrote: ...
As Tim said one day: You need to work/develop on windows on a daily basis to be able to cut releases for it.
I'd second this for any platform actually.
This is why it is critical that the process be automated. This is why distutils is so great, as far as it goes. It is also great because there is a large community working on it and extending it. Phillip Eby and others are doing some great work with eggs and making them work on a variety of platforms. What distutils lacks is a well thought out model for manageing applications rather than libraries. We need to take this to the next step, but we need to work through the distutils SIG, or work with other projects (e.g. Paste Deploy) rather than inventing this ourselves. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On 1/18/06, Jim Fulton <jim@zope.com> wrote:
These were some of my reactions to this first attempt at time-based releases. What do other folks think?
I think early January is an understandable delay, considering that midwinter celebrations came in the way. Great work everyone! -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
--On 18. Januar 2006 07:36:35 -0500 Jim Fulton <jim@zope.com> wrote:
In the future, if someone introduces a major change, they *must* be committed to be available to deal with issues that arise during the release cycle. Perhaps we need to pick different release dates to avoid holidays. Stephan has suggested moving the dates up to November/May rather than December/June.
+1 for moving the dates. Speaking for the Zope 2 release: - the zpkg chances were introduced very late and it was somewhat annoying to deal with this almost untested changes during the betas (not blaming Philipp here, he has done a tremendous lot of work) In addition such major changes should happen on a branch and not on the trunk and such changes should be started early before the next release (not week or two before the first beta).
And then there are the Windows releases. Making Zope 2 windows releases is very painful and there don't seem to be many people willing to help. We've avoided the pain for Zope 3 by being less ambitious. We let distutils do most of the work. The result is that making a windows release takes minutes and is highly automated, but the experience for the end user is less than ideal, Many would rightfully argue that it is inadequate. What we need is a release process that is as easy as the Zope 3 windows release process and produces a result as usable as the Zope 2 windows release. I'm not sure exactly what the answer is, but I am sure we need to take a fresh approach. Whatever approach we take needs to be highly automated and must not require a lot of specialized Windows expertise.
The basic problem with the windows release is that there is currently nobody in charge for the windows release (although Tim is again doing working on the Windows side, ALL HAIL TIM).
Note that I'm banking heavily on eggs without personally having worked with them much. I'm very hopeful that we can make them work for us.
In the end, I consider the "December" release to be largely successful, given the challenges.
It was basically a birth with some pain :-)
These were some of my reactions to this first attempt at time-based releases. What do other folks think?
I think 2.9.0 is the _real_ 2.9 beta which will be widely used by ppl :-) -aj
Andreas Jung wrote:
I think 2.9.0 is the _real_ 2.9 beta which will be widely used by ppl :-)
Isn't this always the case? :) (ie with all 2.x.0 releses) - Rocky -- Rocky Burt ServerZen Software -- http://www.serverzen.com ServerZen Hosting -- http://www.serverzenhosting.net News About The Server -- http://www.serverzen.net
Andreas Jung wrote:
...
The basic problem with the windows release is that there is currently nobody in charge for the windows release (although Tim is again doing working on the Windows side, ALL HAIL TIM).
I'll repeat or emphasis that the windows release process needs to be simple enough that *I* can do it. This means that the process should be simple and well documented enough that someone like me can follow it without thinking. Thinking is hard. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
--On 18. Januar 2006 10:31:03 -0500 Jim Fulton <jim@zope.com> wrote:
Andreas Jung wrote:
...
The basic problem with the windows release is that there is currently nobody in charge for the windows release (although Tim is again doing working on the Windows side, ALL HAIL TIM).
I'll repeat or emphasis that the windows release process needs to be simple enough that *I* can do it.
Well, that's a perfect goal :-) But my experience with doing slightly simple programming tasks on Windows is that Windows will slap you wherever possible - even when you're trying to solve simple problems. I stopped dreaming that anything on Windows works as it should. just-being-a-frustrated-windows-hacker-(sometimes), Andreas
On Wed, Jan 18, 2006 at 04:37:12PM +0100, Andreas Jung wrote: | >I'll repeat or emphasis that the windows release process needs to | >be simple enough that *I* can do it. | | Well, that's a perfect goal :-) But my experience with doing slightly | simple programming tasks on Windows is that Windows will slap you wherever | possible - even when you're trying to solve simple problems. I stopped | dreaming that anything on Windows works as it should. I'll add to that that I used to think this way, too, but Mark Hammond slowly convinced me otherwise. Today I have the opinion that no matter what other people say, Windows is actually superior to *anything* I've seen in Linux or OS X, except for the networking stack and process management. COM, for example, is very cool stuff. You should see some of the stuff Mark has done that allows one to call pretty much any Python object via COM from any language that supports COM as long as the Python object has a interface declaration using 'zope.interface'. I'm still waiting to see something like COM on the Linux world. Over with the bashing, back to the topic now. -- Sidnei da Silva Enfold Systems, LLC. http://enfoldsystems.com
Sidnei da Silva wrote: [snip OS flamewar in the bud] :) Regards, Martijn
Andreas Jung wrote: ...
I think 2.9.0 is the _real_ 2.9 beta which will be widely used by ppl :-)
I could be wrong, but if we stick to a 6-month release cycle for feature releases, I don't think there is going to be much appetite for bug-fix releases, except in extreme cases, and I think it will be more important to get the feature releases right. Of course, this required commitment from the community. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
Andreas Jung wrote: ...
I think 2.9.0 is the _real_ 2.9 beta which will be widely used by ppl :-)
I could be wrong, but if we stick to a 6-month release cycle for feature releases, I don't think there is going to be much appetite for bug-fix releases, except in extreme cases, and I think it will be more important to get the feature releases right.
I expect a fairly normal progression of bugfix releases myself, except that hopefully less features will sneak in as has been the case with older Zope 2.x releases. So we'll see less bugfix releases, but only because we're not "fixing" features in them. :) Regards, Martijn
Hey, First, I'd like to thank you and everyone involved in the Zope 2 and Zope 3 releases for making this time-based release in what I consider to be a smashing success. Thanks for all the hard work! Things were late a bit, some things are imperfect, but we in the community are already feeling the positive effects of being able to plan for releases. I want Jim to be sure he realizes it's a success from other's perspective as well. Several great features are probably in Zope 2 now in part due to the time-based release system. I want to point out for special attention Florent's great work on making Zope 3 style events work with Zope 2. This is no less than fantastic. This kind of work might've lingered for a long time if release dates were uncertain. Jim Fulton wrote:
Now that I've had a week or so to recover from making the Zope 3 releases, I'd like to look at how we did on our first timed releases.
Thanks for doing the post-mortem. Very valuable.
Of course, the releases didn't happen in December. In fact, the Zope 2 Windows release still hasn't happened.
That we were late isn't a great surprise, given that this was our first timed release. The beginning of the release cycle was delayed because people didn't really realize/remember that we needed to freeze by November 1.
[snip]
In the future, if someone introduces a major change, they *must* be committed to be available to deal with issues that arise during the release cycle. Perhaps we need to pick different release dates to avoid holidays. Stephan has suggested moving the dates up to November/May rather than December/June.
+1 to moving the dates to be further away from holiday seasons.
And then there are the Windows releases. Making Zope 2 windows releases is very painful and there don't seem to be many people willing to help. [snip]
Getting a dedicated volunteer to work on Windows releases seems to be hard. I've tried with lxml and while several people have successfully compiled it on Windows, and several people volunteered to help with releases, it still hasn't gotten off the ground. I'm glad the decision was made to let Windows not be a release blocker. We're not the only one with this problem. A few months ago I was reading the Apache HTTP server mailing lists and to my surprise found a thread talking about how their process was failing. The Apache HTTP server has a very strict policy concerning changes to the codebase; they have to be vetted by other programmers before they can go in. There were some problems pointed out about this procedure slowing things down, but interestingly enough one of the blockers for making new releases of Apache was the volunteers not knowing how to do a Windows release. :)
I think that packaging is a significant challenge to our release process. The Zope 2 release was complicated by the use of zpkg. The zpkg system was developed to allow us to decouple the Zope 3 repository and releases. It was too painful to mold the repository to fleeting release decisions. We wanted people to put experimental and add-on applications in the repository so that they would be tested as we rapidly evolved things. While zpkg was crucial for the last 3 releases, I don't think it provides the best long-term approach. Rather than extracting a release from a larger repository tree, I think it would be better, going forward, to assemble a release from smaller individual repository trees or from releases.
I'm glad this is being given a re-think. How do you assemble releases 'from releases'? I'm not sure I understand that. You mean make a Zope 2 release using a Zope 3 release? You know my position concerning the repository and the release; I'd prefer them to be kept as similar as possible to simplify the release process. I hope we can go in that direction. It also makes things more predictable to developers. We noticed that some Zope 3 packages weren't packaged into Zope 2 after the release, even though in a developer's sandbox of Zope 2 they were there. As a side issue: From the perspective of Five, it is beneficial to have as much Zope 3 code included into Zope 2 releases, as that gives us an opportunity to start using this functionality right away, exposing it to Zope 2, without waiting for a new release. [snip]
We have begun to see Zope 3 decomposed into separate projects knit together via svn:externals. I'd like to see that trend continue and to perhaps switch to making the knitting process use eggs, I'd like to leverage eggs to make the release process much simpler. It should be easy for people to make releases so that there could easily be multiple distributions aimed at different needs and expectations.
I'll repeat here that I think there's much value in trying to keep the mapping of the SVN repository very similar to what is actually released. (If you're interested I can try to explain some of my thinking a bit deeply.) Eggs are a nice distribution mechanism, but I'd also want the knitting process to work for a SVN checkout -- developers working with SVN need to be very easily work with a 'knitted' version, so perhaps svn:externals will remain a valuable tool.
As part of this decomposition, we need to make zope.app much smaller. Early in Zope 3 development, I proposed a simple rule for organization of the repository: Anything that depended on zope.app went in zope.app. Anything else went in zope. If there was doubt, put it in zope.app. I think that served us well at the time, but I think we've moved beyond that, I'd like to migrate most of zope.app elsewhere. Zope 2.10 should not include zope.app.
This worries me; Five is currently using massive quantities of code in zope.app, and as expressed above, I value Five having access to potentially all Zope 3 code that's in a Zope 3 release.
Note that I'm banking heavily on eggs without personally having worked with them much. I'm very hopeful that we can make them work for us.
In the end, I consider the "December" release to be largely successful, given the challenges.
Agreed.
These were some of my reactions to this first attempt at time-based releases. What do other folks think?
A great success! Thank you!
Given that the Zope Foudation will be formed during this next release cycle, I think this is a good time to take stock and think about how to move forward,
Included were some of my thoughts. Regards, Martijn
On Wednesday 18 January 2006 11:27, Martijn Faassen wrote:
How do you assemble releases 'from releases'? I'm not sure I understand that. You mean make a Zope 2 release using a Zope 3 release?
I'll note that SchoolTool greatly benefits from the current release building. We simply include all the Zope 3 dependencies in our dependency list and build the release. We can decide to include Zope 3 dependencies or not. Overall I think zpkg is a great win and whatever we do next should not remove those features. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
Stephan Richter wrote:
On Wednesday 18 January 2006 11:27, Martijn Faassen wrote:
How do you assemble releases 'from releases'? I'm not sure I understand that. You mean make a Zope 2 release using a Zope 3 release?
I'll note that SchoolTool greatly benefits from the current release building. We simply include all the Zope 3 dependencies in our dependency list and build the release. We can decide to include Zope 3 dependencies or not. Overall I think zpkg is a great win and whatever we do next should not remove those features.
I agree that dependency based releases -- and development is great. I think eggs are a lot farther along that zpkg. (Eggs weren't around when we started zpkg.) If eggs work out, as I hope they will, I'd like to stop work on zpkg and just use eggs. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On 1/18/06, Jim Fulton <jim@zope.com> wrote:
If eggs work out, as I hope they will, I'd like to stop work on zpkg and just use eggs.
+42 -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "There is no wealth but life." --John Ruskin
Jim Fulton wrote:
Stephan Richter wrote:
On Wednesday 18 January 2006 11:27, Martijn Faassen wrote:
How do you assemble releases 'from releases'? I'm not sure I understand that. You mean make a Zope 2 release using a Zope 3 release?
I'll note that SchoolTool greatly benefits from the current release building. We simply include all the Zope 3 dependencies in our dependency list and build the release. We can decide to include Zope 3 dependencies or not. Overall I think zpkg is a great win and whatever we do next should not remove those features.
I agree that dependency based releases -- and development is great. I think eggs are a lot farther along that zpkg. (Eggs weren't around when we started zpkg.) If eggs work out, as I hope they will, I'd like to stop work on zpkg and just use eggs.
Sure, I support dependencies and separating out Zope into sub projects, I'm just listing an additional use case: the repository state should be similar to release state, to avoid confusion for developers as well as people who want to become developers. I.e. a developer should have the ability to easily see the same (or similar) code layout, dependencies, etc, as in a release, and someone familiar with a release should see the same (or similar) code layout, dependencies, if they become a developer. Another use case, probably mostly in the context of Five, it's nice to have an inclusive release of Zope 3 in Zope 2. The goal of reducing the amount of code included in Zope 2 sounds nice in theory, but it stops Five developers from exposing Zope 3 code in Zope 2 because it simply isn't there in a particular release. It is *nice* to have all of Zope 3 included in Zope 2. I don't want to lose that good thing in the rush to minimize dependencies. Regards, Martijn
Martijn Faassen wrote: ...
Sure, I support dependencies and separating out Zope into sub projects, I'm just listing an additional use case: the repository state should be similar to release state, to avoid confusion for developers as well as people who want to become developers.
I.e. a developer should have the ability to easily see the same (or similar) code layout, dependencies, etc, as in a release, and someone familiar with a release should see the same (or similar) code layout, dependencies, if they become a developer.
Right, got it. A checkout and a release should be very similar. If eggs work out, then they would both use eggs to satisfy their dependencies.
Another use case, probably mostly in the context of Five, it's nice to have an inclusive release of Zope 3 in Zope 2. The goal of reducing the amount of code included in Zope 2 sounds nice in theory, but it stops Five developers from exposing Zope 3 code in Zope 2 because it simply isn't there in a particular release. It is *nice* to have all of Zope 3 included in Zope 2. I don't want to lose that good thing in the rush to minimize dependencies.
Right now Five/Zope2 include lots of packages they don't and may never use. I want Five/Zope2 to not *have* to include packages they don't need just because we've created monoliths. I especially don't want to release experimental code through Five/Zope2 just because we don't have our repository and/or packaging in order. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
Martijn Faassen wrote:
Another use case, probably mostly in the context of Five, it's nice to have an inclusive release of Zope 3 in Zope 2. The goal of reducing the amount of code included in Zope 2 sounds nice in theory, but it stops Five developers from exposing Zope 3 code in Zope 2 because it simply isn't there in a particular release. It is *nice* to have all of Zope 3 included in Zope 2. I don't want to lose that good thing in the rush to minimize dependencies.
Right now Five/Zope2 include lots of packages they don't and may never use. I want Five/Zope2 to not *have* to include packages they don't need just because we've created monoliths. I especially don't want to release experimental code through Five/Zope2 just because we don't have our repository and/or packaging in order.
Hm, some confusion.. Perhaps the cause is this: With "Zope 3", I mean Zope 3 *as released*. I imagine you might think of "Zope 3" differently, i.e. Zope 3 as what's in the repository, which includes things beyond what gets released (i.e experimental packages). I'm talking about a Zope 2 release including (most of) what's in a Zope 3 release, so that Five developers can work on exposing *that* in Zope 2 too (which can then be part of the next Zope 2 release as we integrate the newer Five in it). I'm describing a pattern of working that has worked pretty well for Five for a while now. Of course that doesn't mean I want experimental packages in Zope 2 that are not in a Zope 3 release. Five is about exposing Zope 3 as released in Zope 2, it shouldn't expose *more* functionality than Zope 3 does. :) Regards, Martijn
Martijn Faassen wrote at 2006-1-19 19:37 +0100:
... I'm talking about a Zope 2 release including (most of) what's in a Zope 3 release, so that Five developers can work on exposing *that* in Zope 2 too (which can then be part of the next Zope 2 release as we integrate the newer Five in it).
I very much like Jims idea: having a small core with the ability to add separately released extensions on demand. -- Dieter
Martijn Faassen wrote: ...
How do you assemble releases 'from releases'? I'm not sure I understand that. You mean make a Zope 2 release using a Zope 3 release?
No, I mean using eggs. Zope should be broken into separate projects with their own eggs. A Zope release might just be an egg with dependencies. Or it might just be a collection of eggs.
You know my position concerning the repository and the release; I'd prefer them to be kept as similar as possible to simplify the release process. I hope we can go in that direction. It also makes things more predictable to developers. We noticed that some Zope 3 packages weren't packaged into Zope 2 after the release, even though in a developer's sandbox of Zope 2 they were there.
Right. If eggs work out, then a respository check out will be a lot smaller, but will download needed eggs. This would be a replacement of the use of externals we have now.
As a side issue: From the perspective of Five, it is beneficial to have as much Zope 3 code included into Zope 2 releases, as that gives us an opportunity to start using this functionality right away, exposing it to Zope 2, without waiting for a new release.
Understand though that there is nothing like a backward compatibility promise for something that hasn't been released.
[snip]
We have begun to see Zope 3 decomposed into separate projects knit together via svn:externals. I'd like to see that trend continue and to perhaps switch to making the knitting process use eggs, I'd like to leverage eggs to make the release process much simpler. It should be easy for people to make releases so that there could easily be multiple distributions aimed at different needs and expectations.
I'll repeat here that I think there's much value in trying to keep the mapping of the SVN repository very similar to what is actually released.
I think I understand where you are coming from.
(If you're interested I can try to explain some of my thinking a bit deeply.) Eggs are a nice distribution mechanism, but I'd also want the knitting process to work for a SVN checkout -- developers working with SVN need to be very easily work with a 'knitted' version, so perhaps svn:externals will remain a valuable tool.
Assuming that eggs fullfill their promise, I think I'd rather use eggs than externals.
As part of this decomposition, we need to make zope.app much smaller. Early in Zope 3 development, I proposed a simple rule for organization of the repository: Anything that depended on zope.app went in zope.app. Anything else went in zope. If there was doubt, put it in zope.app. I think that served us well at the time, but I think we've moved beyond that, I'd like to migrate most of zope.app elsewhere. Zope 2.10 should not include zope.app.
This worries me; Five is currently using massive quantities of code in zope.app, and as expressed above, I value Five having access to potentially all Zope 3 code that's in a Zope 3 release.
The code that Five is using will still be available, but it will be somewhere else (with necessary backward compatibility hacks). Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Wednesday 18 January 2006 19:09, Jim Fulton wrote:
You know my position concerning the repository and the release; I'd prefer them to be kept as similar as possible to simplify the release process. I hope we can go in that direction. It also makes things more predictable to developers. We noticed that some Zope 3 packages weren't packaged into Zope 2 after the release, even though in a developer's sandbox of Zope 2 they were there.
Right. If eggs work out, then a respository check out will be a lot smaller, but will download needed eggs. Â This would be a replacement of the use of externals we have now.
Oh, this will make development so much more tedious. Let's say zope.testbrowser is an egg and I discover a bug in zope.textbrowser while doing some other Zope 3 development, I have to check out zope.testbrowser, fix the bug, check it in, download the new egg and hope it fixed my Zope 3 problem. Honestly this is far too much and I will at most make a bug report. I have seen you take a similar approach to zope.testing and I found that painful just by watching the checkins. I feel like an old record, but please let's keep the development process as simple as possible. I rather make some concessions to the packaging and dependency system than spending more time developing. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
Stephan Richter wrote:
Let's say zope.testbrowser is an egg and I discover a bug in zope.textbrowser while doing some other Zope 3 development, I have to check out zope.testbrowser, fix the bug, check it in, download the new egg and hope it fixed my Zope 3 problem.
I'm an egg neophyte, but I believe you can put an egg in "dev mode" (or whatever it's called) and you'll get a subversion (or cvs) checkout. You can then make your changes, etc. -- Benji York Senior Software Engineer Zope Corporation
Stephan Richter wrote:
On Wednesday 18 January 2006 19:09, Jim Fulton wrote:
You know my position concerning the repository and the release; I'd prefer them to be kept as similar as possible to simplify the release process. I hope we can go in that direction. It also makes things more predictable to developers. We noticed that some Zope 3 packages weren't packaged into Zope 2 after the release, even though in a developer's sandbox of Zope 2 they were there.
Right. If eggs work out, then a respository check out will be a lot smaller, but will download needed eggs. This would be a replacement of the use of externals we have now.
Oh, this will make development so much more tedious. Let's say zope.testbrowser is an egg and I discover a bug in zope.textbrowser while doing some other Zope 3 development, I have to check out zope.testbrowser, fix the bug, check it in, download the new egg and hope it fixed my Zope 3 problem. Honestly this is far too much and I will at most make a bug report.
I beliave that Eggs have a development mode in which you can work on the source. This apears to be easy to switch too, perhaps easier than editing externals. I haven't gotten to try this myself. We'll see as we learn more about eggs. I'm at least as lazy as you are, so rest assured, I'm gonna try to come up with a methodology that makes my life as easy as possible. If more than one application uses a package, it is feasible for at most one application to include it. Managing the application that way makes it more complicated for other applications to use. For example, the package's releace cycle becomes bound to the release cycle of the including application, which is cumbersome for other applications.
I have seen you take a similar approach to zope.testing and I found that painful just by watching the checkins.
I don't understand what you mean. Having a separate zope.testing project has been extremely useful. For example, in our comercial apps, we often used newer versions than existed in Zope 3. We often needed enhacements to zope.testing at times that Zope 3 was feature frozen. We could have made a Zope 3 branch just to modify zope.testing, but that would have been a hassle for us and for Zope 3 developers. Note that the new test runner (from zope.testing) was used in ZODB long before it was used in Zope 3.
I feel like an old record, but please let's keep the development process as simple as possible. I rather make some concessions to the packaging and dependency system than spending more time developing.
Perhaps our goals are different. I want Zope's packages to be usable outside of Zope. I also want to make it easier to use external packages. I think that a more package-centric development methodoligy will make achieving these goals much easier. I also think it will make Zope releases go smoother because the scope of the release will be narrower. Someday, A Zope release will include an app-server specific core and a collection of other package releases. To the extend that those packages have their own lifeccyles, and quality control, the amount of newly released code in a Zope release wil be smaller. A package-based approach will also make it easier to release less. We'll be freed from a "batteries included" mentality that encourages large high-risk releases. And it will make it easier for people to make custom releases that have configurations targeted to specific needs. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
On Thursday 19 January 2006 07:00, Jim Fulton wrote:
I have seen you take a similar approach to zope.testing and I found that painful just by watching the checkins.
I don't understand what you mean. Â Having a separate zope.testing project has been extremely useful. Â For example, in our comercial apps, we often used newer versions than existed in Zope 3. Â We often needed enhacements to zope.testing at times that Zope 3 was feature frozen. We could have made a Zope 3 branch just to modify zope.testing, but that would have been a hassle for us and for Zope 3 developers. Â Note that the new test runner (from zope.testing) was used in ZODB long before it was used in Zope 3.
Argument taken and agreed that this is a valuable use case. Regard, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On Thursday 19 January 2006 07:00, Jim Fulton wrote:
 > I feel like an old record, but please
let's keep the development process as simple as possible. I rather make some concessions to the packaging and dependency system than spending more time developing.
Perhaps our goals are different. I want Zope's packages to be usable outside of Zope. Â I also want to make it easier to use external packages. I think that a more package-centric development methodoligy will make achieving these goals much easier. Â I also think it will make Zope releases go smoother because the scope of the release will be narrower. Â Someday, A Zope release will include an app-server specific core and a collection of other package releases. Â To the extend that those packages have their own lifeccyles, and quality control, the amount of newly released code in a Zope release wil be smaller. Â A package-based approach will also make it easier to release less. We'll be freed from a "batteries included" mentality that encourages large high-risk releases. And it will make it easier for people to make custom releases that have configurations targeted to specific needs.
I think we do have different goals here. I would agree with that approach, if we would have a large community where small groups take over the maintenance and development of a package. But that's not the case and it will not change any time soon, I think. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
Stephan Richter wrote:
On Thursday 19 January 2006 07:00, Jim Fulton wrote:
I feel like an old record, but please
let's keep the development process as simple as possible. I rather make some concessions to the packaging and dependency system than spending more time developing.
Perhaps our goals are different. I want Zope's packages to be usable outside of Zope. I also want to make it easier to use external packages. I think that a more package-centric development methodoligy will make achieving these goals much easier. I also think it will make Zope releases go smoother because the scope of the release will be narrower. Someday, A Zope release will include an app-server specific core and a collection of other package releases. To the extend that those packages have their own lifeccyles, and quality control, the amount of newly released code in a Zope release wil be smaller. A package-based approach will also make it easier to release less. We'll be freed from a "batteries included" mentality that encourages large high-risk releases. And it will make it easier for people to make custom releases that have configurations targeted to specific needs.
I think we do have different goals here. I would agree with that approach, if we would have a large community where small groups take over the maintenance and development of a package. But that's not the case and it will not change any time soon, I think.
I disagree. There are sub-projects within The Zope tree that are worked on by a few people. Take viewlets for example. I think we'll have more of this once we have a more supportive environment. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
... [Stephan Ricther]
I have seen you take a similar approach to zope.testing and I found that painful just by watching the checkins.
[Jim Fulton]
I don't understand what you mean. Having a separate zope.testing project has been extremely useful. For example, in our comercial apps, we often used newer versions than existed in Zope 3. We often needed enhacements to zope.testing at times that Zope 3 was feature frozen. We could have made a Zope 3 branch just to modify zope.testing, but that would have been a hassle for us and for Zope 3 developers. Note that the new test runner (from zope.testing) was used in ZODB long before it was used in Zope 3.
I want to note that this was good for Zope3, too: as a willing "early adopter" of zope.testing, ZODB hit bugs and platform-dependent glitches first, and got them fixed before the larger Zope3 development community could be bit by them. Also want to note that ZRS needed to add "new features" to zope.testing, and ZRS doesn't include _any_ Zope code (ZRS builds on ZEO, not Zope). Having zope.testing be its own project without all the adminstrative overheads of having its own "official releases" made it very easy to add new code for ZRS's benefit without disturbing any of zope.testing's other users. In all, zope.testing is a poster child for the value of package development outside of a Zope tree. It's true that, to make changes in zope.testing, I needed to have a distinct checkout of zope.testing. I didn't feel that to be a burden, though.
On 1/20/06, Tim Peters <tim.peters@gmail.com> wrote:
In all, zope.testing is a poster child for the value of package development outside of a Zope tree.
I've been very happy using zope.testing with several non zope projects. Including how easy it is to follow and distribute that package as needed for those projects. Same goes for zdaemon. And of course ZConfig. Michael
Stephan Richter wrote:
On Wednesday 18 January 2006 19:09, Jim Fulton wrote:
You know my position concerning the repository and the release; I'd prefer them to be kept as similar as possible to simplify the release process. I hope we can go in that direction. It also makes things more predictable to developers. We noticed that some Zope 3 packages weren't packaged into Zope 2 after the release, even though in a developer's sandbox of Zope 2 they were there.
Right. If eggs work out, then a respository check out will be a lot smaller, but will download needed eggs. This would be a replacement of the use of externals we have now.
Oh, this will make development so much more tedious. Let's say zope.testbrowser is an egg and I discover a bug in zope.textbrowser while doing some other Zope 3 development, I have to check out zope.testbrowser, fix the bug, check it in, download the new egg and hope it fixed my Zope 3 problem. Honestly this is far too much and I will at most make a bug report.
Ah, exactly the risk I pointed out too, I should've read the thread first before I repeated you. :)
I have seen you take a similar approach to zope.testing and I found that painful just by watching the checkins. I feel like an old record, but please let's keep the development process as simple as possible. I rather make some concessions to the packaging and dependency system than spending more time developing.
What if we can create in SVN the equivalent of what would be an egg + its dependencies for checkout, using externals? I know Jim said he doesn't want to use externals, but I'm thinking in that direction. You'd have one SVN directory for each egg, which then contains the right externals to pull in all the dependencies. Hopefully the process of creating such an SVN directory could be automated from egg dependency metadata. Regards, Martijn
On Thursday 19 January 2006 07:46, Martijn Faassen wrote:
Oh, this will make development so much more tedious. Let's say zope.testbrowser is an egg and I discover a bug in zope.textbrowser while doing some other Zope 3 development, I have to check out zope.testbrowser, fix the bug, check it in, download the new egg and hope it fixed my Zope 3 problem. Honestly this is far too much and I will at most make a bug report.
Ah, exactly the risk I pointed out too, I should've read the thread first before I repeated you. :)
Yeah, I am glad someone else backs my concern. I agreed with everything you said in your previous mail.
I have seen you take a similar approach to zope.testing and I found that painful just by watching the checkins. I feel like an old record, but please let's keep the development process as simple as possible. I rather make some concessions to the packaging and dependency system than spending more time developing.
What if we can create in SVN the equivalent of what would be an egg + its dependencies for checkout, using externals? I know Jim said he doesn't want to use externals, but I'm thinking in that direction. You'd have one SVN directory for each egg, which then contains the right externals to pull in all the dependencies. Hopefully the process of creating such an SVN directory could be automated from egg dependency metadata.
That would work for me. If it resolves the risk and is still pretty automated, SVN checkout or even calling make, then it is fine by me. The others have also pointed out the egg development mode. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
Stephan Richter wrote: [svn reflecting egg dependency structure]
That would work for me. If it resolves the risk and is still pretty automated, SVN checkout or even calling make, then it is fine by me. The others have also pointed out the egg development mode.
Right, I didn't know of that, but if that fulfills the usecases, then that's even better. Regards, Martijn
Martijn Faassen wrote: ...
What if we can create in SVN the equivalent of what would be an egg + its dependencies for checkout, using externals? I know Jim said he doesn't want to use externals, but I'm thinking in that direction. You'd have one SVN directory for each egg, which then contains the right externals to pull in all the dependencies. Hopefully the process of creating such an SVN directory could be automated from egg dependency metadata.
I'm confused. I thought you wanted a checkout to look like a release. Releases won't use svn:externals. What about dependencies that aren't managed in subversion? Will you import them into a repository just so you can use an external? What if a dependency of a dependency changes? Externals don't handle this well. Eggs do. I think we should investigate eggs. Do I know they will work? No. I haven't done much with them yet. Do you know they won't? Obviously not. I suggest we reserve jusdgement until we have had an opportunity for some prototyping. Based on what I've seen so far, I'm very hopeful. And then there's the fact that they come from a much wider community than just Zope. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
Martijn Faassen wrote: ...
What if we can create in SVN the equivalent of what would be an egg + its dependencies for checkout, using externals? I know Jim said he doesn't want to use externals, but I'm thinking in that direction. You'd have one SVN directory for each egg, which then contains the right externals to pull in all the dependencies. Hopefully the process of creating such an SVN directory could be automated from egg dependency metadata.
I'm confused. I thought you wanted a checkout to look like a release. Releases won't use svn:externals. What about dependencies that aren't managed in subversion? Will you import them into a repository just so you can use an external? What if a dependency of a dependency changes? Externals don't handle this well. Eggs do.
I wasn't fully aware when I wrote that that eggs can help during development. I need to read up on development eggs.
I think we should investigate eggs. Do I know they will work? No. I haven't done much with them yet. Do you know they won't? Obviously not. I suggest we reserve jusdgement until we have had an opportunity for some prototyping. Based on what I've seen so far, I'm very hopeful. And then there's the fact that they come from a much wider community than just Zope.
No, I don't know they don't work, I just hadn't considered eggs as a development tool. I agree wholeheartedly eggs should be explored, and I was homing in on a possible solution before I understand what other alternatives exist. :) Regards, Martijn
Jim Fulton wrote:
I think we should investigate eggs. Do I know they will work? No. I haven't done much with them yet. Do you know they won't? Obviously not. I suggest we reserve jusdgement until we have had an opportunity for some prototyping. Based on what I've seen so far, I'm very hopeful. And then there's the fact that they come from a much wider community than just Zope.
I have to admit to being very interested in eggs myself. I like the idea of zope being a collection of independent packages with their own release schedules - breaking down the "monolithic release" problem. I think Zope 3 has done extremely well with all this so far. Personally, I've used Zope 3s' ZPT and testbrowser packages in contexts that have nothing to do with Zope 3 (or even Zope sometimes) and liked the feel of it. I wasn't away the same was true for zope.testing but I'm very glad to hear it. On a side note, I see python now includes doctest.py, why are we still maintaining our own copy in zope.testing? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote: ...
On a side note, I see python now includes doctest.py, why are we still maintaining our own copy in zope.testing?
doctest has been in Python for a long time, far longer than we've been using it. Now that we use it aggressively and are aggressively contributing to it, we usually need a newer version than is available in the Python release we're using. If Python had and used a packaging system, like eggs, this wouldn't be necessary. Someday. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
If Python had and used a packaging system, like eggs, this wouldn't be necessary. Someday.
*grinz* The irony that python is so successfuly _because_ of it's "batteries included" nature isn't lost on me ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Jim Fulton wrote:
If Python had and used a packaging system, like eggs, this wouldn't be necessary. Someday.
*grinz*
The irony that python is so successfuly _because_ of it's "batteries included" nature isn't lost on me ;-)
I would argue that it is successful despite it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
Martijn Faassen wrote: ...
How do you assemble releases 'from releases'? I'm not sure I understand that. You mean make a Zope 2 release using a Zope 3 release?
No, I mean using eggs. Zope should be broken into separate projects with their own eggs. A Zope release might just be an egg with dependencies. Or it might just be a collection of eggs.
Ah, makes sense. A 'meta-package', so to speak. Works well in Debian, so I think that's an interesting pattern to follow. As long as we can do those micro releases quickly -- the problem is that Zope 3 is one project, and we want one communication channel, as we simply don't have enough people otherwise. From another perspective it's many projects, though.
You know my position concerning the repository and the release; I'd prefer them to be kept as similar as possible to simplify the release process. I hope we can go in that direction. It also makes things more predictable to developers. We noticed that some Zope 3 packages weren't packaged into Zope 2 after the release, even though in a developer's sandbox of Zope 2 they were there.
Right. If eggs work out, then a respository check out will be a lot smaller, but will download needed eggs. This would be a replacement of the use of externals we have now.
A risk here is that if I find a bug in package X, I can't easily track it into package Y and fix it there, as package Y is an egg. The current system doesn't have this problem.
As a side issue: From the perspective of Five, it is beneficial to have as much Zope 3 code included into Zope 2 releases, as that gives us an opportunity to start using this functionality right away, exposing it to Zope 2, without waiting for a new release.
Understand though that there is nothing like a backward compatibility promise for something that hasn't been released.
Yes, but Zope 2 included *less* than Zope 3 in the most recent release, and I'd like *all* packages that are in a Zope 3 release to be available in a Zope 2 release. I.e. Five doesn't want packages that aren't in a Zope 3 release, but not less either. [snip]
(If you're interested I can try to explain some of my thinking a bit deeply.) Eggs are a nice distribution mechanism, but I'd also want the knitting process to work for a SVN checkout -- developers working with SVN need to be very easily work with a 'knitted' version, so perhaps svn:externals will remain a valuable tool.
Assuming that eggs fullfill their promise, I think I'd rather use eggs than externals.
Sure, but see the risk I mentioned earlier in this mail.
As part of this decomposition, we need to make zope.app much smaller. Early in Zope 3 development, I proposed a simple rule for organization of the repository: Anything that depended on zope.app went in zope.app. Anything else went in zope. If there was doubt, put it in zope.app. I think that served us well at the time, but I think we've moved beyond that, I'd like to migrate most of zope.app elsewhere. Zope 2.10 should not include zope.app.
This worries me; Five is currently using massive quantities of code in zope.app, and as expressed above, I value Five having access to potentially all Zope 3 code that's in a Zope 3 release.
The code that Five is using will still be available, but it will be somewhere else (with necessary backward compatibility hacks).
As long as Zope 2.10 contains all packages in Zope 3. Regards, Martijn
Martijn Faassen wrote: ...
A risk here is that if I find a bug in package X, I can't easily track it into package Y and fix it there, as package Y is an egg. The current system doesn't have this problem.
There are two issues here: 1. Debugging. Can debugging tools show you code in eggs? They should. If they don't they may need to be improved. If that's not an option, eggs let you expand an egg into a normal directory at installation time. I don't think this will be a problem, although we may need to take some steps to assure that it isn't. 2. Updates. We can't update packages now that we get via externals. If we didn't adopt eggs, I expect that we'd make greater and greater use of externals. Eggs don't make update any harder than externals. ...
Understand though that there is nothing like a backward compatibility promise for something that hasn't been released.
Yes, but Zope 2 included *less* than Zope 3 in the most recent release, and I'd like *all* packages that are in a Zope 3 release to be available in a Zope 2 release. I.e. Five doesn't want packages that aren't in a Zope 3 release, but not less either.
I'm surprised that it included less. I think a more powerful packaging architecture will make it easeir to include what we want. Deciding what we want is another issue. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
Martijn Faassen wrote:
Yes, but Zope 2 included *less* than Zope 3 in the most recent release, and I'd like *all* packages that are in a Zope 3 release to be available in a Zope 2 release. I.e. Five doesn't want packages that aren't in a Zope 3 release, but not less either.
I'm surprised that it included less.
It was a bug, and I think some of it already got resolved (Philipp would know more), but it wasn't noticed until fairly late, as during development such dependencies are development.
I think a more powerful packaging architecture will make it easeir to include what we want. Deciding what we want is another issue.
Agreed. I just wanted to make clear what I want early on. :) Regards, Martijn
Martijn Faassen wrote:
Yes, but Zope 2 included *less* than Zope 3 in the most recent release, and I'd like *all* packages that are in a Zope 3 release to be available in a Zope 2 release. I.e. Five doesn't want packages that aren't in a Zope 3 release, but not less either.
I'm surprised that it included less.
It was a bug, and I think some of it already got resolved (Philipp would know more), but it wasn't noticed until fairly late, as during development such dependencies are development.
It included less due to an oversight on my side. I forgot to make the Zope 2 release depend on certain new zope.app packages that zope.app didn't already depend on. They were not included in Zope 2.9b1, but 2.9b2 quickly fixed that. It was my mistake. I don't think anything would've made it easier catching that mistake other than reports from the people. Also, it's a mute point now (please let's bury it!) as b2 already fixed it. While I got this email open, I'd like to quickly add my EUR0.02 as a summary of this thread: * Breaking Zope (3 that is) into subprojects is extremely useful. zope.testing was a good example, the next things should be zope.interface+zope.schema and zope.pagetemplate+zope.tales+zope.tal etc because there are actually standalone releases of these things or at least people using these things separately from Zope. * Moving things that aren't immanent to the Zope 3 application server out of zope.app is also a good thing. I suspect this is mostly a mechanical change (dropping the 'app' in the import name). Ideally this should indeed leave us with a zope.app that is so Zope3-specific that it isn't of any use for Zope 2. * We should indeed avoid having to include certain Zope 3 things into Zope 2 that don't make any sense in Zope 2. The whole zope.app.twisted machinery is an example. Currently we're not running zope.app tests in Zope 2 because we would have to stitch in twisted into Zope 2 svn! * Zope 2 should continue to use and provide as much Zope 3 functionality as Zope 3 does. This "policy" brought us zope.formlib to Zope 2.9 which is already being enjoyed by some Five developers! * We should get rid of vendor imports such as pytz, docutils and those testbrowser dependencies. Eggs will make this very trivial. We won't even be concerned about development mode or not because we're not modifying them anyways. I think the decision for time-based releases is a success, mostly because I already know when a new Zope release is to expected and roughly what features it'll bring. As someone who puts a lot of effort in Zope 3 documentation I find this extremely valuable as a planning asset. As for the December release itself, I'm not too impressed. On the Five/Zope 2 side of things it was more than people bargained for, namely the port of Five to Zope 3.2 and the packaging of Zope 2 with zpkg. It might be due to the fact that this was our first time-based release and people were just used to releases being postponed when they didn't find the time to commit to a schedule. Obviously this we can't continue like this in the future. If we move the release up to May (I'm +1) then feature freeze is in April. That effectively leaves us not much more than two months worth of time for work on Zope 3.3/2.10. What I want to say with this is that the right time to work on Zope 3.3/2.10 is *now*, not some time in the summer. Philipp
We should distinguish between authoring the Windows build-the-installer code, and running that code. "Making a Zope 2 Windows release" consists of _running_ the build-the-installer code, and is easy. It's actually easier than building a Zope 3 Windows release: once the Python tarball, Zope 2 tarball, and pywin32 installer are downloaded, building a Zope 2 release consists of typing one command in a Cygwin shell: WinBuilders/buildout zope That's it. _Authoring_ the Zope 2 build-the-installer code has been a royal PITA, but work is needed there only when packaging gimmicks change. Like moving to zpkgtools, or moving away from them again, or moving to a different packaging system for one of the components (like Python moving from a Wise installer to an MSI installer for 2.4, or pywin32 earlier moving from a Wise installer to a distutils installer). Across the long life of Zope 2.7, those things changed very little, and as a result very little needed to change in the build-the-installer code across 2.7's life. 2.7 Windows releases remained easy to build across that time (granting that Mark Hammond did the work to adjust to pywin32's packaging changes). 2.8 Windows releases were/are similarly easy to build. A lot of new work was needed for 2.9, due to Python and zpkgtools changes. I also took the opportunity to "upgrade" from InnoSetup 4 to version 5, which introduced a much easier scheme for managing the custom dialog screens in the Zope installer. All of that ended up making the build-the-installer code substantially simpler, to the point where it now seems kinda goofy to use makefiles at all in the process. Makefile mazes are inscrutable, and are particularly ill-suited to the true nature of this task: "unpack various tarballs, and rearrange them like so". Finishing a Windows release of Zope 2 or Zope 3 takes much longer than just minutes, unless you skip testing. If you do skip testing, then it's minutes for both. Testing Zope as a Windows service is harder for Zope 3, because the Zope 3 installer doesn't (but the Zope 2 installer does) install or start the service, or auto-stop and remove the service when Zope is uninstalled. BTW, Zope3 _could_ do that too with a suitable post-install/pre-uninstall script, which is something InnoSetup automates for Zope2. It can take a Windows expert some days to author Zope2 build-the-installer changes when packaging fashions change (whether Zope's or the repackaged components'). InnoSetup is the least of the problems there, and while MSI is the coming fashion, slinging MSI code appears to require much more knowledge than slinging InnoSetup declarations. The idea that it takes "Windows expertise" to write an InnoSetup declaration is wrong: such lines merely say "take the files you find _here_ now, and put them _there_ when the installer runs". For example, this is the entire Inno code needed to install Zope 2's lib/ directory: Source:"lib\*.*"; DestDir: "{app}\lib"; Flags: ignoreversion recursesubdirs That kind of stuff is truly trivial. It does take Inno expertise to create and manage the installer's dialog screens ("manage" == e.g., if the user chooses not to have the installer create an instance home, then the dialog asking for the instance home's directory shouldn't be shown), but those typically don't change across releases.
participants (15)
-
Andreas Jung -
Benji York -
Chris Withers -
Christian Theune -
Dieter Maurer -
Fred Drake -
Jim Fulton -
Lennart Regebro -
Martijn Faassen -
Michael Dunstan -
Philipp von Weitershausen -
Rocky Burt -
Sidnei da Silva -
Stephan Richter -
Tim Peters