At the PIKTipi sprint this past weekend, Andi Zeidler, Andreas Jung and I worked on integrating ZODB 3.8 and Zope 3.4 into the Zope 2 trunk. Zope 3.4 has actually "exploded" into many eggs that are now independently managed and released. I therefore looked into a way to make Zope 2 dependent on these eggs instead of shipping with its own copies of the packages. There are clear benefits to that: * as newer versions of specific zope.* or ZODB packages are released, you can individually install those or not. There'll be no need to wait for another Zope 2 release. * many third-party packages that we had to ship with Zope 2 were actually dependencies of the included Zope 3 (pytz, ClientForm, etc.). The Zope 3 eggs now simply depend on them via the egg dependency mechanism, which means we no longer have to track those projects through vendor imports. * I actually went a bit further and eggified Zope 2 (it seemed the logical conclusion). I didn't really explode Zope 2, it's one big egg (except for a few things that are completely independent, such as ExtensionClass, Acquisition, DateTime, zLOG). Though currently not very comfortable to use, the Zope2 egg seems fully functional and I see it as a (if not *the*) way to distribute Zope 2 in the future. Coming to the actual problem: docutils is one third-party dependency of Zope that we track as a vendor import. Due to certain security problems that can arise when rendering reStructuredText, the docutils code was patched. Therefore, when shipping Zope with the standard docutils distribution (which can be conveniently pulled down as an egg), tests that assert the security of reStructuredText obviously fail. Andi converted the patch that was done to docutils to a monkey patch. It's available on the 'witsch-zope2.11-with-standard-docutils' branch. Not only am I not a big fan of monkey patches, I also think the behaviour of failing with an exception instead of presenting a warning is not very friendly. Tres explained the reason for patching docutils to raise a NotImplementedError in an email message on 8 July 2006:
In Zope2 land, the module is still available, and can be used by other code (which may not know of that issue). I'm *not* in favor of shipping an un-patched docutils until we work this out. For instance, perhaps we should be patching docutils to make the *default* settings disable file inclusion and 'raw'; then the trusted code which wanted to render reST which legitimately needed those features could enable them explicitly.
In my 'philikon-zope2.11-with-standard-docutils' branch, I have done that "working out": making all reStructuredText rendering from Zope immune to 'include' and 'raw' directives. Instead of raising a NotImplementedError, however, I simply changed the default settings in docutils (via monkey patch that's much less intrusive than swapping out methods). I propose to merge my branch to the trunk, therefore effectively changing the way we deal with the forbidden 'include' and 'raw' directives (presenting warning, not raising NotImplementedError), and allowing us to work with a standard docutils distribution. If wanted, I'd also be willing to backport this to older Zopes, though that might change their behaviour regarding reStructuredText in an unwanted way within a bugfix release. -- http://worldcookery.com -- Professional Zope documentation and training
--On 6. Juni 2007 18:01:27 +0200 Philipp von Weitershausen <philipp@weitershausen.de> wrote:
At the PIKTipi sprint this past weekend, Andi Zeidler, Andreas Jung and I worked on integrating ZODB 3.8 and Zope 3.4 into the Zope 2 trunk. Zope 3.4 has actually "exploded" into many eggs that are now independently managed and released. I therefore looked into a way to make Zope 2 dependent on these eggs instead of shipping with its own copies of the packages. There are clear benefits to that:
First, eggs are cool and the ongoing eggification is a good thing. However making such major changes for the next release I would like to discuss a few things or get answers. What will be the impacts on the source code distribution? How would you install "Zope from the sources" (especially if we don't include the 3rd-party eggs like docutils)? Working with eggs in a Zope environment requires that the eggs must be installed isolated within the softwarehome. We can't install Zope eggs within the Python environment (because it causes already a bunch of pain right now). Something like a buildout or workingenv environment would be necessary....also with a Zope-local installation of setuptools/easy_install!? Do we really want to encourage the end-user to update parts of the Zope core or the 3rd-party packages individually? I am skeptical about this because we always shipped Zope as a collection of tested and trusted components. Of course you could also updated package in the past manually however by using eggs we are lowering the barrier for a end-user to mess up his system. However I always see the advantages of the approach to bring updated or fixed packages much easier to the Zope user. Before stepping forward with your work (on the Zope trunk) I would like to get the "big picture" first. Andreas
On 7 Jun 2007, at 08:45 , Andreas Jung wrote:
--On 6. Juni 2007 18:01:27 +0200 Philipp von Weitershausen <philipp@weitershausen.de> wrote:
At the PIKTipi sprint this past weekend, Andi Zeidler, Andreas Jung and I worked on integrating ZODB 3.8 and Zope 3.4 into the Zope 2 trunk. Zope 3.4 has actually "exploded" into many eggs that are now independently managed and released. I therefore looked into a way to make Zope 2 dependent on these eggs instead of shipping with its own copies of the packages. There are clear benefits to that:
First, eggs are cool and the ongoing eggification is a good thing. However making such major changes for the next release I would like to discuss a few things or get answers.
With my email I might've given the wrong impression that I'm going to eggify Zope 2 and that that the Zope 2 release as we know it won't exist. That's not the case. My playground is, in fact, in a separate top-level project called "Zope2.buildout" that simply points externals to the "Zope2" project. "Zope2.buildout" is an experiment (deploying Zope 2 from eggs). I would like to discuss eggification and future releases of Zope 2 separately because I haven't fully worked out all the issues myself yet. My email purely concerns *one* problem (see the subject line) that is related to eggification: shipping with a standard docutils package. Eggs or not, I think being able to ship with an unmodified docutils has advantages (no more having to do vendor imports and applying the patches ourselves).
What will be the impacts on the source code distribution?
None. We can continue making tarballs that contain everything like we do now. I'm not saying we should (I haven't fully made up my mind), but it's certainly possible. The Zope2 egg only contains what is Zope 2, the rest is pulled in as dependencies. In the end you end up with the same amount of code, it's just coupled differently.
How would you install "Zope from the sources" (especially if we don't include the 3rd-party eggs like docutils)?
Again, the standard tarball install with configure; make; make installd doesn't have to go away. As far as the egg is concerned, you install it like you install every egg-based package "from the sources": download the tarball and do python setup.py install.
Working with eggs in a Zope environment requires that the eggs must be installed isolated within the softwarehome. We can't install Zope eggs within the Python environment (because it causes already a bunch of pain right now). Something like a buildout or workingenv environment would be necessary....also with a Zope-local installation of setuptools/easy_install!?
Sure, but that's not a Zope 2-specific problem. Which is why tools like buildout and workingenv exist. So I see no problem here.
Do we really want to encourage the end-user to update parts of the Zope core or the 3rd-party packages individually?
If it's not part of the Zope2 egg, then it's replaceable. I think that has advantages and can be a good thing. In the past people wanted to backport certain features that occurred in a later version of Zope 3 to an earlier version of Zope 2. Well, now they don't have to backport, they can just use a newer version of that particular package they're interested in. As far as 3rd party packages are concerned, it makes even more sense. pytz is a good example. Every so often, some country decides to change their timezone. Software running on old pytz releases (which were coupled to Zope releases) are stuck with old data. With eggs you're able to just upgrade pytz.
I am skeptical about this because we always shipped Zope as a collection of tested and trusted components.
We can still do that, egg-based or not, and attach a sticker "Warranty void if eggs are exchanged".
Of course you could also updated package in the past manually however by using eggs we are lowering the barrier for a end-user to mess up his system. However I always see the advantages of the approach to bring updated or fixed packages much easier to the Zope user.
Before stepping forward with your work (on the Zope trunk) I would like to get the "big picture" first.
Again, I'm not doing any changes to the Zope trunk, except the one thing I clearly laid out and proposed: being able to ship with a standard docutils package.
Philipp von Weitershausen wrote:
* as newer versions of specific zope.* or ZODB packages are released, you can individually install those or not. There'll be no need to wait for another Zope 2 release.
I actually see this as a downside to this. Having a single version number for "zope" is a very good thing imsnho. I have nightmare visions of the future where trying to help people debug their problem involves first trying to figure out what versions of the several million eggs involved they are using and then having to point out that their problem is because eggxyz 2.1.1 isn't compatible with eggabc 0.1a4 and that they really need to move to eggxyz 2.1.0 or eggabc 0.1a5rev2... (this is a problem Plone has really struggled with in the past)
* I actually went a bit further and eggified Zope 2 (it seemed the logical conclusion). I didn't really explode Zope 2, it's one big egg (except for a few things that are completely independent, such as ExtensionClass, Acquisition, DateTime, zLOG). Though currently not very comfortable to use, the Zope2 egg seems fully functional and I see it as a (if not *the*) way to distribute Zope 2 in the future.
I would still like to see a meta-egg for each of Zope 2 and Zope 3 that gives "prefered" versions of all eggs involved and is still the "standard" distribution... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On 7 Jun 2007, at 10:40 , Chris Withers wrote:
Philipp von Weitershausen wrote:
* as newer versions of specific zope.* or ZODB packages are released, you can individually install those or not. There'll be no need to wait for another Zope 2 release.
I actually see this as a downside to this. Having a single version number for "zope" is a very good thing imsnho. I have nightmare visions of the future where trying to help people debug their problem involves first trying to figure out what versions of the several million eggs involved they are using and then having to point out that their problem is because eggxyz 2.1.1 isn't compatible with eggabc 0.1a4 and that they really need to move to eggxyz 2.1.0 or eggabc 0.1a5rev2...
(this is a problem Plone has really struggled with in the past)
* I actually went a bit further and eggified Zope 2 (it seemed the logical conclusion). I didn't really explode Zope 2, it's one big egg (except for a few things that are completely independent, such as ExtensionClass, Acquisition, DateTime, zLOG). Though currently not very comfortable to use, the Zope2 egg seems fully functional and I see it as a (if not *the*) way to distribute Zope 2 in the future.
I would still like to see a meta-egg for each of Zope 2 and Zope 3 that gives "prefered" versions of all eggs involved and is still the "standard" distribution...
Yup, we can do this.
participants (3)
-
Andreas Jung -
Chris Withers -
Philipp von Weitershausen