I've created a bot process (see http://svn.repoze.org/playground/trunk/chris/zopesvnchecker/checker.py) that: - checks out the "trunk" subdir of each top-level directory in svn.zope.org - if the resulting directory does not have a "setup.py", we skip the directory - if the resulting directory does have a setup.py, we create a temporary virtualenv with setuptools installed and run the virtualenv's bin/python passing it "setup.py install". - We capture the results of the install, and print a summary. The "summary" results are at http://www.plope.com/static/misc/checker_results.txt . The "details" pickle generated by the script is at http://www.plope.com/static/misc/results.pickle . My analysis of the results are these: - A version conflict exists in a low-level dependency between a requirement for zope.traversing 3.4.0 and a requirement for zope.traversing>=3.5.0a1.dev-r78730 when installing many zope.* eggs using "setup.py install" (and easy_install), making it impractical for non-Zope people to actually install most of the interesting and released zope.* modules. This conflict needs to get fixed. Additionally, some modules (like zope.pagetemplate) should not fail with this dependency error in the first place; instead their dependencies need to become less conservative. - The only 10 packages (out of 80) in the zope.app namespace can be easy_installed. All the others fail with the above dependency error. Suggestions: - Find and fix the zope.traversing easy_install conflict. I'll try to debug this. - Institute a policy that all distributions that are released to the cheeseshop should be installable via "easy_install". IMO, if they are not installable this way, they should not be released to the cheeseshop, given the larger Python community's expectations. - Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this. - An automated test should ensure that easy_install does something reasonable with the latest release of each distribution. If the test fails, a new release should be made, or the dependency bug tracked down and fixed. My checker script could be made to do this, and I'd provide the work if someone told me which distributions to test, and how to maintain the list of distributions to test over time. - Ditch the idea of releasing separate distributions for each package in zope.app. The individual eggs typically can't be used outside of a zope appserver installation (and if they can, they probably shouldn't be in "zope.app", they should be in "zope" or they should be their own top-level package), and the "namespaciness" of zope.app is suspect when it's unlikely that anyone who is not a Zope committer will release a distribution which makes use of that namespace package. Their current overgranularity makes distributing them as separate eggs and releasing them to the cheeseshop a form of "cheeseshop pollution", especially given that so few of them can actually currently be installed using easy_install or "setup.py install". If cheeseshop is going to continue to be used as the index, I'd suggest creating a zope.app top-level svn module with a single setup.py in containing all the packages that are meant to go into zope.app. Version the resulting "zope.app" distribution as necessary instead of versioning many more granular "zope.app.*" distros. It's OK if some people don't use some of the functionality in the resulting egg, just toss everything in. There is precedent here in the Paste distribution. It has many submodules and does many things, but it comes in the form of a single egg. Yes, you lose the ability to make a bugfix in one subpackage and release it, but IIRC the intent is to trim zope.app down anyway, pushing libraryish things out to top-level or zope.* packages. Issues: - Who's in charge? Whomever you might be, to what extent do you agree/disagree with the above suggestions? If you agree with any, how can I help fix things? - I'm unsure how anybody is able to install Zope 3 right now using eggs, unless there's some fundamental difference in the way easy_install resolves dependencies vs. buildout. I have not looked at how Stephan's KGS works yet, though, so I'm sure I'm just missing some magic. - We should consider fixing setuptools install to detect conflicts before attempting to install anything. The current regime of "find conflicts halfway through an install" is IMO untenable in the long term for using eggs as a distribution mechanism. This may mean a very invasive change to both the package index and the client software, though.
Well, we had a bunch of conversation about this on IRC, and I think we figured out how to at least temporarily fix some of these failures. Someone released an distribution of zope.app.publisher to pypi (3.5.0a2) which requires zope.traversing 3.5.0a1.dev-r78730 but zope.traversing 3.5.0a1.dev-r78730 was never released to the cheeseshop. Buildout-based installs get around this because apparently they all manage to pin zope.traversing to 3.4.0 in some other way than an "install_requires" range dependency. To fix this, we either need to remove the zope.app.publisher 3.5.0a2 release, or release a newer one that depends on a "real" zope.traversing distribution with the new feature in it. Note that a non-solution (in terms of "the right thing", although it would work) would be to just release a zope.traversing 3.5.0-dev-r78730 egg.. dev eggs really shouldn't be registered on the cheese shop. I am leery of to signing up to do this, because I know absolutely zero about the software in question or what impact putting up new releases of zope.app.publisher or zope.traversing to the cheeseshop would have on buildout-based releases, what the release policy is, etc. It would be nice if someone who knows about this software could take the time to roll out new releases of these two packages (or remove the 3.5.0a2 release of zope.app.publisher) and try to do e.g. "easy_install zope.tales" after that to see if it works (although zope.tales shouldn't be depending on either of these packages, but that's a second-level problem). - C On Nov 13, 2007, at 4:29 PM, Chris McDonough wrote:
I've created a bot process (see http://svn.repoze.org/playground/trunk/chris/zopesvnchecker/ checker.py) that:
- checks out the "trunk" subdir of each top-level directory in svn.zope.org
- if the resulting directory does not have a "setup.py", we skip the directory
- if the resulting directory does have a setup.py, we create a temporary virtualenv with setuptools installed and run the virtualenv's bin/python passing it "setup.py install".
- We capture the results of the install, and print a summary.
The "summary" results are at http://www.plope.com/static/misc/checker_results.txt . The "details" pickle generated by the script is at http://www.plope.com/static/misc/results.pickle .
My analysis of the results are these:
- A version conflict exists in a low-level dependency between a requirement for zope.traversing 3.4.0 and a requirement for zope.traversing>=3.5.0a1.dev-r78730 when installing many zope.* eggs using "setup.py install" (and easy_install), making it impractical for non-Zope people to actually install most of the interesting and released zope.* modules. This conflict needs to get fixed. Additionally, some modules (like zope.pagetemplate) should not fail with this dependency error in the first place; instead their dependencies need to become less conservative.
- The only 10 packages (out of 80) in the zope.app namespace can be easy_installed. All the others fail with the above dependency error.
Suggestions:
- Find and fix the zope.traversing easy_install conflict. I'll try to debug this.
- Institute a policy that all distributions that are released to the cheeseshop should be installable via "easy_install". IMO, if they are not installable this way, they should not be released to the cheeseshop, given the larger Python community's expectations.
- Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this.
- An automated test should ensure that easy_install does something reasonable with the latest release of each distribution. If the test fails, a new release should be made, or the dependency bug tracked down and fixed. My checker script could be made to do this, and I'd provide the work if someone told me which distributions to test, and how to maintain the list of distributions to test over time.
- Ditch the idea of releasing separate distributions for each package in zope.app. The individual eggs typically can't be used outside of a zope appserver installation (and if they can, they probably shouldn't be in "zope.app", they should be in "zope" or they should be their own top-level package), and the "namespaciness" of zope.app is suspect when it's unlikely that anyone who is not a Zope committer will release a distribution which makes use of that namespace package. Their current overgranularity makes distributing them as separate eggs and releasing them to the cheeseshop a form of "cheeseshop pollution", especially given that so few of them can actually currently be installed using easy_install or "setup.py install". If cheeseshop is going to continue to be used as the index, I'd suggest creating a zope.app top-level svn module with a single setup.py in containing all the packages that are meant to go into zope.app. Version the resulting "zope.app" distribution as necessary instead of versioning many more granular "zope.app.*" distros. It's OK if some people don't use some of the functionality in the resulting egg, just toss everything in. There is precedent here in the Paste distribution. It has many submodules and does many things, but it comes in the form of a single egg. Yes, you lose the ability to make a bugfix in one subpackage and release it, but IIRC the intent is to trim zope.app down anyway, pushing libraryish things out to top-level or zope.* packages.
Issues:
- Who's in charge? Whomever you might be, to what extent do you agree/disagree with the above suggestions? If you agree with any, how can I help fix things?
- I'm unsure how anybody is able to install Zope 3 right now using eggs, unless there's some fundamental difference in the way easy_install resolves dependencies vs. buildout. I have not looked at how Stephan's KGS works yet, though, so I'm sure I'm just missing some magic.
- We should consider fixing setuptools install to detect conflicts before attempting to install anything. The current regime of "find conflicts halfway through an install" is IMO untenable in the long term for using eggs as a distribution mechanism. This may mean a very invasive change to both the package index and the client software, though.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
I've created a variant of the script I used to generate the results referenced in my original message within this thread. http://svn.repoze.org/playground/trunk/chris/zopesvnchecker/releasechecker.p... The new variant attempts to "easy_install" each cheeseshop project that has 'zope' in its author metadata (found via the cheeseshop's search XML-RPC interface) into a "pristine" virtualenv. This mimics what end-users would see if they attempted to install Zope-related packages from the cheeseshop without using zc.buildout. I'll set this up to run every few days. I'll attempt to send its output to zope-coders. - C On Nov 13, 2007, at 4:29 PM, Chris McDonough wrote:
I've created a bot process (see http://svn.repoze.org/playground/trunk/chris/zopesvnchecker/ checker.py) that:
- checks out the "trunk" subdir of each top-level directory in svn.zope.org
- if the resulting directory does not have a "setup.py", we skip the directory
- if the resulting directory does have a setup.py, we create a temporary virtualenv with setuptools installed and run the virtualenv's bin/python passing it "setup.py install".
- We capture the results of the install, and print a summary.
The "summary" results are at http://www.plope.com/static/misc/checker_results.txt . The "details" pickle generated by the script is at http://www.plope.com/static/misc/results.pickle .
My analysis of the results are these:
- A version conflict exists in a low-level dependency between a requirement for zope.traversing 3.4.0 and a requirement for zope.traversing>=3.5.0a1.dev-r78730 when installing many zope.* eggs using "setup.py install" (and easy_install), making it impractical for non-Zope people to actually install most of the interesting and released zope.* modules. This conflict needs to get fixed. Additionally, some modules (like zope.pagetemplate) should not fail with this dependency error in the first place; instead their dependencies need to become less conservative.
- The only 10 packages (out of 80) in the zope.app namespace can be easy_installed. All the others fail with the above dependency error.
Suggestions:
- Find and fix the zope.traversing easy_install conflict. I'll try to debug this.
- Institute a policy that all distributions that are released to the cheeseshop should be installable via "easy_install". IMO, if they are not installable this way, they should not be released to the cheeseshop, given the larger Python community's expectations.
- Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this.
- An automated test should ensure that easy_install does something reasonable with the latest release of each distribution. If the test fails, a new release should be made, or the dependency bug tracked down and fixed. My checker script could be made to do this, and I'd provide the work if someone told me which distributions to test, and how to maintain the list of distributions to test over time.
- Ditch the idea of releasing separate distributions for each package in zope.app. The individual eggs typically can't be used outside of a zope appserver installation (and if they can, they probably shouldn't be in "zope.app", they should be in "zope" or they should be their own top-level package), and the "namespaciness" of zope.app is suspect when it's unlikely that anyone who is not a Zope committer will release a distribution which makes use of that namespace package. Their current overgranularity makes distributing them as separate eggs and releasing them to the cheeseshop a form of "cheeseshop pollution", especially given that so few of them can actually currently be installed using easy_install or "setup.py install". If cheeseshop is going to continue to be used as the index, I'd suggest creating a zope.app top-level svn module with a single setup.py in containing all the packages that are meant to go into zope.app. Version the resulting "zope.app" distribution as necessary instead of versioning many more granular "zope.app.*" distros. It's OK if some people don't use some of the functionality in the resulting egg, just toss everything in. There is precedent here in the Paste distribution. It has many submodules and does many things, but it comes in the form of a single egg. Yes, you lose the ability to make a bugfix in one subpackage and release it, but IIRC the intent is to trim zope.app down anyway, pushing libraryish things out to top-level or zope.* packages.
Issues:
- Who's in charge? Whomever you might be, to what extent do you agree/disagree with the above suggestions? If you agree with any, how can I help fix things?
- I'm unsure how anybody is able to install Zope 3 right now using eggs, unless there's some fundamental difference in the way easy_install resolves dependencies vs. buildout. I have not looked at how Stephan's KGS works yet, though, so I'm sure I'm just missing some magic.
- We should consider fixing setuptools install to detect conflicts before attempting to install anything. The current regime of "find conflicts halfway through an install" is IMO untenable in the long term for using eggs as a distribution mechanism. This may mean a very invasive change to both the package index and the client software, though.
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Chris McDonough wrote:
- Institute a policy that all distributions that are released to the cheeseshop should be installable via "easy_install". IMO, if they are not installable this way, they should not be released to the cheeseshop, given the larger Python community's expectations.
+1
- Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this.
Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing.
- Ditch the idea of releasing separate distributions for each package in zope.app.
+1
The individual eggs typically can't be used outside of a zope appserver installation (and if they can, they probably shouldn't be in "zope.app", they should be in "zope" or they should be their own top-level package), and the "namespaciness" of zope.app is suspect when it's unlikely that anyone who is not a Zope committer will release a distribution which makes use of that namespace package. Their current overgranularity makes distributing them as separate eggs and releasing them to the cheeseshop a form of "cheeseshop pollution", especially given that so few of them can actually currently be installed using easy_install or "setup.py install". If cheeseshop is going to continue to be used as the index, I'd suggest creating a zope.app top-level svn module with a single setup.py in containing all the packages that are meant to go into zope.app. Version the resulting "zope.app" distribution as necessary instead of versioning many more granular "zope.app.*" distros. It's OK if some people don't use some of the functionality in the resulting egg, just toss everything in.
We can (and should) do that... almost :). For ages, the zope.app package has been treated as a namespace package because it always contained stuff that was releasable and stuff that wasn't (e.g. zope.app.homefolder or zope.app.css were experiments that probably shouldn't have been committed there in the first place, but alas, it happened). I don't think we should start releasing stuff that isn't in a releasable state.
There is precedent here in the Paste distribution. It has many submodules and does many things, but it comes in the form of a single egg.
Except that PasteScript and PasteDeploy are separate, which is just weird, especially considering the awkward naming of the eggs...
Yes, you lose the ability to make a bugfix in one subpackage and release it, but IIRC the intent is to trim zope.app down anyway, pushing libraryish things out to top-level or zope.* packages.
That *was* the plan at least. We moved several packages in the Zope 3.2 -> 3.3 upgrade, but a lot of people complained, so we pretty much stopped doing that. Except when recently the Python parts of zope.app.securitypolicy, zope.app.session were moved to zope.securitypolicy, zope.session in an attempt to be able to reuse the Python stuff without getting the browser views.
- Who's in charge? Whomever you might be, to what extent do you agree/disagree with the above suggestions? If you agree with any, how can I help fix things?
- I'm unsure how anybody is able to install Zope 3 right now using eggs, unless there's some fundamental difference in the way easy_install resolves dependencies vs. buildout. I have not looked at how Stephan's KGS works yet, though, so I'm sure I'm just missing some magic.
- We should consider fixing setuptools install to detect conflicts before attempting to install anything.
+1
The current regime of "find conflicts halfway through an install" is IMO untenable in the long term for using eggs as a distribution mechanism. This may mean a very invasive change to both the package index and the client software, though.
Yup. The problem is that people run custom indexes locally so setuptools would have to be able to deal with different kinds of index implementations.
On Wednesday 14 November 2007, Philipp von Weitershausen wrote:
- Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this.
Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing.
They don't anymore. I removed the find-links in all the packages I touched while creating Zope 3.4.0b2, because distribution contains so many non-ready packages and would have overridden several of the KGS package versions. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On Nov 14, 2007, at 6:05 AM, Stephan Richter wrote:
On Wednesday 14 November 2007, Philipp von Weitershausen wrote:
- Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this.
Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing.
They don't anymore. I removed the find-links in all the packages I touched while creating Zope 3.4.0b2, because distribution contains so many non-ready packages and would have overridden several of the KGS package versions.
So what makes this stuff work via buildout is all version-pinning now? Assuming you agree that packages released to the cheeseshop should be installable via easy_install, if distribution dependencies are version-pinned in a distribution's buildout.cfg, the same pin should be done in the setup.py install_requires= for that distribution. Otherwise running easy_install will never have the same result as running buildout when it comes to fetching dependencies for a particular distribution. In general, buildout.cfg and setup.py should share all dependency information, or the version-pin in buildout.cfg should disappear. - C
On Nov 14, 2007, at 8:38 AM, Chris McDonough wrote:
On Nov 14, 2007, at 6:05 AM, Stephan Richter wrote:
On Wednesday 14 November 2007, Philipp von Weitershausen wrote:
- Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this.
Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing.
They don't anymore. I removed the find-links in all the packages I touched while creating Zope 3.4.0b2, because distribution contains so many non-ready packages and would have overridden several of the KGS package versions.
So what makes this stuff work via buildout is all version-pinning now?
They aren't using the cheeseshop. They are using the KGS instead. Jim -- Jim Fulton Zope Corporation
On Nov 14, 2007, at 9:57 AM, Jim Fulton wrote:
On Nov 14, 2007, at 8:38 AM, Chris McDonough wrote:
On Nov 14, 2007, at 6:05 AM, Stephan Richter wrote:
On Wednesday 14 November 2007, Philipp von Weitershausen wrote:
- Figure out why buildout can (apparently) qsuccesfully install dependencies of currently failing zope.* eggs while easy_install can't. I probably won't be able to do this.
Because most buildouts specifically point to download.zope.org using find-links (so that way the necessary egg that's missing from PyPI is found), or they pin down the version of zope.traversing.
They don't anymore. I removed the find-links in all the packages I touched while creating Zope 3.4.0b2, because distribution contains so many non-ready packages and would have overridden several of the KGS package versions.
So what makes this stuff work via buildout is all version-pinning now?
They aren't using the cheeseshop. They are using the KGS instead.
Yup. If I run: bin/easy_install --index-url=http://download.zope.org/zope3.4 zope.tales It works, good news. I sent a mail before (which didn't appear here, I'm not sure why not, at least I didn't get it back) saying that I was not comfortable making a new release of zope.app.publisher and/or zope.traversing to fix the recurring dependency problem exposed by the bot on the cheeseshop, nor do I have access to do it. Either this needs to get done or the bogus zope.app.publisher 3.5.0a2 release needs to get taken down in order to at least temporarily fix installation from the cheeseshop for arbitrary users. Is someone with familiarity of these packages and access to change those projects on the cheeseshop willing to do that? In the meantime, whose feet would I be shooting if I changed setup.py dependency info so that, e.g. zope.tales (and the like) didn't pull in as many dependencies? Installing zope.tales (as an example, I'm sure its as a result of a transitive dependency lower in the dep chain) right now gets us ~ 75 other distribution, including very unrelated ones: ./docutils-0.4-py2.4.egg ./pytz-2007g-py2.4.egg ./RestrictedPython-3.4.2-py2.4.egg ./ZConfig-2.5-py2.4.egg ./zdaemon-2.0.1-py2.4.egg ./ZODB3-3.8.0b4-py2.4-macosx-10.3-i386.egg ./zodbcode-3.4.0-py2.4.egg ./zope.annotation-3.4.0-py2.4.egg ./zope.app.applicationcontrol-3.4.1-py2.4.egg ./zope.app.appsetup-3.4.1-py2.4.egg ./zope.app.authentication-3.4.1-py2.4.egg ./zope.app.basicskin-3.4.0-py2.4.egg ./zope.app.broken-3.4.0-py2.4.egg ./zope.app.component-3.4.1-py2.4.egg ./zope.app.container-3.5.3-py2.4-macosx-10.3-i386.egg ./zope.app.content-3.4.0-py2.4.egg ./zope.app.debug-3.4.0-py2.4.egg ./zope.app.dependable-3.4.0-py2.4.egg ./zope.app.error-3.5.1-py2.4.egg ./zope.app.exception-3.4.1-py2.4.egg ./zope.app.folder-3.4.0-py2.4.egg ./zope.app.form-3.4.1-py2.4.egg ./zope.app.generations-3.4.1-py2.4.egg ./zope.app.http-3.4.1-py2.4.egg ./zope.app.i18n-3.4.4-py2.4.egg ./zope.app.interface-3.4.0-py2.4.egg ./zope.app.locales-3.4.0-py2.4.egg ./zope.app.pagetemplate-3.4.0-py2.4.egg ./zope.app.principalannotation-3.4.0-py2.4.egg ./zope.app.publication-3.4.2-py2.4.egg ./zope.app.publisher-3.4.1-py2.4.egg ./zope.app.renderer-3.4.0-py2.4.egg ./zope.app.rotterdam-3.4.1-py2.4.egg ./zope.app.schema-3.4.0-py2.4.egg ./zope.app.security-3.4.0-py2.4.egg ./zope.app.session-3.5.1-py2.4.egg ./zope.app.testing-3.4.1-py2.4.egg ./zope.app.wsgi-3.4.0-py2.4.egg ./zope.app.zapi-3.4.0-py2.4.egg ./zope.app.zcmlfiles-3.4.3-py2.4.egg ./zope.app.zopeappgenerations-3.4.0-py2.4.egg ./zope.cachedescriptors-3.4.0-py2.4.egg ./zope.component-3.4.0-py2.4.egg ./zope.configuration-3.4.0-py2.4.egg ./zope.contenttype-3.4.0-py2.4.egg ./zope.copypastemove-3.4.0-py2.4.egg ./zope.datetime-3.4.0-py2.4.egg ./zope.deferredimport-3.4.0-py2.4.egg ./zope.deprecation-3.4.0-py2.4.egg ./zope.dottedname-3.4.2-py2.4.egg ./zope.dublincore-3.4.0-py2.4.egg ./zope.error-3.5.1-py2.4.egg ./zope.event-3.4.0-py2.4.egg ./zope.exceptions-3.4.0-py2.4.egg ./zope.filerepresentation-3.4.0-py2.4.egg ./zope.formlib-3.4.0-py2.4.egg ./zope.hookable-3.4.0-py2.4-macosx-10.3-i386.egg ./zope.i18n-3.4.0-py2.4.egg ./zope.i18nmessageid-3.4.3-py2.4-macosx-10.3-i386.egg ./zope.interface-3.4.0-py2.4-macosx-10.3-i386.egg ./zope.lifecycleevent-3.4.0-py2.4.egg ./zope.location-3.4.0-py2.4.egg ./zope.minmax-1.0-py2.4.egg ./zope.modulealias-3.4.0-py2.4.egg ./zope.pagetemplate-3.4.0-py2.4.egg ./zope.proxy-3.4.0-py2.4-macosx-10.3-i386.egg ./zope.publisher-3.4.1-py2.4.egg ./zope.schema-3.4.0-py2.4.egg ./zope.security-3.4.0-py2.4-macosx-10.3-i386.egg ./zope.session-3.4.1-py2.4.egg ./zope.size-3.4.0-py2.4.egg ./zope.structuredtext-3.4.0-py2.4.egg ./zope.tal-3.4.0-py2.4.egg ./zope.tales-3.4.0-py2.4.egg ./zope.testing-3.5.1-py2.4.egg ./zope.thread-3.4-py2.4.egg ./zope.traversing-3.4.0-py2.4.egg - C
On Wednesday 14 November 2007, Chris McDonough wrote:
In the meantime, whose feet would I be shooting if I changed setup.py dependency info so that, e.g. zope.tales (and the like) didn't pull in as many dependencies? Installing zope.tales (as an example, I'm sure its as a result of a transitive dependency lower in the dep chain) right now gets us ~ 75 other distribution, including very unrelated ones:
I would welcomes this very much, because I use several packages outside of Zope as well. Now that we have the KGS, I feel much more comfortable with playing with the packages and their dependencies. Note that Marius worked on a nifty little gtk-based tool during the Foilage Sprint that shows you the full dependency tree. http://mg.pov.lt/gtkeggdeps/ Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On Nov 14, 2007, at 9:57 AM, Jim Fulton wrote:
So what makes this stuff work via buildout is all version-pinning now?
They aren't using the cheeseshop. They are using the KGS instead.
They are currently also pinned. zope.tal's buildout.cfg contains: [versions] zope.traversing = 3.4.0 zope.app.publisher = 3.4.0 zope.tal shouldn't depend on either of these, so I'd like to remove both the pinning in buildout.cfg and fix the dependency bug that caused them to need to be added in the first place. The actual bug is that zope.tal depends on zope.i18n (which has inappropriate dependencies, but we'll fix that separately). zope.tal only uses the zope.i18n for a single import in dummyengine.py. It imports and uses the zope.i18n.interfaces.ITranslationDomain interface: from zope.i18n.interfaces import ITranslationDomain class DummyTranslationDomain(object): implements(ITranslationDomain) I removed ITranslationDomain implementation from the DummyTranslationDomain, all the tests still pass. I've also removed the pinning in buildout.cfg. I've checked this in. If you do "setup.py install" within zope.tal now, the only distributions required and downloaded during that process are zope.i18nmessageid and zope.interface. Likewise for buildout. I've also made this module possible to test via "setup.py test". - C
On Wednesday 14 November 2007, Chris McDonough wrote:
zope.tal's buildout.cfg contains:
[versions] zope.traversing = 3.4.0 zope.app.publisher = 3.4.0
zope.tal shouldn't depend on either of these, so I'd like to remove both the pinning in buildout.cfg and fix the dependency bug that caused them to need to be added in the first place.
Just remove the section. I removed them in the packages I saw when I was doing the eggs. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On Wednesday 14 November 2007, Philipp von Weitershausen wrote:
Yes, you lose the ability to make a bugfix in one subpackage and release it, but IIRC the intent is to trim zope.app down anyway, pushing libraryish things out to top-level or zope.* packages.
That *was* the plan at least. We moved several packages in the Zope 3.2 -> 3.3 upgrade, but a lot of people complained, so we pretty much stopped doing that. Except when recently the Python parts of zope.app.securitypolicy, zope.app.session were moved to zope.securitypolicy, zope.session in an attempt to be able to reuse the Python stuff without getting the browser views.
There are about 3-4 packages in zope.app with which we have to do the same, so that I can actually create a setup that does not have the ZMI, zope.formlib and zope.app.form. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On Nov 14, 2007, at 4:46 AM, Philipp von Weitershausen wrote: ...
The individual eggs typically can't be used outside of a zope appserver installation (and if they can, they probably shouldn't be in "zope.app", they should be in "zope" or they should be their own top-level package), and the "namespaciness" of zope.app is suspect when it's unlikely that anyone who is not a Zope committer will release a distribution which makes use of that namespace package. Their current overgranularity makes distributing them as separate eggs and releasing them to the cheeseshop a form of "cheeseshop pollution", especially given that so few of them can actually currently be installed using easy_install or "setup.py install". If cheeseshop is going to continue to be used as the index, I'd suggest creating a zope.app top-level svn module with a single setup.py in containing all the packages that are meant to go into zope.app. Version the resulting "zope.app" distribution as necessary instead of versioning many more granular "zope.app.*" distros. It's OK if some people don't use some of the functionality in the resulting egg, just toss everything in.
We can (and should) do that... almost :). For ages, the zope.app package has been treated as a namespace package because it always contained stuff that was releasable and stuff that wasn't (e.g. zope.app.homefolder or zope.app.css were experiments that probably shouldn't have been committed there in the first place, but alas, it happened).
If we put a lot of these packages back together, we should still treat zope.app as a namespace package. Let's not go back on that. One caution is that distutils and setuptools provide no help with multiple eggs that contain the same package. This means that someone can install the combined and separate packages at the same time (like they can now install ZODB3 and transaction) leading to subtle bugs. Probably the biggest mistake we made in this process was moving to fast. Whatever we do to "fix" the current situation, let's be very careful not to make it worse.
I don't think we should start releasing stuff that isn't in a releasable state.
You mean we never should have started. :) ...
Yes, you lose the ability to make a bugfix in one subpackage and release it, but IIRC the intent is to trim zope.app down anyway, pushing libraryish things out to top-level or zope.* packages.
That *was* the plan at least. We moved several packages in the Zope 3.2 -> 3.3 upgrade, but a lot of people complained, so we pretty much stopped doing that. Except when recently the Python parts of zope.app.securitypolicy, zope.app.session were moved to zope.securitypolicy, zope.session in an attempt to be able to reuse the Python stuff without getting the browser views.
IOW, we have to balance cleanliness with disruption. For a while, those of us working on Zope 3 revelled in "refactor mercilessly". This was fine before Zope 3 was widely adopted. We have since leaned the value of mercy. :)
- Who's in charge? Whomever you might be, to what extent do you agree/disagree with the above suggestions? If you agree with any, how can I help fix things? - I'm unsure how anybody is able to install Zope 3 right now using eggs, unless there's some fundamental difference in the way easy_install resolves dependencies vs. buildout. I have not looked at how Stephan's KGS works yet, though, so I'm sure I'm just missing some magic. - We should consider fixing setuptools install to detect conflicts before attempting to install anything.
+1
I strongly suspect that this will be impractical.
The current regime of "find conflicts halfway through an install" is IMO untenable in the long term for using eggs as a distribution mechanism. This may mean a very invasive change to both the package index and the client software, though.
Yup. The problem is that people run custom indexes locally so setuptools would have to be able to deal with different kinds of index implementations.
Huh? There is a defined index API. The custom indexes conform to this afaik. Things could certainly be made faster if the index provided access to more meta data, still resolving conflicts will still be potentially computationally very intensive. I still believe that a setuptools prefer-final switch would go a long way towards addressing this problem. I'll also not that, AFAIK, the current problems weren't due to true conflicts, but due to packages in the cheeseshop requiring package releases that weren't in the cheeseshop. Jim -- Jim Fulton Zope Corporation
On Wednesday 14 November 2007, Jim Fulton wrote:
For a while, those of us working on Zope 3 revelled in "refactor mercilessly". This was fine before Zope 3 was widely adopted. We have since leaned the value of mercy. :)
LOL! (This should make it into the eternal list of Zope quotes.) Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
participants (4)
-
Chris McDonough -
Jim Fulton -
Philipp von Weitershausen -
Stephan Richter