-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi everyone,
Here's a followup on a docs.zope.org automation task I took over during one of the Zope developer IRC metings[1]. The task was to provide individual package documentation, if it exists, directly underneath docs.zope.org, e.g.:
http://docs.zope.org/zope.event/
... and keep those docs updated from the current trunk documentation.
I expanded that task a little bit by generating content that shows the list of packages divided by project and makes it obvious which packages have documentation and which don't. All of that is in a buildout that automatically generates the docs.zope.org website and does away with the single static HTML page with the "dead" new new new new.zope.org design and uses stock Sphinx design.
If you visit docs.zope.org now you'll notice that the content on the main page has not changed much, it's just been updated and re-ordered slightly. What's new are those links I prefixed with "Package documentation", those point to the autogenerated index pages for each project. As noted on top of most of these index pages, if a package is listed but not linked it doesn't have documentation the scripts can recognize.
Currently the buildout can recognize the standard Sphinx documentation layout with a folder named "doc" or "docs" in the package root and a Sphinx configuration and content therein. The buildout can also use a kludge and pull in content that is documented using "z3c.recipe.sphinxdoc" maintained in a separate buildout, but I'm hoping those project authors would consider using the standard layout as exemplified by e.g. zope.event[2] and championed by e.g. Tres, Christian Theune, and several others (including me).
There's a README in the buildout that explains how to do specific tasks[3]. If you have any questions let me know.
jens
[1] http://docs.zope.org/zopetoolkit/zope-dev/zope-dev-20100727.html [2] http://svn.zope.org/zope.event/trunk/docs/ [3] http://svn.zope.org/docs.zope.org_website/
On 07/31/2010 07:22 PM, Jens Vagelpohl wrote:
Here's a followup on a docs.zope.org automation task I took over during one of the Zope developer IRC metings[1]. The task was to provide individual package documentation, if it exists, directly underneath docs.zope.org, e.g.:
Really cool!
Wouldn't it be good to put this under /package/zope.event to avoid potential naming conflicts? I realize they're rare, but I can imagine that a project foo could exist that wants to expose its documentation separately from project foo. Perhaps "Zope" would be a good example. :)
Regards,
Martijn
P.S. It'd be good if the ZODB section had at least a link to zodb.org. Links to buildout.org and grok.zope.org might also be useful.
On 8/2/10 13:40 , Martijn Faassen wrote:
On 07/31/2010 07:22 PM, Jens Vagelpohl wrote:
Here's a followup on a docs.zope.org automation task I took over during one of the Zope developer IRC metings[1]. The task was to provide individual package documentation, if it exists, directly underneath docs.zope.org, e.g.:
Really cool!
Wouldn't it be good to put this under /package/zope.event to avoid potential naming conflicts? I realize they're rare, but I can imagine that a project foo could exist that wants to expose its documentation separately from project foo. Perhaps "Zope" would be a good example. :)
Perhaps we should encourage people to upload documentation to packages.python.org ?
Wichert.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Martijn,
Really cool!
Wouldn't it be good to put this under /package/zope.event to avoid potential naming conflicts? I realize they're rare, but I can imagine that a project foo could exist that wants to expose its documentation separately from project foo. Perhaps "Zope" would be a good example. :)
The specific desire was to offer URLs http://docs.zope.org/<package>, that's why I created it that way. It's not that hard to add a subdirectory, but I'd rather have consensus first. Me personally I like the straight package URLs without an intermediary subfolder.
P.S. It'd be good if the ZODB section had at least a link to zodb.org. Links to buildout.org and grok.zope.org might also be useful.
Take a look at the "Related Links" on the left hand side, they're all in there except for grok.zope.org, which I just added.
jens
On Saturday, July 31, 2010, Jens Vagelpohl wrote:
Currently the buildout can recognize the standard Sphinx documentation layout with a folder named "doc" or "docs" in the package root and a Sphinx configuration and content therein. The buildout can also use a kludge and pull in content that is documented using "z3c.recipe.sphinxdoc" maintained in a separate buildout, but I'm hoping those project authors would consider using the standard layout as exemplified by e.g. zope.event[2] and championed by e.g. Tres, Christian Theune, and several others (including me).
We have put a lot of energy into putting the full documentation of packages into the package's long description and most packages have this layout (I know, because I did it for most. ;-) So it might be good to use the long description of the package, if the other two methods fail. This would turn on documentation for a lot of packages and we can move them over slowly.
But otherwise, it is great to see such great progress on docs.zope.org!
Regards, Stephan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 15:51 , Stephan Richter wrote:
We have put a lot of energy into putting the full documentation of packages into the package's long description and most packages have this layout (I know, because I did it for most. ;-) So it might be good to use the long description of the package, if the other two methods fail. This would turn on documentation for a lot of packages and we can move them over slowly.
I'll have to look at that. Currently, the documentation builder does not do any introspection on the package itself, mostly because I do not want to fully install the package and pull in all dependencies. Maybe there's a simple way that does not require full installation.
jens
On Monday, August 02, 2010, Jens Vagelpohl wrote:
'll have to look at that. Currently, the documentation builder does not do any introspection on the package itself, mostly because I do not want to fully install the package and pull in all dependencies. Maybe there's a simple way that does not require full installation.
I agree. This does not build the package:
python setup.py --long-description
Regards, Stephan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 16:36 , Stephan Richter wrote:
On Monday, August 02, 2010, Jens Vagelpohl wrote:
'll have to look at that. Currently, the documentation builder does not do any introspection on the package itself, mostly because I do not want to fully install the package and pull in all dependencies. Maybe there's a simple way that does not require full installation.
I agree. This does not build the package:
python setup.py --long-description
Thanks for the hint, I'll try that. Can you give me a sample package where the long description is supposed to be the main documentation? And what's the output from that? If it's ReST I'd have to find a way to convert it to HTML on the fly... <sigh>
jens
On 2 August 2010 22:40, Jens Vagelpohl jens@dataflake.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 16:36 , Stephan Richter wrote:
On Monday, August 02, 2010, Jens Vagelpohl wrote:
'll have to look at that. Currently, the documentation builder does not do any introspection on the package itself, mostly because I do not want to fully install the package and pull in all dependencies. Maybe there's a simple way that does not require full installation.
I agree. This does not build the package:
python setup.py --long-description
Thanks for the hint, I'll try that. Can you give me a sample package where the long description is supposed to be the main documentation? And what's the output from that? If it's ReST I'd have to find a way to convert it to HTML on the fly... <sigh>
z3c.form, I'd guess. :)
For other examples, look at e.g. z3c.caching, plone.caching or plone.testing.
You can do:
$ python setup.py --long-description | rst2html.py > doc.html
to create an HTML file.
Martin
On 8/2/10 16:46 , Martin Aspeli wrote:
On 2 August 2010 22:40, Jens Vagelpohljens@dataflake.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 16:36 , Stephan Richter wrote:
On Monday, August 02, 2010, Jens Vagelpohl wrote:
'll have to look at that. Currently, the documentation builder does not do any introspection on the package itself, mostly because I do not want to fully install the package and pull in all dependencies. Maybe there's a simple way that does not require full installation.
I agree. This does not build the package:
python setup.py --long-description
Thanks for the hint, I'll try that. Can you give me a sample package where the long description is supposed to be the main documentation? And what's the output from that? If it's ReST I'd have to find a way to convert it to HTML on the fly...<sigh>
z3c.form, I'd guess. :)
Doesn't z3c.form have sphinx documentation?
Wichert.
On Monday, August 02, 2010, Jens Vagelpohl wrote:
Thanks for the hint, I'll try that. Can you give me a sample package where the long description is supposed to be the main documentation? And what's the output from that? If it's ReST I'd have to find a way to convert it to HTML on the fly... <sigh>
zope.security (http://pypi.python.org/pypi/zope.security)
Once you have docutils installed, you can do:
py26 setup.py --long-description | rst2html
rst2html have many options that allow you to modify the style, including the ability to specify a stylesheet.
Regards, Stephan
On Monday, August 02, 2010, Stephan Richter wrote:
py26 setup.py --long-description | rst2html
rst2html have many options that allow you to modify the style, including the ability to specify a stylesheet.
Oh, btw, this is also the method that we used to verify that the long descriptions are proper ReST. ;-)
Regards, Stephan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 16:50 , Stephan Richter wrote:
On Monday, August 02, 2010, Stephan Richter wrote:
py26 setup.py --long-description | rst2html
rst2html have many options that allow you to modify the style, including the ability to specify a stylesheet.
Oh, btw, this is also the method that we used to verify that the long descriptions are proper ReST. ;-)
I hereby declare that verifying proper ReST is OUT OF SCOPE for docs.zope.org ;-)
jens
On 08/02/2010 04:53 PM, Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 16:50 , Stephan Richter wrote:
On Monday, August 02, 2010, Stephan Richter wrote:
py26 setup.py --long-description | rst2html
rst2html have many options that allow you to modify the style, including the ability to specify a stylesheet.
Oh, btw, this is also the method that we used to verify that the long descriptions are proper ReST. ;-)
I hereby declare that verifying proper ReST is OUT OF SCOPE for docs.zope.org ;-)
It's out of scope for PyPI too, what it does is just display the plain text if there are any ReST errors. And when the developer sees that, that's a clue to go and fix the ReST and use the above technique.
So as long as there's a clue that "it's not working" we'll be fine.
Regards,
Martijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 16:48 , Stephan Richter wrote:
On Monday, August 02, 2010, Jens Vagelpohl wrote:
Thanks for the hint, I'll try that. Can you give me a sample package where the long description is supposed to be the main documentation? And what's the output from that? If it's ReST I'd have to find a way to convert it to HTML on the fly... <sigh>
zope.security (http://pypi.python.org/pypi/zope.security)
Once you have docutils installed, you can do:
py26 setup.py --long-description | rst2html
rst2html have many options that allow you to modify the style, including the ability to specify a stylesheet.
OK, I'll see what I can do with that. I'll probably end up using the long description as last fallback. So you will see every package linked to *something*, even if it's just a page with a few words on it.
jens
On Monday, August 02, 2010, Jens Vagelpohl wrote:
OK, I'll see what I can do with that. I'll probably end up using the long description as last fallback. So you will see every package linked to *something*, even if it's just a page with a few words on it.
Just FYI: If there is no long description (which is very unlikely), then the string "UNKNOWN" is returned from the --long-description call.
Regards, Stephan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 17:00 , Stephan Richter wrote:
On Monday, August 02, 2010, Jens Vagelpohl wrote:
OK, I'll see what I can do with that. I'll probably end up using the long description as last fallback. So you will see every package linked to *something*, even if it's just a page with a few words on it.
Just FYI: If there is no long description (which is very unlikely), then the string "UNKNOWN" is returned from the --long-description call.
OK, that's an excellent flag to give up and say "this package has no documentation I can deal with".
jens
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 16:36 , Stephan Richter wrote:
On Monday, August 02, 2010, Jens Vagelpohl wrote:
'll have to look at that. Currently, the documentation builder does not do any introspection on the package itself, mostly because I do not want to fully install the package and pull in all dependencies. Maybe there's a simple way that does not require full installation.
I agree. This does not build the package:
python setup.py --long-description
Can someone tell me how to do that when I am in Python code already? Given the path to the checkout, can I use some setuptools/pkg_resources or pkginfo magic to get at this data?
My specific problem is this: Many packages define an intermediate folder, like "src", in which the actual code path is stored. When someone runs "setup.py egg_info" then the .egg-info folder ends up in "src" and not at the root. Trying to use e.g. pkginfo.Develop or pkg_resources.find_distributions with the root folder won't find the .egg-info folder in the "src" subfolder. I'd have to do something horrible like os.walk to find distributions, which I really don't want.
jens
On 08/02/2010 05:25 PM, Jens Vagelpohl wrote:
python setup.py --long-description
Can someone tell me how to do that when I am in Python code already? Given the path to the checkout, can I use some setuptools/pkg_resources or pkginfo magic to get at this data?
Oh, this is where I always get scared because I always get lost. :)
The pkg_resources documentation is here:
http://packages.python.org/distribute/pkg_resources.html
You can probably create a Distribution object somehow (handwave) from a path (whether that's the path of the package or the path the package is in, not sure).
Then maybe 'get_metadata('long_description') on the distribution object, but maybe not as one seems to have to supply metadata when constructing a distribution..
But again, I always get lost here, perhaps you need an Environment or a WorkingSet first, etc.
Regards,
Martijn
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 18:26 , Martijn Faassen wrote:
http://packages.python.org/distribute/pkg_resources.html
You can probably create a Distribution object somehow (handwave) from a path (whether that's the path of the package or the path the package is in, not sure).
The issue, just like with pkginfo.Develop, is that I can't find any function or method that finds package information in subfolders. If you look at...
http://packages.python.org/distribute/pkg_resources.html#getting-or-creating...
...there's this function find_distributions which supposedly offers subfolder searching, but it just doesn't work. At least it doesn't do what I would expect when I read the documentation. It finds nothing.
jens
On 08/02/2010 06:34 PM, Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 18:26 , Martijn Faassen wrote:
http://packages.python.org/distribute/pkg_resources.html
You can probably create a Distribution object somehow (handwave) from a path (whether that's the path of the package or the path the package is in, not sure).
The issue, just like with pkginfo.Develop, is that I can't find any function or method that finds package information in subfolders. If you look at...
http://packages.python.org/distribute/pkg_resources.html#getting-or-creating...
...there's this function find_distributions which supposedly offers subfolder searching, but it just doesn't work. At least it doesn't do what I would expect when I read the documentation. It finds nothing.
Yeah, unfortunately this is generally the point where I find myself being lost too.
Regards,
Martijn
On Mon, Aug 02, 2010 at 06:34:21PM +0200, Jens Vagelpohl wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 18:26 , Martijn Faassen wrote:
http://packages.python.org/distribute/pkg_resources.html
You can probably create a Distribution object somehow (handwave) from a path (whether that's the path of the package or the path the package is in, not sure).
The issue, just like with pkginfo.Develop, is that I can't find any function or method that finds package information in subfolders. If you look at...
http://packages.python.org/distribute/pkg_resources.html#getting-or-creating...
...there's this function find_distributions which supposedly offers subfolder searching, but it just doesn't work. At least it doesn't do what I would expect when I read the documentation. It finds nothing.
Do you have access to an source distribution (tarball or zip) for the package involved?
In the source distribution the PKG-INFO file is always at the top level.
jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)
iEYEARECAAYFAkxW840ACgkQRAx5nvEhZLK0MgCfWNsSL47dimbU9TlIZNSRq07h HLIAoIYYLdYUsA5PhYrtqhu9EaVk8Nip =GbbC -----END PGP SIGNATURE----- _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Brian,
Do you have access to an source distribution (tarball or zip) for the package involved?
In the source distribution the PKG-INFO file is always at the top level.
No, these are trunk checkouts.
jens
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jens Vagelpohl wrote:
On 8/2/10 18:26 , Martijn Faassen wrote:
You can probably create a Distribution object somehow (handwave) from a path (whether that's the path of the package or the path the package is in, not sure).
The issue, just like with pkginfo.Develop, is that I can't find any function or method that finds package information in subfolders. If you look at...
http://packages.python.org/distribute/pkg_resources.html#getting-or-creating...
...there's this function find_distributions which supposedly offers subfolder searching, but it just doesn't work. At least it doesn't do what I would expect when I read the documentation. It finds nothing.
The 'pkginfo' utility knows how to extract package metadata from stuff installed on the path:
http://packages.python.org/pkginfo/distributions.html#introspecting-installe...
as well as from checkouts:
http://packages.python.org/pkginfo/distributions.html#introspecting-developm...
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/3/10 00:19 , Tres Seaver wrote:
The 'pkginfo' utility knows how to extract package metadata from stuff installed on the path:
http://packages.python.org/pkginfo/distributions.html#introspecting-installe...
as well as from checkouts:
http://packages.python.org/pkginfo/distributions.html#introspecting-developm...
Just like the counterparts in setuptools this only works if you pass it a path that has a PKG-INFO or .egg-info directly in that folder. The problem is, with many packages those are not in the root folder of the package after doing a setup.py egg_info. Those packages normally use the "package_dir" argument to their setup call in setup.py, but that's information I can't get to, it's in the argument list for the setup call.
jens
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/2/10 15:51 , Stephan Richter wrote:
We have put a lot of energy into putting the full documentation of packages into the package's long description and most packages have this layout (I know, because I did it for most. ;-) So it might be good to use the long description of the package, if the other two methods fail. This would turn on documentation for a lot of packages and we can move them over slowly.
This is now implemented as fallback, take another look at the package listing, like the ZTK list[1]. I had to resort to executing "setup.py - --long-description" in a subprocess, unfortunately. Never found out how to do that programmatically.
jens
[1] http://docs.zope.org/ztkpackages.html
On Tue, Aug 03, 2010 at 10:42:42PM +0200, Jens Vagelpohl wrote:
On 8/2/10 15:51 , Stephan Richter wrote:
We have put a lot of energy into putting the full documentation of packages into the package's long description and most packages have this layout (I know, because I did it for most. ;-) So it might be good to use the long description of the package, if the other two methods fail. This would turn on documentation for a lot of packages and we can move them over slowly.
This is now implemented as fallback, take another look at the package listing, like the ZTK list[1]. I had to resort to executing "setup.py --long-description" in a subprocess, unfortunately. Never found out how to do that programmatically.
From a source checkout? There's no other way. Many packages compute their long_description by running code (which reads and concatenates multiple text files).
jens
Marius Gedminas
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/3/10 22:59 , Marius Gedminas wrote:
On Tue, Aug 03, 2010 at 10:42:42PM +0200, Jens Vagelpohl wrote:
listing, like the ZTK list[1]. I had to resort to executing "setup.py --long-description" in a subprocess, unfortunately. Never found out how to do that programmatically.
From a source checkout? There's no other way. Many packages compute their long_description by running code (which reads and concatenates multiple text files).
I understand that. But it must be possible to do that programatically in my code. I mean, "setup.py --long-description" obviously executes Python code, which I may be able to execute myself in my current interpreter session by importing and executing stuff from setuptools. Having to invoke another Python interpreter in a subshell because that's too complicated to do any other way is awful.
jens
On Tue, Aug 3, 2010 at 11:05 PM, Jens Vagelpohl jens@dataflake.org wrote:
I understand that. But it must be possible to do that programatically in my code. I mean, "setup.py --long-description" obviously executes Python code, which I may be able to execute myself in my current interpreter session by importing and executing stuff from setuptools. Having to invoke another Python interpreter in a subshell because that's too complicated to do any other way is awful.
Maybe this gets you a bit further:
import os from setuptools.sandbox import run_setup
package = os.path.join(os.curdir, 'zope.component') os.chdir(package) run_setup('setup.py', ['--long-description']) os.chdir(os.pardir)
The run_setup call seems to dump the output to sys.stdout which might be a bit painful to grab.
Hanno
Jens Vagelpohl wrote:
I understand that. But it must be possible to do that programatically in my code. I mean, "setup.py --long-description" obviously executes Python code, which I may be able to execute myself in my current interpreter session by importing and executing stuff from setuptools.
Yeah, this is why setup.py sucks ;-)
I've just been doing some build and release tools for a customer that involve this sort of stuff, I resorted to:
with nested(Replacer(),OutputCapture()) as (r,o): r.replace('sys.argv',['X','egg_info']) try: sys.path.insert(0,path) curdir = os.getcwd() os.chdir(path) # yuk, but setup.py is yuk anyway! if 'setup' in sys.modules: del sys.modules['setup'] import setup finally: os.chdir(curdir) sys.path.pop(0)
..in my tests, the del from sys.modules is to allow this to by done multiple times in one process.
Replacer is http://packages.python.org/testfixtures/mocking.html#the-context-manager
OutputCapture looks like:
class OutputCapture:
def __enter__(self): self.original_stdout = sys.stdout self.original_stderr = sys.stderr self.output = sys.stdout = sys.stderr = StringIO() return self
def __exit__(self,*args): sys.stdout = self.original_stdout sys.stderr = self.original_stderr
def compare(self,expected): compare(expected.strip(),self.output.getvalue().strip())
Having to invoke another Python interpreter in a subshell because that's too complicated to do any other way is awful.
Yeah, I do resort to this for some things too ;-)
http://packages.python.org/execute/use.html
cheers,
Chris
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Chris,
I've just been doing some build and release tools for a customer that involve this sort of stuff, I resorted to:
<snip lots of code>
Yikes. Maybe the subshell isn't so bad after all :-P
jens
Jens Vagelpohl wrote:
I've just been doing some build and release tools for a customer that involve this sort of stuff, I resorted to:
<snip lots of code>
Yikes. Maybe the subshell isn't so bad after all :-P
Yeah ;-) Although getting the right python path set up can be a pain...
Here's the actual version I use to do an in-process sdist:
try: orig_argv = sys.argv orig_stdout,orig_stderr = sys.stdout,sys.stderr curdir = os.getcwd() sys.argv = sys.argv[0:1]+['egg_info', '--tag-svn-revision', '--tag-date', 'sdist'] sys.path.insert(0,location) os.chdir(location) sys.stdout,sys.stderr = StringIO(),StringIO() # yuk, but setup.py is yuk anyway! if 'setup' in sys.modules: del sys.modules['setup'] import setup finally: os.chdir(curdir) sys.path.pop(0) sys.stdout,sys.stderr = orig_stdout,orig_stderr sys.argv = orig_argv
cheers,
Chris
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 8/4/10 11:33 , Chris Withers wrote:
Jens Vagelpohl wrote:
I've just been doing some build and release tools for a customer that involve this sort of stuff, I resorted to:
<snip lots of code>
Yikes. Maybe the subshell isn't so bad after all :-P
Yeah ;-) Although getting the right python path set up can be a pain...
In my particular case that's not a problem. I simply re-use sys.path from the running interpreter, which is good enough. Something like this:
cmd = 'PYTHONPATH="%s" %s setup.py --long-description' % ( sys.executable, ':'.join(sys.path)) status, output = commands.getstatusoutput(cmd)
jens
On Tuesday, August 03, 2010, Jens Vagelpohl wrote:
This is now implemented as fallback, take another look at the package listing, like the ZTK list[1]. I had to resort to executing "setup.py --long-description" in a subprocess, unfortunately. Never found out how to do that programmatically.
Yeah, this looks good.
Regards, Stephan