Proposal: cleaning up the content-type story
Following up to Martijn's observations on the ZTK, I'd like to propose a clean-up of how we handle content types. There are several unrelated pieces of code concerned with content types, these include at least zope.contenttype, zope.mimetype and zope.publisher.contenttype. - zope.contenttype does some content-type guessing from file names. It's just a few lines of code and comes with a table of some 30 associations of file-name extensions with content types. - zope.mimetype is a more substantial package that deals with content-type specific marker interfaces. - zope.publisher.contenttype contains a parser for content-type specifications. I propose merging those two packages and one module into one package, zope.contenttype. At present, zope.contenttype doesn't have any dependencies within the ZTK, and zope.mimetype depends on zope.configuration, zope.component and zope.interface. zope.publisher.contenttype doesn't import any zope code. - Switching packages that depend on zope.mimetype would therefore add no additional dependencies to them. - There are three packages within the ZTK and the set of packages under review which depend on zope.contenttype: zope.browserresource, zope.app.file and zope.app.onlinehelp. All of them already depend on zope.mimetype's dependencies, so there's nothing to be lost for them either. - Finally, the same holds for zope.publisher which would only acquire a new dependency on zope.contenttype if zope.publisher.contenttype were to be merged with the latter. Further, the parser in zope.publisher.contenttype needs to be made more conformant with RfC 2045 and be used by other code that does some content-type matching of its own right now, such as zope.publisher.http. -- Thomas
Thomas Lotze wrote:
Following up to Martijn's observations on the ZTK, I'd like to propose a clean-up of how we handle content types. There are several unrelated pieces of code concerned with content types, these include at least zope.contenttype, zope.mimetype and zope.publisher.contenttype.
- zope.contenttype does some content-type guessing from file names. It's just a few lines of code and comes with a table of some 30 associations of file-name extensions with content types.
- zope.mimetype is a more substantial package that deals with content-type specific marker interfaces.
- zope.publisher.contenttype contains a parser for content-type specifications.
I propose merging those two packages and one module into one package, zope.contenttype.
At present, zope.contenttype doesn't have any dependencies within the ZTK, and zope.mimetype depends on zope.configuration, zope.component and zope.interface. zope.publisher.contenttype doesn't import any zope code.
- Switching packages that depend on zope.mimetype would therefore add no additional dependencies to them.
- There are three packages within the ZTK and the set of packages under review which depend on zope.contenttype: zope.browserresource, zope.app.file and zope.app.onlinehelp. All of them already depend on zope.mimetype's dependencies, so there's nothing to be lost for them either.
- Finally, the same holds for zope.publisher which would only acquire a new dependency on zope.contenttype if zope.publisher.contenttype were to be merged with the latter.
Further, the parser in zope.publisher.contenttype needs to be made more conformant with RfC 2045 and be used by other code that does some content-type matching of its own right now, such as zope.publisher.http.
+1 on consolidation. A couple of questions: - How do 'zope.file' and 'zope.filerepresentation' interact with these modules? - How do these modules interact with the standard library's 'mimetypes' module? Tres. -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
Tres Seaver wrote:
Thomas Lotze wrote:
Following up to Martijn's observations on the ZTK, I'd like to propose a clean-up of how we handle content types. There are several unrelated pieces of code concerned with content types, these include at least zope.contenttype, zope.mimetype and zope.publisher.contenttype.
- zope.contenttype does some content-type guessing from file names. It's just a few lines of code and comes with a table of some 30 associations of file-name extensions with content types.
- zope.mimetype is a more substantial package that deals with content-type specific marker interfaces.
I actually forgot to mention that zope.mimetype comes with another table associating content types with file name suffixes and other information. It would certainly be a good idea to have only one such data collection to maintain.
- zope.publisher.contenttype contains a parser for content-type specifications.
I propose merging those two packages and one module into one package, zope.contenttype. [...]
- How do 'zope.file' and 'zope.filerepresentation' interact with these modules?
zope.file uses zope.mimetype to have a structured way of talking about content-type-related properties of file objects, and zope.publisher.contenttype for parsing content-type strings. zope.filerepresentation only defines some interfaces and uses none of the modules in question.
- How do these modules interact with the standard library's 'mimetypes' module?
Both zope.contenttype and zope.mimetype use the mimetypes module to guess content-types from file names. In fact, both even do some guesswork based on the first bytes of file content. Probably merging the two involves essentially dropping one of the implementations. I also wonder whether it might make sense to utilise the /etc/mime.types database (or the system's equivalent) for guessing based on the file-name extension, and libmagic (if available) for magic-number tests based on file contents. But of course, these ideas don't have much to do with the restructuring discussed in this thread. -- Thomas
On Thu, Oct 1, 2009 at 4:44 PM, Thomas Lotze <tl@gocept.com> wrote:
I also wonder whether it might make sense to utilise the /etc/mime.types database (or the system's equivalent) for guessing based on the file-name extension, and libmagic (if available) for magic-number tests based on file contents. But of course, these ideas don't have much to do with the restructuring discussed in this thread.
Not related to the restructuring: Products.MimeTypesRegistry is currently tightly bound to Zope2. But it does have a far more advanced implementation for mimetype related functionality, like type guessing and file extension handling. For mime type information it looks at: - all information found in the http://www.freedesktop.org/standards/shared-mime-info database - magic codes based on gnome-vfs-mime-magic - information form the Apache mime.types (local copy so it works on Windows) - respects zope.contenttype and Python's mimetypes module - on Windows respects the information found in the registry at "HKEY_CLASSES_ROOT\MIME\Database\Content Type" - a couple of its own definitions (for example Markdown, Textile, SVG, new MS Office formats not yet present in the other sources) If someone really wants to extend the functionality, this package might be worth a look. The whole persistent registry idea of MimeTypesRegistry is overkill, but most of the other code should have little or no external dependencies and should be good after a bit of refactoring. Hanno
Hello, That sounds impressing. Definitely something to watch in the refactoring process is having all this working on win32 ;-) Thursday, October 1, 2009, 5:11:10 PM, you wrote: HS> For mime type information it looks at: HS> - all information found in the HS> http://www.freedesktop.org/standards/shared-mime-info database HS> - magic codes based on gnome-vfs-mime-magic HS> - information form the Apache mime.types (local copy so it works on Windows) HS> - respects zope.contenttype and Python's mimetypes module HS> - on Windows respects the information found in the registry at HS> "HKEY_CLASSES_ROOT\MIME\Database\Content Type" HS> - a couple of its own definitions (for example Markdown, Textile, SVG, HS> new MS Office formats not yet present in the other sources) HS> If someone really wants to extend the functionality, this package HS> might be worth a look. The whole persistent registry idea of HS> MimeTypesRegistry is overkill, but most of the other code should have HS> little or no external dependencies and should be good after a bit of HS> refactoring. HS> Hanno HS> _______________________________________________ HS> Zope-Dev maillist - Zope-Dev@zope.org HS> https://mail.zope.org/mailman/listinfo/zope-dev HS> ** No cross posts or HTML encoding! ** HS> (Related lists - HS> https://mail.zope.org/mailman/listinfo/zope-announce HS> https://mail.zope.org/mailman/listinfo/zope ) -- Best regards, Adam GROSZER mailto:agroszer@gmail.com -- Quote of the day: Save yourself! Reboot in 5 seconds!
Thomas Lotze wrote:
I also wonder whether it might make sense to utilise the /etc/mime.types database (or the system's equivalent) for guessing based on the file-name extension,
The Python mimetypes module already uses that file in any of the "standard" Unix locations.
and libmagic (if available) for magic-number tests based on file contents. But of course, these ideas don't have much to do with the restructuring discussed in this thread.
Two PyPI modules claim to do the magic number guessing thing: - http://pypi.python.org/pypi/python-magic/0.1 is a ctypes wrapper around libmagic -- no sdist ;( - http://pypi.python.org/pypi/magic/0.1 has its own database inlined in the module. It has no distributions at all, just a link to a bare magic.py file. Tres. -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
Thomas Lotze wrote:
At present, zope.contenttype doesn't have any dependencies within the ZTK, and zope.mimetype depends on zope.configuration, zope.component and zope.interface. zope.publisher.contenttype doesn't import any zope code.
- Switching packages that depend on zope.mimetype would therefore add no additional dependencies to them.
- There are three packages within the ZTK and the set of packages under review which depend on zope.contenttype: zope.browserresource, zope.app.file and zope.app.onlinehelp. All of them already depend on zope.mimetype's dependencies, so there's nothing to be lost for them either.
A closer look at the actual zope.mimetypes code reveals that it really depends on a great lot of packages which just hadn't been listed among the dependencies. These include zope.app.form so that zope.mimetype ends up with 12 direct dependencies (excluding testing dependencies) and 48 dependencies counting transitive ones, including the ZODB. We should try to resolve some of them, in particular the one on zope.app.form, but we'll likely not get significantly smaller numbers. Therefore I change my original proposal: Let's no longer try to merge all content-type-related functionality into one package but let's rather stick with two packages: - zope.contenttype: parsing of MIME-type identifiers, guessing the MIME type of file contents, preferrably without dependencies within the ZTK - zope.mimetype: Zope interfaces for content types plus all the fancy form and icon stuff, preferrably sane dependencies I'm still going to move the zope.publisher.contenttype functionality to zope.contenttype which will ease some packages' dependencies, and I'll try to move some appropriate bits of code from zope.mimetype to zope.contenttype. -- Thomas
Thomas Lotze wrote:
- zope.contenttype: parsing of MIME-type identifiers, guessing the MIME type of file contents, preferrably without dependencies within the ZTK
Can I suggest that we use a different name? 'content type' to me sounds like CMS-y functionality. We have interfaces like IContentType and methods like queryContentType, neither of which would be relevant to zope.contenttype. :) Maybe zope.mimeparsing or something like that? Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
Martin Aspeli wrote:
Thomas Lotze wrote:
- zope.contenttype: parsing of MIME-type identifiers, guessing the MIME type of file contents, preferrably without dependencies within the ZTK
Can I suggest that we use a different name? 'content type' to me sounds like CMS-y functionality. We have interfaces like IContentType and methods like queryContentType, neither of which would be relevant to zope.contenttype. :)
Maybe zope.mimeparsing or something like that?
Fine with me. OTOH, there's also good reason to talk about content types since at least the relevant RfCs talk about the kind of strings being parsed mostly (only?) in the context of the Content-Type header field. Personally, I don't care enough to risk bike-shedding about this, though. -- Thomas
On Tue, Oct 6, 2009 at 11:56 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Thomas Lotze wrote:
- zope.contenttype: parsing of MIME-type identifiers, guessing the MIME type of file contents, preferrably without dependencies within the ZTK
Can I suggest that we use a different name?
Please don't! We have renamed this package already way too often. This ain't funny anymore: try: from zope.contenttype import guess_content_type except ImportError: # BBB: Zope < 2.10 try: from zope.app.content_types import guess_content_type except ImportError: # BBB: Zope < 2.9 from OFS.content_types import guess_content_type Hanno
Hanno Schlichting wrote:
On Tue, Oct 6, 2009 at 11:56 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Thomas Lotze wrote:
- zope.contenttype: parsing of MIME-type identifiers, guessing the MIME type of file contents, preferrably without dependencies within the ZTK Can I suggest that we use a different name?
Please don't!
We have renamed this package already way too often. This ain't funny anymore:
try: from zope.contenttype import guess_content_type except ImportError: # BBB: Zope < 2.10 try: from zope.app.content_types import guess_content_type except ImportError: # BBB: Zope < 2.9 from OFS.content_types import guess_content_type
Actually, that is kind of funny. :) But yeah, renaming is bad. I thought we were making a new package. Ignore my suggestion. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
Thomas Lotze wrote:
I'm still going to move the zope.publisher.contenttype functionality to zope.contenttype which will ease some packages' dependencies, and I'll try to move some appropriate bits of code from zope.mimetype to zope.contenttype.
Before doing so, however, I'd like to release the current state of some packages and update their versions in ztk.cfg in order to have a clean state to proceed from. The zope.publisher trunk doesn't work with the zope.app.publisher version from the current ZTK. Also, zope.server, zope.app.authentication and zope.app.i18n need some modifications not yet released in order for their tests to pass with the current zope.publisher. So as a first step, I'd like to release the current code of zope.publisher, zope.server, zope.app.authentication and zope.app.i18n. Can somebody please grant me PyPI rights for these packages? Thank you very much. -- Thomas
On Wed, Oct 7, 2009 at 5:03 PM, Thomas Lotze <tl@gocept.com> wrote:
So as a first step, I'd like to release the current code of zope.publisher, zope.server, zope.app.authentication and zope.app.i18n. Can somebody please grant me PyPI rights for these packages? Thank you very much.
Granted :) If someone would document srichter's magic grant-all-powerful PyPi script, I'd run it :) Hanno
On Wed, Oct 7, 2009 at 11:10 AM, Hanno Schlichting <hanno@hannosch.eu> wrote:
If someone would document srichter's magic grant-all-powerful PyPi script, I'd run it :)
That's a horrible thing to do to somebody! Note that I'm not smiling, either. It's too easy to grant people access to way too many packages that way. Somebody ran it for me, without my knowledge, and I learned about a lot of packages I'd never heard of before. That just makes PyPI harder to use when updating something I *am* actually a maintainer for. On the other hand, it got me to prod for a fix to the PyPI bug that didn't let me remove myself from those projects. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On Wed, Oct 7, 2009 at 5:17 PM, Fred Drake <fdrake@gmail.com> wrote:
On Wed, Oct 7, 2009 at 11:10 AM, Hanno Schlichting <hanno@hannosch.eu> wrote:
If someone would document srichter's magic grant-all-powerful PyPi script, I'd run it :)
That's a horrible thing to do to somebody!
Note that I'm not smiling, either. It's too easy to grant people access to way too many packages that way. Somebody ran it for me, without my knowledge, and I learned about a lot of packages I'd never heard of before. That just makes PyPI harder to use when updating something I *am* actually a maintainer for.
On the other hand, it got me to prod for a fix to the PyPI bug that didn't let me remove myself from those projects.
There's both sides: - Once you have access and can release a new version, your mental excuse / barrier for not doing so gets lower. - By having too many, you don't care about any of them anymore. But I think we are still sufficiently few people working on the ZTK, so we can grant all of them access to all ZTK packages. I don't want to give anyone access to all the packages I have. That's Zope + Plone + personal combined and by now about 350 packages. Makes a total mess out of the PyPi UI. But giving people who work on refactoring the ZTK access to the ZTK packages should be fine. Hanno
Fred Drake wrote:
On Wed, Oct 7, 2009 at 11:10 AM, Hanno Schlichting <hanno@hannosch.eu> wrote:
If someone would document srichter's magic grant-all-powerful PyPi script, I'd run it :)
That's a horrible thing to do to somebody!
Note that I'm not smiling, either. It's too easy to grant people access to way too many packages that way. Somebody ran it for me, without my knowledge, and I learned about a lot of packages I'd never heard of before. That just makes PyPI harder to use when updating something I *am* actually a maintainer for.
I think the issue this points at is actually with usability of PyPI. There shouldn't be this miles-long list of packages on the right when logged in in the first place. OTOH, even with good usability I'd rather not have rights for packages I'm not interested in, just to be able to deny responsibility if anything goes wrong with one of them. Having rights for all packages involved with ZTK refactorings would be helpful, though. -- Thomas
On Wed, Oct 7, 2009 at 11:27 AM, Thomas Lotze <tl@gocept.com> wrote:
OTOH, even with good usability I'd rather not have rights for packages I'm not interested in, just to be able to deny responsibility if anything goes wrong with one of them.
It's entirely reasonable for maintainers to have rights for the packages they work on. My objection was to learning about large numbers of packages when someone handed me rights to them on PyPI. Not *exactly* the case here, but I firmly stand against wildly spraying permissions where they're not wanted. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
Thomas Lotze wrote:
I'm still going to move the zope.publisher.contenttype functionality to zope.contenttype which will ease some packages' dependencies,
JFTR: I've done that now, including updates to packages which used to import from zope.publisher.contenttype. -- Thomas
participants (6)
-
Adam GROSZER -
Fred Drake -
Hanno Schlichting -
Martin Aspeli -
Thomas Lotze -
Tres Seaver