Hi, I'm doing some work with WebDAV representations in Zope 2, among other things, and I'm trying to see if we should use zope.filerepresentation as the central abstraction for file read/write operations. However, I find myself lacking a couple of things: 1) A way for an IReadFile to return the file's MIME (content) type (for the Content-Type, for example) and an optional filename (for the Conent-Disposition header, for example) 2) A way for an IWriteFile to be told about the MIME type and encoding of the data it is consuming. I know that IFileFactory() contains this, but data is not always going to be written to a new file, so setting this from the factory won't work. 3) I see this comment: # TODO: We will add ILargeReadFile and ILargeWriteFile to efficiently # handle large data. But it seems this was never implemented. I need a way for the IWriteFile.write() to be passed a file-like object (in fact, there's not really any clear notion of what the 'data' parameter is, but I assume it's meant to be a str object), and for IReadFile.read() to return a file stream iterator of some kind instead of the full raw data. I see two options here: - add a new plone.filerepresentation that extends these interfaces - commit some changes to zope.filerepresentation and make a new release of this Since zope.filerepresentation is really *just* interfaces, #2 seems best. Any objections? Here's an initial take on the interfaces: class IReadFile(Interface): """Provide read access to file data """ def read(): """Return the file data as a str """ def size(): """Return the data length """ class ILargeReadFile(IReadFile): """Provide efficient read access to file data """ def getContentType(): """Get the content/MIME type of the file as a string in the form 'major/minor'. Return None if this is not known. """ def getFilename(): """Get the intended file name for this file. Return None if this is not known. """ def getIterator(): """Return an iterable of the file data """ class IWriteFile(Interface): def write(data): """Update the file data """ class ILargeWriteFile(IWriteFile): def writeFile(data, contentType=None, encoding=None): """Update the file data. data is a file-like object containing input data contentType is the content/MIME type of the data as a string in the form 'major/minor'. It may be None if unknown. encoding is the encoding of the string. It may be None if unknown. """ This should be 100% backwards compatible. The downside is that people will need to implement both write() and writeFile() / read() and getIterator() to support both str based and stream based reading and writing, but that's easily addressed with a StringIO. And of course, nothing in the ZTK will actually use these new interfaces, but that could come later. ;) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
On Wed, Sep 30, 2009 at 5:00 PM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Here's an initial take on the interfaces:
class IReadFile(Interface): """Provide read access to file data """
def read(): """Return the file data as a str """
def size(): """Return the data length """
class ILargeReadFile(IReadFile): """Provide efficient read access to file data """
def getContentType(): """Get the content/MIME type of the file as a string in the form 'major/minor'. Return None if this is not known. """
def getFilename(): """Get the intended file name for this file. Return None if this is not known. """
def getIterator(): """Return an iterable of the file data """
class IWriteFile(Interface):
def write(data): """Update the file data """
class ILargeWriteFile(IWriteFile):
def writeFile(data, contentType=None, encoding=None): """Update the file data.
data is a file-like object containing input data
contentType is the content/MIME type of the data as a string in the form 'major/minor'. It may be None if unknown.
encoding is the encoding of the string. It may be None if unknown. """
Is there any reason to invent a new API and not just use Python's file API? So .name instead of getFilename, iterator per __iter__ and next? I'm not sure about the different read/write methods on file-objects, but I think the standard ones do cover all the same use-cases. The standard file implementation has no knowledge of its size, as this is sometimes impossible to get, when dealing with stream based file-like objects. Do we really need to have files to know their size? Hanno
Hanno Schlichting wrote:
Is there any reason to invent a new API and not just use Python's file API?
I don't know. IReadFile and IWriteFile have been around for ever and are used by a number of things in Zope. They have read(), write() and size(). The first two are on file, the last one isn't. I'd like to be able to use this API as a base, since it's used for things like z3c.blobfile already, and is documented as the way to do this kind of thing in Philipp's book.
So .name instead of getFilename, iterator per __iter__ and next?
Yeah, that's a better idea.
I'm not sure about the different read/write methods on file-objects, but I think the standard ones do cover all the same use-cases.
The standard file implementation has no knowledge of its size, as this is sometimes impossible to get, when dealing with stream based file-like objects. Do we really need to have files to know their size?
Well, for the writeFile() stuff maybe we don't. Thinking through my use cases again, I can't see a need for passing the content type in, and encoding can be set if we support setting the '.encoding' property. It's kind of important to be able to indicate the size and MIME type for a read operation, though. In this case, I want to be able to put that information into the Content-Type and Content-Length headers. The IStreamData stuff in Zope 2 also wants to know the length before it starts streaming. In some cases, it may not be possible to know, in which case we can fall back on len(data), but in other cases, the length is known (plone.namedfile and z3c.blobfile keep track of it, for example), in which case having a way to ask the adapter means it can be done much more efficiently. So, how about this: class IReadFile(Interface): """Provide read access to file data """ def read(): """Return the file data as a str """ def size(): """Return the data length """ class ILargeReadFile(IReadFile): """Provide efficient read access to file data """ def getContentType(): """Get the content/MIME type of the file as a string in the form 'major/minor'. Return None if this is not known or undefined. """ name = schema.TextLine(title=u"Filename", readonly=True) encoding = schema.ASCIILine(title=u"Encoding", readonly=True) def __iter__(): """Get an iterator""" def next(): """See file""" def seek(): """See file""" def tell(): """See file""" class IWriteFile(Interface): def write(data): """Update the file data """ class ILargeWriteFile(IWriteFile): def write(data): """Write a chunk of data """ name = schema.TextLine(title=u"Filename", readonly=False) encoding = schema.ASCIILine(title=u"Encoding", readonly=False) def tell(): """See file""" def close(): """See file"" It may be worth supporting the rest of the functions (readlines, writelines, truncate, readline, isatty, fileno, flush) so that ILargeReadFile + ILargeWriteFile are equivalent to file. I've also allowed here for the name and encoding to be set on an ILargeWriteFile, with the caveat that this has to be done before the writing. In this approach, the caller is responsible for writing large streams in chunks and then calling close, i.e. the iterator isn't passed to the ILargeWriteFile. I think that's OK, though. I've still got the content type as a method (maybe make it a read-only property?) that may return None. That could be in a separate adapter, but that feels like overkill to me. :) Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
On Thu, Oct 1, 2009 at 2:13 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Hanno Schlichting wrote:
Is there any reason to invent a new API and not just use Python's file API?
I don't know. IReadFile and IWriteFile have been around for ever and are used by a number of things in Zope. They have read(), write() and size(). The first two are on file, the last one isn't. I'd like to be able to use this API as a base, since it's used for things like z3c.blobfile already, and is documented as the way to do this kind of thing in Philipp's book.
Ok. I have a feeling that Zope3 at various times invented a new Java-ish API instead of using standard Python protocols. I'd just like to avoid going down that way even more.
class IReadFile(Interface): """Provide read access to file data """
def read(): """Return the file data as a str """
def size(): """Return the data length """
This needs a clarification on what kind of length this is. Is it the length of the binary data? Since these interfaces also know about encoding, it's otherwise not clear if the size for textual data might be its Unicode length.
class ILargeReadFile(IReadFile): """Provide efficient read access to file data """
def getContentType(): """Get the content/MIME type of the file as a string in the form 'major/minor'. Return None if this is not known or undefined. """
name = schema.TextLine(title=u"Filename", readonly=True) encoding = schema.ASCIILine(title=u"Encoding", readonly=True)
encoding only makes sense for text/*, so maybe some small hint at that?
def __iter__(): """Get an iterator"""
def next(): """See file"""
def seek(): """See file"""
def tell(): """See file"""
class IWriteFile(Interface):
def write(data): """Update the file data """
class ILargeWriteFile(IWriteFile):
def write(data): """Write a chunk of data """
name = schema.TextLine(title=u"Filename", readonly=False) encoding = schema.ASCIILine(title=u"Encoding", readonly=False)
def tell(): """See file"""
def close(): """See file""
I've still got the content type as a method (maybe make it a read-only property?) that may return None. That could be in a separate adapter, but that feels like overkill to me. :)
There's no standard way to spell "content type", so I don't really care. On the pure file level it's even called mime type and only internet data handling uses "content type". So you have to look things up anyways. Compared to the other attributes/methods "getContentType" looks like a Java-intruder in otherwise Python naming conventions, though. Hanno
Hanno Schlichting wrote:
On Thu, Oct 1, 2009 at 2:13 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Hanno Schlichting wrote:
Is there any reason to invent a new API and not just use Python's file API? I don't know. IReadFile and IWriteFile have been around for ever and are used by a number of things in Zope. They have read(), write() and size(). The first two are on file, the last one isn't. I'd like to be able to use this API as a base, since it's used for things like z3c.blobfile already, and is documented as the way to do this kind of thing in Philipp's book.
Ok. I have a feeling that Zope3 at various times invented a new Java-ish API instead of using standard Python protocols. I'd just like to avoid going down that way even more.
class IReadFile(Interface): """Provide read access to file data """
def read(): """Return the file data as a str """
def size(): """Return the data length """
This needs a clarification on what kind of length this is. Is it the length of the binary data? Since these interfaces also know about encoding, it's otherwise not clear if the size for textual data might be its Unicode length.
I assume so. Note that this interface already exists, so I'm not proposing to change it. ;-)
class ILargeReadFile(IReadFile): """Provide efficient read access to file data """
def getContentType(): """Get the content/MIME type of the file as a string in the form 'major/minor'. Return None if this is not known or undefined. """
name = schema.TextLine(title=u"Filename", readonly=True) encoding = schema.ASCIILine(title=u"Encoding", readonly=True)
encoding only makes sense for text/*, so maybe some small hint at that?
Sure, yeah.
def __iter__(): """Get an iterator"""
def next(): """See file"""
def seek(): """See file"""
def tell(): """See file"""
class IWriteFile(Interface):
def write(data): """Update the file data """
class ILargeWriteFile(IWriteFile):
def write(data): """Write a chunk of data """
name = schema.TextLine(title=u"Filename", readonly=False) encoding = schema.ASCIILine(title=u"Encoding", readonly=False)
def tell(): """See file"""
def close(): """See file""
I've still got the content type as a method (maybe make it a read-only property?) that may return None. That could be in a separate adapter, but that feels like overkill to me. :)
There's no standard way to spell "content type", so I don't really care. On the pure file level it's even called mime type and only internet data handling uses "content type". So you have to look things up anyways. Compared to the other attributes/methods "getContentType" looks like a Java-intruder in otherwise Python naming conventions, though.
Haha. Maybe I'll have a property mimeType instead? That looks better, I guess. But we *are* in a Zope package, so Zope naming conventions probably apply. Ok, I'm going to do this shortly, unless anyone objects. Martin -- Author of `Professional Plone Development`, a book for developers who want to work with Plone. See http://martinaspeli.net/plone-book
Am 01.10.2009, 15:51 Uhr, schrieb Martin Aspeli <optilude+lists@gmail.com>:
Haha. Maybe I'll have a property mimeType instead? That looks better, I guess.
+1 to use a property rather than a getter. Not sure about content-type vs. mime-type. If this is used only in an HTTP environment then I'd prefer to stick with content-type otherwise I'd suggest the email module's content-type handling.
But we *are* in a Zope package, so Zope naming conventions probably apply.
I go with Pythonic conventions wherever possible. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtztstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
2009/10/1 Martin Aspeli <optilude+lists@gmail.com>:
Hanno Schlichting wrote:
The standard file implementation has no knowledge of its size, as this is sometimes impossible to get, when dealing with stream based file-like objects. Do we really need to have files to know their size?
Well, for the writeFile() stuff maybe we don't. Thinking through my use cases again, I can't see a need for passing the content type in, and encoding can be set if we support setting the '.encoding' property.
It's kind of important to be able to indicate the size and MIME type for a read operation, though. In this case, I want to be able to put that information into the Content-Type and Content-Length headers. The IStreamData stuff in Zope 2 also wants to know the length before it starts streaming.
In some cases, it may not be possible to know, in which case we can fall back on len(data), but in other cases, the length is known (plone.namedfile and z3c.blobfile keep track of it, for example), in which case having a way to ask the adapter means it can be done much more efficiently.
To find the length of a file: f.seek(0,2) # 0 bytes from the end of the file size = f.tell() # position in file. Laurence
participants (4)
-
Charlie Clark -
Hanno Schlichting -
Laurence Rowe -
Martin Aspeli