On 06/12/2012 03:48 PM, Victor Stinner wrote:

...

...

...

1. The most straight-forward way to handle this is to open the file twice, first in binary mode or with latin1 encoding and again in text mode after the encoding has been determined This of course has a performance cost since the data is read twice. Further, it can't be used if the data source is a from a pipe, socket or other non-rewindable source. This includes sys.stdin when it comes from a pipe.

Some months ago, I proposed to automatically detect if a file contains a BOM and uses it to set the encoding. Various methods were proposed but there was no real consensus. One proposition was to use a codec (e.g. "bom") which uses the BOM if it is present, and so don't need to reread the file twice.

For the pipe issue: it depends where the encoding specification is. If the encoding is written at the end of your "file" (stream), you have to store the whole stream content (few MB or maybe much more?) into memory. If it is in the first lines, you have to store these lines in a buffer. It's not easy to decide for the threshold.

That's always a problem. When trying to determine a character encoding one may have to read the entire file because it could consist of all ascii characters except the very last one. (And of course there is no guarantee one can determine *the* encoding at all). Nevertheless, I think thee is a very large class of problems that can be usefully handled by looking at a limited amount of data at the start of a file (or stream). The Python coding declaration in one example (obviously picked hoping it would have some resonance here.) The buffer object used by TextIOWrapper already reads the start of the stream and buffers the first few lines, so why not take advantage of that rather than repeating the work? One of the things I am not sure about is if there are cases when the buffered read returns, say, only one line, as might happen with tty input.

...

I don't like the codec approach because the codec is disconnected from the stream. For example, the codec doesn't know the current position in stream nor can read a few more bytes forward or backward. If you open the file in "append" mode, you are not writing at the beginning but at the end of the file. You may also seek at an arbitrary position before the first read...

There are also some special cases. For example, when a text file is opened in write mode, the file is seekable and the file position is not zero, TextIOWrapper calls encoder.setstate(0) to not write the BOM in the middle of the file. (See also Lib/test/test_io.py for related tests.)

A callable encoding parameter would not be terribly useful with a file opened in write or append mode, but it's behavior would be predictable: a write would result in an error because the encoding hadn't been set. A read in the middle' of the file would work the same way as at the beginning. This is probably not very useful, but is consistent. Of course one could choose to implement a callable encoding parameter such that some or all of these paths are detected at open and declared illegal then. One could prohibit the encoding call after a seek though I'm not sure there is any point to that.

...

...

...

2. Alternatively, with a little more expertise, one can rewrap the open binary stream in a TextIOWrapper to avoid a second OS file open.

That's my favorite method because you have the full control on the stream. (I wrote tokenize.open). But yes, it does not work on non-seekable streams (e.g. pipes).

...

...

This too seems to read the data twice and of course the seek(0) prevents this method also from being usable with pipes, sockets and other non-seekable sources.

Does it really matter? You usually need to read few bytes to get the encoding.

It certainly matters if input is from a pipe. Quoting from my other message: $ cat test.utf8 | python3 stdin.py reopen1 got exception: [Errno 29] Illegal seek The whole point of my suggestion was that you've already read those few bytes -- but by the time you have access to them, you've already been forced to choose an encoding. My suggestion simply defers that encoding setting until after you've had a chance to look at the bytes.

...

...

...

9. In other non-read paths where encoding needs to be known, raise an error if it is still None.

Why not reading data until you the encoding is known instead?

That's how I do it now -- open file in binary mode and read it, buffer it, determine encoding, and henceforth decode the bytes data "by hand" to text. But that's an awful lot like what TextIOWrpper does, yes? Why can't I use TextIOWrapper instead of rewriting it myself? (Yes, I know I can reopen or rewrap the binary stream but as I said, that loses the one-pass processing which breaks pipes.)

...

...

...

I have modified a copy the _pyio module as described and the changes required seemed unsurprising and relatively few, though I am sure there are subtleties and other considerations I am missing. Hence this post seeking feedback...

Can you post the modified somewhere so I can play with it?

I put a diff against the Python-3.2.3 _pyio.py file at: http://pastebin.com/kZHmcBdm Much of the diff is just moving existing stuff around. The note at the bottom says: | It is in no way supposed to be a serious patch. | | It was the minimal changes I could make in order to | see if my suggestion to allow a callable encoding parameter | in TextIOWrapper was feasible, and allow some timing tests. | | I am quite sure it will not pass the Python's tests. | | It does I hope give some idea of the nature and scale of the | code changes needed to implement a callable encodign parameter.