[Python-Dev] PEP 528: Change Windows console encoding to UTF-8

[Steve Dower]

My original plan was to bypass the utf8 encoding step, but that was going to cause major issues with code that blindly assumes it can do things like sys.stdout.buffer.write(b"\n") (rather than b"\n\0" - and who'd imagine you needed to do that). I didn't want to set up secret handshakes either, at least until there's a proven performance issue.

I'd need to test to be sure, but writing an incomplete code point should just truncate to before that point. It may currently raise OSError if that truncated to zero length, as I believe that's not currently distinguished from an error. What behavior would you propose?

Reads of less than four bytes fail instantly, as in the worst case we need four bytes to represent one Unicode character. This is an unfortunate reality of trying to limit it to one system call - you'll never get a full buffer from a single read, as there is no simple mapping between length-as-utf8 and length-as-utf16 for an arbitrary string.

Top-posted from my Windows Phone

-----Original Message-----
From: "Random832" <random832 at fastmail.com>
Sent: ‎9/‎1/‎2016 16:31
To: "python-dev at python.org" <python-dev at python.org>
Subject: Re: [Python-Dev] PEP 528: Change Windows console encoding to UTF-8

On Thu, Sep 1, 2016, at 18:28, Steve Dower wrote:
> This is a raw (bytes) IO class that requires text to be passed encoded
> with utf-8, which will be decoded to utf-16-le and passed to the Windows APIs.
> Similarly, bytes read from the class will be provided by the operating 
> system as utf-16-le and converted into utf-8 when returned to Python.

What happens if a character is broken across a buffer boundary? e.g. if
someone tries to read or write one byte at a time (you can't do a
partial read of zero bytes, there's no way to distinguish that from an
EOF.)

Is there going to be a higher-level text I/O class that bypasses the
UTF-8 encoding step when the underlying bytes stream is a console? What
if we did that but left the encoding as mbcs? I.e. the console is text
stream that can magically handle characters that aren't representable in
its encoding. Note that if anything does os.read/write to the console's
file descriptors, they're gonna get MBCS and there's nothing we can do
about it.
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160901/0dcdee6a/attachment.html>