Thu, Aug 11, 2016 at 6:41 PM, Adam Bartoš <drekin@gmail.com> wrote:

The transcoding wrappers with 'utf-8' encoding are used just as a work around the fact that Python tokenizer cannot use utf-16-le and that the readlinehook machinery is unfortunately bytes-based. The tanscoding wrapper just has encoding 'utf-8' and no buffer attribute, so there is no actual transcoding in sys.std* objects. It's just a signal for PyOS_Readline consumers, and the transcoding occurs in a custom readline hook. Nothing like this would be needed if PyOS_Readline was replaced by some Python API wrapper around sys.readlinehook that would be Unicode string based.

If win_unicode_console gets added to the standard library, I think it should provide at least a std*.buffer interface that transcodes between UTF-16 and UTF-8 (with errors='replace'), to make this as much of a drop-in replacement as possible. I know it's not required. For example, IDLE doesn't implement this. But I'm also sure there's code out there that uses stdout.buffer, including in the standard library. It's mostly test code (not including cases for piping output from a child process) and simple script interfaces, but if we don't have to break people's code, we really shouldn't.

Thu, Aug 11, 2016 at 6:41 PM, Adam Bartoš <drekin@gmail.com> wrote:

The transcoding wrappers with 'utf-8' encoding are used just as a work around the fact that Python tokenizer cannot use utf-16-le and that the readlinehook machinery is unfortunately bytes-based. The tanscoding wrapper just has encoding 'utf-8' and no buffer attribute, so there is no actual transcoding in sys.std* objects. It's just a signal for PyOS_Readline consumers, and the transcoding occurs in a custom readline hook. Nothing like this would be needed if PyOS_Readline was replaced by some Python API wrapper around sys.readlinehook that would be Unicode string based.

If win_unicode_console gets added to the standard library, I think it should provide at least a std*.buffer interface that transcodes between UTF-16 and UTF-8 (with errors='replace'), to make this as much of a drop-in replacement as possible. I know it's not required. For example, IDLE doesn't implement this. But I'm also sure there's code out there that uses stdout.buffer, including in the standard library. It's mostly test code (not including cases for piping output from a child process) and simple script interfaces, but if we don't have to break people's code, we really shouldn't.