From: "random832@fastmail.us" <random832@fastmail.us>

On Mon, Nov 18, 2013, at 7:33, Robin Becker wrote:

...

UTF-8 stuff

This doesn't really solve the issue I was referring to, which is that windows _console_ (i.e. not redirected file or pipe) I/O can only support unicode via wide character (UTF-16) I/O with a special function, not via using byte-based I/O with the normal write function.

The problem is that Windows 16-bit I/O doesn't fit into the usual io module hierarchy. Not because it uses an encoding of UTF-16 (although anyone familiar with ReadConsoleW/WriteConsoleW from other languages may be a bit confused that Python's lowest-level wrappers around them deal in byte counts instead of WCHAR counts), but because you have to use HANDLEs instead of fds. So, there are going to be some compromises and some complexity. One possibility is to use as much of the io hierarchy as possible, but not try to make it flexible enough to be reusable for arbitrary HANDLEs: Add WindowsFileIO and WindowsConsoleIO classes that implement RawIOBase with a native HANDLE and ReadFile/WriteFile and ReadConsoleW/WriteConsoleW respectively. Both work in terms of bytes (which means WindowsConsoleIO.read has to //2 its argument, and write has to *2 the result). You also need a create_windows_io function that wraps a HANDLE by calling GetConsoleMode and constructing a WindowsConsoleIO or WindowsFileIO as appropriate, then creates a BufferedReader/Writer around that, then constructs a TextIOWrapper with UTF-16 or the default encoding around that. At startup, you just do that for the three GetStdHandle handles, and that's your stdin, stdout, and stderr. Besides not being reusable enough for people who want to wrap HANDLEs from other libraries or attach to new consoles from Python, it's not clear what fileno() should return. You could fake it and return the MSVCRT fds that correspond to the same files as the HANDLEs, but it's possible to end up with one redirected and not the other (e.g., if you detach the console), and I'm not sure what happens if you mix and match the two. A more "correct" solution would be to call _open_osfhandle on the HANDLE (and then keep track of the fact that os.close closes the HANDLE, or leave it up to the user to deal with bad handle errors?), but I'm not sure that's any better in practice. Also, should a console HANDLE use _O_WTEXT for its fd (in which case the user has to know that he has a _O_WTEXT handle even though there's no way to see that from Python), or not (in which case he's mixing 8-bit and 16-bit I/O on the same file)? It might be reasonable to just not expose fileno(); most code that wants the fileno() for stdin is just going to do something Unix-y that's not going to work anyway (select it, tcsetattr it, pass it over a socket to another file, …). A different approach would be to reuse as _little_ of io as possible, instead of as much: Windows stdin/stdout/stderr could each be custom TextIOBase implementations that work straight on HANDLEs and don't even support buffer (or detach), much less fileno. That exposes even less functionality to users, of course. It also means we need a parallel implementation of all the buffering logic. (On the other hand, it also leaves the door open to expose some Windows functionality, like async ReadFileEx/WriteFileEx, in a way that would be very hard through the normal layers…) It shouldn't be too hard to write most of these via an extension module or ctypes to experiment with it. As long as you're careful not to mix winsys.stdout and sys.stdout (the module could even set sys.stdin, sys.stdout, sys.stderr=stdin, stdout, stderr at import time, or just del them, for a bit of protection), it should work. It might be worth implementing a few different designs to play with, and putting them through their paces with some modules and scripts that do different things with stdio (including running the scripts with cmd.exe redirected I/O and with subprocess PIPEs) to see which ones have problems or limitations that are hard to foresee in advance. If you have a design that you think sounds good, and are willing to experiment the hell out of it, and don't know how to get started but would be willing to debug and finish a mostly-written/almost-working implementation, I could slap something together with ctypes to get you started.

On Tue, Nov 19, 2013, at 4:46, drekin@gmail.com wrote:

I really wanted to be able to write Unicode in Windows console, so I have written the following code: http://bugs.python.org/file31756/streams.py (based on other samples of code I found), in connection with http://bugs.python.org/issue1602 . It addresses the design of using as much as possible from io hierarchy. It definitely doesn't cover many details but I actually use it in my interactive console.

For this whole tactic of "using as much as possible from the io hierarchy" and acting like a raw stream that reads UTF-16 bytes, I'm worried that at some point it's going to run into something that tries to read a single byte - to which your code will return 0.

From: "random832@fastmail.us" <random832@fastmail.us>

On Mon, Nov 18, 2013, at 7:33, Robin Becker wrote:

...

UTF-8 stuff

This doesn't really solve the issue I was referring to, which is that windows _console_ (i.e. not redirected file or pipe) I/O can only support unicode via wide character (UTF-16) I/O with a special function, not via using byte-based I/O with the normal write function.

The problem is that Windows 16-bit I/O doesn't fit into the usual io module hierarchy. Not because it uses an encoding of UTF-16 (although anyone familiar with ReadConsoleW/WriteConsoleW from other languages may be a bit confused that Python's lowest-level wrappers around them deal in byte counts instead of WCHAR counts), but because you have to use HANDLEs instead of fds. So, there are going to be some compromises and some complexity. One possibility is to use as much of the io hierarchy as possible, but not try to make it flexible enough to be reusable for arbitrary HANDLEs: Add WindowsFileIO and WindowsConsoleIO classes that implement RawIOBase with a native HANDLE and ReadFile/WriteFile and ReadConsoleW/WriteConsoleW respectively. Both work in terms of bytes (which means WindowsConsoleIO.read has to //2 its argument, and write has to *2 the result). You also need a create_windows_io function that wraps a HANDLE by calling GetConsoleMode and constructing a WindowsConsoleIO or WindowsFileIO as appropriate, then creates a BufferedReader/Writer around that, then constructs a TextIOWrapper with UTF-16 or the default encoding around that. At startup, you just do that for the three GetStdHandle handles, and that's your stdin, stdout, and stderr. Besides not being reusable enough for people who want to wrap HANDLEs from other libraries or attach to new consoles from Python, it's not clear what fileno() should return. You could fake it and return the MSVCRT fds that correspond to the same files as the HANDLEs, but it's possible to end up with one redirected and not the other (e.g., if you detach the console), and I'm not sure what happens if you mix and match the two. A more "correct" solution would be to call _open_osfhandle on the HANDLE (and then keep track of the fact that os.close closes the HANDLE, or leave it up to the user to deal with bad handle errors?), but I'm not sure that's any better in practice. Also, should a console HANDLE use _O_WTEXT for its fd (in which case the user has to know that he has a _O_WTEXT handle even though there's no way to see that from Python), or not (in which case he's mixing 8-bit and 16-bit I/O on the same file)? It might be reasonable to just not expose fileno(); most code that wants the fileno() for stdin is just going to do something Unix-y that's not going to work anyway (select it, tcsetattr it, pass it over a socket to another file, …). A different approach would be to reuse as _little_ of io as possible, instead of as much: Windows stdin/stdout/stderr could each be custom TextIOBase implementations that work straight on HANDLEs and don't even support buffer (or detach), much less fileno. That exposes even less functionality to users, of course. It also means we need a parallel implementation of all the buffering logic. (On the other hand, it also leaves the door open to expose some Windows functionality, like async ReadFileEx/WriteFileEx, in a way that would be very hard through the normal layers…) It shouldn't be too hard to write most of these via an extension module or ctypes to experiment with it. As long as you're careful not to mix winsys.stdout and sys.stdout (the module could even set sys.stdin, sys.stdout, sys.stderr=stdin, stdout, stderr at import time, or just del them, for a bit of protection), it should work. It might be worth implementing a few different designs to play with, and putting them through their paces with some modules and scripts that do different things with stdio (including running the scripts with cmd.exe redirected I/O and with subprocess PIPEs) to see which ones have problems or limitations that are hard to foresee in advance. If you have a design that you think sounds good, and are willing to experiment the hell out of it, and don't know how to get started but would be willing to debug and finish a mostly-written/almost-working implementation, I could slap something together with ctypes to get you started.

On Tue, Nov 19, 2013, at 4:46, drekin@gmail.com wrote:

I really wanted to be able to write Unicode in Windows console, so I have written the following code: http://bugs.python.org/file31756/streams.py (based on other samples of code I found), in connection with http://bugs.python.org/issue1602 . It addresses the design of using as much as possible from io hierarchy. It definitely doesn't cover many details but I actually use it in my interactive console.

For this whole tactic of "using as much as possible from the io hierarchy" and acting like a raw stream that reads UTF-16 bytes, I'm worried that at some point it's going to run into something that tries to read a single byte - to which your code will return 0.