[Python-ideas] Fwd: Remove tty module

Sat Aug 3 01:19:46 CEST 2013

Looks like I replied to Random's accidentally-offline reply instead of the right one. So, to amplify the confusion, I'll just forward it here. :)

Sent from a random iPhone

Begin forwarded message:

> From: Andrew Barnert <abarnert at yahoo.com>
> Date: August 2, 2013, 16:18:36 PDT
> To: "random832 at fastmail.us" <random832 at fastmail.us>
> Subject: Re: [Python-ideas] Remove tty module
> 
> On Aug 2, 2013, at 13:05, random832 at fastmail.us wrote:
> 
>> 
>> 
>> On Fri, Aug 2, 2013, at 12:57, Andrew Barnert wrote:
>>> Sure there is. It's harder to implement, and will be less portable. Even
>>> just getting the screen width is tricky without curses.
>> 
>> Why would it be less portable? Right now, we have curses, which only
>> works on unix.
> 
> Because curses isn't available on every platform termios is. And it isn't appropriate for every use case where termios is. (For the most dramatic case, getch makes sense on a serial line; gotoxy does not. But even on an actual terminal, there are many cases where you don't want to take over the terminal, break scrollback, etc.)
> 
>> Implementing a simple set of functions for both unix and
>> windows is more portable.
> 
> Sure, but implementing an even simpler set of functions that works on a broader range of unix plus windows and can be used in a broader range of cases is even _more_ portable.
> 
> And of course it's also simpler, meaning less coding, less debugging, less bikeshedding, etc. 
> 
>> I am thinking in terms of "implement core functionality for multiple
>> platforms, then implement anything more complex on top of that in pure
>> python" - this necessarily means duplicating some of the work that
>> curses does now for unix systems. What's wrong with this approach?
> 
> Well, duplicating the work of curses instead of just using curses may well be a mistake. 
> 
> But otherwise, there's nothing wrong with this; it's just a potentially (and, I think, desirably) separate idea from implementing basic raw terminal I/O.
> 
> As I said before, I think we ultimately may want three packages:
> 
> * a raw console I/O package
> * a simple formatting (color) package
> * a simple foundation for console GUIs
> 
> The first two are nearly orthogonal, and could easily be built to work together when appropriate but also work separately when desired. Also, the second one already has dozens of existing implementations to choose from, and the first took me a few minutes to hack up a prototype.
> 
> The third, meanwhile, is going to require a lot more work. It may make use of the first two always, only on Windows (using curses on Unix), or never, but an end user is rarely if ever going to use it in the same program as the others.
> 
>>> And more importantly, a gotoxy function can only work after you've taken
>>> over the whole terminal in the same way curses does.
>> 
>> Uh, all you have to do for that is clear the screen... and clrscr() was
>> going to be the next function I was going to propose
>> 
>> There are a lot of
>>> things you may want to do--from getwch to setting colors--that don't
>>> require that. So, a module that didn't let you getwch unless you enter a
>>> curses-like mode would be less useful.
>>> 
>>> I think a simple consoleio module that just does nonblocking I/O is a
>>> useful thing. A separate module that does non-full-screen formatting (and
>>> there are dozens of these on PyPI) makes a nice complement. (Especially
>>> since there are good use cases for using that _without_ a terminal--e.g.,
>>> creating ANSI art text files--but also of course many good use cases for
>>> doing both together.) A curses/conio wrapper for full-screen GUIs seems
>>> like an almost entirely separate thing, except for the fact that both
>>> would happen to use some of the same msvcrt calls on Windows.
>>>> As for kbhit, you could probably implement it on unix with a call to
>>>> select. If the tty file descriptor is ready for reading, then return
>>>> true. The one possible wrinkle is that getwch could block if an
>>>> incomplete multibyte character is read - something that cannot happen on
>>>> windows.
>>> 
>>> There are other wrinkles. For example, on some posix platforms you also
>>> need to fcntl the fd into nonblocking mode.
>> 
>> What platforms are those? I thought the whole POINT of select was that
>> the file descriptor doesn't have to be in nonblocking mode, since
>> otherwise you could just attempt to read and have it return
>> EWOULDBLOCK/EAGAIN.
>> 
>>> Meanwhile, multibyte characters are not actually a problem. At an actual
>>> console, if you type one, all of the bytes become ready at the same time,
>>> so you can getwch.
>> 
>> Yes, the problem is if you type a single character in a non-UTF8
>> character set that python _thinks_ is the first byte of a UTF8
>> character.
> 
> Only if it thinks your terminal is UTF-8. Which it shouldn't.
> 
>> This is really more of an issue for escape sequences than
>> multibyte characters, since in the multibyte case you could just say
>> it's a misconfiguration so it's only natural that it leads to bad
>> behavior.
> 
> Yes, escape sequences are a problem even with proper configuration.
> 
> But again, this is a problem that conio-style code had always had, from the DOS days up to the current msvcrt implementation.
> 
>>> On a serial line, that isn't true, but it isn't true
>>> on Windows either,
>> 
>> How is it not true on windows? The physical input on windows is unicode
>> characters; any translation to multibyte characters happens within
>> getch, getwch will never even _see_ anything that's not a whole unicode
>> codepoint. The only reason you get multiple values for arrow keys is
>> because getwch translates it _into_ multiple values from a lower-level
>> source that generates a single event (see my other post where I propose
>> bypassing this)
> 
> But people writing conio-style code today are using either msvcrt or libraries like python-conio, where it _is_ a problem.
> 
>>> There are also problems that are Windows specific that already affect
>>> msvcrt and fancier implementations that I haven't made any attempt to
>>> deal with, like the fact that getwch can return half a surrogate pair.
>> 
>> Surrogate pair support on the console is terrible in general. You might
>> get half a surrogate pair and never get the other half, because there is
>> no part of the data path that actually deals with whole code points [so
>> it's possible the pair never existed]
>> 
>>> See the caveats in my readme, and the todo file, for everything I've
>>> discovered so far. And please experiment with the code and find all the
>>> problems I haven't discovered, because I'm sure there are plenty.
>> 
>> The way I see it, there are five possible types of keyboard events:
>> 
>> A unicode character is typed (if multiple characters are typed, this can
>> be multiple events, even if it came from one key) - you still might want
>> additional info to differentiate ctrl-h from ctrl-shift-h or backspace.
>> Obviously, this is a normal event that you should be able to read and
>> kbhit should return true.
>> 
>> An "action" key is typed, e.g. arrows, home, end, etc. You almost always
>> want to be able to read this and it should trigger kbhit.
>> 
>> A modifier key or dead key is pressed. Generally, you don't want to read
>> this or trigger kbhit, and on unix systems it is impossible to do so.
>> Escape sequence on unix.
>> 
>> A key is released. Same as above, you don't want this event, and it's
>> not possible on unix, barring some seriously esoteric xterm feature I'm
>> not aware of.
>> 
>> Mouse events. On either unix or windows, this is part of the same
>> "stream" as keyboard events, and you only get it if you ask for it.
>> 
>>> What about cbreak mode? It's a useful thing, there's no obvious way to fit it into the conio-style paradigm, and tty wraps it up for you.
>> 
>> I'd think cbreak mode basically consists of calling getche() all the
>> time. What's the difference, other than that?
> 
> No, raw/cbreak/cooked is entirely orthogonal to echo/noecho. For example, in raw mode, ^C is just a character; in cbreak mode, as in cooked mode, it's a signal. Cbreak basically means to turn off line discipline, but leave on everything else.
> 
>> Windows acts weird if you mix line buffering and getch, by the way,
>> which we may have to simply tolerate: type a line that's 10 characters,
>> read 5, call getch, then read 5 more, and the second read will actually
>> get the next 5 characters from the first line, even with no userspace
>> buffering [directly calling os.read; haven't tried kernel32.ReadFile or
>> ReadConsole yet - it's _possible_ there's a buffer we can flush
>> somewhere].
> 
> This is a big part of the reason I decided to require enable/disable pairs and just say that normal stdin/out is illegal (but not necessarily checked) inside an enable and consoleio illegal outside of one.
> 
>>> As I said in my last email, that implies that we need either curses or a whole lot of code rather than just termios, and more importantly that nobody can use consoleio without going into a curses full-screen mode.
>>> 
>>> We could of course have two different modes that you can enable (just raw I/O vs. curses full screen), where the functionality that's common to both has the same names both ways, which you suggest later. But I'm wary about that, because getwch, kbhit, etc. based on curses will have many subtle differences from implementations based on select and raw mode.
>> 
>> I still don't understand your objection. People would be able to use the
>> rest of consoleio all the time, just without using those functions (one
>> of which, clrscr, basically _is_ "going into a curses full-screen mode"
>> - i'm not sure what else you think going into a full-screen mode
>> consists of)
> 
> Read the source to curses. It consists of doing a bunch of additional termios stuff you don't need for raw mode, doing various bizarre workarounds for different platforms, setting env variables to interact with a variety of different terminal emulators, sending a variety of escape sequences to control things like soft labels and take over scrolling and scrollback on terminals that support it, etc. There's a reason lib_newterm.c is 352 lines long.
> 
>> Input and output are basically completely independent
>> (other than echoing), anyway, why would they have to use an output
>> function to be able to use an input function? Why would being in full
>> screen output mode have any effect on getwch or kbhit? Unless you're
>> proposing using the _actual_ curses input functions, which I never so
>> much as breathed a word of.
> 
> But I've repeatedly said that if you want full-screen graphics, you _do_ want to use curses, rather than try to reproduce decades worth of development and debugging to get it to work on a wide variety of platforms, handle features you haven't thought of, etc.
> 
>>> For that matter, there's no reason you shouldn't be able to consoleio on a second tty or pty if you've got a handle to one.
>> 
>> I agree, but this is tricky to define in a cross-platform way,
>> particularly since on windows you A) absolutely cannot have a second
>> console without doing _seriously_ tricky things with multiple processes
>> [you can attach to any process's console, so we could spawn a subprocess
>> just to get its console] and B) even if you could, the console consists
>> of a pair of handles, not just one (but see A, what you really need is a
>> process ID...), C) using a serial port is very different from using the
>> console.
>> 
>> Also, /dev/tty has the advantage of being available to a program which
>> has its input and output redirected, and python already uses it for
>> getpass, so "you really should have to know a little bit about what
>> you're doing." is already a lost cause. And I say this despite
>> considering and rejecting calling AllocConsole if the windows version of
>> the API is enabled from a pythonw process - redirection is a more common
>> use case than wanting to do console I/O from a windows program, and
>> msvcrt already has about half a dozen pitfalls we _can't_ fix from
>> python, not least being the fact that calling msvcrt.getwch [etc] for
>> the first time is a bell you can't unring.
>> 
>>> Although, come to think of it, I think windows has APIs to set a terminal HANDLE as your terminal for getwch and friends, so maybe a param for that as well?
>> 
>> Not really. There's an API to _close_ the console and open a new one or
>> attach to the one belonging to a different process, but it's very
>> global, there's no way to actually pass it in. And if you're the only
>> process attached to a console and you close it, it goes away. So you
>> could do a very hacky thing with subprocesses and switching on every
>> call, I suppose.
>> 
>> -- 
>> Random832
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20130802/13010d9a/attachment-0001.html>