[Python-ideas] Remove tty module

[Andrew Barnert]

On Aug 2, 2013, at 20:03, random832 at fastmail.us wrote:

> Eh, and everyone _insists_ it's a good idea not to have reply-to
> headers.
> 
> On Fri, Aug 2, 2013, at 19:18, Andrew Barnert wrote:
>> On Aug 2, 2013, at 13:05, random832 at fastmail.us wrote:
>>> Why would it be less portable? Right now, we have curses, which only
>>> works on unix.
>> 
>> Because curses isn't available on every platform termios is.
> 
> Once again, I'm not suggesting _actually using curses_, I'm talking
> about implementing basic functionality from scratch, using just
> terminfo/termcap data.

And I'm suggesting that if you really want to do curses type stuff, you _do_ want to use curses, for the reasons I explained.

> And I don't know what you think is hard about
> getting the screen width... it's just in an environment variable (it's
> handling resize events that is complicated, but we could simply not
> handle them on systems where it doesn't work by the known mechanisms)

First, COLUMNS is not an environment variable, it's a shell variable. Which is why os.environ['COLUMNS'] will raise a KeyError. Or even running a bash subshell. Try this:

    cat >foo
    #!/bin/bash
    echo $COLUMNS
    ^d
    chmod +x foo
    ./foo

Does it print 80, or a blank line?

On top of that, even if you do know how to get it, it's not always correct. For example, if you run a program that ignores SIGWINCH, resize the window, and have it run a child process, the child will get the original width, not the new one.

The usual way around both problems in shell scripts is to use the tput tool... Which is part of curses.

Curses has the appropriate solutions for all of these kinds of problems on all terminals. If you want to reimplement it from scratch, you will have to discover and solve all of them yourself.

> Reading $COLUMNS is not hard. Sending escape sequences is not hard.
> 99.99% of apps on 99.99% of terminals don't need anything else in terms
> of actual output to the terminal.

If you're willing to break Mac scrolling, not interoperable properly with the scrollback buffer in various terminals, hang on Cygwin, not handle terminal width properly in a variety of common cases, ...

>> And it isn't appropriate for every use case where termios is. (For the
>> most dramatic case, getch makes sense on a serial line; gotoxy does
>> not. But even on an actual terminal, there are many cases where you
>> don't want to take over the terminal, break scrollback, etc.)
> 
> I don't get what you think that "taking over the terminal" actually
> involves.

I explained it repeatedly; I'm not sure how else I can explain it.

> And that wouldn't happen unless those functions are actually
> called, anyway. You wouldn't have to go into that mode just to use
> getch. like I said, except for echo, input is independent of output, and
> screen manipulation functions are purely an output thing.
> 
> Also, getch doesn't really make sense on a non-terminal because its
> meaning is tied up with the terminal line discipline. On a non-terminal
> you would use os.read. getch is all about turning on cbreak and turning
> off echo before calling read, then changing it back after.

Raw mode and cbreak mode are not the same thing. Every implementation of getch I've found turns on raw mode (that is, turns off icanon). Do you think this isn't necessary?

>>> Implementing a simple set of functions for both unix and
>>> windows is more portable.
>> 
>> Sure, but implementing an even simpler set of functions that works on a
>> broader range of unix plus windows and can be used in a broader range of
>> cases is even _more_ portable.
> 
> I don't know why you are arguing against implementing these other
> functions. They don't affect the functions you want to implement.

But they do affect getting a module designed, built, tested, used, maintained, and accepted into the stdlib. If the module isn't considered done until it can be used to build portable console GUIs, it will be much longer before it's done. Which is why I think they belong as separate modules. I've never said I don't think your functionality shouldn't exist, just that it shouldn't be shoehorned into the much simpler and smaller module I want to build.

>> And of course it's also simpler, meaning less coding, less debugging,
>> less bikeshedding, etc. 
>> 
>>> I am thinking in terms of "implement core functionality for multiple
>>> platforms, then implement anything more complex on top of that in pure
>>> python" - this necessarily means duplicating some of the work that
>>> curses does now for unix systems. What's wrong with this approach?
>> 
>> Well, duplicating the work of curses instead of just using curses may
>> well be a mistake.
> 
> curses isn't available on windows.

Sure. Which is exactly why I suggested that a fullscreen GUI module should implement a subset of what curses can do instead of all of it, and should do so by using curses on Unix and different functions on Windows.

> The best curses-like implementation
> available for windows doesn't use the console. There is _nothing_
> cross-platform available right now. And curses is overkill for
> _precisely_ the reasons you think it's necessary - because it requires
> you to take over the screen and use its special input functions if you
> want to do any cursor movement.

Exactly. Which is a reason the full screen GUI module should be separate from the raw I/O module.

>> But otherwise, there's nothing wrong with this; it's just a potentially
>> (and, I think, desirably) separate idea from implementing basic raw
>> terminal I/O.
> 
> I don't see why they shouldn't go in the same module.

Again, so the one can get finished quickly while you're just getting started on the other.

> 
>> As I said before, I think we ultimately may want three packages:
>> 
>> * a raw console I/O package
> 
> You keep saying "raw I/O", but this functionality is all about input.
> There's no "O" in your I/O.

putch and friends may not be very powerful, maybe not even very useful, but they are certainly output.

>> Only if it thinks your terminal is UTF-8. Which it shouldn't.
> 
> I see it as an extension of the win32 issue with surrogate pairs - some
> terminal environments may not provide validation for pasted data (or for
> something like a "stuff" keybinding on screen), so even when it's
> supposed to be UTF-8 the next byte may never come. But it's _something_
> that's misconfigured, so it's a low priority.
> 
>> But people writing conio-style code today are using either msvcrt or
>> libraries like python-conio, where it _is_ a problem.
> 
> It's still not the same problem. If you type a genuine multibyte
> character, like in shift-JIS or something, then the second byte is
> available immediately even though you read it in another call, just like
> what we're talking about on unix. There's never an "orphan lead byte"
> like I was saying, where the trail byte might block or might never come.
> The fact that it's a second call is immaterial.
> 
>>>> What about cbreak mode? It's a useful thing, there's no obvious way
>>>> to fit it into the conio-style paradigm, and tty wraps it up for
>>>> you.
>>> 
>>> I'd think cbreak mode basically consists of calling getche() all the
>>> time. What's the difference, other than that?
>> 
>> No, raw/cbreak/cooked is entirely orthogonal to echo/noecho.
> 
> I am not talking about echo/noecho, I just thought by cbreak you meant
> cbreak+echo.

Well, I didn't. By cbreak, I meant cbreak. As opposed to raw and canon/cooked. Any of the three can be used with or without echo.

> The point is both getch and getche are inherently cbreak.

I think they're inherently raw. But whichever one you choose, the point is that there are three primary modes, and my suggested module will only handle one of them, and therefore tty will still be the only easy way to get the third.

> But it sounds like you're actually talking about not disabling signals.

Nobody said anything about disabling signals. The issue (or rather this small subissue) is that different terminal modes map different sequences to signals. How (or whether) you deal with those signals is a separate question that I don't think is even relevant here.

> I actually don't think getch should disable signals, or it should be an
> option passed to it. It was actually implementation-dependent on DOS.
> (and it doesn't disable control-break on windows, only control-C) But in
> both cases you're reading a single character immediately, there's no
> reason to have a persistent "mode" that alters the behavior of os.read.

Yes there is. 

With the obvious posix implementation, if you switch to raw mode for kbhit, switch back to cooked mode, and switch back to raw mode for getch, the character may no longer be immediately available. If we used higher-level functions instead of a direct mapping of conio, so you could getch with a timeout instead of polling kbhit, it would be possible to work around that. But otherwise, it isn't.

And meanwhile, windows has exactly the issue you raised in your previous email if you mix conio and stdio, and I don't know of any way around that except to declare that illegal.

> You're already looking at a kernel context switch on every keystroke, so
> switching the terminal mode on every call isn't a huge cost on top of
> that.

It's a correctness issue, not a performance one.

>> For example,
>> in raw mode, ^C is just a character; in cbreak mode, as in cooked mode,
>> it's a signal. Cbreak basically means to turn off line discipline, but
>> leave on everything else.
>> 
>>> Windows acts weird if you mix line buffering and getch, by the way,
>> 
>> This is a big part of the reason I decided to require enable/disable
>> pairs and just say that normal stdin/out is illegal (but not necessarily
>> checked) inside an enable and consoleio illegal outside of one.
> 
> I don't think that was really necessary. And I think it's contributed to
> your thinking in terms of "taking over the screen", thinking that if we
> add gotoxy you will have to add that to your enable function and it will
> happen to people who don't want it.
> 
>> Read the source to curses. It consists of doing a bunch of additional
>> termios stuff you don't need for raw mode, doing various bizarre
>> workarounds for different platforms, setting env variables to interact
>> with a variety of different terminal emulators, sending a variety of
>> escape sequences to control things like soft labels and take over
>> scrolling and scrollback on terminals that support it, etc. There's a
>> reason lib_newterm.c is 352 lines long.
> 
> And my whole point is NOT DOING those things. None of those are required
> to clear the screen and go to a coordinate point on 99.999% of
> terminals;

So Cygwin makes up less than 0.001%? The first of those workarounds is to prevent a hang on Cygwin. (IIRC, you have to leak a /dev/tty handle before you can use stdin as a tty in some fares. IIR incorrectly... Then I'd write code that hangs.)

> all you have to do is send two escape sequences. In most of
> the remaining .001%, all you have to do is send another escape sequence
> beforehand. You don't even have to turn off cooked mode. People put this
> stuff in their prompts.

Which is why the fancy prompt I built on my old Fedora box doesn't work right when I ssh in from my Mac or my Ubuntu box...

> A lot of curses is also geared to performance, making sure you have the
> shortest possible byte sequence for a screen update, especially on
> terminals
> that don't support region scrolling.

Yes, a lot of it is. But there's also a lot that's about working correctly on different platforms/terminals.

>> But I've repeatedly said that if you want full-screen graphics, you _do_
>> want to use curses, rather than try to reproduce decades worth of
>> development and debugging to get it to work on a wide variety of
>> platforms, handle features you haven't thought of, etc.
> 
> That's cargo-cult thinking. Soft labels? When have you ever used
> a terminal with them? When have you ever heard of an app that sets them?
> And you just listed it off like something naturally no-one could do
> without.

If you're just going to pick one piece out of a paragraph and ignore the rest of it, it's going to be very hard to have a productive discussion. Sure, it's been a long time since I've even thought about soft labels. But scrollback buffers, to take the very next phrase, are something we both use every day. I would also like to have scrolling integrate properly with OS scrolling in Terminal.app, as emacs or any curses or app does, but many other fullscreen apps do not. And so on.

If you want to go through all the features and workarounds curses deals with and decide which ones you do and don't need, of course you can. You also probably want to explore the various alternatives like termbox to see where they fall down and whether you care. I think you'll have a much easier time wrapping a small subset of curses to implement the functionality you want (and writing a separate Windows implementation, of course) than doing it from scratch. But I'm not planning to write that module, so what I think isn't as important here.