[Python-ideas] Suggestion: Clear screen command for the REPL

[eryk sun]

On Tue, Oct 4, 2016 at 2:22 PM, Random832 <random832 at fastmail.com> wrote:
> On Wed, Sep 28, 2016, at 23:36, Chris Angelico wrote:
>> On Thu, Sep 29, 2016 at 12:04 PM, Steven D'Aprano <steve at pearwood.info>
>> wrote:
>> > (Also, it seems a shame that Ctrl-D is EOF in Linux and Mac, but Windows
>> > is Ctrl-Z + Return. Can that be standardized to Ctrl-D everywhere?)
>>
>> Sadly, I suspect not. If you're running in the default Windows
>> terminal emulator (the one a normal user will get by invoking
>> cmd.exe), you're running under a lot of restrictions, and I believe
>> one of them is that you can't get Ctrl-D without an enter.
>
> Well, we could read _everything_ in character-at-a-time mode, and
> implement our own line editing. In effect, that's what readline is
> doing.

3.6+ switched to calling ReadConsoleW, which allows using a 32-bit
control mask to indicate which ASCII control codes should terminate a
read. The control character is left in the input string, so it's
possible to define custom behavior for multiple control characters.
Here's a basic ctypes example of how this feature works. In each case,
after calling ReadConsoleW I enter "spam" and then type a control
character to terminate the read.

    import sys
    import msvcrt
    import ctypes

    kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
    ReadConsoleW = kernel32.ReadConsoleW

    CTRL_MASK = 2 ** 32 - 1 # all ctrl codes

    hin = msvcrt.get_osfhandle(sys.stdin.fileno())
    buf = (ctypes.c_wchar * 10)(*('-' * 10))
    pn = (ctypes.c_ulong * 1)()
    ctl = (ctypes.c_ulong * 4)(16, 0, CTRL_MASK, 0)

    >>> # Ctrl+2 or Ctrl+@ (i.e. NUL)
    ... ret = ReadConsoleW(hin, buf, 10, pn, ctl); print()
    spam
    >>> buf[:]
    'spam\x00-----'

    >>> # Ctrl+D
    ... ret = ReadConsoleW(hin, buf, 10, pn, ctl); print()
    spam
    >>> buf[:]
    'spam\x04-----'

    >>> # Ctrl+[
    ... ret = ReadConsoleW(hin, buf, 10, pn, ctl); print()
    spam
    >>> buf[:]
    'spam\x1b-----'

This could be used to implement Ctrl+D and Ctrl+L support in
PyOS_Readline. Supporting Ctrl+L to work like GNU readline wouldn't be
a trivial one-liner, but it's doable. It has to clear the screen and
also write the input (except the Ctrl+L) back to the input buffer.

> The main consequence of reading everything in character-at-a-time mode
> is that we'd have to implement everything ourselves, and the line
> editing you get *without* doing it yourself is somewhat nicer on Windows
> than on Linux (it supports cursor movement, inserting characters, and
> history).

Line-input mode also supports F7 for a history popup window to select
a previous command; Ctrl+F to search the screen text; text selection
(e.g. shift+arrows or Ctrl+A); copy/paste via Ctrl+C and Ctrl+V (or
Ctrl+Insert and Shift+Insert); and parameterized input aliases ($1-$9
and $* for parameters).

https://technet.microsoft.com/en-us/library/mt427362
https://technet.microsoft.com/en-us/library/cc753867

>> "Bash on Ubuntu on windows" responds to CTRL+D just fine. I don't really
>> know how it works, but it looks like it is based on the Windows terminal
>> emulator.
>
> It runs inside it, but it's using the "Windows Subsystem for Linux",
> which (I assume) reads character-at-a-time and feeds it to a Unix-like
> terminal driver, (which Bash then has incidentally also put in
> character-at-a-time mode by using readline - to see what you get on WSL
> *without* doing this, try running "cat" under bash.exe)

Let's take a look at how WSL modifies the console's global state.
Here's a simple function to print the console's input and output modes
and codepages, which we can call in the background to monitor the
console state:

    def report():
        hin = msvcrt.get_osfhandle(0)
        hout = msvcrt.get_osfhandle(1)
        modeIn = (ctypes.c_ulong * 1)()
        modeOut = (ctypes.c_ulong * 1)()
        kernel32.GetConsoleMode(hin, modeIn)
        kernel32.GetConsoleMode(hout, modeOut)
        cpIn = kernel32.GetConsoleCP()
        cpOut = kernel32.GetConsoleOutputCP()
        print('\nmodeIn=%x, modeOut=%x, cpIn=%d, cpOut=%d' %
              (modeIn[0], modeOut[0], cpIn, cpOut))

    def monitor():
        report()
        t = threading.Timer(10, monitor, ())
        t.start()

    >>> monitor(); subprocess.call('bash.exe')

    modeIn=f7, modeOut=3, cpIn=437, cpOut=437
    ...
    modeIn=2d8, modeOut=f, cpIn=65001, cpOut=65001

See the following page for a description of the mode flags:

    https://msdn.microsoft.com/en-us/library/ms686033

The output mode changed from 0x3 to 0xf, enabling

    DISABLE_NEWLINE_AUTO_RETURN (0x8)
    ENABLE_VIRTUAL_TERMINAL_PROCESSING (0x4)

The input mode changed from 0xf7 to 0x2d8, enabling

    ENABLE_VIRTUAL_TERMINAL_INPUT (0x200)
    ENABLE_WINDOW_INPUT (0x8, probably for SIGWINCH)

and disabling

    ENABLE_INSERT_MODE (0x20)
    ENABLE_ECHO_INPUT (0x4)
    ENABLE_LINE_INPUT (0x2)
    ENABLE_PROCESSED_INPUT (0x1)

So you're correct that it's basically using a raw read, except it's
also translating some input keys to VT100 sequences.

If you Ctrl+Break out of WSL, don't plan to reuse the console for
regular Windows console programs. You could reset the modes and
codepages, but it'll simpler to just open a new console. Here's an
example of the VT100 sequences for the arrow keys after breaking out
of WSL:

    C:\>^[[A^[[B^[[C^[[D

WSL also changes the input and output codepages to 65001 (UTF-8). It
hasn't done anything to fix the console's broken support for non-ASCII
input when using UTF-8. But instead of getting an empty read (i.e.
EOF) like what we see in this case with the cooked read used by
Windows Python, WSL's raw read simply strips out non-ASCII input.
That's simply brilliant. /s