built in to clear terminal
Sorry -- I can't find the thread, but there was a discussion on this list about whether there should be a built in for clearing the terminal screen. I'm was on the fence about it -- but one more data point: One of my beginning Python students just submitted code with this in it: def clear_screen(): ''' Function: clear_screen() (Windows/Unix) Input: None Process: Clear Screen Output: None ''' if (opSys == 'Windows'): os.system('cls') else: os.system('clear') return None Which makes me think -- yes, it would be nice to have that available in the standard lib somewhere. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Thread is here: https://mail.python.org/archives/list/python-ideas@python.org/message/EWQ2BO... I don't think this needs to be a literal built-in, part of the interpreter. I think it could be a library function which is added to builtins on startup, like `help` and `quit`. -- Steve
On Sat, Dec 19, 2020 at 7:43 PM Steven D'Aprano <steve@pearwood.info> wrote:
That sounds like a very special status. Why not os.clear()? I do agree with the motivation for doing this. (I've started doing more exploratory programming and I've felt this need clearly a few times.) I can't comment on the exact semantics or implementation (though I'd prefer it if it didn't have to run a subprocess). IDLE should probably monkey-patch this so it does something reasonable in its shell window. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Sat, Dec 19, 2020 at 8:53 PM Guido van Rossum <guido@python.org> wrote:
Frankly, you can't do hardly any scripting without importing at least os or sys anyway. And "scripting" environments could pre-import whatever they want. I can't comment on the exact semantics or implementation (though I'd prefer
I'm not sure if there's anything to do to make that easier, but it would be good if it were easy for any terminal emulator in IDEs, etc to have it to hook into. I also have no idea about implementation, but I"m sure there's a few folks with platform expertise that could make this work on many systems out of the box. This SO post: ( https://stackoverflow.com/questions/55771796/how-to-clear-the-screen-of-inte...) both provided one more data point about folks wanting this, and a pointer to the fact that iPython already includes something -- though I just found that code, and it's no fancier than what my student came up with: if os.name == 'posix': def _term_clear(): os.system('clear') elif sys.platform == 'win32': def _term_clear(): os.system('cls') else: def _term_clear(): pass (https://github.com/ipython/ipython/blob/master/IPython/utils/terminal.py) is it so bad to use a subprocess? -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 20Dec2020 08:51, Christopher Barker <pythonchb@gmail.com> wrote:
On Sat, Dec 19, 2020 at 8:53 PM Guido van Rossum <guido@python.org> wrote:
at sounds like a very special status. Why not os.clear()?
My anger at programmes which gratuitously clear the screen is large. (Years of anger watching IBM-derived PCs boot, particularly, when I want to see some diagnostic.) One problem is: what does it mean? On a terminal, easy. But in a GUI? Clear the screen (possibly forbidden)? A window? The "main" window? Etc. Anyway, I think it should be in curses (or be loaded via curses on demand), and just have a clear_screen function thus: def clear_screen(): setupterm() print(ti_getstr('cl'), end='', flush=True) I can already see many ways to bikeshed on that, alas. But the point here is that this is trivial function (albeit often wanted, however misguides I might personally consider that want to often be).
On terminals, see above. In a GUI, who knows? And how does one tell the programme which it is talking to?
is it so bad to use a subprocess?
Yes. It is _really slow_, depends on external reaources which might not be there, and subprocess brings other burdens too. Python comes with curses and that knows directly how to do this. Cheers, Cameron Simpson <cs@cskk.id.au>
On Sun, Dec 20, 2020 at 1:23 PM Cameron Simpson <cs@cskk.id.au> wrote:
There are a LOT of bad things one can do with Python, I don't think we need to make something difficult simply because it can be abused. One problem is: what does it mean? On a terminal, easy. But in a GUI?
Clear the screen (possibly forbidden)? A window? The "main" window? Etc.
It would be meaningless outside of a terminal -- call it `clear_term` if you want. Even print() doesn't have a consistent useful meaning in a GUI program. Anyway, I think it should be in curses (or be loaded via curses on
demand),
That would be great, though I just looked at the 3.9 docs and saw: "The Windows version of Python doesn’t include the curses module." So we're pretty much back to square one. I think the idea here is that we want a simple way, out of the box that people can use to clear the terminal, that will work on most commonly configured systems out of the box, and can be overridden (monkey patched) to work in IDEs (e.g. Idle), custom terminal emulators (e.g. iPython Qtconsole), etc. We don't want to require a full curses implementation. and just have a clear_screen function thus:
I just tried that on my mac and tigetstr('cl') simply returns None with no effect. I've only looked for a coupe minutes, but it seems you need to set up curses in order ot use it, so maybe not teh best way to go for a single action? The tutorial seems to indicate there is a clear method available, so I did try that, and it does nothing on my Mac with the default terminal app: def clear_screen(): from curses import wrapper def clr(stdscr): # Clear screen stdscr.clear() wrapper(clr) clear_screen() I may have done something wrong, so anyone please correct -- but it's sure looking to me like curses is not a solution to this problem. here is that this is trivial function clearly not trivial to write for someone new to curses :-)
On terminals, see above. In a GUI, who knows? And how does one tell the programme which it is talking to?
That would be up to the GUI toolkit to provide if it includes a terminal emulator of some sort.
is it so bad to use a subprocess?
Yes. It is _really slow_,
could it possible be slow enough to matter? not the kind of thing that's in a tight loop. depends on external reaources which might not
be there,
oh, like the curses lib ;-)
and subprocess brings other burdens too. Python comes with curses and that knows directly how to do this.
not that I could figure out, and apparently not on Windows. Anyway, the idea (I have) is that this would be somewhere like os.clear_term(), and the implementation would be platform specific -- if on *nix systems, it used curses, great! The other thought is that this has been in iPython, with the simple os.system() implementation, for years, so it eather works well enough, or no one ever uses it :-) I looked at iPython issues, and apparently there is (or was) an issue with Windows if the current working dir is a network mount, but that's all I found. Of course, there are 1400 open issues, so I may have missed one :-) Anyway, not my area of expertise, I just think it's a good idea if someone with the appropriate expertise wants to step up and write it. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 20Dec2020 15:48, Christopher Barker <pythonchb@gmail.com> wrote:
True.
+1
Yeah, Windows.
So we're pretty much back to square one.
No, for Windows we shell out to its command line unless a Windows person knows what to do. We could hardwire the ANSI clear screen sequence as a fallback when there's no curses module, that's easy with a try/import.
Ok. I still think on UNIX we want curses as the preferred choice since to accomodates all terminal types. But a fallback assuming ANSI seems sensible to me, unless there's a Windows counter example. Untested: cls_bs = None def clear_screen(): if cls_bs is None: try: import curses except ImportError: cls_bs=curses.tigetstr('clear') print(cls_bs.encode(), end='', flush=True)
I just tried that on my mac and tigetstr('cl') simply returns None with no effect.
Yeah. 'clear' works though.
Nonono. That requires a full curses init, "full screen curses mode programme" etc. We just want the terminal escape sequence. setupterm() inits the terminal db access, then off we go. As above.
In your imagined use case. Guarenteed one day this will be in a tight loop. In a simulation or something.
Hahaha. Batteries supplied. Cheers, Cameron Simpson <cs@cskk.id.au>
On 12/20/20, Cameron Simpson <cs@cskk.id.au> wrote:
A C or ctypes implementation is required in Windows. Virtual terminal mode is supported in Windows 10, for which it *might* be enabled. This can be queried via GetConsoleMode. If virtual terminal mode isn't enabled, then clearing the screen has to be implemented by scrolling the console screen buffer. The screen buffer size, visible rectangle, and current character attributes can be queried via GetConsoleScreenBufferInfo. The window can be scrolled via ScrollConsoleScreenBuffer. The entire buffer can be scrolled out, like the CMD shell's CLS command, or one can just scroll the buffer enough to clear the visible window. The cursor can be set to the home position via SetConsoleCursorPosition.
On Sun, Dec 20, 2020 at 8:06 PM Eryk Sun <eryksun@gmail.com> wrote:
The Windows console model is a bit different from POSIX terminals. Instead of having a screen and a scrollback buffer, you have a buffer and a window/view into it. If you want clear() to just erase the screen and preserve "scrollback", the obvious thing to do is to clear the window/view. The problem is that if the user happens to be looking at "scrollback" when clear() is called, whatever they're looking at will be erased, not whatever was written since the last clear(). I wonder if the best approach wouldn't be to set the top-of-screen line on the first call of clear() (perhaps to one more than the current cursor line) and then use that line on all subsequent calls regardless of the current window or cursor position. That way scrollback would be safe both from accidental erasure and from pollution by programs that use clear() as a poor man's curses by erasing and redrawing everything whenever anything changes. I don't think there would need to be any interface to clear the first-line cache, since the whole thing is just a cheap hack anyway. I think it'd be a good idea to include some sort of proper Windows console interface in the standard library. It's only fair since other platforms already have curses. Then os.clear() could be implemented on top of it, and if you didn't like its behavior you could import the real library and do what you want.
On Sun, Dec 20, 2020 at 4:12 PM Cameron Simpson <cs@cskk.id.au> wrote:
darn that untested code -- a lot wrong here :-( But unpacking it, I've come up with this, which seems to work. import os import sys # Define clear_screen function: try: import curses curses.setupterm() CLS_BS = curses.tigetstr('clear') def clear_screen(): sys.stdout.buffer.write(CLS_BS) except: # if anything went wrong, fall back to os.system call if os.name == 'posix': def clear_screen(): os.system('clear') elif sys.platform == 'win32': # ideally something lower level on Windows def clear_screen(): os.system('cls') else: def clear_screen(): pass On Sun, Dec 20, 2020 at 8:06 PM Eryk Sun <eryksun@gmail.com> wrote:
A C or ctypes implementation is required in Windows.
<snip> so that could be plugged in to the above, with (or not) a fallback to the system calls. Anyway, this has been interesting enough to distract me from other things I should be doing, but I'm moving on now ... -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
The simplest answer is print('`\x1b[2J\x1b[H') Are there any terminals that this does not work on that are in active use? Is using curses that uses termcap needed these days? Of course Windows is the outlier, but the new Windows Terminal supports ANSI escapes sequences and utf-8. I tested the above with Windows Terminal 1.4 on Windows 10 and it just works. Otherwise os.system('cls') works for windows terminal and the old windows console stuff. Barry
It turns out that on Windows 10 it works for old console API if you do this: ----- import sys def clear_terminal(): if sys.platform == 'win32': import ctypes kernel32 = ctypes.windll.kernel32 # turn on the console ANSI colour handling kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7) sys.stdout.write('\x1b[2J' '\x1b[H') ----- The above should work in all but the old none ANSI terminals. Barry
On 12/22/20, Barry Scott <barry@barrys-emacs.org> wrote:
Here are some concerns I have: * Does not support Windows 8 * Does not support legacy console in Windows 10 (on the "options" tab) * Does not check for SetConsoleMode failure * Does not support a different active screen buffer * Assumes StandardOutput is a screen buffer for the current console * Assumes the current mode of the screen buffer is 3 or 7. New modes have been added, and even more may be added * Sets a global console setting that persists after Python exits Like the CRT's "conio" API, clear_screen() should open "conout$" (temporarily), which will succeed if Python is attached to a console, regardless of the state of the standard handles, file descriptors, or sys.stdout, and will always open the currently active screen buffer, regardless of how many screen buffers exist in the current console session. The current mode should be queried for ENABLE_VIRTUAL_TERMINAL_PROCESSING (4) via GetConsoleMode(). If it's not enabled, bitwise OR it into the mode and try to enable it via SetConsoleMode(). If VT mode is enabled, write '\x1b[2J\x1b[H' to the file. If VT mode can't be enabled, then fall back on the legacy console API. In particular, some people mentioned not wanting to spawn a cmd.exe process just to use its CLS command. Even if spawning a process is okay, the CLS command clears the scrollback, which is inconsistent with ESC[2J. If clear_screen() is going to add ESC[3J to clear the scrollback, then it's at least consistent, but I'd rather not clear the scrollback. clear_screen() should be able to emulate ESC[2J via GetConsoleScreenBufferInfoEx (get the screen buffer size, window, cursor position, and default character attributes), ScrollConsoleScreenBuffer (if the screen buffer has to be scrolled up to make space), and SetConsoleScreenBufferInfoEx (shift the visible window in the buffer and set the cursor position). This can be implemented in ctypes or C. But normally the standard library avoids using ctypes. Finally, if VT mode was temporarily enabled, revert to the original mode, and always close the "conout$" file. Off topic comment:
kernel32 = ctypes.windll.kernel32
I recommend the following instead: kernel32 = ctypes.WinDLL('kernel32', use_last_error=True) The global library loaders such as ctypes.cdll and ctypes.windll are not reliable for production code in the wild. They cache CDLL library instances, which cache function pointers, which may have argtypes, restype, and errcheck prototypes. Another imported package might set function prototypes that break your code, which is an actual problem that I've seen a few times, particularly with common routines from kernel32, advapi32, and user32. It's not worth taking the chance of a conflict with another package just to save a few keystrokes. The global loaders also don't allow setting use_errno=True or use_last_error=True, so the function pointers they create don't capture the C errno value for ctypes.get_errno() or Windows last error value for ctypes.get_last_error(). Calling kernel32.GetLastError() after the fact may not be reliable in a scripting environment even if it's called directly after the previous FFI call.
snip... Eryk, Does this work for you: import sys, os def clear(): if sys.playform == 'win32': os.system('cls') else: sys.stdout.write('\x1b[2J' '\x1b[H') No need to address the list above because CLS does it. No concern about CLS being a trojan becuse as a builtin to CMD and PowerShell and it takes priority over cls.bat etc. For all other platforms I'm assuming the VT100 level of ANSI escape sequences is supported. Barry
On Thu, Dec 24, 2020 at 1:34 AM Barry Scott <barry@barrys-emacs.org> wrote:
Because it's a shell builtin, you have to run the shell to use it. If it wasn't a shell builtin, you could run it directly. Either way you have to make sure you're running the right executable. I suppose finding the right shell is easier since you can use COMSPEC, but if you don't trust the path then I'm not sure you should trust COMSPEC. As Eryk Sun said, cls erases the scrollback, while your non-win32 code doesn't. The popular cmd.exe replacement TCC/LE seems to have different default behavior for cls, erasing only the visible text by default. You can add /c to match cmd.exe's behavior, but cls/c is an error in cmd.exe. TCC also has cls/s which moves the visible text into the scrollback, which might be the best behavior for Python, but if we want that we'll have to do it in-process since we can't rely on TCC being present. Spawning processes that run for 1ms and do a trivial thing is common in the Unix world but it's not idiomatic in Windows. It feels like a CVE in the making. (Even on Unix, to be honest.)
well, my two cents here is that I do clear my screen - a lot. In the same use case as Guido's mentioned - exploratory programming. But when I do it, I do in a manner that was discussed in the original thread - by using CTRL+L, which is of course terminal-specific but clears everything both in Python's REPL and ipython. As a comparison, I also code almost daily in Elixir, which has a REPL as well, but for some weird reason, CTRL+L does not work there. [image: --] Felipe V. Rodrigues [image: https://]felipevr.com <https://felipevr.com> On Sun, Dec 20, 2020 at 8:48 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Mon, Dec 21, 2020 at 08:22:35AM +1100, Cameron Simpson wrote:
[...]
Do you have benchmarks showing that shelling out to "clear" or "cls" is actually slower than running the full curses machinery for this? IPython claims that the magic `%clear` command shells out to `clear`, are there complaints about the time it takes to clear the screen? On my computer, shelling out to "clear" takes about one hundredth of a millisecond, which in interactive use is pretty much instantaneous. Do you have examples of tight loops where this would be a bottleneck? Other than performance, are there any other reasons why we shouldn't just use `os.system('clear')` (or equivalent on Windows) and delegate to the shell command for this? I guess there could be unusual platforms with no shell clear command, so we might need a fallback. -- Steve
On Wed, Dec 23, 2020 at 9:15 AM Steven D'Aprano <steve@pearwood.info> wrote:
A tight loop clearing the screen? If you have that, you have much bigger problems :) And yes, I've seen that happen, usually because something doesn't know about cursor movement calls. (It looks UGLY.) But shelling out to clear/cls has another problem: it's a big ol' dependency. You need to have the corresponding command, and it needs to be accessible on $PATH, and so on and so forth. I'd rather a more complicated system of ANSI escape sequence management than something that depends on an external command; it's only going to be a half page of code to do it inside Python. ChrisA
On Tue, Dec 22, 2020 at 10:26 PM Chris Angelico <rosuav@gmail.com> wrote:
I worry about the security concerns of having a common interactive command actually utilize an external executable. Yes, I trust the version of `clear` packaged with my Linux distribution, but it feels like sneaking in something malicious onto the path is a possible problem. I don't have a precise threat model in mind, but if you think of things like online hosted interpreters, or remote limited access to shells, or stuff like that, it starts to feel plausible that someone could do something bad. I'm not sure about Windows. Is 'cls' built into the command-line executable itself (like Busybox) or is it an exe? Somewhat supporting my concern, I just was surprised to find this: % ll `which clear` -rwxrwxr-x 2 dmertz 14344 Nov 14 17:07 /home/dmertz/miniconda3/bin/clear % ll /usr/bin/clear -rwxr-xr-x 1 root 14656 Feb 29 2020 /usr/bin/clear Mind you, I generally trust the conda/conda-forge folks. But I absolutely would not have guessed that I run a different `clear` depending on whether I'm in an environment (and different environments each have their own `clear` it seems). This is not a symlink, but an actual different executable with a different size. In fact: % find ~/miniconda3/envs -name 'clear' | xargs wc -c 14344 /home/dmertz/miniconda3/envs/cleaning3.8/bin/clear 14344 /home/dmertz/miniconda3/envs/INE/bin/clear 14344 /home/dmertz/miniconda3/envs/Pearson-ML/bin/clear 14344 /home/dmertz/miniconda3/envs/cleaning3.9/bin/clear 8012 /home/dmertz/miniconda3/envs/py3.4/bin/clear 14288 /home/dmertz/miniconda3/envs/cleaning/bin/clear 14296 /home/dmertz/miniconda3/envs/pypy/bin/clear 14344 /home/dmertz/miniconda3/envs/py3.9/bin/clear 14296 /home/dmertz/miniconda3/envs/py3.8/bin/clear 8012 /home/dmertz/miniconda3/envs/py3.5/bin/clear 14296 /home/dmertz/miniconda3/envs/play/bin/clear 144920 total Apparently I have at least 5 different `clear` executables installed on my system... and I only learned that in the last 2 minutes. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Tue, Dec 22, 2020 at 2:58 PM David Mertz <mertz@gnosis.cx> wrote:
that is, indeed surprising. But also maybe an argument for using it -- it can be supplied in by conda (and others) , and thus we don't need to count on the system having it. Though, yeah shelling out to an arbitrary command on the system is a bit scary -- does the Python stdlib currently do that anywhere? In fact:
interesting .. any idea what conda package is providing it? -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
As I say, I just discovered the fact about conda vendoring `clear`. I haven't looked further into what includes it as dependency... although it seems to be in pretty much all my environments, so maybe IPython? On Wed, Dec 23, 2020 at 12:46 AM Christopher Barker <pythonchb@gmail.com> wrote:
-- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Tue, Dec 22, 2020 at 4:46 PM Christopher Barker <pythonchb@gmail.com> wrote:
Though, yeah shelling out to an arbitrary command on the system is a bit scary -- does the Python stdlib currently do that anywhere?
Here's what I found by grepping the sources for uses of os.system and subprocess: * pydoc shells out to "more" on win32 and "less" or "more" elsewhere if no pager is configured. Incidentally, on Windows it passes the docs to the pager via a temp file, with the comment "pipes completely broken in Windows" - is this left over from the Win9x days? Also the temp-file code runs the pager with os.system instead of Popen, and doesn't properly quote it. * ctypes.util has a ton of hacky code for finding libraries, which includes calling out to gcc, cc, ld, objdump, /sbin/ldconfig, and /usr/ccs/bin/dump on various platforms. * platform.architecture() calls file and ad-hoc parses its output, except if sys.platform in ('dos', 'win32', 'win16'). On those three platforms it seems to be totally broken, always returning the pointer size of the current Python process no matter what executable you pass to it. * webbrowser looks for a bunch of specific named browsers. On Windows it'll run any of ("firefox", "firebird", "seamonkey", "mozilla", "netscape", "opera") from the insecure search path that starts with the current directory.
On 12/22/20, David Mertz <mertz@gnosis.cx> wrote:
CLS is an internal command of the CMD shell. An internal command takes precedence as long as the executed name is unquoted and has no extension. My concern with cls/clear is that they clear the scrollback. Is that what most people want from a clear_screen() function?
On Sat, Dec 19, 2020 at 08:50:57PM -0800, Guido van Rossum wrote:
I don't care where the implementation lives. But I think that it should be added to builtins for interactive use. It seems to me that the primary use for this will be in the interactive interpreter. People are used to something like "cls" or "clear" from various shells. Using it in scripting is, I think, a secondary use-case. Given that, I think that like other conveniences for interactive use like `help`, it should be available without an import. -- Steve
On 2020-12-22 14:01, Steven D'Aprano wrote:
Given that, I think that like other conveniences for interactive use like `help`, it should be available without an import.
Agreed in general, however Ctrl+L is mostly available for this purpose. There are two major exceptions unfortunately. One is when using cmd.exe. However, every other combination of shell and console/terminal on Windows does support it. Not to mention everywhere Unix has an influence. Second is the Python REPL console on Windows. I'm guessing it wouldn't be too hard to add Ctrl+L handling to it. Would be useful I think, regardless of some of the other decisions that need to be made. -Mike
' I disagree. yes, I want to clear the screen a lot during interactive use, but I'd rather do that with a keystroke (cmd+K on the Mac) -- and indeed, do that with iPython all the time, even though iPython has the command built in. So I think the reason to have it in the std lib is for scripting, and I think putting it in os or sys is just fine. -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sat, Dec 26, 2020 at 05:58:16PM -0800, Christopher Barker wrote:
And nobody is going to take that away from you, unless you choose to go to Windows, where ctrl-L doesn't work. But the ctrl-L trick has no discoverability. It took me close to twenty years of using Linux before I discovered it, and I still don't remember to use it when I need it. Beginners and casual users aren't going to know ctrl-L, or stumble across it through experimentation. People like me aren't going to remember to use it. For interactive use, os.clear is barely any better. (Me, I'll be looking in shutils, not os, because this is a shell feature, not an os feature.)
So I think the reason to have it in the std lib is for scripting, and I think putting it in os or sys is just fine.
Putting it in sys requires it to be built into the interpreter, which means it has to be written in C (for CPython). Putting it in os is fine, that can easily delegate to platform specific code in posix.py, nt.py etc, with a pure-Python lowest common denominator fallback for weird platforms that don't support anything sensible. I just want site.py to import it at startup for the sake of beginners. Is that so bad? It adds one line to site.py, four if you want to make it conditional: try: from os import clear except: pass I can easily add it to my own personal PYTHONSTARTUP file, but the average beginner can't. Let's throw newbies a bone and pre-import it for them, they're the ones who seem to expect it the most. So that's the easy part: just agree with me and import it in site.py, wherever it happens to come from. *wink* The hard part is deciding exactly what it should do, e.g. clear the scrollback or not. I think Chris makes a good point that real beginners will misuse this to hide the evidence whenever they get an error, so I think the default should be to *not* clear the scrollback, for the sake of educators, teachers and trainers. People who know what the scrollback is and definitely want to clear it can just provide an extra argument to the function. So let's just all agree I'm right *wink* and move on to the *really* hard part, which is ensuring it works on Windows and weird terminals. -- Steve
On Sun, Dec 27, 2020 at 1:25 PM Steven D'Aprano <steve@pearwood.info> wrote:
True. Plus, despite my misgivings about how it'd be misused, this IS a useful feature to have within a program, and making a program press Ctrl-L isn't reliable.
os or shutil would be fine with me, and I know for sure that a lot of people will look in the wrong one. My inclination would be for os, since shutil is described as "high level FILE operations", and is mainly focused on file system rather than console. But that's bikeshedding.
This is actually the exact part that I think should *not* happen, for several reasons: 1) Stuff that exists interactively and doesn't exist in a script is a barrier to that migration. I don't think this feature justifies that barrier. 2) The builtins are already incredibly busy, and this, again, doesn't justify being one 3) Beginners already have to learn about importing (eg "from math import sin") and this is just one more importable.
So that's the easy part: just agree with me and import it in site.py, wherever it happens to come from. *wink*
This is python-ideas! We don't "just agree" here, we argue! Even if we don't have to!
Okay, so I think there are a few things we can agree on. * Having a function to clear the screen can be abused, but is still worth having * It shouldn't be based on os.system, which top SO results are advocating. * It should work on all OSes, consoles, etc, if possible * It doesn't have to work in a GUI; Idle can monkeypatch it My personal shed colour preferences: * os module, not sys, not shutil, not builtin * clearscreen(), not cls(), although I'd accept clear() * Suppressing scrollback is a downside, but better than not having the feature at all * Beginners should NOT be encouraged to use this too much The last one is hard to define, but I'm generally disinclined towards any proposal that makes it too easy for a beginner to clear the screen all the time. Life is SO much easier as an educator if you can do post-mortem analysis, not just of the one most recent action, but of all previous ones. Even among more advanced students, I see far too many cases where they try to "tidy up" (by clearing the screen, or closing the terminal and opening a new one, or whatever), and in doing so, throw away valuable error messages or indicative commands (like if someone wrote "os = 'Linux'" and then broke all os.x() calls). ChrisA
On Sun, Dec 27, 2020 at 02:00:00PM +1100, Chris Angelico wrote:
It will still exist in a script, you can still import it from the os module if you care about supporting cases where site.py doesn't get run. This is no different from `exit`: [steve ~]$ python3.9 -c "print(exit)" Use exit() or Ctrl-D (i.e. EOF) to exit [steve ~]$ python3.9 -S -c "print(exit)" Traceback (most recent call last): File "<string>", line 1, in <module> NameError: name 'exit' is not defined Well, actually, it will be slightly different: the pseudo-builtin `exit` is a special object created for interactive use, it's not the same as `sys.exit`. Whereas I invisage that the interactive `clear` will be identical (same object) as `os.clear`.
2) The builtins are already incredibly busy, and this, again, doesn't justify being one
It's not an *actual* builtin, it's just monkey-patched into the builtins by site.py. I don't think there is sufficient justification for this to make it a genuine builtin like `len` or `iter`, but I think loading it into the builtin namespace for the convenience of the user is plenty justified. Having clear available immediately in the interactive interpreter is certainly more useful than having the licence, credits and copyright notice, and yet there they are. Probably the three least used builtins ever.
3) Beginners already have to learn about importing (eg "from math import sin") and this is just one more importable.
As far as I can tell, every language that supports a clear/cls command, from old BASIC to modern shells, makes it available immediately. You don't have to adjust the PATH in bash to get access to clear, it just works. Think of this as the companion to print. If you think about Python as an interactive interpreter, its weird to have print, help, quit available but not clear. Its a fairly fundamental interactive command.
I disagree! *wink* -- Steve
On Sun, Dec 27, 2020 at 4:26 PM Steven D'Aprano <steve@pearwood.info> wrote:
Oh, sorry, my bad. Withdraw point 1 then.
But how's that different? Either it's a builtin or it's not. How it gets there is mostly irrelevant, other than to make a separate monkeypatching layer (such as would need to be done by Idle) more complicated.
Ohh, do you mean how some functions are intrinsic parts of the interpreter? I'm not talking about that; I'm talking about how it's accessed, ie the namespacing.
Perhaps, but they have legal reasons for existing.
Old BASIC didn't *have* an import feature (at least, the BASICs that I used didn't), and modern shells have a ton of shell-related stuff in their "builtins" but nothing else, because that's their focus. You can't, for instance, do trig functions in the basic shell, nor even arithmetic, because those aren't builtins. Different environments focus on different things.
I don't actually agree. Having the ability to print? Yes, that's fundamental. Being able to quit? Of course you need that. But there's no cursor movement function in the builtins, nor is there a "show this image" function, nor "display this web page", or various things like that. In terms of console actions, the two most fundamental are print and input, and we get both of those. Everything else has to justify itself in terms of utility and convenience; getting documentation, for instance, could easily have been buried away in a module, but it is *incredibly* helpful to be able to pull up help(x) without an import. We don't need a full set of cursor movement and screen manipulation features as builtins - that's the job of curses. I don't think we really need screen clearing that way either, particularly as the obvious name "clear" is also used in various other contexts, such as lists and dicts. (How confusing would it be to say clear(lst) instead of lst.clear() and then try to figure out what's going on?)
Is this one of those cases where we "agree to disagree"? *wink* ChrisA
On 2020-12-27 at 13:24:34 +1100, Steven D'Aprano <steve@pearwood.info> wrote:
The first time a sysadmin added readline to one of our work computers, it took me *minutes* to be stunned that my terminal window was being cleared for no reason. It took me longer to discover that the reason was accidental Ctrl-L's (no doubt an off-by-one error trying to press Ctrl-K), and way longer to figure out what piece of [software] was intercepting that keystroke and clearning my screen. The very first line in my .inputrc file is as follows: Control-L: redraw-current-line
Technically, I was a professional, and I stumbled on it through fat fingers and not knowing that readline had been installed and did horrible things like that out of the box. And then I disabled it as quickly as I could figure out how. Call me 0 (because I don't care enough to be +0 or -0) on the whole thing, as long as it's spelled with more than one keystroke (to prevent accidental invocation).
On 2020-12-26 19:45, 2QdxY4RzWzUUiLuE@potatochowder.com wrote:
No offense, but if you're a developer who can't handle hotkeys or came of age before the "glass tty" and won't learn any new tricks… well, that's just *"your problem, mayan."* The rest of us are capable. :D Presumably you'd have to find the appropriate help page to discover this feature, whether the os module, interactive interpreter page, or stackoverflow. Discoverability is not a huge problem when just about every interpreter implements the hotkey, save python on Windows and legacy cmd.exe, it's once and done. (At least cmd.exe can be replaced with the yori shell, which I often install when forced to use Windows.) I entered a bug for it, and found a related one for IDLE: https://bugs.python.org/issue42771 (Windows interactive console) https://bugs.python.org/issue6143 (Idle) Would be great if these could get looked at. Implementation doesn't mean other enhancements couldn't be made as well. -Mike p.s. Ctrl+L meant form-feed in ASCII so there is prior history.
Mike Miller writes:
The thing is, this is a destructive capability. In some cases it only clears the "screen" (whatever that means), which may cost you some diagnostics. Probably (but not always) you can recreate them by repeating a command from history. But in some cases it also clears the scrollback buffer. That makes it kinda hard to figure out what's going on unless you know what's going on. I don't need it ever and I don't hit Ctrl-L accidentally, so I'm =0 on adding the feature. But it's something that needs some care in implementation. Steve
On 2020-12-28 23:33, Stephen J. Turnbull wrote:
Well, the time to push back on the feature was in the mid 1970s. Today, if you're at a command prompt or repl, there's a very high chance the feature already exists, with few exceptions. Consistency in UI is still a goal though it has taken a beating the last few years. Also, I'd argue the likelihood of a newbie clearing important work, or an expert making a significant mistake in this manner is not very high. The readline (or equivalent) configuration can mitigate the latter if need be. -Mike
On 2020-12-29 15:07, Chris Angelico wrote:
Note, I added the word *important* on purpose. ;-) Reminds me of a story. As an teen I learned the lesson of what happens when you flip the power switch of the computer in the middle of writing an as-yet-unsaved document. It was an important lesson, but the particular document wasn't. Nor were any for a number of years. But, ever since I make sure work is saved frequently, and avoid power buttons. :D I don't try to prevent their existence, due to their utility. -Mike
Mike Miller writes:
Note, I added the word *important* on purpose. ;-)
Ie, since you were lucky enough the time you did something like it, it's not important to anybody?
Reminds me of a fact: modern personal computers don't really have an off switch. They have menu items and keys that take several hundred milliseconds to react. Seems to me that that is most likely because a lot of folks weren't as lucky with at least one of the times they shutdown in the middle of a save. And then what was that fysnc-on-close thread about? Again, I'm not against the Ctrl-L feature or some clear function (I'm -0.5 on a builtin, but if the module maintainers for sys or os are willing, I'm perfectly happy to ignore it), but some care should be taken that it (1) not clear scrollback or history unless explicitly requested, and (2) not be easy to inadvertantly invoke. I don't think (2) is at all difficult, but it should be checked before distribution. Steve
Thread is here: https://mail.python.org/archives/list/python-ideas@python.org/message/EWQ2BO... I don't think this needs to be a literal built-in, part of the interpreter. I think it could be a library function which is added to builtins on startup, like `help` and `quit`. -- Steve
On Sat, Dec 19, 2020 at 7:43 PM Steven D'Aprano <steve@pearwood.info> wrote:
That sounds like a very special status. Why not os.clear()? I do agree with the motivation for doing this. (I've started doing more exploratory programming and I've felt this need clearly a few times.) I can't comment on the exact semantics or implementation (though I'd prefer it if it didn't have to run a subprocess). IDLE should probably monkey-patch this so it does something reasonable in its shell window. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On Sat, Dec 19, 2020 at 8:53 PM Guido van Rossum <guido@python.org> wrote:
Frankly, you can't do hardly any scripting without importing at least os or sys anyway. And "scripting" environments could pre-import whatever they want. I can't comment on the exact semantics or implementation (though I'd prefer
I'm not sure if there's anything to do to make that easier, but it would be good if it were easy for any terminal emulator in IDEs, etc to have it to hook into. I also have no idea about implementation, but I"m sure there's a few folks with platform expertise that could make this work on many systems out of the box. This SO post: ( https://stackoverflow.com/questions/55771796/how-to-clear-the-screen-of-inte...) both provided one more data point about folks wanting this, and a pointer to the fact that iPython already includes something -- though I just found that code, and it's no fancier than what my student came up with: if os.name == 'posix': def _term_clear(): os.system('clear') elif sys.platform == 'win32': def _term_clear(): os.system('cls') else: def _term_clear(): pass (https://github.com/ipython/ipython/blob/master/IPython/utils/terminal.py) is it so bad to use a subprocess? -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 20Dec2020 08:51, Christopher Barker <pythonchb@gmail.com> wrote:
On Sat, Dec 19, 2020 at 8:53 PM Guido van Rossum <guido@python.org> wrote:
at sounds like a very special status. Why not os.clear()?
My anger at programmes which gratuitously clear the screen is large. (Years of anger watching IBM-derived PCs boot, particularly, when I want to see some diagnostic.) One problem is: what does it mean? On a terminal, easy. But in a GUI? Clear the screen (possibly forbidden)? A window? The "main" window? Etc. Anyway, I think it should be in curses (or be loaded via curses on demand), and just have a clear_screen function thus: def clear_screen(): setupterm() print(ti_getstr('cl'), end='', flush=True) I can already see many ways to bikeshed on that, alas. But the point here is that this is trivial function (albeit often wanted, however misguides I might personally consider that want to often be).
On terminals, see above. In a GUI, who knows? And how does one tell the programme which it is talking to?
is it so bad to use a subprocess?
Yes. It is _really slow_, depends on external reaources which might not be there, and subprocess brings other burdens too. Python comes with curses and that knows directly how to do this. Cheers, Cameron Simpson <cs@cskk.id.au>
On Sun, Dec 20, 2020 at 1:23 PM Cameron Simpson <cs@cskk.id.au> wrote:
There are a LOT of bad things one can do with Python, I don't think we need to make something difficult simply because it can be abused. One problem is: what does it mean? On a terminal, easy. But in a GUI?
Clear the screen (possibly forbidden)? A window? The "main" window? Etc.
It would be meaningless outside of a terminal -- call it `clear_term` if you want. Even print() doesn't have a consistent useful meaning in a GUI program. Anyway, I think it should be in curses (or be loaded via curses on
demand),
That would be great, though I just looked at the 3.9 docs and saw: "The Windows version of Python doesn’t include the curses module." So we're pretty much back to square one. I think the idea here is that we want a simple way, out of the box that people can use to clear the terminal, that will work on most commonly configured systems out of the box, and can be overridden (monkey patched) to work in IDEs (e.g. Idle), custom terminal emulators (e.g. iPython Qtconsole), etc. We don't want to require a full curses implementation. and just have a clear_screen function thus:
I just tried that on my mac and tigetstr('cl') simply returns None with no effect. I've only looked for a coupe minutes, but it seems you need to set up curses in order ot use it, so maybe not teh best way to go for a single action? The tutorial seems to indicate there is a clear method available, so I did try that, and it does nothing on my Mac with the default terminal app: def clear_screen(): from curses import wrapper def clr(stdscr): # Clear screen stdscr.clear() wrapper(clr) clear_screen() I may have done something wrong, so anyone please correct -- but it's sure looking to me like curses is not a solution to this problem. here is that this is trivial function clearly not trivial to write for someone new to curses :-)
On terminals, see above. In a GUI, who knows? And how does one tell the programme which it is talking to?
That would be up to the GUI toolkit to provide if it includes a terminal emulator of some sort.
is it so bad to use a subprocess?
Yes. It is _really slow_,
could it possible be slow enough to matter? not the kind of thing that's in a tight loop. depends on external reaources which might not
be there,
oh, like the curses lib ;-)
and subprocess brings other burdens too. Python comes with curses and that knows directly how to do this.
not that I could figure out, and apparently not on Windows. Anyway, the idea (I have) is that this would be somewhere like os.clear_term(), and the implementation would be platform specific -- if on *nix systems, it used curses, great! The other thought is that this has been in iPython, with the simple os.system() implementation, for years, so it eather works well enough, or no one ever uses it :-) I looked at iPython issues, and apparently there is (or was) an issue with Windows if the current working dir is a network mount, but that's all I found. Of course, there are 1400 open issues, so I may have missed one :-) Anyway, not my area of expertise, I just think it's a good idea if someone with the appropriate expertise wants to step up and write it. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 20Dec2020 15:48, Christopher Barker <pythonchb@gmail.com> wrote:
True.
+1
Yeah, Windows.
So we're pretty much back to square one.
No, for Windows we shell out to its command line unless a Windows person knows what to do. We could hardwire the ANSI clear screen sequence as a fallback when there's no curses module, that's easy with a try/import.
Ok. I still think on UNIX we want curses as the preferred choice since to accomodates all terminal types. But a fallback assuming ANSI seems sensible to me, unless there's a Windows counter example. Untested: cls_bs = None def clear_screen(): if cls_bs is None: try: import curses except ImportError: cls_bs=curses.tigetstr('clear') print(cls_bs.encode(), end='', flush=True)
I just tried that on my mac and tigetstr('cl') simply returns None with no effect.
Yeah. 'clear' works though.
Nonono. That requires a full curses init, "full screen curses mode programme" etc. We just want the terminal escape sequence. setupterm() inits the terminal db access, then off we go. As above.
In your imagined use case. Guarenteed one day this will be in a tight loop. In a simulation or something.
Hahaha. Batteries supplied. Cheers, Cameron Simpson <cs@cskk.id.au>
On 12/20/20, Cameron Simpson <cs@cskk.id.au> wrote:
A C or ctypes implementation is required in Windows. Virtual terminal mode is supported in Windows 10, for which it *might* be enabled. This can be queried via GetConsoleMode. If virtual terminal mode isn't enabled, then clearing the screen has to be implemented by scrolling the console screen buffer. The screen buffer size, visible rectangle, and current character attributes can be queried via GetConsoleScreenBufferInfo. The window can be scrolled via ScrollConsoleScreenBuffer. The entire buffer can be scrolled out, like the CMD shell's CLS command, or one can just scroll the buffer enough to clear the visible window. The cursor can be set to the home position via SetConsoleCursorPosition.
On Sun, Dec 20, 2020 at 8:06 PM Eryk Sun <eryksun@gmail.com> wrote:
The Windows console model is a bit different from POSIX terminals. Instead of having a screen and a scrollback buffer, you have a buffer and a window/view into it. If you want clear() to just erase the screen and preserve "scrollback", the obvious thing to do is to clear the window/view. The problem is that if the user happens to be looking at "scrollback" when clear() is called, whatever they're looking at will be erased, not whatever was written since the last clear(). I wonder if the best approach wouldn't be to set the top-of-screen line on the first call of clear() (perhaps to one more than the current cursor line) and then use that line on all subsequent calls regardless of the current window or cursor position. That way scrollback would be safe both from accidental erasure and from pollution by programs that use clear() as a poor man's curses by erasing and redrawing everything whenever anything changes. I don't think there would need to be any interface to clear the first-line cache, since the whole thing is just a cheap hack anyway. I think it'd be a good idea to include some sort of proper Windows console interface in the standard library. It's only fair since other platforms already have curses. Then os.clear() could be implemented on top of it, and if you didn't like its behavior you could import the real library and do what you want.
On Sun, Dec 20, 2020 at 4:12 PM Cameron Simpson <cs@cskk.id.au> wrote:
darn that untested code -- a lot wrong here :-( But unpacking it, I've come up with this, which seems to work. import os import sys # Define clear_screen function: try: import curses curses.setupterm() CLS_BS = curses.tigetstr('clear') def clear_screen(): sys.stdout.buffer.write(CLS_BS) except: # if anything went wrong, fall back to os.system call if os.name == 'posix': def clear_screen(): os.system('clear') elif sys.platform == 'win32': # ideally something lower level on Windows def clear_screen(): os.system('cls') else: def clear_screen(): pass On Sun, Dec 20, 2020 at 8:06 PM Eryk Sun <eryksun@gmail.com> wrote:
A C or ctypes implementation is required in Windows.
<snip> so that could be plugged in to the above, with (or not) a fallback to the system calls. Anyway, this has been interesting enough to distract me from other things I should be doing, but I'm moving on now ... -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
The simplest answer is print('`\x1b[2J\x1b[H') Are there any terminals that this does not work on that are in active use? Is using curses that uses termcap needed these days? Of course Windows is the outlier, but the new Windows Terminal supports ANSI escapes sequences and utf-8. I tested the above with Windows Terminal 1.4 on Windows 10 and it just works. Otherwise os.system('cls') works for windows terminal and the old windows console stuff. Barry
It turns out that on Windows 10 it works for old console API if you do this: ----- import sys def clear_terminal(): if sys.platform == 'win32': import ctypes kernel32 = ctypes.windll.kernel32 # turn on the console ANSI colour handling kernel32.SetConsoleMode(kernel32.GetStdHandle(-11), 7) sys.stdout.write('\x1b[2J' '\x1b[H') ----- The above should work in all but the old none ANSI terminals. Barry
On 12/22/20, Barry Scott <barry@barrys-emacs.org> wrote:
Here are some concerns I have: * Does not support Windows 8 * Does not support legacy console in Windows 10 (on the "options" tab) * Does not check for SetConsoleMode failure * Does not support a different active screen buffer * Assumes StandardOutput is a screen buffer for the current console * Assumes the current mode of the screen buffer is 3 or 7. New modes have been added, and even more may be added * Sets a global console setting that persists after Python exits Like the CRT's "conio" API, clear_screen() should open "conout$" (temporarily), which will succeed if Python is attached to a console, regardless of the state of the standard handles, file descriptors, or sys.stdout, and will always open the currently active screen buffer, regardless of how many screen buffers exist in the current console session. The current mode should be queried for ENABLE_VIRTUAL_TERMINAL_PROCESSING (4) via GetConsoleMode(). If it's not enabled, bitwise OR it into the mode and try to enable it via SetConsoleMode(). If VT mode is enabled, write '\x1b[2J\x1b[H' to the file. If VT mode can't be enabled, then fall back on the legacy console API. In particular, some people mentioned not wanting to spawn a cmd.exe process just to use its CLS command. Even if spawning a process is okay, the CLS command clears the scrollback, which is inconsistent with ESC[2J. If clear_screen() is going to add ESC[3J to clear the scrollback, then it's at least consistent, but I'd rather not clear the scrollback. clear_screen() should be able to emulate ESC[2J via GetConsoleScreenBufferInfoEx (get the screen buffer size, window, cursor position, and default character attributes), ScrollConsoleScreenBuffer (if the screen buffer has to be scrolled up to make space), and SetConsoleScreenBufferInfoEx (shift the visible window in the buffer and set the cursor position). This can be implemented in ctypes or C. But normally the standard library avoids using ctypes. Finally, if VT mode was temporarily enabled, revert to the original mode, and always close the "conout$" file. Off topic comment:
kernel32 = ctypes.windll.kernel32
I recommend the following instead: kernel32 = ctypes.WinDLL('kernel32', use_last_error=True) The global library loaders such as ctypes.cdll and ctypes.windll are not reliable for production code in the wild. They cache CDLL library instances, which cache function pointers, which may have argtypes, restype, and errcheck prototypes. Another imported package might set function prototypes that break your code, which is an actual problem that I've seen a few times, particularly with common routines from kernel32, advapi32, and user32. It's not worth taking the chance of a conflict with another package just to save a few keystrokes. The global loaders also don't allow setting use_errno=True or use_last_error=True, so the function pointers they create don't capture the C errno value for ctypes.get_errno() or Windows last error value for ctypes.get_last_error(). Calling kernel32.GetLastError() after the fact may not be reliable in a scripting environment even if it's called directly after the previous FFI call.
snip... Eryk, Does this work for you: import sys, os def clear(): if sys.playform == 'win32': os.system('cls') else: sys.stdout.write('\x1b[2J' '\x1b[H') No need to address the list above because CLS does it. No concern about CLS being a trojan becuse as a builtin to CMD and PowerShell and it takes priority over cls.bat etc. For all other platforms I'm assuming the VT100 level of ANSI escape sequences is supported. Barry
On Thu, Dec 24, 2020 at 1:34 AM Barry Scott <barry@barrys-emacs.org> wrote:
Because it's a shell builtin, you have to run the shell to use it. If it wasn't a shell builtin, you could run it directly. Either way you have to make sure you're running the right executable. I suppose finding the right shell is easier since you can use COMSPEC, but if you don't trust the path then I'm not sure you should trust COMSPEC. As Eryk Sun said, cls erases the scrollback, while your non-win32 code doesn't. The popular cmd.exe replacement TCC/LE seems to have different default behavior for cls, erasing only the visible text by default. You can add /c to match cmd.exe's behavior, but cls/c is an error in cmd.exe. TCC also has cls/s which moves the visible text into the scrollback, which might be the best behavior for Python, but if we want that we'll have to do it in-process since we can't rely on TCC being present. Spawning processes that run for 1ms and do a trivial thing is common in the Unix world but it's not idiomatic in Windows. It feels like a CVE in the making. (Even on Unix, to be honest.)
well, my two cents here is that I do clear my screen - a lot. In the same use case as Guido's mentioned - exploratory programming. But when I do it, I do in a manner that was discussed in the original thread - by using CTRL+L, which is of course terminal-specific but clears everything both in Python's REPL and ipython. As a comparison, I also code almost daily in Elixir, which has a REPL as well, but for some weird reason, CTRL+L does not work there. [image: --] Felipe V. Rodrigues [image: https://]felipevr.com <https://felipevr.com> On Sun, Dec 20, 2020 at 8:48 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Mon, Dec 21, 2020 at 08:22:35AM +1100, Cameron Simpson wrote:
[...]
Do you have benchmarks showing that shelling out to "clear" or "cls" is actually slower than running the full curses machinery for this? IPython claims that the magic `%clear` command shells out to `clear`, are there complaints about the time it takes to clear the screen? On my computer, shelling out to "clear" takes about one hundredth of a millisecond, which in interactive use is pretty much instantaneous. Do you have examples of tight loops where this would be a bottleneck? Other than performance, are there any other reasons why we shouldn't just use `os.system('clear')` (or equivalent on Windows) and delegate to the shell command for this? I guess there could be unusual platforms with no shell clear command, so we might need a fallback. -- Steve
On Wed, Dec 23, 2020 at 9:15 AM Steven D'Aprano <steve@pearwood.info> wrote:
A tight loop clearing the screen? If you have that, you have much bigger problems :) And yes, I've seen that happen, usually because something doesn't know about cursor movement calls. (It looks UGLY.) But shelling out to clear/cls has another problem: it's a big ol' dependency. You need to have the corresponding command, and it needs to be accessible on $PATH, and so on and so forth. I'd rather a more complicated system of ANSI escape sequence management than something that depends on an external command; it's only going to be a half page of code to do it inside Python. ChrisA
On Tue, Dec 22, 2020 at 10:26 PM Chris Angelico <rosuav@gmail.com> wrote:
I worry about the security concerns of having a common interactive command actually utilize an external executable. Yes, I trust the version of `clear` packaged with my Linux distribution, but it feels like sneaking in something malicious onto the path is a possible problem. I don't have a precise threat model in mind, but if you think of things like online hosted interpreters, or remote limited access to shells, or stuff like that, it starts to feel plausible that someone could do something bad. I'm not sure about Windows. Is 'cls' built into the command-line executable itself (like Busybox) or is it an exe? Somewhat supporting my concern, I just was surprised to find this: % ll `which clear` -rwxrwxr-x 2 dmertz 14344 Nov 14 17:07 /home/dmertz/miniconda3/bin/clear % ll /usr/bin/clear -rwxr-xr-x 1 root 14656 Feb 29 2020 /usr/bin/clear Mind you, I generally trust the conda/conda-forge folks. But I absolutely would not have guessed that I run a different `clear` depending on whether I'm in an environment (and different environments each have their own `clear` it seems). This is not a symlink, but an actual different executable with a different size. In fact: % find ~/miniconda3/envs -name 'clear' | xargs wc -c 14344 /home/dmertz/miniconda3/envs/cleaning3.8/bin/clear 14344 /home/dmertz/miniconda3/envs/INE/bin/clear 14344 /home/dmertz/miniconda3/envs/Pearson-ML/bin/clear 14344 /home/dmertz/miniconda3/envs/cleaning3.9/bin/clear 8012 /home/dmertz/miniconda3/envs/py3.4/bin/clear 14288 /home/dmertz/miniconda3/envs/cleaning/bin/clear 14296 /home/dmertz/miniconda3/envs/pypy/bin/clear 14344 /home/dmertz/miniconda3/envs/py3.9/bin/clear 14296 /home/dmertz/miniconda3/envs/py3.8/bin/clear 8012 /home/dmertz/miniconda3/envs/py3.5/bin/clear 14296 /home/dmertz/miniconda3/envs/play/bin/clear 144920 total Apparently I have at least 5 different `clear` executables installed on my system... and I only learned that in the last 2 minutes. -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Tue, Dec 22, 2020 at 2:58 PM David Mertz <mertz@gnosis.cx> wrote:
that is, indeed surprising. But also maybe an argument for using it -- it can be supplied in by conda (and others) , and thus we don't need to count on the system having it. Though, yeah shelling out to an arbitrary command on the system is a bit scary -- does the Python stdlib currently do that anywhere? In fact:
interesting .. any idea what conda package is providing it? -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
As I say, I just discovered the fact about conda vendoring `clear`. I haven't looked further into what includes it as dependency... although it seems to be in pretty much all my environments, so maybe IPython? On Wed, Dec 23, 2020 at 12:46 AM Christopher Barker <pythonchb@gmail.com> wrote:
-- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
On Tue, Dec 22, 2020 at 4:46 PM Christopher Barker <pythonchb@gmail.com> wrote:
Though, yeah shelling out to an arbitrary command on the system is a bit scary -- does the Python stdlib currently do that anywhere?
Here's what I found by grepping the sources for uses of os.system and subprocess: * pydoc shells out to "more" on win32 and "less" or "more" elsewhere if no pager is configured. Incidentally, on Windows it passes the docs to the pager via a temp file, with the comment "pipes completely broken in Windows" - is this left over from the Win9x days? Also the temp-file code runs the pager with os.system instead of Popen, and doesn't properly quote it. * ctypes.util has a ton of hacky code for finding libraries, which includes calling out to gcc, cc, ld, objdump, /sbin/ldconfig, and /usr/ccs/bin/dump on various platforms. * platform.architecture() calls file and ad-hoc parses its output, except if sys.platform in ('dos', 'win32', 'win16'). On those three platforms it seems to be totally broken, always returning the pointer size of the current Python process no matter what executable you pass to it. * webbrowser looks for a bunch of specific named browsers. On Windows it'll run any of ("firefox", "firebird", "seamonkey", "mozilla", "netscape", "opera") from the insecure search path that starts with the current directory.
On 12/22/20, David Mertz <mertz@gnosis.cx> wrote:
CLS is an internal command of the CMD shell. An internal command takes precedence as long as the executed name is unquoted and has no extension. My concern with cls/clear is that they clear the scrollback. Is that what most people want from a clear_screen() function?
On Sat, Dec 19, 2020 at 08:50:57PM -0800, Guido van Rossum wrote:
I don't care where the implementation lives. But I think that it should be added to builtins for interactive use. It seems to me that the primary use for this will be in the interactive interpreter. People are used to something like "cls" or "clear" from various shells. Using it in scripting is, I think, a secondary use-case. Given that, I think that like other conveniences for interactive use like `help`, it should be available without an import. -- Steve
On 2020-12-22 14:01, Steven D'Aprano wrote:
Given that, I think that like other conveniences for interactive use like `help`, it should be available without an import.
Agreed in general, however Ctrl+L is mostly available for this purpose. There are two major exceptions unfortunately. One is when using cmd.exe. However, every other combination of shell and console/terminal on Windows does support it. Not to mention everywhere Unix has an influence. Second is the Python REPL console on Windows. I'm guessing it wouldn't be too hard to add Ctrl+L handling to it. Would be useful I think, regardless of some of the other decisions that need to be made. -Mike
' I disagree. yes, I want to clear the screen a lot during interactive use, but I'd rather do that with a keystroke (cmd+K on the Mac) -- and indeed, do that with iPython all the time, even though iPython has the command built in. So I think the reason to have it in the std lib is for scripting, and I think putting it in os or sys is just fine. -CHB
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sat, Dec 26, 2020 at 05:58:16PM -0800, Christopher Barker wrote:
And nobody is going to take that away from you, unless you choose to go to Windows, where ctrl-L doesn't work. But the ctrl-L trick has no discoverability. It took me close to twenty years of using Linux before I discovered it, and I still don't remember to use it when I need it. Beginners and casual users aren't going to know ctrl-L, or stumble across it through experimentation. People like me aren't going to remember to use it. For interactive use, os.clear is barely any better. (Me, I'll be looking in shutils, not os, because this is a shell feature, not an os feature.)
So I think the reason to have it in the std lib is for scripting, and I think putting it in os or sys is just fine.
Putting it in sys requires it to be built into the interpreter, which means it has to be written in C (for CPython). Putting it in os is fine, that can easily delegate to platform specific code in posix.py, nt.py etc, with a pure-Python lowest common denominator fallback for weird platforms that don't support anything sensible. I just want site.py to import it at startup for the sake of beginners. Is that so bad? It adds one line to site.py, four if you want to make it conditional: try: from os import clear except: pass I can easily add it to my own personal PYTHONSTARTUP file, but the average beginner can't. Let's throw newbies a bone and pre-import it for them, they're the ones who seem to expect it the most. So that's the easy part: just agree with me and import it in site.py, wherever it happens to come from. *wink* The hard part is deciding exactly what it should do, e.g. clear the scrollback or not. I think Chris makes a good point that real beginners will misuse this to hide the evidence whenever they get an error, so I think the default should be to *not* clear the scrollback, for the sake of educators, teachers and trainers. People who know what the scrollback is and definitely want to clear it can just provide an extra argument to the function. So let's just all agree I'm right *wink* and move on to the *really* hard part, which is ensuring it works on Windows and weird terminals. -- Steve
On Sun, Dec 27, 2020 at 1:25 PM Steven D'Aprano <steve@pearwood.info> wrote:
True. Plus, despite my misgivings about how it'd be misused, this IS a useful feature to have within a program, and making a program press Ctrl-L isn't reliable.
os or shutil would be fine with me, and I know for sure that a lot of people will look in the wrong one. My inclination would be for os, since shutil is described as "high level FILE operations", and is mainly focused on file system rather than console. But that's bikeshedding.
This is actually the exact part that I think should *not* happen, for several reasons: 1) Stuff that exists interactively and doesn't exist in a script is a barrier to that migration. I don't think this feature justifies that barrier. 2) The builtins are already incredibly busy, and this, again, doesn't justify being one 3) Beginners already have to learn about importing (eg "from math import sin") and this is just one more importable.
So that's the easy part: just agree with me and import it in site.py, wherever it happens to come from. *wink*
This is python-ideas! We don't "just agree" here, we argue! Even if we don't have to!
Okay, so I think there are a few things we can agree on. * Having a function to clear the screen can be abused, but is still worth having * It shouldn't be based on os.system, which top SO results are advocating. * It should work on all OSes, consoles, etc, if possible * It doesn't have to work in a GUI; Idle can monkeypatch it My personal shed colour preferences: * os module, not sys, not shutil, not builtin * clearscreen(), not cls(), although I'd accept clear() * Suppressing scrollback is a downside, but better than not having the feature at all * Beginners should NOT be encouraged to use this too much The last one is hard to define, but I'm generally disinclined towards any proposal that makes it too easy for a beginner to clear the screen all the time. Life is SO much easier as an educator if you can do post-mortem analysis, not just of the one most recent action, but of all previous ones. Even among more advanced students, I see far too many cases where they try to "tidy up" (by clearing the screen, or closing the terminal and opening a new one, or whatever), and in doing so, throw away valuable error messages or indicative commands (like if someone wrote "os = 'Linux'" and then broke all os.x() calls). ChrisA
On Sun, Dec 27, 2020 at 02:00:00PM +1100, Chris Angelico wrote:
It will still exist in a script, you can still import it from the os module if you care about supporting cases where site.py doesn't get run. This is no different from `exit`: [steve ~]$ python3.9 -c "print(exit)" Use exit() or Ctrl-D (i.e. EOF) to exit [steve ~]$ python3.9 -S -c "print(exit)" Traceback (most recent call last): File "<string>", line 1, in <module> NameError: name 'exit' is not defined Well, actually, it will be slightly different: the pseudo-builtin `exit` is a special object created for interactive use, it's not the same as `sys.exit`. Whereas I invisage that the interactive `clear` will be identical (same object) as `os.clear`.
2) The builtins are already incredibly busy, and this, again, doesn't justify being one
It's not an *actual* builtin, it's just monkey-patched into the builtins by site.py. I don't think there is sufficient justification for this to make it a genuine builtin like `len` or `iter`, but I think loading it into the builtin namespace for the convenience of the user is plenty justified. Having clear available immediately in the interactive interpreter is certainly more useful than having the licence, credits and copyright notice, and yet there they are. Probably the three least used builtins ever.
3) Beginners already have to learn about importing (eg "from math import sin") and this is just one more importable.
As far as I can tell, every language that supports a clear/cls command, from old BASIC to modern shells, makes it available immediately. You don't have to adjust the PATH in bash to get access to clear, it just works. Think of this as the companion to print. If you think about Python as an interactive interpreter, its weird to have print, help, quit available but not clear. Its a fairly fundamental interactive command.
I disagree! *wink* -- Steve
On Sun, Dec 27, 2020 at 4:26 PM Steven D'Aprano <steve@pearwood.info> wrote:
Oh, sorry, my bad. Withdraw point 1 then.
But how's that different? Either it's a builtin or it's not. How it gets there is mostly irrelevant, other than to make a separate monkeypatching layer (such as would need to be done by Idle) more complicated.
Ohh, do you mean how some functions are intrinsic parts of the interpreter? I'm not talking about that; I'm talking about how it's accessed, ie the namespacing.
Perhaps, but they have legal reasons for existing.
Old BASIC didn't *have* an import feature (at least, the BASICs that I used didn't), and modern shells have a ton of shell-related stuff in their "builtins" but nothing else, because that's their focus. You can't, for instance, do trig functions in the basic shell, nor even arithmetic, because those aren't builtins. Different environments focus on different things.
I don't actually agree. Having the ability to print? Yes, that's fundamental. Being able to quit? Of course you need that. But there's no cursor movement function in the builtins, nor is there a "show this image" function, nor "display this web page", or various things like that. In terms of console actions, the two most fundamental are print and input, and we get both of those. Everything else has to justify itself in terms of utility and convenience; getting documentation, for instance, could easily have been buried away in a module, but it is *incredibly* helpful to be able to pull up help(x) without an import. We don't need a full set of cursor movement and screen manipulation features as builtins - that's the job of curses. I don't think we really need screen clearing that way either, particularly as the obvious name "clear" is also used in various other contexts, such as lists and dicts. (How confusing would it be to say clear(lst) instead of lst.clear() and then try to figure out what's going on?)
Is this one of those cases where we "agree to disagree"? *wink* ChrisA
On 2020-12-27 at 13:24:34 +1100, Steven D'Aprano <steve@pearwood.info> wrote:
The first time a sysadmin added readline to one of our work computers, it took me *minutes* to be stunned that my terminal window was being cleared for no reason. It took me longer to discover that the reason was accidental Ctrl-L's (no doubt an off-by-one error trying to press Ctrl-K), and way longer to figure out what piece of [software] was intercepting that keystroke and clearning my screen. The very first line in my .inputrc file is as follows: Control-L: redraw-current-line
Technically, I was a professional, and I stumbled on it through fat fingers and not knowing that readline had been installed and did horrible things like that out of the box. And then I disabled it as quickly as I could figure out how. Call me 0 (because I don't care enough to be +0 or -0) on the whole thing, as long as it's spelled with more than one keystroke (to prevent accidental invocation).
On 2020-12-26 19:45, 2QdxY4RzWzUUiLuE@potatochowder.com wrote:
No offense, but if you're a developer who can't handle hotkeys or came of age before the "glass tty" and won't learn any new tricks… well, that's just *"your problem, mayan."* The rest of us are capable. :D Presumably you'd have to find the appropriate help page to discover this feature, whether the os module, interactive interpreter page, or stackoverflow. Discoverability is not a huge problem when just about every interpreter implements the hotkey, save python on Windows and legacy cmd.exe, it's once and done. (At least cmd.exe can be replaced with the yori shell, which I often install when forced to use Windows.) I entered a bug for it, and found a related one for IDLE: https://bugs.python.org/issue42771 (Windows interactive console) https://bugs.python.org/issue6143 (Idle) Would be great if these could get looked at. Implementation doesn't mean other enhancements couldn't be made as well. -Mike p.s. Ctrl+L meant form-feed in ASCII so there is prior history.
Mike Miller writes:
The thing is, this is a destructive capability. In some cases it only clears the "screen" (whatever that means), which may cost you some diagnostics. Probably (but not always) you can recreate them by repeating a command from history. But in some cases it also clears the scrollback buffer. That makes it kinda hard to figure out what's going on unless you know what's going on. I don't need it ever and I don't hit Ctrl-L accidentally, so I'm =0 on adding the feature. But it's something that needs some care in implementation. Steve
On 2020-12-28 23:33, Stephen J. Turnbull wrote:
Well, the time to push back on the feature was in the mid 1970s. Today, if you're at a command prompt or repl, there's a very high chance the feature already exists, with few exceptions. Consistency in UI is still a goal though it has taken a beating the last few years. Also, I'd argue the likelihood of a newbie clearing important work, or an expert making a significant mistake in this manner is not very high. The readline (or equivalent) configuration can mitigate the latter if need be. -Mike
On 2020-12-29 15:07, Chris Angelico wrote:
Note, I added the word *important* on purpose. ;-) Reminds me of a story. As an teen I learned the lesson of what happens when you flip the power switch of the computer in the middle of writing an as-yet-unsaved document. It was an important lesson, but the particular document wasn't. Nor were any for a number of years. But, ever since I make sure work is saved frequently, and avoid power buttons. :D I don't try to prevent their existence, due to their utility. -Mike
Mike Miller writes:
Note, I added the word *important* on purpose. ;-)
Ie, since you were lucky enough the time you did something like it, it's not important to anybody?
Reminds me of a fact: modern personal computers don't really have an off switch. They have menu items and keys that take several hundred milliseconds to react. Seems to me that that is most likely because a lot of folks weren't as lucky with at least one of the times they shutdown in the middle of a save. And then what was that fysnc-on-close thread about? Again, I'm not against the Ctrl-L feature or some clear function (I'm -0.5 on a builtin, but if the module maintainers for sys or os are willing, I'm perfectly happy to ignore it), but some care should be taken that it (1) not clear scrollback or history unless explicitly requested, and (2) not be easy to inadvertantly invoke. I don't think (2) is at all difficult, but it should be checked before distribution. Steve
participants (13)
-
2QdxY4RzWzUUiLuE＠potatochowder.com -
Barry Scott -
Ben Rudiak-Gould -
Cameron Simpson -
Chris Angelico -
Christopher Barker -
David Mertz -
Eryk Sun -
Felipe Rodrigues -
Guido van Rossum -
Mike Miller -
Stephen J. Turnbull -
Steven D'Aprano