Option of running shell/console commands inside the REPL
Hello, Consider adding the option of running shell/console commands inside the REPL. Something like
!dir
Best regards, João Matos
Further magic comes with https://pypi.org/project/xonsh/ ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
I always use ptipython (ptpython shell over the ipython console) for my REPLs. The built-in python repl is not *batteries included* in the sense that it already has what you need to explore the language. I wonder, what do the python committers think about including a stripped-down version of ipython and ptpython in the built-in python? On Fri, Feb 1, 2019 at 6:15 AM Lele Gaifax <lele@metapensiero.it> wrote:
On 2/1/2019 9:07 AM, James Lu wrote:
I'm opposed to it. I think the current situation is fine, where CPython ships with a basic REPL, and other REPLs exist. I don't want to see any complexity added to CPython. I'm not opposed to hooks to make it easier for third parties to add functionality to the CPython REPL, but I'd have to see how they complicate things before making up my mind. Eric
On 2/1/2019 1:50 PM, James Lu wrote:
It’s difficult to learn anything with a body (such as a loop or a class or a function) with the built in REPL because you can’t edit lines you’ve already written.
I presume you mean that it is 'difficult to learn about compound statements and multiline simple statements with standard interactive python because one cannot edit lines already entered. True. This is one of the great advantages of IDLE and any other IDE in which one keys in and can edit a complete multiline statement before submitting it to Python to run It is also difficult to edit and rerun previously run multiline statements in the standard REPL when one must retrieve, edit, and enter previous lines one at a time. Again, IDLE and anything similar makes this much easier by retrieving and allowing the editing and resubmission of complete multiline statements. However, I don't see what this has to do with running system console commands in a Python shell. -- Terry Jan Reedy
On Fri, Feb 01, 2019 at 03:12:42PM -0500, Terry Reedy <tjreedy@udel.edu> wrote:
Python REPL is missing the following batteries: * Persistent history; * Syntax highlighting; * Clear separation (using, for example, different colors) between input, output and errors; * Paging of very long output/errors.
-- Terry Jan Reedy
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sat, Feb 2, 2019 at 7:32 AM Oleg Broytman <phd@phdru.name> wrote:
Dunno about that - it persists just fine on my system. Maybe it's a Windows problem? On all my Windows systems, I just use IDLE instead of the default REPL and have done with it.
* Syntax highlighting;
Not usually something I need from a REPL. That's a feature for an editor.
* Clear separation (using, for example, different colors) between input, output and errors;
Input is marked with a prompt (">>>" or "..."). Errors are just another form of output.
* Paging of very long output/errors.
This could definitely be improved on. I don't know that this counts as "missing batteries" though, as the standard library DOES include a hook that you could use for the purpose (sys.displayhook); it's just a matter of figuring out a suitable renderer. ChrisA
On Sat, Feb 02, 2019 at 07:45:43AM +1100, Chris Angelico <rosuav@gmail.com> wrote:
Yes, that the thing - it's *another* form of output.
I use it, of course: https://phdru.name/Software/dotfiles/python/init.py.html The question is - does builtin REPL need more batteries?
ChrisA
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On 2/1/2019 3:31 PM, Oleg Broytman wrote:
Python REPL is missing the following batteries:
That is why, on Windows, I nearly always use IDLE.
* Persistent history;
IDLE's Shell history persists across restarts (which are not available is the standard shell). I cannot remember wanting history saved between IDLE sessions. I also cannot remember seeing a request for this feature. If I want a sequence of commands saved, I copy them into an editor window and save in a named file.
* Syntax highlighting;
IDLE does this.
* Clear separation (using, for example, different colors) between input, output and errors;
IDLE does this.
* Paging of very long output/errors.
Help output *is* paged in the standard shell. For comparison with IDLE, for issue 35855, I tried out help(tkinter) in standard Python on both Windows and Mac. For the most part, I found the pagers to be somewhat obnoxious and deficient*. Except for one thing that I want to fix, paging up an down a window with the standard PageUp and PageDown keys, as in IDLE, is easier and better. And it is not limited * Windows Console holds a maximum of 9999 characters, with the default being 300, at least for Command Prompt. The less pager, at least on Mac, when invoked by help when running Python in Terminal, displays the paged output in a separate window. When less is exited, the window closes and the paged output *disappears*. You left out what I think is a very important 'battery': * Works with Python statements instead of physical lines of text. As I discussed elsewhere, this includes editing entire multiline statements before submitting and storing and retrieving complete statements to and from the history list. Here is another: * Clicking on a traceback item can open the file in an editor at the indicated line. -- Terry Jan Reedy
On 2/1/19, Terry Reedy <tjreedy@udel.edu> wrote:
Python's built-in REPL relies on the readline module for history. In Windows you'll need to install pyreadline, an implementation that uses the Windows console API via ctypes. Out of the box, Python uses the the built-in line editing and history that's provided by the Windows console host (conhost.exe). There's an undocumented function to read this history (as used by doskey.exe), but there's no function to load lines into it. I suppose it could be replayed manually in a loop that calls WriteConsoleInputW and ReadConsoleW.
* Windows Console holds a maximum of 9999 characters,
9999 lines, not characters.
On Fri, Feb 01, 2019 at 08:43:53PM -0600, eryk sun <eryksun@gmail.com> wrote:
Yep, I use it. The question is: Does Python REPL need this battery preinstalled and preconfigured; or any other battery as well?
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Fri, Feb 01, 2019 at 08:40:22PM -0500, Terry Reedy <tjreedy@udel.edu> wrote:
For the question "Does Python REPL need more batteries?" is your answer "No, just point people to IDLE"? If it is - well, I disagree. I implemented a lot of enhancements for REPL myself, and I don't like and avoid GUI programs.
I found pager ``less`` quite good. I configured it in a complex way, learned to use it and use in many different programs, not only Python.
Yep, would be nice to have. Bash implements it somehow.
That would be too much. Requires a traceback-sensitive pager. Also not very useful for situations where developer works on one host and runs programs on another (containers/virtual machines/remote hosts are widely used these days).
-- Terry Jan Reedy
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sat, Feb 2, 2019, 8:15 AM Oleg Broytman <phd@phdru.name wrote:
IPython and BPython and Xonsh are projects that exist. I doubt the Python REPL will ever be as full featured as those, and I don't really think it should hope to. Similarly for IDLE or PyCharm or Spyder or Jupyter for folks who like GUIs. Apart from the maintenance work required for CPython (and for other Python implementations), the feature cadence is likely to be faster on those projects than is the Python core. That said, I definitely appreciate some small improvements in the REPL sooner the years, probably ones you have added, Oleg. So it's a balancing act, I recognize.
On 2/2/2019 8:13 AM, Oleg Broytman wrote:
I believe this was Chris, not me.
To do this correctly requires a separate programmable repl that sees each character as it is entered. To me, this is as useful in the repl as in an editor.
If one want these batteries *today*, that is one sane answer, especially on Windows and, it seems so far, Mac.
If it is - well, I disagree. I implemented a lot of enhancements for REPL myself, and I don't like and avoid GUI programs.
Whereas, in spite of (or perhaps because of) possibly being older than you, I am the opposite. Many beginners, even after multiple college CS classes, have never used a TUI (text user interface) program like Command Prompt before encountering Python and wanting to do something that requires the TUI, such as using pip. Anti-GUI attitudes make Python harder to use for many people. Pip, for instance, desperately needs a GUI front end. I believe that PIP problems are the most common Python question on StackOverflow. However, the pip people will not write one and I was prohibited from adding one to the stdlib.
I don't know what less configuration one can do on Mac, but its default of erasing help output before one writes the statement one wanted help for is quite bad to me.
Mac terminal does not.
Now that I am used to this, I consider it essential.
Requires a traceback-sensitive pager.
It require something that can match the regex "File ([~,]*), line ([0-9]*)," to a selected line or the one above it and pass the captured file name and line number (if successful) to something that can open the file at the line).
I and many other people edit and run on the same machine. However, editing and running on different machines is to me an argument for keeping the UI separate from the execution binary. What matters is where the user interface presenting the traceback is, not where the program was run. IDLE runs user code in a separate process, with a socket and rpc connection to the GUI. It could potentially run the separate process in a separate machine. IDLE compiles user code in the GUI before sending the code object to the execution process. The 'instant' syntax check would be even more relatively useful if execution were on a remote machine, with the attendant delay. Another IDLE feature I miss in the default REPL is smart auto-indentation. This includes following some PEP 8 recommendations for non-significant indentation of continuation lines. Thinking about it, the IDLE code either is or could be independent of the tkinter text widget. With a little work, it could be moved into the code module for use by InteractiveInterpreter, which IDLE uses, and by anyone else, including, maybe, python itself. Since it enforces a particular style, it would have to be optional at least for InteractiveInterpreter and python. -- Terry Jan Reedy
On Sat, Feb 02, 2019 at 07:37:56PM -0500, Terry Reedy <tjreedy@udel.edu> wrote:
Oh, "my beard is longer than your" game! :-))) I'm 51 y.o. But I always was like that. When all sane people were switching from DOS to Windows 3.0 I switched to Unix. First, BSD/OS, later SunOS, later FreeBSD, after that Linux, Linux, Linux. At DOS times I was writing huge .bat-files using all possible tools I can collect in pre-Internet era. Small utilities, alternative command line interpreters (4DOS), .bat compilers (turbobat). I tolerated TUI but hated GUI even then. Switching to shell scripting was pretty natural for me with awk/find/grep/sed/etc replacing all those tools.
Anti-GUI attitudes make Python harder to use for many people. Pip, for instance, desperately needs a GUI front end.
For me it's hard to believe ``pip`` needs any UI. I run ``pip`` in command line, in scripts and at remote servers, often completely unattended (Travis CI and AppVeyor, e.g.) I don't see how I'd use an UI.
I believe that PIP problems are the most common Python question on StackOverflow.
Problems - yes. But not because it lacks an UI. I read SO every day for a few years now and answer questions every few days. I don't remember people ever asked about any UI for pip. pip problems, as far as I can recollect the problems, are: * SSL; recently pypi.org and github.com switched to TLS1.2-only and the change broke a lot of sites where people still run CentOS 5 and other old Python versions without any possibility of upgrading. * A few pythons at the host: run ``pip install module``, ``import module``, got ``ImportError/NoModuleFoundError`` because ``pip`` and ``import`` are ran with 2 different pythons. * Incompatible upgrades: ``pip`` was upgraded but the user didn't get that it was ``/usr/bin/pip`` that was upgraded and ``/usr/local/bin/pip`` is now outdated but is found first in ``$PATH``. Hence the error ``AttributeError: main``. * The absence of a compiler on Windows (or a wrong version of it) to install a C extension without a binary wheel. * Offline installation. Downloading binary wheels for a different platform (different from the host where ``pip`` is running).
-- Terry Jan Reedy
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sun, Feb 03, 2019 at 02:52:31AM +0100, Oleg Broytman <phd@phdru.name> wrote:
Sorry, was not completely clear here: * Downloading the tree of dependencies (binary wheels or source distributions if there are no binary wheels) for a different platform.
-- Terry Jan Reedy
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sat, Feb 02, 2019 at 07:37:56PM -0500, Terry Reedy wrote: [...]
That's an ambiguous sentence. It could mean that syntax highlighting in both are as useful as a screen door on a submarine :-) I don't know how you use the REPL, but for me, highlighting in the REPL is much less useful because I'm not looking at large chunks of code at a time. If it is more than, oh, two or three lines, it is too big to be editing in the clunky editing environment of a REPL and I'll use an editor to read/edit the code.
Did you put it on PyPI? That's the usual suggestion for something that might be useful but isn't believed to be needed for the stdlib. As Nick said, Rather than "rejected" per se, I've closed this as "postponed" https://bugs.python.org/issue27051#msg286634 https://summerofcode.withgoogle.com/archive/2016/projects/5063140450500608/ [...]
I'd like to see the help pager code made into a public API, and help itself made configurable by the user to choose a pager (auto-detecting a pager by default, as it does now). That would allow people to choose whichever pager works best for them -- possibly including a seperate GUI window which they could leave open all the time.
If you are running a sufficiently recent bash and readline is supported, it should support any feature the POSIX bash supports, regardless of the OS or the terminal. (Although I think readline on Mac is slightly different from other Unix systems? And maybe less functional?) I'm not sure what multiline editing is being referred to above. I personally don't know of one in bash. It may be that you have to turn the feature on -- by default, bash ships with a fair amount of stuff turned off, and the rest hidden behind cryptic keystroke commands. One readline feature I know of which I would love to see supported in the REPL is the "execute and next line" command. On my bash system, it is Ctrl-O. What it does: - use the up-arrow to move through the history to the line you want to execute; - type Ctrl-O to execute that line; - the *next* line is automatically loaded, ready to be executed. E.g. suppose my history consists of five lines: 1: print('hello') 2: def spam(): 3: x = 1 4: return x + 1 5: foo = spam() and I want to re-load the spam function (perhaps after editing it) in the REPL. From a bare prompt, I can: * hit UP-ARROW four times to get to line 2 of the history, then ENTER; * this pushes a new line into the history; * hit UP-ARROW four times again to get to line 3, then ENTER; * hit UP-ARROW four more times to get to line 4, then ENTER. With execute-and-next-line, I could: * hit UP-ARROW four times to get to line 2 of the history, then Ctrl-O; * this loads line 3, so I can immediately hit Ctrl-O; * this loads line 4, so I can immediately hit ENTER. Much easier to re-execute or edit a multiline block! As far as I can tell, it is impossible to add this feature to the REPL from pure Python (corrections welcome!) but I imagine it would be something IDLE could do. -- Steve
On Sun, Feb 03, 2019 at 10:24:12PM +1100, Steven D'Aprano <steve@pearwood.info> wrote:
shopt -s cmdhist From ``man bash``: "If set, bash attempts to save all lines of a multiple-line command in the same history entry. This allows easy re-editing of multi-line commands."
-- Steve
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
Steven D'Aprano <steve@pearwood.info> writes:
I implemented and offered the feature, but unfortunately it is just a couple of lines copied from bash, and that being GPL was considered unfair to be relicensed under current Python license. Maybe sooner or later I will try to repackage it as a standalone extension. ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
Lele Gaifax <lele@metapensiero.it> writes:
Steven D'Aprano <steve@pearwood.info> writes:
One readline feature
To be exact, it's not a feature of the readline library: see https://bugs.python.org/issue22228 (sigh, my GH fork of cpython has been recreated so some refs are broken). ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
On Fri, Feb 01, 2019 at 09:31:21PM +0100, Oleg Broytman wrote:
Python REPL is missing the following batteries:
* Persistent history;
On Linux/Unix systems, that has been available for about 18+ years, since the rlcompleter module was introduced in Python 2.0: https://docs.python.org/release/2.0/lib/module-rlcompleter.html It's been automatically enabled for more than six years: https://hg.python.org/cpython/rev/d5ef330bac50 Windows is another story.
* Syntax highlighting;
bpython provides syntax highlighting in the REPL, but I've never seen the point. In an editor, perhaps, but why bother in the REPL? https://bpython-interpreter.org/ So does DreamPie: http://www.dreampie.org/
* Clear separation (using, for example, different colors) between input, output and errors;
Input always starts with a prompt; exception tracebacks always start with the line "Traceback..."; other errors are generally not programmatically distinguishable from non-errors. One could modify the displayhook and excepthook to deal with the usual cases, but colour codes are not just platform-specific but console- specific. https://docs.python.org/3.5/library/sys.html#sys.displayhook https://docs.python.org/3.5/library/sys.html#sys.excepthook
* Paging of very long output/errors.
I don't know how I feel about that. Now that Python shrinks long tracebacks filled with identical lines, I'm not sure that this is so important or desirable.
-- Steve
Sorry, I was not as clear as I ought to have been: On Sat, Feb 02, 2019 at 02:10:06PM +1100, Steven D'Aprano wrote:
By which I mean, if a script or module reports an error by just printing a message to the screen: print("sorry, an error occurred") sort of thing. Obviously the interpreter cannot guess which messages represent errors in that sense and which are non-error output. (Of course, a script could format its own error messages, say by using terminal-specific colour codes.) -- Steve
On Sat, Feb 02, 2019 at 03:37:36PM +1100, Steven D'Aprano <steve@pearwood.info> wrote:
Scripts that report errors this way are broken and must be fixed: print("sorry, an error occurred", file=sys.stderr) Now it's easy to separate normal output from errors and colorize them differently.
-- Steve
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sat, Feb 02, 2019 at 02:10:06PM +1100, Steven D'Aprano <steve@pearwood.info> wrote:
I've configured my own .pythonrc long before that so I missed that it's now finally enabled. Thanks for pointing this out.
And I have a different color just for prompt.
-- Steve
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Fri, Feb 01, 2019 at 08:26:03AM +0000, João Matos wrote:
That's a pretty big change, arguably best left for a third-party REPL like iPython (which already supports "magic" commands using ! and ? syntax). It is hard to tell precisely where to draw the line between adding useful functionality to the standard Python interpreter, and where to leave it for "advanced" REPLs or IDEs like iPython. My personal opinion is that specifically supporting shell commands crosses that line, but adding a REPL hook for ! commands which the user can customise does not. Hypothetically speaking, the Python interactive interpreter, and *only* the interactive interpreter, might define hooks for lines starting with a leading ! or ? and users could define their own magic functions for those. Or one could use the code module to define your own custom REPL, and automatically run that on startup: https://docs.python.org/3/library/code.html Actually, adding a code.InteractiveConsole subclass with hooks for lines beginning with ! and ? might be a nice addition to the stdlib without adding complexity to the builtin interpreter. -- Steve
On 2/1/19 2:26 AM, João Matos wrote:
I first ran into this in the days of teletypes and dumb terminals, where other programs let you run shell commands from inside them. Now the shoe appears to be on the other foot. So why not turn that around? ksh (since way back when) and bash (since 2008, according to what I read somewhere online) have "co-processes," which allow you to run a command "in the background," and send commands and receive replies from it. So I tried it with Python, but it didn't work: $ coproc P3 { python3; } $ echo 'import sys; print(sys.version)' >&${P3[1]} $ read v <&${P3[0]} [the read command just waits forever] A pile of experiments and examples from web pages later, I think it's Python and not me. My example, with suitable changes to the literal in the echo command, works with sbcl and erl, but not python3. If I start python3 as follows: $ coproc P3 { python3 | tee /tmp/P3; } then I can see the empty /tmp/P3 file, and the python3 and tee processes, but /tmp/P3 remains empty. Any ideas as to why not? Thanks, Dan
On Sat, Feb 2, 2019 at 7:39 AM Dan Sommers <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:
(This might be better on python-list rather than here)
Looks like you may have a problem with output buffering. Try running "python3 -u" to tell it to run stdout/stderr unbuffered - otherwise, Python assumes that its output is going to a file and it won't matter. Alternatively, explicitly flush stdout after output, which could be done with a display hook: _old_hook = sys.displayhook def display(obj): _old_hook(obj) sys.stdout.flush() sys.displayhook = display But unless you're expecting a lot of output, it's probably easier and just as effective to simply use "-u". ChrisA
On Fri, Feb 01, 2019 at 02:38:43PM -0600, Dan Sommers wrote:
This is another good example of the problem James was referring to in the thread about clearer communication. Don't assume we all know what coproc does.
What are sbcl and erl? I'm guessing you don't mean antimony pentachloride and a municipality in Austria. Possibly Steel Bank Common Lisp and Erlang? But I'm not confident about that. Does your example work with more well-known interpreted languages with interactive interpreters such as Ruby, Lua, Javascript (outside of the browser), etc? -- Steve
On 2/1/19 3:48 PM, Steven D'Aprano wrote:
As I indicated in what you quoted, shell co-processes allow you to run a command in the background and interact with that command from your shell.
Yes, the Steel Bank Common Lisp and Erlang REPLs, respectively.
I don't know (I don't write software in any of those languages, and I don't have them imstalled on my computer), but adding the "-i" flag to my python3 command makes it work (thanks to ChrisA for suggesting "-u"; it was a short leap from there to "-i.") Dan
On Fri, Feb 01, 2019 at 04:28:25PM -0600, Dan Sommers wrote:
Okay, but what does that mean in practice? What does it require to make it work with Python? What is your expected input and output? This is a Python forum. You can assume your readers have a good level of knowledge about Python. Outside of that, you can expect to lose a significant portion of your audience if you start talking about features, practices etc that aren't supported by the Python language, stdlib and perhaps a few of the better-known third-party libraries. E.g. me, I have no idea what you mean by "interact with that command from your shell". If you're a sys admin, you might do that fifty times a day for all I know, but I've never knowingly done it. And are we supposed to know what ">&${P3[1]}" does? It looks like your cat walked over your keyboard. -- Steve
On 2/1/19, Steven D'Aprano <steve@pearwood.info> wrote:
bash coproc runs a process in the background with stdin and stdout redirected to pipes. The file descriptors for our end of the pipes are available in an array with the given name (e.g. P3). The default array name is COPROC. As soon as "pipe" is mentioned, anyone familiar with the REPL's behavior with pipes should know that making this work will require the -i command-line option to force interactive mode. Otherwise stdout will be fully buffered. For example: $ coproc P3 { python3 -qi 2>&1; } [1] 16923 $ echo 'import sys; print(sys.version)' >&${P3[1]} $ read -t 1 <&${P3[0]} && echo $REPLY >>> 3.6.7 (default, Oct 22 2018, 11:32:17) $ read -t 1 <&${P3[0]} && echo $REPLY [GCC 8.2.0] $ read -t 1 <&${P3[0]} && echo $REPLY $ echo 'sys.exit(42)' >&${P3[1]} $ [1]+ Exit 42 coproc P3 { python3 -qi 2>&1; }
And are we supposed to know what ">&${P3[1]}" does? It looks like your cat walked over your keyboard.
It redirects the command's standard output (>) to the file descriptor (&) in index 1 of the P3 array (${P3[1]}), which is our end of the pipe that's connected to stdin of the co-process.
On Fri, Feb 01, 2019 at 07:21:47PM -0600, eryk sun wrote:
Thanks for the explanation.
I wonder... could Python automatically detect when it is connected to pipes and switch buffering off?
And this is why I don't program in bash :-) -- Steven
On 2/1/19, Steven D'Aprano <steve@pearwood.info> wrote:
In most cases we want full buffering when standard I/O is a pipe or disk file. It's more efficient to read/write large chunks from/to the OS. In another message I saw -u mentioned to disable buffering. But that's not sufficient. We need -i to force running the built-in REPL over a pipe, and optionally -q to quiet the initial banner message.
Further magic comes with https://pypi.org/project/xonsh/ ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
I always use ptipython (ptpython shell over the ipython console) for my REPLs. The built-in python repl is not *batteries included* in the sense that it already has what you need to explore the language. I wonder, what do the python committers think about including a stripped-down version of ipython and ptpython in the built-in python? On Fri, Feb 1, 2019 at 6:15 AM Lele Gaifax <lele@metapensiero.it> wrote:
On 2/1/2019 9:07 AM, James Lu wrote:
I'm opposed to it. I think the current situation is fine, where CPython ships with a basic REPL, and other REPLs exist. I don't want to see any complexity added to CPython. I'm not opposed to hooks to make it easier for third parties to add functionality to the CPython REPL, but I'd have to see how they complicate things before making up my mind. Eric
On 2/1/2019 1:50 PM, James Lu wrote:
It’s difficult to learn anything with a body (such as a loop or a class or a function) with the built in REPL because you can’t edit lines you’ve already written.
I presume you mean that it is 'difficult to learn about compound statements and multiline simple statements with standard interactive python because one cannot edit lines already entered. True. This is one of the great advantages of IDLE and any other IDE in which one keys in and can edit a complete multiline statement before submitting it to Python to run It is also difficult to edit and rerun previously run multiline statements in the standard REPL when one must retrieve, edit, and enter previous lines one at a time. Again, IDLE and anything similar makes this much easier by retrieving and allowing the editing and resubmission of complete multiline statements. However, I don't see what this has to do with running system console commands in a Python shell. -- Terry Jan Reedy
On Fri, Feb 01, 2019 at 03:12:42PM -0500, Terry Reedy <tjreedy@udel.edu> wrote:
Python REPL is missing the following batteries: * Persistent history; * Syntax highlighting; * Clear separation (using, for example, different colors) between input, output and errors; * Paging of very long output/errors.
-- Terry Jan Reedy
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sat, Feb 2, 2019 at 7:32 AM Oleg Broytman <phd@phdru.name> wrote:
Dunno about that - it persists just fine on my system. Maybe it's a Windows problem? On all my Windows systems, I just use IDLE instead of the default REPL and have done with it.
* Syntax highlighting;
Not usually something I need from a REPL. That's a feature for an editor.
* Clear separation (using, for example, different colors) between input, output and errors;
Input is marked with a prompt (">>>" or "..."). Errors are just another form of output.
* Paging of very long output/errors.
This could definitely be improved on. I don't know that this counts as "missing batteries" though, as the standard library DOES include a hook that you could use for the purpose (sys.displayhook); it's just a matter of figuring out a suitable renderer. ChrisA
On Sat, Feb 02, 2019 at 07:45:43AM +1100, Chris Angelico <rosuav@gmail.com> wrote:
Yes, that the thing - it's *another* form of output.
I use it, of course: https://phdru.name/Software/dotfiles/python/init.py.html The question is - does builtin REPL need more batteries?
ChrisA
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On 2/1/2019 3:31 PM, Oleg Broytman wrote:
Python REPL is missing the following batteries:
That is why, on Windows, I nearly always use IDLE.
* Persistent history;
IDLE's Shell history persists across restarts (which are not available is the standard shell). I cannot remember wanting history saved between IDLE sessions. I also cannot remember seeing a request for this feature. If I want a sequence of commands saved, I copy them into an editor window and save in a named file.
* Syntax highlighting;
IDLE does this.
* Clear separation (using, for example, different colors) between input, output and errors;
IDLE does this.
* Paging of very long output/errors.
Help output *is* paged in the standard shell. For comparison with IDLE, for issue 35855, I tried out help(tkinter) in standard Python on both Windows and Mac. For the most part, I found the pagers to be somewhat obnoxious and deficient*. Except for one thing that I want to fix, paging up an down a window with the standard PageUp and PageDown keys, as in IDLE, is easier and better. And it is not limited * Windows Console holds a maximum of 9999 characters, with the default being 300, at least for Command Prompt. The less pager, at least on Mac, when invoked by help when running Python in Terminal, displays the paged output in a separate window. When less is exited, the window closes and the paged output *disappears*. You left out what I think is a very important 'battery': * Works with Python statements instead of physical lines of text. As I discussed elsewhere, this includes editing entire multiline statements before submitting and storing and retrieving complete statements to and from the history list. Here is another: * Clicking on a traceback item can open the file in an editor at the indicated line. -- Terry Jan Reedy
On 2/1/19, Terry Reedy <tjreedy@udel.edu> wrote:
Python's built-in REPL relies on the readline module for history. In Windows you'll need to install pyreadline, an implementation that uses the Windows console API via ctypes. Out of the box, Python uses the the built-in line editing and history that's provided by the Windows console host (conhost.exe). There's an undocumented function to read this history (as used by doskey.exe), but there's no function to load lines into it. I suppose it could be replayed manually in a loop that calls WriteConsoleInputW and ReadConsoleW.
* Windows Console holds a maximum of 9999 characters,
9999 lines, not characters.
On Fri, Feb 01, 2019 at 08:43:53PM -0600, eryk sun <eryksun@gmail.com> wrote:
Yep, I use it. The question is: Does Python REPL need this battery preinstalled and preconfigured; or any other battery as well?
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Fri, Feb 01, 2019 at 08:40:22PM -0500, Terry Reedy <tjreedy@udel.edu> wrote:
For the question "Does Python REPL need more batteries?" is your answer "No, just point people to IDLE"? If it is - well, I disagree. I implemented a lot of enhancements for REPL myself, and I don't like and avoid GUI programs.
I found pager ``less`` quite good. I configured it in a complex way, learned to use it and use in many different programs, not only Python.
Yep, would be nice to have. Bash implements it somehow.
That would be too much. Requires a traceback-sensitive pager. Also not very useful for situations where developer works on one host and runs programs on another (containers/virtual machines/remote hosts are widely used these days).
-- Terry Jan Reedy
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sat, Feb 2, 2019, 8:15 AM Oleg Broytman <phd@phdru.name wrote:
IPython and BPython and Xonsh are projects that exist. I doubt the Python REPL will ever be as full featured as those, and I don't really think it should hope to. Similarly for IDLE or PyCharm or Spyder or Jupyter for folks who like GUIs. Apart from the maintenance work required for CPython (and for other Python implementations), the feature cadence is likely to be faster on those projects than is the Python core. That said, I definitely appreciate some small improvements in the REPL sooner the years, probably ones you have added, Oleg. So it's a balancing act, I recognize.
On 2/2/2019 8:13 AM, Oleg Broytman wrote:
I believe this was Chris, not me.
To do this correctly requires a separate programmable repl that sees each character as it is entered. To me, this is as useful in the repl as in an editor.
If one want these batteries *today*, that is one sane answer, especially on Windows and, it seems so far, Mac.
If it is - well, I disagree. I implemented a lot of enhancements for REPL myself, and I don't like and avoid GUI programs.
Whereas, in spite of (or perhaps because of) possibly being older than you, I am the opposite. Many beginners, even after multiple college CS classes, have never used a TUI (text user interface) program like Command Prompt before encountering Python and wanting to do something that requires the TUI, such as using pip. Anti-GUI attitudes make Python harder to use for many people. Pip, for instance, desperately needs a GUI front end. I believe that PIP problems are the most common Python question on StackOverflow. However, the pip people will not write one and I was prohibited from adding one to the stdlib.
I don't know what less configuration one can do on Mac, but its default of erasing help output before one writes the statement one wanted help for is quite bad to me.
Mac terminal does not.
Now that I am used to this, I consider it essential.
Requires a traceback-sensitive pager.
It require something that can match the regex "File ([~,]*), line ([0-9]*)," to a selected line or the one above it and pass the captured file name and line number (if successful) to something that can open the file at the line).
I and many other people edit and run on the same machine. However, editing and running on different machines is to me an argument for keeping the UI separate from the execution binary. What matters is where the user interface presenting the traceback is, not where the program was run. IDLE runs user code in a separate process, with a socket and rpc connection to the GUI. It could potentially run the separate process in a separate machine. IDLE compiles user code in the GUI before sending the code object to the execution process. The 'instant' syntax check would be even more relatively useful if execution were on a remote machine, with the attendant delay. Another IDLE feature I miss in the default REPL is smart auto-indentation. This includes following some PEP 8 recommendations for non-significant indentation of continuation lines. Thinking about it, the IDLE code either is or could be independent of the tkinter text widget. With a little work, it could be moved into the code module for use by InteractiveInterpreter, which IDLE uses, and by anyone else, including, maybe, python itself. Since it enforces a particular style, it would have to be optional at least for InteractiveInterpreter and python. -- Terry Jan Reedy
On Sat, Feb 02, 2019 at 07:37:56PM -0500, Terry Reedy <tjreedy@udel.edu> wrote:
Oh, "my beard is longer than your" game! :-))) I'm 51 y.o. But I always was like that. When all sane people were switching from DOS to Windows 3.0 I switched to Unix. First, BSD/OS, later SunOS, later FreeBSD, after that Linux, Linux, Linux. At DOS times I was writing huge .bat-files using all possible tools I can collect in pre-Internet era. Small utilities, alternative command line interpreters (4DOS), .bat compilers (turbobat). I tolerated TUI but hated GUI even then. Switching to shell scripting was pretty natural for me with awk/find/grep/sed/etc replacing all those tools.
Anti-GUI attitudes make Python harder to use for many people. Pip, for instance, desperately needs a GUI front end.
For me it's hard to believe ``pip`` needs any UI. I run ``pip`` in command line, in scripts and at remote servers, often completely unattended (Travis CI and AppVeyor, e.g.) I don't see how I'd use an UI.
I believe that PIP problems are the most common Python question on StackOverflow.
Problems - yes. But not because it lacks an UI. I read SO every day for a few years now and answer questions every few days. I don't remember people ever asked about any UI for pip. pip problems, as far as I can recollect the problems, are: * SSL; recently pypi.org and github.com switched to TLS1.2-only and the change broke a lot of sites where people still run CentOS 5 and other old Python versions without any possibility of upgrading. * A few pythons at the host: run ``pip install module``, ``import module``, got ``ImportError/NoModuleFoundError`` because ``pip`` and ``import`` are ran with 2 different pythons. * Incompatible upgrades: ``pip`` was upgraded but the user didn't get that it was ``/usr/bin/pip`` that was upgraded and ``/usr/local/bin/pip`` is now outdated but is found first in ``$PATH``. Hence the error ``AttributeError: main``. * The absence of a compiler on Windows (or a wrong version of it) to install a C extension without a binary wheel. * Offline installation. Downloading binary wheels for a different platform (different from the host where ``pip`` is running).
-- Terry Jan Reedy
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sun, Feb 03, 2019 at 02:52:31AM +0100, Oleg Broytman <phd@phdru.name> wrote:
Sorry, was not completely clear here: * Downloading the tree of dependencies (binary wheels or source distributions if there are no binary wheels) for a different platform.
-- Terry Jan Reedy
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sat, Feb 02, 2019 at 07:37:56PM -0500, Terry Reedy wrote: [...]
That's an ambiguous sentence. It could mean that syntax highlighting in both are as useful as a screen door on a submarine :-) I don't know how you use the REPL, but for me, highlighting in the REPL is much less useful because I'm not looking at large chunks of code at a time. If it is more than, oh, two or three lines, it is too big to be editing in the clunky editing environment of a REPL and I'll use an editor to read/edit the code.
Did you put it on PyPI? That's the usual suggestion for something that might be useful but isn't believed to be needed for the stdlib. As Nick said, Rather than "rejected" per se, I've closed this as "postponed" https://bugs.python.org/issue27051#msg286634 https://summerofcode.withgoogle.com/archive/2016/projects/5063140450500608/ [...]
I'd like to see the help pager code made into a public API, and help itself made configurable by the user to choose a pager (auto-detecting a pager by default, as it does now). That would allow people to choose whichever pager works best for them -- possibly including a seperate GUI window which they could leave open all the time.
If you are running a sufficiently recent bash and readline is supported, it should support any feature the POSIX bash supports, regardless of the OS or the terminal. (Although I think readline on Mac is slightly different from other Unix systems? And maybe less functional?) I'm not sure what multiline editing is being referred to above. I personally don't know of one in bash. It may be that you have to turn the feature on -- by default, bash ships with a fair amount of stuff turned off, and the rest hidden behind cryptic keystroke commands. One readline feature I know of which I would love to see supported in the REPL is the "execute and next line" command. On my bash system, it is Ctrl-O. What it does: - use the up-arrow to move through the history to the line you want to execute; - type Ctrl-O to execute that line; - the *next* line is automatically loaded, ready to be executed. E.g. suppose my history consists of five lines: 1: print('hello') 2: def spam(): 3: x = 1 4: return x + 1 5: foo = spam() and I want to re-load the spam function (perhaps after editing it) in the REPL. From a bare prompt, I can: * hit UP-ARROW four times to get to line 2 of the history, then ENTER; * this pushes a new line into the history; * hit UP-ARROW four times again to get to line 3, then ENTER; * hit UP-ARROW four more times to get to line 4, then ENTER. With execute-and-next-line, I could: * hit UP-ARROW four times to get to line 2 of the history, then Ctrl-O; * this loads line 3, so I can immediately hit Ctrl-O; * this loads line 4, so I can immediately hit ENTER. Much easier to re-execute or edit a multiline block! As far as I can tell, it is impossible to add this feature to the REPL from pure Python (corrections welcome!) but I imagine it would be something IDLE could do. -- Steve
On Sun, Feb 03, 2019 at 10:24:12PM +1100, Steven D'Aprano <steve@pearwood.info> wrote:
shopt -s cmdhist From ``man bash``: "If set, bash attempts to save all lines of a multiple-line command in the same history entry. This allows easy re-editing of multi-line commands."
-- Steve
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
Steven D'Aprano <steve@pearwood.info> writes:
I implemented and offered the feature, but unfortunately it is just a couple of lines copied from bash, and that being GPL was considered unfair to be relicensed under current Python license. Maybe sooner or later I will try to repackage it as a standalone extension. ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
Lele Gaifax <lele@metapensiero.it> writes:
Steven D'Aprano <steve@pearwood.info> writes:
One readline feature
To be exact, it's not a feature of the readline library: see https://bugs.python.org/issue22228 (sigh, my GH fork of cpython has been recreated so some refs are broken). ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele@metapensiero.it | -- Fortunato Depero, 1929.
On Fri, Feb 01, 2019 at 09:31:21PM +0100, Oleg Broytman wrote:
Python REPL is missing the following batteries:
* Persistent history;
On Linux/Unix systems, that has been available for about 18+ years, since the rlcompleter module was introduced in Python 2.0: https://docs.python.org/release/2.0/lib/module-rlcompleter.html It's been automatically enabled for more than six years: https://hg.python.org/cpython/rev/d5ef330bac50 Windows is another story.
* Syntax highlighting;
bpython provides syntax highlighting in the REPL, but I've never seen the point. In an editor, perhaps, but why bother in the REPL? https://bpython-interpreter.org/ So does DreamPie: http://www.dreampie.org/
* Clear separation (using, for example, different colors) between input, output and errors;
Input always starts with a prompt; exception tracebacks always start with the line "Traceback..."; other errors are generally not programmatically distinguishable from non-errors. One could modify the displayhook and excepthook to deal with the usual cases, but colour codes are not just platform-specific but console- specific. https://docs.python.org/3.5/library/sys.html#sys.displayhook https://docs.python.org/3.5/library/sys.html#sys.excepthook
* Paging of very long output/errors.
I don't know how I feel about that. Now that Python shrinks long tracebacks filled with identical lines, I'm not sure that this is so important or desirable.
-- Steve
Sorry, I was not as clear as I ought to have been: On Sat, Feb 02, 2019 at 02:10:06PM +1100, Steven D'Aprano wrote:
By which I mean, if a script or module reports an error by just printing a message to the screen: print("sorry, an error occurred") sort of thing. Obviously the interpreter cannot guess which messages represent errors in that sense and which are non-error output. (Of course, a script could format its own error messages, say by using terminal-specific colour codes.) -- Steve
On Sat, Feb 02, 2019 at 03:37:36PM +1100, Steven D'Aprano <steve@pearwood.info> wrote:
Scripts that report errors this way are broken and must be fixed: print("sorry, an error occurred", file=sys.stderr) Now it's easy to separate normal output from errors and colorize them differently.
-- Steve
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Sat, Feb 02, 2019 at 02:10:06PM +1100, Steven D'Aprano <steve@pearwood.info> wrote:
I've configured my own .pythonrc long before that so I missed that it's now finally enabled. Thanks for pointing this out.
And I have a different color just for prompt.
-- Steve
Oleg. -- Oleg Broytman https://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.
On Fri, Feb 01, 2019 at 08:26:03AM +0000, João Matos wrote:
That's a pretty big change, arguably best left for a third-party REPL like iPython (which already supports "magic" commands using ! and ? syntax). It is hard to tell precisely where to draw the line between adding useful functionality to the standard Python interpreter, and where to leave it for "advanced" REPLs or IDEs like iPython. My personal opinion is that specifically supporting shell commands crosses that line, but adding a REPL hook for ! commands which the user can customise does not. Hypothetically speaking, the Python interactive interpreter, and *only* the interactive interpreter, might define hooks for lines starting with a leading ! or ? and users could define their own magic functions for those. Or one could use the code module to define your own custom REPL, and automatically run that on startup: https://docs.python.org/3/library/code.html Actually, adding a code.InteractiveConsole subclass with hooks for lines beginning with ! and ? might be a nice addition to the stdlib without adding complexity to the builtin interpreter. -- Steve
On 2/1/19 2:26 AM, João Matos wrote:
I first ran into this in the days of teletypes and dumb terminals, where other programs let you run shell commands from inside them. Now the shoe appears to be on the other foot. So why not turn that around? ksh (since way back when) and bash (since 2008, according to what I read somewhere online) have "co-processes," which allow you to run a command "in the background," and send commands and receive replies from it. So I tried it with Python, but it didn't work: $ coproc P3 { python3; } $ echo 'import sys; print(sys.version)' >&${P3[1]} $ read v <&${P3[0]} [the read command just waits forever] A pile of experiments and examples from web pages later, I think it's Python and not me. My example, with suitable changes to the literal in the echo command, works with sbcl and erl, but not python3. If I start python3 as follows: $ coproc P3 { python3 | tee /tmp/P3; } then I can see the empty /tmp/P3 file, and the python3 and tee processes, but /tmp/P3 remains empty. Any ideas as to why not? Thanks, Dan
On Sat, Feb 2, 2019 at 7:39 AM Dan Sommers <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:
(This might be better on python-list rather than here)
Looks like you may have a problem with output buffering. Try running "python3 -u" to tell it to run stdout/stderr unbuffered - otherwise, Python assumes that its output is going to a file and it won't matter. Alternatively, explicitly flush stdout after output, which could be done with a display hook: _old_hook = sys.displayhook def display(obj): _old_hook(obj) sys.stdout.flush() sys.displayhook = display But unless you're expecting a lot of output, it's probably easier and just as effective to simply use "-u". ChrisA
On Fri, Feb 01, 2019 at 02:38:43PM -0600, Dan Sommers wrote:
This is another good example of the problem James was referring to in the thread about clearer communication. Don't assume we all know what coproc does.
What are sbcl and erl? I'm guessing you don't mean antimony pentachloride and a municipality in Austria. Possibly Steel Bank Common Lisp and Erlang? But I'm not confident about that. Does your example work with more well-known interpreted languages with interactive interpreters such as Ruby, Lua, Javascript (outside of the browser), etc? -- Steve
On 2/1/19 3:48 PM, Steven D'Aprano wrote:
As I indicated in what you quoted, shell co-processes allow you to run a command in the background and interact with that command from your shell.
Yes, the Steel Bank Common Lisp and Erlang REPLs, respectively.
I don't know (I don't write software in any of those languages, and I don't have them imstalled on my computer), but adding the "-i" flag to my python3 command makes it work (thanks to ChrisA for suggesting "-u"; it was a short leap from there to "-i.") Dan
On Fri, Feb 01, 2019 at 04:28:25PM -0600, Dan Sommers wrote:
Okay, but what does that mean in practice? What does it require to make it work with Python? What is your expected input and output? This is a Python forum. You can assume your readers have a good level of knowledge about Python. Outside of that, you can expect to lose a significant portion of your audience if you start talking about features, practices etc that aren't supported by the Python language, stdlib and perhaps a few of the better-known third-party libraries. E.g. me, I have no idea what you mean by "interact with that command from your shell". If you're a sys admin, you might do that fifty times a day for all I know, but I've never knowingly done it. And are we supposed to know what ">&${P3[1]}" does? It looks like your cat walked over your keyboard. -- Steve
On 2/1/19, Steven D'Aprano <steve@pearwood.info> wrote:
bash coproc runs a process in the background with stdin and stdout redirected to pipes. The file descriptors for our end of the pipes are available in an array with the given name (e.g. P3). The default array name is COPROC. As soon as "pipe" is mentioned, anyone familiar with the REPL's behavior with pipes should know that making this work will require the -i command-line option to force interactive mode. Otherwise stdout will be fully buffered. For example: $ coproc P3 { python3 -qi 2>&1; } [1] 16923 $ echo 'import sys; print(sys.version)' >&${P3[1]} $ read -t 1 <&${P3[0]} && echo $REPLY >>> 3.6.7 (default, Oct 22 2018, 11:32:17) $ read -t 1 <&${P3[0]} && echo $REPLY [GCC 8.2.0] $ read -t 1 <&${P3[0]} && echo $REPLY $ echo 'sys.exit(42)' >&${P3[1]} $ [1]+ Exit 42 coproc P3 { python3 -qi 2>&1; }
And are we supposed to know what ">&${P3[1]}" does? It looks like your cat walked over your keyboard.
It redirects the command's standard output (>) to the file descriptor (&) in index 1 of the P3 array (${P3[1]}), which is our end of the pipe that's connected to stdin of the co-process.
On Fri, Feb 01, 2019 at 07:21:47PM -0600, eryk sun wrote:
Thanks for the explanation.
I wonder... could Python automatically detect when it is connected to pipes and switch buffering off?
And this is why I don't program in bash :-) -- Steven
On 2/1/19, Steven D'Aprano <steve@pearwood.info> wrote:
In most cases we want full buffering when standard I/O is a pipe or disk file. It's more efficient to read/write large chunks from/to the OS. In another message I saw -u mentioned to disable buffering. But that's not sufficient. We need -i to force running the built-in REPL over a pipe, and optionally -q to quiet the initial banner message.
participants (12)
-
Chris Angelico -
Dan Sommers -
David Mertz -
Eric V. Smith -
eryk sun -
James Lu -
João Matos -
Lele Gaifax -
Mike Müller -
Oleg Broytman -
Steven D'Aprano -
Terry Reedy