Incorrect documentation of the raw_input built-in function
Recently I was trying to debug an old python program who's maintenance I inherited. I was using the quick-and-dirty method of putting some 'print
The latest documentation for raw_input states "If the prompt argument is present, it is written to standard output without a trailing newline." I posted a question regarding the observed behavior to comp.lang.python and Gabriel Genellina (thanks Gabriel!) pointed out that despite the documentation, raw_input was hard-coded to always output its prompt text to stderr. This raises two questions: 1. Shouldn't the current documentation be corrected to state that raw_input writes its prompt to standard error? 2. Is this really the hard-coded behavior we want? I don't think my use-case is that odd; in fact, what I find very odd is that the prompt output is send to stderr. I mean, I'm printing the prompt for a question, not some error message. Can there not at least be an optional parameter to indicate that you want the output sent to stdout rather than stderr?
Mike> 2. Is this really the hard-coded behavior we want? I don't think Mike> my use-case is that odd; in fact, what I find very odd is that Mike> the prompt output is send to stderr. I mean, I'm printing the Mike> prompt for a question, not some error message. Can there not at Mike> least be an optional parameter to indicate that you want the Mike> output sent to stdout rather than stderr? I can think of situations where you don't want the output to go to stdout either (suppose it's the regular output of the file you want to save to a file). Having a choice seems the best route.
On Thu, 24 Jan 2008, skip@pobox.com wrote:
What about an option (maybe even a default) to send the prompt to stdin? The Postgres command line interface psql appears to do this: $ psql 2>&1 >/dev/null Password: $ (I typed my password and then I quit by typing ^D; if I type the wrong password, it looks the same on screen but it quits right away without waiting for ^D) I think ssh also does this when it needs to prompt for a password. Really the prompt is part of the input, not part of the output, in a certain sense. Have people actually verified that the prompt is really sent to stderr right now by using 2>/dev/null to attempt to suppress it? Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist
On Thu, Jan 24, 2008 at 12:36:51PM -0500, Isaac Morland wrote:
One cannot write to stdin:
But it is possible to explicitly open the current console for reading and writing and do I/O regardless of stdin/stdout/stderr redirects:
Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
Isaac> Have people actually verified that the prompt is really sent to Isaac> stderr right now by using 2>/dev/null to attempt to suppress it? Good point. On my machine at work (Solaris), Python 2.4 seems to send its raw_input prompt to stdout, not stderr: % python Python 2.4.2 (#1, Feb 23 2006, 12:48:31) [GCC 3.4.1] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> raw_input("?") ?z 'z' >>> ink% python 2>/dev/null >>> raw_input("?") ?sdf 'sdf' >>> ink% python >/dev/null Python 2.4.2 (#1, Feb 23 2006, 12:48:31) [GCC 3.4.1] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> raw_input("?") wer >>> ^D Same for 2.6a0 on my Mac at home. Mike, are you sure about it prompting to stderr? If so, what's your setup? Skip
Beware: this may depend on whether GNU readline is enabled or not -- under many circumstances, raw_input() calls GNU readline instead of using sys.stdin/stdout. I do agree that if there are any conditions where it uses stderr instead of stdout those are mistakes. On Jan 24, 2008 9:47 AM, <skip@pobox.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
<skip <at> pobox.com> writes:
Skip, Guido and others: Interesting point about whether GNU readline is installed. My setup is RedHat Linux, with Python 2.5 that I built and installed myself. GNU readline is not, in fact, installed. If you look at Python2.5/Parser/myreadline.c, function PyOS_StdioReadline, line 125, you will see that prompt output is being sent to stderr. As best as my Python-fu can determine, this is the code used to output a raw_input prompt (thanks again to Gabriel Genellina for pointing me in the right direction.) It's entirely likely that the difference in what I am seeing and what you guys are seeing is caused by my not having GNU readline installed. Nevertheless, the behavior without it seems wrong, and is certainly different from the documentation.
On Jan 24, 2008 11:41 AM, Mike Kent <mike.kent@sage.com> wrote:
Agreed. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <guido <at> python.org> writes:
Being that Guido agrees, should I put a bug report into the system for it, and if so, which should it report as the bug: the actual behavior of the raw_input prompt when GNU readline is not installed, or the documentation?
IMO the actual behavior is incorrect. On Jan 24, 2008 12:14 PM, Mike Kent <mike.kent@sage.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
"Guido" == Guido van Rossum <guido@python.org> writes:
>> It's entirely likely that the difference in what I am seeing and what >> you guys are seeing is caused by my not having GNU readline >> installed. Nevertheless, the behavior without it seems wrong, and is >> certainly different from the documentation. Guido> Agreed. http://bugs.python.org/issue1927 Skip
Incodentally, I think your example is wrong. It should be either psql >/dev/null 2>&1 or psql &>/dev/null Notice the reversal of redirects in the first one. Of course, this doesn't change anything about the core discussion... - C On 1/24/08, Isaac Morland <ijmorlan@cs.uwaterloo.ca> wrote:
-- Sent from Gmail for mobile | mobile.google.com
Isaac Morland wrote:
No, it's probably using the C stdlib routine getpass(). From the man page: The getpass() function displays a prompt to, and reads in a password from, /dev/tty. If this file is not accessible, getpass() displays the prompt on the standard error output and reads from the standard input. So it appears that the official Unix Way prefers using stderr over stdout for prompting, if using the std files for it at all. Writing to stdin would be wrong, since it's usually read-only, even when connected to a terminal. -- Greg
On Jan 28, 2008 12:35 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
That's a dangerous generalization from just one example. I'd prefer it if you could unearth some POSIX or Linux base document saying this.
Writing to stdin would be wrong, since it's usually read-only, even when connected to a terminal.
Nowadays, it often is writable I've found, but we still shouldn't assume that. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
-On [20080128 18:57], Guido van Rossum (guido@python.org) wrote:
I cannot find anything explicitly in favour of or against this in POSIX. The stuff quoted below is what I could find. I do not think there's a hard mandate either way, but from my experience of having been a committer for both FreeBSD and DragonFly BSD for a number of years stderr was not preferred for prompting. stderr was always the domain of diagnostics. "3.105 Command Language Interpreter An interface that interprets sequences of text input as commands. It may operate on an input stream or it may interactively prompt and read commands from a terminal. It is possible for applications to invoke utilities through a number of interfaces, which are collectively considered to act as command interpreters." POSIX defines per utility what stdin, stdout and stderr do. The default definition for these is: STDIN There is no additional rationale provided for this section. STDOUT The format description is intended to be sufficiently rigorous to allow post-processing of output by other programs, particularly by an awk or lex parser. STDERR This section does not describe error messages that refer to incorrect operation of the utility. Consider a utility that processes program source code as its input. This section is used to describe messages produced by a correctly operating utility that encounters an error in the program source code on which it is processing. However, a message indicating that the utility had insufficient memory in which to operate would not be described. Some utilities have traditionally produced warning messages without returning a non-zero exit status; these are specifically noted in their sections. Other utilities shall not write to standard error if they complete successfully, unless the implementation provides some sort of extension to increase the verbosity or debugging level. The format descriptions are intended to be sufficiently rigorous to allow post-processing of output by other programs. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ We have met the enemy and they are ours...
-On [20080128 19:51], Jeroen Ruigrok van der Werven (asmodai@in-nomine.org) wrote:
Sorry, I made a mental typo here: but from my experience of having been a committer for both FreeBSD and DragonFly BSD for a number of years stdout was not preferred for prompting. stderr was always the domain of diagnostics. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ We have met the enemy and they are ours...
Jeroen Ruigrok van der Werven wrote:
What document did this come from? This sounds more like it's talking about what should be described in various sections of a man page, not what should be written to the various streams by a program. Otherwise,
a message indicating that the utility had insufficient memory in which to operate would not be described.
sounds like an out-of-memory message shouldn't be written to stderr, which I'm fairly sure is not the case! -- Greg
-On [20080129 00:13], Greg Ewing (greg.ewing@canterbury.ac.nz) wrote:
It did, it's from the C. Rationale for Shell and Utilities (XCU).
It's only talking about what will be described in the various sections of the utilities' documentation, yes. But it was the closest I could find to *anything* explicitly mentioning stdin, stdout, or stderr and their function. I could find nothing with regard to prompting or other such descriptions. So, don't count on POSIX to mandate anything on this (as far as I can tell). -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ We have met the enemy and they are ours...
Guido van Rossum wrote:
While I couldn't locate such a document, it makes sense when you consider that if such a process is part of a pipeline you really don't want the prompts being handled as input by the downstream processes. That's why mv and similar utilities prompt on standard error. I'd be very surprised if you can find a UNIX or Linux shell, for example, that prompts on standard output.
regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/
On Jan 28, 2008 11:17 AM, Steve Holden <steve@holdenweb.com> wrote:
Ah, indeed. It looks like Python's prompt indeed gets written to stderr, at least when stdout is redirected to a file. Oddly, when stderr is redirected to a file, the prompt gets written to stdout. This seems backwards then (and not what the shells do -- when stderr isn't interactive, they don't write a prompt at all). But fixing it is delicate -- e.g. getpass very carefully makes sure to write to stdout, not stderr (and to avoid raw_input() and hence GNU readline). And cmd.py (used by pdb.py) also assumes the prompt should go to stdout when it is not using raw_input. I'm beginning to think that, despite the example of the shells, we'd be better off if Python *always* prompted to stdout (or not at all if that's not a tty). -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On Jan 28, 2008 12:35 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Well, it makes sense in the context of the general Unix philosophy that stdout is for the result that the program is computing, and therefore shouldn't be polluted by error messages or other extraneous things. Also, the Python interpreter itself appears to write its prompts to stderr, both with and without readline. -- Greg
Mike> 2. Is this really the hard-coded behavior we want? I don't think Mike> my use-case is that odd; in fact, what I find very odd is that Mike> the prompt output is send to stderr. I mean, I'm printing the Mike> prompt for a question, not some error message. Can there not at Mike> least be an optional parameter to indicate that you want the Mike> output sent to stdout rather than stderr? I can think of situations where you don't want the output to go to stdout either (suppose it's the regular output of the file you want to save to a file). Having a choice seems the best route.
On Thu, 24 Jan 2008, skip@pobox.com wrote:
What about an option (maybe even a default) to send the prompt to stdin? The Postgres command line interface psql appears to do this: $ psql 2>&1 >/dev/null Password: $ (I typed my password and then I quit by typing ^D; if I type the wrong password, it looks the same on screen but it quits right away without waiting for ^D) I think ssh also does this when it needs to prompt for a password. Really the prompt is part of the input, not part of the output, in a certain sense. Have people actually verified that the prompt is really sent to stderr right now by using 2>/dev/null to attempt to suppress it? Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist
On Thu, Jan 24, 2008 at 12:36:51PM -0500, Isaac Morland wrote:
One cannot write to stdin:
But it is possible to explicitly open the current console for reading and writing and do I/O regardless of stdin/stdout/stderr redirects:
Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
Isaac> Have people actually verified that the prompt is really sent to Isaac> stderr right now by using 2>/dev/null to attempt to suppress it? Good point. On my machine at work (Solaris), Python 2.4 seems to send its raw_input prompt to stdout, not stderr: % python Python 2.4.2 (#1, Feb 23 2006, 12:48:31) [GCC 3.4.1] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> raw_input("?") ?z 'z' >>> ink% python 2>/dev/null >>> raw_input("?") ?sdf 'sdf' >>> ink% python >/dev/null Python 2.4.2 (#1, Feb 23 2006, 12:48:31) [GCC 3.4.1] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> raw_input("?") wer >>> ^D Same for 2.6a0 on my Mac at home. Mike, are you sure about it prompting to stderr? If so, what's your setup? Skip
Beware: this may depend on whether GNU readline is enabled or not -- under many circumstances, raw_input() calls GNU readline instead of using sys.stdin/stdout. I do agree that if there are any conditions where it uses stderr instead of stdout those are mistakes. On Jan 24, 2008 9:47 AM, <skip@pobox.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
<skip <at> pobox.com> writes:
Skip, Guido and others: Interesting point about whether GNU readline is installed. My setup is RedHat Linux, with Python 2.5 that I built and installed myself. GNU readline is not, in fact, installed. If you look at Python2.5/Parser/myreadline.c, function PyOS_StdioReadline, line 125, you will see that prompt output is being sent to stderr. As best as my Python-fu can determine, this is the code used to output a raw_input prompt (thanks again to Gabriel Genellina for pointing me in the right direction.) It's entirely likely that the difference in what I am seeing and what you guys are seeing is caused by my not having GNU readline installed. Nevertheless, the behavior without it seems wrong, and is certainly different from the documentation.
On Jan 24, 2008 11:41 AM, Mike Kent <mike.kent@sage.com> wrote:
Agreed. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum <guido <at> python.org> writes:
Being that Guido agrees, should I put a bug report into the system for it, and if so, which should it report as the bug: the actual behavior of the raw_input prompt when GNU readline is not installed, or the documentation?
IMO the actual behavior is incorrect. On Jan 24, 2008 12:14 PM, Mike Kent <mike.kent@sage.com> wrote:
-- --Guido van Rossum (home page: http://www.python.org/~guido/)
"Guido" == Guido van Rossum <guido@python.org> writes:
>> It's entirely likely that the difference in what I am seeing and what >> you guys are seeing is caused by my not having GNU readline >> installed. Nevertheless, the behavior without it seems wrong, and is >> certainly different from the documentation. Guido> Agreed. http://bugs.python.org/issue1927 Skip
Incodentally, I think your example is wrong. It should be either psql >/dev/null 2>&1 or psql &>/dev/null Notice the reversal of redirects in the first one. Of course, this doesn't change anything about the core discussion... - C On 1/24/08, Isaac Morland <ijmorlan@cs.uwaterloo.ca> wrote:
-- Sent from Gmail for mobile | mobile.google.com
Isaac Morland wrote:
No, it's probably using the C stdlib routine getpass(). From the man page: The getpass() function displays a prompt to, and reads in a password from, /dev/tty. If this file is not accessible, getpass() displays the prompt on the standard error output and reads from the standard input. So it appears that the official Unix Way prefers using stderr over stdout for prompting, if using the std files for it at all. Writing to stdin would be wrong, since it's usually read-only, even when connected to a terminal. -- Greg
On Jan 28, 2008 12:35 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
That's a dangerous generalization from just one example. I'd prefer it if you could unearth some POSIX or Linux base document saying this.
Writing to stdin would be wrong, since it's usually read-only, even when connected to a terminal.
Nowadays, it often is writable I've found, but we still shouldn't assume that. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
-On [20080128 18:57], Guido van Rossum (guido@python.org) wrote:
I cannot find anything explicitly in favour of or against this in POSIX. The stuff quoted below is what I could find. I do not think there's a hard mandate either way, but from my experience of having been a committer for both FreeBSD and DragonFly BSD for a number of years stderr was not preferred for prompting. stderr was always the domain of diagnostics. "3.105 Command Language Interpreter An interface that interprets sequences of text input as commands. It may operate on an input stream or it may interactively prompt and read commands from a terminal. It is possible for applications to invoke utilities through a number of interfaces, which are collectively considered to act as command interpreters." POSIX defines per utility what stdin, stdout and stderr do. The default definition for these is: STDIN There is no additional rationale provided for this section. STDOUT The format description is intended to be sufficiently rigorous to allow post-processing of output by other programs, particularly by an awk or lex parser. STDERR This section does not describe error messages that refer to incorrect operation of the utility. Consider a utility that processes program source code as its input. This section is used to describe messages produced by a correctly operating utility that encounters an error in the program source code on which it is processing. However, a message indicating that the utility had insufficient memory in which to operate would not be described. Some utilities have traditionally produced warning messages without returning a non-zero exit status; these are specifically noted in their sections. Other utilities shall not write to standard error if they complete successfully, unless the implementation provides some sort of extension to increase the verbosity or debugging level. The format descriptions are intended to be sufficiently rigorous to allow post-processing of output by other programs. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ We have met the enemy and they are ours...
-On [20080128 19:51], Jeroen Ruigrok van der Werven (asmodai@in-nomine.org) wrote:
Sorry, I made a mental typo here: but from my experience of having been a committer for both FreeBSD and DragonFly BSD for a number of years stdout was not preferred for prompting. stderr was always the domain of diagnostics. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ We have met the enemy and they are ours...
Jeroen Ruigrok van der Werven wrote:
What document did this come from? This sounds more like it's talking about what should be described in various sections of a man page, not what should be written to the various streams by a program. Otherwise,
a message indicating that the utility had insufficient memory in which to operate would not be described.
sounds like an out-of-memory message shouldn't be written to stderr, which I'm fairly sure is not the case! -- Greg
-On [20080129 00:13], Greg Ewing (greg.ewing@canterbury.ac.nz) wrote:
It did, it's from the C. Rationale for Shell and Utilities (XCU).
It's only talking about what will be described in the various sections of the utilities' documentation, yes. But it was the closest I could find to *anything* explicitly mentioning stdin, stdout, or stderr and their function. I could find nothing with regard to prompting or other such descriptions. So, don't count on POSIX to mandate anything on this (as far as I can tell). -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ We have met the enemy and they are ours...
Guido van Rossum wrote:
While I couldn't locate such a document, it makes sense when you consider that if such a process is part of a pipeline you really don't want the prompts being handled as input by the downstream processes. That's why mv and similar utilities prompt on standard error. I'd be very surprised if you can find a UNIX or Linux shell, for example, that prompts on standard output.
regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/
On Jan 28, 2008 11:17 AM, Steve Holden <steve@holdenweb.com> wrote:
Ah, indeed. It looks like Python's prompt indeed gets written to stderr, at least when stdout is redirected to a file. Oddly, when stderr is redirected to a file, the prompt gets written to stdout. This seems backwards then (and not what the shells do -- when stderr isn't interactive, they don't write a prompt at all). But fixing it is delicate -- e.g. getpass very carefully makes sure to write to stdout, not stderr (and to avoid raw_input() and hence GNU readline). And cmd.py (used by pdb.py) also assumes the prompt should go to stdout when it is not using raw_input. I'm beginning to think that, despite the example of the shells, we'd be better off if Python *always* prompted to stdout (or not at all if that's not a tty). -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
On Jan 28, 2008 12:35 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Well, it makes sense in the context of the general Unix philosophy that stdout is for the result that the program is computing, and therefore shouldn't be polluted by error messages or other extraneous things. Also, the Python interpreter itself appears to write its prompts to stderr, both with and without readline. -- Greg
participants (10)
-
"Martin v. Löwis" -
Chris Monson -
Greg Ewing -
Guido van Rossum -
Isaac Morland -
Jeroen Ruigrok van der Werven -
Mike Kent -
Oleg Broytmann -
skip＠pobox.com -
Steve Holden