So, I was just thinking, maybe we don't want an errfile arg, but an arg that is a sequence of file objects that need to be flushed before showing the prompt. That's certainly more complicated, but it seems more general if we want to cover this case.


Thoughts?


wt


From: Python-ideas <python-ideas-bounces+w00t=fb.com@python.org> on behalf of Wren Turkal <w00t@fb.com>
Sent: Friday, September 29, 2017 9:53:01 AM
To: Amit Green
Cc: python-ideas@python.org
Subject: [Potential Spoof] Re: [Python-ideas] allow overriding files used for the input builtin
 

Oh, I thought that stderr was unbuffered. Like the following C program:


#include <stdio.h>

int main() {
  fprintf(stdout, "stdout0");
  fprintf(stderr, "stderr0");
  fprintf(stdout, "stdout1");
  return 0;
}

This outputs:

stderr0stdout0stdout1


Turns out fprintf in glibc will also implicitly flush at newlines also. So, the following program:

#include <stdio.h>

int main() {
  fprintf(stdout, "stdout0\nstdout12");
  fprintf(stderr, "stderr0");
  fprintf(stdout, "stdout1");
  return 0;
}

outputs:

stdout0
stderr0stdout12stdout1

This is in line with python. The following program:

import sys

sys.stdout.write('stdout0\nstdout01')
sys.stderr.write('stderr0')
sys.stdout.write('stdout1')

outputs:

stdout0
stderr0stdout01stdout1

While the following program:

import sys

sys.stdout.write('stdout0')
sys.stderr.write('stderr0')
sys.stdout.write('stdout1')

outputs:

stderr0stdout0stdout1


So, I guess that the errfile is explicitly flushed due to the fact that stdout may or may not be flushed during output.


So, I have 2 questions: (1) do we need to keep the errfile param to make this useful (I think we do), and if #1 is yes, (2) what is an appropriate way to doc this?


wt


From: Amit Green <amit.mixie@gmail.com>
Sent: Friday, September 29, 2017 9:17:39 AM
To: Wren Turkal
Cc: Steven D'Aprano; python-ideas@python.org
Subject: Re: [Python-ideas] allow overriding files used for the input builtin
 
Hmm, very good point.

Flushing standard error is essential:

  • This is because output of standard output & standard error are often redirected to the same file -- frequently the terminal -- and its important to flush one of them before output to the other (since these are typically line buffered you only see this if you output partial lines).
  • Here is an example of not flushing & the mess it causes:

>>> import sys
>>> sys.stdout.write('hi'); sys.stderr.write(' HMM '); sys.stdout.write('there\n')
 HMM hithere

  • Here is the same example with flushing:

>>> sys.stdout.write('hi'); sys.stdout.flush(); sys.stderr.write(' HMM '); sys.stderr.flush(); sys.stdout.write('there\n')
hi HMM there

In fact, I think you need to improve your interface and documentation:

  • If either of the parameters 'outfile', or 'errfile' are passed in & not the 'none' value then the following happens:
    • Both sys.stdout & sys.stderr are flushed (in that order) before being redirected.  (NOTE: Both are flushed, even if only one is redirected).
    • Before restoring sys.stdout & sys.stderr; then outfile & errfile are flushed (if both have been used then they are flushed in that order).

You would of course need to write the documentation clearer than I did above (writing documentation well is not my skill) -- I wrote it to convey exactly what has to happen.


On Fri, Sep 29, 2017 at 12:01 PM, Wren Turkal <w00t@fb.com> wrote:

Steven and Amit,


I originally configured the mailing list for digest delivery and can't reply directly to his message. However, I now seen it, I will update the PR with the suggested name changes as soon as my "make test" finishes. FWIW, I've changed to direct message delivery.


With regard to ferr (the corresponding variable in the builtin_input c function before the change), I saw this code:


    /* First of all, flush stderr */
    tmp = _PyObject_CallMethodId(ferr, &PyId_flush, NULL);
    if (tmp == NULL)
        PyErr_Clear();
    else
        Py_DECREF(tmp);

And I assumed that it was important to be able to override a stderr as a result.


I think there are few options here to resolve that:

  1. Remove the errfile parameter and just do what happened before.
  2. Remove the errfile and the above code (FWIW, I am not sure I understand the importance of flushing stderr before taking input).
  3. Document the explicit purpose of the errfile. Amit, you'd responded with something about this (again digest reply, sorry). I am not sure how to concisely do that given Amit's descriptions. 😊

I am honestly leaning toward 2 unless I can figure out why the flushing of stderr is actually needed.


wt



From: Amit Green <amit.mixie@gmail.com>
Sent: Friday, September 29, 2017 8:13:21 AM

To: Wren Turkal
Cc: python-ideas@python.org
Subject: Re: [Python-ideas] allow overriding files used for the input builtin
 
Yes, infile, outfile & errfile would be consistent with python naming convention (also mentioned by Steven D'Aprano above)

One of python's greatest strength is its library, the consistency of the library, and how well documented the library is (in fact, I think the library is a greater strength than even the very nice syntax of python in general).

By "consistency of the library" I mean: functions do pretty much what you expect, they use consistent error mechanism & the documentation pretty much accurately documents what the function does -- especially as to showing its results & how it handles errors.

Regarding this though, this then brings up the question (above from Steven D'Aprano) -- what would the the "errfile" parameter do?
  • As a general principle of consistency python library functions, and python itself, do not output to errfile, but instead throw errors.
  • (There are very minor exceptions such as exceptions thrown in __del__ functions; which are caught by python & then printed to standard error).

I would thus think you don't want the errfile parameter -- unless it would be for catching these __del__ method that get triggered by input failing (for example your 'infile' parameter when called, allocated an object, which gets deallocated & throws an exception inside of its __del__ method).

If this is the purpose, then (back to 'how well documented the library is') -- it should be documented this is the purpose of the "errfile" parameter ;-)

[A secondary reason you might want to redirect "errfile" is that the passed in input or output file's, themselves do output to standard error ...]



On Fri, Sep 29, 2017 at 10:50 AM, Wren Turkal <w00t@fb.com> wrote:

I am happy to rename the args. What do you think about infile, outfile, and errfile?


FWIW, I did consider "in", "out", and "err", but "in" is a keyword, and I didn't think those quite captured the full meaning.


wt


From: Amit Green <amit.mixie@gmail.com>
Sent: Thursday, September 28, 2017 11:18:18 PM
To: Wren Turkal
Cc: python-ideas@python.org
Subject: Re: [Python-ideas] allow overriding files used for the input builtin
 
I'm fine with the idea in general of extra keyword parameters to the input function.

A few points:

Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style).

Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'.  This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important).

Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time.

On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com> wrote:

Hi there,


I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603.


The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output.


This is meant to turn code like the following:

orig_stdin = sys.stdin

orig_stdout = sys.stdout

with open('/dev/tty', 'r+') as f:

    sys.stdin = f

    sys.stdout = f

    name = input('Name? ')

sys.stdin = orig_stdin

sys.stdout = orig_stdout

print(name)


into something more like this:

with open('/dev/tty', 'r+') as f:

    name = input('Name? ', fin=f, fout=f)

print(name)


It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed.


I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also.


Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args.


FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther.


Wren Turkal

Existential Production Engineer of the Ages

Facebook, Inc.


_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/