allow overriding files used for the input builtin
Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc.
I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com> wrote:
I am happy to rename the args. What do you think about infile, outfile, and errfile? FWIW, I did consider "in", "out", and "err", but "in" is a keyword, and I didn't think those quite captured the full meaning. wt ________________________________ From: Amit Green <amit.mixie@gmail.com> Sent: Thursday, September 28, 2017 11:18:18 PM To: Wren Turkal Cc: python-ideas@python.org Subject: Re: [Python-ideas] allow overriding files used for the input builtin I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603<https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue31603&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=sjDmBcI00MbWPPjuLMPlBnZHoFZOHoTxaCo2KCYlEd4&e=>. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc. _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_python-2Dideas&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=dvbtuL0kPe5UxakqosCnQQCIxlKprMP6JTqx4ZLXx4g&e=> Code of Conduct: http://python.org/psf/codeofconduct/<https://urldefense.proofpoint.com/v2/url?u=http-3A__python.org_psf_codeofconduct_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=px67QeYtdnaejiTP9VcY5IuSiJ4pk3XEXcbDzZWbsnI&e=>
Yes, infile, outfile & errfile would be consistent with python naming convention (also mentioned by Steven D'Aprano above) One of python's greatest strength is its library, the consistency of the library, and how well documented the library is (in fact, I think the library is a greater strength than even the very nice syntax of python in general). By "consistency of the library" I mean: functions do pretty much what you expect, they use consistent error mechanism & the documentation pretty much accurately documents what the function does -- especially as to showing its results & how it handles errors. Regarding this though, this then brings up the question (above from Steven D'Aprano) -- what would the the "errfile" parameter do? - As a general principle of consistency python library functions, and python itself, do not output to errfile, but instead throw errors. - (There are very minor exceptions such as exceptions thrown in __del__ functions; which are caught by python & then printed to standard error). I would thus think you don't want the errfile parameter -- unless it would be for catching these __del__ method that get triggered by input failing (for example your 'infile' parameter when called, allocated an object, which gets deallocated & throws an exception inside of its __del__ method). If this is the purpose, then (back to 'how well documented the library is') -- it should be documented this is the purpose of the "errfile" parameter ;-) [A secondary reason you might want to redirect "errfile" is that the passed in input or output file's, themselves do output to standard error ...] On Fri, Sep 29, 2017 at 10:50 AM, Wren Turkal <w00t@fb.com> wrote:
Steven and Amit, I originally configured the mailing list for digest delivery and can't reply directly to his message. However, I now seen it, I will update the PR with the suggested name changes as soon as my "make test" finishes. FWIW, I've changed to direct message delivery. With regard to ferr (the corresponding variable in the builtin_input c function before the change), I saw this code: /* First of all, flush stderr */ tmp = _PyObject_CallMethodId(ferr, &PyId_flush, NULL); if (tmp == NULL) PyErr_Clear(); else Py_DECREF(tmp); And I assumed that it was important to be able to override a stderr as a result. I think there are few options here to resolve that: 1. Remove the errfile parameter and just do what happened before. 2. Remove the errfile and the above code (FWIW, I am not sure I understand the importance of flushing stderr before taking input). 3. Document the explicit purpose of the errfile. Amit, you'd responded with something about this (again digest reply, sorry). I am not sure how to concisely do that given Amit's descriptions. ๐ I am honestly leaning toward 2 unless I can figure out why the flushing of stderr is actually needed. wt ________________________________ From: Amit Green <amit.mixie@gmail.com> Sent: Friday, September 29, 2017 8:13:21 AM To: Wren Turkal Cc: python-ideas@python.org Subject: Re: [Python-ideas] allow overriding files used for the input builtin Yes, infile, outfile & errfile would be consistent with python naming convention (also mentioned by Steven D'Aprano above) One of python's greatest strength is its library, the consistency of the library, and how well documented the library is (in fact, I think the library is a greater strength than even the very nice syntax of python in general). By "consistency of the library" I mean: functions do pretty much what you expect, they use consistent error mechanism & the documentation pretty much accurately documents what the function does -- especially as to showing its results & how it handles errors. Regarding this though, this then brings up the question (above from Steven D'Aprano) -- what would the the "errfile" parameter do? * As a general principle of consistency python library functions, and python itself, do not output to errfile, but instead throw errors. * (There are very minor exceptions such as exceptions thrown in __del__ functions; which are caught by python & then printed to standard error). I would thus think you don't want the errfile parameter -- unless it would be for catching these __del__ method that get triggered by input failing (for example your 'infile' parameter when called, allocated an object, which gets deallocated & throws an exception inside of its __del__ method). If this is the purpose, then (back to 'how well documented the library is') -- it should be documented this is the purpose of the "errfile" parameter ;-) [A secondary reason you might want to redirect "errfile" is that the passed in input or output file's, themselves do output to standard error ...] On Fri, Sep 29, 2017 at 10:50 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: I am happy to rename the args. What do you think about infile, outfile, and errfile? FWIW, I did consider "in", "out", and "err", but "in" is a keyword, and I didn't think those quite captured the full meaning. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Thursday, September 28, 2017 11:18:18 PM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603<https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue31603&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=sjDmBcI00MbWPPjuLMPlBnZHoFZOHoTxaCo2KCYlEd4&e=>. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc. _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_python-2Dideas&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=dvbtuL0kPe5UxakqosCnQQCIxlKprMP6JTqx4ZLXx4g&e=> Code of Conduct: http://python.org/psf/codeofconduct/<https://urldefense.proofpoint.com/v2/url?u=http-3A__python.org_psf_codeofconduct_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=px67QeYtdnaejiTP9VcY5IuSiJ4pk3XEXcbDzZWbsnI&e=>
Hmm, very good point. Flushing standard error is essential: - This is because output of standard output & standard error are often redirected to the same file -- frequently the terminal -- and its important to flush one of them before output to the other (since these are typically line buffered you only see this if you output partial lines). - Here is an example of not flushing & the mess it causes:
- Here is the same example with flushing:
In fact, I think you need to improve your interface and documentation: - If either of the parameters 'outfile', or 'errfile' are passed in & not the 'none' value then the following happens: - Both sys.stdout & sys.stderr are flushed (in that order) before being redirected. (NOTE: Both are flushed, even if only one is redirected). - Before restoring sys.stdout & sys.stderr; then outfile & errfile are flushed (if both have been used then they are flushed in that order). You would of course need to write the documentation clearer than I did above (writing documentation well is not my skill) -- I wrote it to convey exactly what has to happen. On Fri, Sep 29, 2017 at 12:01 PM, Wren Turkal <w00t@fb.com> wrote:
Oh, I thought that stderr was unbuffered. Like the following C program: #include <stdio.h> int main() { fprintf(stdout, "stdout0"); fprintf(stderr, "stderr0"); fprintf(stdout, "stdout1"); return 0; } This outputs: stderr0stdout0stdout1 Turns out fprintf in glibc will also implicitly flush at newlines also. So, the following program: #include <stdio.h> int main() { fprintf(stdout, "stdout0\nstdout12"); fprintf(stderr, "stderr0"); fprintf(stdout, "stdout1"); return 0; } outputs: stdout0 stderr0stdout12stdout1 This is in line with python. The following program: import sys sys.stdout.write('stdout0\nstdout01') sys.stderr.write('stderr0') sys.stdout.write('stdout1') outputs: stdout0 stderr0stdout01stdout1 While the following program: import sys sys.stdout.write('stdout0') sys.stderr.write('stderr0') sys.stdout.write('stdout1') outputs: stderr0stdout0stdout1 So, I guess that the errfile is explicitly flushed due to the fact that stdout may or may not be flushed during output. So, I have 2 questions: (1) do we need to keep the errfile param to make this useful (I think we do), and if #1 is yes, (2) what is an appropriate way to doc this? wt ________________________________ From: Amit Green <amit.mixie@gmail.com> Sent: Friday, September 29, 2017 9:17:39 AM To: Wren Turkal Cc: Steven D'Aprano; python-ideas@python.org Subject: Re: [Python-ideas] allow overriding files used for the input builtin Hmm, very good point. Flushing standard error is essential: * This is because output of standard output & standard error are often redirected to the same file -- frequently the terminal -- and its important to flush one of them before output to the other (since these are typically line buffered you only see this if you output partial lines). * Here is an example of not flushing & the mess it causes:
* Here is the same example with flushing:
sys.stdout.write('hi'); sys.stdout.flush(); sys.stderr.write(' HMM '); sys.stderr.flush(); sys.stdout.write('there\n') hi HMM there
In fact, I think you need to improve your interface and documentation: * If either of the parameters 'outfile', or 'errfile' are passed in & not the 'none' value then the following happens: * Both sys.stdout & sys.stderr are flushed (in that order) before being redirected. (NOTE: Both are flushed, even if only one is redirected). * Before restoring sys.stdout & sys.stderr; then outfile & errfile are flushed (if both have been used then they are flushed in that order). You would of course need to write the documentation clearer than I did above (writing documentation well is not my skill) -- I wrote it to convey exactly what has to happen. On Fri, Sep 29, 2017 at 12:01 PM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Steven and Amit, I originally configured the mailing list for digest delivery and can't reply directly to his message. However, I now seen it, I will update the PR with the suggested name changes as soon as my "make test" finishes. FWIW, I've changed to direct message delivery. With regard to ferr (the corresponding variable in the builtin_input c function before the change), I saw this code: /* First of all, flush stderr */ tmp = _PyObject_CallMethodId(ferr, &PyId_flush, NULL); if (tmp == NULL) PyErr_Clear(); else Py_DECREF(tmp); And I assumed that it was important to be able to override a stderr as a result. I think there are few options here to resolve that: 1. Remove the errfile parameter and just do what happened before. 2. Remove the errfile and the above code (FWIW, I am not sure I understand the importance of flushing stderr before taking input). 3. Document the explicit purpose of the errfile. Amit, you'd responded with something about this (again digest reply, sorry). I am not sure how to concisely do that given Amit's descriptions. ๐ I am honestly leaning toward 2 unless I can figure out why the flushing of stderr is actually needed. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Friday, September 29, 2017 8:13:21 AM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin Yes, infile, outfile & errfile would be consistent with python naming convention (also mentioned by Steven D'Aprano above) One of python's greatest strength is its library, the consistency of the library, and how well documented the library is (in fact, I think the library is a greater strength than even the very nice syntax of python in general). By "consistency of the library" I mean: functions do pretty much what you expect, they use consistent error mechanism & the documentation pretty much accurately documents what the function does -- especially as to showing its results & how it handles errors. Regarding this though, this then brings up the question (above from Steven D'Aprano) -- what would the the "errfile" parameter do? * As a general principle of consistency python library functions, and python itself, do not output to errfile, but instead throw errors. * (There are very minor exceptions such as exceptions thrown in __del__ functions; which are caught by python & then printed to standard error). I would thus think you don't want the errfile parameter -- unless it would be for catching these __del__ method that get triggered by input failing (for example your 'infile' parameter when called, allocated an object, which gets deallocated & throws an exception inside of its __del__ method). If this is the purpose, then (back to 'how well documented the library is') -- it should be documented this is the purpose of the "errfile" parameter ;-) [A secondary reason you might want to redirect "errfile" is that the passed in input or output file's, themselves do output to standard error ...] On Fri, Sep 29, 2017 at 10:50 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: I am happy to rename the args. What do you think about infile, outfile, and errfile? FWIW, I did consider "in", "out", and "err", but "in" is a keyword, and I didn't think those quite captured the full meaning. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Thursday, September 28, 2017 11:18:18 PM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603<https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue31603&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=sjDmBcI00MbWPPjuLMPlBnZHoFZOHoTxaCo2KCYlEd4&e=>. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc. _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_python-2Dideas&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=dvbtuL0kPe5UxakqosCnQQCIxlKprMP6JTqx4ZLXx4g&e=> Code of Conduct: http://python.org/psf/codeofconduct/<https://urldefense.proofpoint.com/v2/url?u=http-3A__python.org_psf_codeofconduct_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=px67QeYtdnaejiTP9VcY5IuSiJ4pk3XEXcbDzZWbsnI&e=>
So, I was just thinking, maybe we don't want an errfile arg, but an arg that is a sequence of file objects that need to be flushed before showing the prompt. That's certainly more complicated, but it seems more general if we want to cover this case. Thoughts? wt ________________________________ From: Python-ideas <python-ideas-bounces+w00t=fb.com@python.org> on behalf of Wren Turkal <w00t@fb.com> Sent: Friday, September 29, 2017 9:53:01 AM To: Amit Green Cc: python-ideas@python.org Subject: [Potential Spoof] Re: [Python-ideas] allow overriding files used for the input builtin Oh, I thought that stderr was unbuffered. Like the following C program: #include <stdio.h> int main() { fprintf(stdout, "stdout0"); fprintf(stderr, "stderr0"); fprintf(stdout, "stdout1"); return 0; } This outputs: stderr0stdout0stdout1 Turns out fprintf in glibc will also implicitly flush at newlines also. So, the following program: #include <stdio.h> int main() { fprintf(stdout, "stdout0\nstdout12"); fprintf(stderr, "stderr0"); fprintf(stdout, "stdout1"); return 0; } outputs: stdout0 stderr0stdout12stdout1 This is in line with python. The following program: import sys sys.stdout.write('stdout0\nstdout01') sys.stderr.write('stderr0') sys.stdout.write('stdout1') outputs: stdout0 stderr0stdout01stdout1 While the following program: import sys sys.stdout.write('stdout0') sys.stderr.write('stderr0') sys.stdout.write('stdout1') outputs: stderr0stdout0stdout1 So, I guess that the errfile is explicitly flushed due to the fact that stdout may or may not be flushed during output. So, I have 2 questions: (1) do we need to keep the errfile param to make this useful (I think we do), and if #1 is yes, (2) what is an appropriate way to doc this? wt ________________________________ From: Amit Green <amit.mixie@gmail.com> Sent: Friday, September 29, 2017 9:17:39 AM To: Wren Turkal Cc: Steven D'Aprano; python-ideas@python.org Subject: Re: [Python-ideas] allow overriding files used for the input builtin Hmm, very good point. Flushing standard error is essential: * This is because output of standard output & standard error are often redirected to the same file -- frequently the terminal -- and its important to flush one of them before output to the other (since these are typically line buffered you only see this if you output partial lines). * Here is an example of not flushing & the mess it causes:
* Here is the same example with flushing:
sys.stdout.write('hi'); sys.stdout.flush(); sys.stderr.write(' HMM '); sys.stderr.flush(); sys.stdout.write('there\n') hi HMM there
In fact, I think you need to improve your interface and documentation: * If either of the parameters 'outfile', or 'errfile' are passed in & not the 'none' value then the following happens: * Both sys.stdout & sys.stderr are flushed (in that order) before being redirected. (NOTE: Both are flushed, even if only one is redirected). * Before restoring sys.stdout & sys.stderr; then outfile & errfile are flushed (if both have been used then they are flushed in that order). You would of course need to write the documentation clearer than I did above (writing documentation well is not my skill) -- I wrote it to convey exactly what has to happen. On Fri, Sep 29, 2017 at 12:01 PM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Steven and Amit, I originally configured the mailing list for digest delivery and can't reply directly to his message. However, I now seen it, I will update the PR with the suggested name changes as soon as my "make test" finishes. FWIW, I've changed to direct message delivery. With regard to ferr (the corresponding variable in the builtin_input c function before the change), I saw this code: /* First of all, flush stderr */ tmp = _PyObject_CallMethodId(ferr, &PyId_flush, NULL); if (tmp == NULL) PyErr_Clear(); else Py_DECREF(tmp); And I assumed that it was important to be able to override a stderr as a result. I think there are few options here to resolve that: 1. Remove the errfile parameter and just do what happened before. 2. Remove the errfile and the above code (FWIW, I am not sure I understand the importance of flushing stderr before taking input). 3. Document the explicit purpose of the errfile. Amit, you'd responded with something about this (again digest reply, sorry). I am not sure how to concisely do that given Amit's descriptions. ๐ I am honestly leaning toward 2 unless I can figure out why the flushing of stderr is actually needed. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Friday, September 29, 2017 8:13:21 AM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin Yes, infile, outfile & errfile would be consistent with python naming convention (also mentioned by Steven D'Aprano above) One of python's greatest strength is its library, the consistency of the library, and how well documented the library is (in fact, I think the library is a greater strength than even the very nice syntax of python in general). By "consistency of the library" I mean: functions do pretty much what you expect, they use consistent error mechanism & the documentation pretty much accurately documents what the function does -- especially as to showing its results & how it handles errors. Regarding this though, this then brings up the question (above from Steven D'Aprano) -- what would the the "errfile" parameter do? * As a general principle of consistency python library functions, and python itself, do not output to errfile, but instead throw errors. * (There are very minor exceptions such as exceptions thrown in __del__ functions; which are caught by python & then printed to standard error). I would thus think you don't want the errfile parameter -- unless it would be for catching these __del__ method that get triggered by input failing (for example your 'infile' parameter when called, allocated an object, which gets deallocated & throws an exception inside of its __del__ method). If this is the purpose, then (back to 'how well documented the library is') -- it should be documented this is the purpose of the "errfile" parameter ;-) [A secondary reason you might want to redirect "errfile" is that the passed in input or output file's, themselves do output to standard error ...] On Fri, Sep 29, 2017 at 10:50 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: I am happy to rename the args. What do you think about infile, outfile, and errfile? FWIW, I did consider "in", "out", and "err", but "in" is a keyword, and I didn't think those quite captured the full meaning. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Thursday, September 28, 2017 11:18:18 PM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603<https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue31603&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=sjDmBcI00MbWPPjuLMPlBnZHoFZOHoTxaCo2KCYlEd4&e=>. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc. _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_python-2Dideas&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=dvbtuL0kPe5UxakqosCnQQCIxlKprMP6JTqx4ZLXx4g&e=> Code of Conduct: http://python.org/psf/codeofconduct/<https://urldefense.proofpoint.com/v2/url?u=http-3A__python.org_psf_codeofconduct_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=px67QeYtdnaejiTP9VcY5IuSiJ4pk3XEXcbDzZWbsnI&e=>
On Fri, Sep 29, 2017 at 05:53:58AM +0000, Wren Turkal wrote: [...]
I've done the whole "set stdout and stdin, call input, reset stdout and stdin" dance, and so I like this concept of being able to easily use input() non-interactively. I think your basic idea is a good one. I wonder what you think the ferr parameter will do? As far as I know, input() doesn't use stderr. I also don't think much of your parameter names. They strike me as very C-like, rather than Pythonic. So my proposal is: input(prompt, *, infile, outfile) where infile must be a file-like object with a read() method, suitable for replacing stdin, and outfile must be a file-like object with a write() method suitable for replacing stdout.
For production use, that should be wrapped in a try...finally: try: ... finally: sys.stdin = orig_stdin sys.stdout = orig_stdout
I like it very much. But as an alternative, perhaps all we really need is a context manager to set the std* files: with open('/dev/tty', 'r+') as f: with stdio(stdin=f, stdout=f): name = input('Name? ') print(name) That's nearly as nice, and is possibly useful in more situations. Or maybe we should have both? [...]
Definitely interested! -- Steve
On 29 September 2017 at 11:25, Steven D'Aprano <steve@pearwood.info> wrote:
Agreed - a general way of redirecting stdio would be more generally useful - I've often replaced stdio, but as far as I can recall never for input(). There's already contextlib.redirect_stdout() and contextlib.redirect_stderr(). Adding contextlib.redirect_stdin() would be logical, but I think a more flexible contextlib.redirect_stdio(stdin=None, stdout=None, stderr=None) would be better - where None (the default) means "leave this alone".
I'm interested in the general context manager. I don't use input() enough to be particularly interested in a solution specific to that function. Paul
On Fri, Sep 29, 2017 at 01:45:05PM +0300, Serhiy Storchaka wrote:
Because the two-liner doesn't do what input() does. Testing it at the interactive interpreter gives me: py> def myinput(): ... sys.stdout.write("Name? ") ... return sys.stdin.readline() ... py> x = myinput() Steve py> ? py> The output isn't displayed until the input is entered, and then the prompt messes it up. Admittedly this isn't likely to be an issue if you're redirecting to another file, but it demonstrates that your suggested replacement is not equivalent to the feature request. There's no support for arrow keys, even when readline is available: My name^[[D^[[C^[[D
This falls to me in the category "not every two lines of the code should be added as a builtin".
The built-in already exists. This is making it more useful, just like adding file to print made print more useful. -- Steve
29.09.17 14:40, Steven D'Aprano ะฟะธัะต:
Well, I forgot about a flush(). Now this is just a three-liner. fout.write('Name? ') fout.flush() name = fin.readline()
There's no support for arrow keys, even when readline is available:
My name^[[D^[[C^[[D
Did you check that arrow keys are supported with the proposed PR?
I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com> wrote:
I am happy to rename the args. What do you think about infile, outfile, and errfile? FWIW, I did consider "in", "out", and "err", but "in" is a keyword, and I didn't think those quite captured the full meaning. wt ________________________________ From: Amit Green <amit.mixie@gmail.com> Sent: Thursday, September 28, 2017 11:18:18 PM To: Wren Turkal Cc: python-ideas@python.org Subject: Re: [Python-ideas] allow overriding files used for the input builtin I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603<https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue31603&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=sjDmBcI00MbWPPjuLMPlBnZHoFZOHoTxaCo2KCYlEd4&e=>. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc. _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_python-2Dideas&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=dvbtuL0kPe5UxakqosCnQQCIxlKprMP6JTqx4ZLXx4g&e=> Code of Conduct: http://python.org/psf/codeofconduct/<https://urldefense.proofpoint.com/v2/url?u=http-3A__python.org_psf_codeofconduct_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=px67QeYtdnaejiTP9VcY5IuSiJ4pk3XEXcbDzZWbsnI&e=>
Yes, infile, outfile & errfile would be consistent with python naming convention (also mentioned by Steven D'Aprano above) One of python's greatest strength is its library, the consistency of the library, and how well documented the library is (in fact, I think the library is a greater strength than even the very nice syntax of python in general). By "consistency of the library" I mean: functions do pretty much what you expect, they use consistent error mechanism & the documentation pretty much accurately documents what the function does -- especially as to showing its results & how it handles errors. Regarding this though, this then brings up the question (above from Steven D'Aprano) -- what would the the "errfile" parameter do? - As a general principle of consistency python library functions, and python itself, do not output to errfile, but instead throw errors. - (There are very minor exceptions such as exceptions thrown in __del__ functions; which are caught by python & then printed to standard error). I would thus think you don't want the errfile parameter -- unless it would be for catching these __del__ method that get triggered by input failing (for example your 'infile' parameter when called, allocated an object, which gets deallocated & throws an exception inside of its __del__ method). If this is the purpose, then (back to 'how well documented the library is') -- it should be documented this is the purpose of the "errfile" parameter ;-) [A secondary reason you might want to redirect "errfile" is that the passed in input or output file's, themselves do output to standard error ...] On Fri, Sep 29, 2017 at 10:50 AM, Wren Turkal <w00t@fb.com> wrote:
Steven and Amit, I originally configured the mailing list for digest delivery and can't reply directly to his message. However, I now seen it, I will update the PR with the suggested name changes as soon as my "make test" finishes. FWIW, I've changed to direct message delivery. With regard to ferr (the corresponding variable in the builtin_input c function before the change), I saw this code: /* First of all, flush stderr */ tmp = _PyObject_CallMethodId(ferr, &PyId_flush, NULL); if (tmp == NULL) PyErr_Clear(); else Py_DECREF(tmp); And I assumed that it was important to be able to override a stderr as a result. I think there are few options here to resolve that: 1. Remove the errfile parameter and just do what happened before. 2. Remove the errfile and the above code (FWIW, I am not sure I understand the importance of flushing stderr before taking input). 3. Document the explicit purpose of the errfile. Amit, you'd responded with something about this (again digest reply, sorry). I am not sure how to concisely do that given Amit's descriptions. ๐ I am honestly leaning toward 2 unless I can figure out why the flushing of stderr is actually needed. wt ________________________________ From: Amit Green <amit.mixie@gmail.com> Sent: Friday, September 29, 2017 8:13:21 AM To: Wren Turkal Cc: python-ideas@python.org Subject: Re: [Python-ideas] allow overriding files used for the input builtin Yes, infile, outfile & errfile would be consistent with python naming convention (also mentioned by Steven D'Aprano above) One of python's greatest strength is its library, the consistency of the library, and how well documented the library is (in fact, I think the library is a greater strength than even the very nice syntax of python in general). By "consistency of the library" I mean: functions do pretty much what you expect, they use consistent error mechanism & the documentation pretty much accurately documents what the function does -- especially as to showing its results & how it handles errors. Regarding this though, this then brings up the question (above from Steven D'Aprano) -- what would the the "errfile" parameter do? * As a general principle of consistency python library functions, and python itself, do not output to errfile, but instead throw errors. * (There are very minor exceptions such as exceptions thrown in __del__ functions; which are caught by python & then printed to standard error). I would thus think you don't want the errfile parameter -- unless it would be for catching these __del__ method that get triggered by input failing (for example your 'infile' parameter when called, allocated an object, which gets deallocated & throws an exception inside of its __del__ method). If this is the purpose, then (back to 'how well documented the library is') -- it should be documented this is the purpose of the "errfile" parameter ;-) [A secondary reason you might want to redirect "errfile" is that the passed in input or output file's, themselves do output to standard error ...] On Fri, Sep 29, 2017 at 10:50 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: I am happy to rename the args. What do you think about infile, outfile, and errfile? FWIW, I did consider "in", "out", and "err", but "in" is a keyword, and I didn't think those quite captured the full meaning. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Thursday, September 28, 2017 11:18:18 PM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603<https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue31603&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=sjDmBcI00MbWPPjuLMPlBnZHoFZOHoTxaCo2KCYlEd4&e=>. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc. _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_python-2Dideas&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=dvbtuL0kPe5UxakqosCnQQCIxlKprMP6JTqx4ZLXx4g&e=> Code of Conduct: http://python.org/psf/codeofconduct/<https://urldefense.proofpoint.com/v2/url?u=http-3A__python.org_psf_codeofconduct_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=px67QeYtdnaejiTP9VcY5IuSiJ4pk3XEXcbDzZWbsnI&e=>
Hmm, very good point. Flushing standard error is essential: - This is because output of standard output & standard error are often redirected to the same file -- frequently the terminal -- and its important to flush one of them before output to the other (since these are typically line buffered you only see this if you output partial lines). - Here is an example of not flushing & the mess it causes:
- Here is the same example with flushing:
In fact, I think you need to improve your interface and documentation: - If either of the parameters 'outfile', or 'errfile' are passed in & not the 'none' value then the following happens: - Both sys.stdout & sys.stderr are flushed (in that order) before being redirected. (NOTE: Both are flushed, even if only one is redirected). - Before restoring sys.stdout & sys.stderr; then outfile & errfile are flushed (if both have been used then they are flushed in that order). You would of course need to write the documentation clearer than I did above (writing documentation well is not my skill) -- I wrote it to convey exactly what has to happen. On Fri, Sep 29, 2017 at 12:01 PM, Wren Turkal <w00t@fb.com> wrote:
Oh, I thought that stderr was unbuffered. Like the following C program: #include <stdio.h> int main() { fprintf(stdout, "stdout0"); fprintf(stderr, "stderr0"); fprintf(stdout, "stdout1"); return 0; } This outputs: stderr0stdout0stdout1 Turns out fprintf in glibc will also implicitly flush at newlines also. So, the following program: #include <stdio.h> int main() { fprintf(stdout, "stdout0\nstdout12"); fprintf(stderr, "stderr0"); fprintf(stdout, "stdout1"); return 0; } outputs: stdout0 stderr0stdout12stdout1 This is in line with python. The following program: import sys sys.stdout.write('stdout0\nstdout01') sys.stderr.write('stderr0') sys.stdout.write('stdout1') outputs: stdout0 stderr0stdout01stdout1 While the following program: import sys sys.stdout.write('stdout0') sys.stderr.write('stderr0') sys.stdout.write('stdout1') outputs: stderr0stdout0stdout1 So, I guess that the errfile is explicitly flushed due to the fact that stdout may or may not be flushed during output. So, I have 2 questions: (1) do we need to keep the errfile param to make this useful (I think we do), and if #1 is yes, (2) what is an appropriate way to doc this? wt ________________________________ From: Amit Green <amit.mixie@gmail.com> Sent: Friday, September 29, 2017 9:17:39 AM To: Wren Turkal Cc: Steven D'Aprano; python-ideas@python.org Subject: Re: [Python-ideas] allow overriding files used for the input builtin Hmm, very good point. Flushing standard error is essential: * This is because output of standard output & standard error are often redirected to the same file -- frequently the terminal -- and its important to flush one of them before output to the other (since these are typically line buffered you only see this if you output partial lines). * Here is an example of not flushing & the mess it causes:
* Here is the same example with flushing:
sys.stdout.write('hi'); sys.stdout.flush(); sys.stderr.write(' HMM '); sys.stderr.flush(); sys.stdout.write('there\n') hi HMM there
In fact, I think you need to improve your interface and documentation: * If either of the parameters 'outfile', or 'errfile' are passed in & not the 'none' value then the following happens: * Both sys.stdout & sys.stderr are flushed (in that order) before being redirected. (NOTE: Both are flushed, even if only one is redirected). * Before restoring sys.stdout & sys.stderr; then outfile & errfile are flushed (if both have been used then they are flushed in that order). You would of course need to write the documentation clearer than I did above (writing documentation well is not my skill) -- I wrote it to convey exactly what has to happen. On Fri, Sep 29, 2017 at 12:01 PM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Steven and Amit, I originally configured the mailing list for digest delivery and can't reply directly to his message. However, I now seen it, I will update the PR with the suggested name changes as soon as my "make test" finishes. FWIW, I've changed to direct message delivery. With regard to ferr (the corresponding variable in the builtin_input c function before the change), I saw this code: /* First of all, flush stderr */ tmp = _PyObject_CallMethodId(ferr, &PyId_flush, NULL); if (tmp == NULL) PyErr_Clear(); else Py_DECREF(tmp); And I assumed that it was important to be able to override a stderr as a result. I think there are few options here to resolve that: 1. Remove the errfile parameter and just do what happened before. 2. Remove the errfile and the above code (FWIW, I am not sure I understand the importance of flushing stderr before taking input). 3. Document the explicit purpose of the errfile. Amit, you'd responded with something about this (again digest reply, sorry). I am not sure how to concisely do that given Amit's descriptions. ๐ I am honestly leaning toward 2 unless I can figure out why the flushing of stderr is actually needed. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Friday, September 29, 2017 8:13:21 AM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin Yes, infile, outfile & errfile would be consistent with python naming convention (also mentioned by Steven D'Aprano above) One of python's greatest strength is its library, the consistency of the library, and how well documented the library is (in fact, I think the library is a greater strength than even the very nice syntax of python in general). By "consistency of the library" I mean: functions do pretty much what you expect, they use consistent error mechanism & the documentation pretty much accurately documents what the function does -- especially as to showing its results & how it handles errors. Regarding this though, this then brings up the question (above from Steven D'Aprano) -- what would the the "errfile" parameter do? * As a general principle of consistency python library functions, and python itself, do not output to errfile, but instead throw errors. * (There are very minor exceptions such as exceptions thrown in __del__ functions; which are caught by python & then printed to standard error). I would thus think you don't want the errfile parameter -- unless it would be for catching these __del__ method that get triggered by input failing (for example your 'infile' parameter when called, allocated an object, which gets deallocated & throws an exception inside of its __del__ method). If this is the purpose, then (back to 'how well documented the library is') -- it should be documented this is the purpose of the "errfile" parameter ;-) [A secondary reason you might want to redirect "errfile" is that the passed in input or output file's, themselves do output to standard error ...] On Fri, Sep 29, 2017 at 10:50 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: I am happy to rename the args. What do you think about infile, outfile, and errfile? FWIW, I did consider "in", "out", and "err", but "in" is a keyword, and I didn't think those quite captured the full meaning. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Thursday, September 28, 2017 11:18:18 PM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603<https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue31603&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=sjDmBcI00MbWPPjuLMPlBnZHoFZOHoTxaCo2KCYlEd4&e=>. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc. _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_python-2Dideas&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=dvbtuL0kPe5UxakqosCnQQCIxlKprMP6JTqx4ZLXx4g&e=> Code of Conduct: http://python.org/psf/codeofconduct/<https://urldefense.proofpoint.com/v2/url?u=http-3A__python.org_psf_codeofconduct_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=px67QeYtdnaejiTP9VcY5IuSiJ4pk3XEXcbDzZWbsnI&e=>
So, I was just thinking, maybe we don't want an errfile arg, but an arg that is a sequence of file objects that need to be flushed before showing the prompt. That's certainly more complicated, but it seems more general if we want to cover this case. Thoughts? wt ________________________________ From: Python-ideas <python-ideas-bounces+w00t=fb.com@python.org> on behalf of Wren Turkal <w00t@fb.com> Sent: Friday, September 29, 2017 9:53:01 AM To: Amit Green Cc: python-ideas@python.org Subject: [Potential Spoof] Re: [Python-ideas] allow overriding files used for the input builtin Oh, I thought that stderr was unbuffered. Like the following C program: #include <stdio.h> int main() { fprintf(stdout, "stdout0"); fprintf(stderr, "stderr0"); fprintf(stdout, "stdout1"); return 0; } This outputs: stderr0stdout0stdout1 Turns out fprintf in glibc will also implicitly flush at newlines also. So, the following program: #include <stdio.h> int main() { fprintf(stdout, "stdout0\nstdout12"); fprintf(stderr, "stderr0"); fprintf(stdout, "stdout1"); return 0; } outputs: stdout0 stderr0stdout12stdout1 This is in line with python. The following program: import sys sys.stdout.write('stdout0\nstdout01') sys.stderr.write('stderr0') sys.stdout.write('stdout1') outputs: stdout0 stderr0stdout01stdout1 While the following program: import sys sys.stdout.write('stdout0') sys.stderr.write('stderr0') sys.stdout.write('stdout1') outputs: stderr0stdout0stdout1 So, I guess that the errfile is explicitly flushed due to the fact that stdout may or may not be flushed during output. So, I have 2 questions: (1) do we need to keep the errfile param to make this useful (I think we do), and if #1 is yes, (2) what is an appropriate way to doc this? wt ________________________________ From: Amit Green <amit.mixie@gmail.com> Sent: Friday, September 29, 2017 9:17:39 AM To: Wren Turkal Cc: Steven D'Aprano; python-ideas@python.org Subject: Re: [Python-ideas] allow overriding files used for the input builtin Hmm, very good point. Flushing standard error is essential: * This is because output of standard output & standard error are often redirected to the same file -- frequently the terminal -- and its important to flush one of them before output to the other (since these are typically line buffered you only see this if you output partial lines). * Here is an example of not flushing & the mess it causes:
* Here is the same example with flushing:
sys.stdout.write('hi'); sys.stdout.flush(); sys.stderr.write(' HMM '); sys.stderr.flush(); sys.stdout.write('there\n') hi HMM there
In fact, I think you need to improve your interface and documentation: * If either of the parameters 'outfile', or 'errfile' are passed in & not the 'none' value then the following happens: * Both sys.stdout & sys.stderr are flushed (in that order) before being redirected. (NOTE: Both are flushed, even if only one is redirected). * Before restoring sys.stdout & sys.stderr; then outfile & errfile are flushed (if both have been used then they are flushed in that order). You would of course need to write the documentation clearer than I did above (writing documentation well is not my skill) -- I wrote it to convey exactly what has to happen. On Fri, Sep 29, 2017 at 12:01 PM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Steven and Amit, I originally configured the mailing list for digest delivery and can't reply directly to his message. However, I now seen it, I will update the PR with the suggested name changes as soon as my "make test" finishes. FWIW, I've changed to direct message delivery. With regard to ferr (the corresponding variable in the builtin_input c function before the change), I saw this code: /* First of all, flush stderr */ tmp = _PyObject_CallMethodId(ferr, &PyId_flush, NULL); if (tmp == NULL) PyErr_Clear(); else Py_DECREF(tmp); And I assumed that it was important to be able to override a stderr as a result. I think there are few options here to resolve that: 1. Remove the errfile parameter and just do what happened before. 2. Remove the errfile and the above code (FWIW, I am not sure I understand the importance of flushing stderr before taking input). 3. Document the explicit purpose of the errfile. Amit, you'd responded with something about this (again digest reply, sorry). I am not sure how to concisely do that given Amit's descriptions. ๐ I am honestly leaning toward 2 unless I can figure out why the flushing of stderr is actually needed. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Friday, September 29, 2017 8:13:21 AM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin Yes, infile, outfile & errfile would be consistent with python naming convention (also mentioned by Steven D'Aprano above) One of python's greatest strength is its library, the consistency of the library, and how well documented the library is (in fact, I think the library is a greater strength than even the very nice syntax of python in general). By "consistency of the library" I mean: functions do pretty much what you expect, they use consistent error mechanism & the documentation pretty much accurately documents what the function does -- especially as to showing its results & how it handles errors. Regarding this though, this then brings up the question (above from Steven D'Aprano) -- what would the the "errfile" parameter do? * As a general principle of consistency python library functions, and python itself, do not output to errfile, but instead throw errors. * (There are very minor exceptions such as exceptions thrown in __del__ functions; which are caught by python & then printed to standard error). I would thus think you don't want the errfile parameter -- unless it would be for catching these __del__ method that get triggered by input failing (for example your 'infile' parameter when called, allocated an object, which gets deallocated & throws an exception inside of its __del__ method). If this is the purpose, then (back to 'how well documented the library is') -- it should be documented this is the purpose of the "errfile" parameter ;-) [A secondary reason you might want to redirect "errfile" is that the passed in input or output file's, themselves do output to standard error ...] On Fri, Sep 29, 2017 at 10:50 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: I am happy to rename the args. What do you think about infile, outfile, and errfile? FWIW, I did consider "in", "out", and "err", but "in" is a keyword, and I didn't think those quite captured the full meaning. wt ________________________________ From: Amit Green <amit.mixie@gmail.com<mailto:amit.mixie@gmail.com>> Sent: Thursday, September 28, 2017 11:18:18 PM To: Wren Turkal Cc: python-ideas@python.org<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] allow overriding files used for the input builtin I'm fine with the idea in general of extra keyword parameters to the input function. A few points: Your example code, needs try/catch to match what the input with parameters does -- and yes, its way nicer to be able to use it the example you have shown than play games with try/catch (Personally I also refuse to ever change sys.stdin, or sys.stdout, as I consider that a bad coding style). Mostly though I would like to ask, please do not name keyword arguments with names like 'fin' & 'fout'. This is almost unreadable and make's code almost indecipherable to others the first time they see the function & its keyword arguments (First impressions are very important). Both a function name & its keyword parameters need to be as understandable as possible when a user encounters them for the first time. On Fri, Sep 29, 2017 at 1:53 AM, Wren Turkal <w00t@fb.com<mailto:w00t@fb.com>> wrote: Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603<https://urldefense.proofpoint.com/v2/url?u=https-3A__bugs.python.org_issue31603&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=sjDmBcI00MbWPPjuLMPlBnZHoFZOHoTxaCo2KCYlEd4&e=>. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc. _______________________________________________ Python-ideas mailing list Python-ideas@python.org<mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas<https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_python-2Dideas&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=dvbtuL0kPe5UxakqosCnQQCIxlKprMP6JTqx4ZLXx4g&e=> Code of Conduct: http://python.org/psf/codeofconduct/<https://urldefense.proofpoint.com/v2/url?u=http-3A__python.org_psf_codeofconduct_&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=OAN5uLR4JWXbIgcvx315Zw&m=K1-OD0dhslOiAqYQdEyr0Oppl8TaroBPvXr8h_Z8XxM&s=px67QeYtdnaejiTP9VcY5IuSiJ4pk3XEXcbDzZWbsnI&e=>
On Fri, Sep 29, 2017 at 05:53:58AM +0000, Wren Turkal wrote: [...]
I've done the whole "set stdout and stdin, call input, reset stdout and stdin" dance, and so I like this concept of being able to easily use input() non-interactively. I think your basic idea is a good one. I wonder what you think the ferr parameter will do? As far as I know, input() doesn't use stderr. I also don't think much of your parameter names. They strike me as very C-like, rather than Pythonic. So my proposal is: input(prompt, *, infile, outfile) where infile must be a file-like object with a read() method, suitable for replacing stdin, and outfile must be a file-like object with a write() method suitable for replacing stdout.
For production use, that should be wrapped in a try...finally: try: ... finally: sys.stdin = orig_stdin sys.stdout = orig_stdout
I like it very much. But as an alternative, perhaps all we really need is a context manager to set the std* files: with open('/dev/tty', 'r+') as f: with stdio(stdin=f, stdout=f): name = input('Name? ') print(name) That's nearly as nice, and is possibly useful in more situations. Or maybe we should have both? [...]
Definitely interested! -- Steve
On 29 September 2017 at 11:25, Steven D'Aprano <steve@pearwood.info> wrote:
Agreed - a general way of redirecting stdio would be more generally useful - I've often replaced stdio, but as far as I can recall never for input(). There's already contextlib.redirect_stdout() and contextlib.redirect_stderr(). Adding contextlib.redirect_stdin() would be logical, but I think a more flexible contextlib.redirect_stdio(stdin=None, stdout=None, stderr=None) would be better - where None (the default) means "leave this alone".
I'm interested in the general context manager. I don't use input() enough to be particularly interested in a solution specific to that function. Paul
On Fri, Sep 29, 2017 at 01:45:05PM +0300, Serhiy Storchaka wrote:
Because the two-liner doesn't do what input() does. Testing it at the interactive interpreter gives me: py> def myinput(): ... sys.stdout.write("Name? ") ... return sys.stdin.readline() ... py> x = myinput() Steve py> ? py> The output isn't displayed until the input is entered, and then the prompt messes it up. Admittedly this isn't likely to be an issue if you're redirecting to another file, but it demonstrates that your suggested replacement is not equivalent to the feature request. There's no support for arrow keys, even when readline is available: My name^[[D^[[C^[[D
This falls to me in the category "not every two lines of the code should be added as a builtin".
The built-in already exists. This is making it more useful, just like adding file to print made print more useful. -- Steve
29.09.17 14:40, Steven D'Aprano ะฟะธัะต:
Well, I forgot about a flush(). Now this is just a three-liner. fout.write('Name? ') fout.flush() name = fin.readline()
There's no support for arrow keys, even when readline is available:
My name^[[D^[[C^[[D
Did you check that arrow keys are supported with the proposed PR?
participants (5)
-
Amit Green -
Paul Moore -
Serhiy Storchaka -
Steven D'Aprano -
Wren Turkal