Backwards incompatible sys.stdout.write() behavior in Python 3 (Was: [Python-ideas] Pythonic buffering in Py3 print())
Posting to python-dev as it is no more relates to the idea of improving print(). sys.stdout.write() in Python 3 causes backwards incompatible behavior that breaks recipe for unbuffered character reading from stdin on Linux - http://code.activestate.com/recipes/134892/ At first I though that the problem is in the new print() function, but it appeared that the culprit is sys.stdout.write() Attached is a test script which is a stripped down version of the recipe above. If executed with Python 2, you can see the prompt to press a key (even though output on Linux is buffered in Python 2). With Python 3, there is not prompt until you press a key. Is it a bug or intended behavior? What is the cause of this break? -- anatoly t.
I think this may be because in Python 2, there is a coupling between stdin and stderr (in the C stdlib code) that flushes stdout when you read stdin. This doesn't seem to be required by the C std, but most implementations seem to do it. http://stackoverflow.com/questions/2123528/does-reading-from-stdin-flush-std... I think it was a nice feature but I can see problems with it; apps that want this behavior ought to bite the bullet and flush stdout. On Fri, Jan 13, 2012 at 7:34 AM, anatoly techtonik <techtonik@gmail.com>wrote:
-- --Guido van Rossum (python.org/~guido)
On 2012-01-13, at 16:34 , anatoly techtonik wrote: the change with 3 lines import sys sys.stdout.write('promt>') sys.stdin.read(1) Python 2 displays "prompt" and terminates execution on [Return], Python 3 does not display anything until [Return] is pressed. Interestingly, the `-u` option is not sufficient to make "prompt>" appear in Python 3, the stream has to be flushed explicitly unless the input is ~16k characters (I guess that's an internal buffer size of some sort)
On Fri, 13 Jan 2012 17:00:57 +0100 Xavier Morel <python-dev@masklinn.net> wrote:
"-u" forces line-buffering mode for stdout/stderr, which is already the default if they are wired to an interactive device (isattr() returning True). But this was already rehashed on python-ideas and the bug tracker, and apparently Anatoly thought it would be a good idea to post on a third medium. Sigh. Regards Antoine.
On 2012-01-13, at 17:19 , Antoine Pitrou wrote:
Oh, I had not noticed the documentation had changed in Python 3 (in Python 2 it stated that `-u` made IO unbuffered, on Python 3 it now states that only binary IO is unbuffered and text IO remains line-buffered). Sorry about that.
On Fri, Jan 13, 2012 at 7:19 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
If you track this more closely, you'll notice there are four issues (surprises) from the user point of view: 1. print() buffers output on Python3 2. print() also buffers output on Python2, but only on Linux 3. there is some useless '-u' command line parameter (useless, because the last thing user wants is not only care about Python 2/3, but also how to invoke them) 4. print() is not guilty - it is sys.stdout.write() that buffers output 1-2 discussion was about idea to make new print() function behavior more 'pythonic', i.e. 'user-friendly' or just KISS, which resulted in adding a flush parameter 3 is a just a side FYI remark 4 doesn't relate to python-ideas anymore about fixing print() - it is about the *cause* of the problem with print() UX, which is underlying sys.stdout.write() behavior I asked 4 here, because it is the more appropriate place not only to ask if it can be/will be fixed, but also why. The target audience of the question are developers. Hope that helps Antoine recover from the sorrow. ;) -- anatoly t.
On 17 Jan, 2012, at 11:59, anatoly techtonik wrote:
All four "issues" are related to output buffering and how that is not user-friendly. The new issue you raise is the same as before: sys.stdout is line buffered when writing to a tty, which means that you have to explictly flush output when you want to output a partial line. Why is this a problem for you? Is that something that bothers you personally or do you have data that suggests that this is a problem for a significant amount of (new) users? Ronald
On 1/17/2012 5:59 AM, anatoly techtonik wrote:
1. print() buffers output on Python3 2. print() also buffers output on Python2, but only on Linux
No, print() does not buffer output. It merely sends it to a file.
4. print() is not guilty - it is sys.stdout.write() that buffers output
Oh, you already know that 1&2 are false. So is 4, if interpreted as saying that sys.stdout.write() *will* buffer output. sys.stdout can be *any* file-like object. Its .write method *may* buffer output, or it *may not*. With IDLE, it does not. We have been over this before. At your instigation, the doc has been changed to make this clearer. At your request, a new feature has been added to force flushing. By most people's standards, you won. -- Terry Jan Reedy
I think this may be because in Python 2, there is a coupling between stdin and stderr (in the C stdlib code) that flushes stdout when you read stdin. This doesn't seem to be required by the C std, but most implementations seem to do it. http://stackoverflow.com/questions/2123528/does-reading-from-stdin-flush-std... I think it was a nice feature but I can see problems with it; apps that want this behavior ought to bite the bullet and flush stdout. On Fri, Jan 13, 2012 at 7:34 AM, anatoly techtonik <techtonik@gmail.com>wrote:
-- --Guido van Rossum (python.org/~guido)
On 2012-01-13, at 16:34 , anatoly techtonik wrote: the change with 3 lines import sys sys.stdout.write('promt>') sys.stdin.read(1) Python 2 displays "prompt" and terminates execution on [Return], Python 3 does not display anything until [Return] is pressed. Interestingly, the `-u` option is not sufficient to make "prompt>" appear in Python 3, the stream has to be flushed explicitly unless the input is ~16k characters (I guess that's an internal buffer size of some sort)
On Fri, 13 Jan 2012 17:00:57 +0100 Xavier Morel <python-dev@masklinn.net> wrote:
"-u" forces line-buffering mode for stdout/stderr, which is already the default if they are wired to an interactive device (isattr() returning True). But this was already rehashed on python-ideas and the bug tracker, and apparently Anatoly thought it would be a good idea to post on a third medium. Sigh. Regards Antoine.
On 2012-01-13, at 17:19 , Antoine Pitrou wrote:
Oh, I had not noticed the documentation had changed in Python 3 (in Python 2 it stated that `-u` made IO unbuffered, on Python 3 it now states that only binary IO is unbuffered and text IO remains line-buffered). Sorry about that.
On Fri, Jan 13, 2012 at 7:19 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
If you track this more closely, you'll notice there are four issues (surprises) from the user point of view: 1. print() buffers output on Python3 2. print() also buffers output on Python2, but only on Linux 3. there is some useless '-u' command line parameter (useless, because the last thing user wants is not only care about Python 2/3, but also how to invoke them) 4. print() is not guilty - it is sys.stdout.write() that buffers output 1-2 discussion was about idea to make new print() function behavior more 'pythonic', i.e. 'user-friendly' or just KISS, which resulted in adding a flush parameter 3 is a just a side FYI remark 4 doesn't relate to python-ideas anymore about fixing print() - it is about the *cause* of the problem with print() UX, which is underlying sys.stdout.write() behavior I asked 4 here, because it is the more appropriate place not only to ask if it can be/will be fixed, but also why. The target audience of the question are developers. Hope that helps Antoine recover from the sorrow. ;) -- anatoly t.
On 17 Jan, 2012, at 11:59, anatoly techtonik wrote:
All four "issues" are related to output buffering and how that is not user-friendly. The new issue you raise is the same as before: sys.stdout is line buffered when writing to a tty, which means that you have to explictly flush output when you want to output a partial line. Why is this a problem for you? Is that something that bothers you personally or do you have data that suggests that this is a problem for a significant amount of (new) users? Ronald
On 1/17/2012 5:59 AM, anatoly techtonik wrote:
1. print() buffers output on Python3 2. print() also buffers output on Python2, but only on Linux
No, print() does not buffer output. It merely sends it to a file.
4. print() is not guilty - it is sys.stdout.write() that buffers output
Oh, you already know that 1&2 are false. So is 4, if interpreted as saying that sys.stdout.write() *will* buffer output. sys.stdout can be *any* file-like object. Its .write method *may* buffer output, or it *may not*. With IDLE, it does not. We have been over this before. At your instigation, the doc has been changed to make this clearer. At your request, a new feature has been added to force flushing. By most people's standards, you won. -- Terry Jan Reedy
participants (6)
-
anatoly techtonik -
Antoine Pitrou -
Guido van Rossum -
Ronald Oussoren -
Terry Reedy -
Xavier Morel