In Python 2 "print 'something', statement calls were unbuffered and immediately appeared on screen. In Python 3 "print('something', end='')" calls are buffered. This breaks progress bars and other stuff. 1. What is more "Pythonic" and why? 2. Should Python 3 be fixed ASAP? -- anatoly t.
anatoly techtonik, 09.01.2012 11:30:
I find the Py3 behaviour more pythonic: flushing should be explicit (which is better than implicit). The mere fact that the user did *not* provide a terminator shows that there are likely other things to follow. A progress bar is just one specific case where flushing is actually desired. In other cases, it may not be desired and would rather lead to performance issues. Making it implicit if a flush happens or not prevents users from controlling it.
2. Should Python 3 be fixed ASAP?
IMHO, no. Stefan
On Mon, Jan 9, 2012 at 1:51 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Do you find pythonic that `print` statement should output string to the screen (according to docs), but it doesn't do this? Do you find ability to stack chars into screen buffer before the output (instead of doing stacking to a string) more important than hiding the low level output implementation details from the users? A progress
Can you give some examples of these 'other cases'? Maybe 95% of users are using print for progress bars, and enabling flushing by default will make it more pythonic by making code more beautiful (which is better than ugly).
anatoly techtonik, 09.01.2012 13:16:
Well, it does it if you tell it to. Lacking an explicit request, the rest is platform specific, as usual with multi-level output buffering. Actually, print() is often the first place where new users get in touch with I/O "specialties" such as flush() and buffering/queuing, and it's helpful to learn about them. Not only in the context of terminal I/O but also for disk or network I/O.
All I'm saying is that there is no "right" way to do it, so choosing to flush implicitly will a) break user code and b) keep some (or many?) users from getting the behaviour they want. And that without good reason and without making it "better" or "more correct" through such a change.
... or maybe not ...
and enabling flushing by default will make it more pythonic by making code more beautiful (which is better than ugly).
print() is just one way of pushing output "somewhere", and often not the best one. It's helpful for debugging and mostly acceptable in short scripts, but it's otherwise useless for most applications. That being said, I certainly don't claim to have an idea of what "most" users use print() for and what behaviour "most" users expect... Stefan
In Python 2 "print 'something', statement calls were unbuffered and immediately appeared on screen.
No, they weren't. Python doesn't do any extra buffering or flushing. Usually your terminal emulator line-buffers stdout. If you don't print a newline (in Python 2 OR 3) then it doesn't show up until you either flush (sys.stdout.flush()) or send a newline. -- Devin On Mon, Jan 9, 2012 at 5:30 AM, anatoly techtonik <techtonik@gmail.com> wrote:
On Mon, Jan 9, 2012 at 1:53 PM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
--- cut py2print.py --- from time import sleep while 1: sleep(1) print ".", --- cut --- This produces one dot every second with Python 2.7.2 on Windows. Python doesn't do any extra buffering or flushing. Usually your
I assume you use Linux. Do you agree that for cross-platform language some behavior should be made consistent?
On 9 January 2012 12:21, anatoly techtonik <techtonik@gmail.com> wrote:
No. (At least not for 2.x). The Python 2.x position has always been that the IO layer is a relatively thin wrapper over the platform behaviour, and so this sort of platform dependent behaviour should not come as a surprise to anyone. For Python 3.x, the IO layer was reworked to be more platform-neutral, and I would expect consistent behaviour across platforms there. And the behaviour I would expect (given that stdout is a buffered text stream) is that output from print would be buffered until an implicit or explicit flush. And that is exactly what happens. So I'd say that you're 100% right in what you say as regards Python 3, and that is exactly what is happening. The difference with Python 2 is precisely one of those backward-incompatible breaks that required the 2->3 major version change. Characterising a carefully thought out change implemented in a way that minimises backward compatibility issues as a "bug", is at best unfair and dismissive of the work that was put into this. Paul.
This produces one dot every second with Python 2.7.2 on Windows.
Weird. Must be a windows thing.
I assume you use Linux. Do you agree that for cross-platform language some behavior should be made consistent?
Yes. I guess that's why they changed the behavior of print on Windows in Python 3. -- Devin On Mon, Jan 9, 2012 at 7:21 AM, anatoly techtonik <techtonik@gmail.com> wrote:
Read also the discussion on the Python bug tracker: http://bugs.python.org/issue11633 Victor
This has nothing to do with print or print(). There are no flush() calls implied by print/print() calls, nor should there be(*). print/print() simply calls sys.stdout.write() one or more times, the final call usually being sys.stdout.write('\n') which will trigger a flush() if the file is line-buffered -- but if you use a trailing comma (Python 2) or end=... (Python 3) the last thing written is not '\n' and no flush() will be triggered. Whether writing \n flushes or not depends on the platform, the Python version and implementation, and the environment (e.g. most platforms automatically turn on line buffering for files known to be associated with a terminal emulator or other interactive device, assuming there's an easy way to tell). __________ (*) There are plenty of apps that construct their output using many print/print() calls, and for which flushing after each call would be quite unnecessary and potentially cause lots of redundant I/O operations, ultimately slowing down the app. Apps that need flushing should call flush(). -- --Guido van Rossum (python.org/~guido)
On 1/9/2012 12:52 PM, Guido van Rossum wrote:
Thank you for the clarification. I noted on http://bugs.python.org/issue11633 that there should be no code change
Apps that need flushing should call flush().
and that saying something like this in the print doc entry is the only possible change. -- Terry Jan Reedy
I don't think it would be straight forward or reasonable to _prevent_ flushing from occurring. Forcing a flush, is only useful when the file object to which you're printing isn't already handy, and this only occurs for sys.stdout. It's not worth adding a "flush-in-case" flag just for this one stream, especially if it's unable to prevent flushing. On Tue, Jan 10, 2012 at 8:05 PM, INADA Naoki <songofacandy@gmail.com> wrote:
On Tue, Jan 10, 2012 at 8:15 PM, Masklinn <masklinn@masklinn.net> wrote:
+0 for being able to write print("whatever", force_flush=True) instead of having to do: import sys # somewhere in the file print("whatever") sys.stdout.flush() Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 10, 2012 at 6:36 PM, MRAB <python@mrabarnett.plus.com> wrote:
Heh, that's exactly why we made it a function. So is adding e.g. a force_flush flag -- though personally I would be fine calling it "flush" despite the possibility that the underlying stream might still flush when it is not explicitly requests. -- --Guido van Rossum (python.org/~guido)
On 09Jan2012 22:49, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote: | On Jan 9, 2012, at 7:52 PM, INADA Naoki wrote: | > What about adding flush=False keyword argument to print()? | | I would expect flush=False to mean "please don't flush this right now, | even if I do use a newline." | | flush=None? +1 from me, if None means "honour the file's default flushing behaviour". -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Since I've mentioned the subject of geneology, I'll repeat a story I heard about a poor fellow over on airstrip one. Seems he spent the most recent thirty years of his life tracking down his family history. Spent hundreds of pounds, traveled, devoted his life to it. Then, last month, a cousin told him he was adopted. Ahhh, sweet irony. - Tim_Mefford <tim@physics.orst.edu>
I see that Python 3 behavior is consistent on Windows too meaning it doesn't output anything on the screen when end='' It seems that basic function print() that was created to just "print a string" got over-engineered breaking unix principle of "doing one thing". It reminds me of subprocess.Popen which was also designed to just "run a process". I would keep it as simple as possible for printing stuff to the screen without any implicit buffering. You called print() - you get the output. KISS. If people need more control with output buffering - provide a pointer to sys.stdout with all gory details about streams etc. To get back on topic, let me give a definition. To me "pythonic" means "intuitive" in the first place. And "intuitive" means "up to user expectations". What does user expect from "print()" function? Writing to a file? I don't think so. Output to printer? Maybe. Printing to the screen? Definitely. "intuitive" also means that users won't have to consult user documentation for every basic thing it needs, especially for such basic thing such as getting some stuff printed to the screen output - the stuff that is essential in the coding process. It is already less convenient in Py3k, because you need to type extra Shift-9 and Shift-0 in the process, and it risks to become completely weird: print('calling external process... ', end='') sys.output.flush() ... print('done') So, py3k print() is not pythonic. To make it pythonic, the output should be immediately available after you call it. If you need to buffer output - do this explicitly, because print() doesn't guarantee that it will be buffered anyway. If you care about performance - use sys.stdout directly and tune it explicitly. On Mon, Jan 9, 2012 at 4:25 PM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
Am 10.01.2012 22:36, schrieb Guido van Rossum:
I've created an issue to track this: http://bugs.python.org/issue13761 Georg
There is already an option for flush: -u command line option. Can't we change print() to flush if end is set and -u option is used, instead of introducing a new option? I understand that a command line option is less practical than a new keyword, especially on Windows.
Victor Stinner, 11.01.2012 09:46:
For one, it doesn't allow the program code to explicitly enforce the flush at a specific point if it depends on a command line switch. Note that the caller of print() may not even know what "print" really is at a given point in the execution, and what (if any) output stream it will write to. Doesn't have to be sys.stdout. So a new keyword argument makes sense to me (and I agree that calling it "flush" should be enough). Stefan
On Jan 10, 11:27 pm, anatoly techtonik <techto...@gmail.com> wrote:
No, what "pythonic" means here is "up to anatoly's expectations". If I could kill one idea dead about Python it's this persistent belief that you shouldn't need to read any documentation in order to be able to use it.
"intuitive" also means that users won't have to consult user documentation for every basic thing it needs
Well, clearly that's not true, as your intuition about print's behaviour was wrong. Other people's "intuitive" understanding of mutable default arguments gets them into trouble; you can either "fix" this behaviour so it's inconsistent with the rest of Python's object model, or you can recommend they read the docs so their understanding changes and they know what it means. The latter is the only sane method, IMO. Similarly, every time your intuition is wrong doesn't make it a regression or bug.
Em Mon, 9 Jan 2012 13:30:55 +0300 anatoly techtonik <techtonik@gmail.com> escreveu:
In Python 2 "print 'something', statement calls were unbuffered and immediately appeared on screen.
Only if Python is called with -u command line option, from the man page: -u Force stdin, stdout and stderr to be totally unbuffered. Regards, -- .:''''':. .:' ` Sérgio Surkamp | Administrador de Redes :: ........ sergio@gruposinternet.com.br `:. .:' `:, ,.:' *Grupos Internet S.A.* `: :' R. Lauro Linhares, 2123 Torre B - Sala 201 : : Trindade - Florianópolis - SC :.' :: +55 48 3234-4109 : ' http://www.gruposinternet.com.br
anatoly techtonik, 09.01.2012 11:30:
I find the Py3 behaviour more pythonic: flushing should be explicit (which is better than implicit). The mere fact that the user did *not* provide a terminator shows that there are likely other things to follow. A progress bar is just one specific case where flushing is actually desired. In other cases, it may not be desired and would rather lead to performance issues. Making it implicit if a flush happens or not prevents users from controlling it.
2. Should Python 3 be fixed ASAP?
IMHO, no. Stefan
On Mon, Jan 9, 2012 at 1:51 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Do you find pythonic that `print` statement should output string to the screen (according to docs), but it doesn't do this? Do you find ability to stack chars into screen buffer before the output (instead of doing stacking to a string) more important than hiding the low level output implementation details from the users? A progress
Can you give some examples of these 'other cases'? Maybe 95% of users are using print for progress bars, and enabling flushing by default will make it more pythonic by making code more beautiful (which is better than ugly).
anatoly techtonik, 09.01.2012 13:16:
Well, it does it if you tell it to. Lacking an explicit request, the rest is platform specific, as usual with multi-level output buffering. Actually, print() is often the first place where new users get in touch with I/O "specialties" such as flush() and buffering/queuing, and it's helpful to learn about them. Not only in the context of terminal I/O but also for disk or network I/O.
All I'm saying is that there is no "right" way to do it, so choosing to flush implicitly will a) break user code and b) keep some (or many?) users from getting the behaviour they want. And that without good reason and without making it "better" or "more correct" through such a change.
... or maybe not ...
and enabling flushing by default will make it more pythonic by making code more beautiful (which is better than ugly).
print() is just one way of pushing output "somewhere", and often not the best one. It's helpful for debugging and mostly acceptable in short scripts, but it's otherwise useless for most applications. That being said, I certainly don't claim to have an idea of what "most" users use print() for and what behaviour "most" users expect... Stefan
In Python 2 "print 'something', statement calls were unbuffered and immediately appeared on screen.
No, they weren't. Python doesn't do any extra buffering or flushing. Usually your terminal emulator line-buffers stdout. If you don't print a newline (in Python 2 OR 3) then it doesn't show up until you either flush (sys.stdout.flush()) or send a newline. -- Devin On Mon, Jan 9, 2012 at 5:30 AM, anatoly techtonik <techtonik@gmail.com> wrote:
On Mon, Jan 9, 2012 at 1:53 PM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
--- cut py2print.py --- from time import sleep while 1: sleep(1) print ".", --- cut --- This produces one dot every second with Python 2.7.2 on Windows. Python doesn't do any extra buffering or flushing. Usually your
I assume you use Linux. Do you agree that for cross-platform language some behavior should be made consistent?
On 9 January 2012 12:21, anatoly techtonik <techtonik@gmail.com> wrote:
No. (At least not for 2.x). The Python 2.x position has always been that the IO layer is a relatively thin wrapper over the platform behaviour, and so this sort of platform dependent behaviour should not come as a surprise to anyone. For Python 3.x, the IO layer was reworked to be more platform-neutral, and I would expect consistent behaviour across platforms there. And the behaviour I would expect (given that stdout is a buffered text stream) is that output from print would be buffered until an implicit or explicit flush. And that is exactly what happens. So I'd say that you're 100% right in what you say as regards Python 3, and that is exactly what is happening. The difference with Python 2 is precisely one of those backward-incompatible breaks that required the 2->3 major version change. Characterising a carefully thought out change implemented in a way that minimises backward compatibility issues as a "bug", is at best unfair and dismissive of the work that was put into this. Paul.
This produces one dot every second with Python 2.7.2 on Windows.
Weird. Must be a windows thing.
I assume you use Linux. Do you agree that for cross-platform language some behavior should be made consistent?
Yes. I guess that's why they changed the behavior of print on Windows in Python 3. -- Devin On Mon, Jan 9, 2012 at 7:21 AM, anatoly techtonik <techtonik@gmail.com> wrote:
Read also the discussion on the Python bug tracker: http://bugs.python.org/issue11633 Victor
This has nothing to do with print or print(). There are no flush() calls implied by print/print() calls, nor should there be(*). print/print() simply calls sys.stdout.write() one or more times, the final call usually being sys.stdout.write('\n') which will trigger a flush() if the file is line-buffered -- but if you use a trailing comma (Python 2) or end=... (Python 3) the last thing written is not '\n' and no flush() will be triggered. Whether writing \n flushes or not depends on the platform, the Python version and implementation, and the environment (e.g. most platforms automatically turn on line buffering for files known to be associated with a terminal emulator or other interactive device, assuming there's an easy way to tell). __________ (*) There are plenty of apps that construct their output using many print/print() calls, and for which flushing after each call would be quite unnecessary and potentially cause lots of redundant I/O operations, ultimately slowing down the app. Apps that need flushing should call flush(). -- --Guido van Rossum (python.org/~guido)
On 1/9/2012 12:52 PM, Guido van Rossum wrote:
Thank you for the clarification. I noted on http://bugs.python.org/issue11633 that there should be no code change
Apps that need flushing should call flush().
and that saying something like this in the print doc entry is the only possible change. -- Terry Jan Reedy
I don't think it would be straight forward or reasonable to _prevent_ flushing from occurring. Forcing a flush, is only useful when the file object to which you're printing isn't already handy, and this only occurs for sys.stdout. It's not worth adding a "flush-in-case" flag just for this one stream, especially if it's unable to prevent flushing. On Tue, Jan 10, 2012 at 8:05 PM, INADA Naoki <songofacandy@gmail.com> wrote:
On Tue, Jan 10, 2012 at 8:15 PM, Masklinn <masklinn@masklinn.net> wrote:
+0 for being able to write print("whatever", force_flush=True) instead of having to do: import sys # somewhere in the file print("whatever") sys.stdout.flush() Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jan 10, 2012 at 6:36 PM, MRAB <python@mrabarnett.plus.com> wrote:
Heh, that's exactly why we made it a function. So is adding e.g. a force_flush flag -- though personally I would be fine calling it "flush" despite the possibility that the underlying stream might still flush when it is not explicitly requests. -- --Guido van Rossum (python.org/~guido)
On 09Jan2012 22:49, Carl M. Johnson <cmjohnson.mailinglist@gmail.com> wrote: | On Jan 9, 2012, at 7:52 PM, INADA Naoki wrote: | > What about adding flush=False keyword argument to print()? | | I would expect flush=False to mean "please don't flush this right now, | even if I do use a newline." | | flush=None? +1 from me, if None means "honour the file's default flushing behaviour". -- Cameron Simpson <cs@zip.com.au> DoD#743 http://www.cskk.ezoshosting.com/cs/ Since I've mentioned the subject of geneology, I'll repeat a story I heard about a poor fellow over on airstrip one. Seems he spent the most recent thirty years of his life tracking down his family history. Spent hundreds of pounds, traveled, devoted his life to it. Then, last month, a cousin told him he was adopted. Ahhh, sweet irony. - Tim_Mefford <tim@physics.orst.edu>
I see that Python 3 behavior is consistent on Windows too meaning it doesn't output anything on the screen when end='' It seems that basic function print() that was created to just "print a string" got over-engineered breaking unix principle of "doing one thing". It reminds me of subprocess.Popen which was also designed to just "run a process". I would keep it as simple as possible for printing stuff to the screen without any implicit buffering. You called print() - you get the output. KISS. If people need more control with output buffering - provide a pointer to sys.stdout with all gory details about streams etc. To get back on topic, let me give a definition. To me "pythonic" means "intuitive" in the first place. And "intuitive" means "up to user expectations". What does user expect from "print()" function? Writing to a file? I don't think so. Output to printer? Maybe. Printing to the screen? Definitely. "intuitive" also means that users won't have to consult user documentation for every basic thing it needs, especially for such basic thing such as getting some stuff printed to the screen output - the stuff that is essential in the coding process. It is already less convenient in Py3k, because you need to type extra Shift-9 and Shift-0 in the process, and it risks to become completely weird: print('calling external process... ', end='') sys.output.flush() ... print('done') So, py3k print() is not pythonic. To make it pythonic, the output should be immediately available after you call it. If you need to buffer output - do this explicitly, because print() doesn't guarantee that it will be buffered anyway. If you care about performance - use sys.stdout directly and tune it explicitly. On Mon, Jan 9, 2012 at 4:25 PM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
Am 10.01.2012 22:36, schrieb Guido van Rossum:
I've created an issue to track this: http://bugs.python.org/issue13761 Georg
There is already an option for flush: -u command line option. Can't we change print() to flush if end is set and -u option is used, instead of introducing a new option? I understand that a command line option is less practical than a new keyword, especially on Windows.
Victor Stinner, 11.01.2012 09:46:
For one, it doesn't allow the program code to explicitly enforce the flush at a specific point if it depends on a command line switch. Note that the caller of print() may not even know what "print" really is at a given point in the execution, and what (if any) output stream it will write to. Doesn't have to be sys.stdout. So a new keyword argument makes sense to me (and I agree that calling it "flush" should be enough). Stefan
On Jan 10, 11:27 pm, anatoly techtonik <techto...@gmail.com> wrote:
No, what "pythonic" means here is "up to anatoly's expectations". If I could kill one idea dead about Python it's this persistent belief that you shouldn't need to read any documentation in order to be able to use it.
"intuitive" also means that users won't have to consult user documentation for every basic thing it needs
Well, clearly that's not true, as your intuition about print's behaviour was wrong. Other people's "intuitive" understanding of mutable default arguments gets them into trouble; you can either "fix" this behaviour so it's inconsistent with the rest of Python's object model, or you can recommend they read the docs so their understanding changes and they know what it means. The latter is the only sane method, IMO. Similarly, every time your intuition is wrong doesn't make it a regression or bug.
Em Mon, 9 Jan 2012 13:30:55 +0300 anatoly techtonik <techtonik@gmail.com> escreveu:
In Python 2 "print 'something', statement calls were unbuffered and immediately appeared on screen.
Only if Python is called with -u command line option, from the man page: -u Force stdin, stdout and stderr to be totally unbuffered. Regards, -- .:''''':. .:' ` Sérgio Surkamp | Administrador de Redes :: ........ sergio@gruposinternet.com.br `:. .:' `:, ,.:' *Grupos Internet S.A.* `: :' R. Lauro Linhares, 2123 Torre B - Sala 201 : : Trindade - Florianópolis - SC :.' :: +55 48 3234-4109 : ' http://www.gruposinternet.com.br
participants (18)
-
alex23 -
anatoly techtonik -
Barry Warsaw -
Cameron Simpson -
Carl M. Johnson -
Devin Jeanpierre -
Georg Brandl -
Guido van Rossum -
INADA Naoki -
Masklinn -
Matt Joiner -
MRAB -
Nick Coghlan -
Paul Moore -
Stefan Behnel -
Sérgio Surkamp -
Terry Reedy -
Victor Stinner