On 9 June 2012 12:00, Paul Moore wrote:

On 9 June 2012 10:55, Nick Coghlan wrote:

...

So, after much digging, it appears the *right* way to replace a standard stream in Python 3 after application start is to do the following:

   sys.stdin = open(sys.stdin.fileno(), 'r', <new settings>)    sys.stdout = open(sys.stdout.fileno(), 'w', <new settings>)    sys.stderr = open(sys.stderr.fileno(), 'w', <new settings>)

Ditto for the other standard streams. It seems it already *is* as simple as with any other file, we just collectively forgot about:

One minor point - if sys.stdout is redirected, *and* you have already written to sys.stdout, this resets the file pointer. With test.py as

import sys print("Hello!") sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8') print("Hello!")

test.py >a gives one line in a, not two (tested on Windows, Unix may be different). And changing to "a" doesn't resolve this...

Ignore me - you need to flush stdout before repoening it, is all. Dumb mistake, sorry for the noise :-( Paul.

On 10.06.12 00:22, Mark Lawrence wrote:

On 09/06/2012 21:02, Serhiy Storchaka wrote:

...

None of these methods are not guaranteed to work if the input or output have occurred before.

That's a double negative so I'm not sure what you meant to say. Can you please rephrase it. I assume that English is not your native language, so I'll let you off :)

open(sys.stdin.fileno()) is not guaranteed to work if the input or output have occurred before. And io.TextIOWrapper(sys.stdin.detach()) is not guaranteed to work if the input or output have occurred before. sys.stdin internal buffer can contains read by not used characters. sys.stdin.buffer internal buffer can contains read by not used bytes. With multibyte encoding sys.stdin.decoder internal buffer can contains uncompleted multibyte character.

^^^^^^^[[[[[[[[[@[@Nick Coghlan writes:

On Mon, Jun 11, 2012 at 12:34 AM, Serhiy Storchaka wrote:

...

open(sys.stdin.fileno()) is not guaranteed to work if the input or output have occurred before. [...]

Right, but the point of this discussion is to document the cleanest available way for an application to change these settings at *application start* (e.g. to support an "--encoding" parameter). Yes, there are potential issues if you use any of these mechanisms while there is data in the buffers,

+1 The OP's problem is a real one. His use case (the "--encoding" parameter) seems to be the most likely one in production use, so the loss of buffered data issue should rarely come up. Changing encodings on the fly offers plenty of ways to lose data besides incomplete buffers, anyway. I am a little concerned with MRAB's report that import sys print("hello") sys.stdout.flush() sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8') print("hello") doesn't work as expected, though. (It does work for me on Mac OS X, both as above -- of course there are no '\r's in the output -- and with 'print("hello", end="\r\n")'.)

On 10 June 2012 19:12, MRAB wrote:

On 10/06/2012 17:41, Stephen J. Turnbull wrote:

...

I am a little concerned with MRAB's report that

    import sys     print("hello")     sys.stdout.flush()     sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8')     print("hello")

doesn't work as expected, though.  (It does work for me on Mac OS X, both as above -- of course there are no '\r's in the output -- and with 'print("hello", end="\r\n")'.)

That's actually Python 3.1. From Python 3.2 it's slightly different, but still not quite right:

Python 3.1:     "hello\r\nhello\r\r\n" Python 3.2:     "hello\nhello\r\n" Python 3.3.0a4: "hello\nhello\r\n"

All on Windows.

Not here (Win 7 32-bit): PS D:\Data> type t.py import sys print("Hello!") sys.stdout.flush() sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8') print("Hello!") PS D:\Data> py -3.2 t.py | od -c 0000000 H e l l o ! \r \n H e l l o ! \r \n 0000020 Paul.

On 10 June 2012 20:01, MRAB wrote:

On 10/06/2012 19:34, Paul Moore wrote:

...

On 10 June 2012 19:12, MRAB  wrote:

...

 On 10/06/2012 17:41, Stephen J. Turnbull wrote:

...

 I am a little concerned with MRAB's report that

     import sys      print("hello")      sys.stdout.flush()      sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8')      print("hello")

 doesn't work as expected, though.  (It does work for me on Mac OS X,  both as above -- of course there are no '\r's in the output -- and  with 'print("hello", end="\r\n")'.)

 That's actually Python 3.1. From Python 3.2 it's slightly different,  but still not quite right:

 Python 3.1:     "hello\r\nhello\r\r\n"  Python 3.2:     "hello\nhello\r\n"  Python 3.3.0a4: "hello\nhello\r\n"

 All on Windows.

Not here (Win 7 32-bit):

PS D:\Data>  type t.py import sys print("Hello!") sys.stdout.flush()

sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8') print("Hello!") PS D:\Data>  py -3.2 t.py | od -c 0000000   H   e   l   l   o   !  \r  \n   H   e   l   l   o   !  \r  \n 0000020

I'm using Windows XP Pro (32-bit), initially sys.stdout.encoding == "cp1252".

PS D:\Data> py -3 -c "import sys; print(sys.stdout.encoding)" cp850 This is at the console (Powershell) - are you running from within something like idle, or a GUI environment? Paul.

On 10 June 2012 21:28, MRAB wrote:

On 10/06/2012 21:07, Paul Moore wrote:

...

On 10 June 2012 20:01, MRAB  wrote:

...

 On 10/06/2012 19:34, Paul Moore wrote:

...

 On 10 June 2012 19:12, MRAB    wrote:

...

  On 10/06/2012 17:41, Stephen J. Turnbull wrote:

...

  I am a little concerned with MRAB's report that

      import sys       print("hello")       sys.stdout.flush()       sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8')       print("hello")

  doesn't work as expected, though.  (It does work for me on Mac OS X,   both as above -- of course there are no '\r's in the output -- and   with 'print("hello", end="\r\n")'.)

  That's actually Python 3.1. From Python 3.2 it's slightly different,   but still not quite right:

  Python 3.1:     "hello\r\nhello\r\r\n"   Python 3.2:     "hello\nhello\r\n"   Python 3.3.0a4: "hello\nhello\r\n"

  All on Windows.

 Not here (Win 7 32-bit):

 PS D:\Data>    type t.py  import sys  print("Hello!")  sys.stdout.flush()

 sys.stdout = open(sys.stdout.fileno(), 'w', encoding='utf-8')  print("Hello!")  PS D:\Data>    py -3.2 t.py | od -c  0000000   H   e   l   l   o   !  \r  \n   H   e   l   l   o   !  \r  \n  0000020

 I'm using Windows XP Pro (32-bit), initially sys.stdout.encoding ==  "cp1252".

PS D:\Data>  py -3 -c "import sys; print(sys.stdout.encoding)" cp850

This is at the console (Powershell) - are you running from within something like idle, or a GUI environment?

It's at the system command prompt. When I redirect the script's stdout to a file (on the command line using ">output.txt") I get those 15 bytes from Python 3.2.

Your output appears to be 32 bytes (the second line starts with "0000020").

Well spotted - PowerShell does funny things with Unicode in pipes, I'd forgotten. Indeed, I get the same output as you from cmd. Odd. Paul

On 11 June 2012 07:16, Stephen J. Turnbull wrote:

MRAB writes:

 > That's actually Python 3.1. From Python 3.2 it's slightly different,  > but still not quite right:  >  > Python 3.1:     "hello\r\nhello\r\r\n"  > Python 3.2:     "hello\nhello\r\n"  > Python 3.3.0a4: "hello\nhello\r\n"  >  > All on Windows.

<stifle o="self"/>

Hm.  Maybe it's that port's implementation of universal newlines or something like that?  What happens if you use an explicit "end=" argument?  (I don't have a Python 3 to check on Windows easily available.)

Explicit end= makes no difference to the behaviour. In fact, a minimal test suggests that universal newline mode is not enabled on Windows in Python 3. That's a regression from 2.x. See below. D:\Data>py -3 -c "print('x')" | od -c 0000000 x \n 0000002 D:\Data>py -2 -c "print('x')" | od -c 0000000 x \r \n 0000003 D:\Data>py -3 -V Python 3.2.2 D:\Data>py -2 -V Python 2.7.2 Paul.

On 10.06.12 18:44, Nick Coghlan wrote:

This approach also has the advantage of leaving sys.__std(in/out/err)__ in a somewhat usable state.

And then sys.std* and sys.__std*__ have their own inconsistent buffers.

Calling detach() on the standard streams is a bad idea - the interpreter uses the originals internally, and calling detach() breaks them. -- Sent from my phone, thus the relative brevity :) On Jun 10, 2012 6:03 AM, "Serhiy Storchaka" wrote:

On 09.06.12 12:55, Nick Coghlan wrote:

...

So, after much digging, it appears the *right* way to replace a standard stream in Python 3 after application start is to do the following:

sys.stdin = open(sys.stdin.fileno(), 'r',<new settings>) sys.stdout = open(sys.stdout.fileno(), 'w',<new settings>) sys.stderr = open(sys.stderr.fileno(), 'w',<new settings>)

sys.stdin = io.TextIOWrapper(sys.stdin.**detach(), <new settings>) sys.stdout = io.TextIOWrapper(sys.stdout.**detach(), <new settings>) ...

None of these methods are not guaranteed to work if the input or output have occurred before.

______________________________**_________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/**mailman/listinfo/python-ideashttp://mail.python.org/mailman/listinfo/python-ideas

On Sun, Jun 10, 2012 at 5:17 PM, Antoine Pitrou wrote:

Le 10/06/2012 04:26, Nick Coghlan a écrit :

...

Calling detach() on the standard streams is a bad idea - the interpreter uses the originals internally, and calling detach() breaks them.

Where does it do that? The interpreter certainly shouldn't hardwire the original objects internally.

At the very least, sys.__std(in/out/err)__. Doing "sys.stderr = io.TextIOWrapper(sys.stderr.detach(), line_buffering=True)" also seems to suppress display of exception tracebacks at the interactive prompt (perhaps the default except hook is using a cached reference?). I believe PyFatalError and other APIs that are used deep in the interpreter won't respect the module level setting. Basically, it's dangerous to use detach() on a stream where you don't hold the sole reference, and the safest approach with the standard streams is to assume that other code is holding references to them. Detaching the standard streams is just as likely to cause problems as closing them.

Moreover, your snippet is wrong because if someone replaces the streams for a second time, garbage collecting the previous streams will close the file descriptors. You should use closefd=False.

True, although that nicety is all the more reason to encapsulate this idiom in a new IOBase.reopen() method: def reopen(self, mode=None, buffering=-1, encoding=None, errors=None, newline=None, closefd=False): if mode is None: mode = getattr(mode, self, 'r') return open(self.fileno(), mode, buffering, encoding, errors, newline, closefd) Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

...

    sys.stdin = open(sys.stdin.fileno(), 'r',<new settings>)     sys.stdout = open(sys.stdout.fileno(), 'w',<new settings>)     sys.stderr = open(sys.stderr.fileno(), 'w',<new settings>)

   sys.stdin = io.TextIOWrapper(sys.stdin.detach(), <new settings>)    sys.stdout = io.TextIOWrapper(sys.stdout.detach(), <new settings>)    ...

None of these methods are not guaranteed to work if the input or output have occurred before.

You should set the newline option for sys.std* files. Python 3 does something like this: if os.name == "win32: # translate "\r\n" to "\n" for sys.stdin on Windows newline = None else: newline = "\n" sys.stdin = io.TextIOWrapper(sys.stdin.detach(), newline=newline, <new settings>) sys.stdout = io.TextIOWrapper(sys.stdout.detach(), newline="\n", <new settings>) sys.stderr = io.TextIOWrapper(sys.stderr.detach(), newline="\n", <new settings>) -- Lib/test/regrtest.py uses the following code which is not exactly correct (it creates a new buffered writer instead of reusing sys.stdout buffered writer): def replace_stdout(): """Set stdout encoder error handler to backslashreplace (as stderr error handler) to avoid UnicodeEncodeError when printing a traceback""" import atexit stdout = sys.stdout sys.stdout = open(stdout.fileno(), 'w', encoding=stdout.encoding, errors="backslashreplace", closefd=False, newline='\n') def restore_stdout(): sys.stdout.close() sys.stdout = stdout atexit.register(restore_stdout) Victor