easier writing to multiple streams
hi first post here :) i have a recurring use pattern where i want to save to file everything i print to the screen. i have used the standard logging module and found it too cumbersome for my simple use case, and i find myself changing all my print()'s to custom myprint()'s. i am interested in suggesting several additions to the standard libraries, as i feel this use case is common enough, important enough, and currently painful enough to justify such changes, and that all four of the following suggestions could be useful in a complementary fashion. 1. allow the print() "file" argument to take a list of streams (currently it can take only a single stream). e.g: with open('output.txt', 'w') as f: print('hello', file=[sys.stdout, f]) this seems like the quickest and least resistant path for the user. need to decide how multiple streams will work with the flush argument. 2. modify contextlib.redirect_stdout() to take an additional boolean argument which we can call tee/replicate/duplicate/clone etc. when True it will duplicate instead of just redirecting: with open('output.txt', 'w') as f: with redirect_stdout(f, tee=True): # will duplicate instead of redirecting print('hello') or we could add a new context manager contextlib.copy_stdout() 3. allow the contextlib.redirect_stdout() "new_target" argument to take a list of streams, or allow specifying multiple arguments for multiple streams. e.g: with open('output.txt', 'w') as f: with redirect_stdout(sys.stdout, f): # or redirect_stdout([sys.stdout, f]) print('hello') this has the benefit over the tee argument of allowing more than one additional stream, but you need to explicitly specify stdout. so would be nice to have both modifications. 4. add to the standard io library a new class which gives you the write interface of a single stream, but is a wrapper that will write to multiple streams: with open('output.txt', 'w') as f: multi = io.OutputStreamCollection(sys.stdout, f) multi.write('hello\n') # or: print('hello', file=multi) this is similar to: https://stackoverflow.com/questions/9130755/wrapper-to-write-to-multiple-str..., and we need to decide about the rest of the stream methods. eyal.
On 25 Nov 2021, at 13:07, Eyal Gruss <eyalgruss@gmail.com> wrote:
hi
first post here :) i have a recurring use pattern where i want to save to file everything i print to the screen. i have used the standard logging module and found it too cumbersome for my simple use case, and i find myself changing all my print()'s to custom myprint()'s. i am interested in suggesting several additions to the standard libraries, as i feel this use case is common enough, important enough, and currently painful enough to justify such changes, and that all four of the following suggestions could be useful in a complementary fashion.
1. allow the print() "file" argument to take a list of streams (currently it can take only a single stream). e.g:
with open('output.txt', 'w') as f: print('hello', file=[sys.stdout, f])
You can replace sys.stdout with your own object that prints to a list of streams. I have used this in the past. The implementation can be as simple as supplying an object with a write method that writes to each of the steams I turn. Barry
this seems like the quickest and least resistant path for the user. need to decide how multiple streams will work with the flush argument.
2. modify contextlib.redirect_stdout() to take an additional boolean argument which we can call tee/replicate/duplicate/clone etc. when True it will duplicate instead of just redirecting:
with open('output.txt', 'w') as f: with redirect_stdout(f, tee=True): # will duplicate instead of redirecting print('hello')
or we could add a new context manager contextlib.copy_stdout()
3. allow the contextlib.redirect_stdout() "new_target" argument to take a list of streams, or allow specifying multiple arguments for multiple streams. e.g:
with open('output.txt', 'w') as f: with redirect_stdout(sys.stdout, f): # or redirect_stdout([sys.stdout, f]) print('hello')
this has the benefit over the tee argument of allowing more than one additional stream, but you need to explicitly specify stdout. so would be nice to have both modifications.
4. add to the standard io library a new class which gives you the write interface of a single stream, but is a wrapper that will write to multiple streams:
with open('output.txt', 'w') as f: multi = io.OutputStreamCollection(sys.stdout, f) multi.write('hello\n') # or: print('hello', file=multi)
this is similar to: https://stackoverflow.com/questions/9130755/wrapper-to-write-to-multiple-str..., and we need to decide about the rest of the stream methods.
eyal. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7I55RD... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Nov 25, 2021 at 9:52 AM Barry <barry@barrys-emacs.org> wrote:
You can replace sys.stdout with your own object that prints to a list of streams. I have used this in the past.
I think was essentially the OP's suggestion #4: 4. add to the standard io library a new class which gives you the write
interface of a single stream, but is a wrapper that will write to multiple streams:
Except that it would be in the stdlib. "note very two line function needs to be in the stlib" -- but I'm guessing that there might be some subtleties to make it truly robust, so it may be worth adding. Also, from a newbie script-writers perspective, that few-line class is a bit complex. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Nov 25, 2021, 6:06 AM Eyal Gruss <eyalgruss@gmail.com> wrote:
hi
first post here :) i have a recurring use pattern where i want to save to file everything i print to the screen. i have used the standard logging module and found it too cumbersome for my simple use case, and i find myself changing all my print()'s to custom myprint()'s. i am interested in suggesting several additions to the standard libraries, as i feel this use case is common enough, important enough, and currently painful enough to justify such changes, and that all four of the following suggestions could be useful in a complementary fashion.
I'm not sure this is actually so common. How do you know it is? Also, it's not very painful to do, just occasionally annoying. If you want, you could write`_print = print` and then redefine print() as you want. I doubt that's good practice, but it works. 1. allow the print() "file" argument to take a list of streams (currently
it can take only a single stream). e.g:
with open('output.txt', 'w') as f: print('hello', file=[sys.stdout, f])
this seems like the quickest and least resistant path for the user. need to decide how multiple streams will work with the flush argument.
This could very possibly be backwards-incompatible. 2. modify contextlib.redirect_stdout() to take an additional boolean
argument which we can call tee/replicate/duplicate/clone etc. when True it will duplicate instead of just redirecting:
with open('output.txt', 'w') as f: with redirect_stdout(f, tee=True): # will duplicate instead of redirecting print('hello')
or we could add a new context manager contextlib.copy_stdout()
3. allow the contextlib.redirect_stdout() "new_target" argument to take a list of streams, or allow specifying multiple arguments for multiple streams. e.g:
with open('output.txt', 'w') as f: with redirect_stdout(sys.stdout, f): # or redirect_stdout([sys.stdout, f]) print('hello')
this has the benefit over the tee argument of allowing more than one additional stream, but you need to explicitly specify stdout. so would be nice to have both modifications.
4. add to the standard io library a new class which gives you the write interface of a single stream, but is a wrapper that will write to multiple streams:
with open('output.txt', 'w') as f: multi = io.OutputStreamCollection(sys.stdout, f) multi.write('hello\n') # or: print('hello', file=multi)
Is this necessary if the other ideas are implemented? this is similar to:
https://stackoverflow.com/questions/9130755/wrapper-to-write-to-multiple-str..., and we need to decide about the rest of the stream methods.
eyal. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7I55RD... Code of Conduct: http://python.org/psf/codeofconduct/
-- Finn (Mobile)
On 24Nov2021 19:26, Eyal Gruss <eyalgruss@gmail.com> wrote:
i find myself changing all my print()'s to custom myprint()'s.
Had you tried: from myutils import myprint as print I've got a terminal status line module (cs.upd), and to print things one needs to withdraw the status lines, run the normal print(), restore the lines (or nip to the top status line, insert a blank, run the normal print, redraw below it). This approach lets one interoperate with code already using print() without changing that code.
4. add to the standard io library a new class which gives you the write interface of a single stream, but is a wrapper that will write to multiple streams:
with open('output.txt', 'w') as f: multi = io.OutputStreamCollection(sys.stdout, f) multi.write('hello\n') # or: print('hello', file=multi)
I've got one of those in my cs.fileutils module, a Tee(*files) class and a tee(file1, file2) context manager. I find I pretty much never use them. While that's unfair to you, since you seem to have this situation, it is to me an argument that such a thing does not need to be in the stdlib. You mentioned logging. Doesn't just adding a second root logger (or handler) get you what you'd want there? They accumulate, and messages go to all lthe loggers. BTW, can you elaborate on when you find yourself wanting to write the same message t multiple files? Other than logging, I've almost never wanted to do that. Cheers, Cameron Simpson <cs@cskk.id.au>
i wonder whether: from myutils import myprint as print or _print = print print = myprint is really the pythonic way? my use case for multiple files on top of the stdout, is when using e.g. wandb which is a popular ML dashboard and experiment logging platform. i want to write my log file both the a local log.txt and to a second copy in the temporary local wandb folder that later gets synced to the cloud. otherwise i have to take care of copying over the file later including in cases of exceptions. more generally: writing a log to both a local and a remote location. thanks eyal
On Sat, Nov 27, 2021 at 3:03 AM <eyalgruss@gmail.com> wrote:
i wonder whether:
from myutils import myprint as print
or
_print = print print = myprint
is really the pythonic way?
Seems fine to me, what's unpythonic about it?
my use case for multiple files on top of the stdout, is when using e.g. wandb which is a popular ML dashboard and experiment logging platform. i want to write my log file both the a local log.txt and to a second copy in the temporary local wandb folder that later gets synced to the cloud. otherwise i have to take care of copying over the file later including in cases of exceptions. more generally: writing a log to both a local and a remote location.
The other option would be to pipe your script through tee and do the whole thing externally. That way, if something goes horribly wrong, the output is still captured. ChrisA
On 26Nov2021 16:00, eyalgruss@gmail.com <eyalgruss@gmail.com> wrote:
i wonder whether:
from myutils import myprint as print
or
_print = print print = myprint
is really the pythonic way?
Well, I prefer the former - short and clear. You can always get the "old" print from the builtins module. In the setup above, myutils.print would do the _print=print shuffle in its implementation. In my own code that often looks like: from builtins import print as builtin_print def print(...): ... call buildin_print as needed ... Then I import print from that module. Example: from cs.upd import Upd, print "Upd" for the class the module provides, and print for the interoperating print() function from that module.
my use case for multiple files on top of the stdout, is when using e.g. wandb which is a popular ML dashboard and experiment logging platform. i want to write my log file both the a local log.txt and to a second copy in the temporary local wandb folder that later gets synced to the cloud. otherwise i have to take care of copying over the file later including in cases of exceptions. more generally: writing a log to both a local and a remote location.
This sounds to me lke you want a logger setup with multiple handlers. Add a handler to log to your cloud log file in addition to the normal log file. Then you'd just use ordinary logging calls in your code elsewhere. No special print() at all. Cheers, Cameron Simpson <cs@cskk.id.au>
On 26 Nov 2021, at 16:00, eyalgruss@gmail.com wrote:
i wonder whether:
from myutils import myprint as print
or
_print = print print = myprint
is really the pythonic way?
The problem with this is that there are more ways to output then the print command. Only replacing print will not catch any code that uses sys.stdout.write() for example. That is why when I need this I replace sys.stdout and usually also sys.stderr with a file like object. BTW did you know that you can get the original sys.stdout is also available in sys.__stdout__?
my use case for multiple files on top of the stdout, is when using e.g. wandb which is a popular ML dashboard and experiment logging platform. i want to write my log file both the a local log.txt and to a second copy in the temporary local wandb folder that later gets synced to the cloud. otherwise i have to take care of copying over the file later including in cases of exceptions. more generally: writing a log to both a local and a remote location.
As has been already suggested, you can use the logging module and add handlers for each of the log files that you want to create at the same time, I have also replaced sys.stdout with an object that takes all lines written to sys.stdout and sends them to the logger. Barry
thanks eyal _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PEH5SG... Code of Conduct: http://python.org/psf/codeofconduct/
participants (8)
-
Barry -
Barry Scott
-
Cameron Simpson -
Chris Angelico -
Christopher Barker -
Eyal Gruss -
eyalgruss＠gmail.com
-
Finn Mason