execute function on iterator items without changing or consuming iterator
Python has many tools for iteration, such as map and filter, that change an iterator into another iterator without consuming the iterator. These make it simple to deal with many items in a list without consuming excessive memory. Occasionally it is useful to be able to tap into iterator items and execute a function (such as a side effect to validate elements or print them) without making any changes to the overall iterator. This is similar to the idea of a tap in rxjs: https://rxjs-dev.firebaseapp.com/api/operators/tap. The proposed interface would be: def generate_items() -> list[int]: some_iter = range(10) some_iter = tap(assert_int, some_iter) some_iter = tap(print, some_iter) return list(some_iter) This would be useful to (for example): 1. Debug chained iterators without a debugger (at the moment you would have to convert the list and then print the whole list or include a print statement in one of the chained functions) 2. Check that items in an iterator conform to assumptions and raising exceptions if they do not 3. Improve type hints in editors since after the assert is executed all items conform to the assert in the tap (for example, they are an integer) The implementation would be quite simple (at least in python): def tap(func: typing.Callable[[T], typing.Any], iter: typing.Iterable[T]) -> typing.iterable[T]: for item in iter: func(item) yield item Thoughts?
This seems like it might be a reasonable thing to have in intertools? It's not hard to write this sort of thing with map: some_iter = map(lambda x: (print(x), x)[1], some_iter) But it's a little awkward. (Though maybe I'm missing a less awkward way to do that.) Java has Stream.peek for similar functionality, while Stream.forEach is analogous to Python's map. On Sun, Oct 25, 2020 at 9:02 AM <anderssonpublic@gmail.com> wrote:
Python has many tools for iteration, such as map and filter, that change an iterator into another iterator without consuming the iterator. These make it simple to deal with many items in a list without consuming excessive memory.
Occasionally it is useful to be able to tap into iterator items and execute a function (such as a side effect to validate elements or print them) without making any changes to the overall iterator. This is similar to the idea of a tap in rxjs: https://rxjs-dev.firebaseapp.com/api/operators/tap.
The proposed interface would be: def generate_items() -> list[int]: some_iter = range(10) some_iter = tap(assert_int, some_iter) some_iter = tap(print, some_iter) return list(some_iter)
This would be useful to (for example): 1. Debug chained iterators without a debugger (at the moment you would have to convert the list and then print the whole list or include a print statement in one of the chained functions) 2. Check that items in an iterator conform to assumptions and raising exceptions if they do not 3. Improve type hints in editors since after the assert is executed all items conform to the assert in the tap (for example, they are an integer)
The implementation would be quite simple (at least in python): def tap(func: typing.Callable[[T], typing.Any], iter: typing.Iterable[T]) -> typing.iterable[T]: for item in iter: func(item) yield item
Thoughts? _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7Q3NR4... Code of Conduct: http://python.org/psf/codeofconduct/
less awkward is: some_iter = map(lambda x: x if print(x) else x, some_iter) The tuple has a ~50% overhead, the case statement ~15%, compared to the generator. I think that the less awkward syntax solves the problem fine (if you can come up with it). I like that it's explicit rather than requiring someone to know what itertools.tap does. On Sun, Oct 25, 2020 at 4:27 PM Samuel Freilich via Python-ideas < python-ideas@python.org> wrote:
This seems like it might be a reasonable thing to have in intertools? It's not hard to write this sort of thing with map:
some_iter = map(lambda x: (print(x), x)[1], some_iter)
But it's a little awkward. (Though maybe I'm missing a less awkward way to do that.)
Java has Stream.peek for similar functionality, while Stream.forEach is analogous to Python's map.
On Sun, Oct 25, 2020 at 9:02 AM <anderssonpublic@gmail.com> wrote:
Python has many tools for iteration, such as map and filter, that change an iterator into another iterator without consuming the iterator. These make it simple to deal with many items in a list without consuming excessive memory.
Occasionally it is useful to be able to tap into iterator items and execute a function (such as a side effect to validate elements or print them) without making any changes to the overall iterator. This is similar to the idea of a tap in rxjs: https://rxjs-dev.firebaseapp.com/api/operators/tap.
The proposed interface would be: def generate_items() -> list[int]: some_iter = range(10) some_iter = tap(assert_int, some_iter) some_iter = tap(print, some_iter) return list(some_iter)
This would be useful to (for example): 1. Debug chained iterators without a debugger (at the moment you would have to convert the list and then print the whole list or include a print statement in one of the chained functions) 2. Check that items in an iterator conform to assumptions and raising exceptions if they do not 3. Improve type hints in editors since after the assert is executed all items conform to the assert in the tap (for example, they are an integer)
The implementation would be quite simple (at least in python): def tap(func: typing.Callable[[T], typing.Any], iter: typing.Iterable[T]) -> typing.iterable[T]: for item in iter: func(item) yield item
Thoughts? _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7Q3NR4... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7AUUGT... Code of Conduct: http://python.org/psf/codeofconduct/
On 2020-10-25 at 16:34:14 +0000, George Harding <george.winton.harding@gmail.com> wrote:
some_iter = map(lambda x: x if print(x) else x, some_iter)
The tuple has a ~50% overhead, the case statement ~15%, compared to the generator.
def print_first(x): print(x) return x new_iter = map(print_first, some_iter) No extranous tuple, no extranous case stament. Newlines are cheap these days. The call to print, however, is possibly unbounded in time and space. If you really want to go overboard, put something like the following peeker function into your personal toolbox (untested): def peeker(function): def peeker(x): function(x) return x return peeker And use it like this: new_iter = map(peeker(print), some_iter)
this seems nice. It just seems that the obvious way to do it would be: def tap(iter_, func): for item in iter_: func(item) yield item I think it can be nice, but more as cookbook material than an actual itertools function. On Sun, 25 Oct 2020 at 14:38, <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:
On 2020-10-25 at 16:34:14 +0000, George Harding <george.winton.harding@gmail.com> wrote:
some_iter = map(lambda x: x if print(x) else x, some_iter)
The tuple has a ~50% overhead, the case statement ~15%, compared to the generator.
def print_first(x): print(x) return x new_iter = map(print_first, some_iter)
No extranous tuple, no extranous case stament. Newlines are cheap these days. The call to print, however, is possibly unbounded in time and space.
If you really want to go overboard, put something like the following peeker function into your personal toolbox (untested):
def peeker(function): def peeker(x): function(x) return x return peeker
And use it like this:
new_iter = map(peeker(print), some_iter) _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RG2ZQD... Code of Conduct: http://python.org/psf/codeofconduct/
participants (5)
-
2QdxY4RzWzUUiLuE＠potatochowder.com -
anderssonpublic＠gmail.com -
George Harding -
Joao S. O. Bueno -
Samuel Freilich