Support reversed(itertools.chain(x, y, z))
Today I had a need: I had a tuple of dynamic sequence-like objects. I wanted to iterate on them reversed, starting with items of the last one and slowly making my way towards the first one. In short, I want `reversed(itertools.chain(x, y, z))` that behaves like `itertools.chain(map(reversed, (z, y, x)))`. What do you think?
On 9/01/21 10:19 am, Ram Rachum wrote:
In short, I want `reversed(itertools.chain(x, y, z))` that behaves like `itertools.chain(map(reversed, (z, y, x)))`.
I think you mean `itertools.chain(*map(reversed, (z, y, x)))` You can get this with itertools.chain(*map(reversed, reversed(t))) Making `reversed(itertools.chain(x, y, z))` do this would be a backwards incompatible change. Also it's hard to see how it could be made to work, because the argument to reversed() necessarily has to be a sequence, not an iterator. -- Greg
On Sat, Jan 09, 2021 at 11:22:35AM +1300, Greg Ewing wrote:
Also it's hard to see how it could be made to work, because the argument to reversed() necessarily has to be a sequence, not an iterator.
No, it just needs a `__reversed__` dunder. It doesn't even need to be an iterable or sequence at all!
class X: ... def __reversed__(self): ... return iter("Bet you didn't see this coming.".split()) ... list(reversed(X())) ['Bet', 'you', "didn't", 'see', 'this', 'coming.']
-- Steve
On Fri, 8 Jan 2021 at 22:25, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 9/01/21 10:19 am, Ram Rachum wrote:
In short, I want `reversed(itertools.chain(x, y, z))` that behaves like `itertools.chain(map(reversed, (z, y, x)))`.
I think you mean `itertools.chain(*map(reversed, (z, y, x)))`
You can get this with
itertools.chain(*map(reversed, reversed(t)))
Making `reversed(itertools.chain(x, y, z))` do this would be a backwards incompatible change.
Also it's hard to see how it could be made to work, because the argument to reversed() necessarily has to be a sequence, not an iterator.
The argument to reversed either needs to be a sequence with __len__ and __getitem__ or an object with a __reversed__ method that returns an iterator. The arguments to chain have to be iterables. Every sequence is an iterable so there is a significant intersection between the possible inputs to chain and reversed. Also some non-sequences such as dict can work with reversed. You say it's hard to see how it could be made to work but you've shown precisely how it can already be done above: reversed(chain(*args)) == chain(*map(reversed, reversed(args))) We can try that out and it certainly seems to work: >>> from itertools import chain >>> args = [[1, 2], [3, 4]] >>> list(chain(*args)) [1, 2, 3, 4] >>> list(chain(*map(reversed, reversed(args)))) [4, 3, 2, 1] This wouldn't work with chain.from_iterable without preconsuming the top-level iterable but in the case of chain the iterables are already in a *args tuple so flipping that order is always possible in a lazy way. That means the operation works fine if each arg in args is reversible. Otherwise if any arg is not reversible then it should give a TypeError just like reversed(set()) does except the error would potentially be delayed if some of the args are reversible and some are not. I haven't ever wanted to reverse a chain but I have wanted to be able to reverse an enumerate many times: >>> reversed(enumerate([1, 2, 3])) ... TypeError The alternative zip(range(len(obj)-1, -1, -1), reversed(obj)) is fairly cryptic in comparison as well as probably being less efficient. There could be a __reversed__ method for enumerate with the same caveat as for chain: if the underlying object is not reversible then you get a TypeError. Otherwise reversed(enumerate(seq)) works fine for any sequence seq. The thornier issue is how to handle reversed if the chain/enumerate iterator has already been partially consumed. If it's possible just to give an error in that case then reversed could still be useful in the common case. -- Oscar
On Sat, 9 Jan 2021 at 13:29, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
The argument to reversed either needs to be a sequence with __len__ and __getitem__ or an object with a __reversed__ method that returns an iterator. The arguments to chain have to be iterables. Every sequence is an iterable so there is a significant intersection between the possible inputs to chain and reversed. Also some non-sequences such as dict can work with reversed.
You say it's hard to see how it could be made to work but you've shown precisely how it can already be done above:
reversed(chain(*args)) == chain(*map(reversed, reversed(args)))
We can try that out and it certainly seems to work:
>>> from itertools import chain >>> args = [[1, 2], [3, 4]] >>> list(chain(*args)) [1, 2, 3, 4] >>> list(chain(*map(reversed, reversed(args)))) [4, 3, 2, 1]
This wouldn't work with chain.from_iterable without preconsuming the top-level iterable but in the case of chain the iterables are already in a *args tuple so flipping that order is always possible in a lazy way. That means the operation works fine if each arg in args is reversible. Otherwise if any arg is not reversible then it should give a TypeError just like reversed(set()) does except the error would potentially be delayed if some of the args are reversible and some are not.
I haven't ever wanted to reverse a chain but I have wanted to be able to reverse an enumerate many times:
>>> reversed(enumerate([1, 2, 3])) ... TypeError
The alternative zip(range(len(obj)-1, -1, -1), reversed(obj)) is fairly cryptic in comparison as well as probably being less efficient. There could be a __reversed__ method for enumerate with the same caveat as for chain: if the underlying object is not reversible then you get a TypeError. Otherwise reversed(enumerate(seq)) works fine for any sequence seq.
The thornier issue is how to handle reversed if the chain/enumerate iterator has already been partially consumed. If it's possible just to give an error in that case then reversed could still be useful in the common case.
I think you're about right here - both chain and enumerate could reasonably be expected to be reversible. There are some fiddly edge cases, and some potentially weird situations (as soon as we assume no-one would ever expect to reverse a partially consumed iterator, I bet someone will...) which probably warrant no more than "don't do that then" but will end up being the subject of questions/confusion. The question is whether the change is worth the cost. For me: 1. enumerate is probably more important than chain. I use enumerate a *lot* and I've very rarely used chain. 2. Consistency is a benefit - as we've already seen, people assume things work by analogy with other cases, and waste time when they don't. 3. How easy it is to write your own matters. If chain or enumerate objects exposed the iterables they were based on, you could write your own reverser more easily. 4. How problematic are the workarounds? reversed(list(some_iter)) works fine - is turning the iterator into a concrete list that much of an issue? And of course, the key point - how often do people want to do this anyway? If someone wants to do the work to implement this, I would say go for it - raise a bpo issue and create a PR, and see what the response is. Getting "community support" via this list is probably not crucial for something like this. It's more of a quality of life change than a big feature. Paul
On Sun, Jan 10, 2021 at 12:29 AM Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
I haven't ever wanted to reverse a chain but I have wanted to be able to reverse an enumerate many times:
>>> reversed(enumerate([1, 2, 3])) ... TypeError
The alternative zip(range(len(obj)-1, -1, -1), reversed(obj)) is fairly cryptic in comparison as well as probably being less efficient. There could be a __reversed__ method for enumerate with the same caveat as for chain: if the underlying object is not reversible then you get a TypeError. Otherwise reversed(enumerate(seq)) works fine for any sequence seq.
To clarify, you want reversed(enumerate(x)) to yield the exact same pairs that reversed(list(enumerate(x))) would return, yes? If so, it absolutely must have a length, AND be reversible. I don't think spelling it reversed(enumerate(x)) will work, due to issues with partial consumption; but it wouldn't be too hard to define a renumerate function: def renumerate(seq): """Equivalent to reversed(list(enumerate(seq))) but more efficient""" return zip(range(len(seq))[::-1], reversed(seq)) (I prefer spelling it [::-1] than risking getting the range args wrong, but otherwise it's the same as you had) But if you'd rather not do this, then a cleaner solution might be for enumerate() to grow a step parameter: enumerate(iterable, start=0, step=1) And then, what you want is simply: enumerate(reversed(seq), len(seq) - 1, -1) I'm +0 on enumerate gaining a parameter, and otherwise, -1 on actual changes to the stdlib - this is a one-liner that you can have in your personal library if you need it. Might be a cool recipe for itertools docs, or one of the third-party more-itertools packages, or something, though. ChrisA
On Fri, Jan 08, 2021 at 11:19:40PM +0200, Ram Rachum wrote:
Today I had a need: I had a tuple of dynamic sequence-like objects. I wanted to iterate on them reversed, starting with items of the last one and slowly making my way towards the first one.
In short, I want `reversed(itertools.chain(x, y, z))` that behaves like `itertools.chain(map(reversed, (z, y, x)))`.
That would break backwards compatibility, because `reversed(itertools.chain(x, y, z))` is already possible today, and it does *not* behave in that fashion. reversed reverses whatever iterable it is given, it doesn't single out chain objects for special magical handling. If I write this: reversed([None, 'abc', 'def', 'ghi']) I expect to get 'ghi', 'def', 'abc', None and not 'i', 'h', 'g', 'f', 'e', 'd', 'c', 'b', 'a', raise TypeError If I replace the list with itertools.chain, I should still get exactly the same results, and I do. Breaking backwards compatibility for your special case is not going to happen.
What do you think?
I think you should just write `itertools.chain(map(reversed, (z, y, x)))`. If you don't have the individual x, y, z sequences, but only their tuple t = (x, y, z), you have two choices: itertools.chain(map(reversed, t[::-1])) itertools.chain(map(reversed, reversed(t))) They have slightly different meanings, so you get to choose whichever one suits your use-case better. Not every trivial combination of functions needs to be given a built-in or standard library solution. Especially not if doing so will break backwards compatibility. "I had three numbers in a tuple, and wanted half of twice the first number added to the difference of the remaining two. What do you think about making `len((a,b,c))` return `(2*a + abs(b - c))/2`?" *wink* -- Steve
On Sat, Jan 9, 2021 at 10:21 AM Steven D'Aprano <steve@pearwood.info> wrote:
Not every trivial combination of functions needs to be given a built-in or standard library solution. Especially not if doing so will break backwards compatibility.
"I had three numbers in a tuple, and wanted half of twice the first number added to the difference of the remaining two. What do you think about making `len((a,b,c))` return `(2*a + abs(b - c))/2`?"
*wink*
Certainly not! It should be the square root of the sum of the squares of all the numbers!! ChrisA
Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'itertools.chain' object is not reversible
On Sat, Jan 09, 2021 at 10:20:27AM +1100, Steven D'Aprano wrote:
If I replace the list with itertools.chain, I should still get exactly the same results, and I do.
I spoke too soon, I don't. You cannot compose reversed() with chain(). (I don't often make definitive statements without testing them first, but when I do, I'm invariably wrong.) (Ram, I guess you had probably already discovered that reversed and chain can't be composed. It would have been nice for you to have mention this fact in your proposal, rather than expect every single of your readers to rediscover it for themselves.) So in principle we could make the requested change. However it would still be surprising. I think most people would expect that if we could compose reversed and chain, the result would be closest to this: iterables = (a, b, c) # for example reversed(list(chain(*iterables))) rather than your proposal: iterables = (a, b, c) rev_iters = tuple(map(reversed, iterables)) reversed(list(*rev_iters)) or equivalent. Your proposal would still have the surprising consequences that reversing a chain that includes a string would surprisingly split the string into a sequence of characters in reverse order. -- Steve
On 9/01/21 12:54 pm, Steven D'Aprano wrote:
Your proposal would still have the surprising consequences that reversing a chain that includes a string would surprisingly split the string into a sequence of characters in reverse order.
Not that surprising, since chain already splits strings into sequences of characters:
list(itertools.chain("abc", "def")) ['a', 'b', 'c', 'd', 'e', 'f']
-- Greg
participants (7)
-
Chris Angelico -
Greg Ewing -
Oscar Benjamin -
Paul Moore -
Ram Rachum -
Steven D'Aprano -
William Pickard