Re: [Python-Dev] reversed enumerate

Hi Ilya, I'm not sure that this mailing list (Python-Dev) is the right place for this discussion, I think that Python-Ideas (CCed) is the correct place. For the benefit of Python-Ideas, I have left your entire post below, to establish context. [Ilya]
It isn't really well-defined, since enumerate can operate on infinite iterators, and you cannot reverse an infinite stream. Consider: def values(): while True: yield random.random() a, b = reversed(enumerate(values()) What should the first pair of (a, b) be? However, having said that, I think that your idea is not unreasonable. `enumerate(it)` in the most general case isn't reversable, but if `it` is reversable and sized, there's no reason why `enumerate(it)` shouldn't be too. My personal opinion is that this is a fairly obvious and straightforward enhancement, one which (hopefully!) shouldn't require much, if any, debate. I don't think we need a new class for this, I think enhancing enumerate to be reversable if its underlying iterator is reversable makes good sense. But if you can show some concrete use-cases, especially one or two from the standard library, that would help your case. Or some other languages which offer this functionality as standard. On the other hand, I think that there is a fairly lightweight work around. Define a helper function: def countdown(n): while True: yield n n -= 1 then call it like this: # reversed(enumerate(seq)) zip(countdown(len(seq)-1), reversed(seq))) So it isn't terribly hard to work around this. But I agree that it would be nice if enumerate encapsulated this for the caller. One potentially serious question: what should `enumerate.__reversed__` do when given a starting value? reversed(enumerate('abc', 1)) Should that yield...? # treat the start value as a start value (1, 'c'), (0, 'b'), (-1, 'a') # treat the start value as an end value (3, 'c'), (2, 'b'), (1, 'a') Something else? My preference would be to treat the starting value as an ending value. Steven On Wed, Apr 01, 2020 at 08:45:34PM +0200, Ilya Kamenshchikov wrote:

Before jumping in: In many cases, when you want to reverse an enumerate, it’s small and fixed-sized, so there’s a trivial way to do this: Just store the enumerate iterator in a tuple, and tuples are reversible. for idx, value in reversed(tuple(enumerate(stuff))): But of course there are some cases where this isn’t appropriate, like enumerating a fixed-size but huge input.
...
Agreed—but this is just a small piece of a much wider issue. Today, enumerate is always an Iterator. It’s never reversible. But it’s also not sized, or subscriptable, or in-testable, even if you give it inputs that are. And it’s not just enumerate—the same is true for map, filter, zip, itertools.islice, itertools.dropwhile, etc. There’s no reason these things couldn’t all be views, just like the existing dict views (and other things like memoryview and third-party things like numpy array slices). In fact, they already are in Swift, and will be in C++20.
Actually, that doesn’t work—it has to be Sized as well. More generally, it’s rarely _quite_ as simple as just “views support the same operations as the things they view”. An enumerate can be a Sequence if its input is, but a filter can’t. A map with multiple inputs isn’t Reversible unless they’re all not just Reversible but Sized, although a map with only one input doesn’t need it to be Sized. And so on. But none of these things are hard, it’s just a bunch of work to go through all the input types for all the view types and write up the rules. (Or steal them from another language or library that already did that work…)
Agreed. I don’t think we need to wait until someone designs and writes a complete viewtools library and submits it for stdlib inclusion before we can consider adding just one extension to one iterator. But I do think we want to add the one(s) that are most useful if any, not just whichever ones people think of first. I’ve personally wanted to reverse a map or a filter more often than an enumerate, but examples would easily convince me that that’s just me, and reversing enumerate is more needed.
I don’t think this is a problem. When you reversed(tuple(enumerate('abc', 1))) today, what do you get? You presumably don’t even need to look that up or try it out. It would be pretty confusing if it were different without the tuple.

So, what is the conclusion? I also think reversed(enumerate(some_seq)) will be very useful in many cases. It should: 1) work the same as reversed(tuple(enumerate(...))) for "reversible" objects as argument of enumerate, 2) raise TypeError if the object is not reversible. Or, another option would be adding a "step" keyword argument to enumerate. Then, reversing enumerate would be much easier, like this: enumerate(reversed(some_seq), step=-1)

I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences. I think it's good idea, but it adds complexity and work, which I guess needs to be justified on a case-by-case basis. In short, this has nothing to do with reversed. If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34], which could also be useful. I think it makes Python slightly more perfect and more beautiful. Best, Neil On Thursday, January 21, 2021 at 6:23:20 AM UTC-5 Nuri Jung wrote:

On Fri, Jan 22, 2021 at 5:21 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences. I think it's good idea, but it adds complexity and work, which I guess needs to be justified on a case-by-case basis.
In short, this has nothing to do with reversed. If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34], which could also be useful. I think it makes Python slightly more perfect and more beautiful.
list(enumerate(some_list)) will give you a sequence, if that's what you want. ChrisA

On Thu, Jan 21, 2021 at 1:26 PM Chris Angelico <rosuav@gmail.com> wrote:
Right. And reversed(list(enumerate(some_list)) also works. The point is that if you return sequence views (as Andrew mentioned), you don't need to cast to list explicitly, and you don't pay the computational cost of that either.

On Fri, Jan 22, 2021 at 5:44 AM Neil Girdhar <mistersheik@gmail.com> wrote:
With enumerate, you might be able to make a lazy sequence, but then people will ask for map to be able to return a sequence too - and that's going to definitely run into problems. It's much safer to be explicit about it: if you want a list, ask for a list. In the rare instances where you actually need an enumerated lazy list (ie where the cost of an eager list is too high AND it needs to be enumerated), it's not that hard to whip up a thing that gives back the index as well: class Enumerated: def __init__(self, basis): self.basis = basis def __getitem__(self, idx): return (idx, self.basis[idx]) def __len__(self): return len(self.basis) That ought to do most or all of what you want. It's iterable and reversible, and you can index it directly. ChrisA

On Thu, Jan 21, 2021 at 10:43 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I'm not so sure -- I don't think I want a sequence returned. In fact, in this case, just the opposite. If you wanted a sequence, you could simply do: reversed(list(enumerate(the_iterable))) Rather, you want (or I want, anyway) is for various iterators to be able to do things that require a Sequence, when given a Sequence. Or, maybe not even a full Sequence, but rather, some particular aspect of a sequence -- like the length, or indexability. enumerate() with a reversed flag is one. Another, I'd like is islice to support negative indexing for indexing from the end. There are probably many others. I'd love to see these supported. For this example: enumerate(an_iterable, reversed=True) would "just work" if an_iterable had a __len__ and a __getitem__. It would raise a TypeError if passed a non-lengthed, or non-indexable iterable. -CHB
In short, this has nothing to do with reversed.
I agree.
If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34],
As Chris A mentioned, it's quite easy to wrap list() or tuple() around it if you want that. - Chris B -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Thu, Jan 21, 2021 at 3:13 PM Christopher Barker <pythonchb@gmail.com> wrote:
Don't you think this is over-complicated? It's possible for reversed(enumerate(...)) to just work if enumerate of a sequence were to return a sequence view. Then you would also get all the other sequence operations for free like enumerate(...)[23:27], len(enumerate(...)), etc.
Yes, you can alternatively wrap with list and pay the computational cost. However, as was suggested by Andrew in this thread, but has come up many times before, the native iteration tools could return views. This saves the user having to build her own custom classes (along with tests, etc.). I happen to think this is simpler and more beautiful. Best, Neil

On Fri, Jan 22, 2021 at 8:31 AM Neil Girdhar <mistersheik@gmail.com> wrote:
That would break backward compatibility. The behaviour of enumerate() at the moment isn't compatible with it just suddenly being a sequence. It would have to have some way to be told "please return a sequence". We currently have a way to say "please return an eager sequence", so the only thing we don't easily have is "please return a lazy sequence", and that's possible with about four lines of custom class. Note that slicing is NOT easy. The proposed semantics for a reversed enumeration would make slicing extremely odd.
It wouldn't work. With enumerate, you could get away with returning a view, but let's look at map. Should that return a view? If so, does map(...)[4] cache the result or not? If it does, you're paying the memory cost of (potentially) the full eager list, in which case you may as well just make the list. If it doesn't, you break expectations badly by having it call the function more than once, potentially giving different results. Either way, it's far better to write your own, giving you the freedom to define the semantics the way you want to. It's even worse with filter(). It can't possibly know which is the 5th element without counting from the start ("hmm, that's Fire... Earth... Water... Air... Boron. Ah! Boron!"). What about itertools.tee()? I think the whole point of that is to hybridize laziness and caching, but if you're going to give it a sequence, you may as well just get multiple iterators. I'm eyeballing itertools and really not seeing any that would benefit from magically returning sequences. Several of them specifically exist to provide sequence-like behaviour on non-sequence iterables (eg islice), and others don't have a direct correlation between input index and output index (eg dropwhile) or have other discrepancies (eg takewhile). Rather than try to propose that ALL native iteration tools return views, find one that would actually benefit, and propose that. Unfortunately, I think you'll run into the same problem that it's no longer backward compatible, so it's still going to need a spelling to request that it behave in sequence view mode, which is probably best written as something like "import seqview; seqview.map(...)". ChrisA

On Thu, Jan 21, 2021, at 18:48, Chris Angelico wrote:
Note that slicing is NOT easy. The proposed semantics for a reversed enumeration would make slicing extremely odd.
What proposed semantics? You were the one who posted a pure-python implementation that didn't bother to implement slicing. It's easy enough to add slicing to your Enumerated class concept, though. class Enumerated: def __init__(self, basis, indices=None): self.basis = basis self.indices = indices if indices is not None else range(len(basis)) def __getitem__(self, idx): if isinstance(idx, slice): return Enumerated(self.basis, indices=self.indices[idx]) else: return (self.indices[idx], self.basis[self.indices[idx]]) def __len__(self): return len(self.indices)

On Fri, Jan 22, 2021 at 11:10 AM Random832 <random832@fastmail.com> wrote:
Ah, my bad. This thread started out on python-dev before migrating to -ideas, and I believe the proposed semantics may have been on one of the early -dev posts. The proposal was, effectively, to be the same as enumerating, making a concrete list, and then reversing (only, without actually building the list, of course). So enumerate("abc") yields [(0,'a'),(1,'b'), (2,'c')], and iterating backwards should yield those same tuples in reverse order. But if you take the reversed enumeration, iterate over it a bit, and then slice it, the semantics will be extremely bizarre.
Yeah, I don't think that'll work if you slice more than once, especially with some iteration in between. The precise semantics for mixing iteration and slicing are complex enough and variable enough that you may as well just go for a concrete list at that point. ChrisA

On Thu, Jan 21, 2021 at 7:19 PM Chris Angelico <rosuav@gmail.com> wrote:
Can you give an example of what you mean?--because I'm pretty sure that we're not talking about the same thing at all. If enumerate(some_sequence) returns a sequence view, iterating over that sequence view does not advance it—just like how DictViews are not altered by iteration. Same thing if reversed(some_sequence) returns a sequence view. So, reversed(enumerate("abc"))[2] would just be (0, 'a'). You're right that there are some minor incongruities. The only one I can think of off-hand is bool(enumerate(some_sequence)) would be false when the sequence is empty. As for candidates for sequence views, I'd like to see reversed and enumerate to start since I can remember having to cast to list many times and it would be convenient not to have to. From itertools, it would be useful for thecombinatorics operators (combinations, permutations, product, etc.) to be sequence views. In fact, there are convenience functions for looking them up by index in more-itertools as this comes up a lot (at least it did for me back when I did contests).

On Fri, Jan 22, 2021 at 11:48 AM Neil Girdhar <mistersheik@gmail.com> wrote:
Then that's backward incompatible with current behaviour, and cannot be done just as "it's a sequence so return a sequence view". Hence the suggestion of a parameter to control this - or, my preferred way: a "seqview" module from which you can import a sequence-view-only version of various iterator functions. ChrisA

On Thu, Jan 21, 2021, at 19:19, Chris Angelico wrote:
Yeah, I don't think that'll work if you slice more than once, especially with some iteration in between.
I think part of the point of this implementation [or the other person's suggestion of having a "collection view", which amounts to the same thing] is that iterating wouldn't consume items anymore. Which does make it incompatible with enumerate currently returning an iterator, though, in situations where the iteration is dropped and picked up again.
I do think it's possible to have well-defined semantics [next() could simply return self[0] and increment a value to be added to input indices], but it's probably not worth it. I'd say this and e.g. similar map/zip functions belong in a "sequencetools" module rather than replacing the builtin, so that compatibility in these situations doesn't need to be maintained.

Before jumping in: In many cases, when you want to reverse an enumerate, it’s small and fixed-sized, so there’s a trivial way to do this: Just store the enumerate iterator in a tuple, and tuples are reversible. for idx, value in reversed(tuple(enumerate(stuff))): But of course there are some cases where this isn’t appropriate, like enumerating a fixed-size but huge input.
...
Agreed—but this is just a small piece of a much wider issue. Today, enumerate is always an Iterator. It’s never reversible. But it’s also not sized, or subscriptable, or in-testable, even if you give it inputs that are. And it’s not just enumerate—the same is true for map, filter, zip, itertools.islice, itertools.dropwhile, etc. There’s no reason these things couldn’t all be views, just like the existing dict views (and other things like memoryview and third-party things like numpy array slices). In fact, they already are in Swift, and will be in C++20.
Actually, that doesn’t work—it has to be Sized as well. More generally, it’s rarely _quite_ as simple as just “views support the same operations as the things they view”. An enumerate can be a Sequence if its input is, but a filter can’t. A map with multiple inputs isn’t Reversible unless they’re all not just Reversible but Sized, although a map with only one input doesn’t need it to be Sized. And so on. But none of these things are hard, it’s just a bunch of work to go through all the input types for all the view types and write up the rules. (Or steal them from another language or library that already did that work…)
Agreed. I don’t think we need to wait until someone designs and writes a complete viewtools library and submits it for stdlib inclusion before we can consider adding just one extension to one iterator. But I do think we want to add the one(s) that are most useful if any, not just whichever ones people think of first. I’ve personally wanted to reverse a map or a filter more often than an enumerate, but examples would easily convince me that that’s just me, and reversing enumerate is more needed.
I don’t think this is a problem. When you reversed(tuple(enumerate('abc', 1))) today, what do you get? You presumably don’t even need to look that up or try it out. It would be pretty confusing if it were different without the tuple.

So, what is the conclusion? I also think reversed(enumerate(some_seq)) will be very useful in many cases. It should: 1) work the same as reversed(tuple(enumerate(...))) for "reversible" objects as argument of enumerate, 2) raise TypeError if the object is not reversible. Or, another option would be adding a "step" keyword argument to enumerate. Then, reversing enumerate would be much easier, like this: enumerate(reversed(some_seq), step=-1)

I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences. I think it's good idea, but it adds complexity and work, which I guess needs to be justified on a case-by-case basis. In short, this has nothing to do with reversed. If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34], which could also be useful. I think it makes Python slightly more perfect and more beautiful. Best, Neil On Thursday, January 21, 2021 at 6:23:20 AM UTC-5 Nuri Jung wrote:

On Fri, Jan 22, 2021 at 5:21 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences. I think it's good idea, but it adds complexity and work, which I guess needs to be justified on a case-by-case basis.
In short, this has nothing to do with reversed. If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34], which could also be useful. I think it makes Python slightly more perfect and more beautiful.
list(enumerate(some_list)) will give you a sequence, if that's what you want. ChrisA

On Thu, Jan 21, 2021 at 1:26 PM Chris Angelico <rosuav@gmail.com> wrote:
Right. And reversed(list(enumerate(some_list)) also works. The point is that if you return sequence views (as Andrew mentioned), you don't need to cast to list explicitly, and you don't pay the computational cost of that either.

On Fri, Jan 22, 2021 at 5:44 AM Neil Girdhar <mistersheik@gmail.com> wrote:
With enumerate, you might be able to make a lazy sequence, but then people will ask for map to be able to return a sequence too - and that's going to definitely run into problems. It's much safer to be explicit about it: if you want a list, ask for a list. In the rare instances where you actually need an enumerated lazy list (ie where the cost of an eager list is too high AND it needs to be enumerated), it's not that hard to whip up a thing that gives back the index as well: class Enumerated: def __init__(self, basis): self.basis = basis def __getitem__(self, idx): return (idx, self.basis[idx]) def __len__(self): return len(self.basis) That ought to do most or all of what you want. It's iterable and reversible, and you can index it directly. ChrisA

On Thu, Jan 21, 2021 at 10:43 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I'm not so sure -- I don't think I want a sequence returned. In fact, in this case, just the opposite. If you wanted a sequence, you could simply do: reversed(list(enumerate(the_iterable))) Rather, you want (or I want, anyway) is for various iterators to be able to do things that require a Sequence, when given a Sequence. Or, maybe not even a full Sequence, but rather, some particular aspect of a sequence -- like the length, or indexability. enumerate() with a reversed flag is one. Another, I'd like is islice to support negative indexing for indexing from the end. There are probably many others. I'd love to see these supported. For this example: enumerate(an_iterable, reversed=True) would "just work" if an_iterable had a __len__ and a __getitem__. It would raise a TypeError if passed a non-lengthed, or non-indexable iterable. -CHB
In short, this has nothing to do with reversed.
I agree.
If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34],
As Chris A mentioned, it's quite easy to wrap list() or tuple() around it if you want that. - Chris B -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Thu, Jan 21, 2021 at 3:13 PM Christopher Barker <pythonchb@gmail.com> wrote:
Don't you think this is over-complicated? It's possible for reversed(enumerate(...)) to just work if enumerate of a sequence were to return a sequence view. Then you would also get all the other sequence operations for free like enumerate(...)[23:27], len(enumerate(...)), etc.
Yes, you can alternatively wrap with list and pay the computational cost. However, as was suggested by Andrew in this thread, but has come up many times before, the native iteration tools could return views. This saves the user having to build her own custom classes (along with tests, etc.). I happen to think this is simpler and more beautiful. Best, Neil

On Fri, Jan 22, 2021 at 8:31 AM Neil Girdhar <mistersheik@gmail.com> wrote:
That would break backward compatibility. The behaviour of enumerate() at the moment isn't compatible with it just suddenly being a sequence. It would have to have some way to be told "please return a sequence". We currently have a way to say "please return an eager sequence", so the only thing we don't easily have is "please return a lazy sequence", and that's possible with about four lines of custom class. Note that slicing is NOT easy. The proposed semantics for a reversed enumeration would make slicing extremely odd.
It wouldn't work. With enumerate, you could get away with returning a view, but let's look at map. Should that return a view? If so, does map(...)[4] cache the result or not? If it does, you're paying the memory cost of (potentially) the full eager list, in which case you may as well just make the list. If it doesn't, you break expectations badly by having it call the function more than once, potentially giving different results. Either way, it's far better to write your own, giving you the freedom to define the semantics the way you want to. It's even worse with filter(). It can't possibly know which is the 5th element without counting from the start ("hmm, that's Fire... Earth... Water... Air... Boron. Ah! Boron!"). What about itertools.tee()? I think the whole point of that is to hybridize laziness and caching, but if you're going to give it a sequence, you may as well just get multiple iterators. I'm eyeballing itertools and really not seeing any that would benefit from magically returning sequences. Several of them specifically exist to provide sequence-like behaviour on non-sequence iterables (eg islice), and others don't have a direct correlation between input index and output index (eg dropwhile) or have other discrepancies (eg takewhile). Rather than try to propose that ALL native iteration tools return views, find one that would actually benefit, and propose that. Unfortunately, I think you'll run into the same problem that it's no longer backward compatible, so it's still going to need a spelling to request that it behave in sequence view mode, which is probably best written as something like "import seqview; seqview.map(...)". ChrisA

On Thu, Jan 21, 2021, at 18:48, Chris Angelico wrote:
Note that slicing is NOT easy. The proposed semantics for a reversed enumeration would make slicing extremely odd.
What proposed semantics? You were the one who posted a pure-python implementation that didn't bother to implement slicing. It's easy enough to add slicing to your Enumerated class concept, though. class Enumerated: def __init__(self, basis, indices=None): self.basis = basis self.indices = indices if indices is not None else range(len(basis)) def __getitem__(self, idx): if isinstance(idx, slice): return Enumerated(self.basis, indices=self.indices[idx]) else: return (self.indices[idx], self.basis[self.indices[idx]]) def __len__(self): return len(self.indices)

On Fri, Jan 22, 2021 at 11:10 AM Random832 <random832@fastmail.com> wrote:
Ah, my bad. This thread started out on python-dev before migrating to -ideas, and I believe the proposed semantics may have been on one of the early -dev posts. The proposal was, effectively, to be the same as enumerating, making a concrete list, and then reversing (only, without actually building the list, of course). So enumerate("abc") yields [(0,'a'),(1,'b'), (2,'c')], and iterating backwards should yield those same tuples in reverse order. But if you take the reversed enumeration, iterate over it a bit, and then slice it, the semantics will be extremely bizarre.
Yeah, I don't think that'll work if you slice more than once, especially with some iteration in between. The precise semantics for mixing iteration and slicing are complex enough and variable enough that you may as well just go for a concrete list at that point. ChrisA

On Thu, Jan 21, 2021 at 7:19 PM Chris Angelico <rosuav@gmail.com> wrote:
Can you give an example of what you mean?--because I'm pretty sure that we're not talking about the same thing at all. If enumerate(some_sequence) returns a sequence view, iterating over that sequence view does not advance it—just like how DictViews are not altered by iteration. Same thing if reversed(some_sequence) returns a sequence view. So, reversed(enumerate("abc"))[2] would just be (0, 'a'). You're right that there are some minor incongruities. The only one I can think of off-hand is bool(enumerate(some_sequence)) would be false when the sequence is empty. As for candidates for sequence views, I'd like to see reversed and enumerate to start since I can remember having to cast to list many times and it would be convenient not to have to. From itertools, it would be useful for thecombinatorics operators (combinations, permutations, product, etc.) to be sequence views. In fact, there are convenience functions for looking them up by index in more-itertools as this comes up a lot (at least it did for me back when I did contests).

On Fri, Jan 22, 2021 at 11:48 AM Neil Girdhar <mistersheik@gmail.com> wrote:
Then that's backward incompatible with current behaviour, and cannot be done just as "it's a sequence so return a sequence view". Hence the suggestion of a parameter to control this - or, my preferred way: a "seqview" module from which you can import a sequence-view-only version of various iterator functions. ChrisA

On Thu, Jan 21, 2021, at 19:19, Chris Angelico wrote:
Yeah, I don't think that'll work if you slice more than once, especially with some iteration in between.
I think part of the point of this implementation [or the other person's suggestion of having a "collection view", which amounts to the same thing] is that iterating wouldn't consume items anymore. Which does make it incompatible with enumerate currently returning an iterator, though, in situations where the iteration is dropped and picked up again.
I do think it's possible to have well-defined semantics [next() could simply return self[0] and increment a value to be added to input indices], but it's probably not worth it. I'd say this and e.g. similar map/zip functions belong in a "sequencetools" module rather than replacing the builtin, so that compatibility in these situations doesn't need to be maintained.
participants (7)
-
Andrew Barnert
-
Chris Angelico
-
Christopher Barker
-
Neil Girdhar
-
Nuri Jung
-
Random832
-
Steven D'Aprano