Re: [Python-Dev] reversed enumerate
Hi Ilya, I'm not sure that this mailing list (Python-Dev) is the right place for this discussion, I think that Python-Ideas (CCed) is the correct place. For the benefit of Python-Ideas, I have left your entire post below, to establish context. [Ilya]
I needed reversed(enumerate(x: list)) in my code, and have discovered that it wound't work. This is disappointing because operation is well defined.
It isn't really well-defined, since enumerate can operate on infinite iterators, and you cannot reverse an infinite stream. Consider: def values(): while True: yield random.random() a, b = reversed(enumerate(values()) What should the first pair of (a, b) be? However, having said that, I think that your idea is not unreasonable. `enumerate(it)` in the most general case isn't reversable, but if `it` is reversable and sized, there's no reason why `enumerate(it)` shouldn't be too. My personal opinion is that this is a fairly obvious and straightforward enhancement, one which (hopefully!) shouldn't require much, if any, debate. I don't think we need a new class for this, I think enhancing enumerate to be reversable if its underlying iterator is reversable makes good sense. But if you can show some concrete use-cases, especially one or two from the standard library, that would help your case. Or some other languages which offer this functionality as standard. On the other hand, I think that there is a fairly lightweight work around. Define a helper function: def countdown(n): while True: yield n n -= 1 then call it like this: # reversed(enumerate(seq)) zip(countdown(len(seq)-1), reversed(seq))) So it isn't terribly hard to work around this. But I agree that it would be nice if enumerate encapsulated this for the caller. One potentially serious question: what should `enumerate.__reversed__` do when given a starting value? reversed(enumerate('abc', 1)) Should that yield...? # treat the start value as a start value (1, 'c'), (0, 'b'), (-1, 'a') # treat the start value as an end value (3, 'c'), (2, 'b'), (1, 'a') Something else? My preference would be to treat the starting value as an ending value. Steven On Wed, Apr 01, 2020 at 08:45:34PM +0200, Ilya Kamenshchikov wrote:
Hi,
I needed reversed(enumerate(x: list)) in my code, and have discovered that it wound't work. This is disappointing because operation is well defined. It is also well defined for str type, range, and - in principle, but not yet in practice - on dictionary iterators - keys(), values(), items() as dictionaries are ordered now. It would also be well defined on any user type implementing __iter__, __len__, __reversed__ - think numpy arrays, some pandas dataframes, tensors.
That's plenty of usecases, therefore I guess it would be quite useful to avoid hacky / inefficient solutions like described here: https://code.activestate.com/lists/python-list/706205/.
If deemed useful, I would be interested in implementing this, maybe together with __reversed__ on dict keys, values, items.
Best Regards, -- Ilya Kamen
----------- p.s.
*Sketch* of what I am proposing:
class reversible_enumerate:
def __init__(self, iterable): self.iterable = iterable self.ctr = 0
def __iter__(self): for e in self.iterable: yield self.ctr, e self.ctr += 1
def __reversed__(self): try: ri = reversed(self.iterable) except Exception as e: raise Exception( "enumerate can only be reversed if iterable to enumerate can be reversed and has defined length." ) from e
try: l = len(self.iterable) except Exception as e: raise Exception( "enumerate can only be reversed if iterable to enumerate can be reversed and has defined length." ) from e
indexes = range(l-1, -1, -1) for i, e in zip(indexes, ri): yield i, e
for i, c in reversed(reversible_enumerate("Hello World")): print(i, c)
for i, c in reversed(reversible_enumerate([11, 22, 33])):
print(i, c)
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NDDKDUDH... Code of Conduct: http://python.org/psf/codeofconduct/
Before jumping in: In many cases, when you want to reverse an enumerate, it’s small and fixed-sized, so there’s a trivial way to do this: Just store the enumerate iterator in a tuple, and tuples are reversible. for idx, value in reversed(tuple(enumerate(stuff))): But of course there are some cases where this isn’t appropriate, like enumerating a fixed-size but huge input.
On Apr 1, 2020, at 19:23, Steven D'Aprano <steve@pearwood.info> wrote:
[Ilya]
I needed reversed(enumerate(x: list)) in my code, and have discovered that it wound't work. This is disappointing because operation is well defined.
It isn't really well-defined, since enumerate can operate on infinite iterators, and you cannot reverse an infinite stream.
...
However, having said that, I think that your idea is not unreasonable. `enumerate(it)` in the most general case isn't reversable, but if `it` is reversable and sized, there's no reason why `enumerate(it)` shouldn't be too.
Agreed—but this is just a small piece of a much wider issue. Today, enumerate is always an Iterator. It’s never reversible. But it’s also not sized, or subscriptable, or in-testable, even if you give it inputs that are. And it’s not just enumerate—the same is true for map, filter, zip, itertools.islice, itertools.dropwhile, etc. There’s no reason these things couldn’t all be views, just like the existing dict views (and other things like memoryview and third-party things like numpy array slices). In fact, they already are in Swift, and will be in C++20.
My personal opinion is that this is a fairly obvious and straightforward enhancement, one which (hopefully!) shouldn't require much, if any, debate. I don't think we need a new class for this, I think enhancing enumerate to be reversable if its underlying iterator is reversable makes good sense.
Actually, that doesn’t work—it has to be Sized as well. More generally, it’s rarely _quite_ as simple as just “views support the same operations as the things they view”. An enumerate can be a Sequence if its input is, but a filter can’t. A map with multiple inputs isn’t Reversible unless they’re all not just Reversible but Sized, although a map with only one input doesn’t need it to be Sized. And so on. But none of these things are hard, it’s just a bunch of work to go through all the input types for all the view types and write up the rules. (Or steal them from another language or library that already did that work…)
But if you can show some concrete use-cases, especially one or two from the standard library, that would help your case. Or some other languages which offer this functionality as standard.
Agreed. I don’t think we need to wait until someone designs and writes a complete viewtools library and submits it for stdlib inclusion before we can consider adding just one extension to one iterator. But I do think we want to add the one(s) that are most useful if any, not just whichever ones people think of first. I’ve personally wanted to reverse a map or a filter more often than an enumerate, but examples would easily convince me that that’s just me, and reversing enumerate is more needed.
One potentially serious question: what should `enumerate.__reversed__` do when given a starting value?
reversed(enumerate('abc', 1))
I don’t think this is a problem. When you reversed(tuple(enumerate('abc', 1))) today, what do you get? You presumably don’t even need to look that up or try it out. It would be pretty confusing if it were different without the tuple.
So, what is the conclusion? I also think reversed(enumerate(some_seq)) will be very useful in many cases. It should: 1) work the same as reversed(tuple(enumerate(...))) for "reversible" objects as argument of enumerate, 2) raise TypeError if the object is not reversible. Or, another option would be adding a "step" keyword argument to enumerate. Then, reversing enumerate would be much easier, like this: enumerate(reversed(some_seq), step=-1)
I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences. I think it's good idea, but it adds complexity and work, which I guess needs to be justified on a case-by-case basis. In short, this has nothing to do with reversed. If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34], which could also be useful. I think it makes Python slightly more perfect and more beautiful. Best, Neil On Thursday, January 21, 2021 at 6:23:20 AM UTC-5 Nuri Jung wrote:
So, what is the conclusion? I also think reversed(enumerate(some_seq)) will be very useful in many cases. It should: 1) work the same as reversed(tuple(enumerate(...))) for "reversible" objects as argument of enumerate, 2) raise TypeError if the object is not reversible.
Or, another option would be adding a "step" keyword argument to enumerate. Then, reversing enumerate would be much easier, like this: enumerate(reversed(some_seq), step=-1) _______________________________________________ Python-ideas mailing list -- python...@python.org To unsubscribe send an email to python-id...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python...@python.org/message/BW2XKRZ4T... <https://mail.python.org/archives/list/python-ideas@python.org/message/BW2XKR...> Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Jan 22, 2021 at 5:21 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences. I think it's good idea, but it adds complexity and work, which I guess needs to be justified on a case-by-case basis.
In short, this has nothing to do with reversed. If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34], which could also be useful. I think it makes Python slightly more perfect and more beautiful.
list(enumerate(some_list)) will give you a sequence, if that's what you want. ChrisA
On Thu, Jan 21, 2021 at 1:26 PM Chris Angelico <rosuav@gmail.com> wrote:
On Fri, Jan 22, 2021 at 5:21 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences. I think it's good idea, but it adds complexity and work, which I guess needs to be justified on a case-by-case basis.
In short, this has nothing to do with reversed. If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34], which could also be useful. I think it makes Python slightly more perfect and more beautiful.
list(enumerate(some_list)) will give you a sequence, if that's what you want.
Right. And reversed(list(enumerate(some_list)) also works. The point is that if you return sequence views (as Andrew mentioned), you don't need to cast to list explicitly, and you don't pay the computational cost of that either.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WW3JHS... Code of Conduct: http://python.org/psf/codeofconduct/
--
--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/3wGART2ECXQ/unsubscribe. To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/python-ideas/CAPTjJmpPchBQEWVP1Mt8UW3aaquT....
On Fri, Jan 22, 2021 at 5:44 AM Neil Girdhar <mistersheik@gmail.com> wrote:
On Thu, Jan 21, 2021 at 1:26 PM Chris Angelico <rosuav@gmail.com> wrote:
On Fri, Jan 22, 2021 at 5:21 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences. I think it's good idea, but it adds complexity and work, which I guess needs to be justified on a case-by-case basis.
In short, this has nothing to do with reversed. If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34], which could also be useful. I think it makes Python slightly more perfect and more beautiful.
list(enumerate(some_list)) will give you a sequence, if that's what you want.
Right. And reversed(list(enumerate(some_list)) also works. The point is that if you return sequence views (as Andrew mentioned), you don't need to cast to list explicitly, and you don't pay the computational cost of that either.
With enumerate, you might be able to make a lazy sequence, but then people will ask for map to be able to return a sequence too - and that's going to definitely run into problems. It's much safer to be explicit about it: if you want a list, ask for a list. In the rare instances where you actually need an enumerated lazy list (ie where the cost of an eager list is too high AND it needs to be enumerated), it's not that hard to whip up a thing that gives back the index as well: class Enumerated: def __init__(self, basis): self.basis = basis def __getitem__(self, idx): return (idx, self.basis[idx]) def __len__(self): return len(self.basis) That ought to do most or all of what you want. It's iterable and reversible, and you can index it directly. ChrisA
On Thu, Jan 21, 2021 at 10:43 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences.
I'm not so sure -- I don't think I want a sequence returned. In fact, in this case, just the opposite. If you wanted a sequence, you could simply do: reversed(list(enumerate(the_iterable))) Rather, you want (or I want, anyway) is for various iterators to be able to do things that require a Sequence, when given a Sequence. Or, maybe not even a full Sequence, but rather, some particular aspect of a sequence -- like the length, or indexability. enumerate() with a reversed flag is one. Another, I'd like is islice to support negative indexing for indexing from the end. There are probably many others. I'd love to see these supported. For this example: enumerate(an_iterable, reversed=True) would "just work" if an_iterable had a __len__ and a __getitem__. It would raise a TypeError if passed a non-lengthed, or non-indexable iterable. -CHB
In short, this has nothing to do with reversed.
I agree.
If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34],
As Chris A mentioned, it's quite easy to wrap list() or tuple() around it if you want that. - Chris B -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Thu, Jan 21, 2021 at 3:13 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Thu, Jan 21, 2021 at 10:43 AM Neil Girdhar <mistersheik@gmail.com> wrote:
I've seen this proposed here before. The general idea is that some iterator transformations (like enumerate) should return sequences when they're applied to sequences.
I'm not so sure -- I don't think I want a sequence returned. In fact, in this case, just the opposite. If you wanted a sequence, you could simply do:
reversed(list(enumerate(the_iterable)))
Rather, you want (or I want, anyway) is for various iterators to be able to do things that require a Sequence, when given a Sequence. Or, maybe not even a full Sequence, but rather, some particular aspect of a sequence -- like the length, or indexability.
enumerate() with a reversed flag is one. Another, I'd like is islice to support negative indexing for indexing from the end. There are probably many others.
I'd love to see these supported. For this example:
enumerate(an_iterable, reversed=True)
Don't you think this is over-complicated? It's possible for reversed(enumerate(...)) to just work if enumerate of a sequence were to return a sequence view. Then you would also get all the other sequence operations for free like enumerate(...)[23:27], len(enumerate(...)), etc.
would "just work" if an_iterable had a __len__ and a __getitem__.
It would raise a TypeError if passed a non-lengthed, or non-indexable iterable.
-CHB
In short, this has nothing to do with reversed.
I agree.
If you made enumerate return a sequence when its input is a sequence, you would also be able to do enumerate(some_list)[34],
As Chris A mentioned, it's quite easy to wrap list() or tuple() around it if you want that.
Yes, you can alternatively wrap with list and pay the computational cost. However, as was suggested by Andrew in this thread, but has come up many times before, the native iteration tools could return views. This saves the user having to build her own custom classes (along with tests, etc.). I happen to think this is simpler and more beautiful. Best, Neil
- Chris B
-- Christopher Barker, PhD (Chris)
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Fri, Jan 22, 2021 at 8:31 AM Neil Girdhar <mistersheik@gmail.com> wrote:
It's possible for reversed(enumerate(...)) to just work if enumerate of a sequence were to return a sequence view. Then you would also get all the other sequence operations for free like enumerate(...)[23:27], len(enumerate(...)), etc.
That would break backward compatibility. The behaviour of enumerate() at the moment isn't compatible with it just suddenly being a sequence. It would have to have some way to be told "please return a sequence". We currently have a way to say "please return an eager sequence", so the only thing we don't easily have is "please return a lazy sequence", and that's possible with about four lines of custom class. Note that slicing is NOT easy. The proposed semantics for a reversed enumeration would make slicing extremely odd.
As Chris A mentioned, it's quite easy to wrap list() or tuple() around it if you want that.
Yes, you can alternatively wrap with list and pay the computational cost. However, as was suggested by Andrew in this thread, but has come up many times before, the native iteration tools could return views. This saves the user having to build her own custom classes (along with tests, etc.). I happen to think this is simpler and more beautiful.
It wouldn't work. With enumerate, you could get away with returning a view, but let's look at map. Should that return a view? If so, does map(...)[4] cache the result or not? If it does, you're paying the memory cost of (potentially) the full eager list, in which case you may as well just make the list. If it doesn't, you break expectations badly by having it call the function more than once, potentially giving different results. Either way, it's far better to write your own, giving you the freedom to define the semantics the way you want to. It's even worse with filter(). It can't possibly know which is the 5th element without counting from the start ("hmm, that's Fire... Earth... Water... Air... Boron. Ah! Boron!"). What about itertools.tee()? I think the whole point of that is to hybridize laziness and caching, but if you're going to give it a sequence, you may as well just get multiple iterators. I'm eyeballing itertools and really not seeing any that would benefit from magically returning sequences. Several of them specifically exist to provide sequence-like behaviour on non-sequence iterables (eg islice), and others don't have a direct correlation between input index and output index (eg dropwhile) or have other discrepancies (eg takewhile). Rather than try to propose that ALL native iteration tools return views, find one that would actually benefit, and propose that. Unfortunately, I think you'll run into the same problem that it's no longer backward compatible, so it's still going to need a spelling to request that it behave in sequence view mode, which is probably best written as something like "import seqview; seqview.map(...)". ChrisA
On Thu, Jan 21, 2021, at 18:48, Chris Angelico wrote:
Note that slicing is NOT easy. The proposed semantics for a reversed enumeration would make slicing extremely odd.
What proposed semantics? You were the one who posted a pure-python implementation that didn't bother to implement slicing. It's easy enough to add slicing to your Enumerated class concept, though. class Enumerated: def __init__(self, basis, indices=None): self.basis = basis self.indices = indices if indices is not None else range(len(basis)) def __getitem__(self, idx): if isinstance(idx, slice): return Enumerated(self.basis, indices=self.indices[idx]) else: return (self.indices[idx], self.basis[self.indices[idx]]) def __len__(self): return len(self.indices)
On Fri, Jan 22, 2021 at 11:10 AM Random832 <random832@fastmail.com> wrote:
On Thu, Jan 21, 2021, at 18:48, Chris Angelico wrote:
Note that slicing is NOT easy. The proposed semantics for a reversed enumeration would make slicing extremely odd.
What proposed semantics? You were the one who posted a pure-python implementation that didn't bother to implement slicing.
Ah, my bad. This thread started out on python-dev before migrating to -ideas, and I believe the proposed semantics may have been on one of the early -dev posts. The proposal was, effectively, to be the same as enumerating, making a concrete list, and then reversing (only, without actually building the list, of course). So enumerate("abc") yields [(0,'a'),(1,'b'), (2,'c')], and iterating backwards should yield those same tuples in reverse order. But if you take the reversed enumeration, iterate over it a bit, and then slice it, the semantics will be extremely bizarre.
It's easy enough to add slicing to your Enumerated class concept, though.
class Enumerated: def __init__(self, basis, indices=None): self.basis = basis self.indices = indices if indices is not None else range(len(basis)) def __getitem__(self, idx): if isinstance(idx, slice): return Enumerated(self.basis, indices=self.indices[idx]) else: return (self.indices[idx], self.basis[self.indices[idx]]) def __len__(self): return len(self.indices)
Yeah, I don't think that'll work if you slice more than once, especially with some iteration in between. The precise semantics for mixing iteration and slicing are complex enough and variable enough that you may as well just go for a concrete list at that point. ChrisA
On Thu, Jan 21, 2021 at 7:19 PM Chris Angelico <rosuav@gmail.com> wrote:
On Fri, Jan 22, 2021 at 11:10 AM Random832 <random832@fastmail.com> wrote:
On Thu, Jan 21, 2021, at 18:48, Chris Angelico wrote:
Note that slicing is NOT easy. The proposed semantics for a reversed enumeration would make slicing extremely odd.
What proposed semantics? You were the one who posted a pure-python implementation that didn't bother to implement slicing.
Ah, my bad. This thread started out on python-dev before migrating to -ideas, and I believe the proposed semantics may have been on one of the early -dev posts.
The proposal was, effectively, to be the same as enumerating, making a concrete list, and then reversing (only, without actually building the list, of course). So enumerate("abc") yields [(0,'a'),(1,'b'), (2,'c')], and iterating backwards should yield those same tuples in reverse order.
But if you take the reversed enumeration, iterate over it a bit, and then slice it, the semantics will be extremely bizarre.
It's easy enough to add slicing to your Enumerated class concept, though.
class Enumerated: def __init__(self, basis, indices=None): self.basis = basis self.indices = indices if indices is not None else range(len(basis)) def __getitem__(self, idx): if isinstance(idx, slice): return Enumerated(self.basis, indices=self.indices[idx]) else: return (self.indices[idx], self.basis[self.indices[idx]]) def __len__(self): return len(self.indices)
Yeah, I don't think that'll work if you slice more than once, especially with some iteration in between.
The precise semantics for mixing iteration and slicing are complex enough and variable enough that you may as well just go for a concrete list at that point.
Can you give an example of what you mean?--because I'm pretty sure that we're not talking about the same thing at all. If enumerate(some_sequence) returns a sequence view, iterating over that sequence view does not advance it—just like how DictViews are not altered by iteration. Same thing if reversed(some_sequence) returns a sequence view. So, reversed(enumerate("abc"))[2] would just be (0, 'a'). You're right that there are some minor incongruities. The only one I can think of off-hand is bool(enumerate(some_sequence)) would be false when the sequence is empty. As for candidates for sequence views, I'd like to see reversed and enumerate to start since I can remember having to cast to list many times and it would be convenient not to have to. From itertools, it would be useful for thecombinatorics operators (combinations, permutations, product, etc.) to be sequence views. In fact, there are convenience functions for looking them up by index in more-itertools as this comes up a lot (at least it did for me back when I did contests).
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VTSZZK... Code of Conduct: http://python.org/psf/codeofconduct/
--
--- You received this message because you are subscribed to a topic in the Google Groups "python-ideas" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-ideas/3wGART2ECXQ/unsubscribe. To unsubscribe from this group and all its topics, send an email to python-ideas+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/python-ideas/CAPTjJmpboVPni3FimaTh%3DNo_GB....
On Fri, Jan 22, 2021 at 11:48 AM Neil Girdhar <mistersheik@gmail.com> wrote:
If enumerate(some_sequence) returns a sequence view, iterating over that sequence view does not advance it—just like how DictViews are not altered by iteration. Same thing if reversed(some_sequence) returns a sequence view.
Then that's backward incompatible with current behaviour, and cannot be done just as "it's a sequence so return a sequence view". Hence the suggestion of a parameter to control this - or, my preferred way: a "seqview" module from which you can import a sequence-view-only version of various iterator functions. ChrisA
On Thu, Jan 21, 2021, at 19:19, Chris Angelico wrote:
Yeah, I don't think that'll work if you slice more than once, especially with some iteration in between.
I think part of the point of this implementation [or the other person's suggestion of having a "collection view", which amounts to the same thing] is that iterating wouldn't consume items anymore. Which does make it incompatible with enumerate currently returning an iterator, though, in situations where the iteration is dropped and picked up again.
The precise semantics for mixing iteration and slicing are complex enough and variable enough that you may as well just go for a concrete list at that point.
I do think it's possible to have well-defined semantics [next() could simply return self[0] and increment a value to be added to input indices], but it's probably not worth it. I'd say this and e.g. similar map/zip functions belong in a "sequencetools" module rather than replacing the builtin, so that compatibility in these situations doesn't need to be maintained.
participants (7)
-
Andrew Barnert -
Chris Angelico -
Christopher Barker -
Neil Girdhar -
Nuri Jung -
Random832 -
Steven D'Aprano