Mailman 3 Use the plus operator to concatenate iterators - Python-ideas

Use the plus operator to concatenate iterators

older
Re: [Python-ideas] fork - other...

Grayson, Samuel Andrew

5 Aug 2015 5 Aug '15

2:22 a.m.

Concatenation is the most fundamental operation that can be done on iterators. In fact, we already do that with lists. [1, 2, 3] + [4, 5, 6] # evaluates to [1, 2, 3, 4, 5, 6] I propose: iter([1, 2, 3]) + iter([4, 5, 6]) # evaluates to something like itertools.chain(iter([1, 2, 3]), iter([4, 5, 6])) # equivalent to iter([1, 2, 3, 4, 5, 6]) There is some python2 code where: a = dict(zip('abcd', range(4))) isinstance(a.values(), list) alphabet = a.keys() + a.values() In python2, this `alphabet` becomes a list of all values and keys In current python3, this raises: TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_values' But in my proposal, it works just fine. `alphabet` becomes an iterator over all values and keys (similar to the python2 case). Sincerely, Sam G

Attachments:

attachment.htm (text/html — 3.6 KB)

Show replies by thread

Joseph Jevnik

5 Aug 5 Aug

2:43 a.m.

Iterators all all different types though. iter(list) returns a list_iterator type, iter(dict.keys()) returns a dict_keys_iterator type and so on. Is your suggestion that the standard lib types do this? How do we update all of the existing iterators not in the stdlib that do not do this? Finally, how is this better than itertools.chain? On Tue, Aug 4, 2015 at 8:22 PM, Grayson, Samuel Andrew < sag150430@utdallas.edu> wrote:

...

Concatenation is the most fundamental operation that can be done on iterators. In fact, we already do that with lists.

[1, 2, 3] + [4, 5, 6] # evaluates to [1, 2, 3, 4, 5, 6]

I propose:

iter([1, 2, 3]) + iter([4, 5, 6]) # evaluates to something like itertools.chain(iter([1, 2, 3]), iter([4, 5, 6])) # equivalent to iter([1, 2, 3, 4, 5, 6])

There is some python2 code where:

a = dict(zip('abcd', range(4))) isinstance(a.values(), list) alphabet = a.keys() + a.values()

In python2, this `alphabet` becomes a list of all values and keys

In current python3, this raises:

TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_values'

But in my proposal, it works just fine. `alphabet` becomes an iterator over all values and keys (similar to the python2 case).

Sincerely, Sam G

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Alexander Belopolsky

3:01 a.m.

On Tue, Aug 4, 2015 at 8:43 PM, Joseph Jevnik wrote:

...

Iterators all all different types though. iter(list) returns a list_iterator type, iter(dict.keys()) returns a dict_keys_iterator type and so on. Is your suggestion that the standard lib types do this? How do we update all of the existing iterators not in the stdlib that do not do this?

In theory, this can be done inside PyNumber_Add(x, y). It already checks for numbers or sequences and failing that can check for the __next__ method on its first operand and return itertools.chain(x, y).

...

Finally, how is this better than itertools.chain?

Shorter. Especially when you chain more than two iterators. Nevertheless, I am -1 on the idea. It is bad enough that Python abuses + as sequences concatenation operator.

Chris Angelico

3:01 a.m.

On Wed, Aug 5, 2015 at 10:22 AM, Grayson, Samuel Andrew wrote:

...

I propose:

iter([1, 2, 3]) + iter([4, 5, 6]) # evaluates to something like itertools.chain(iter([1, 2, 3]), iter([4, 5, 6])) # equivalent to iter([1, 2, 3, 4, 5, 6])

Try this: class iter: iter = iter # snapshot the original iter() def __init__(self, iterable): self.it = self.iter(iterable) self.next = [] def __iter__(self): return self def __next__(self): while self.next: try: return next(self.it) except StopIteration: self.it, *self.next = self.next return next(self.it) # Allow StopIteration to bubble when it's the last one def __add__(self, other): result = self.__class__(self.it) result.next = self.next + [self.iter(other)] return result As long as you explicitly call iter() on something, you get the ability to add two iterators together. I haven't checked for odd edge cases, but something like this does work, and should work on all Python versions. ChrisA

Joseph Jevnik

3:02 a.m.

That would get _really_ messy with iterators that define an __add__, sequence concat or number add function On Tue, Aug 4, 2015 at 9:01 PM, Alexander Belopolsky < alexander.belopolsky@gmail.com> wrote:

...

...
Iterators all all different types though. iter(list) returns a

...
type, iter(dict.keys()) returns a dict_keys_iterator type and so on. Is your suggestion that the standard lib types do this? How do we update all of

On Tue, Aug 4, 2015 at 8:43 PM, Joseph Jevnik wrote: list_iterator the

...
existing iterators not in the stdlib that do not do this?

In theory, this can be done inside PyNumber_Add(x, y). It already checks for numbers or sequences and failing that can check for the __next__ method on its first operand and return itertools.chain(x, y).

...
Finally, how is this better than itertools.chain?

Shorter. Especially when you chain more than two iterators.

Nevertheless, I am -1 on the idea. It is bad enough that Python abuses + as sequences concatenation operator.

Chris Angelico

3:06 a.m.

On Wed, Aug 5, 2015 at 11:01 AM, Alexander Belopolsky wrote:

...

Nevertheless, I am -1 on the idea. It is bad enough that Python abuses + as sequences concatenation operator.

I'm -1 on this idea, but I disagree that concrete sequence concatenation is bad. (I'm not sure it applies to sequence protocol, incidentally; it's specific to lists and tuples.) Being able to add a list and a list is a perfectly reasonable feature. ChrisA

Alexander Belopolsky

3:17 a.m.

On Tue, Aug 4, 2015 at 9:06 PM, Chris Angelico wrote:

...

(I'm not sure it applies to sequence protocol, incidentally; it's specific to lists and tuples.)

https://docs.python.org/3/c-api/sequence.html#c.PySequence_Concat

Alexander Belopolsky

3:21 a.m.

On Tue, Aug 4, 2015 at 9:06 PM, Chris Angelico wrote:

...

Being able to add a list and a list is a perfectly reasonable feature.

Sure

...

...
...
from numpy import array array([1,2,3]) + array([3,2,1]) array([4, 4, 4])

.. and how do I concatenate those things?

Grayson, Samuel Andrew

3:21 a.m.

In the case that `__add__` is already defined by the iterator, I propose to use that. Otherwise, I propose to concatenate the iterators. This serves two purposes: convenience and backwards-compatibility/consistency. Convenience: Imagine having to do `operator.mul(5, 4)` instead of `5 * 4`, or `list_chain([1, 2, 3], [4, 5, 6])` instead of `[1, 2, 3] + [4, 5, 6]`. Following this pattern for commonly used operators, `range(5) + 'abc'` instead of `itertools.chain(range(5), 'abc'`. (notice also that if you act like a list, but redefine the __add__ method, then the default behavior is overridden. In the same way, I propose that if you redefine the __add__ method, then the proposed default behavior (concatenanation) is overridden. Backwards-compatibility: This helps backwards-compatibility where lists in python2 changed to iterators in python3, everywhere except for concatenation via the plus operator.

Grayson, Samuel Andrew

3:31 a.m.

I propose that if the iterator class overrides the __add__ method, then use that instead of concatenation. Therefore numpy.array([1, 2, 3]) + numpy.array([3, 2, 1]) # evaluates to numpy.array([4, 4, 4]) It is already done with the __mul__ function a = [1, 2, 3] # or a tuple a * 2 # evaluates to [1, 2, 3, 1, 2, 3] b = np.array([1, 2, 3]) b * 2 # evaluates to np.array([2, 4, 6]) Sincerely, Sam G

Joonas Liik

3:32 a.m.

no opinion if its a good idea or not, however, if iterator + iterator... iterator * number

Chris Angelico

3:34 a.m.

On Wed, Aug 5, 2015 at 11:32 AM, Joonas Liik wrote:

...

no opinion if its a good idea or not, however, if iterator + iterator...

iterator * number

Almost completely useless. Multiplying an *iterable* by an integer will often be useful (eg multiplying list by int), but multiplying an *iterator* (or even just adding one to itself) is going to be useless, because the first time through it will exhaust it, and any well-behaved iterator will remain exhausted once it's ever raised StopIteration. (Note that the 'class iter' that I posted earlier is NOT well-behaved. You can add something onto an exhausted iterator and rejuvenate it. It'd take a couple extra lines of code to fix that.) ChrisA

Steven D'Aprano

4:15 a.m.

On Wed, Aug 05, 2015 at 12:22:51AM +0000, Grayson, Samuel Andrew wrote:

...

Concatenation is the most fundamental operation that can be done on iterators.

Surely "get next value" is the most fundamental operation that can be done on interators. Supporting concatenation is not even part of the definition of iterator. But having said that, concatenation does make sense as an iterator method. Python chooses to make that a function, itertools.chain, rather than a method or operator. See below.

...

In fact, we already do that with lists.

[1, 2, 3] + [4, 5, 6] # evaluates to [1, 2, 3, 4, 5, 6]

I propose:

iter([1, 2, 3]) + iter([4, 5, 6]) # evaluates to something like itertools.chain(iter([1, 2, 3]), iter([4, 5, 6])) # equivalent to iter([1, 2, 3, 4, 5, 6])

I don't entirely dislike this. I'm not a big fan of Python's choice to use + for concatenation, but the principle of supporting concatenation for iterators makes sense. But, "iterator" isn't a type in Python, it is a protocol, so there are a whole lot of *different* types that count as iterators, and as far as I can see, they don't share any common superclass apart from object itself. I count at least nine in the builtins alone: range_iterator, list_iterator, tuple_iterator, str_iterator, set_iterator, dict_keyiterator, dict_valueiterator, dict_itemiterator, generator (These are distinct from the types range, list, tuple, etc.) and custom-made iterators don't have to inherit from any special class, they just need to obey the protocol. So where would you put the __add__ and __radd__ methods? The usual Pythonic solution to the problem of where to put a method that needs to operate on a lot of disparate types with no shared superclass is to turn it into a function. We already have that: itertools.chain. A bonus with chain is that there is no need to manually convert each argument to an iterator first, it does it for you: chain(this, that, another) versus iter(this) + iter(that) + iter(another) And the bonus with chain() is that you can start using it *right now*, and not wait another two years for Python 3.6.

...

There is some python2 code where:

a = dict(zip('abcd', range(4))) isinstance(a.values(), list) alphabet = a.keys() + a.values()

In python2, this `alphabet` becomes a list of all values and keys

In current python3, this raises:

TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_values'

But in my proposal, it works just fine. `alphabet` becomes an iterator over all values and keys (similar to the python2 case).

dict_keys and dict_values are not iterators, they are set-like views, and concatenating them does not make sense. The Python 2 equivalent of Python 3's `a.keys() + a.values()` is a.viewkeys() + a.viewvalues() which also raises TypeError, as it should. Or to put it another way, the Python 3 equivalent of the Python 2 code is this: list(a.keys()) + list(a.values()) Either way, since dict keys and values aren't iterators, any change to the iterator protocol or support for iterator concatenation won't change them. -- Steve

random832＠fastmail.us

4:37 a.m.

On Tue, Aug 4, 2015, at 21:34, Chris Angelico wrote:

...

Almost completely useless. Multiplying an *iterable* by an integer will often be useful (eg multiplying list by int), but multiplying an *iterator* (or even just adding one to itself) is going to be useless, because the first time through it will exhaust it, and any well-behaved iterator will remain exhausted once it's ever raised StopIteration. (Note that the 'class iter' that I posted earlier is NOT well-behaved. You can add something onto an exhausted iterator and rejuvenate it. It'd take a couple extra lines of code to fix that.)

Suppose iterator * number returns a new iterator which will iterate through the original iterator once, caching the results, and then yield the cached results n-1 times.

Grayson, Samuel Andrew

4:37 a.m.

If we are barking up that tree, import itertools _old_iter = iter class iter (object): def __init__(self, it): self.it = _old_iter(it) def __iter__(self): return self def __next__(self): return next(self.it) def __add__(self, other): return iter(itertools.chain(self.it, other)) def __radd__(self, other): return iter(itertools.chain(other, self.it)) >>> list('wxy' + iter([1, 2]) + range(3, 5) + 'abc') ['w', 'x', 'y', 1, 2, 3, 4, 'a', 'b', 'c']

Chris Angelico

4:44 a.m.

On Wed, Aug 5, 2015 at 12:37 PM, wrote:

...

On Tue, Aug 4, 2015, at 21:34, Chris Angelico wrote:

...
Almost completely useless. Multiplying an *iterable* by an integer will often be useful (eg multiplying list by int), but multiplying an *iterator* (or even just adding one to itself) is going to be useless, because the first time through it will exhaust it, and any well-behaved iterator will remain exhausted once it's ever raised StopIteration. (Note that the 'class iter' that I posted earlier is NOT well-behaved. You can add something onto an exhausted iterator and rejuvenate it. It'd take a couple extra lines of code to fix that.)

Suppose iterator * number returns a new iterator which will iterate through the original iterator once, caching the results, and then yield the cached results n-1 times.

That's easily spelled "list(iterator) * number", apart from the fact that it makes a concrete result list. I don't think it needs language support. ChrisA

Steven D'Aprano

5:01 a.m.

On Tue, Aug 04, 2015 at 10:37:00PM -0400, random832@fastmail.us wrote:

...

Suppose iterator * number returns a new iterator which will iterate through the original iterator once, caching the results, and then yield the cached results n-1 times.

Repetition on an arbitrary iterator is ambiguous. If I say, "repeat the list [1,2,3] twice" there is no ambiguity, I must get [1, 2, 3, 1, 2, 3] or there is some problem. But iterators can have non-deterministic lengths and values: def gen(): while random.random() < 0.9: yield random.random() What does it mean to "repeat gen twice"? It might mean either of: - generate one run of values using gen, then repeat those same values; - generate two runs of values using gen. Both are easy to write, e.g.: list(gen())*2 # explicitly cache the values, then repeat chain(gen(), gen()) # explicitly use two separate runs In the face of ambiguity, resist the temptation to guess. There's no obvious right behaviour here, whichever you bake into iterator * you will make about half the users unhappy because it doesn't support their use-case. Not every simple expression needs to be an operator. -- Steve

Chris Angelico

6:29 a.m.

On Wed, Aug 5, 2015 at 1:01 PM, Steven D'Aprano wrote:

...

Repetition on an arbitrary iterator is ambiguous. If I say, "repeat the list [1,2,3] twice" there is no ambiguity, I must get [1, 2, 3, 1, 2, 3] or there is some problem. But iterators can have non-deterministic lengths and values:

def gen(): while random.random() < 0.9: yield random.random()

What does it mean to "repeat gen twice"? It might mean either of:

- generate one run of values using gen, then repeat those same values;

- generate two runs of values using gen.

Actually, that's a generator, which is an example of an *iterable*, not an *iterator*. An iterator would be gen(), not gen. There's no logical way to go back to the iterator's source and say "Please sir, I want some more"; compare: x = iter([1,2,3,4,5]) next(x); next(x) print(list(x*2)) What does "doubling" an iterator that's already partly consumed do? Asking to go back to the original list is wrong; asking to duplicate (by caching) the results that we'd already get makes sense. With a generator, it's no different - you could chain two generator objects called from the same function, but the generator object doesn't know what function it was called from (at least, I don't think it does). So there's only one possible meaning for doubling an iterator, and it's list(x). ChrisA

Sven R. Kunze

12:49 p.m.

1) It always sucks when moving from lists to iterators. 2) As you showed, transition to Python 3 is made easier. Thus: +1 for me On 05.08.2015 02:22, Grayson, Samuel Andrew wrote:

...

Concatenation is the most fundamental operation that can be done on iterators. In fact, we already do that with lists.

[1, 2, 3] + [4, 5, 6] # evaluates to [1, 2, 3, 4, 5, 6]

I propose:

iter([1, 2, 3]) + iter([4, 5, 6]) # evaluates to something like itertools.chain(iter([1, 2, 3]), iter([4, 5, 6])) # equivalent to iter([1, 2, 3, 4, 5, 6])

There is some python2 code where:

a = dict(zip('abcd', range(4))) isinstance(a.values(), list) alphabet = a.keys() + a.values()

In python2, this `alphabet` becomes a list of all values and keys

In current python3, this raises:

TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_values'

But in my proposal, it works just fine. `alphabet` becomes an iterator over all values and keys (similar to the python2 case).

Sincerely, Sam G

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Guido van Rossum

1:13 p.m.

Honestly, this has been brought up and rejected so many times I don't think it's even worth discussing. Maybe someone should write a PEP with the proposal so that we can reject it and direct future discussion to the PEP. -- --Guido van Rossum (python.org/~guido)

Sven R. Kunze

1:34 p.m.

Good idea. What are the reasons for the rejections? On 05.08.2015 13:13, Guido van Rossum wrote:

...

Honestly, this has been brought up and rejected so many times I don't think it's even worth discussing. Maybe someone should write a PEP with the proposal so that we can reject it and direct future discussion to the PEP.

-- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)

Guido van Rossum

1:47 p.m.

Read the thread. On Wed, Aug 5, 2015 at 1:34 PM, Sven R. Kunze wrote:

...

Good idea.

What are the reasons for the rejections?

On 05.08.2015 13:13, Guido van Rossum wrote:

Honestly, this has been brought up and rejected so many times I don't think it's even worth discussing. Maybe someone should write a PEP with the proposal so that we can reject it and direct future discussion to the PEP.

-- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)

-- --Guido van Rossum (python.org/~guido)

Sven R. Kunze

4 p.m.

60% of the thread are about the ominous * operator on lists. << Seriously? The thread started with a simple and useful idea; not with some rarely used feature. 35% of the thread are about some internal implementation issues. <<< Maybe not easy but I am certain you will sort that out. 5% about its usefulness and simplicity. <<< I stick to that. On 05.08.2015 13:47, Guido van Rossum wrote:

...

Read the thread.

On Wed, Aug 5, 2015 at 1:34 PM, Sven R. Kunze mailto:srkunze@mail.de> wrote:

Good idea.

What are the reasons for the rejections?

On 05.08.2015 13:13, Guido van Rossum wrote:

...
Honestly, this has been brought up and rejected so many times I don't think it's even worth discussing. Maybe someone should write a PEP with the proposal so that we can reject it and direct future discussion to the PEP.

-- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)

-- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)

Ed Kellett

4:15 p.m.

On Wed, 5 Aug 2015 at 15:01 Sven R. Kunze wrote:

...

60% of the thread are about the ominous * operator on lists. << Seriously? The thread started with a simple and useful idea; not with some rarely used feature. 35% of the thread are about some internal implementation issues. <<< Maybe not easy but I am certain you will sort that out. 5% about its usefulness and simplicity. <<< I stick to that.

I don't think the type issue is an "internal implementation issue". I don't think it should get in the way of doing something useful, but it is a problem that'd need to be solved. That said, there are solutions, and as far as I can tell it's the only problem that's been raised in the thread. My own view is that this would be a good thing, but only because of the deeper problem that it's inconvenient to call functions in Python (they quickly make for rparen soup) and impossible to define custom operators. `+` shouldn't really mean concatenation, but it *does* mean concatenation in Python, and I think concatenation is a pretty reasonable thing to want to do. The clumsiness of itertools.chain on iterators compared with + on lists feels like discouragement from using iterators, even where they're clearly the better solution. edk

Paul Moore

4:16 p.m.

On 5 August 2015 at 15:00, Sven R. Kunze wrote:

...

60% of the thread are about the ominous * operator on lists. << Seriously? The thread started with a simple and useful idea; not with some rarely used feature. 35% of the thread are about some internal implementation issues. <<< Maybe not easy but I am certain you will sort that out. 5% about its usefulness and simplicity. <<< I stick to that.

There is no single iterator type that can have a "+" operator defined on it. If you pick one or more such types, other (possibly user defined) types will not work with "+" and this will confuse users. itertools.chain is available to do this in an unambiguous and straightforward manner, its only downside (for some people) is that it's slightly more verbose. Paul

Paul Moore

4:26 p.m.

On 5 August 2015 at 15:15, Ed Kellett wrote:

...

That said, there are solutions, and as far as I can tell it's the only problem that's been raised in the thread.

Are there solutions? No-one has come up with one, to my knowledge. You need to cover 1. All of the many internal iterator types in Python:

...

...
...
type(iter([])) type({}.keys()) type({}.values()) type({}.items()) type(iter((1,2)))

Please don't dismiss this as "just a matter of coding". It's also necessary to remember to do this for every new custom iterator type. 2. Generator functions and expressions:

...

...
...
def f(): ... yield 1 ... yield 2 ... type(f()) type((i for i in [1,2,3]))

"just another internal type", yeah I know... 3. User defined iterators:

...

...
...
class I: ... def __next__(self): ... return 1 ... type(I())

It is *not* acceptable to ask all users to retro-fit an "__add__ method to all their custom iterator types. Even if it were, this would fail for types which are their own iterator and have a custom __add__ of their own. 4. Iterators defined in 3rd party modules, which is similar to user-defined iterators, but with a much wider user base, and much more stringent compatibility issues to consider. Hand-waving away the real issues here is not an option. Paul

Andrew Barnert

4:34 p.m.

On Aug 5, 2015, at 07:26, Paul Moore wrote:

...

...
On 5 August 2015 at 15:15, Ed Kellett wrote: That said, there are solutions, and as far as I can tell it's the only problem that's been raised in the thread.

Are there solutions? No-one has come up with one, to my knowledge.

You need to cover

1. All of the many internal iterator types in Python:

...
...
...
type(iter([])) type({}.keys()) type({}.values()) type({}.items())

These last three are not iterators, they're views. The fact that the OP and at least one person explaining the problem both seem to think otherwise implies that the problem is even bigger: we'd need to add the operator to not just all possible iterator types, but all possible iterable types. That's an even more insurmountable task--especially since many iterable types already have a perfectly good meaning for the + operator.

Ed Kellett

4:38 p.m.

On Wed, 5 Aug 2015 at 15:26 Paul Moore wrote:

...

On 5 August 2015 at 15:15, Ed Kellett wrote:

...
That said, there are solutions, and as far as I can tell it's the only problem that's been raised in the thread.

Are there solutions? No-one has come up with one, to my knowledge.

You need to cover

1. All of the many internal iterator types in Python:

...
...
...
type(iter([])) type({}.keys()) type({}.values()) type({}.items()) type(iter((1,2)))

Please don't dismiss this as "just a matter of coding". It's also necessary to remember to do this for every new custom iterator type.

2. Generator functions and expressions:

...
...
...
def f(): ... yield 1 ... yield 2 ... type(f()) type((i for i in [1,2,3]))

"just another internal type", yeah I know...

3. User defined iterators:

...
...
...
class I: ... def __next__(self): ... return 1 ... type(I())

It is *not* acceptable to ask all users to retro-fit an "__add__ method to all their custom iterator types. Even if it were, this would fail for types which are their own iterator and have a custom __add__ of their own.

4. Iterators defined in 3rd party modules, which is similar to user-defined iterators, but with a much wider user base, and much more stringent compatibility issues to consider.

Hand-waving away the real issues here is not an option.

Yes, this is why I replied to say they were real issues. Well, one solution would be to have + special-case iterators, and try an iterator-add if there would be a TypeError otherwise. I think this is horrible; I'm just mentioning it for completeness. The other solution would be an Iterator ABC that you'd inherit as a mixin, and recognizes anything as a subclass that implements the current iterator protocol + __add__. Python could do this for 1 and 2. 3 and 4 would be more difficult and ultimately require the author of the class to either inherit Iterator or write an __add__, but this might be eased a bit by having iter() wrap iterators that aren't instances of Iterator.

...

Even if it were, this would fail for types which are their own iterator and have a custom __add__ of their own.

Yes. I don't think there's a solution to this part—as I said before, I think the root of the problem is Python's lack of user-defined operators. edk

Mark Lawrence

5:04 p.m.

On 05/08/2015 15:16, Paul Moore wrote:

...

On 5 August 2015 at 15:00, Sven R. Kunze wrote:

...
60% of the thread are about the ominous * operator on lists. << Seriously? The thread started with a simple and useful idea; not with some rarely used feature. 35% of the thread are about some internal implementation issues. <<< Maybe not easy but I am certain you will sort that out. 5% about its usefulness and simplicity. <<< I stick to that.

There is no single iterator type that can have a "+" operator defined on it. If you pick one or more such types, other (possibly user defined) types will not work with "+" and this will confuse users.

itertools.chain is available to do this in an unambiguous and straightforward manner, its only downside (for some people) is that it's slightly more verbose.

Paul

c = itertools.chain should be short enough for those people. I've discarded ch as it's too often used as a character, and cn as it's likely to get confused with some type of connector. Me, I'll stick with the readable version. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

Chris Angelico

5:07 p.m.

On Thu, Aug 6, 2015 at 12:38 AM, Ed Kellett wrote:

...

Python could do this for 1 and 2. 3 and 4 would be more difficult and ultimately require the author of the class to either inherit Iterator or write an __add__, but this might be eased a bit by having iter() wrap iterators that aren't instances of Iterator.

I offered a solution along these lines early in the thread - effectively, you replace the builtin iter() with a class which wraps the iterator in a way that allows addition as chaining. As far as I can see, there *is no solution* that will accept arbitrary iterables and meaningfully add them, so you're going to have to call iter() at some point anyway. Why not use that as the hook that adds your new method? Note that a production-ready version of 'class iter' would need a couple of additional features. One would be a "pass-through" mode - have __new__ recognize that it's being called on an instance of itself, and swiftly return it, thus making iter() idempotent. Another would be supporting the iter(callable, sentinel) form, which shouldn't be hard (just tweak __new__ and __init__ to allow two parameters). There may be other requirements too, but probably nothing insurmountable. ChrisA

Paul Moore

6:43 p.m.

On 5 August 2015 at 15:34, Andrew Barnert wrote:

...

On Aug 5, 2015, at 07:26, Paul Moore wrote:

...
...
On 5 August 2015 at 15:15, Ed Kellett wrote: That said, there are solutions, and as far as I can tell it's the only problem that's been raised in the thread.

Are there solutions? No-one has come up with one, to my knowledge.

You need to cover

1. All of the many internal iterator types in Python:

...
...
...
type(iter([])) type({}.keys()) type({}.values()) type({}.items())

These last three are not iterators, they're views. The fact that the OP and at least one person explaining the problem both seem to think otherwise implies that the problem is even bigger: we'd need to add the operator to not just all possible iterator types, but all possible iterable types. That's an even more insurmountable task--especially since many iterable types already have a perfectly good meaning for the + operator.

I understand that - that was really my point, that befor anyone can claim that "there are solutions" they need to be sure they understand what the problem is - and part of that is getting people to be clear on what they mean when they say "iterators" - as you say, iterator vs iterable is a big problem here (and the OP specifically wanted this for views, *not* iterators...) Sorry for not being clear that I was explaining that other people weren't being clear :-) Paul

Guido van Rossum

6:48 p.m.

On Wed, Aug 5, 2015 at 4:00 PM, Sven R. Kunze wrote:

...

60% of the thread are about the ominous * operator on lists. << Seriously? The thread started with a simple and useful idea; not with some rarely used feature. 35% of the thread are about some internal implementation issues. <<< Maybe not easy but I am certain you will sort that out.

What seems an internal implementation issue to you is a philosophical issue to others. You dismiss it at your peril (i.e. a waste of your time). Binary operators in Python are always implemented by letting one or the other argument provide the implementation, never by having the language pick an implementation (not even a default implementation). What you see in the C code that might seem to contradict this is either an optimization or an ancient artifact, not to be copied. 5% about its usefulness and simplicity. <<< I stick to that.

...

Good luck. I'm checking out of this thread until a PEP is deemed ready for review. -- --Guido van Rossum (python.org/~guido)

3184

Age (days ago)

3184

Last active (days ago)

List overview

Download

31 comments

13 participants

participants (13)

Alexander Belopolsky
Andrew Barnert
Chris Angelico
Ed Kellett
Grayson, Samuel Andrew
Guido van Rossum
Joonas Liik
Joseph Jevnik
Mark Lawrence
Paul Moore
random832＠fastmail.us
Steven D'Aprano
Sven R. Kunze

Use the plus operator to concatenate iterators

Grayson, Samuel Andrew

Grayson, Samuel Andrew

Grayson, Samuel Andrew

random832＠fastmail.us

Grayson, Samuel Andrew

Mark Lawrence

tags

participants (13)