I would like to add that for example numpy ndarrays are iterables, but they have an __add__ with completely different semantics, namely element-wise ( numerical) addition. So this proposal would conflict with existing libraries with iterable objects. Stephan Op 25 jun. 2017 2:51 p.m. schreef "Serhiy Storchaka" :

25.06.17 15:06, lucas via Python-ideas пише:

...

I often use generators, and itertools.chain on them. What about providing something like the following:

a = (n for n in range(2)) b = (n for n in range(2, 4)) tuple(a + b) # -> 0 1 2 3

This, from user point of view, is just as how the __add__ operator works on lists and tuples. Making generators works the same way could be a great way to avoid calls to itertools.chain everywhere, and to limits the differences between generators and other "linear" collections.

I do not know exactly how to implement that (i'm not that good at C, nor CPython source itself), but by seeing the sources, i imagine that i could do something like the list_concat function at Objects/listobject.c:473, but in the Objects/genobject.c file, where instead of copying elements i'm creating and initializing a new chainobject as described at Modules/itertoolsmodule.c:1792.

(In pure python, the implementation would be something like `def __add__(self, othr): return itertools.chain(self, othr)`)

It would be weird if the addition is only supported for instances of the generator class, but not for other iterators. Why (n for n in range(2)) + (n for n in range(2, 4)) works, but iter(range(2)) + iter(range(2, 4)) and iter([0, 1]) + iter((2, 3)) don't? itertools.chain() supports arbitrary iterators. Therefore you will need to implement the __add__ method for *all* iterators in the world.

However itertools.chain() accepts not just *iterators*. It works with *iterables*. Therefore you will need to implement the __add__ method also for all iterables in the world. But __add__ already is implemented for list and tuple, and many other sequences, and your definition conflicts with this.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas < python-ideas@python.org> wrote:

I often use generators, and itertools.chain on them. What about providing something like the following:

a = (n for n in range(2)) b = (n for n in range(2, 4)) tuple(a + b) # -> 0 1 2 3

AudioLazy does that: https://github.com/danilobellini/audiolazy -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap)

On Sun, Jun 25, 2017 at 02:06:54PM +0200, lucas via Python-ideas wrote:

What about providing something like the following:

a = (n for n in range(2)) b = (n for n in range(2, 4)) tuple(a + b) # -> 0 1 2 3

As Serhiy points out, this is going to conflict with existing use of + operator for string and sequence concatenation. I have a counter-proposal: introduce the iterator chaining operator "&": iterable & iterable --> itertools.chain(iterable, iterable) The reason I choose & rather than + is that & is less likely to conflict with any existing string/sequence types. None of the built-in or std lib sequences that I can think of support the & operator. Also, & is used for (string?) concatenation in some languages, such as VB.Net, some BASIC dialects, Hypertalk, AppleScript, and Ada. Iterator chaining is more like concatenation than (numeric) addition. However, the & operator is already used for bitwise-AND. Under my proposal that behaviour will continue, and will take priority over chaining. Currently, the & operator does something similar to (but significantly more complex) to this: # simplified pseudo-code of existing behaviour if hasattr(x, '__and__'): return x.__and__(y) elif hasattr(y, '__rand__'): return y.__rand__(x) else: raise TypeError The key is to insert the new behaviour after the existing __(r)and__ code, just before TypeError is raised: attempt existing __(r)and__ behaviour if and only if that fails to apply: return itertools.chain(iter(x), iter(y)) So classes that define a __(r)and__ method will keep their existing behaviour. This implies that we cannot use & to chain sets and frozen sets, since they already define __(r)and__. This has an easy work-around: just call iter() on the set first. Applying & to objects which don't define __(r)and__ and aren't iterable will continue to raise TypeError, just as it does now. The only backwards incompatibility this proposal introduces is for any code which relies on `iterable & iterable` to raise TypeError. Frankly I can't imagine that there is any such code, outside of the Python test suite, but if there is, and people think it is worth it, we could make this a __future__ import. But I think that's overkill. The downside to this proposal is that it adds some conceptual complexity to Python operators. Apart from `is` and `is not`, all Python operators call one or more dunder methods. This is (as far as I know) the first operator which has fall-back functionality if the dunder methods aren't defined. Up to now, I've talked about & chaining being equivalent to the itertools.chain function. That glosses over one difference which needs to be mentioned. The chain function currently doesn't attempt to iterate over its arguments until needed: py> x = itertools.chain("a", 1, "c") py> next(x) 'a' py> next(x) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'int' object is not iterable Any proposal to change this behaviour for the itertools.chain function should be kept separate from this one. But for the & chaining operator, I think that behaviour must change: if we have an operand that is neither iterable nor defines __(r)and__, the & operator should fail early: [1, 2, 3] & None should raise TypeError immediately, unlike itertools.chain(). -- Steve

26.06.17 13:47, Joao S. O. Bueno пише:

On 25 June 2017 at 20:55, Danilo J. S. Bellini mailto:danilo.bellini@gmail.com> wrote:

On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas mailto:python-ideas@python.org>wrote:

I often use generators, and itertools.chain on them. What about providing something like the following:

a = (n for n in range(2)) b = (n for n in range(2, 4)) tuple(a + b) # -> 0 1 2 3

You know you can do `tuple(*a, *b)` , right?

The problem with the "*" notation is that it will actually render the iterable contents eagerly - unlike something that would just chain them. But for creating tuples, it just works.

Even the tuple constructor is not needed.

...

...

*a, *b (0, 1, 2, 3)

On Sun, Jun 25, 2017 at 8:23 PM, Steven D'Aprano wrote:

On Sun, Jun 25, 2017 at 02:06:54PM +0200, lucas via Python-ideas wrote:

I have a counter-proposal: introduce the iterator chaining operator "&":

iterable & iterable --> itertools.chain(iterable, iterable)

In [1]: import numpy as np In [2]: import itertools In [3]: a, b = np.array([1,2,3]), np.array([4,5,6]) In [4]: a & b Out[4]: array([0, 0, 2]) In [5]: a + b Out[5]: array([5, 7, 9]) In [6]: list(itertools.chain(a, b)) Out[6]: [1, 2, 3, 4, 5, 6] These are all distinct, useful, and well-defined behaviors. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

On Sunday, June 25, 2017, Wes Turner wrote:

On Sunday, June 25, 2017, Danilo J. S. Bellini javascript:_e(%7B%7D,'cvml','danilo.bellini@gmail.com');> wrote:

...

On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas < python-ideas@python.org> wrote:

...

I often use generators, and itertools.chain on them. What about providing something like the following:

a = (n for n in range(2)) b = (n for n in range(2, 4)) tuple(a + b) # -> 0 1 2 3

AudioLazy does that: https://github.com/danilobellini/audiolazy

- http://toolz.readthedocs.io/en/latest/api.html#toolz.itertoolz.concat and concatv

- https://github.com/kachayev/fn.py#streams-and-infinite- sequences-declaration - Stream() << obj

<< is __lshift__() <<= is __ilshift__() https://docs.python.org/2/library/operator.html Do Stream() and __lshift__() from fn.py not solve here?

...

-- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap)

On Tue, Jun 27, 2017 at 12:12 AM, Steven D'Aprano wrote:

...

...

I have a counter-proposal: introduce the iterator chaining operator "&": iterable & iterable --> itertools.chain(iterable, iterable)

In [1]: import numpy as np In [2]: import itertools In [3]: a, b = np.array([1,2,3]), np.array([4,5,6]) In [4]: a & b Out[4]: array([0, 0, 2]) In [5]: a + b Out[5]: array([5, 7, 9]) In [6]: list(itertools.chain(a, b)) Out[6]: [1, 2, 3, 4, 5, 6]

These are all distinct, useful, and well-defined behaviors.

- keep the existing __and__ and __rand__ behaviour; - if they are not defined, and both operands x, y are iterable, return chain(x, y);

I understand. But it invites confusion about just what the `&` operator will do for a given iterable. For NumPy itself, you don't really want to spell `chain(a, b)` so much. But you CAN, they are iterables. The idiomatic way is: >>> np.concat((a,b)) array([1, 2, 3, 4, 5, 6]) However, for "any old iterable" it feels very strange to need to inspect the .__and__() and .__rand__ () methods of the things on both sides before you know WHAT operator it is. Maybe if you are confident a and b are exactly NumPy arrays it is obvious, but what about: from some_array_library import a from other_array_library import b What do you think `a & b` will do under your proposal? Yes, I understand it's deterministic... but it's far from *obvious*. This isn't even doing something pathological like defining both `.__iter__()` and `.__and__()` on the same class... which honestly, isn't even all that pathological; I can imagine real-world use cases. I think that chaining iterators is common enough and important enough in

Python 3 to deserve an operator. While lists are still important, a lot of things which were lists are now lazily generated iterators, and we often need to concatenate them. itertools.chain() is less convenient than it should be.

I actually completely agree! I just wish I could think of a good character that doesn't have some very different meaning in other well-known contexts (even among iterables). Some straw men: both = a ⊕ b both = a ⇢ b Either of those look pretty nice to me, but neither is easy to enter on most keyboards. I think I wouldn't mind `&` if it only worked on iteraTORS. But then it loses many of the use cases. I'd like this, after all: for i in range(10)⇢[20,19,18]⇢itertools.count(100): if i>N: break ... -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

Hi all, Is "itertools.chain" actually that common? Sufficiently common to warrant its own syntax? In my experience, "enumerate" is far more common among the iterable operations. And that doesn't have special syntax either. A minimal proposal would be to promote "chain" to builtins. Stephan 2017-06-27 10:40 GMT+02:00 Greg Ewing :

David Mertz wrote:

...

I just wish I could think of a good character that doesn't have some very different meaning in other well-known contexts (even among iterables).

(a;b)

Should be unambiguous as long as the parens are required.

-- Greg

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jun 27, 2017 at 11:01:32AM +0200, Stephan Houben wrote:

Hi all,

Is "itertools.chain" actually that common? Sufficiently common to warrant its own syntax?

I think it's much more common than (say) sequence repetition: a = [None]*5 which has had an operator for a long, long time.

In my experience, "enumerate" is far more common among the iterable operations. And that doesn't have special syntax either.

True. But enumerate is a built-in, and nearly always used in a single context: for i, x in enumerate(sequence): ... A stranger to Python could almost be forgiven for thinking that enumerate is part of the for-loop syntax. In contrast, chaining (while not as common as, say, numeric addition) happens in variable contexts: in expressions, as arguments to function calls, etc. It is absloutely true that this proposal brings nothing new to the language that cannot already be done. It's syntactic sugar. So I guess the value of it depends on whether or not you chain iterables enough that you would rather use an operator rather than a function.

A minimal proposal would be to promote "chain" to builtins.

Better than nothing, I suppose. -- Steve

Hi Steven, To put this into perspective, I did some greps on Sagemath, being the largest Python project I have installed on this machine (1955 .py files). Occurrences: enumerate: 922 zip: 585 itertools.product: 67 itertools.combinations: 18 itertools.islice: 17 chain: 14 (with or without itertools. prefix) This seems to confirm my gut feeling that "chain" just isn't that common an operation; even among itertools functions, product, combinations and islice are more common. Based on this I would say there is little justification to even put "chain" in builtins, let alone to give it dedicated syntax. I also note that * for repetition is only supported for a few iterables (list, tuple), incidentally the same ones which support + for sequence chaining. Stephan 2017-06-27 12:38 GMT+02:00 Steven D'Aprano :

On Tue, Jun 27, 2017 at 11:01:32AM +0200, Stephan Houben wrote:

...

Hi all,

Is "itertools.chain" actually that common? Sufficiently common to warrant its own syntax?

I think it's much more common than (say) sequence repetition:

a = [None]*5

which has had an operator for a long, long time.

...

In my experience, "enumerate" is far more common among the iterable operations. And that doesn't have special syntax either.

True. But enumerate is a built-in, and nearly always used in a single context:

for i, x in enumerate(sequence): ...

A stranger to Python could almost be forgiven for thinking that enumerate is part of the for-loop syntax.

In contrast, chaining (while not as common as, say, numeric addition) happens in variable contexts: in expressions, as arguments to function calls, etc.

It is absloutely true that this proposal brings nothing new to the language that cannot already be done. It's syntactic sugar. So I guess the value of it depends on whether or not you chain iterables enough that you would rather use an operator rather than a function.

...

A minimal proposal would be to promote "chain" to builtins.

Better than nothing, I suppose.

-- Steve _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jun 27, 2017 at 01:32:05PM +0200, Stephan Houben wrote:

Hi Steven,

To put this into perspective, I did some greps on Sagemath, being the largest Python project I have installed on this machine (1955 .py files).

And one which is especially focused on numerical processing, not really the sort of thing that does a much iterator chaining. That's hardly a fair test -- we know there are applications where chaining is not important at all. Its the applications where it *is* important that we should be looking at. -- Steve

Just another syntactical suggestion: the binary ++ operator is used as concat in various contexts in various languages, and is probably less likely to confuse people as being either a logical or binary &. On Tue, Jun 27, 2017 at 6:53 AM Stephan Houben wrote:

...

Its the applications where it *is* important that we should be looking at.

Um, yes, but given our relative positions in this debate, the onus is not really on *me* to demonstrate such an application, right? That would just confuse everbody ;-)

(FWIW, Sagemath is not mostly "numerical processing", it is mostly *symbolic* calculations and involves a lot of complex algorithms and datastructures, including sequences.)

Stephan

2017-06-27 13:48 GMT+02:00 Steven D'Aprano :

...

On Tue, Jun 27, 2017 at 01:32:05PM +0200, Stephan Houben wrote:

...

Hi Steven,

To put this into perspective, I did some greps on Sagemath, being the largest Python project I have installed on this machine (1955 .py files).

And one which is especially focused on numerical processing, not really the sort of thing that does a much iterator chaining. That's hardly a fair test -- we know there are applications where chaining is not important at all. Its the applications where it *is* important that we should be looking at.

-- Steve _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

---

Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Argh, you're correct. Thanks for the catch. On Tue, Jun 27, 2017 at 10:06 AM Stephan Houben wrote:

Unfortunately this is existing syntax:

a++b is parsed as a+(+b)

Stephan

Op 27 jun. 2017 6:03 p.m. schreef "Joshua Morton" < joshua.morton13@gmail.com>:

Just another syntactical suggestion: the binary ++ operator is used as concat in various contexts in various languages, and is probably less likely to confuse people as being either a logical or binary &.

On Tue, Jun 27, 2017 at 6:53 AM Stephan Houben wrote:

...

...

Its the applications where it *is* important that we should be looking at.

Um, yes, but given our relative positions in this debate, the onus is not really on *me* to demonstrate such an application, right? That would just confuse everbody ;-)

(FWIW, Sagemath is not mostly "numerical processing", it is mostly *symbolic* calculations and involves a lot of complex algorithms and datastructures, including sequences.)

Stephan

2017-06-27 13:48 GMT+02:00 Steven D'Aprano :

...

On Tue, Jun 27, 2017 at 01:32:05PM +0200, Stephan Houben wrote:

...

Hi Steven,

To put this into perspective, I did some greps on Sagemath, being the largest Python project I have installed on this machine (1955 .py files).

And one which is especially focused on numerical processing, not really the sort of thing that does a much iterator chaining. That's hardly a fair test -- we know there are applications where chaining is not important at all. Its the applications where it *is* important that we should be looking at.

-- Steve _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

---

Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jun 27, 2017 at 4:44 AM, Steven D'Aprano wrote:

But that strikes me as overkill. You don't normally check for dunders before using an operator, and we already have operators that can return different types depending on the operands:

% can mean modulo division or string interpolation * can mean sequence repetition or numeric multiplication + can mean numeric addition or sequence concatenation

Why is

& can mean iterable chaining or bitwise-AND

I don't think it's "uniquely confusing." Just more so than the other examples you give. For example, I might write functions like these (untested): def modulo1(i: int, j: int) -> int: return i % j def modulo2(s: str, t: tuple) -> str: return s % t And similar examples for `*` and `+`. When I try to write this: def ampersand(x: Iterable, y: Iterable) -> Iterable: return x & y More ambiguity exists. The type signature works for both Numpy arrays and generators (under the proposed language feature), but the function does something different... in a way that is "more different" than I'd expect. That said, I like the idea of having iterators that act magically to fold in general iterables after an .__and__() or .__add__() as proposed by Brendan down-thread. Without any language change we could have: chainable(x for x in blah) + [1, 2, 3] + "hello" And I would like a language change that made a number of common iterable objects "chainable" without the wrapper. This wrapper could of course be used as a decorator too. E.g. generator comprehensions, things returned by itertools functions, range(), enumerate(), zip(), etc. This wouldn't promise that EVERY iterable or iterator had that "chainable" behavior, but it would cover 90% of the use cases. And I wouldn't find it confusing because the leftmost object would be the one determining the behavior, which feels more intuitive and predictable. I don't hate `&&`, but I think this approach makes more sense. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

On 2017-06-25 20:23, Steven D'Aprano wrote:

I have a counter-proposal: introduce the iterator chaining operator "&":

iterable & iterable --> itertools.chain(iterable, iterable)

I like this suggestion. Here's another color that might be less controversial: iterable3 = iterable1.chain(iterable2) Perhaps more obvious than &, easier to use than "from itertools import chain...". -Mike

On 2017-06-27 14:02, David Mertz wrote:

On Tue, Jun 27, 2017 at 1:57 PM, Mike Miller mailto:python-ideas@mgmiller.net> wrote:

I like this suggestion. Here's another color that might be less controversial:

iterable3 = iterable1.chain(iterable2)

How do you chain it1, it2, it3, etc?

I guess `it1.chain(it2.chain(it3)))` ... but that starts to become distinctly less readable IMO. I'd much rather spell `chain(it1, it2, it3)`.

Even if this "chain" only took one argument, you could do it1.chain(it2).chain(it3). But I don't see why it couldn't take multiple arguments as you suggest. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On 2017-06-27 14:05, Brendan Barnwell wrote:

Even if this "chain" only took one argument, you could do it1.chain(it2).chain(it3). But I don't see why it couldn't take multiple arguments as you suggest.

Right, and as I forgot to mention, making it a built-in is an uphill battle with higher backward compatibility concerns.

On 28 June 2017 at 07:13, Mike Miller wrote:

On 2017-06-27 14:05, Brendan Barnwell wrote:

...

Even if this "chain" only took one argument, you could do it1.chain(it2).chain(it3). But I don't see why it couldn't take multiple arguments as you suggest.

Right, and as I forgot to mention, making it a built-in is an uphill battle with higher backward compatibility concerns.

While I haven't been following this thread closely, I'd like to note that arguing for a "chain()" builtin has the virtue that would just be arguing for the promotion of the existing itertools.chain function into the builtin namespace. Such an approach has a lot to recommend it: 1. It has precedent, in that Python 3's map(), filter(), and zip(), are essentially Python 2's itertools.imap(), ifilter(), and izip() 2. There's no need for a naming or semantics debate, as we'd just be promoting an established standard library API into the builtin namespace 3. Preserving compatibility with older versions is straightforward: just do an unconditional "from itertools import chain" 4. As an added bonus, we'd also get "chain.from_iterable" as a builtin API So it would be good to have a short PEP that argued that since chaining arbitrary iterables is at least as important as mapping, filtering, and zipping them, itertools.chain should be added to the builtin namespace in 3.7+ (but no, I'm not volunteering to write that myself). As a *separate* discussion, folks could then also argue for the additional of a `__lshift__` operator implementation specifically to iterator chains that let you write: full_chain = chain(it1) << it2 << it3 # Incrementally create new chains full_chain <<= it4 # Extend an existing chain I'd be surprised if such a proposal got accepted for 3.7, but it would make a good follow-up discussion for 3.8 (assuming chain() made it into the 3.7 builtins). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 June 2017 at 14:30, Terry Reedy wrote:

On 6/27/2017 10:47 PM, Nick Coghlan wrote:

...

Such an approach has a lot to recommend it:

1. It has precedent, in that Python 3's map(), filter(), and zip(), are essentially Python 2's itertools.imap(), ifilter(), and izip() 2. There's no need for a naming or semantics debate, as we'd just be promoting an established standard library API into the builtin namespace

A counter-argument is that there are other itertools that deserve promotion, by usage, even more. But we need to see comparisons from more that one limited corpus.

On the other hand, there might be a theory argument that chain is somehow more basic, akin to map, etc, in a way that others are not.

The main rationale I see is the one that kicked off the most recent discussion, which is that in Python 2, you could readily chain the output of map(), filter(), zip(), range(), dict.keys(), dict.values(), dict.items(), etc together with "+", simply because they all returned concrete lists. In Python 3, you can't do that as easily anymore, since they all return either iterators or computed containers that don't support "+". While there are good reasons not to implement "+" on those iterators and custom containers, we *can* fairly easily restore builtin concatentation support for their outputs, and we can do it in a way that's friendly to all implementations that already provide the itertools.chain API. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 June 2017 at 18:54, Paul Moore wrote:

On 28 June 2017 at 05:30, Terry Reedy wrote:

...

A counter-argument is that there are other itertools that deserve promotion, by usage, even more. But we need to see comparisons from more that one limited corpus.

Indeed. I don't recall *ever* using itertools.chain myself. I'd be interested in seeing some usage stats to support this proposal. As an example, I see 8 uses of itertools.chain in pip and its various vendored packages, as opposed to around 30 uses of map (plus however many list comprehensions are used in place of maps). On a very brief scan, it looks like the various other itertools are used less than chain, but with only 8 uses of chain, it's not really possible to read anything more into the relative frequencies.

The other thing to look for would be list() and list.extend() calls. I know I use those quite a bit in combination with str.join, where I don't actually *need* a list, it's just currently the most convenient way to accumulate all the components I'm planning to combine. And if you're converting from Python 2 code, then adding a few list() calls in critical places in order to keep the obj.extend() calls working is likely to be easier in many cases than switching over to using itertools.chain. For simple cases, that's fine (since a list of direct references will be lower overhead than accumulating a chain of short iterables), but without builtin support for iterator concatenation, it's currently a nonlocal refactoring (to add the "from itertools import chain" at the top of the file) to switch completely to the "pervasive iterators" model when folks actually do want to do that. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Jun 28, 2017 at 11:54 AM, Paul Moore wrote:

Indeed. I don't recall *ever* using itertools.chain myself.

​ In fact, me neither. Or maybe a couple of times. For such a basic task, it feels more natural to write a generator function, or even turn it into a list, if you can be sure that the 'unnecessary' lists will be small and that the code won't be a performance bottle neck. To reiterate on this some more: One of the nice things of Python 3 is (or could be) the efficiency of not making unnecessary lists by default. But for the programmer/beginner it's not nearly as convenient with the views as it is with lists. Beginners quickly need to learn about Also generators are really nice, and chaining them is just as useful/necessary as extending or concatenating lists. Chaining generators with other iterables is certainly useful, but when all the parts of the chained object are iterable but not sequences, that seems like an invitation to use list() at some point in the code. So whatever the outcome of this discussion (I hope there is one, whether it is by adding iterator-related builtins or something more sophisticated), it should probably take into account possible future ways of dealing with some kind of "lazy lists". However, I'm actually not sure the syntax of chaining generators/iterables should necessarily be the same as for chaining arbitrary sequences. The programmer needs to be well aware of whether the resulting object is a Sequence or 'just' a generator. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven +

On Wed, Jun 28, 2017 at 3:18 PM, Sven R. Kunze wrote:

On 28.06.2017 14:00, Koos Zevenhoven wrote:

The programmer needs to be well aware of whether the resulting object is a Sequence or 'just' a generator.

Could you elaborate more on **why**?

​For a moment, I was wondering what the double emphasis was for, but then I realized you are simply calling `statement.__why__()`​ directly instead of the recommended `spoiler(statement)`. But sure, I just got on vacation and I even found a power extension cord to use my laptop at the pool, so what else would I do ;). It all depends on what you need to do with the result of the concatenation. When all you need is something to iterate over, a generator-like thingy is fine. But when you need something for indexing and slicing or len etc., you want to be sure that that is what you're getting. But maybe someone passed you an argument that is not a sequence, or you forgot if a function returns a sequence or a generator. In that case, you want the error right away, instead of from some completely different piece of code somewhere that thinks it's getting a sequence. I don't think Python should depend on a static type checker to catch the error early. After all, type hints are optional. PS: I consider this proposal to be like allowing adding floats and ints

together. If I don't know if there was a float in the sum, don't know if my result will be a float.

​Not to say that the float/int case is never problematic, but the situation is still different. Often when a float makes any sense, you can work with either floats or ints and it doesn't really matter. But if you specifically *need* an int, you usually don't call functions that return floats. But if you do use division etc., you probably need to think about floor/ceil/closest anyway. And yes, there have probably been Python 2->3 porting bugs where / division was not appropriately replaced with //. But regarding containers, it often makes just as much sense for a function to return a generator as it does to return a sequence. The name or purpose of a function may give no hint about whether an iterable or sequence is returned, and you can't expect everyone to prefix their function names with iter_ and seq_ etc. And it's not just function return values, it's also arguments passed into your function. ​​ ​-- Koos​ -- + Koos Zevenhoven + http://twitter.com/k7hoven +

On 28.06.2017 16:01, Koos Zevenhoven wrote:

For a moment, I was wondering what the double emphasis was for, but then I realized you are simply calling `statement.__why__()`​ directly instead of the recommended `spoiler(statement)`.

Doing this for years now. Sometimes, when 'statement.__why__()' returns None, 'spoiler(statement)' returns some thought-terminating cliché. ;)

But sure, I just got on vacation and I even found a power extension cord to use my laptop at the pool, so what else would I do ;).

Good man. Today, a colleague of mine showed me a mobile mini-keyboard with a phone bracket (not even a dock). So, having his 7'' smartphone, he can work from his vacations and answer emails as well. ;) Cheap notebook replacement, if you don't prefer large screens and keyboards. :D

It all depends on what you need to do with the result of the concatenation. When all you need is something to iterate over, a generator-like thingy is fine. But when you need something for indexing and slicing or len etc., you want to be sure that that is what you're getting. But maybe someone passed you an argument that is not a sequence, or you forgot if a function returns a sequence or a generator. In that case, you want the error right away, instead of from some completely different piece of code somewhere that thinks it's getting a sequence. I don't think Python should depend on a static type checker to catch the error early. After all, type hints are optional.

I understand that. In the end, I remember people on this mailing-list recommending me to use "list(...)" to make sure you got one in your hands. I remember this being necessary in the conversion process from Python2 to 3. The pattern is already here.

PS: I consider this proposal to be like allowing adding floats and ints together. If I don't know if there was a float in the sum, don't know if my result will be a float.

​Not to say that the float/int case is never problematic, but the situation is still different. Often when a float makes any sense, you can work with either floats or ints and it doesn't really matter. But if you specifically *need* an int, you usually don't call functions that return floats. But if you do use division etc., you probably need to think about floor/ceil/closest anyway. And yes, there have probably been Python 2->3 porting bugs where / division was not appropriately replaced with //.

Division is one thing, numeric input parameters from unknown sources is another. In this regard, calling "int(...)" or "list(...)" follows the same scheme IMO.

But regarding containers, it often makes just as much sense for a function to return a generator as it does to return a sequence. The name or purpose of a function may give no hint about whether an iterable or sequence is returned, and you can't expect everyone to prefix their function names with iter_ and seq_ etc. And it's not just function return values, it's also arguments passed into your function.

Exactly. Neither want I those prefixes. And I can tell you they aren't necessary in practice at all. Just my 2 cents on this: At work, we heavily rely on Django. Django provides a so-called QuerySet type, its db-result abstraction. Among those querysets, our functions return lists and sets with no indication of whatsoever type it may be. It works quite well and we didn't had any issues with that. If we need a list, we wrap it with "list(...)". It's as simple as that. The valid concern, that it could be confusing which type the return value might have, is usually an abstract one. I can tell you that in practice it's not really an issue to talk about. Regards, Sven

On Tuesday, June 27, 2017, Wes Turner wrote:

On Monday, June 26, 2017, Wes Turner javascript:_e(%7B%7D,'cvml','wes.turner@gmail.com');> wrote:

...

On Sunday, June 25, 2017, Wes Turner wrote:

...

On Sunday, June 25, 2017, Danilo J. S. Bellini wrote:

...

On Sun, Jun 25, 2017 at 3:06 PM, lucas via Python-ideas < python-ideas@python.org> wrote:

...

I often use generators, and itertools.chain on them. What about providing something like the following:

a = (n for n in range(2)) b = (n for n in range(2, 4)) tuple(a + b) # -> 0 1 2 3

AudioLazy does that: https://github.com/danilobellini/audiolazy

- http://toolz.readthedocs.io/en/latest/api.html#toolz.itertoolz.concat and concatv

""" We use chain.from_iterable rather than chain(*seqs) so that seqs can be a generator. """ ... ``chain.from_iterable()``

...

...

- https://github.com/kachayev/fn.py#streams-and-infinite-seque nces-declaration - Stream() << obj

<< is __lshift__() <<= is __ilshift__()

https://docs.python.org/2/library/operator.html

Do Stream() and __lshift__() from fn.py not solve here?

In this syntax example, iter1 is mutated before iteration:

iter_x = Stream(iter1) << iter2 << iter3 iter1 <<= iter5 # IDK if fn.py yet has <<= list(iter1) list(iter_x)

2017-06-27 Stephan Houben dixit:

Is "itertools.chain" actually that common? Sufficiently common to warrant its own syntax?

Please, note that it can be upturned: maybe they are not so common as they could be because of all that burden with importing from separate module -- after all we are saying about somewhat very simple operation, so using lists and `+` just wins because of our (programmers') laziness. :-) Cheers. *j

On Fri, Jun 30, 2017 at 1:09 AM, Jan Kaliszewski  wrote:
> 2017-06-25 Serhiy Storchaka  dixit:
>
>> 25.06.17 15:06, lucas via Python-ideas пише:
>
>> > I often use generators, and itertools.chain on them.
>> > What about providing something like the following:
>> >
>> >      a = (n for n in range(2))
>> >      b = (n for n in range(2, 4))
>> >      tuple(a + b)  # -> 0 1 2 3
> [...]
>> It would be weird if the addition is only supported for instances of
>> the generator class, but not for other iterators. Why (n for n in
>> range(2))
>> + (n for n in range(2, 4)) works, but iter(range(2)) + iter(range(2,
>> 4)) and iter([0, 1]) + iter((2, 3)) don't? itertools.chain() supports
>> arbitrary iterators. Therefore you will need to implement the __add__
>> method for *all* iterators in the world.
>>
>> However itertools.chain() accepts not just *iterators*.
> [...]
>
> But implementation of the OP's proposal does not need to be based on
> __add__ at all.  It could be based on extending the current behaviour of
> the `+` operator itself.
>
> Now this behavior is (roughly): try left side's __add__, if failed try
> right side's __radd__, if failed raise TypeError.
>
> New behavior could be (again: roughly): try left side's __add__, if
> failed try right side's __radd__, if failed try __iter__ of both sides
> and chain them (creating a new iterator¹), if failed raise TypeError.
>
> And similarly, for `+=`: try __iadd__..., try __add__..., try
> __iter__..., raise TypeError.

I actually really like this proposal, in additional to the original
proposal of using '+' to chain generators--I don't think it
necessarily needs to be extended to *all* iterables.  But this
proposal goes one better.  I just have to wonder what kind of strange
unexpected bugs would result.  For example now you could add a list to
a string:

    >>> list(['a', 'b', 'c'] + 'def')
    ['a', 'b', 'c', 'd', 'e', 'f']

Personally, I really like this and find it natural.  But it will break
anything expecting this to be a TypeError.

On 28Jun2017 09:54, Paul Moore wrote:

On 28 June 2017 at 05:30, Terry Reedy wrote:

...

On 6/27/2017 10:47 PM, Nick Coghlan wrote:

...

While I haven't been following this thread closely, I'd like to note that arguing for a "chain()" builtin has the virtue that would just be arguing for the promotion of the existing itertools.chain function into the builtin namespace.

Such an approach has a lot to recommend it:

1. It has precedent, in that Python 3's map(), filter(), and zip(), are essentially Python 2's itertools.imap(), ifilter(), and izip() 2. There's no need for a naming or semantics debate, as we'd just be promoting an established standard library API into the builtin namespace

A counter-argument is that there are other itertools that deserve promotion, by usage, even more. But we need to see comparisons from more that one limited corpus.

Indeed. I don't recall *ever* using itertools.chain myself. I'd be interested in seeing some usage stats to support this proposal. As an example, I see 8 uses of itertools.chain in pip and its various vendored packages, as opposed to around 30 uses of map (plus however many list comprehensions are used in place of maps). On a very brief scan, it looks like the various other itertools are used less than chain, but with only 8 uses of chain, it's not really possible to read anything more into the relative frequencies.

I don't use it often, but when I do it is very handy. While I'm not arguing for making it a builtin on the basis of my own use (though I've no objections either), a quick grep shows: My maildb kit uses chain to assemble multiple related header values: *chain( msg.get_all(hdr, []) for hdr in ('to', 'cc', 'bcc', 'resent-to', 'resent-cc') ) Two examples where I use it to insert items in front of an iterable: chunks = chain( [data], chunks ) blocks = indirect_blocks(chain( ( topblock, nexttopblock ), blocks )) Neither of these is amenable to list rephrasings because the tail iterables ("chunks" and "blocks") are of unknown and potentially large size. And a few other cases whose uses are harder to succinctly describe, but generally "iterable flattening". So it is uncommon for me, but very useful when I want it. Just some (small) data points. Cheers, Cameron Simpson

We HAVE spellings for these things: c from:

...

yield from a yield from b

c = chain(a, b)

c as list from:

...

yield from a yield from b

c = list(chain(a, b))

c as sum from:

...

yield from a yield from b

c = sum(chain(a, b)) Those really are not "crazy generator expressions." -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

On Saturday, July 1, 2017, Steven D'Aprano wrote:

On Fri, Jun 30, 2017 at 01:09:51AM +0200, Jan Kaliszewski wrote:

[...]

But the more I think about it the more I agree with Nick. Let's start by moving itertools.chain into built-ins, with zip and map, and only consider giving it an operator after we've had a few years of experience with chain as a built-in. We might even find that an operator doesn't add any real value.

- Would that include chain.from_iterable? - So there's then a new conditional import (e.g. in a compat package)? What does this add?

...

¹ Preferably using the existing `yield from` mechanism -- because, in case of generators, it would provide a way to combine ("concatenate") *generators*, preserving semantics of all that their __next__(), send(), throw() nice stuff...

I don't think that would be generally useful.

Flatten one level?

If you're sending values into an arbitrary generator, who knows what you're getting? chain() will operate on arbitrary iterables, you can't expect to send values into chain([1, 2, 3], my_generator(), "xyz") and have anything sensible occur.

- is my_generator() mutable (e.g. before or during iteration)? - https://docs.python.org/2/reference/expressions.html#generator.send

On Sat, Jul 1, 2017 at 6:11 PM, Steven D'Aprano wrote:

...

- So there's then a new conditional import (e.g. in a compat package)? What does this add?

try: chain except NameError: from itertools import chain

Two lines, if and only if you both need chain and want to support versions of Python older than 3.7.

There's no need to import it if you aren't going to use it.

It'd be even simpler. If you want to support <3.7 and 3.7+, you write: from itertools import chain At least, I presume it isn't going to be *removed* from itertools. Promotion to builtin shouldn't break pre-existing code, so the way to be compatible with pre-promotion Pythons is simply to code for those and not take advantage of the new builtin. ChrisA

On Saturday, July 1, 2017, Steven D'Aprano wrote:

On Sat, Jul 01, 2017 at 01:35:29AM -0500, Wes Turner wrote:

...

On Saturday, July 1, 2017, Steven D'Aprano javascript:;> wrote:

...

On Fri, Jun 30, 2017 at 01:09:51AM +0200, Jan Kaliszewski wrote:

[...]

But the more I think about it the more I agree with Nick. Let's start by moving itertools.chain into built-ins, with zip and map, and only consider giving it an operator after we've had a few years of experience with chain as a built-in. We might even find that an operator doesn't add any real value.

- Would that include chain.from_iterable?

Yes.

...

- So there's then a new conditional import (e.g. in a compat package)? What does this add?

try: chain except NameError: from itertools import chain

Two lines, if and only if you both need chain and want to support versions of Python older than 3.7.

There's no need to import it if you aren't going to use it.

Or, can I just continue to import the same function from the same place: from itertools import chain Nice, simple, easy. There's even (for all you functional lovers): from itertools import * And, again, this works today: from fn import Stream itr = Stream() << my_generator() << (8,9,0) - https://github.com/kachayev/fn.py/blob/master/README.rst#streams-and-infinit... - https://github.com/kachayev/fn.py/blob/master/fn/stream.py - AFAIU, + doesn't work because e.g. numpy already defines + and & for Iterable arrays.

...

...

...

¹ Preferably using the existing `yield from` mechanism -- because, in case of generators, it would provide a way to combine ("concatenate") *generators*, preserving semantics of all that their __next__(), send(), throw() nice stuff...

I don't think that would be generally useful.

Flatten one level?

Flattening typically applies to lists and sequences.

I'm not saying that chain shouldn't support generators. That would be silly: a generator is an iterable and chaining supports iterables. I'm saying that it wouldn't be helpful to require chain objects to support send(), throw() etc.

So the argspec is/shouldbe Iterables with __iter__ (but not necessarily __len__)?

...

...

If you're sending values into an arbitrary generator, who knows what you're getting? chain() will operate on arbitrary iterables, you can't expect to send values into chain([1, 2, 3], my_generator(), "xyz") and have anything sensible occur.

- is my_generator() mutable (e.g. before or during iteration)?

It doesn't matter. Sending into a chain of arbitrary iterators isn't a useful thing to do.

So, with a generator function, I get a traceback at the current yield statement. With chain() I get whatever line the chain call is on.

...

- https://docs.python.org/2/reference/expressions.html#generator.send

Why are you linking to the 2 version of the docs? We're discusing a hypotheticial new feature which must go into 3, not 2.

In your opinion, has the send() functionality changed at all?

-- Steve _______________________________________________ Python-ideas mailing list Python-ideas@python.org javascript:; https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/