On 11 October 2016 at 23:50, אלעזר wrote:

I thought about it a lot recently. Specifically on your proposal, and in general. Unpacking expression can have a much more uniform treatment in the language, as an expression with special "bare tuple" type - like tuple, but "without the braces".

That's a recipe for much deeper confusion, as it would make "*expr" and "*expr," semantically identical

...

...

*range(3), (0, 1, 2)

As things stand, the above makes tuple expansion the same as any other expression: you need a comma to actually make it a tuple. If you allow a bare "*" to imply the trailing comma, then it immediately becomes confusing when you actually *do* have a comma present, as the "*" no longer implies a new tuple, it gets absorbed into the surrounding one. That's outright backwards incompatible with the status quo once you take parentheses into account: >>> (*range(3)), (0, 1, 2) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 12 October 2016 at 23:58, Sven R. Kunze wrote:

Reading PEP448 it seems to me that it's already been considered: https://www.python.org/dev/peps/pep-0448/#variations

The reason for not-inclusion were about concerns about acceptance because of "strong concerns about readability" but also received "mild support". I think your post strengthens the support given that you "expected it to just work". This shows at least to me that the concerns about readability/understandability are not justified much.

Readability isn't about "Do some people guess the same semantics for what it would mean?", as when there are only a few plausible interpretations, all the possibilities are going to get a respectable number of folks picking them as reasonable behaviour. Instead, readability is about: - Do people consistently guess the *same* interpretation? - Is that interpretation consistent with other existing uses of the syntax? - Is it more readily comprehensible than existing alternatives, or is it brevity for brevity's sake? This particular proposal fails on the first question (as too many people would expect it to mean the same thing as either "[*expr, for expr in iterable]" or "[*(expr for expr in iterable)]"), but it fails on the other two grounds as well. In most uses of *-unpacking it's adding entries to a comma-delimited sequence, or consuming entries in a comma delimited sequence (the commas are optional in some cases, but they're still part of the relevant contexts). The expansions removed the special casing of functions, and made these capabilities generally available to all sequence definition operations. Comprehensions and generator expressions, by contrast, dispense with the comma delimited format entirely, and instead use a format inspired by mathematical set builder notation (just modified to use keywords and Python expressions rather than symbols and mathematical expressions): https://en.wikipedia.org/wiki/Set-builder_notation#Sets_defined_by_a_predica... However, set builder notation doesn't inherently include the notion of flattening lists-of-lists. Instead, that's a *consumption* operation that happens externally after the initial list-of-lists has been built, and that's exactly how it's currently spelled in Python: "itertools.chain.from_iterable(subiter for subiter in iterable)". Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Oct 12, 2016 at 06:32:12PM +0200, Sven R. Kunze wrote:

On 12.10.2016 17:41, Nick Coghlan wrote:

...

This particular proposal fails on the first question (as too many people would expect it to mean the same thing as either "[*expr, for expr in iterable]" or "[*(expr for expr in iterable)]")

So, my reasoning would tell me: where have I seen * so far? *args and **kwargs!

And multiplication. And sequence unpacking.

[...] is just the list constructor.

Also indexing: dict[key] or sequence[item or slice]. The list constructor would be either list(...) or possibly list.__new__. [...] is either a list display: [1, 2, 3, 4] or a list comprehension. They are not the same thing, and they don't work the same way. The only similarity is that they use [ ] as delimiters, just like dict and sequence indexing. That doesn't mean that you can write: mydict[x for x in seq if condition] Not everything with [ ] is the same.

So, putting those two pieces together is quite simple.

I don't see that it is simple at all. I don't see any connection between function *args and list comprehension loop variables.

Furthermore, your two "interpretations" would yield the very same result as [expr for expr in iterable] which doesn't match with my experience with Python so far; especially when it comes to special characters. They must mean something. So, a simple "no-op" would not match my expectations.

Just because something would otherwise be a no-op doesn't mean that it therefore has to have some magical meaning. Python has a few no-ops which are allowed, or required, by syntax but don't do anything. pass (x) # same as just x +1 # no difference between literals +1 and 1 -0 func((expr for x in iterable)) # redundant parens for generator expr There may be more.

...

but it fails on the other two grounds as well.

Here I disagree with you. We use *args all the time, so we know what * does. I don't understand why this should not work in between brackets [...].

By this logic, *t should work... everywhere? while *args: try: raise *args except *args: del *args That's not how Python works. Just because syntax is common, doesn't mean it has to work everywhere. We cannot write: for x in import math: ... even though importing is common. *t doesn't work as the expression inside a list comprehension because that's not how list comps work. To make it work requires making this a special case and mapping [expr for t in iterable] to a list append, while [*expr for t in iterable] gets mapped to a list extend. Its okay to want that as a special feature, but understand what you are asking for: you're not asking for some restriction to be lifted, which will then automatically give you the functionality you expect. You're asking for new functionality to be added. Sequence unpacking inside list comprehensions as a way of flattening a sequence is completely new functionality which does not logically follow from the current semantics of comprehensions.

...

In most uses of *-unpacking it's adding entries to a comma-delimited sequence, or consuming entries in a comma delimited sequence (the commas are optional in some cases, but they're still part of the relevant contexts). The expansions removed the special casing of functions, and made these capabilities generally available to all sequence definition operations.

I don't know what you mean by comma-delimited sequence. There are no commas. It's just a list of entries. * adds entries to this list. (At least from my point of view.)

Not all points of view are equally valid. -- Steve

On Thu, Oct 13, 2016 at 10:37:35AM +0200, Sven R. Kunze wrote:

On 13.10.2016 01:29, Steven D'Aprano wrote:

...

On Wed, Oct 12, 2016 at 06:32:12PM +0200, Sven R. Kunze wrote:

...

So, my reasoning would tell me: where have I seen * so far? *args and **kwargs!

And multiplication.

Multiplication with only a single argument? Come on.

You didn't say anything about a single argument. Your exact words are shown above: "where have I seen * so far?". I'm pretty sure you've seen * used for multiplication. I also could have mentioned regexes, globs, and exponentiation. I cannot respond to your intended meaning, only to what you actually write. Don't blame the reader if you failed to communicate clearly and were misunderstood. [...]

About the list constructor: we construct a list by writing [a,b,c] or by writing [b for b in bs]. The end result is a list

I construct lists using all sorts of ways: list(argument) map(func, sequence) zip(a, b) file.readlines() dict.items() os.listdir('.') sorted(values) and so on. Should I call them all "list constructors" just because they return a list? No, I don't think so. Constructor has a specific meaning, and these are not all constructors -- and neither are list comprehensions.

and that matters from the end developer's point of view, no matter how fancy words you choose for it.

These "fancy words" that you dismiss are necessary technical terms. Precision in language is something we should aim for, not dismiss as unimportant. List comprehensions and list displays have different names, not to show off our knowledge of "fancy terms", but because they are different things which just happen to both return lists. Neither of them are what is commonly called a constructor. List displays are, in some senses, like a literal; list comprehensions are not, and are better understood as list builders, a process which builds a list. Emphasis should be on the *process* part: a comprehension is syntactic sugar for building a list using for-loop, not for a list display or list constructor. The bottom line is that when you see a comprehension (list, set or dict) or a generator expression, you shouldn't think of list displays, but of a for-loop. That's one of the reasons why the analogy with argument unpacking fails: it doesn't match what comprehensions *actually* are. -- Steve

On 13.10.2016 16:10, Steven D'Aprano wrote:
> On Thu, Oct 13, 2016 at 10:37:35AM +0200, Sven R. Kunze wrote:
>> Multiplication with only a single argument? Come on. 
> You didn't say anything about a single argument. Your exact words are
> shown above: "where have I seen * so far?". I'm pretty sure you've seen
> * used for multiplication. I also could have mentioned regexes, globs,
> and exponentiation.
>
> I cannot respond to your intended meaning, only to what you actually
> write. Don't blame the reader if you failed to communicate clearly and
> were misunderstood.

Steven, please. You seemed to struggle to understand the notion of the 
[*....] construct and many people (not just me) here tried their best to 
explain their intuition to you. But now it seems you don't even try to 
come behind the idea and instead try hard not to understand the help 
offered. If you don't want help or don't really want to understand the 
proposal, that's fine but please, do us a favor and don't divert the 
thread with nitpicking nonsensical details (like multiplication) and 
waste everybody's time.


The context of the proposal is about lists/dicts and the */** unpacking 
syntax. So, I actually expect you to put every post here into this very 
context. Discussions without context don't make much sense. So, I won't 
reply further.


Best,
Sven

On Thu, Oct 13, 2016 at 6:19 PM Paul Moore wrote:

On 13 October 2016 at 15:32, Sven R. Kunze wrote:

...

Steven, please. You seemed to struggle to understand the notion of the [*....] construct and many people (not just me) here tried their best to explain their intuition to you.

And yet, the fact that it's hard to explain your intuition to others (Steven is not the only one who's finding this hard to follow) surely implies that it's merely that - personal intuition - and not universal understanding.

I fail to see this implication. Perhaps you mean that the universal understanding is hard to get, intuitively. And trying to explain them is the way to figure out howw hard can this difficulty be overcome.

The *whole point* here is that not everyone understands the proposed notation the way the proposers do, and it's *hard to explain* to those people. Blaming the people who don't understand does not support the position that this notation should be added to the language. Rather, it reinforces the idea that the new proposal is hard to teach (and consequently, it may be a bad idea for Python).

It may also suggest that there are currently two ways to understand the *[...] construct, and only one of them can be generalized to lead the new proposal. So people that are *used* to the other way may have harder time than people coming with a clean slate. So it might or might not be hard to teach. (I'm not saying that's necessarily the case) I will be happy to understand that "other way" that is harder to generalize; I think this discussion may be fruitful in making these different understandings explicit. Elazar

Exactly with Paul! As I mentioned, I teach software developers and scientists Python for a living. I get paid a lot of money to do that, and have a good sense of what learners can easily understand and not (I've also written hundred of articles and a few books about Python). The people I write for and teach are educated, smart, and generally have familiarity with multiple programming languages. In my opinion, this new construct—if added to the language—would be difficult to teach, and most of my students would get it wrong most of the time. Yes, I understand the proposed semantics. It is not *intuitive* to me, but I could file the rule about the behavior if I had to. But if I were forced to teach it, it would always be "Here's a Python wart to look out for if you see it in other code... you should not ever use it yourself." On Thu, Oct 13, 2016 at 8:18 AM, Paul Moore wrote:

On 13 October 2016 at 15:32, Sven R. Kunze wrote:

...

Steven, please. You seemed to struggle to understand the notion of the [*....] construct and many people (not just me) here tried their best to explain their intuition to you.

And yet, the fact that it's hard to explain your intuition to others (Steven is not the only one who's finding this hard to follow) surely implies that it's merely that - personal intuition - and not universal understanding.

The *whole point* here is that not everyone understands the proposed notation the way the proposers do, and it's *hard to explain* to those people. Blaming the people who don't understand does not support the position that this notation should be added to the language. Rather, it reinforces the idea that the new proposal is hard to teach (and consequently, it may be a bad idea for Python).

Paul _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

I've never used nor taught a * in a list display. I don't think they seem so bad, but it's a step down a slippery slope towards forms that might as well be Perl. On Oct 13, 2016 10:33 PM, "Greg Ewing" wrote:

David Mertz wrote:

...

it would always be "Here's a Python wart to look out for if you see it in other code... you should not ever use it yourself."

Do you currently tell them the same thing about the use of * in a list display?

-- Greg _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

אלעזר writes:

Steve, you only need to allow multiple arguments to append(), then it makes perfect sense.

No, because that would be explicit. Here it's implicit and ambiguous. Specifically, it requires guessing "operator associativity". That is something people have different intuitions about.

...

On Tue, Oct 11, 2016 at 02:42:54PM +0200, Martti Kühne wrote:

...

Hello list

I love the "new" unpacking generalisations as of pep448. And I found myself using them rather regularly, both with lists and dict. Today I somehow expected that [*foo for foo in bar] was equivalent to itertools.chain(*[foo for foo in bar]), which it turned out to be a SyntaxError.

Which is what I myself would expect, same as *(1, 2) + 3 is a SyntaxError. I could see Nick's interpretation that *foo in such a context would actually mean (*foo,) (i.e., it casts iterables to tuples). I would certainly want [i, j for i, j in [[1, 2], [3, 4]]] to evaluate to [(1, 2), (3, 4)] (if it weren't a SyntaxError).

...

To me, that's a very strange thing to expect. Why would you expect that unpacking items in a list comprehension would magically lead to extra items in the resulting list? I don't think that makes any sense.

Well, that's what it does in display syntax for sequences. If you think of a comprehension as a "macro" that expands to display syntax, makes some sense. But as you and Nick point out, comprehensions are real operations, not macros which implicitly construct displays, then evaluate them to get the actual sequence.

...

Wishful thinking perhaps?

That was unnecessary. I know sometimes I fall into the trap of thinking there really ought to be concise syntax for a "simple" idea, and then making one up rather than looking it up.

On Thu, Oct 13, 2016 at 2:35 AM Steven D'Aprano wrote:

On Wed, Oct 12, 2016 at 04:11:55PM +0000, אלעזר wrote:

...

Steve, you only need to allow multiple arguments to append(), then it makes perfect sense.

I think you're missing a step. What will multiple arguments given to append do? There are two obvious possibilities:

- collect all the arguments into a tuple, and append the tuple;

- duplicate the functionality of list.extend

neither of which appeals to me.

The latter, of course. Similar to max(). Not unheard of. Elazar

What is the intuition behind [1, *x, 5]? The starred expression is replaced with a comma-separated sequence of its elements. The trailing comma Nick referred to is there, with the rule that [1,, 5] is the same as [1, 5]. All the examples follow this intuition, IIUC. Elazar בתאריך יום ד׳, 12 באוק' 2016, 22:22, מאת David Mertz ‏:

I've followed this discussion some, and every example given so far completely mystifies me and I have no intuition about what they should mean.

On Oct 12, 2016 8:43 AM, "Steven D'Aprano" wrote:

On Tue, Oct 11, 2016 at 02:42:54PM +0200, Martti Kühne wrote:

...

Hello list

I love the "new" unpacking generalisations as of pep448. And I found myself using them rather regularly, both with lists and dict. Today I somehow expected that [*foo for foo in bar] was equivalent to itertools.chain(*[foo for foo in bar]), which it turned out to be a SyntaxError.

To me, that's a very strange thing to expect. Why would you expect that unpacking items in a list comprehension would magically lead to extra items in the resulting list? I don't think that makes any sense.

Obviously we could program list comprehensions to act that way if we wanted to, but that would not be consistent with the ordinary use of list comprehensions. It would introduce a special case of magical behaviour that people will have to memorise, because it doesn't follow logically from the standard list comprehension design.

The fundamental design principle of list comps is that they are equivalent to a for-loop with a single append per loop:

[expr for t in iterable]

is equivalent to:

result = [] for t in iterable: result.append(expr)

If I had seen a list comprehension with an unpacked loop variable:

[*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]

I never in a million years would expect that running a list comprehension over a three-item sequence would magically expand to six items:

[1, 'a', 2, 'b', 3, 'c']

I would expect that using the unpacking operator would give some sort of error, or *at best*, be a no-op and the result would be:

[(1, 'a'), (2, 'b'), (3, 'c')]

append() doesn't take multiple arguments, hence a error should be the most obvious result. But if not an error, imagine the tuple unpacked to two arguments 1 and 'a' (on the first iteration), then automatically packed back into a tuple (1, 'a') just as you started with.

I think it is a clear, obvious and, most importantly, desirable property of list comprehensions with a single loop that they cannot be longer than the initial iterable that feeds them. They might be shorter, if you use the form

[expr for t in iterable if condition]

but they cannot be longer.

So I'm afraid I cannot understand what reasoning lead you to expect that unpacking would apply this way. Wishful thinking perhaps?

-- Steve _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, Oct 12, 2016 at 11:26 PM David Mertz wrote:

On Wed, Oct 12, 2016 at 12:38 PM, אלעזר wrote:

What is the intuition behind [1, *x, 5]? The starred expression is replaced with a comma-separated sequence of its elements.

I've never actually used the `[1, *x, 5]` form. And therefore, of course, I've never taught it either (I teach Python for a living nowadays). I think that syntax already perhaps goes too far, actually; but I can understand it relatively easily by analogy with:

a, *b, c = range(10)

It's not exactly "analogy" as such - it is the dual notion. Here you are using the "destructor" (functional terminology) but we are talking about "constructors". But nevermind.

But the way I think about or explain either of those is "gather the extra items from the sequence." That works in both those contexts. In contrast:

>>> *b = range(10) SyntaxError: starred assignment target must be in a list or tuple

Since nothing was assigned to a non-unpacked variable, nothing is "extra items" in the same sense. So failure feels right to me. I understand that "convert an iterable to a list" is conceptually available for that line, but we already have `list(it)` around, so it would be redundant and slightly confusing.

But that's not a uniform treatment. It might have good reasons from readability point of view, but it is an explicit exception for the rule. The desired behavior would be equivalent to b = tuple(range(10)) and yes, there are Two Ways To Do It. I would think it should have been prohibited by PEP-8 and not by the compiler. Oh well. What seems to be wanted with `[*foo for foo in bar]` is basically just

`flatten(bar)`. The latter feels like a better spelling, and the recipes in itertools docs give an implementation already (a one-liner).

We do have a possibility of writing this:

>>> [(*stuff,) for stuff in [range(-5,-1), range(5)]] [(-5, -4, -3, -2), (0, 1, 2, 3, 4)]

That's not flattened, as it should not be. But it is very confusing to have `[(*stuff) for stuff in ...]` behave differently than that. It's much more natural—and much more explicit—to write:

>>> [item for seq in [range(-5,-1), range(5)] for item in seq] [-5, -4, -3, -2, 0, 1, 2, 3, 4]

The distinction between (x) and (x,) is already deep in the language. It has nothing to do with this thread

...

...

[1, *([2],), 3] [1, [2], 3] [1, *([2]), 3] [1, 2, 3]

So there. Just like in this proposal. Elazar.

On 12 October 2016 at 20:22, David Mertz wrote:

I've followed this discussion some, and every example given so far completely mystifies me and I have no intuition about what they should mean.

Same here. On 12 October 2016 at 20:38, אלעזר wrote:

What is the intuition behind [1, *x, 5]? The starred expression is replaced with a comma-separated sequence of its elements.

The trailing comma Nick referred to is there, with the rule that [1,, 5] is the same as [1, 5].

All the examples follow this intuition, IIUC.

But intuition is precisely that - it's not based on rules, but on people's instinctive understanding. When evaluating whether something is intuitive, the *only* thing that matters is what people tell you they do or don't understand by a given construct. And in this case, people have been expressing differing interpretations, and confusion. That says "not intuitive" loud and clear to me. And yes, I find [1, *x, 5] intuitive. And I can't tell you why I find it OK, but I find {**x for x in d.items()} non-intuitive. But just because I can't explain it doesn't mean it's not true, or you can "change my mind" about how I feel. Paul

From a CPython implementation standpoint, we specifically blocked this code

First of all: +1 to Sven's very well-expressed support of the proposal, and +1 to Nick's very well-explained reasons for rejecting it. As one of the main implementers of PEP 448, I have always liked this, but I suggested that we leave this out when there was opposition since there's no rush for it. Regarding Steven's example, like Sven, I also see it this way: [*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]] should mean: [*(1, 'a'), *(2, 'b'), *(3, 'c')]] Which coincides with what the OP is asking for. At the end of this discussion it might be good to get a tally of how many people think the proposal is reasonable and logical. I imagine people will be asking this same question next year and the year after, and so it will be good to see if as familiarity with PEP 448 expands, more people will find this intuitive and logical. path, and it is only a matter of unblocking it if we want to support this. Best, Neil

Neil Girdhar wrote:

At the end of this discussion it might be good to get a tally of how many people think the proposal is reasonable and logical.

I think it's reasonable and logical. -- Greg

On Wed, Oct 26, 2016 at 01:25:48AM +0100, Rob Cliffe wrote:

(2) This is admittedly a somewhat tangential argument, but: I didn't really know what "yield from" meant. But when I read in an earlier post that someone had proposed "yield *" for it, I had a Eureka moment.

Are you aware that "yield from it" does not just mean this...? for x in it: yield x "yield from ..." is not just "some sort of unpacking". If all it did was iterate over an iterable and yield the values, it would not have been given special syntax just to save one line. It does *much* more than just those two lines. For start, it is an expression which returns a value, so you can write: result = yield from it A full implementation of "yield from ..." would be 39 lines of Python code, according to the PEP, not two. It has to handle delegating send(), throw() and close() messages, exceptions, plus of course the obvious iteration.

Which suggests if "*" is used to mean some sort of unpacking in more contexts, the more familiar and intuitive it may become. I guess the word I'm groping for is 'consistency'.

I think that there is zero hope of consistency for * the star operator. That horse has bolted. It is already used for: - multiplication and exponentiation - repetition - "zero or more of the previous element" in regular expressions - "zero or more of any character" in globbing - "everything" in imports - sequence unpacking - sequence packing - collecting positional and keyword arguments Some of which are admittedly *similar* uses, but the * operator does get overloaded for so many unrelated uses. -- Steve

On Thu, Oct 13, 2016 at 01:30:45PM -0700, Neil Girdhar wrote:

From a CPython implementation standpoint, we specifically blocked this code path, and it is only a matter of unblocking it if we want to support this.

I find that difficult to believe. The suggested change seems like it should be much bigger than just removing a block. Can you point us to the relevant code? In any case, it isn't really the difficulty of implementation that is being questioned. Many things are easy to implement, but we still don't do them. The real questions here are: (1) Should we overload list comprehensions as sugar for a flatten() function? (2) If so, should we spell that [*t for t in iterable]? Actually the answer to (1) should be "we already do". We just spell it: [x for t in iterable for x in t] -- Steve

On Sat, Oct 15, 2016 at 05:38:15PM +0000, Neil Girdhar wrote:

In ast.c, you can find:

if (is_dict) { ast_error(c, n, "dict unpacking cannot be used in " "dict comprehension"); return NULL; } res = ast_for_dictcomp(c, ch);

[...] Thanks for the pointer.

...

In any case, it isn't really the difficulty of implementation that is being questioned. Many things are easy to implement, but we still don't do them.

If it doesn't matter, why bring it up?

I never said it doesn't matter. I brought it up because I'm curious. Because if it turns out that it actually is *difficult* to implement, that would be a point in my favour that *t doesn't naturally apply in list comps. And on the other hand, if it is *easy* to implement, that's a hint that perhaps I'm missing something and there is some natural interpretation of *t in a list comp that I've missed. Perhaps. Just because I have a strong opinion doesn't mean I'm not willing to consider the possibility that I'm wrong. -- Steve