Fwd: Fwd: unpacking generalisations for list comprehension
On Wed, Oct 12, 2016 at 5:41 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Oct 12, 2016 at 5:42 PM, Steven D'Aprano <steve@pearwood.info> wrote:
As it happens, python does have an external consumption operation that happens externally with an iteration implied: for t in iterable: yield t For your example [t for t in [(1, 'a'), (2, 'b'), (3, 'c')]] that would mean: for t in [(1, 'a'), (2, 'b'), (3, 'c')]: yield t And accordingly, for the latter case [*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]] it would be: for item in [(1, 'a'), (2, 'b'), (3, 'c')]: for t in item: yield t cheers! mar77i
On Thu, Oct 13, 2016 at 04:34:49PM +0200, Martti Kühne wrote:
If I had seen a list comprehension with an unpacked loop variable:
[t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]
Marttii, somehow you have lost the leading * when quoting me. What I actually wrote was: [*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]
If you replace the t with *t, you get a syntax error: py> def gen(): ... for t in [(1, 'a'), (2, 'b'), (3, 'c')]: ... yield *t File "<stdin>", line 3 yield *t ^ SyntaxError: invalid syntax Even if it was allowed, what would it mean? It could only mean "unpack the sequence t, and collect the values into a tuple; then yield the tuple".
No it wouldn't. Where does the second for loop come from? The list comprehension shown only has one loop, not nested loops. -- Steve
On Thu, Oct 13, 2016 at 6:55 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Sorry for misquoting you. Can I fix my name, though? Also, this mail was too long in my outbox so the context was lost on it. I reiterate it, risking that I would annoy some, but to be absolutely clear.
I meant that statement in context of the examples which were brought up: the occurrence of a list comprehension inside an array have the following effect: 1) [ ..., [expr for t in iterable] ] is equivalent to: def expr_long(iterable, result): result.append(iterable) return result expr_long(iterable, [ ..., ]) so, if you make the case for pep448, you might arrive at the following: 2) [ ..., *[expr for expr in iterable] ] which would be, if I'm typing it correctly, equivalent to, what resembles an external collection: def expr_star(list_comp, result): result.extend(list(list_comp)) return result expr_star(iterable, [ ..., ]) Having this in mind, the step to making: [ ..., [*expr for expr in iterable], ] from: def expr_insidestar(iterable, result): for expr in iterable: result.extend(expr) return result does not appear particularly far-fetched, at least not to me and a few people on this list. cheers! mar77i
On Thu, Oct 13, 2016 at 08:15:36PM +0200, Martti Kühne wrote:
Can I fix my name, though?
I don't understand what you mean. Your email address says your name is Martti Kühne. Is that incorrect? [...]
The good thing about this example is that it is actual runnable code that we can run to see if they are equivalent. They are not equivalent. py> def expr_long(iterable, result): ... result.append(iterable) ... return result ... py> iterable = (100, 200, 300) py> a = [..., [2*x for x in iterable]] py> b = expr_long(iterable, [...]) py> a == b False py> print(a, b) [Ellipsis, [200, 400, 600]] [Ellipsis, (100, 200, 300)] For this to work, you have to evaluate the list comprehension first, then pass the resulting list to be appended to the result. I don't think this is very insightful. All you have demonstrated is that a list display [a, b, c, ...] is equivalent to: result = [] for x in [a, b, c, ...]: result.append(x) except that you have written it in a slightly functional form.
so, if you make the case for pep448, you might arrive at the following:
2) [ ..., *[expr for expr in iterable] ]
That syntax already works (in Python 3.5): py> [1, 2, 3, *[x+1 for x in (100, 200, 300)], 4, 5] [1, 2, 3, 101, 201, 301, 4, 5]
But you don't have [..., list_comp, ] you just have the list comp. You are saying: (1) List displays [a, b, c, d, ...] are like this; (2) we can sensibly extend that to the case [a, b, *c, d, ...] I agree with (1) and (2). But then you have a leap: (3) therefore [*t for t in iterable] should mean this. There's a huge leap between the two. To even begin to make sense of this, you have to unroll the list comprehension into a list display. But that's not very helpful: [expr for t in iterable] Would you rather see that explained as: [expr, expr, expr, expr, ...] or as this? result = [] for t in iterable: result.append(expr) The second form, the standard, documented explanation for comprehensions, also applies easily to more complex examples: [expr for t in iter1 for u in iter2 for v in iter3 if condition] result = [] for t in iter1: for u in iter2: for v in iter3: if condition: result.append(expr) -- Steve
[*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]
Another problem with this is that it is very hard to generalize to the case where the item included in a comprehension is a transformation on iterated values. E.g. what does this do? [math.exp(*t) for t in [(1,2),(3,4)]] Maybe that somehow magically gets us: [2.7182, 7.38905, 20.0855, 54.5981] Or maybe the syntax would be: [*math.exp(t) for t in [(1,2),(3,4)]] Neither of those follows conventional Python semantics for function calling or sequence unpacking. So maybe that remains a type error or syntax error. But then we exclude a very common pattern of using comprehensions to create collections of *transformed* data, not simply of filtered data. In contrast, either of these are unambiguous and obvious: [math.exp(t) for t in flatten([(1,2),(3,4)])] Or: [math.exp(n) for t in [(1,2),(3,4)] for n in t] Obviously, picking math.exp() is arbitrary and any unary function would be the same issue. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Thu, Oct 13, 2016, at 14:50, David Mertz wrote:
[*map(math.exp, t) for t in [(1, 2), (3, 4)]] [*(math.exp(x) for x in t) for t in [(1, 2), (3, 4)]] I think "excluding" is a bit of a strong word - just because something doesn't address a mostly unrelated need doesn't mean it doesn't have any merit in its own right. Not every proposal is going to do everything. I think the key is that the person originally asking this thought of *x as a generalized "yield from x"-ish thing, for example: "a, *b, c" becomes "def f(): yield a; yield from b; yield c;" [a, *b, c] == list(f()) (a, *b, c) == tuple(f()) so, under a similar 'transformation', "*foo for foo in bar" likewise becomes "def f(): for foo in bar: yield from foo" bar = [(1, 2), (3, 4)] (*(1, 2), *(3, 4)) == == tuple(f()) [*(1, 2), *(3, 4)] == == list(f())
On Thu, Oct 13, 2016, at 15:46, Random832 wrote:
I accidentally hit ctrl-enter while copying and pasting, causing my message to go out while my example was less thorough than intended and containing syntax errors. It was intended to read as follows: ..."*foo for foo in bar" likewise becomes def f(): for foo in bar: yield from foo a, b = (1, 2), (3, 4) bar = [a, b] (*a, *b) == (1, 2, 3, 4) == tuple(f()) # tuple(*foo for foo in bar) [*a, *b] == [1, 2, 3, 4] == list(f()) # [*foo for foo in bar]
On 13 October 2016 at 20:51, Random832 <random832@fastmail.com> wrote:
I remain puzzled. Given the well-documented and understood transformation: [fn(x) for x in lst if cond] translates to result = [] for x in lst: if cond: result.append(fn(x)) please can you explain how to modify that translation rule to incorporate the suggested syntax? Personally, I'm not even sure any more that I can *describe* the suggested syntax. Where in [fn(x) for x in lst if cond] is the * allowed? fn(*x)? *fn(x)? Only as *x with a bare variable, but no expression? Only in certain restricted types of construct which aren't expressions but are some variation on an unpacking construct? We've had a lot of examples. I think it's probably time for someone to describe the precise syntax (as BNF, like the syntax in the Python docs at https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-se... and following sections) and semantics (as an explanation of how to rewrite any syntactically valid display as a loop). It'll have to be done in the end, as part of any implementation, so why not now? Paul
On Thu, Oct 13, 2016 at 11:42 PM Paul Moore <p.f.moore@gmail.com> wrote:
if you allow result.append(1, 2, 3) to mean result.extend([1,2,3]) # which was discussed before result = [] for x in lst: if cond: result.append(*fn(x)) Or simply use result.extend([*fn(x)]) Personally, I'm not even sure any more that I can *describe* the
The star is always exactly at the place that should "handle" it. which means [*(fn(x)) for x in lst if cond]. fn(x) must be iterable as always.
I will be happy to do so, and will be happy to work with anyone else interested. Elazar
On 13 October 2016 at 21:47, אלעזר <elazarg@gmail.com> wrote:
if you allow result.append(1, 2, 3) to mean result.extend([1,2,3]) # which was discussed before
I don't (for the reasons raised before). But thank you for your explanation, it clarifies what you were proposing. And it does so within the *current* uses of the * symbol, which is good. But: 1. I'm not keen on extending append's meaning to overlap with extend's like this. 2. Your proposal does not generalise to generator expressions, set displays (without similarly modifying the set.add() method) or dictionary displays. 3. *fn(x) isn't an expression, and yet it *looks* like it should be, and in the current syntax, an expression is required in that position. To me, that suggests it would be hard to teach. [1] You can of course generalise Sjoerd's "from" proposal and then just replace "from" with "*" throughout. That avoids your requirement to change append, but at the cost of the translation no longer being a parallel to an existing use of "*". Paul [1] On a purely personal note, I'd say it's confusing, but I don't want to go back to subjective arguments, so I only note that here as an opinion, not an argument.
On Thu, Oct 13, 2016 at 11:59 PM Paul Moore <p.f.moore@gmail.com> wrote:
I think it is an unfortunate accident of syntax, the use of "yield from foo()" instead of "yield *foo()". These "mean" the same: a syntactic context that directly handles iterable as repetition, (with some guarantees regarding exceptions etc.). Alternatively, we could be writing [1, 2, from [3, 4], 5, 6]. Whether it is "from x" or "*x" is just an accident. In my mind. As you said, the proposal should be written in a much more formal way, so that it could be evaluated without confusion. I completely agree. Elazar
אלעזר wrote:
I think it is an unfortunate accident of syntax, the use of "yield from foo()" instead of "yield *foo()".
I think that was actually discussed back when yield-from was being thrashed out, but as far as I remember we didn't have * in list displays then, so the argument for it was weaker. If we had, it might have been given more serious consideration. -- Greg
On Thu, Oct 13, 2016, at 16:59, Paul Moore wrote:
I think the "append(*x)" bit was just a flourish to try to explain it in terms of the current use of * since you don't seem to understand it any other way, rather than an actual proposal to actually change the append method.
Basically it would make the following substitutions in the conventional "equivalent loops" generator yield => yield from list append => extend set add => update dict __setitem__ => update dict comprehensions would need to use **x - {*x for x in y} would be a set comprehension.
I can think of another position an expression used to be required in: Python 3.5.2
[1, *(2, 3), 4] [1, 2, 3, 4]
Python 2.7.11
Was that hard to teach? Maybe. But it's a bit late to object now, and every single expression on the right hand side in my examples below already has a meaning. Frankly, I don't see why the pattern isn't obvious [and why people keep assuming there will be a new meaning of f(*x) as if it doesn't already have a meaning] Lists, present: [x for x in [a, b, c]] == [a, b, c] [f(x) for x in [a, b, c]] == [f(a), f(b), f(c)] [f(*x) for x in [a, b, c]] == [f(*a), f(*b), f(*c)] [f(**x) for x in [a, b, c]] == [f(**a), f(**b), f(**c)] Lists, future: [*x for x in [a, b, c]] == [*a, *b, *c] [*f(x) for x in [a, b, c]] == [*f(a), *f(b), *f(c)] [*f(*x) for x in [a, b, c]] == [*f(*a), *f(*b), *f(*c)] [*f(**x) for x in [a, b, c]] == [*f(**a), *f(**b), *f(**c)] Sets, present: {x for x in [a, b, c]} == {a, b, c} {f(x) for x in [a, b, c]} == {f(a), f(b), f(c)} {f(*x) for x in [a, b, c]} == {f(*a), f(*b), f(*c)} {f(**x) for x in [a, b, c]} == {f(**a), f(**b), f(**c)} Sets, future: {*x for x in [a, b, c]} == {*a, *b, *c} {*f(x) for x in [a, b, c]} == {*f(a), *f(b), *f(c)} {*f(*x) for x in [a, b, c]} == {*f(*a), *f(*b), *f(*c)} {*f(**x) for x in [a, b, c]} == {*f(**a), *f(**b), *f(**c)} Dicts, future: {**x for x in [a, b, c]} == {**a, **b, **c} {**f(x) for x in [a, b, c]} == {**f(a), **f(b), **f(c)} {**f(*x) for x in [a, b, c]} == {**f(*a), **f(*b), **f(*c)} {**f(**x) for x in [a, b, c]} == {**f(**a), **f(**b), **f(**c)}
Trying to restate the proposal, somewhat more formal following Random832 and Paul's suggestion. I only speak about the single star. --- *The suggested change of syntax:* comprehension ::= starred_expression comp_for *Semantics:* (In the following, f(x) must always evaluate to an iterable) 1. List comprehension: result = [*f(x) for x in iterable if cond] Translates to result = [] for x in iterable: if cond: result.extend(f(x)) 2. Set comprehension: result = {*f(x) for x in iterable if cond} Translates to result = set() for x in iterable: if cond: result.update(f(x)) 3. Generator expression: (*f(x) for x in iterable if cond) Translates to for x in iterable: if cond: yield from f(x) Elazar
Regarding all those examples: Le 14/10/2016 à 00:08, אלעזר a écrit :
Please note that we already have a way to do those. E.G: result = [*f(x) for x in iterable if cond] can currently been expressed as: >>> iterable = range(10) >>> f = lambda x: [x] * x >>> [y for x in iterable if x % 2 == 0 for y in f(x)] [2, 2, 4, 4, 4, 4, 6, 6, 6, 6, 6, 6, 8, 8, 8, 8, 8, 8, 8, 8] Now I do like the new extension syntax. I find it more natural, and more readable: >>> [*f(x) for x in iterable if x % 2 == 0] But it's not a missing feature, it's really just a (rather nice) syntaxic improvement.
בתאריך יום ו׳, 14 באוק' 2016, 12:19, מאת Michel Desmoulin < desmoulinmichel@gmail.com>:
It is about lifting restrictions from an existing syntax. That this behavior is being *explicitly disabled* in the implementation is a strong evidence, in my mind. (There are more restrictions I was asked not to divert this thread, which makes sense) Elazar
On Thu, Oct 13, 2016 at 05:30:49PM -0400, Random832 wrote:
Frankly, I don't see why the pattern isn't obvious
*shrug* Maybe your inability to look past your assumptions and see things from other people's perspective is just as much a blind spot as our inability to see why you think the pattern is obvious. We're *all* having difficulty in seeing things from the other side's perspective here. Let me put it this way: as far as I am concerned, sequence unpacking is equivalent to manually replacing the sequence with its items: t = (1, 2, 3) [100, 200, *t, 300] is equivalent to replacing "*t" with "1, 2, 3", which gives us: [100, 200, 1, 2, 3, 300] That's nice, simple, it makes sense, and it works in sufficiently recent Python versions. It applies to function calls and assignments: func(100, 200, *t) # like func(100, 200, 1, 2, 3) a, b, c, d, e = 100, 200, *t # like a, b, c, d, e = 100, 200, 1, 2, 3 although it doesn't apply when the star is on the left hand side: a, b, *x, e = 1, 2, 3, 4, 5, 6, 7 That requires a different model for starred names, but *that* model is similar to its role in function parameters: def f(*args). But I digress. Now let's apply that same model of "starred expression == expand the sequence in place" to a list comp: iterable = [t] [*t for t in iterable] If you do the same manual replacement, you get: [1, 2, 3 for t in iterable] which isn't legal since it looks like a list display [1, 2, ...] containing invalid syntax. The only way to have this make sense is to use parentheses: [(1, 2, 3) for t in iterable] which turns [*t for t in iterable] into a no-op. Why should the OP's complicated, hard to understand (to many of us) interpretation take precedence over the simple, obvious, easy to understand model of sequence unpacking that I describe here? That's not a rhetorical question. If you have a good answer, please share it. But I strongly believe that on the evidence of this thread, [a, b, *t, d] is easy to explain, teach and understand, while: [*t for t in iterable] will be confusing, hard to teach and understand except as "magic syntax" -- it works because the interpreter says it works, not because it follows from the rules of sequence unpacking or comprehensions. It might as well be spelled: [ MAGIC!!!! HAPPENS!!!! HERE!!!! t for t in iterable] except it is shorter. Of course, ultimately all syntax is "magic", it all needs to be learned. There's nothing about + that inherently means plus. But we should strongly prefer to avoid overloading the same symbol with distinct meanings, and * is one of the most heavily overloaded symbols in Python: - multiplication and exponentiation - wildcard imports - globs, regexes - collect arguments and kwargs - sequence unpacking - collect unused elements from a sequence and maybe more. This will add yet another special meaning: - expand the comprehension ("extend instead of append"). If we're going to get this (possibly useful?) functionality, I'd rather see an explicit flatten() builtin, or see it spelled: [from t for t in sequence] which at least is *obviously* something magical, than yet another magic meaning to the star operator. Its easy to look it up in the docs or google for it, and doesn't look like Perlish line noise. -- Steve
On Fri, Oct 14, 2016, at 22:38, Steven D'Aprano wrote:
And as far as I am concerned, comprehensions are equivalent to manually creating a sequence/dict/set consisting of repeating the body of the comprehension to the left of "for" with the iteration variable[s] replaced in turn with each actual value.
I don't understand why it's not _just as simple_ to say: t = ('abc', 'def', 'ghi') [*x for x in t] is equivalent to replacing "x" in "*x" with, each in turn, 'abc', 'def', and 'ghi', which gives us: [*'abc', *'def', *'ghi'] just like [f(x) for x in t] would give you [f('abc'), f('def'), f('ghi')]
That's nice, simple, it makes sense, and it works in sufficiently recent Python versions.
That last bit is not an argument - every new feature works in sufficiently recent python versions. The only difference for this proposal (provided it is approved) is that the sufficiently recent python versions simply don't exist yet.
Steven D'Aprano wrote:
Um, no, you need to also *remove the for loop*, otherwise you get complete nonsense, whether * is used or not. Let's try a less degenerate example, both ways. iterable = [1, 2, 3] [t for t in iterable] To expand that, we replace t with each of the values generated by the loop and put commas between them: [1, 2, 3] Now with the star: iterable = [(1, 2, 3), (4, 5, 6), (7, 8, 9)] [*t for t in iterable] Replace *t with each of the sequence generated by the loop, with commas between: [1,2,3 , 4,5,6 , 7,8,9]
It's obvious that you're having difficulty seeing what we're seeing, but I don't know how to explain it any more clearly, I'm sorry. -- Greg
On Thu, Oct 13, 2016, at 16:42, Paul Moore wrote:
In this case * would change this to result.extend (or +=) just as result = [a, *b, c] is equivalent to: result = [] result.append(a) result.extend(b) result.append(c) result = [*x for x in lst if cond] would become: result = [] for x in lst: if cond: result.extend(x) I used yield from as my original example to include generator expressions, which should also support this.
This already has a meaning, so it's obviously "allowed", but not in a way relevant to this proposal. The elements of x are passed to fn as arguments rather than being inserted into the list. Ultimately the meaning is the same.
*fn(x)? Only as *x with a bare variable, but no expression?
Both of these would be allowed. Any expression would be allowed, but at runtime its value must be iterable, the same as other places that you can use *x.
Paul Moore wrote:
please can you explain how to modify that translation rule to incorporate the suggested syntax?
It's quite simple: when there's a '*', replace 'append' with 'extend': [*fn(x) for x in lst if cond] expands to result = [] for x in lst: if cond: result.extend(fn(x)) The people thinking that you should just stick the '*x' in as an argument to append() are misunderstanding the nature of the expansion. You can't do that, because the current expansion is based on the assumption that the thing being substituted is an expression, and '*x' is not a valid expression on its own. A new rule is needed to handle that case. And I'm the one who *invented* that expansion, so I get to say what it means. :-) -- Greg
Paul Moore wrote:
Where in [fn(x) for x in lst if cond] is the * allowed? fn(*x)? *fn(x)?
Obviously you're *allowed* to put fn(*x), because that's already a valid function call, but the only *new* place we're talking about, and proposing new semantics for, is in front of the expression representing items to be added to the list, i.e. [*fn(x) for ...]
Replace comprehension ::= expression comp_for with comprehension ::= (expression | "*" expression) comp_for
and semantics (as an explanation of how to rewrite any syntactically valid display as a loop).
The expansion of the "*" case is the same as currently except that 'append' is replaced by 'extend' in a list comprehension, 'yield' is replaced by 'yield from' in a generator comprehension. If we decided to also allow ** in dict comprehensions, then the expansion would use 'update'. -- Greg
On 14 October 2016 at 07:54, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Thanks. That does indeed clarify. Part of my confusion was that I'm sure I'd seen someone give an example along the lines of [(x, *y, z) for ...] which *doesn't* conform to the above syntax. OTOH, it is currently valid syntax, just not an example of *this* proposal (that's part of why all this got very confusing). So now I understand what's being proposed, which is good. I don't (personally) find it very intuitive, although I'm completely capable of using the rules given to establish what it means. In practical terms, I'd be unlikely to use or recommend it - not because of anything specific about the proposal, just because it's "confusing". I would say the same about [(x, *y, z) for ...]. IMO, combining unpacking and list (or other types of) comprehensions leads to obfuscated code. Each feature is fine in isolation, but over-enthusiastic use of the ability to combine them harms readability. So I'm now -0 on this proposal. There's nothing *wrong* with it, and I now see how it can be justified as a generalisation of current rules. But I can't think of any real-world case where using the proposed syntax would measurably improve code maintainability or comprehensibility. Paul
On 14 October 2016 at 10:48, Paul Moore <p.f.moore@gmail.com> wrote:
Thinking some more about this, is it not true that [ *expression for var in iterable ] is the same as [ x for var in iterable for x in expression ] ? If so, then this proposal adds no new expressiveness, merely a certain amount of "compactness". Which isn't necessarily a bad thing, but it's clearly controversial whether the compact version is more readable / "intuitive" in this case. Given the lack of any clear improvement, I'd be inclined to think that "explicit is better than implicit" applies here, and reject the new proposal. Paul.
On Sat, Oct 15, 2016 at 3:06 PM, Paul Moore <p.f.moore@gmail.com> wrote:
is the same as
[ x for var in iterable for x in expression ]
correction, that would be: [var for expression in iterable for var in expression] you are right, though. List comprehensions are already stackable. TIL. cheers! mar77i
On 15.10.2016 16:47, Martti Kühne wrote:
Good catch, Paul. Comprehensions appear to be a special case when it comes to unpacking as they provide an alternative path. So, nested comprehensions seem to unintuitive to those who actually favor the *-variant. ;) Anyway, I don't think that it's a strong argument against the proposal. ~10 other ways are available to do what * does and this kind of argument did not prevent PEP448. What's more (and which I think is a more important response to the nested comprehension alternative) is that nested comprehensions are rarely used, and usually get long quite easily. To be practical here, let's look at an example I remembered this morning (taken from real-world code I needed to work with lately): return [(language, text) for language, text in fulltext_tuples] That's the minimum comprehension. So, you need to make it longer already to do **actual** work like filtering or mapping (otherwise, just return fulltext_tuples). So, we go even longer (and/or less readable): return [t for t in tuple for tuple in fulltext_tuples if tuple[0] == 'english'] return chain.from_iterable((language, text) for language, text in fulltext_tuples if language == 'english']) I still think the * variant would have its benefits here: return [*(language, text) for language, text in fulltext_tuples if language == 'english'] (Why it should be unpacked, you wonder? It's because of executemany of psycopg2.] Cheers, Sven
Random832 wrote:
[*map(math.exp, t) for t in [(1, 2), (3, 4)]]
[*(math.exp(x) for x in t) for t in [(1, 2), (3, 4)]]
Or more simply, [math.exp(x) for t in [(1, 2), (3, 4)] for x in t] I think this brings out an important point. While it would be nice to allow * unpacking in comprehensions for consistency with displays, it's not strictly necessary, since you can always get the same effect with another level of looping. So it comes down to whether you think added conistency, plus maybe some efficiency gains in some cases, are worth making the change. -- Greg
After having followed this thread for a while, it occured to me that the reason that the idea is confusing, is because the spelling is confusing. I think the suggested spelling (`*`) is the confusing part. If it were to be spelled `from ` instead, it would be less confusing. Consider this: g = (f(t) for t in iterable) is "merely" sugar for def gen(): for t in iterable: yield f(t) g = gen() Likewise, l = [f(t) for t in iterable] can be seen as sugar for def gen(): for t in iterable: yield f(t) l = list(gen()) Now the suggested spelling l = [*f(t) for t in iterable] is very confusing, from what I understand: what does the `*` even mean here. However, consider the following spelling: l = [from f(t) for t in iterable] To me, it does not seem far-fetched that this would mean: def gen(): for t in iterable: yield from f(t) l = list(gen()) It follows the "rule" quite well: given a generator display, everything before the first "for" gets placed after "yield ", and all the `for`/`if`s are expanded to suites. Now I'm not sure if I'm a fan of the idea, but I think that at least the `from `-spelling is less confusing than the `*`-spelling. (Unless I totally misunderstood what the `*`-spelling was about, given how confusing it supposedly is. Maybe it confused me.) On Fri, Oct 14, 2016 at 03:55:46AM +1100, Steven D'Aprano wrote:
On 13 October 2016 at 21:40, Sjoerd Job Postmus <sjoerdjob@sjoerdjob.com> wrote:
Thank you. This is the type of precise definition I was asking for in my previous post (your timing was superb!) I'm not sure I *like* the proposal, but I need to come up with some reasonable justification for my feeling, whereas for previous proposals the "I don't understand what you're suggesting" was the overwhelming feeling, and stifled any genuine discussion of merits or downsides. Paul PS I can counter a suggestion of using *f(t) rather than from f(t) in the above, by saying that it adds yet another meaning to the already heavily overloaded * symbol. The suggestion of "from" avoids this as "from" only has a few meanings already. (You can agree or disagree with my view, but at least we're debating the point objectively at that point!)
Paul Moore wrote:
We've *already* given it that meaning in non-comprehension list displays, though, so we're not really adding any new meanings for it -- just allowing it to have that meaning in a place where it's currently disallowed. Something I've just noticed -- the Language Reference actually defines both ordinary list displays and list comprehensions as "displays", and says that a display can contain either a comprehension or an explicit list of values. It has to go out of its way a bit to restrict the * form to non-comprehensions. -- Greg
On Thu, Oct 13, 2016 at 10:40:19PM +0200, Sjoerd Job Postmus wrote:
But that is *not* how list comprehensions are treated today. Perhaps they should be? https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-se... (Aside: earlier I contrasted "list display" from "list comprehension". In fact according to the docs, a comprehension is a kind of display, a special case of display. Nevertheless, my major point still holds: a list display like [1, 2, 3] is not the same as a list comprehension like [a+1 for a in (0, 1, 2)].) There may be some conceptual benefits to switching to a model where list/set/dict displays are treated as list(gen_expr) etc. But that still leaves the question of what "yield *t" is supposed to mean? Consider the analogy with f(*t), where t = (a, b, c). We *don't* have: f(*t) is equivalent to f(a); f(b); f(c) So why would yield *t give us this? yield a; yield b; yield c By analogy with the function call syntax, it should mean: yield (a, b, c) That is, t is unpacked, then repacked to a tuple, then yielded.
Indeed. The reader may be forgiven for thinking that this is yet another unrelated and arbitrary use of * to join the many other uses: - mathematical operator; - glob and regex wild-card; - unpacking; - import all - and now yield from
Now we're starting to move towards a reasonable proposal. It still requires a conceptual shift in how list comprehensions are documented, but at least now the syntax is no longer so arbitrary. -- Steve
On Thu, Oct 13, 2016, at 18:15, Steven D'Aprano wrote:
Consider the analogy with f(*t), where t = (a, b, c). We *don't* have:
f(*t) is equivalent to f(a); f(b); f(c)
I don't know where this "analogy" is coming from. f(*t) == f(a, b, c) [*t] == [a, b, c] {*t} == {a, b, c} All of this is true *today*. t, u, v = (a, b, c), (d, e, f), (g, h, i) f(*t, *u, *v) == f(a, b, c, d, e, f, g, h, i) [*t, *u, *v] == [a, b, c, d, e, f, g, h, i]
How is it arbitrary?
This is unpacking. It unpacks the results into the destination. There's a straight line from [*t, *u, *v] to [*x for x in (t, u, v)]. What's surprising is that it doesn't work now. I think last month we even had someone who didn't know about 'yield from' propose 'yield *x' for exactly this feature. It is intuitive - it is a straight-line extension of the unpacking syntax.
- import all - and now yield from
On Thu, Oct 13, 2016 at 11:32:49PM -0400, Random832 wrote:
I'm explicitly saying that we DON'T have that behaviour with function calls. f(*t) is NOT expanded to f(a), f(b), f(c). I even emphasised the "don't" part of my sentence above. And yet, this proposal wants to expand [*t for t in iterable] into the equivalent of: result = [] for t in iterable: a, b, c = *t result.append(a) result.append(b) result.append(c) Three separate calls to append, analogous to three separate calls to f(). The point I am making is that this proposed change is *not* analogous to the way sequence unpacking works in other contexts. I'm sorry if I wasn't clear enough. [...]
It is arbitrary because the suggested use of *t in list comprehensions has no analogy to the use of *t in other contexts. As far as I can see, this is not equivalent to the way sequence (un)packing works on *either* side of assignment. It's not equivalent to the way sequence unpacking works in function calls, or in list displays. It's this magical syntax which turns a virtual append() into extend(): # [t for t in iterable] result = [] for t in iterable: result.append(t) # but [*t for t in iterable] result = [] for t in iterable: result.extend(t) or, if you prefer, keep the append but magical add an extra for-loop: # [*t for t in iterable] result = [] for t in iterable: for x in t: result.append(x)
If it were unpacking as it is understood today, with no other changes, it would be a no-op. (To be technical, it would convert whatever iterable t is into a tuple.) I've covered that in an earlier post: if you replace *t with the actual items of t, you DON'T get: result = [] for t in iterable: a, b, c = *t # assuming t has three items, as per above result.append(a) result.append(b) result.append(c) as desired, but: result = [] for t in iterable: a, b, c = *t result.append((a, b, c)) which might as well be a no-op. To make this work, the "unpacking operator" needs to do more than just unpack. It has to either change append into extend, or equivalently, add an extra for loop into the list comprehension.
There's a straight line from [*t, *u, *v] to [*x for x in (t, u, v)]. What's surprising is that it doesn't work now.
I'm not surprised that it doesn't work. I expected that it wouldn't work. When I first saw the suggestion, I thought "That can't possibly be meaningful, it should be an error." Honestly Random832, I cannot comprehend how you see this as a straightforward obvious extension from existing behaviour. To me, this is nothing like the existing behaviour, and it contradicts the way sequence unpacking works everywhere else. I do not understand the reasoning you use to conclude that this is a straight-line extension to the current behaviour. Nothing I have seen in any of this discussion justifies that claim to me. I don't know what you are seeing that I cannot see. My opinion is that you're seeing things that aren't there -- I expect that your opinion is that I'm blind.
Except for all the folks who have repeatedly said that it is counter-intuitive, that it is a twisty, unexpected, confusing path from the existing behaviour to this proposal. -- Steve
On Sat, Oct 15, 2016, at 04:00, Steven D'Aprano wrote:
If that were true, it would be a no-op everywhere.
I've covered that in an earlier post: if you replace *t with the actual items of t, you DON'T get:
Replacing it _with the items_ is not the same thing as replacing it _with a sequence containing the items_, and you're trying to pull a fast one by claiming it is by using the fact that the "equivalent loop" (which is and has always been a mere fiction, not a real transformation actually performed by the interpreter) happens to use a sequence of tokens that would cause a tuple to be created if a comma appears in the relevant position.
To make this work, the "unpacking operator" needs to do more than just unpack. It has to either change append into extend,
Yes, that's what unpacking does. In every context where unpacking means anything at all, it does something to arrange for the sequence's elements to be included "unbracketed" in the context it's being ultimately used in. It's no different from changing BUILD_LIST (equivalent to creating an empty list and appending each item) to BUILD_LIST_UNPACK (equivalent to creating an empty list and extending with each item). Imagine that we were talking about ordinary list displays, and for some reason had developed a tradition of explaining them in terms of "equivalent" code the way we do for comprehensions. x = [a, b, c] is equivalent to: x = list() x.append(a) x.append(b) x.append(c) So now if we replace c with *c [where c == [d, e]], must we now say this? x = list() x.append(a) x.append(b) x.append(d, e) Well, that's just not valid at all. Clearly we must reject this ridiculous notion of allowing starred expressions within list displays, because we _can't possibly_ change the transformation to accommodate the new feature.
On Sat, Oct 15, 2016 at 04:42:13AM -0400, Random832 wrote:
That's clearly not the case. x = (1, 2, 3) [100, 200, *x, 300] If you do it the way I say, and replace *x with the individual items of x, you get this: [100, 200, 1, 2, 3, 300] which conveniently happens to be what you already get in Python. You claim that if I were write it should be a no-op -- that doesn't follow. Why would it be a no-op? I've repeatedly shown the transformation to use, and it clearly does what I say it should. How could it not?
I don't think I ever used the phrasing "a sequence containing the items". I think that's *your* phrase, not mine. I may have said "with the sequence of items" or words to that effect. These two phrases do have different meanings: x = (1, 2, 3) [100, 200, *x, 300] # Replace *x with "a sequence containing items of x" [100, 200, [1, 2, 3], 300] # Replace *x with "the sequence of items of x" [100, 200, 1, 2, 3, 300] Clearly they have different meanings. I'm confident that I've always made it clear that I'm referring to the second, not the first, but I'm only human and if I've been unclear or used the wrong phrasing, my apologies. But nit-picking about the exact phrasing used aside, it is clear that expanding the *t in a list comprehension: [*t for t in iterable] to flatten the iterable cannot be analogous to this. Whatever explanation you give for why *t expands the list comprehension, it cannot be given in terms of replacing *t with the items of t. There has to be some magic to give it the desired special behaviour.
I don't know what "equivalent loop" you are accusing me of misusing. The core developers have made it absolutely clear that changing the fundamental equivalence of list comps as syntactic sugar for: result = [] for t in iterable: result.append(t) is NOT NEGOTIABLE. (That is much to my disappointment -- I would love to introduce a "while" version of list comps to match the "if" version, but that's not an option.) So regardless of whether it is a fiction or an absolute loop, Python's list comprehensions are categorically limited to behave equivalently to the loop above (modulo scope, temporary variables, etc). If you want to change that -- change the append to an extend, for example -- you need to make a case for that change which is strong enough to overcome Guido's ruling. (Maybe Guido will be willing to bend his ruling to allow extend as well.) There are three ways to get the desired flatten() behaviour from a list comp. One way is to explicitly add a second loop, which has the benefit of already working: [x for t in iterable for x in t] Another is to swap out the append for an extend: [*t for t in iterable] # real or virtual transformation, it doesn't matter result = [] for t in iterable: result.extend(t) And the third is to keep the append but insert an extra virtual loop: # real or virtual transformation, it still doesn't matter result = [] for t in iterable: for x in t: result.append(x) Neither of the second or third suggestions match the equivalent loop form given above. Neither the second nor third is an obvious extension of the way sequence unpacking works in other contexts. [...]
Right. And if we had a tradition of saying that list displays MUST be equivalent to the unpacked sequence of appends, then sequence unpacking inside a list display would be prohibited. But we have no such tradition, and sequence unpacking inside the list really is an obvious straight line extrapolation from (say) sequence unpacking inside a function call. Fortunately, we have a *different* tradition when it comes to list displays, and no ruling that *c must turn into append with multiple arguments. Our explanation of [a, b, *c] occurs at an earlier level: replace the *c with the items of c: c = [d, e] [a, b, *c] ==> [a, b, d, e] And there is no problem. Unfortuantely for you, none of this is the case for list comps. We DO have a tradition and a BDFL ruling that list comps are strictly equivalent to a loop with append. And the transformation of *t for the items of t (I don't care if it is a real transformation in the implementation, or only a fictional transformation) cannot work in a list comp. Let's make the number of items of t explicit so we don't have to worry about variable item counts: [*t for t in iterable] # t has three items [a, b, c for (a, b, c) in iterable] That's a syntax error. To avoid the syntax error, we need parentheses: [(a, b, c) for (a, b, c) in iterable] and that's a no-op. So we're back to my first response to this thread: why on earth would you expect *t in a list comprehension to flatten the iterable? It should be either an error, or a no-op.
Of course we can. I've repeatedly said we can do anything we want. If we want, we can have *t in a list comprehension be sugar for importing the sys module, or erasing your home directory. What we can't say is that "erasing your home directory" is an obvious straight-line extrapolation from existing uses of the star operator. There's nothing obvious here: this thread is proof that whatever connection (if any) between the two is non-obvious, twisted, even strange and bizarre. I have never argued against this suggested functionality: flattening iterables is obviously a useful thing to do. But: - we can already use a list comp to flatten: [x for t in iterable for x in t] - there's no obvious or clear connection between the *t in the suggested syntax and existing uses of the star operator; it might as well be spelled [magic!!!! t for t in iterable] for all the relevance sequence unpacking has; - if anyone can explain the connection they see, I'm listening; (believe me, I am *trying to understand* -- but none of the given explanations for a connection hold up as far as I am concerned) - even if we agree that there is a connection, this thread is categorical proof that it is not obvious: it has taken DOZENS of emails to (allegedly) get the message across; - if we do get syntactic sugar for flatten(), why does it have to overload the star operator for yet another meaning? Hence my earlier questions: do we really need this, and if so, does it have to be spelled *t? Neither of those questions are obviously answered with a "Yes". -- Steve
On Sat, Oct 15, 2016, at 06:38, Steven D'Aprano wrote:
It's not your phrasing, it's the actual semantic content of your claim that it would have to wrap them in a tuple.
I've never heard of this. It certainly never came up in this discussion. And it was negotiable just fine when they got rid of the leaked loop variable.
See, there it is. Why are *those* things that are allowed to be differences, but this (which could be imagined as "result += [t]" if you _really_ need a single statement where the left-hand clause is substituted in, or otherwise) is not?
On 16 October 2016 at 03:17, Random832 <random832@fastmail.com> wrote:
It's not negotiable, and that most recently came up just a few weeks ago: https://mail.python.org/pipermail/python-ideas/2016-September/042602.html
And it was negotiable just fine when they got rid of the leaked loop variable.
We wrapped it in a nested function scope without otherwise changing the syntactic equivalence, and only after generator expressions already introduced the notion of using a nested scope (creating Python 2's scoping discrepancy between "[x for x in iterable]" and "list(x for x in iterable)", which was eliminated in Python 3). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Oct 15, 2016 at 1:49 PM Steven D'Aprano <steve@pearwood.info> wrote: ...
You are confusing here two distinct roles of the parenthesis: disambiguation as in "(1 + 2) * 2", and tuple construction as in (1, 2, 3). This overload is the reason that (1) is not a 1-tuple and we must write (1,). You may argue that this overloading causes confusion and make this construct hard to understand, but please be explicit about that; even if <1, 2,3 > was the syntax for tuples, the expansion was still [(a, b, c) for (a, b, c) in iterable] Since no tuple is constructed here. Elazar
On Sun, Oct 16, 2016 at 4:38 AM, אלעזר <elazarg@gmail.com> wrote:
Square brackets create a list. I'm not sure what you're not understanding, here. The comma does have other meanings in other contexts (list/dict/set display, function parameters), but outside of those, it means "create tuple". ChrisA
On Sat, Oct 15, 2016 at 8:45 PM Chris Angelico <rosuav@gmail.com> wrote:
On a second thought you may decide whether the rule is tuple and there are exceptions, or the other way around. The point was, conceptual expansion does not "fail" just because there is an overloading in the meaning of the tokens ( and ). It might make it harder to understand, though. Elazar
On Sun, Oct 16, 2016 at 12:10 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Yes, in the same way that other operators can need to be disambiguated. You need to say (1).bit_length() because otherwise "1." will be misparsed. You need parens to say x = (yield 5) + 2, else it'd yield 7. But that's not because a tuple fundamentally needs parentheses. ChrisA
Steven D'Aprano wrote: So why would yield *t give us this?
This is a false analogy, because yield is not a function.
However, consider the following spelling:
l = [from f(t) for t in iterable]
That sentence no verb! In English, 'from' is a preposition, so one expects there to be a verb associated with it somewhere. We currently have 'from ... import' and 'yield from'. But 'from f(t) for t in iterable' ... do what? -- Greg
On Fri, Oct 14, 2016 at 08:29:28PM +1300, Greg Ewing wrote:
Neither are list comprehensions or sequence unpacking in the context of assignment: a, b, c = *t Not everything is a function. What's your point? As far as I can see, in *every* other use of sequence unpacking, *t is conceptually replaced by a comma-separated sequence of items from t. If the starred item is on the left-hand side of the = sign, we might call it "sequence packing" rather than unpacking, and it operates to collect unused items, just like *args does in function parameter lists. Neither of these are even close to what the proposed [*t for t in iterable] will do.
*shrug* I'm not married to this suggestion. It could be written [MAGIC!!! HAPPENS!!! HERE!!! t for t in iterable] if you prefer. The suggestion to use "from" came from Sjoerd Job Postmus, not me. -- Steve
Sjoerd Job Postmus wrote:
I think the suggested spelling (`*`) is the confusing part. If it were to be spelled `from ` instead, it would be less confusing.
Are you suggesting this spelling just for generator comprehensions, or for list comprehensions as well? What about dict comprehensions? -- Greg
On Fri, Oct 14, 2016 at 07:06:12PM +1300, Greg Ewing wrote:
For both generator, list and set comprehensions it makes sense, I think. For dict comprehensions: not so much. That in itself is already sign enough that probably the */** spelling would make more sense, while also allowing the `yield *foo` alternative to `yield from foo`. But what would be the meaning of `yield **foo`? Would that be `yield *foo.items()`? I have no idea.
Here's an interesting idea regarding yield **x: Right now a function containing any yield returns a generator. Therefore, it works like a generator expression, which is the lazy version of a list display. lists can only contain elements x and unpackings *x. Therefore, it would make sense to only have "yield x" and "yield *xs" (currently spelled "yield from xs") If one day, there was a motivation to provide a lazy version of a dict display, then such a function would probably have "yield key: value" or "yield **d". Such a lazy dictionary is the map stage of the famous mapreduce algorithm. It might not make sense in single processor python, but it might in distributed Python. Best, Neil On Fri, Oct 14, 2016 at 3:34 AM Sjoerd Job Postmus <sjoerdjob@sjoerdjob.com> wrote:
On Fri, Oct 14, 2016 at 07:51:18AM +0000, Neil Girdhar wrote:
No, there's no "therefore" about it. "yield from x" is not the same as "yield *x". *x is conceptually equivalent to replacing "*x" with a comma-separated sequence of individual items from x. Given x = (1, 2, 3): f(*x) is like f(1, 2, 3) [100, 200, *x, 300] is like [100, 200, 1, 2, 3, 300] a, b, c, d = 100, *x is like a, b, c, d = 100, 1, 2, 3 Now replace "yield *x" with "yield 1, 2, 3". Conveniently, that syntax already works: py> def gen(): ... yield 1, 2, 3 ... py> it = gen() py> next(it) (1, 2, 3) "yield *x" should not be the same as "yield from x". Yielding a starred expression currently isn't allowed, but if it were allowed, it would be pointless: it would be the same as unpacking x, then repacking it into a tuple. Either that, or we would have yet another special meaning for * unrelated to the existing meanings. -- Steve
On Fri, Oct 14, 2016 at 06:23:32PM +1300, Greg Ewing wrote:
Oh man, you're not even trying to be persuasive any more. You're just assuming the result that you want, then declaring that it is "necessary". :-( I have a counter proposal: suppose *x is expanded to the string literal "Nope!". Then, given y = (1, 2, 3) (say): list(*x for x in y) gives ["Nope!", "Nope!", "Nope!"], and [*x for x in y] also gives ["Nope!", "Nope!", "Nope!"]. Thus the identity is kept, and your claim of "necessity" is disproven. We already know what *x should expand to: nearly everywhere else, *x is conceptually replaced by a comma-separated sequence of the items of x. That applies to function calls, sequence unpacking and list displays. The only exceptions I can think of are *args parameters in function parameter lists, and sequence packing on the left side of an assignment, both of which work in similar fashions. But not this proposal: it wouldn't work like either of the above, hence it would be yet another unrelated use of the * operator for some special meaning. -- Steve
On Thu, Oct 13, 2016 at 04:34:49PM +0200, Martti Kühne wrote:
If I had seen a list comprehension with an unpacked loop variable:
[t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]
Marttii, somehow you have lost the leading * when quoting me. What I actually wrote was: [*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]
If you replace the t with *t, you get a syntax error: py> def gen(): ... for t in [(1, 'a'), (2, 'b'), (3, 'c')]: ... yield *t File "<stdin>", line 3 yield *t ^ SyntaxError: invalid syntax Even if it was allowed, what would it mean? It could only mean "unpack the sequence t, and collect the values into a tuple; then yield the tuple".
No it wouldn't. Where does the second for loop come from? The list comprehension shown only has one loop, not nested loops. -- Steve
On Thu, Oct 13, 2016 at 6:55 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Sorry for misquoting you. Can I fix my name, though? Also, this mail was too long in my outbox so the context was lost on it. I reiterate it, risking that I would annoy some, but to be absolutely clear.
I meant that statement in context of the examples which were brought up: the occurrence of a list comprehension inside an array have the following effect: 1) [ ..., [expr for t in iterable] ] is equivalent to: def expr_long(iterable, result): result.append(iterable) return result expr_long(iterable, [ ..., ]) so, if you make the case for pep448, you might arrive at the following: 2) [ ..., *[expr for expr in iterable] ] which would be, if I'm typing it correctly, equivalent to, what resembles an external collection: def expr_star(list_comp, result): result.extend(list(list_comp)) return result expr_star(iterable, [ ..., ]) Having this in mind, the step to making: [ ..., [*expr for expr in iterable], ] from: def expr_insidestar(iterable, result): for expr in iterable: result.extend(expr) return result does not appear particularly far-fetched, at least not to me and a few people on this list. cheers! mar77i
On Thu, Oct 13, 2016 at 08:15:36PM +0200, Martti Kühne wrote:
Can I fix my name, though?
I don't understand what you mean. Your email address says your name is Martti Kühne. Is that incorrect? [...]
The good thing about this example is that it is actual runnable code that we can run to see if they are equivalent. They are not equivalent. py> def expr_long(iterable, result): ... result.append(iterable) ... return result ... py> iterable = (100, 200, 300) py> a = [..., [2*x for x in iterable]] py> b = expr_long(iterable, [...]) py> a == b False py> print(a, b) [Ellipsis, [200, 400, 600]] [Ellipsis, (100, 200, 300)] For this to work, you have to evaluate the list comprehension first, then pass the resulting list to be appended to the result. I don't think this is very insightful. All you have demonstrated is that a list display [a, b, c, ...] is equivalent to: result = [] for x in [a, b, c, ...]: result.append(x) except that you have written it in a slightly functional form.
so, if you make the case for pep448, you might arrive at the following:
2) [ ..., *[expr for expr in iterable] ]
That syntax already works (in Python 3.5): py> [1, 2, 3, *[x+1 for x in (100, 200, 300)], 4, 5] [1, 2, 3, 101, 201, 301, 4, 5]
But you don't have [..., list_comp, ] you just have the list comp. You are saying: (1) List displays [a, b, c, d, ...] are like this; (2) we can sensibly extend that to the case [a, b, *c, d, ...] I agree with (1) and (2). But then you have a leap: (3) therefore [*t for t in iterable] should mean this. There's a huge leap between the two. To even begin to make sense of this, you have to unroll the list comprehension into a list display. But that's not very helpful: [expr for t in iterable] Would you rather see that explained as: [expr, expr, expr, expr, ...] or as this? result = [] for t in iterable: result.append(expr) The second form, the standard, documented explanation for comprehensions, also applies easily to more complex examples: [expr for t in iter1 for u in iter2 for v in iter3 if condition] result = [] for t in iter1: for u in iter2: for v in iter3: if condition: result.append(expr) -- Steve
[*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]
Another problem with this is that it is very hard to generalize to the case where the item included in a comprehension is a transformation on iterated values. E.g. what does this do? [math.exp(*t) for t in [(1,2),(3,4)]] Maybe that somehow magically gets us: [2.7182, 7.38905, 20.0855, 54.5981] Or maybe the syntax would be: [*math.exp(t) for t in [(1,2),(3,4)]] Neither of those follows conventional Python semantics for function calling or sequence unpacking. So maybe that remains a type error or syntax error. But then we exclude a very common pattern of using comprehensions to create collections of *transformed* data, not simply of filtered data. In contrast, either of these are unambiguous and obvious: [math.exp(t) for t in flatten([(1,2),(3,4)])] Or: [math.exp(n) for t in [(1,2),(3,4)] for n in t] Obviously, picking math.exp() is arbitrary and any unary function would be the same issue. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Thu, Oct 13, 2016, at 14:50, David Mertz wrote:
[*map(math.exp, t) for t in [(1, 2), (3, 4)]] [*(math.exp(x) for x in t) for t in [(1, 2), (3, 4)]] I think "excluding" is a bit of a strong word - just because something doesn't address a mostly unrelated need doesn't mean it doesn't have any merit in its own right. Not every proposal is going to do everything. I think the key is that the person originally asking this thought of *x as a generalized "yield from x"-ish thing, for example: "a, *b, c" becomes "def f(): yield a; yield from b; yield c;" [a, *b, c] == list(f()) (a, *b, c) == tuple(f()) so, under a similar 'transformation', "*foo for foo in bar" likewise becomes "def f(): for foo in bar: yield from foo" bar = [(1, 2), (3, 4)] (*(1, 2), *(3, 4)) == == tuple(f()) [*(1, 2), *(3, 4)] == == list(f())
On Thu, Oct 13, 2016, at 15:46, Random832 wrote:
I accidentally hit ctrl-enter while copying and pasting, causing my message to go out while my example was less thorough than intended and containing syntax errors. It was intended to read as follows: ..."*foo for foo in bar" likewise becomes def f(): for foo in bar: yield from foo a, b = (1, 2), (3, 4) bar = [a, b] (*a, *b) == (1, 2, 3, 4) == tuple(f()) # tuple(*foo for foo in bar) [*a, *b] == [1, 2, 3, 4] == list(f()) # [*foo for foo in bar]
On 13 October 2016 at 20:51, Random832 <random832@fastmail.com> wrote:
I remain puzzled. Given the well-documented and understood transformation: [fn(x) for x in lst if cond] translates to result = [] for x in lst: if cond: result.append(fn(x)) please can you explain how to modify that translation rule to incorporate the suggested syntax? Personally, I'm not even sure any more that I can *describe* the suggested syntax. Where in [fn(x) for x in lst if cond] is the * allowed? fn(*x)? *fn(x)? Only as *x with a bare variable, but no expression? Only in certain restricted types of construct which aren't expressions but are some variation on an unpacking construct? We've had a lot of examples. I think it's probably time for someone to describe the precise syntax (as BNF, like the syntax in the Python docs at https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-se... and following sections) and semantics (as an explanation of how to rewrite any syntactically valid display as a loop). It'll have to be done in the end, as part of any implementation, so why not now? Paul
On Thu, Oct 13, 2016 at 11:42 PM Paul Moore <p.f.moore@gmail.com> wrote:
if you allow result.append(1, 2, 3) to mean result.extend([1,2,3]) # which was discussed before result = [] for x in lst: if cond: result.append(*fn(x)) Or simply use result.extend([*fn(x)]) Personally, I'm not even sure any more that I can *describe* the
The star is always exactly at the place that should "handle" it. which means [*(fn(x)) for x in lst if cond]. fn(x) must be iterable as always.
I will be happy to do so, and will be happy to work with anyone else interested. Elazar
On 13 October 2016 at 21:47, אלעזר <elazarg@gmail.com> wrote:
if you allow result.append(1, 2, 3) to mean result.extend([1,2,3]) # which was discussed before
I don't (for the reasons raised before). But thank you for your explanation, it clarifies what you were proposing. And it does so within the *current* uses of the * symbol, which is good. But: 1. I'm not keen on extending append's meaning to overlap with extend's like this. 2. Your proposal does not generalise to generator expressions, set displays (without similarly modifying the set.add() method) or dictionary displays. 3. *fn(x) isn't an expression, and yet it *looks* like it should be, and in the current syntax, an expression is required in that position. To me, that suggests it would be hard to teach. [1] You can of course generalise Sjoerd's "from" proposal and then just replace "from" with "*" throughout. That avoids your requirement to change append, but at the cost of the translation no longer being a parallel to an existing use of "*". Paul [1] On a purely personal note, I'd say it's confusing, but I don't want to go back to subjective arguments, so I only note that here as an opinion, not an argument.
On Thu, Oct 13, 2016 at 11:59 PM Paul Moore <p.f.moore@gmail.com> wrote:
I think it is an unfortunate accident of syntax, the use of "yield from foo()" instead of "yield *foo()". These "mean" the same: a syntactic context that directly handles iterable as repetition, (with some guarantees regarding exceptions etc.). Alternatively, we could be writing [1, 2, from [3, 4], 5, 6]. Whether it is "from x" or "*x" is just an accident. In my mind. As you said, the proposal should be written in a much more formal way, so that it could be evaluated without confusion. I completely agree. Elazar
אלעזר wrote:
I think it is an unfortunate accident of syntax, the use of "yield from foo()" instead of "yield *foo()".
I think that was actually discussed back when yield-from was being thrashed out, but as far as I remember we didn't have * in list displays then, so the argument for it was weaker. If we had, it might have been given more serious consideration. -- Greg
On Thu, Oct 13, 2016, at 16:59, Paul Moore wrote:
I think the "append(*x)" bit was just a flourish to try to explain it in terms of the current use of * since you don't seem to understand it any other way, rather than an actual proposal to actually change the append method.
Basically it would make the following substitutions in the conventional "equivalent loops" generator yield => yield from list append => extend set add => update dict __setitem__ => update dict comprehensions would need to use **x - {*x for x in y} would be a set comprehension.
I can think of another position an expression used to be required in: Python 3.5.2
[1, *(2, 3), 4] [1, 2, 3, 4]
Python 2.7.11
Was that hard to teach? Maybe. But it's a bit late to object now, and every single expression on the right hand side in my examples below already has a meaning. Frankly, I don't see why the pattern isn't obvious [and why people keep assuming there will be a new meaning of f(*x) as if it doesn't already have a meaning] Lists, present: [x for x in [a, b, c]] == [a, b, c] [f(x) for x in [a, b, c]] == [f(a), f(b), f(c)] [f(*x) for x in [a, b, c]] == [f(*a), f(*b), f(*c)] [f(**x) for x in [a, b, c]] == [f(**a), f(**b), f(**c)] Lists, future: [*x for x in [a, b, c]] == [*a, *b, *c] [*f(x) for x in [a, b, c]] == [*f(a), *f(b), *f(c)] [*f(*x) for x in [a, b, c]] == [*f(*a), *f(*b), *f(*c)] [*f(**x) for x in [a, b, c]] == [*f(**a), *f(**b), *f(**c)] Sets, present: {x for x in [a, b, c]} == {a, b, c} {f(x) for x in [a, b, c]} == {f(a), f(b), f(c)} {f(*x) for x in [a, b, c]} == {f(*a), f(*b), f(*c)} {f(**x) for x in [a, b, c]} == {f(**a), f(**b), f(**c)} Sets, future: {*x for x in [a, b, c]} == {*a, *b, *c} {*f(x) for x in [a, b, c]} == {*f(a), *f(b), *f(c)} {*f(*x) for x in [a, b, c]} == {*f(*a), *f(*b), *f(*c)} {*f(**x) for x in [a, b, c]} == {*f(**a), *f(**b), *f(**c)} Dicts, future: {**x for x in [a, b, c]} == {**a, **b, **c} {**f(x) for x in [a, b, c]} == {**f(a), **f(b), **f(c)} {**f(*x) for x in [a, b, c]} == {**f(*a), **f(*b), **f(*c)} {**f(**x) for x in [a, b, c]} == {**f(**a), **f(**b), **f(**c)}
Trying to restate the proposal, somewhat more formal following Random832 and Paul's suggestion. I only speak about the single star. --- *The suggested change of syntax:* comprehension ::= starred_expression comp_for *Semantics:* (In the following, f(x) must always evaluate to an iterable) 1. List comprehension: result = [*f(x) for x in iterable if cond] Translates to result = [] for x in iterable: if cond: result.extend(f(x)) 2. Set comprehension: result = {*f(x) for x in iterable if cond} Translates to result = set() for x in iterable: if cond: result.update(f(x)) 3. Generator expression: (*f(x) for x in iterable if cond) Translates to for x in iterable: if cond: yield from f(x) Elazar
Regarding all those examples: Le 14/10/2016 à 00:08, אלעזר a écrit :
Please note that we already have a way to do those. E.G: result = [*f(x) for x in iterable if cond] can currently been expressed as: >>> iterable = range(10) >>> f = lambda x: [x] * x >>> [y for x in iterable if x % 2 == 0 for y in f(x)] [2, 2, 4, 4, 4, 4, 6, 6, 6, 6, 6, 6, 8, 8, 8, 8, 8, 8, 8, 8] Now I do like the new extension syntax. I find it more natural, and more readable: >>> [*f(x) for x in iterable if x % 2 == 0] But it's not a missing feature, it's really just a (rather nice) syntaxic improvement.
בתאריך יום ו׳, 14 באוק' 2016, 12:19, מאת Michel Desmoulin < desmoulinmichel@gmail.com>:
It is about lifting restrictions from an existing syntax. That this behavior is being *explicitly disabled* in the implementation is a strong evidence, in my mind. (There are more restrictions I was asked not to divert this thread, which makes sense) Elazar
On Thu, Oct 13, 2016 at 05:30:49PM -0400, Random832 wrote:
Frankly, I don't see why the pattern isn't obvious
*shrug* Maybe your inability to look past your assumptions and see things from other people's perspective is just as much a blind spot as our inability to see why you think the pattern is obvious. We're *all* having difficulty in seeing things from the other side's perspective here. Let me put it this way: as far as I am concerned, sequence unpacking is equivalent to manually replacing the sequence with its items: t = (1, 2, 3) [100, 200, *t, 300] is equivalent to replacing "*t" with "1, 2, 3", which gives us: [100, 200, 1, 2, 3, 300] That's nice, simple, it makes sense, and it works in sufficiently recent Python versions. It applies to function calls and assignments: func(100, 200, *t) # like func(100, 200, 1, 2, 3) a, b, c, d, e = 100, 200, *t # like a, b, c, d, e = 100, 200, 1, 2, 3 although it doesn't apply when the star is on the left hand side: a, b, *x, e = 1, 2, 3, 4, 5, 6, 7 That requires a different model for starred names, but *that* model is similar to its role in function parameters: def f(*args). But I digress. Now let's apply that same model of "starred expression == expand the sequence in place" to a list comp: iterable = [t] [*t for t in iterable] If you do the same manual replacement, you get: [1, 2, 3 for t in iterable] which isn't legal since it looks like a list display [1, 2, ...] containing invalid syntax. The only way to have this make sense is to use parentheses: [(1, 2, 3) for t in iterable] which turns [*t for t in iterable] into a no-op. Why should the OP's complicated, hard to understand (to many of us) interpretation take precedence over the simple, obvious, easy to understand model of sequence unpacking that I describe here? That's not a rhetorical question. If you have a good answer, please share it. But I strongly believe that on the evidence of this thread, [a, b, *t, d] is easy to explain, teach and understand, while: [*t for t in iterable] will be confusing, hard to teach and understand except as "magic syntax" -- it works because the interpreter says it works, not because it follows from the rules of sequence unpacking or comprehensions. It might as well be spelled: [ MAGIC!!!! HAPPENS!!!! HERE!!!! t for t in iterable] except it is shorter. Of course, ultimately all syntax is "magic", it all needs to be learned. There's nothing about + that inherently means plus. But we should strongly prefer to avoid overloading the same symbol with distinct meanings, and * is one of the most heavily overloaded symbols in Python: - multiplication and exponentiation - wildcard imports - globs, regexes - collect arguments and kwargs - sequence unpacking - collect unused elements from a sequence and maybe more. This will add yet another special meaning: - expand the comprehension ("extend instead of append"). If we're going to get this (possibly useful?) functionality, I'd rather see an explicit flatten() builtin, or see it spelled: [from t for t in sequence] which at least is *obviously* something magical, than yet another magic meaning to the star operator. Its easy to look it up in the docs or google for it, and doesn't look like Perlish line noise. -- Steve
On Fri, Oct 14, 2016, at 22:38, Steven D'Aprano wrote:
And as far as I am concerned, comprehensions are equivalent to manually creating a sequence/dict/set consisting of repeating the body of the comprehension to the left of "for" with the iteration variable[s] replaced in turn with each actual value.
I don't understand why it's not _just as simple_ to say: t = ('abc', 'def', 'ghi') [*x for x in t] is equivalent to replacing "x" in "*x" with, each in turn, 'abc', 'def', and 'ghi', which gives us: [*'abc', *'def', *'ghi'] just like [f(x) for x in t] would give you [f('abc'), f('def'), f('ghi')]
That's nice, simple, it makes sense, and it works in sufficiently recent Python versions.
That last bit is not an argument - every new feature works in sufficiently recent python versions. The only difference for this proposal (provided it is approved) is that the sufficiently recent python versions simply don't exist yet.
Steven D'Aprano wrote:
Um, no, you need to also *remove the for loop*, otherwise you get complete nonsense, whether * is used or not. Let's try a less degenerate example, both ways. iterable = [1, 2, 3] [t for t in iterable] To expand that, we replace t with each of the values generated by the loop and put commas between them: [1, 2, 3] Now with the star: iterable = [(1, 2, 3), (4, 5, 6), (7, 8, 9)] [*t for t in iterable] Replace *t with each of the sequence generated by the loop, with commas between: [1,2,3 , 4,5,6 , 7,8,9]
It's obvious that you're having difficulty seeing what we're seeing, but I don't know how to explain it any more clearly, I'm sorry. -- Greg
On Thu, Oct 13, 2016, at 16:42, Paul Moore wrote:
In this case * would change this to result.extend (or +=) just as result = [a, *b, c] is equivalent to: result = [] result.append(a) result.extend(b) result.append(c) result = [*x for x in lst if cond] would become: result = [] for x in lst: if cond: result.extend(x) I used yield from as my original example to include generator expressions, which should also support this.
This already has a meaning, so it's obviously "allowed", but not in a way relevant to this proposal. The elements of x are passed to fn as arguments rather than being inserted into the list. Ultimately the meaning is the same.
*fn(x)? Only as *x with a bare variable, but no expression?
Both of these would be allowed. Any expression would be allowed, but at runtime its value must be iterable, the same as other places that you can use *x.
Paul Moore wrote:
please can you explain how to modify that translation rule to incorporate the suggested syntax?
It's quite simple: when there's a '*', replace 'append' with 'extend': [*fn(x) for x in lst if cond] expands to result = [] for x in lst: if cond: result.extend(fn(x)) The people thinking that you should just stick the '*x' in as an argument to append() are misunderstanding the nature of the expansion. You can't do that, because the current expansion is based on the assumption that the thing being substituted is an expression, and '*x' is not a valid expression on its own. A new rule is needed to handle that case. And I'm the one who *invented* that expansion, so I get to say what it means. :-) -- Greg
Paul Moore wrote:
Where in [fn(x) for x in lst if cond] is the * allowed? fn(*x)? *fn(x)?
Obviously you're *allowed* to put fn(*x), because that's already a valid function call, but the only *new* place we're talking about, and proposing new semantics for, is in front of the expression representing items to be added to the list, i.e. [*fn(x) for ...]
Replace comprehension ::= expression comp_for with comprehension ::= (expression | "*" expression) comp_for
and semantics (as an explanation of how to rewrite any syntactically valid display as a loop).
The expansion of the "*" case is the same as currently except that 'append' is replaced by 'extend' in a list comprehension, 'yield' is replaced by 'yield from' in a generator comprehension. If we decided to also allow ** in dict comprehensions, then the expansion would use 'update'. -- Greg
On 14 October 2016 at 07:54, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Thanks. That does indeed clarify. Part of my confusion was that I'm sure I'd seen someone give an example along the lines of [(x, *y, z) for ...] which *doesn't* conform to the above syntax. OTOH, it is currently valid syntax, just not an example of *this* proposal (that's part of why all this got very confusing). So now I understand what's being proposed, which is good. I don't (personally) find it very intuitive, although I'm completely capable of using the rules given to establish what it means. In practical terms, I'd be unlikely to use or recommend it - not because of anything specific about the proposal, just because it's "confusing". I would say the same about [(x, *y, z) for ...]. IMO, combining unpacking and list (or other types of) comprehensions leads to obfuscated code. Each feature is fine in isolation, but over-enthusiastic use of the ability to combine them harms readability. So I'm now -0 on this proposal. There's nothing *wrong* with it, and I now see how it can be justified as a generalisation of current rules. But I can't think of any real-world case where using the proposed syntax would measurably improve code maintainability or comprehensibility. Paul
On 14 October 2016 at 10:48, Paul Moore <p.f.moore@gmail.com> wrote:
Thinking some more about this, is it not true that [ *expression for var in iterable ] is the same as [ x for var in iterable for x in expression ] ? If so, then this proposal adds no new expressiveness, merely a certain amount of "compactness". Which isn't necessarily a bad thing, but it's clearly controversial whether the compact version is more readable / "intuitive" in this case. Given the lack of any clear improvement, I'd be inclined to think that "explicit is better than implicit" applies here, and reject the new proposal. Paul.
On Sat, Oct 15, 2016 at 3:06 PM, Paul Moore <p.f.moore@gmail.com> wrote:
is the same as
[ x for var in iterable for x in expression ]
correction, that would be: [var for expression in iterable for var in expression] you are right, though. List comprehensions are already stackable. TIL. cheers! mar77i
On 15.10.2016 16:47, Martti Kühne wrote:
Good catch, Paul. Comprehensions appear to be a special case when it comes to unpacking as they provide an alternative path. So, nested comprehensions seem to unintuitive to those who actually favor the *-variant. ;) Anyway, I don't think that it's a strong argument against the proposal. ~10 other ways are available to do what * does and this kind of argument did not prevent PEP448. What's more (and which I think is a more important response to the nested comprehension alternative) is that nested comprehensions are rarely used, and usually get long quite easily. To be practical here, let's look at an example I remembered this morning (taken from real-world code I needed to work with lately): return [(language, text) for language, text in fulltext_tuples] That's the minimum comprehension. So, you need to make it longer already to do **actual** work like filtering or mapping (otherwise, just return fulltext_tuples). So, we go even longer (and/or less readable): return [t for t in tuple for tuple in fulltext_tuples if tuple[0] == 'english'] return chain.from_iterable((language, text) for language, text in fulltext_tuples if language == 'english']) I still think the * variant would have its benefits here: return [*(language, text) for language, text in fulltext_tuples if language == 'english'] (Why it should be unpacked, you wonder? It's because of executemany of psycopg2.] Cheers, Sven
Random832 wrote:
[*map(math.exp, t) for t in [(1, 2), (3, 4)]]
[*(math.exp(x) for x in t) for t in [(1, 2), (3, 4)]]
Or more simply, [math.exp(x) for t in [(1, 2), (3, 4)] for x in t] I think this brings out an important point. While it would be nice to allow * unpacking in comprehensions for consistency with displays, it's not strictly necessary, since you can always get the same effect with another level of looping. So it comes down to whether you think added conistency, plus maybe some efficiency gains in some cases, are worth making the change. -- Greg
After having followed this thread for a while, it occured to me that the reason that the idea is confusing, is because the spelling is confusing. I think the suggested spelling (`*`) is the confusing part. If it were to be spelled `from ` instead, it would be less confusing. Consider this: g = (f(t) for t in iterable) is "merely" sugar for def gen(): for t in iterable: yield f(t) g = gen() Likewise, l = [f(t) for t in iterable] can be seen as sugar for def gen(): for t in iterable: yield f(t) l = list(gen()) Now the suggested spelling l = [*f(t) for t in iterable] is very confusing, from what I understand: what does the `*` even mean here. However, consider the following spelling: l = [from f(t) for t in iterable] To me, it does not seem far-fetched that this would mean: def gen(): for t in iterable: yield from f(t) l = list(gen()) It follows the "rule" quite well: given a generator display, everything before the first "for" gets placed after "yield ", and all the `for`/`if`s are expanded to suites. Now I'm not sure if I'm a fan of the idea, but I think that at least the `from `-spelling is less confusing than the `*`-spelling. (Unless I totally misunderstood what the `*`-spelling was about, given how confusing it supposedly is. Maybe it confused me.) On Fri, Oct 14, 2016 at 03:55:46AM +1100, Steven D'Aprano wrote:
On 13 October 2016 at 21:40, Sjoerd Job Postmus <sjoerdjob@sjoerdjob.com> wrote:
Thank you. This is the type of precise definition I was asking for in my previous post (your timing was superb!) I'm not sure I *like* the proposal, but I need to come up with some reasonable justification for my feeling, whereas for previous proposals the "I don't understand what you're suggesting" was the overwhelming feeling, and stifled any genuine discussion of merits or downsides. Paul PS I can counter a suggestion of using *f(t) rather than from f(t) in the above, by saying that it adds yet another meaning to the already heavily overloaded * symbol. The suggestion of "from" avoids this as "from" only has a few meanings already. (You can agree or disagree with my view, but at least we're debating the point objectively at that point!)
Paul Moore wrote:
We've *already* given it that meaning in non-comprehension list displays, though, so we're not really adding any new meanings for it -- just allowing it to have that meaning in a place where it's currently disallowed. Something I've just noticed -- the Language Reference actually defines both ordinary list displays and list comprehensions as "displays", and says that a display can contain either a comprehension or an explicit list of values. It has to go out of its way a bit to restrict the * form to non-comprehensions. -- Greg
On Thu, Oct 13, 2016 at 10:40:19PM +0200, Sjoerd Job Postmus wrote:
But that is *not* how list comprehensions are treated today. Perhaps they should be? https://docs.python.org/3.6/reference/expressions.html#displays-for-lists-se... (Aside: earlier I contrasted "list display" from "list comprehension". In fact according to the docs, a comprehension is a kind of display, a special case of display. Nevertheless, my major point still holds: a list display like [1, 2, 3] is not the same as a list comprehension like [a+1 for a in (0, 1, 2)].) There may be some conceptual benefits to switching to a model where list/set/dict displays are treated as list(gen_expr) etc. But that still leaves the question of what "yield *t" is supposed to mean? Consider the analogy with f(*t), where t = (a, b, c). We *don't* have: f(*t) is equivalent to f(a); f(b); f(c) So why would yield *t give us this? yield a; yield b; yield c By analogy with the function call syntax, it should mean: yield (a, b, c) That is, t is unpacked, then repacked to a tuple, then yielded.
Indeed. The reader may be forgiven for thinking that this is yet another unrelated and arbitrary use of * to join the many other uses: - mathematical operator; - glob and regex wild-card; - unpacking; - import all - and now yield from
Now we're starting to move towards a reasonable proposal. It still requires a conceptual shift in how list comprehensions are documented, but at least now the syntax is no longer so arbitrary. -- Steve
On Thu, Oct 13, 2016, at 18:15, Steven D'Aprano wrote:
Consider the analogy with f(*t), where t = (a, b, c). We *don't* have:
f(*t) is equivalent to f(a); f(b); f(c)
I don't know where this "analogy" is coming from. f(*t) == f(a, b, c) [*t] == [a, b, c] {*t} == {a, b, c} All of this is true *today*. t, u, v = (a, b, c), (d, e, f), (g, h, i) f(*t, *u, *v) == f(a, b, c, d, e, f, g, h, i) [*t, *u, *v] == [a, b, c, d, e, f, g, h, i]
How is it arbitrary?
This is unpacking. It unpacks the results into the destination. There's a straight line from [*t, *u, *v] to [*x for x in (t, u, v)]. What's surprising is that it doesn't work now. I think last month we even had someone who didn't know about 'yield from' propose 'yield *x' for exactly this feature. It is intuitive - it is a straight-line extension of the unpacking syntax.
- import all - and now yield from
On Thu, Oct 13, 2016 at 11:32:49PM -0400, Random832 wrote:
I'm explicitly saying that we DON'T have that behaviour with function calls. f(*t) is NOT expanded to f(a), f(b), f(c). I even emphasised the "don't" part of my sentence above. And yet, this proposal wants to expand [*t for t in iterable] into the equivalent of: result = [] for t in iterable: a, b, c = *t result.append(a) result.append(b) result.append(c) Three separate calls to append, analogous to three separate calls to f(). The point I am making is that this proposed change is *not* analogous to the way sequence unpacking works in other contexts. I'm sorry if I wasn't clear enough. [...]
It is arbitrary because the suggested use of *t in list comprehensions has no analogy to the use of *t in other contexts. As far as I can see, this is not equivalent to the way sequence (un)packing works on *either* side of assignment. It's not equivalent to the way sequence unpacking works in function calls, or in list displays. It's this magical syntax which turns a virtual append() into extend(): # [t for t in iterable] result = [] for t in iterable: result.append(t) # but [*t for t in iterable] result = [] for t in iterable: result.extend(t) or, if you prefer, keep the append but magical add an extra for-loop: # [*t for t in iterable] result = [] for t in iterable: for x in t: result.append(x)
If it were unpacking as it is understood today, with no other changes, it would be a no-op. (To be technical, it would convert whatever iterable t is into a tuple.) I've covered that in an earlier post: if you replace *t with the actual items of t, you DON'T get: result = [] for t in iterable: a, b, c = *t # assuming t has three items, as per above result.append(a) result.append(b) result.append(c) as desired, but: result = [] for t in iterable: a, b, c = *t result.append((a, b, c)) which might as well be a no-op. To make this work, the "unpacking operator" needs to do more than just unpack. It has to either change append into extend, or equivalently, add an extra for loop into the list comprehension.
There's a straight line from [*t, *u, *v] to [*x for x in (t, u, v)]. What's surprising is that it doesn't work now.
I'm not surprised that it doesn't work. I expected that it wouldn't work. When I first saw the suggestion, I thought "That can't possibly be meaningful, it should be an error." Honestly Random832, I cannot comprehend how you see this as a straightforward obvious extension from existing behaviour. To me, this is nothing like the existing behaviour, and it contradicts the way sequence unpacking works everywhere else. I do not understand the reasoning you use to conclude that this is a straight-line extension to the current behaviour. Nothing I have seen in any of this discussion justifies that claim to me. I don't know what you are seeing that I cannot see. My opinion is that you're seeing things that aren't there -- I expect that your opinion is that I'm blind.
Except for all the folks who have repeatedly said that it is counter-intuitive, that it is a twisty, unexpected, confusing path from the existing behaviour to this proposal. -- Steve
On Sat, Oct 15, 2016, at 04:00, Steven D'Aprano wrote:
If that were true, it would be a no-op everywhere.
I've covered that in an earlier post: if you replace *t with the actual items of t, you DON'T get:
Replacing it _with the items_ is not the same thing as replacing it _with a sequence containing the items_, and you're trying to pull a fast one by claiming it is by using the fact that the "equivalent loop" (which is and has always been a mere fiction, not a real transformation actually performed by the interpreter) happens to use a sequence of tokens that would cause a tuple to be created if a comma appears in the relevant position.
To make this work, the "unpacking operator" needs to do more than just unpack. It has to either change append into extend,
Yes, that's what unpacking does. In every context where unpacking means anything at all, it does something to arrange for the sequence's elements to be included "unbracketed" in the context it's being ultimately used in. It's no different from changing BUILD_LIST (equivalent to creating an empty list and appending each item) to BUILD_LIST_UNPACK (equivalent to creating an empty list and extending with each item). Imagine that we were talking about ordinary list displays, and for some reason had developed a tradition of explaining them in terms of "equivalent" code the way we do for comprehensions. x = [a, b, c] is equivalent to: x = list() x.append(a) x.append(b) x.append(c) So now if we replace c with *c [where c == [d, e]], must we now say this? x = list() x.append(a) x.append(b) x.append(d, e) Well, that's just not valid at all. Clearly we must reject this ridiculous notion of allowing starred expressions within list displays, because we _can't possibly_ change the transformation to accommodate the new feature.
On Sat, Oct 15, 2016 at 04:42:13AM -0400, Random832 wrote:
That's clearly not the case. x = (1, 2, 3) [100, 200, *x, 300] If you do it the way I say, and replace *x with the individual items of x, you get this: [100, 200, 1, 2, 3, 300] which conveniently happens to be what you already get in Python. You claim that if I were write it should be a no-op -- that doesn't follow. Why would it be a no-op? I've repeatedly shown the transformation to use, and it clearly does what I say it should. How could it not?
I don't think I ever used the phrasing "a sequence containing the items". I think that's *your* phrase, not mine. I may have said "with the sequence of items" or words to that effect. These two phrases do have different meanings: x = (1, 2, 3) [100, 200, *x, 300] # Replace *x with "a sequence containing items of x" [100, 200, [1, 2, 3], 300] # Replace *x with "the sequence of items of x" [100, 200, 1, 2, 3, 300] Clearly they have different meanings. I'm confident that I've always made it clear that I'm referring to the second, not the first, but I'm only human and if I've been unclear or used the wrong phrasing, my apologies. But nit-picking about the exact phrasing used aside, it is clear that expanding the *t in a list comprehension: [*t for t in iterable] to flatten the iterable cannot be analogous to this. Whatever explanation you give for why *t expands the list comprehension, it cannot be given in terms of replacing *t with the items of t. There has to be some magic to give it the desired special behaviour.
I don't know what "equivalent loop" you are accusing me of misusing. The core developers have made it absolutely clear that changing the fundamental equivalence of list comps as syntactic sugar for: result = [] for t in iterable: result.append(t) is NOT NEGOTIABLE. (That is much to my disappointment -- I would love to introduce a "while" version of list comps to match the "if" version, but that's not an option.) So regardless of whether it is a fiction or an absolute loop, Python's list comprehensions are categorically limited to behave equivalently to the loop above (modulo scope, temporary variables, etc). If you want to change that -- change the append to an extend, for example -- you need to make a case for that change which is strong enough to overcome Guido's ruling. (Maybe Guido will be willing to bend his ruling to allow extend as well.) There are three ways to get the desired flatten() behaviour from a list comp. One way is to explicitly add a second loop, which has the benefit of already working: [x for t in iterable for x in t] Another is to swap out the append for an extend: [*t for t in iterable] # real or virtual transformation, it doesn't matter result = [] for t in iterable: result.extend(t) And the third is to keep the append but insert an extra virtual loop: # real or virtual transformation, it still doesn't matter result = [] for t in iterable: for x in t: result.append(x) Neither of the second or third suggestions match the equivalent loop form given above. Neither the second nor third is an obvious extension of the way sequence unpacking works in other contexts. [...]
Right. And if we had a tradition of saying that list displays MUST be equivalent to the unpacked sequence of appends, then sequence unpacking inside a list display would be prohibited. But we have no such tradition, and sequence unpacking inside the list really is an obvious straight line extrapolation from (say) sequence unpacking inside a function call. Fortunately, we have a *different* tradition when it comes to list displays, and no ruling that *c must turn into append with multiple arguments. Our explanation of [a, b, *c] occurs at an earlier level: replace the *c with the items of c: c = [d, e] [a, b, *c] ==> [a, b, d, e] And there is no problem. Unfortuantely for you, none of this is the case for list comps. We DO have a tradition and a BDFL ruling that list comps are strictly equivalent to a loop with append. And the transformation of *t for the items of t (I don't care if it is a real transformation in the implementation, or only a fictional transformation) cannot work in a list comp. Let's make the number of items of t explicit so we don't have to worry about variable item counts: [*t for t in iterable] # t has three items [a, b, c for (a, b, c) in iterable] That's a syntax error. To avoid the syntax error, we need parentheses: [(a, b, c) for (a, b, c) in iterable] and that's a no-op. So we're back to my first response to this thread: why on earth would you expect *t in a list comprehension to flatten the iterable? It should be either an error, or a no-op.
Of course we can. I've repeatedly said we can do anything we want. If we want, we can have *t in a list comprehension be sugar for importing the sys module, or erasing your home directory. What we can't say is that "erasing your home directory" is an obvious straight-line extrapolation from existing uses of the star operator. There's nothing obvious here: this thread is proof that whatever connection (if any) between the two is non-obvious, twisted, even strange and bizarre. I have never argued against this suggested functionality: flattening iterables is obviously a useful thing to do. But: - we can already use a list comp to flatten: [x for t in iterable for x in t] - there's no obvious or clear connection between the *t in the suggested syntax and existing uses of the star operator; it might as well be spelled [magic!!!! t for t in iterable] for all the relevance sequence unpacking has; - if anyone can explain the connection they see, I'm listening; (believe me, I am *trying to understand* -- but none of the given explanations for a connection hold up as far as I am concerned) - even if we agree that there is a connection, this thread is categorical proof that it is not obvious: it has taken DOZENS of emails to (allegedly) get the message across; - if we do get syntactic sugar for flatten(), why does it have to overload the star operator for yet another meaning? Hence my earlier questions: do we really need this, and if so, does it have to be spelled *t? Neither of those questions are obviously answered with a "Yes". -- Steve
On Sat, Oct 15, 2016, at 06:38, Steven D'Aprano wrote:
It's not your phrasing, it's the actual semantic content of your claim that it would have to wrap them in a tuple.
I've never heard of this. It certainly never came up in this discussion. And it was negotiable just fine when they got rid of the leaked loop variable.
See, there it is. Why are *those* things that are allowed to be differences, but this (which could be imagined as "result += [t]" if you _really_ need a single statement where the left-hand clause is substituted in, or otherwise) is not?
On 16 October 2016 at 03:17, Random832 <random832@fastmail.com> wrote:
It's not negotiable, and that most recently came up just a few weeks ago: https://mail.python.org/pipermail/python-ideas/2016-September/042602.html
And it was negotiable just fine when they got rid of the leaked loop variable.
We wrapped it in a nested function scope without otherwise changing the syntactic equivalence, and only after generator expressions already introduced the notion of using a nested scope (creating Python 2's scoping discrepancy between "[x for x in iterable]" and "list(x for x in iterable)", which was eliminated in Python 3). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Oct 15, 2016 at 1:49 PM Steven D'Aprano <steve@pearwood.info> wrote: ...
You are confusing here two distinct roles of the parenthesis: disambiguation as in "(1 + 2) * 2", and tuple construction as in (1, 2, 3). This overload is the reason that (1) is not a 1-tuple and we must write (1,). You may argue that this overloading causes confusion and make this construct hard to understand, but please be explicit about that; even if <1, 2,3 > was the syntax for tuples, the expansion was still [(a, b, c) for (a, b, c) in iterable] Since no tuple is constructed here. Elazar
On Sun, Oct 16, 2016 at 4:38 AM, אלעזר <elazarg@gmail.com> wrote:
Square brackets create a list. I'm not sure what you're not understanding, here. The comma does have other meanings in other contexts (list/dict/set display, function parameters), but outside of those, it means "create tuple". ChrisA
On Sat, Oct 15, 2016 at 8:45 PM Chris Angelico <rosuav@gmail.com> wrote:
On a second thought you may decide whether the rule is tuple and there are exceptions, or the other way around. The point was, conceptual expansion does not "fail" just because there is an overloading in the meaning of the tokens ( and ). It might make it harder to understand, though. Elazar
On Sun, Oct 16, 2016 at 12:10 PM, Steven D'Aprano <steve@pearwood.info> wrote:
Yes, in the same way that other operators can need to be disambiguated. You need to say (1).bit_length() because otherwise "1." will be misparsed. You need parens to say x = (yield 5) + 2, else it'd yield 7. But that's not because a tuple fundamentally needs parentheses. ChrisA
Steven D'Aprano wrote: So why would yield *t give us this?
This is a false analogy, because yield is not a function.
However, consider the following spelling:
l = [from f(t) for t in iterable]
That sentence no verb! In English, 'from' is a preposition, so one expects there to be a verb associated with it somewhere. We currently have 'from ... import' and 'yield from'. But 'from f(t) for t in iterable' ... do what? -- Greg
On Fri, Oct 14, 2016 at 08:29:28PM +1300, Greg Ewing wrote:
Neither are list comprehensions or sequence unpacking in the context of assignment: a, b, c = *t Not everything is a function. What's your point? As far as I can see, in *every* other use of sequence unpacking, *t is conceptually replaced by a comma-separated sequence of items from t. If the starred item is on the left-hand side of the = sign, we might call it "sequence packing" rather than unpacking, and it operates to collect unused items, just like *args does in function parameter lists. Neither of these are even close to what the proposed [*t for t in iterable] will do.
*shrug* I'm not married to this suggestion. It could be written [MAGIC!!! HAPPENS!!! HERE!!! t for t in iterable] if you prefer. The suggestion to use "from" came from Sjoerd Job Postmus, not me. -- Steve
Sjoerd Job Postmus wrote:
I think the suggested spelling (`*`) is the confusing part. If it were to be spelled `from ` instead, it would be less confusing.
Are you suggesting this spelling just for generator comprehensions, or for list comprehensions as well? What about dict comprehensions? -- Greg
On Fri, Oct 14, 2016 at 07:06:12PM +1300, Greg Ewing wrote:
For both generator, list and set comprehensions it makes sense, I think. For dict comprehensions: not so much. That in itself is already sign enough that probably the */** spelling would make more sense, while also allowing the `yield *foo` alternative to `yield from foo`. But what would be the meaning of `yield **foo`? Would that be `yield *foo.items()`? I have no idea.
Here's an interesting idea regarding yield **x: Right now a function containing any yield returns a generator. Therefore, it works like a generator expression, which is the lazy version of a list display. lists can only contain elements x and unpackings *x. Therefore, it would make sense to only have "yield x" and "yield *xs" (currently spelled "yield from xs") If one day, there was a motivation to provide a lazy version of a dict display, then such a function would probably have "yield key: value" or "yield **d". Such a lazy dictionary is the map stage of the famous mapreduce algorithm. It might not make sense in single processor python, but it might in distributed Python. Best, Neil On Fri, Oct 14, 2016 at 3:34 AM Sjoerd Job Postmus <sjoerdjob@sjoerdjob.com> wrote:
On Fri, Oct 14, 2016 at 07:51:18AM +0000, Neil Girdhar wrote:
No, there's no "therefore" about it. "yield from x" is not the same as "yield *x". *x is conceptually equivalent to replacing "*x" with a comma-separated sequence of individual items from x. Given x = (1, 2, 3): f(*x) is like f(1, 2, 3) [100, 200, *x, 300] is like [100, 200, 1, 2, 3, 300] a, b, c, d = 100, *x is like a, b, c, d = 100, 1, 2, 3 Now replace "yield *x" with "yield 1, 2, 3". Conveniently, that syntax already works: py> def gen(): ... yield 1, 2, 3 ... py> it = gen() py> next(it) (1, 2, 3) "yield *x" should not be the same as "yield from x". Yielding a starred expression currently isn't allowed, but if it were allowed, it would be pointless: it would be the same as unpacking x, then repacking it into a tuple. Either that, or we would have yet another special meaning for * unrelated to the existing meanings. -- Steve
On Fri, Oct 14, 2016 at 06:23:32PM +1300, Greg Ewing wrote:
Oh man, you're not even trying to be persuasive any more. You're just assuming the result that you want, then declaring that it is "necessary". :-( I have a counter proposal: suppose *x is expanded to the string literal "Nope!". Then, given y = (1, 2, 3) (say): list(*x for x in y) gives ["Nope!", "Nope!", "Nope!"], and [*x for x in y] also gives ["Nope!", "Nope!", "Nope!"]. Thus the identity is kept, and your claim of "necessity" is disproven. We already know what *x should expand to: nearly everywhere else, *x is conceptually replaced by a comma-separated sequence of the items of x. That applies to function calls, sequence unpacking and list displays. The only exceptions I can think of are *args parameters in function parameter lists, and sequence packing on the left side of an assignment, both of which work in similar fashions. But not this proposal: it wouldn't work like either of the above, hence it would be yet another unrelated use of the * operator for some special meaning. -- Steve
participants (14)
-
Chris Angelico -
David Mertz -
Greg Ewing -
Martti Kühne -
Michel Desmoulin -
MRAB -
Neil Girdhar -
Nick Coghlan -
Paul Moore -
Random832 -
Sjoerd Job Postmus -
Steven D'Aprano -
Sven R. Kunze -
אלעזר