Fwd: Fwd: Fwd: unpacking generalisations for list comprehension
On Sat, Oct 15, 2016 at 10:09 AM, Steven D'Aprano <steve@pearwood.info> wrote:
You brush over the fact that *t is not limited to a replacement by a comma-separated sequence of items from t, but *t is actually a replacement by that comma-separated sequence of items from t INTO an external context. For func(*t) to work, all the elements of t are kind of "leaked externally" into the function argument list's context, and for {**{'a': 1, 'b': 2, ...}} the inner dictionary's items are kind of "leaked externally" into the outer's context. You can think of the */** operators as a promotion from append to extend, but another way to see this is as a promotion from yield to yield from. So if you want to instead of append items to a comprehension, as is done with [yield_me for yield_me in iterator], you can see this new piece as a means to [*yield_from_me for yield_from_me in iterator]. Therefore I think it's a bit confusing that yield needs a different keyword if these asterisk operators already have this intuitive promotion effect. Besides, [*thing for thing in iterable_of_iters if cond] has this cool potential for the existing any() and all() builtins for cond, where a decision can be made based on the composition of the in itself iterable thing. cheers! mar77i
On Sat, Oct 15, 2016 at 11:36:28PM +1300, Greg Ewing wrote:
Indeed. In situations where there isn't any context for the interpretation of *, it's not allowed.
You mean like in list comprehensions? Are you now supporting my argument that starring the list comprehension expression isn't meaningful? Not if star is defined as sequence unpacking in the usual way. If you want to invent a new meaning for * to make this work, to join all the other special case magic meanings for the * symbol, that's another story.
Oh look, just like now: py> iterable = [(1, 'a'), (2, 'b')] py> [(100, *t) for t in iterable] [(100, 1, 'a'), (100, 2, 'b')] Hands up anyone who expected to flatten the iterable and get [100, 1, 'a', 100, 2, 'b'] instead? Anyone? No? Take out the (100, ...) and just leave the *t, and why should it be different? It is my position that: (quote) there isn't any context for the interpretation of * for [*t for t in iterable]. Writing that is the list comp equivalent of writing x = *t. -- Steve
On Sat, Oct 15, 2016 at 12:48 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I don't know whether that should be provocating or beside the poinnt. It's probably both. You're putting two expectations on the same example: first, you make the reasonable expectation that results in [(100, 1, 'a'), (100, 2, 'b')], and then you ask whether anyone expected [100, 1, 'a', 100, 2, 'b'], but don't add or remove anything from the same example. Did you forget to put a second example using the new notation in there? Then you'd have to spell it out and start out with [*(100, *t) for t in iterable]. And then you can ask who expected [100, 1, 'a', 100, 2, 'b']. Which is what this thread is all about. cheers! mar77i
Steven D'Aprano wrote:
Are you now supporting my argument that starring the list comprehension expression isn't meaningful?
The context it's in (a form of list display) has a clear meaning for a comma-separated list of values, so there is a reasonable interpretation that it *could* be given.
The * there is in the context of constructing a tuple, not the list into which the tuple is placed. The difference is the same as the difference between these:
-- Greg
On Sun, Oct 16, 2016 at 12:48:36PM +1300, Greg Ewing wrote:
This thread is a huge, multi-day proof that people do not agree that this is a "reasonable" interpretation.
Right: the context of the star is meaningful. We all agree that *t in a list display [a, b, c, ...] is meaningful; same for tuples; same for function calls; same for sequence unpacking for assignment. What is not meaningful (except as a Perlish line-noise special case to be memorised) is *t as the list comprehension expression. I've never disputed that we could *assert* that *t in a list comp means "flatten". We could assert that it means anything we like. But it doesn't follow from the usual meaning of sequence unpacking anywhere else -- that's why it is currently a SyntaxError, and that's why people reacted with surprise at the OP who assumed that *t would magically flatten his iterable. Why would you assume that? It makes no sense to me -- that's not how sequence unpacking works in any other context, it isn't how list comprehensions work. Right from the beginning I called this "wishful thinking", and *nothing* since then has changed my mind. This proposal only makes even a little bit of sense if you imagine list comprehensions [*t for a in it1 for b in it2 for c in it3 ... for t in itN] completely unrolled into a list display: [*t, *t, *t, *t, ... ] but who does that? Why would you reason about your list comps like that? If you think about list comps as we're expected to think of them -- as list builders equivalent to a for-loop -- the use of *t there is invalid. Hence it is a SyntaxError. You want a second way to flatten your iterables? A cryptic, mysterious, Perlish line-noise way? Okay, fine, but don't pretend it is sequence unpacking -- in the context of a list comprehension, sequence unpacking doesn't make sense, it is invalid. Call it something else: the new "flatten" operator: [^t for t in iterable] for example, which magically adds an second invisible for-loop to your list comps: # expands to for t in iterable: for x in t: result.append(x) Because as your own email inadvertently reinforces, if sequence unpacking made sense in the context of a list comprehension, it would already be allowed rather than a SyntaxError: it is intentionally prohibited because it doesn't make sense in the context of list comps. -- Steve
Steven D'Aprano wrote:
This thread is a huge, multi-day proof that people do not agree that this is a "reasonable" interpretation.
So far I've seen one very vocal person who disgrees, and maybe one other who isn't sure.
Many people do, and it's a perfectly valid way to think about them. They're meant to admit a declarative reading; that's the reason they exist in the first place. The expansion in terms of for-loops and appends is just *one* way to describe the current semantics. It's not written on stone tablets brought down from a mountain. Any other way of thinking about it that gives the same result is equally valid.
magically adds an second invisible for-loop to your list comps:
You might as well say that the existing * in a list display magically inserts a for-loop into it. You can think of it that way if you want, but you don't have to.
it is intentionally prohibited because it doesn't make sense in the context of list comps.
I don't know why it's currently prohibited. You would have to ask whoever put that code in, otherwise you're just guessing about the motivation. -- Greg
On Sun, Oct 16, 2016 at 3:44 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
And what you're NOT seeing is a whole lot of people (myself included) who have mostly glazed over, unsure what is and isn't reasonable, and not clear enough on either side of the debate to weigh in. (Or not even clear what the two sides are.) ChrisA
On 16 October 2016 at 06:47, Chris Angelico <rosuav@gmail.com> wrote:
+1 There are lots of arguments of whether the new syntax is readable or not etc., but not so many arguments why we need this and what kind of problems it would solve. What I have learned from this megathread is that the syntax [*foo for foo in bar] is proposed as a replacement for a one-liner itertools.chain(*[foo for foo in bar]). I do not have any strong opinion on this, because I simply do not use such constructs frequently (if ever). -- Ivan
On Sun, Oct 16, 2016 at 03:02:55PM +0200, Ivan Levkivskyi wrote:
If people take away nothing else from this thread, it should be that flattening an iterable is as easy as: [x for t in iterable for x in t] which corresponds neatly to: for t in iterable: for x in t: result.append(x) -- Steve
On Oct 15, 2016 9:45 PM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
Steven D'Aprano wrote:
This thread is a huge, multi-day proof that people do not agree that
this is a "reasonable" interpretation.
So far I've seen one very vocal person who disgrees, and maybe one other who isn't sure.
In case it wasn't entirely clear, I strongly and vehemently opposed this unnecessary new syntax. It is confusing, bug prone, and would be difficult to teach. Or am I that very vocal person? I was thinking your meant Steven.
On 16.10.2016 07:08, David Mertz wrote:
As this discussion won't come to an end, I decided to consult my girlfriend. I started with (btw. she learned basic Python to solve some math quizzes): """ Let's install a list in another one.
Maybe, we want to remove the brackets.
['a', *meine_liste, 'b'] ['a', 2, 3, 4, 5, 'b']
Now, the problem of the discussion is the following:
[(i,i,i) for i in range(4)] [(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3)]
Let's remove these inner parentheses again.
Some guy wanted to remove that restriction. """ I said a teacher contributed to the discussion and he finds this too complicated and confusing and does not even teach * in list displays at all. Her reaction was hilarious: "Whom does he teach? Children?" Me: "What? No, everybody I think. Why?" She: "It's easy enough to remember what the star does." She also asked what would the alternative would look like. I wrote: """ the star easier." In the end, she also added: "Not everybody drives a car but they still exist." Cheers, Sven PS: off to the weekend. She's already complaint that I should spend less time inside my mailbox and more with her. ;)
On Sun, Oct 16, 2016 at 5:34 AM, Sven R. Kunze <srkunze@mail.de> wrote:
As I've said, the folks I teach are mostly working scientists with doctorates in scientific fields and years of programming experience in languages other than Python. For example, rocket scientists at NASA. Now I admit that I don't specifically know how quickly they would pick up on something I've never taught them. But I've written enough teaching materials and articles and books and I have a certain intuition. The way you explained the special case you built up is pretty good, but it's very tailored to making that specific case plausible, and is not general.
That is an absolutely terrible construct, obviously. I have to pause and think a while myself to understand what it does. It also answers a very different use case than the one that has been mostly discussed in this thread. A much better spelling is:
[i for i in range(4) for _ in range(3)] [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3]
For some related uses, itertools.repeat() is very useful. But the case that has been discussed far more often is like this:
listOfLists = [[1,2,3], [4,5,6,7], [8,9]] flatten(listOfLists)
The very long argument is that somehow that would be easier to spell as:
[*i for i in listOfLists]
It's just not easier. And there are contrary intuitions that will occur to many people based on the several other related-but-different uses of * for packing/unpacking in other contexts. Also, this simplest case might be teachable, but the more general "exactly where can I use that star in a comprehensions" will be far harder to explain plausibly. What's the pattern here?
Yes! I know those first few are actually doing something different than the proposed new syntax. But explaining that to my rocket scientists or your girlfriend in a consistent and accurate way would be a huge challenge. It would mostly come down to "don't do that, it's too confusing." -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
Her reaction was hilarious:
"Whom does he teach? Children?"
I sense mockery in your email, and it does not conform to the PSF code of conduct. Please read the CoC before posting in this mailing list. The link is available at the bottom of every python mailing list email. https://www.python.org/psf/codeofconduct/ I don't find teaching children is a laughing matter, neither is the idea of children learning to code. In Canada, we have initiatives like Girls Learning Code and Kids Learning Code. I mentored in a couple of those events and the students are girls aged 8-14. They surprised me with their abilities to learn. I would suggest looking for such mentoring opportunities in your area to gain appreciation with this regard. Thanks. (Sorry to derail everyone from the topic of list comprehension. Please continue!)
On 16/10/2016 16:41, Mariatta Wijaya wrote:
The RUE was allowed to insult the community for years and got away with it. I'm autistic, stepped across the line, and got hammered. Hypocrisy at its best. Even funnier, the BDFL has asked for my advice in recent weeks with respect to the bug tracker. I've replied, giving the bare minimum that I feel I can give within the circumstances. Yours most disgustingly. Mark Lawrence.
Actually, I agree with Marietta. I don't care whatsoever about mocking me, which was a certain element of it. I have thick skin and am confident in these conversations. The part that was probably over the line was mocking children who learn to program or those who teach them. That's a huge and great job. I know I would not have the skill to teach children effectively. Adults with technical expertise are much easier for me. That said, thank you Mark for your empirical research with a test subject. Best, David On Oct 16, 2016 9:39 AM, "Mark Lawrence via Python-ideas" < python-ideas@python.org> wrote:
On Sun, 16 Oct 2016 at 09:39 Mark Lawrence via Python-ideas < python-ideas@python.org> wrote:
What is the "RUE"?
I'm autistic, stepped across the line, and got hammered. Hypocrisy at its best.
While some of us know your background, Mark, not everyone on this list does as people join at different times, so please try to give the benefit of the doubt to people. Marietta obviously takes how children are reflected personally and was trying to point out that fact. I don't think she meant for the CoC reference to come off as threatening, just to back up why she was taking the time out to speak up that she was upset by what was said.
On 17/10/2016 19:29, Brett Cannon wrote:
Not what, who, the Resident Unicode Expert who spent two years spewing his insults at the entire Python community until the moderators finally woke up, did their job, and kicked him into touch.
This list is irrelevant. The PSF has to be consistent across all of its lists. Who the hell is Marietta, I don't recall a single post from her in 16 years of using Python? -- Hell I'm confused, why the hell do I bother???
On 10/21/2016 12:13 PM, Mark Lawrence via Python-ideas wrote:
This list is irrelevant. The PSF has to be consistent across all of its lists.
This list is not irrelevant, and yes *volunteers who moderate* should be consistent.
Who the hell is Marietta, I don't recall a single post from her in 16 years of using Python?
She is a fellow Pythonista, and you owe her an apology. -- ~Ethan~ P.S. Should you decide to bring my own stupidity a few months of ago of being insulting, you should also take note that I took responsibility for it and apologized myself.
On 10/21/16 3:13 PM, Mark Lawrence via Python-ideas wrote:
Mark, I know you viewed "Python is broken" as an insult to the maintainers, but others do not. He certainly never singled out any individual for abuse.
The fact that you haven't seen a post from Marietta before is irrelevant. She is new to this list, and has been traveling in different Python circles than you have. That doesn't make her less of a member of this community, and it doesn't diminish her point. Your vitriol towards her diminishes yours. --Ned.
On Sun, Oct 16, 2016 at 02:34:58PM +0200, Sven R. Kunze wrote:
Did you remember to tell your girlfriend that a critical property of the "??? for i in range(4)" construct is that it generates one value per loop? It loops four times, so it generates exactly four values (in this case, each value is a bracketed term).
It loops four times, so it must generate four values. What's the star supposed to do? Turn four loops into twelve? -- Steve
On 16 October 2016 at 14:44, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
"Language design by whoever shouts the loudest and the longest" is a terrible idea, and always has been. It's why "Has Guido already muted the thread?" is a useful metric for whether or not people are wasting their time in an unproductive argument (I don't know if he's muted this particular thread by now, but I'd be genuinely surprised if he hasn't) Remember that what we're arguing about is that existing instances of: [x for subiterable in iterable for x in subiterable] or: list(itertools.chain.from_iterable(iterable)) would be easier to read and maintain if they were instead written as: [*subiter for subiter in iterable] That's the bar people have to reach - if we're going to add a 3rd spelling for something that already has two spellings, then a compelling argument needs to be presented that the new spelling is *always* preferable to the existing ones, *not* merely "some people already agree that this 3rd spelling should mean the same thing as the existing spellings". The only proposal in this thread that has come close to reaching that bar is David Mertz's proposal to reify single level flattening as a flatten() builtin: [x for x in flatten(iterable)] or, equivalently: list(flatten(iterable)) Then the only thing that folks need to learn is that Python's builtin "flatten()" is a *non-recursive* operation that consistently flattens one layer of iterables with no special casing (not even of strings or bytes-like objects).
This is why I brought up mathematical set builder notation early in the thread, and requested that people present prior art for this proposal from that domain. It's the inspiration for comprehensions, so if a proposal to change comprehensions: - can't be readily explained in terms of their syntactic sugar for Python statements - can't be readily explained in terms of mathematical set builder notation then it's on incredibly shaky ground.
No need to guess, PEP 448 says why they're prohibited: https://www.python.org/dev/peps/pep-0448/#variations "This was met with a mix of strong concerns about readability and mild support. " Repeatedly saying "All of you people who find it unreadable are wrong, it's perfectly readable to *me*" does nothing except exacerbate the readability concerns, as folks who find it intuitive will use it over the more explicit existing alternatives, creating exactly the readability and maintainability problem we're worried about. Cryptic syntactic abbreviations are sometimes worthwhile when they're summarising something that can't otherwise be expressed easily in the form of an expression, but that isn't the case here - the existing alternatives are already expressions, and one of them already doesn't require any imports. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Oct 17, 2016, at 00:06, Nick Coghlan wrote:
Nothing is *always* preferable. That's an impossible bar for any feature that is already in python to have reached. Case in point - neither of the two spellings that you just gave is always preferable to the other.
It's the same bar that [a, b, *c, d] had to reach over list(itertools.chain((a, b), c, (d,))). You've also artificially constructed the examples to make the proposal look worse - there is only one layer of the comprehension so adding a second one doesn't look so bad. Meanwhile with the itertools.chain, the fact that it's just "*subiter" rather than [*some_function(item) for item in iterable] allows your chain example to be artificially short, it'd have to be list(itertools.chain.from_iterable(some_function(item) for item in iterable))
Once again, this alleged simplicity relies on the chosen example "x for x" rather than "f(x) for x" - this one doesn't even put the use of flatten in the right place to be generalized to the more complex cases. You'd need list(flatten(f(x) for x in iterable)) And this is where the "flatten" proposal fails - there's no way to use it alongside the list comprehension construct - either you can use a list comprehension, or you have to use the list constructor with a generator expression and flatten.
Repeatedly saying "All of you people who find it unreadable are wrong, it's perfectly readable to *me*"
Honestly, it goes beyond just being "wrong". The repeated refusal to even acknowledge any equivalence between [...x... for x in [a, b, c]] and [...a..., ...b..., ...c...] truly makes it difficult for me to accept some people's _sincerity_. The only other interpretation I see as possible is if they _also_ think [a, *b, c] is unreadable (something hinted at with the complaint that this is "hard to teach" because "in the current syntax, an expression is required in that position." something that was obviously also true of all the other places that unpacking generalization was added]) and are fighting a battle they already lost.
On Mon, Oct 17, 2016 at 12:11:46PM -0400, Random832 wrote:
While we're talking about people being insincere, how about if you take a look at your own comments? This "repeated refusal" that you accuse us (opponents of this proposal) of is more of a rhetorical fiction than an actual reality. Paul, David and I have all acknowledged the point you are trying to make. I won't speak for Paul or David, but speaking for myself, it isn't that I don't understand the point you're trying to make, but that I do not understand why you think that point is meaningful or desirable. I have acknowledged that starring the expression in a list comprehension makes sense if you think of the comprehension as a fully unrolled list display: [*expr, *expr *expr, *expr, ...] What I don't believe is: (1) that the majority of Python programmers (or even a large minority) regularly and consistently think of comprehensions as syntactic sugar for a completely unrolled list display; rather, I expect that they usually think of them as sugar for a for-loop; (2) that we should encourage people to think of comprehensions as sugar for a completely unrolled list display rather than a for-loop; (3) that we should accept syntax which makes no sense in the context of a for-loop-with-append (i.e. the thing which comprehensions are sugar for). But if we *do* accept this syntax, then I believe that we should drop the pretense that it is a natural extension of sequence unpacking in the context of a for-loop-with-append (i.e. list comprehensions) and accept that it will be seen by people as a magical "flatten" operator. And, in my opinion, rightly so: the semantic distance between *expr in a list comp and the level of explanation where it makes sense is so great that thinking of it as just special syntax for flattening is the simplest way of looking at it. So, yet again for emphasis: I see what you mean about unrolling the list comprehension into a list display. But I believe that's not a helpful way to think about list comprehensions. The way we should be thinking about them is as for-loops with append, and in *that* context, sequence unpacking doesn't make sense. In a list comprehension, we expect the invariant that the number of items produced will equal the number of loops performed. (Less if there are any "if" clauses.) There is one virtual append per loop. You cannot get the behaviour you want without breaking that invariant: either the append has to be replaced by extend, or you have so insert an extra loop into your mental picture of comprehensions. Yet again, for emphasis: I understand that you don't believe that invariant is important, or at least you are willing to change it. But drop the pretense that this is an obvious extension to the well- established behaviour of list comprehensions and sequence unpacking. If you think you can convince people (particularly Guido) that this flattening behaviour is important enough to give up the invariant "one append per loop", then by all means try. For all I know, Guido might agree with you and love this idea! But while you're accusing us of refusing to acknowledge the point you make about unrolling the loop to a list display (what I maintain is an unhelpful and non-obvious way of thinking about this), you in turn seem to be refusing to acknowledge the points we have made. This isn't a small change: it requires not insignificant changes to people's understanding of what list comprehension syntax means and does. -- Steve
On Mon, Oct 17, 2016, at 13:32, Steven D'Aprano wrote:
Only if their understanding is limited to a sequence of tokens that it supposedly expands to [except for all the little differences like whether a variable actually exists] - like your argument that it should just convert to a tuple because "yield x, y" happens to yield a tuple - rather than actual operations with real semantic meaning.
On 18 October 2016 at 03:49, Random832 <random832@fastmail.com> wrote:
Hi, I contributed the current list comprehension implementation (when refactoring it for Python 3 to avoid leaking the iteration variable, as requested in PEP 3100 [1]), and "comprehensions are syntactic sugar for a series of nested for and if statements" is precisely my understanding of how they work, and what they mean. It is also how they are frequently explained to new Python users. Directly insulting me and many of the educators who do so much to bring new users to Python by calling our understanding of a construct I implemented (and that you apparently love using) limited, is *not* doing your cause any favours, and is incredibly inappropriate behaviour for this list. Regards, Nick. [1] https://www.python.org/dev/peps/pep-3100/#core-language -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Oct 18, 2016, at 02:10, Nick Coghlan wrote:
But it's simply not true. It has never been true and it will never be true. Something is not "syntactic sugar" if it doesn't compile to the exact same sequence of operations as the thing it is supposedly syntactic sugar for. It's a useful teaching tool (though you've eventually got to teach the differences), but claiming that it's "syntactic sugar" - and "non-negotiably" so, at that - implies that it is a literal transformation. I said "limited" in reference to the specific claim - which was not yours - that since "yield a, b" yields a tuple, "yield *x" [and therefore (*x for...)] ought to also yield a tuple, and I stand by it. It's the same kind of simplistic understanding that should lead one to believe that not only the loop variable but also the "result" temporary ought to exist after the comprehension is executed. I was being entirely serious in saying that this is like objecting to normal unpacking on the grounds that an ordinary list display should be considered syntactic sugar for an unrolled sequence of append calls. In both cases, the equivalence is not exact, and there should be room to at least discuss things that would merely require an additional rule to be added (or changed, technically making it "result += [...]" would cover both cases) to the transformation - a transformation which already results in three different statements depending on whether it is a list comprehension, a set comprehension, or a generator expression (and a fourth if you count dict comprehensions, though that's a different syntax too) - rather than simply declaring them "not negotiable". Declaring this "not negotiable" was an incredibly hostile dismissal of everyone else's position. Especially when what's being proposed wouldn't invalidate the concept, it would just change the exact details of what the transformation is. Which is more than can be said for not leaking the variable.
On Fri, Oct 21, 2016 at 6:09 AM, Random832 <random832@fastmail.com> wrote:
But it is. There are two caveats to the transformation: firstly, it's done in a nested function (as of Py3), and secondly, the core operations are done with direct opcodes rather than looking up the ".append" method; but other than that, yes, it's exactly the same. Here's the disassembly (in 2.7, to avoid the indirection of the nested function):
3 6 SETUP_LOOP 33 (to 42) 9 LOAD_FAST 0 (x) 12 GET_ITER >> 13 FOR_ITER 25 (to 41) 16 STORE_FAST 2 (ch) 4 19 LOAD_FAST 1 (ret) 22 LOAD_ATTR 0 (append) 25 LOAD_GLOBAL 1 (ord) 28 LOAD_FAST 2 (ch) 31 CALL_FUNCTION 1 34 CALL_FUNCTION 1 37 POP_TOP 38 JUMP_ABSOLUTE 13 >> 41 POP_BLOCK 5 >> 42 LOAD_FAST 1 (ret) 45 RETURN_VALUE
Okay, so what exactly is going on here? Looks to me like there's some optimization happening in the list comp, but you can see that the same code is being emitted. It's not *perfectly* identical, but that's mainly because CPython doesn't take advantage of the fact that 'ret' was initialized to a list - it still does the full "look up 'append', then call it" work. I'm not sure why SETUP_LOOP exists in the full version and not the comprehension, but I believe it's to do with the break and continue keywords, which can't happen inside a comprehension. So, again, it's optimizations that are possible in the comprehension, but otherwise, the code is identical. Maybe "syntactic sugar" is pushing it a bit, but there's no fundamental difference between the two. Imagine if an optimizing compiler could (a) notice that there's no use of break/continue, and (b) do some static type analysis to see that 'ret' is always a list (and not a subclass thereof), and optimize the multi-line version. At that point, the two forms would look almost, or maybe completely, identical. So I'd support the "syntactic sugar" label here. Why is this discussion still on python-ideas? Shouldn't it be on python-demanding-explanations-for-status-quo by now? ChrisA
On 21 October 2016 at 05:09, Random832 <random832@fastmail.com> wrote:
We don't need to guess about this, since we can consult the language reference and see how comprehension semantics are specified for language implementors: https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets... Firstly, container displays are broken out into two distinct kinds: For constructing a list, a set or a dictionary Python provides special syntax called “displays”, each of them in two flavors: - either the container contents are listed explicitly, or - they are computed via a set of looping and filtering instructions, called a comprehension. Secondly, the meaning of the clauses in comprehensions is spelled out a little further down: The comprehension consists of a single expression followed by at least one for clause and zero or more for or if clauses. In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. We can also go back and read the design PEPs that added these features to the language: * List comprehensions: https://www.python.org/dev/peps/pep-0202/ * Generator expressions: https://www.python.org/dev/peps/pep-0289/ PEP 202 defined the syntax in terms of its proposed behaviour rather than a syntactic, with the only reference to the nesting equivalence being this BDFL pronouncement: - The form [... for x... for y...] nests, with the last index varying fastest, just like nested for loops. PEP 289, by contrast, fully spells out the implied generator definition that was used to guide the implementation of generator expressions in the code generator: g = (tgtexp for var1 in exp1 if exp2 for var2 in exp3 if exp4) is equivalent to: def __gen(bound_exp): for var1 in bound_exp: if exp2: for var2 in exp3: if exp4: yield tgtexp g = __gen(iter(exp1)) del __gen When I implemented the comprehension index variable hiding for Python 3.0, the final version was the one where I blended those two definitions to end up with the situation where: data = [(var1, var2) for var1 in exp1 if exp2 for var2 in exp3 if exp4)] is now equivalent to: def __comprehension(bound_exp): __hidden_var = [] for var1 in bound_exp: if exp2: for var2 in exp3: if exp4: __hidden_var.append((var1, var2)) return __hidden_var data = __comprehension(iter(exp1)) del __comprehension While it's pretty dated now (I wrote it circa 2.5 as part of a draft book manuscript that was never published), if you'd like to learn more about this, you may want to check out the section on "Initialising standard containers" in http://svn.python.org/view/sandbox/trunk/userref/ODF/Chapter02_StatementsAnd... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2016-10-17 10:32, Steven D'Aprano wrote:
It seems to me that this difference is fundamental. The entire point of this type of generalization is to break that invariant and allow the number of elements in the result list to vary independently of the number of iterations in the comprehension. It seems that a lot of this thread is talking at cross purposes, because the specifics of the syntax don't matter if you insist on that invariant. For instance, there's been a lot of discussion about whether this use of * is or isn't parallel to argument unpacking or assignment unpacking, or whether it's "intuitive" to some people or all people. But none of that matters if you insist on this invariant. If you insist on this invariant, no syntax will be acceptable; what is at issue is the semantics of enlarging the resulting list by more than one element. Now, personally, I don't insist on that invariant. I would certainly like to be able to do more general things in a list comprehension, and many times I have been irritated by the fact that the one-item-per-loop invariant exists. I'm not sure whether I'm in favor of this particular syntax, but I'd like to be able to do the kind of things it allows. But doing them inherently requires breaking the invariant you describe. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
But the proposal has explicit syntax that point the reader to the fact that the invariant doesn't hold. Same as other unpacking occurences: [x, *y] The invariant does not hold. And that's explicit. Elazar בתאריך יום ב׳, 17 באוק' 2016, 21:16, מאת Brendan Barnwell < brenbarn@brenbarn.net>:
On Mon, Oct 17, 2016 at 11:16:03AM -0700, Brendan Barnwell wrote:
I hear you, because I too would like to introduce a variant comprehension that uses a while instead of if. So don't think I'm not sympathetic. But that's not an option, and given the official position on comprehensions, I don't think this should be either. Officially, list comprehensions are not a replacement for general for-loops. Python is not Perl, where we encourage people to write one-liners, nor is it Haskell, where everything is an expression. If you want to do "more general things", use a real for-loop. Comprehensions are targetted at a narrow but important and common set of use-cases.
That last point is incorrect. You already can do the kind of things this thread is about: [*t for t in iterable] # proposed syntax: flatten can be written as: [x for t in iterable for x in t] If you want to apply a function to each element of t: [func(x) for x in [*t for t in iterable]] # proposed syntax can be written today: [func(x) for t in iterable for x in t] If you want to apply a function or expression to t first: [*func(t) for t in iterable] # proposed syntax [*expression for t in iterable] # proposed syntax this too can be written today: [x for t in iterable for x in func(t)] [x for t in iterable for x in expression] You might have an opinion on whether it is better to have an explicit extra loop (more verbose, but less magical) or special syntax (more compact, but more magical), but I don't think this proposal adds anything that cannot be done today. -- Steve
On 2016-10-17 16:35, Steven D'Aprano wrote:
Right, but by "doing those kinds of things" I mean doing them more in a more conise way without an extra level of iteration. (You can "do multiplication" by adding repeatedly, but it's still nice to have multiplication as an operation.) -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Mon, Oct 17, 2016 at 10:32 AM, Steven D'Aprano <steve@pearwood.info> wrote:
So, yet again for emphasis: I see what you mean about unrolling the list
comprehension into a list display. But I believe that's not a helpful way to think about list comprehensions.
Moreover, this "magical flatten" operator will crash in bad ways that a regular flatten() will not. I.e. this is fine (if strange):
It's hard to see how that won't blow up under the new syntax (i.e. generally for all infinite sequences). Try running this, for example:
a, *b = count()
Syntactically valid... but doesn't terminate. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 17 October 2016 at 18:32, Steven D'Aprano <steve@pearwood.info> wrote:
For my part: 1. I've acknowledged that equivalence. As well as the fact that the proposal (specifically, as explained formally by Greg) is understandable and a viable possible extension. 2. I don't find the "interpolation" equivalence a *good* way of interpreting list comprehensions, any more than I think that loops should be explained by demonstrating how to unroll them. 3. I've even explicitly revised my position on the proposal from -1 to -0 (although I'm tending back towards -1, if I'm honest...). 4. Whether you choose to believe me or not, I've sincerely tried to understand the proposal, but I pretty much had to insist on a formal definition of syntax and semantics before I got an explanation that I could follow. However: 1. I'm tired of hearing that the syntax is "obvious". This whole thread proves otherwise, and I've yet to hear anyone from the "obvious" side of the debate acknowledge that. 2. Can someone summarise the *other* arguments for the proposal? I'm genuinely struggling to recall what they are (assuming they exist). It feels like I'm hearing nothing more than "it's obvious what this does, it's obvious that it's needed and the people saying it isn't are wrong". That may well not be the truth, but *it's the impression I'm getting*. I've tried to take a step back and summarise my side of the debate a couple of times now. I don't recall seeing anyone doing the same from the other side (Greg's summarised the proposal, but I don't recall anyone doing the same with the justification for it). 3. The fact is that the proposed behaviour was *specifically* blocked, *precisely* because of strong concerns that it would cause readability issues and only had "mild" support. I'm not hearing any reason to change that decision (sure, there are a few people here offering something stronger than "mild" support, but it's only a few voices, and they are not addressing the readability concerns at all). There was no suggestion in the PEP that this decision was expected to be revisited later. Maybe there was an *intention* to do so, but the PEP didn't state it. I'd suggest that this fact alone implies that the people proposing this change need to write a new PEP for it, but honestly I don't think the way the current discussion has gone suggests that there's any chance of putting together a persuasive PEP, much less a consensus decision. And finally, no-one has even *tried* to explain why we need a third way of expressing this construction. Nick made this point, and basically got told that his condition was too extreme. He essentially got accused of constructing an impossible test. And yet it's an entirely fair test, and one that's applied regularly to proposals - and many *do* pass the test. It's worth noting here that we have had no real-world use cases, so the common approach of demonstrating real code, and showing how the proposal improves it, is not available. Also, there's no evidence that this is a common need, and so it's not clear to what extent any sort of special language support is warranted. We don't (as far as I know, and no-one's provided evidence otherwise) see people routinely writing workarounds for this construct. We don't hear of trainers saying that pupils routinely try to do this, and are surprised when it doesn't work (I'm specifically talking about students *deducing* this behaviour, not being asked if they think it's reasonable once explained). These are all arguments that have been used in the past to justify new syntax (and so reach Nick's "bar"). And we've had a special-case function (flatten) proposed to cover the most common cases (taking the approach of the 80-20 rule) - but the only response to that proposal has been "but it doesn't cover <artificial example>". If it didn't cover a demonstrably common real-world problem, that would be a different matter - but anyone can construct cases that aren't covered by *any* given proposal. That doesn't prove anything. I don't see any signs of progress here. And I'm pretty much at the point where I'm losing interest in having the same points repeated at me over and over, as if repetition and volume will persuade me. Sorry. Paul
On Mon, Oct 17, 2016, at 16:12, Paul Moore wrote:
As the one who made that accusation, my objection was specifically to the word "always" - which was emphasized - and which is something that I don't believe is actually a component of the test that is normally applied. His words, specifically, were "a compelling argument needs to be presented that the new spelling is *always* preferable to the existing ones" List comprehensions themselves aren't even always preferable to loops.
On 17 October 2016 at 21:30, Random832 <random832@fastmail.com> wrote:
Sigh. And no-one else in this debate has ever used exaggerated language. I have no idea if Nick would reject an argument that had any exceptions at all, but I don't think it's unreasonable to ask that people at least *try* to formulate an argument that demonstrates that the two existing ways we have are inferior to the proposal. Stating that you're not even willing to try is hardly productive. Paul
On Mon, Oct 17, 2016 at 11:13 PM Paul Moore <p.f.moore@gmail.com> wrote: ...
2. Can someone summarise the *other* arguments for the proposal? I'm genuinely struggling to recall what they are (assuming they exist).
My own argument was uniformity: allowing starred expression in other places, and I claim that the None-aware operator "come out" naturally from this uniformity. I understand that uniformity is not held high as far as decision making goes, and I was also kindly asked not to divert this thread, so I did not repeat it, but the argument is there. Elazar
On 17.10.2016 22:12, Paul Moore wrote: that's okay.
If it doesn't meet your standards of real-world code, okay. I meets mine. there were trainers who routinely reported this, that would be a strong argument for it. However, the absence of this signal, is not an argument against it IMHO.
Same here. The discussion is inconclusive. I think it's best to drop it for the time being. Best, Sven
On 17 October 2016 at 21:33, Sven R. Kunze <srkunze@mail.de> wrote:
Thank you. You're correct that was mentioned. I infer from the responses that it isn't sufficient, but I should have noted it explicitly. Elazar also mentioned consistency, which I had also forgotten. He noted in his comment (and I agree) that consistency isn't a compelling argument in itself. I'd generalise that point and say that theoretical arguments are typically considered secondary to real-world requirements.
Apologies. I had completely missed that example. Personally, I'd be inclined to argue that you shouldn't try so hard to build the list you want to return in a single statement. You can build the return value using a loop. Or maybe even write a function designed to filter Postgres result sets, which might even be reusable in other parts of your program. It's not my place to tell you how to redesign your code, or to insist that you have to use a particular style, but if I were writing that code, I wouldn't look for an unpacking syntax.
If this proposal had been described as "a syntax to replace chain.from_iterable", then it might have been received differently. I doubt it would have succeeded even so, but people would have understood the use case better. For my part, I find the name chain.from_iterable to be non-obvious. But if I needed to use it a lot (I don't!) I'd be more likely to simply come up with a better name, and rename it. Naming is *hard*, but it's worthwhile. One problem (IMO) of the "propose some syntax" approach is that it avoids the issue of thinking up a good name, by throwing symbols at the problem. (And you can't google for a string of symbols...)
No-one is looking for arguments *against* the proposal. Like it or not "status quo wins" is the reality. People need to look for arguments in favour of the proposal.
Thanks for the reasoned response. Paul
On Mon, Oct 17, 2016 at 10:33:32PM +0200, Sven R. Kunze wrote:
Sorry? You know, I am all for real-world code and I also delivered: https://mail.python.org/pipermail/python-ideas/2016-October/043030.html
Your example shows the proposed: [*(language, text) for language, text in fulltext_tuples if language == 'english'] which can be written as: [x for language, text in fulltext_tuples for x in (language, text) if language == 'english'] which is only ten characters longer. To me, though, there's simply no nice way of writing this: the repetition of "language, text" reads poorly regardless of whether there is a star or no star. If I were doing this more than once, I'd be strongly inclined to invest in a simple helper function to make this more readable: def filter_and_flatten(language, fulltext): for lang, text in fulltext: if lang == language: yield lang yield text filter_and_flatten('english', fulltext_tuples) In some ways, list comprehensions are a trap: their convenience and ease of use for the easy cases lure us into using them when we ought to be using a generator. But that's just my opinion. -- Steve
Steven D'Aprano wrote:
I think the ugliness of this particular example has roots in the fact that a tuple rather than an object with named fields is being used, which is going to make *any* piece of code that touches it a bit awkward. If it were a namedtuple, for example, you could write [*t for t in fulltext_tuples if t.language == 'english'] or [x for t in fulltext_tuples if t.language == 'english' for x in t] The latter is a bit unsatisfying, because we are having to make up an arbitrary name 'x' to stand for an element of t. Even though the two elements of t have quite different roles, we can't use names that reflect those roles. Because of that, to my eyes the version with * makes it easier to see what is going on. -- Greg
On 18.10.2016 08:23, Greg Ewing wrote:
It's an intriguing idea to use namedtuples but in this case one should not over-engineer. What I dislike most are the names of "fulltext_tuple", "x", "t". If I were to use it, I think my coworkers would tar and feather me. ;) This is one of the cases where it makes absolutely no sense to invent artificial names for the sake of naming. I can name a lot of (internal) examples where we tried really hard at inventing named concepts which make absolutely no sense half a year later even to those who invented them. Repeatedly, in the same part of the code. Each newly named concept introduces another indirection. Thus, we always need to find a middle ground between naming and using language features, so I (personally) would be grateful for this particular feature. :)
Because of that, to my eyes the version with * makes it easier to see what is going on.
That's a very nice phrase: "makes it easier to see what is going on". I need to remember that. Cheers, Sven
Steven D'Aprano wrote:
You don't have to believe that, because thinking about it as a for-loop works equally well. Without the star, it means "insert each of these things into a list". With the star, it means "unpack each of these things into a list".
In a list comprehension, we expect the invariant that the number of items produced will equal the number of loops performed.
There's a corresponding invariant for list displays: the number of items produced is equal to the number of expressions in the display. But that doesn't hold when the display includes unpacking, for obvious reasons. For the same reasons, we shouldn't expect it to hold for comprehensions with unpacking. -- Greg
On Mon, Oct 17, 2016 at 9:11 AM, Random832 <random832@fastmail.com> wrote:
What you're saying is EXACTLY 180 deg reversed from the truth. It's *precisely* because it doesn't need the extra complication that `flatten()` is more flexible and powerful. I have no idea what your example is meant to do, but the actual correspondence is: [f(x) for x in flatten(it)] Under my proposed "more flexible recursion levels" idea, it could even be: [f(x) for x in flatten(it, levels=3)] There would simply be NO WAY to get that out of the * comprehension syntax at all. But a decent flatten() function gets all the flexibility.
I am absolutely sincere in disliking and finding hard-to-teach this novel use of * in comprehensions. Yours, David... P.S. It's very artificial to assume user are unable to use 'from itertools import chain' to try to make chain() seem more cumbersome than it is. Likewise, I would like flatten() in itertools, but I assume the usual pattern would be importing the function itself. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 17.10.2016 20:38, David Mertz wrote:
I see what you are trying to do here and I appreciate it. Just one thought from my practical experience: I haven't had a single usage for levels > 1. levels==1 is basically * which I have at least one example for. Maybe, that relates to the fact that we asked our devs to use names (as in attributes or dicts) instead of deeply nested list/tuple structures. Do you think it would make sense to start a new thread just for the sake of readability?
You are consistent at least. You don't teach * in list displays, no matter if regular lists or comprehensions. +1
I am sorry but it is cumbersome. Regards, Sven
On 17 October 2016 at 20:35, Sven R. Kunze <srkunze@mail.de> wrote:
Imports are a fundamental part of Python. How are they "cumbersome"? Is it cumbersome to have to import sys to get access to argv? To import re to use regular expressions? To import subprocess to run an external program? Importing the features you use (and having an extensive standard library of tools you might want, but which don't warrant being built into the language) is, to me, a basic feature of Python. Certainly having to add an import statement is extra typing. But terseness was *never* a feature of Python. In many ways, a resistance to overly terse (I could say "Perl-like") constructs is one of the defining features of the language - and certainly, it's one that drew me to Python, and one that I value. Paul
On 17.10.2016 22:26, Paul Moore wrote:
The statement about "cumbersomeness" was specific to this whole issue. Of course, importing feature-rich pieces from the stdlib is really cool. It was more the missed ability to do the same with list comprehensions of what is possible with list displays today. List displays feature * without importing anything fancy from the stdlib. Nevermind, it seems we need to wait longer for this issue to come up again and maybe again to solve it eventually. Best, Sven
On 17 October 2016 at 21:43, Sven R. Kunze <srkunze@mail.de> wrote:
In your other post you specifically mentioned itertools.chain.from_iterable. I'd have to agree with you that this specific name feels clumsy to me as well. But I'd argue for finding a better name, not replacing the function with syntax :-) Cheers, Paul
On Mon, Oct 17, 2016, at 14:38, David Mertz wrote:
No, it's not. For a more concrete example: [*range(x) for x in range(4)] [*(),*(0,),*(0,1),*(0,1,2)] [0, 0, 1, 0, 1, 2] There is simply no way to get there by using flatten(range(4)). The only way flatten *without* a generator expression can serve the same use cases as this proposal is for comprehensions of the *exact* form [*x for x in y]. For all other cases you'd need list(flatten(...generator expression without star...)).
As Paul or someone pointed out, that's a fairly odd thing to do. It's the first time that use case has been mentioned in this thread. It's true you've managed to construct something that isn't done by flatten(). I would have had to think a while to see what you meant by the original if you haven't provided the intermediate interpretations. Of course, it's *really simple* to spell that in a natural way with existing syntax that isn't confusing like yours: [x for end in range(4) for x in range(end)] There is no possible way to construct something that would use the proposed syntax that can't be expressed more naturally with a nested loop... because it's just confusing syntax sugar for exactly that. Your example looks like some sort of interview quiz question to see if someone knows obscure and unusual syntax.
On Mon, Oct 17, 2016, at 22:17, David Mertz wrote:
[*range(x) for x in range(4)]
As Paul or someone pointed out, that's a fairly odd thing to do.
I agree with the specific example of it being an odd thing to do with range, it was just an attempt to illustrate with a concrete example.
It's the first time that use case has been mentioned in this thread.
I think that in general the "body involves a subexpression returning an iterable other than the bare loop variable" has been covered before, though it might not have been clear at all times that that was what was being discussed. Frankly, I think it's rare that something of the form "x for x ..." is best written with a comprehension in the first place, and the same would be true for "*x for x..." so I didn't like that some of the translations being discussed only work well for that case.
I feel like I should be honest about something else - I'm always a little bit confused by the ordering for comprehensions involving multiple clauses. For me, it's the fact that: [[a for a in b] for b in ['uvw', 'xyz']] == [['u', 'v', 'w'], ['x', 'y', 'z']] which makes me want to write: [a for a in b for b in ['uvw', 'xyz']] but that's an error, and it actually needs to be [a for b in ['uvw', 'xyz'] for a in b] == ['u', 'v', 'w', 'x', 'y', 'z'] So when this talk of readability issues comes up and the recommended alternative is something that I don't really find readable, it's frustrating. To me this proposal is something that would allow for more things to be expressed without resorting to multi-loop comprehensions.
On Mon, Oct 17, 2016 at 7:50 PM, Random832 <random832@fastmail.com> wrote:
It's also easy to construct examples where the hypothetical * syntax can't handle a requirement. E.g. flatten() with levels>1 (yes, of course you can find some way to nest more loops or more comprehensions-within-comprehensions to make it work in some way that still uses the * by force). I feel like I should be honest about something else - I'm always a
little bit confused by the ordering for comprehensions involving multiple clauses.
Me too! I get the order of nested loops in comprehensions wrong about 25% of the time. Then it's a NameError, and I fix it. This is a lot of why I like a utility function like `flatten()` that is pretty much self-documenting. Perhaps a couple other itertools helpers would be nice. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 18 October 2016 at 13:32, David Mertz <mertz@gnosis.cx> wrote:
This is also one of the main reasons that named generator expression pipelines can sometimes be easier to read than nested comprehensions: incrementally_increasing_ranges = (range(end) for end in itertools.count()) flatten = itertools.chain.from_iterable incrementally_increasing_cycles = flatten(incrementally_increasing_ranges()) Forcing ourselves to come up with a name for the series of values produced by the outer iteration then makes that name available as documentation of our intent for future readers of the code. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 18 October 2016 at 07:31, Nick Coghlan <ncoghlan@gmail.com> wrote:
This is a key point, that is frequently missed when people propose new "shorter" syntax, or constructs that "reduce indentation levels". Make no mistake, coming up with good names is *hard* (and I have enormous respect for the (often unrecognised) people who come up with intuitive APIs and names for functions). So it's very easy to be tempted by "concise" constructs that offer the option of not naming an operation. But as someone who spends 99% of his time doing maintenance programming, be sure that the people who support your code will thank you for spending time naming your abstractions (and avoiding using constructs like the one proposed here). Paul
Random832 wrote:
You're not alone! Lately I've been becoming convinced that this is the way we should have done it right back at the beginning. But it's far too late to change it now, sadly. Our only hope would be to introduce a new syntax, maybe [a for a in b; for b in ['uvw', 'xyz']] Inserting the semicolons mightn't be such a bad idea, because when doing the reversal it would be necessary to keep any 'if' clauses together with their preceding 'for' clause. So if we wrote [a for a in b if cond(a); for b in things] we could say the rule is that you split at the semicolons and then reverse the clauses. -- Greg
On 18/10/2016 07:40, Greg Ewing wrote:
If the syntax were [ x for x in alist for alist in list-of-lists ] there is a smooth (or rather, monotonic) gradation from the smallest object (x) to the next biggest object (alist) to the biggest object (list-of-lists), which IMHO is easier to follow. Each object is conceptually zero or one steps from its neighbour. But with the actual syntax [ x for alist in list-of-lists for x in alist ] there is a conceptual hiatus after "x" ("what on earth are alist and list-of-lists, and what have they got do to with x?"). This would be even more marked with more than 2 loops: we jump from the lowest level object to the two highest level objects, and it all seems like a disorienting non-sequitur until the very last loop "joins the dots". You have to identify the "x" at the beginning with the "x" near (but not at!) the end. Instead of (ideally, if not always in practice) reading the expression from left-to-right in one go, your eyes are forced to jump around in order for your brain to assimilate it. A possible alternative syntax might be to follow more closely the for-loop syntax, e.g. [ for alist in list-of-lists: for x in alist: x ] Here the "conceptual jump" between each object and the next is either 1 or 2, which for me makes this a "second best" option. But at least the "conceptual jump" is bounded (regardless of the number of loops), and this syntax has the advantage of familiarity.
Best wishes Rob Cliffe
For me the current behaviour does not seem unreasonable as it resembles the order in which you write out loops outside a comprehension except that the expression for generated values is provided first. On 21 October 2016 at 05:03, Sven R. Kunze <srkunze@mail.de> wrote:
Alexander Heger wrote:
For me the current behaviour does not seem unreasonable as it resembles the order in which you write out loops outside a comprehension
That's true, but the main reason for having comprehensions syntax in the first place is so that it can be read declaratively -- as a description of the list you want, rather than a step-by-step sequence of instructions for building it up. If you have to stop and mentally transform it into nested for-statements, that very purpose is undermined. -- Greg
Well, an argument that was often brought up on this forum is that Python should do things consistently, and not in one way in one place and in another way in another place, for the same thing. Here it is about the order of loop execution. The current behaviour in comprehension is that is ts being done the same way as in nested for loops. Which is easy enough to remember. Same way, everywhere. -Alexander
On 22.10.2016 09:50, Alexander Heger wrote:
Like * in list displays? ;-)
It still would. Just read it from right to left. The order stays the same.
Which is easy enough to remember. Same way, everywhere.
I am sorry but many disagree with you on this thread. I still don't understand why the order needs to be one-way anyway. Comprehensions are a declarative construct, so it should be possible to mix those "for .. in .."s up in an arbitrary order. Cheers, Sven
Thinking about it, though, I ended up exactly where you are now, except that I then thought about where an item would be known, and it seemed to me, yes, an item would be more likely known *after* looping over it rather than before: [bi for bi in before for before in iterable] # why should "before" exist before it is looped over? correctly: [bi for before in iterable for bi in before] it doesn't, it should be declared to the right hand side and only the result is kept over at the left hand edge. On a same note, with if expressions the picture might look different: [bi for bi in before for before in iterable if before[0] < 3] # ... is bi filtered now or not? correctly: [bi for before in iterable if before[0] < 3 for bi in before] it is filtered very clearly this way. cheers! mar77i
On 17 October 2016 at 21:22, Random832 <random832@fastmail.com> wrote:
the equivalent flatten for that is: flatten(range(x) for x in range(4)) ; flatten has no magic so will not replace a piece of code with two range calls (like your example) for code with one. I see some mention that flatten does not cover all cases; but correct me if I'm wrong with this statement: Any case of [*<exp1> for <vars> in <exp2>] could be replaced with flatten(<exp1> for <vars> in <exp2>). Where flatten is defined as def flatten(it): return [x for for subit in it for x in subit] (there is a slight difference where I'm making flatten iterable instead of a list) What perhaps was confusing is that in the case where <exp1> and <vars> are the same, you can also write flatten(<exp2>). So, for me, this feature is something that could be covered with a (new) function with no new syntax required. All you have to learn is that instead of [*...] you use flatten(...) Am I wrong? I keep reading people on both sides saying "flatten is not enough in all cases", and I can find a counterexample (even for 1 level flatten which is what I used here) PS: or alternatively, flatten = lambda it: list(itertools.chain(it)) # :) PPS: or if you prefer to work with iterators, flatten = itertools.chain -- Daniel F. Moisset - UK Country Manager www.machinalis.com Skype: @dmoisset
On Tue, Oct 18, 2016, at 04:01, Daniel Moisset wrote:
That is correct - though flatten as previously discussed did not return a list, so list(flatten(...)) was required, though I suppose you could use [*flatten(...)] - my point was that [especially with the list constructor] this is significantly more verbose than the proposed syntax.
On 18.10.2016 10:01, Daniel Moisset wrote:
The main motivation is not "hey we need yet another way for xyz" but "could we remove that inconsistency?". You wrote "[*...]"; which already works. It just that it does not work for all kinds of "..." . I hope that makes the motivation clearer. I for one don't need yet another function or function cascade to make things work. "list(itertools.chain.from_iterable(...))" just works fine for all kinds of "...". Cheers, Sven
On Sun, Oct 16, 2016 at 9:06 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I don't think I'd actually propose a builtin for this. For me, including the recipe that is in the itertools documentation into a function in the module would be plenty. Writing "from itertools import flatten" is not hard.
Actually, I think that if we added `flatten()` to itertools, I'd like a more complex implementation that had a signature like: def flatten(it, levels=1): # Not sure the best implementation for clever use of other itertools ... I'm not quite sure how one would specify "flatten all the way down." In practice, `sys.maxsize` is surely large enough; but semantically it feels weird to use an arbitrary large number to mean "infinitely if necessary." -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Sat, Oct 15, 2016 at 9:42 PM Steven D'Aprano <steve@pearwood.info> wrote:
Whoa, hang on a second there. It is intentionally prohibited because Joshua Landau (who helped a lot with writing and implementing the PEP) and I felt like there was going to be a long debate and we wanted to get PEP 448 checked in. If it "didn't make sense" as you say, then we would have said so in the PEP. Instead, Josh wrote: This was met with a mix of strong concerns about readability and mild support. In order not to disadvantage the less controversial aspects of the PEP, this was not accepted with the rest of the proposal. I don't remember who it was who had those strong concerns (maybe you?) But that's why we didn't include it. Best, Neil
On Sun, Oct 16, 2016 at 05:14:54AM +0000, Neil Girdhar wrote: [Steven (me), refering to Greg]
Okay, interesting, and thanks for the correction.
I don't remember who it was who had those strong concerns (maybe you?) But that's why we didn't include it.
I'm pretty sure it wasn't me. I don't recall being involved at all with any discussions about PEP 448, and a quick search has failed to come up with anything relevant. I think I sat that one out. -- Steve
On Sat, Oct 15, 2016 at 11:36:28PM +1300, Greg Ewing wrote:
Indeed. In situations where there isn't any context for the interpretation of *, it's not allowed.
You mean like in list comprehensions? Are you now supporting my argument that starring the list comprehension expression isn't meaningful? Not if star is defined as sequence unpacking in the usual way. If you want to invent a new meaning for * to make this work, to join all the other special case magic meanings for the * symbol, that's another story.
Oh look, just like now: py> iterable = [(1, 'a'), (2, 'b')] py> [(100, *t) for t in iterable] [(100, 1, 'a'), (100, 2, 'b')] Hands up anyone who expected to flatten the iterable and get [100, 1, 'a', 100, 2, 'b'] instead? Anyone? No? Take out the (100, ...) and just leave the *t, and why should it be different? It is my position that: (quote) there isn't any context for the interpretation of * for [*t for t in iterable]. Writing that is the list comp equivalent of writing x = *t. -- Steve
On Sat, Oct 15, 2016 at 12:48 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I don't know whether that should be provocating or beside the poinnt. It's probably both. You're putting two expectations on the same example: first, you make the reasonable expectation that results in [(100, 1, 'a'), (100, 2, 'b')], and then you ask whether anyone expected [100, 1, 'a', 100, 2, 'b'], but don't add or remove anything from the same example. Did you forget to put a second example using the new notation in there? Then you'd have to spell it out and start out with [*(100, *t) for t in iterable]. And then you can ask who expected [100, 1, 'a', 100, 2, 'b']. Which is what this thread is all about. cheers! mar77i
Steven D'Aprano wrote:
Are you now supporting my argument that starring the list comprehension expression isn't meaningful?
The context it's in (a form of list display) has a clear meaning for a comma-separated list of values, so there is a reasonable interpretation that it *could* be given.
The * there is in the context of constructing a tuple, not the list into which the tuple is placed. The difference is the same as the difference between these:
-- Greg
On Sun, Oct 16, 2016 at 12:48:36PM +1300, Greg Ewing wrote:
This thread is a huge, multi-day proof that people do not agree that this is a "reasonable" interpretation.
Right: the context of the star is meaningful. We all agree that *t in a list display [a, b, c, ...] is meaningful; same for tuples; same for function calls; same for sequence unpacking for assignment. What is not meaningful (except as a Perlish line-noise special case to be memorised) is *t as the list comprehension expression. I've never disputed that we could *assert* that *t in a list comp means "flatten". We could assert that it means anything we like. But it doesn't follow from the usual meaning of sequence unpacking anywhere else -- that's why it is currently a SyntaxError, and that's why people reacted with surprise at the OP who assumed that *t would magically flatten his iterable. Why would you assume that? It makes no sense to me -- that's not how sequence unpacking works in any other context, it isn't how list comprehensions work. Right from the beginning I called this "wishful thinking", and *nothing* since then has changed my mind. This proposal only makes even a little bit of sense if you imagine list comprehensions [*t for a in it1 for b in it2 for c in it3 ... for t in itN] completely unrolled into a list display: [*t, *t, *t, *t, ... ] but who does that? Why would you reason about your list comps like that? If you think about list comps as we're expected to think of them -- as list builders equivalent to a for-loop -- the use of *t there is invalid. Hence it is a SyntaxError. You want a second way to flatten your iterables? A cryptic, mysterious, Perlish line-noise way? Okay, fine, but don't pretend it is sequence unpacking -- in the context of a list comprehension, sequence unpacking doesn't make sense, it is invalid. Call it something else: the new "flatten" operator: [^t for t in iterable] for example, which magically adds an second invisible for-loop to your list comps: # expands to for t in iterable: for x in t: result.append(x) Because as your own email inadvertently reinforces, if sequence unpacking made sense in the context of a list comprehension, it would already be allowed rather than a SyntaxError: it is intentionally prohibited because it doesn't make sense in the context of list comps. -- Steve
Steven D'Aprano wrote:
This thread is a huge, multi-day proof that people do not agree that this is a "reasonable" interpretation.
So far I've seen one very vocal person who disgrees, and maybe one other who isn't sure.
Many people do, and it's a perfectly valid way to think about them. They're meant to admit a declarative reading; that's the reason they exist in the first place. The expansion in terms of for-loops and appends is just *one* way to describe the current semantics. It's not written on stone tablets brought down from a mountain. Any other way of thinking about it that gives the same result is equally valid.
magically adds an second invisible for-loop to your list comps:
You might as well say that the existing * in a list display magically inserts a for-loop into it. You can think of it that way if you want, but you don't have to.
it is intentionally prohibited because it doesn't make sense in the context of list comps.
I don't know why it's currently prohibited. You would have to ask whoever put that code in, otherwise you're just guessing about the motivation. -- Greg
On Sun, Oct 16, 2016 at 3:44 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
And what you're NOT seeing is a whole lot of people (myself included) who have mostly glazed over, unsure what is and isn't reasonable, and not clear enough on either side of the debate to weigh in. (Or not even clear what the two sides are.) ChrisA
On 16 October 2016 at 06:47, Chris Angelico <rosuav@gmail.com> wrote:
+1 There are lots of arguments of whether the new syntax is readable or not etc., but not so many arguments why we need this and what kind of problems it would solve. What I have learned from this megathread is that the syntax [*foo for foo in bar] is proposed as a replacement for a one-liner itertools.chain(*[foo for foo in bar]). I do not have any strong opinion on this, because I simply do not use such constructs frequently (if ever). -- Ivan
On Sun, Oct 16, 2016 at 03:02:55PM +0200, Ivan Levkivskyi wrote:
If people take away nothing else from this thread, it should be that flattening an iterable is as easy as: [x for t in iterable for x in t] which corresponds neatly to: for t in iterable: for x in t: result.append(x) -- Steve
On Oct 15, 2016 9:45 PM, "Greg Ewing" <greg.ewing@canterbury.ac.nz> wrote:
Steven D'Aprano wrote:
This thread is a huge, multi-day proof that people do not agree that
this is a "reasonable" interpretation.
So far I've seen one very vocal person who disgrees, and maybe one other who isn't sure.
In case it wasn't entirely clear, I strongly and vehemently opposed this unnecessary new syntax. It is confusing, bug prone, and would be difficult to teach. Or am I that very vocal person? I was thinking your meant Steven.
On 16.10.2016 07:08, David Mertz wrote:
As this discussion won't come to an end, I decided to consult my girlfriend. I started with (btw. she learned basic Python to solve some math quizzes): """ Let's install a list in another one.
Maybe, we want to remove the brackets.
['a', *meine_liste, 'b'] ['a', 2, 3, 4, 5, 'b']
Now, the problem of the discussion is the following:
[(i,i,i) for i in range(4)] [(0, 0, 0), (1, 1, 1), (2, 2, 2), (3, 3, 3)]
Let's remove these inner parentheses again.
Some guy wanted to remove that restriction. """ I said a teacher contributed to the discussion and he finds this too complicated and confusing and does not even teach * in list displays at all. Her reaction was hilarious: "Whom does he teach? Children?" Me: "What? No, everybody I think. Why?" She: "It's easy enough to remember what the star does." She also asked what would the alternative would look like. I wrote: """ the star easier." In the end, she also added: "Not everybody drives a car but they still exist." Cheers, Sven PS: off to the weekend. She's already complaint that I should spend less time inside my mailbox and more with her. ;)
On Sun, Oct 16, 2016 at 5:34 AM, Sven R. Kunze <srkunze@mail.de> wrote:
As I've said, the folks I teach are mostly working scientists with doctorates in scientific fields and years of programming experience in languages other than Python. For example, rocket scientists at NASA. Now I admit that I don't specifically know how quickly they would pick up on something I've never taught them. But I've written enough teaching materials and articles and books and I have a certain intuition. The way you explained the special case you built up is pretty good, but it's very tailored to making that specific case plausible, and is not general.
That is an absolutely terrible construct, obviously. I have to pause and think a while myself to understand what it does. It also answers a very different use case than the one that has been mostly discussed in this thread. A much better spelling is:
[i for i in range(4) for _ in range(3)] [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3]
For some related uses, itertools.repeat() is very useful. But the case that has been discussed far more often is like this:
listOfLists = [[1,2,3], [4,5,6,7], [8,9]] flatten(listOfLists)
The very long argument is that somehow that would be easier to spell as:
[*i for i in listOfLists]
It's just not easier. And there are contrary intuitions that will occur to many people based on the several other related-but-different uses of * for packing/unpacking in other contexts. Also, this simplest case might be teachable, but the more general "exactly where can I use that star in a comprehensions" will be far harder to explain plausibly. What's the pattern here?
Yes! I know those first few are actually doing something different than the proposed new syntax. But explaining that to my rocket scientists or your girlfriend in a consistent and accurate way would be a huge challenge. It would mostly come down to "don't do that, it's too confusing." -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
Her reaction was hilarious:
"Whom does he teach? Children?"
I sense mockery in your email, and it does not conform to the PSF code of conduct. Please read the CoC before posting in this mailing list. The link is available at the bottom of every python mailing list email. https://www.python.org/psf/codeofconduct/ I don't find teaching children is a laughing matter, neither is the idea of children learning to code. In Canada, we have initiatives like Girls Learning Code and Kids Learning Code. I mentored in a couple of those events and the students are girls aged 8-14. They surprised me with their abilities to learn. I would suggest looking for such mentoring opportunities in your area to gain appreciation with this regard. Thanks. (Sorry to derail everyone from the topic of list comprehension. Please continue!)
On 16/10/2016 16:41, Mariatta Wijaya wrote:
The RUE was allowed to insult the community for years and got away with it. I'm autistic, stepped across the line, and got hammered. Hypocrisy at its best. Even funnier, the BDFL has asked for my advice in recent weeks with respect to the bug tracker. I've replied, giving the bare minimum that I feel I can give within the circumstances. Yours most disgustingly. Mark Lawrence.
Actually, I agree with Marietta. I don't care whatsoever about mocking me, which was a certain element of it. I have thick skin and am confident in these conversations. The part that was probably over the line was mocking children who learn to program or those who teach them. That's a huge and great job. I know I would not have the skill to teach children effectively. Adults with technical expertise are much easier for me. That said, thank you Mark for your empirical research with a test subject. Best, David On Oct 16, 2016 9:39 AM, "Mark Lawrence via Python-ideas" < python-ideas@python.org> wrote:
On Sun, 16 Oct 2016 at 09:39 Mark Lawrence via Python-ideas < python-ideas@python.org> wrote:
What is the "RUE"?
I'm autistic, stepped across the line, and got hammered. Hypocrisy at its best.
While some of us know your background, Mark, not everyone on this list does as people join at different times, so please try to give the benefit of the doubt to people. Marietta obviously takes how children are reflected personally and was trying to point out that fact. I don't think she meant for the CoC reference to come off as threatening, just to back up why she was taking the time out to speak up that she was upset by what was said.
On 17/10/2016 19:29, Brett Cannon wrote:
Not what, who, the Resident Unicode Expert who spent two years spewing his insults at the entire Python community until the moderators finally woke up, did their job, and kicked him into touch.
This list is irrelevant. The PSF has to be consistent across all of its lists. Who the hell is Marietta, I don't recall a single post from her in 16 years of using Python? -- Hell I'm confused, why the hell do I bother???
On 10/21/2016 12:13 PM, Mark Lawrence via Python-ideas wrote:
This list is irrelevant. The PSF has to be consistent across all of its lists.
This list is not irrelevant, and yes *volunteers who moderate* should be consistent.
Who the hell is Marietta, I don't recall a single post from her in 16 years of using Python?
She is a fellow Pythonista, and you owe her an apology. -- ~Ethan~ P.S. Should you decide to bring my own stupidity a few months of ago of being insulting, you should also take note that I took responsibility for it and apologized myself.
On 10/21/16 3:13 PM, Mark Lawrence via Python-ideas wrote:
Mark, I know you viewed "Python is broken" as an insult to the maintainers, but others do not. He certainly never singled out any individual for abuse.
The fact that you haven't seen a post from Marietta before is irrelevant. She is new to this list, and has been traveling in different Python circles than you have. That doesn't make her less of a member of this community, and it doesn't diminish her point. Your vitriol towards her diminishes yours. --Ned.
On Sun, Oct 16, 2016 at 02:34:58PM +0200, Sven R. Kunze wrote:
Did you remember to tell your girlfriend that a critical property of the "??? for i in range(4)" construct is that it generates one value per loop? It loops four times, so it generates exactly four values (in this case, each value is a bracketed term).
It loops four times, so it must generate four values. What's the star supposed to do? Turn four loops into twelve? -- Steve
On 16 October 2016 at 14:44, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
"Language design by whoever shouts the loudest and the longest" is a terrible idea, and always has been. It's why "Has Guido already muted the thread?" is a useful metric for whether or not people are wasting their time in an unproductive argument (I don't know if he's muted this particular thread by now, but I'd be genuinely surprised if he hasn't) Remember that what we're arguing about is that existing instances of: [x for subiterable in iterable for x in subiterable] or: list(itertools.chain.from_iterable(iterable)) would be easier to read and maintain if they were instead written as: [*subiter for subiter in iterable] That's the bar people have to reach - if we're going to add a 3rd spelling for something that already has two spellings, then a compelling argument needs to be presented that the new spelling is *always* preferable to the existing ones, *not* merely "some people already agree that this 3rd spelling should mean the same thing as the existing spellings". The only proposal in this thread that has come close to reaching that bar is David Mertz's proposal to reify single level flattening as a flatten() builtin: [x for x in flatten(iterable)] or, equivalently: list(flatten(iterable)) Then the only thing that folks need to learn is that Python's builtin "flatten()" is a *non-recursive* operation that consistently flattens one layer of iterables with no special casing (not even of strings or bytes-like objects).
This is why I brought up mathematical set builder notation early in the thread, and requested that people present prior art for this proposal from that domain. It's the inspiration for comprehensions, so if a proposal to change comprehensions: - can't be readily explained in terms of their syntactic sugar for Python statements - can't be readily explained in terms of mathematical set builder notation then it's on incredibly shaky ground.
No need to guess, PEP 448 says why they're prohibited: https://www.python.org/dev/peps/pep-0448/#variations "This was met with a mix of strong concerns about readability and mild support. " Repeatedly saying "All of you people who find it unreadable are wrong, it's perfectly readable to *me*" does nothing except exacerbate the readability concerns, as folks who find it intuitive will use it over the more explicit existing alternatives, creating exactly the readability and maintainability problem we're worried about. Cryptic syntactic abbreviations are sometimes worthwhile when they're summarising something that can't otherwise be expressed easily in the form of an expression, but that isn't the case here - the existing alternatives are already expressions, and one of them already doesn't require any imports. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Oct 17, 2016, at 00:06, Nick Coghlan wrote:
Nothing is *always* preferable. That's an impossible bar for any feature that is already in python to have reached. Case in point - neither of the two spellings that you just gave is always preferable to the other.
It's the same bar that [a, b, *c, d] had to reach over list(itertools.chain((a, b), c, (d,))). You've also artificially constructed the examples to make the proposal look worse - there is only one layer of the comprehension so adding a second one doesn't look so bad. Meanwhile with the itertools.chain, the fact that it's just "*subiter" rather than [*some_function(item) for item in iterable] allows your chain example to be artificially short, it'd have to be list(itertools.chain.from_iterable(some_function(item) for item in iterable))
Once again, this alleged simplicity relies on the chosen example "x for x" rather than "f(x) for x" - this one doesn't even put the use of flatten in the right place to be generalized to the more complex cases. You'd need list(flatten(f(x) for x in iterable)) And this is where the "flatten" proposal fails - there's no way to use it alongside the list comprehension construct - either you can use a list comprehension, or you have to use the list constructor with a generator expression and flatten.
Repeatedly saying "All of you people who find it unreadable are wrong, it's perfectly readable to *me*"
Honestly, it goes beyond just being "wrong". The repeated refusal to even acknowledge any equivalence between [...x... for x in [a, b, c]] and [...a..., ...b..., ...c...] truly makes it difficult for me to accept some people's _sincerity_. The only other interpretation I see as possible is if they _also_ think [a, *b, c] is unreadable (something hinted at with the complaint that this is "hard to teach" because "in the current syntax, an expression is required in that position." something that was obviously also true of all the other places that unpacking generalization was added]) and are fighting a battle they already lost.
On Mon, Oct 17, 2016 at 12:11:46PM -0400, Random832 wrote:
While we're talking about people being insincere, how about if you take a look at your own comments? This "repeated refusal" that you accuse us (opponents of this proposal) of is more of a rhetorical fiction than an actual reality. Paul, David and I have all acknowledged the point you are trying to make. I won't speak for Paul or David, but speaking for myself, it isn't that I don't understand the point you're trying to make, but that I do not understand why you think that point is meaningful or desirable. I have acknowledged that starring the expression in a list comprehension makes sense if you think of the comprehension as a fully unrolled list display: [*expr, *expr *expr, *expr, ...] What I don't believe is: (1) that the majority of Python programmers (or even a large minority) regularly and consistently think of comprehensions as syntactic sugar for a completely unrolled list display; rather, I expect that they usually think of them as sugar for a for-loop; (2) that we should encourage people to think of comprehensions as sugar for a completely unrolled list display rather than a for-loop; (3) that we should accept syntax which makes no sense in the context of a for-loop-with-append (i.e. the thing which comprehensions are sugar for). But if we *do* accept this syntax, then I believe that we should drop the pretense that it is a natural extension of sequence unpacking in the context of a for-loop-with-append (i.e. list comprehensions) and accept that it will be seen by people as a magical "flatten" operator. And, in my opinion, rightly so: the semantic distance between *expr in a list comp and the level of explanation where it makes sense is so great that thinking of it as just special syntax for flattening is the simplest way of looking at it. So, yet again for emphasis: I see what you mean about unrolling the list comprehension into a list display. But I believe that's not a helpful way to think about list comprehensions. The way we should be thinking about them is as for-loops with append, and in *that* context, sequence unpacking doesn't make sense. In a list comprehension, we expect the invariant that the number of items produced will equal the number of loops performed. (Less if there are any "if" clauses.) There is one virtual append per loop. You cannot get the behaviour you want without breaking that invariant: either the append has to be replaced by extend, or you have so insert an extra loop into your mental picture of comprehensions. Yet again, for emphasis: I understand that you don't believe that invariant is important, or at least you are willing to change it. But drop the pretense that this is an obvious extension to the well- established behaviour of list comprehensions and sequence unpacking. If you think you can convince people (particularly Guido) that this flattening behaviour is important enough to give up the invariant "one append per loop", then by all means try. For all I know, Guido might agree with you and love this idea! But while you're accusing us of refusing to acknowledge the point you make about unrolling the loop to a list display (what I maintain is an unhelpful and non-obvious way of thinking about this), you in turn seem to be refusing to acknowledge the points we have made. This isn't a small change: it requires not insignificant changes to people's understanding of what list comprehension syntax means and does. -- Steve
On Mon, Oct 17, 2016, at 13:32, Steven D'Aprano wrote:
Only if their understanding is limited to a sequence of tokens that it supposedly expands to [except for all the little differences like whether a variable actually exists] - like your argument that it should just convert to a tuple because "yield x, y" happens to yield a tuple - rather than actual operations with real semantic meaning.
On 18 October 2016 at 03:49, Random832 <random832@fastmail.com> wrote:
Hi, I contributed the current list comprehension implementation (when refactoring it for Python 3 to avoid leaking the iteration variable, as requested in PEP 3100 [1]), and "comprehensions are syntactic sugar for a series of nested for and if statements" is precisely my understanding of how they work, and what they mean. It is also how they are frequently explained to new Python users. Directly insulting me and many of the educators who do so much to bring new users to Python by calling our understanding of a construct I implemented (and that you apparently love using) limited, is *not* doing your cause any favours, and is incredibly inappropriate behaviour for this list. Regards, Nick. [1] https://www.python.org/dev/peps/pep-3100/#core-language -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Oct 18, 2016, at 02:10, Nick Coghlan wrote:
But it's simply not true. It has never been true and it will never be true. Something is not "syntactic sugar" if it doesn't compile to the exact same sequence of operations as the thing it is supposedly syntactic sugar for. It's a useful teaching tool (though you've eventually got to teach the differences), but claiming that it's "syntactic sugar" - and "non-negotiably" so, at that - implies that it is a literal transformation. I said "limited" in reference to the specific claim - which was not yours - that since "yield a, b" yields a tuple, "yield *x" [and therefore (*x for...)] ought to also yield a tuple, and I stand by it. It's the same kind of simplistic understanding that should lead one to believe that not only the loop variable but also the "result" temporary ought to exist after the comprehension is executed. I was being entirely serious in saying that this is like objecting to normal unpacking on the grounds that an ordinary list display should be considered syntactic sugar for an unrolled sequence of append calls. In both cases, the equivalence is not exact, and there should be room to at least discuss things that would merely require an additional rule to be added (or changed, technically making it "result += [...]" would cover both cases) to the transformation - a transformation which already results in three different statements depending on whether it is a list comprehension, a set comprehension, or a generator expression (and a fourth if you count dict comprehensions, though that's a different syntax too) - rather than simply declaring them "not negotiable". Declaring this "not negotiable" was an incredibly hostile dismissal of everyone else's position. Especially when what's being proposed wouldn't invalidate the concept, it would just change the exact details of what the transformation is. Which is more than can be said for not leaking the variable.
On Fri, Oct 21, 2016 at 6:09 AM, Random832 <random832@fastmail.com> wrote:
But it is. There are two caveats to the transformation: firstly, it's done in a nested function (as of Py3), and secondly, the core operations are done with direct opcodes rather than looking up the ".append" method; but other than that, yes, it's exactly the same. Here's the disassembly (in 2.7, to avoid the indirection of the nested function):
3 6 SETUP_LOOP 33 (to 42) 9 LOAD_FAST 0 (x) 12 GET_ITER >> 13 FOR_ITER 25 (to 41) 16 STORE_FAST 2 (ch) 4 19 LOAD_FAST 1 (ret) 22 LOAD_ATTR 0 (append) 25 LOAD_GLOBAL 1 (ord) 28 LOAD_FAST 2 (ch) 31 CALL_FUNCTION 1 34 CALL_FUNCTION 1 37 POP_TOP 38 JUMP_ABSOLUTE 13 >> 41 POP_BLOCK 5 >> 42 LOAD_FAST 1 (ret) 45 RETURN_VALUE
Okay, so what exactly is going on here? Looks to me like there's some optimization happening in the list comp, but you can see that the same code is being emitted. It's not *perfectly* identical, but that's mainly because CPython doesn't take advantage of the fact that 'ret' was initialized to a list - it still does the full "look up 'append', then call it" work. I'm not sure why SETUP_LOOP exists in the full version and not the comprehension, but I believe it's to do with the break and continue keywords, which can't happen inside a comprehension. So, again, it's optimizations that are possible in the comprehension, but otherwise, the code is identical. Maybe "syntactic sugar" is pushing it a bit, but there's no fundamental difference between the two. Imagine if an optimizing compiler could (a) notice that there's no use of break/continue, and (b) do some static type analysis to see that 'ret' is always a list (and not a subclass thereof), and optimize the multi-line version. At that point, the two forms would look almost, or maybe completely, identical. So I'd support the "syntactic sugar" label here. Why is this discussion still on python-ideas? Shouldn't it be on python-demanding-explanations-for-status-quo by now? ChrisA
On 21 October 2016 at 05:09, Random832 <random832@fastmail.com> wrote:
We don't need to guess about this, since we can consult the language reference and see how comprehension semantics are specified for language implementors: https://docs.python.org/3/reference/expressions.html#displays-for-lists-sets... Firstly, container displays are broken out into two distinct kinds: For constructing a list, a set or a dictionary Python provides special syntax called “displays”, each of them in two flavors: - either the container contents are listed explicitly, or - they are computed via a set of looping and filtering instructions, called a comprehension. Secondly, the meaning of the clauses in comprehensions is spelled out a little further down: The comprehension consists of a single expression followed by at least one for clause and zero or more for or if clauses. In this case, the elements of the new container are those that would be produced by considering each of the for or if clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. We can also go back and read the design PEPs that added these features to the language: * List comprehensions: https://www.python.org/dev/peps/pep-0202/ * Generator expressions: https://www.python.org/dev/peps/pep-0289/ PEP 202 defined the syntax in terms of its proposed behaviour rather than a syntactic, with the only reference to the nesting equivalence being this BDFL pronouncement: - The form [... for x... for y...] nests, with the last index varying fastest, just like nested for loops. PEP 289, by contrast, fully spells out the implied generator definition that was used to guide the implementation of generator expressions in the code generator: g = (tgtexp for var1 in exp1 if exp2 for var2 in exp3 if exp4) is equivalent to: def __gen(bound_exp): for var1 in bound_exp: if exp2: for var2 in exp3: if exp4: yield tgtexp g = __gen(iter(exp1)) del __gen When I implemented the comprehension index variable hiding for Python 3.0, the final version was the one where I blended those two definitions to end up with the situation where: data = [(var1, var2) for var1 in exp1 if exp2 for var2 in exp3 if exp4)] is now equivalent to: def __comprehension(bound_exp): __hidden_var = [] for var1 in bound_exp: if exp2: for var2 in exp3: if exp4: __hidden_var.append((var1, var2)) return __hidden_var data = __comprehension(iter(exp1)) del __comprehension While it's pretty dated now (I wrote it circa 2.5 as part of a draft book manuscript that was never published), if you'd like to learn more about this, you may want to check out the section on "Initialising standard containers" in http://svn.python.org/view/sandbox/trunk/userref/ODF/Chapter02_StatementsAnd... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 2016-10-17 10:32, Steven D'Aprano wrote:
It seems to me that this difference is fundamental. The entire point of this type of generalization is to break that invariant and allow the number of elements in the result list to vary independently of the number of iterations in the comprehension. It seems that a lot of this thread is talking at cross purposes, because the specifics of the syntax don't matter if you insist on that invariant. For instance, there's been a lot of discussion about whether this use of * is or isn't parallel to argument unpacking or assignment unpacking, or whether it's "intuitive" to some people or all people. But none of that matters if you insist on this invariant. If you insist on this invariant, no syntax will be acceptable; what is at issue is the semantics of enlarging the resulting list by more than one element. Now, personally, I don't insist on that invariant. I would certainly like to be able to do more general things in a list comprehension, and many times I have been irritated by the fact that the one-item-per-loop invariant exists. I'm not sure whether I'm in favor of this particular syntax, but I'd like to be able to do the kind of things it allows. But doing them inherently requires breaking the invariant you describe. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
But the proposal has explicit syntax that point the reader to the fact that the invariant doesn't hold. Same as other unpacking occurences: [x, *y] The invariant does not hold. And that's explicit. Elazar בתאריך יום ב׳, 17 באוק' 2016, 21:16, מאת Brendan Barnwell < brenbarn@brenbarn.net>:
On Mon, Oct 17, 2016 at 11:16:03AM -0700, Brendan Barnwell wrote:
I hear you, because I too would like to introduce a variant comprehension that uses a while instead of if. So don't think I'm not sympathetic. But that's not an option, and given the official position on comprehensions, I don't think this should be either. Officially, list comprehensions are not a replacement for general for-loops. Python is not Perl, where we encourage people to write one-liners, nor is it Haskell, where everything is an expression. If you want to do "more general things", use a real for-loop. Comprehensions are targetted at a narrow but important and common set of use-cases.
That last point is incorrect. You already can do the kind of things this thread is about: [*t for t in iterable] # proposed syntax: flatten can be written as: [x for t in iterable for x in t] If you want to apply a function to each element of t: [func(x) for x in [*t for t in iterable]] # proposed syntax can be written today: [func(x) for t in iterable for x in t] If you want to apply a function or expression to t first: [*func(t) for t in iterable] # proposed syntax [*expression for t in iterable] # proposed syntax this too can be written today: [x for t in iterable for x in func(t)] [x for t in iterable for x in expression] You might have an opinion on whether it is better to have an explicit extra loop (more verbose, but less magical) or special syntax (more compact, but more magical), but I don't think this proposal adds anything that cannot be done today. -- Steve
On 2016-10-17 16:35, Steven D'Aprano wrote:
Right, but by "doing those kinds of things" I mean doing them more in a more conise way without an extra level of iteration. (You can "do multiplication" by adding repeatedly, but it's still nice to have multiplication as an operation.) -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Mon, Oct 17, 2016 at 10:32 AM, Steven D'Aprano <steve@pearwood.info> wrote:
So, yet again for emphasis: I see what you mean about unrolling the list
comprehension into a list display. But I believe that's not a helpful way to think about list comprehensions.
Moreover, this "magical flatten" operator will crash in bad ways that a regular flatten() will not. I.e. this is fine (if strange):
It's hard to see how that won't blow up under the new syntax (i.e. generally for all infinite sequences). Try running this, for example:
a, *b = count()
Syntactically valid... but doesn't terminate. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 17 October 2016 at 18:32, Steven D'Aprano <steve@pearwood.info> wrote:
For my part: 1. I've acknowledged that equivalence. As well as the fact that the proposal (specifically, as explained formally by Greg) is understandable and a viable possible extension. 2. I don't find the "interpolation" equivalence a *good* way of interpreting list comprehensions, any more than I think that loops should be explained by demonstrating how to unroll them. 3. I've even explicitly revised my position on the proposal from -1 to -0 (although I'm tending back towards -1, if I'm honest...). 4. Whether you choose to believe me or not, I've sincerely tried to understand the proposal, but I pretty much had to insist on a formal definition of syntax and semantics before I got an explanation that I could follow. However: 1. I'm tired of hearing that the syntax is "obvious". This whole thread proves otherwise, and I've yet to hear anyone from the "obvious" side of the debate acknowledge that. 2. Can someone summarise the *other* arguments for the proposal? I'm genuinely struggling to recall what they are (assuming they exist). It feels like I'm hearing nothing more than "it's obvious what this does, it's obvious that it's needed and the people saying it isn't are wrong". That may well not be the truth, but *it's the impression I'm getting*. I've tried to take a step back and summarise my side of the debate a couple of times now. I don't recall seeing anyone doing the same from the other side (Greg's summarised the proposal, but I don't recall anyone doing the same with the justification for it). 3. The fact is that the proposed behaviour was *specifically* blocked, *precisely* because of strong concerns that it would cause readability issues and only had "mild" support. I'm not hearing any reason to change that decision (sure, there are a few people here offering something stronger than "mild" support, but it's only a few voices, and they are not addressing the readability concerns at all). There was no suggestion in the PEP that this decision was expected to be revisited later. Maybe there was an *intention* to do so, but the PEP didn't state it. I'd suggest that this fact alone implies that the people proposing this change need to write a new PEP for it, but honestly I don't think the way the current discussion has gone suggests that there's any chance of putting together a persuasive PEP, much less a consensus decision. And finally, no-one has even *tried* to explain why we need a third way of expressing this construction. Nick made this point, and basically got told that his condition was too extreme. He essentially got accused of constructing an impossible test. And yet it's an entirely fair test, and one that's applied regularly to proposals - and many *do* pass the test. It's worth noting here that we have had no real-world use cases, so the common approach of demonstrating real code, and showing how the proposal improves it, is not available. Also, there's no evidence that this is a common need, and so it's not clear to what extent any sort of special language support is warranted. We don't (as far as I know, and no-one's provided evidence otherwise) see people routinely writing workarounds for this construct. We don't hear of trainers saying that pupils routinely try to do this, and are surprised when it doesn't work (I'm specifically talking about students *deducing* this behaviour, not being asked if they think it's reasonable once explained). These are all arguments that have been used in the past to justify new syntax (and so reach Nick's "bar"). And we've had a special-case function (flatten) proposed to cover the most common cases (taking the approach of the 80-20 rule) - but the only response to that proposal has been "but it doesn't cover <artificial example>". If it didn't cover a demonstrably common real-world problem, that would be a different matter - but anyone can construct cases that aren't covered by *any* given proposal. That doesn't prove anything. I don't see any signs of progress here. And I'm pretty much at the point where I'm losing interest in having the same points repeated at me over and over, as if repetition and volume will persuade me. Sorry. Paul
On Mon, Oct 17, 2016, at 16:12, Paul Moore wrote:
As the one who made that accusation, my objection was specifically to the word "always" - which was emphasized - and which is something that I don't believe is actually a component of the test that is normally applied. His words, specifically, were "a compelling argument needs to be presented that the new spelling is *always* preferable to the existing ones" List comprehensions themselves aren't even always preferable to loops.
On 17 October 2016 at 21:30, Random832 <random832@fastmail.com> wrote:
Sigh. And no-one else in this debate has ever used exaggerated language. I have no idea if Nick would reject an argument that had any exceptions at all, but I don't think it's unreasonable to ask that people at least *try* to formulate an argument that demonstrates that the two existing ways we have are inferior to the proposal. Stating that you're not even willing to try is hardly productive. Paul
On Mon, Oct 17, 2016 at 11:13 PM Paul Moore <p.f.moore@gmail.com> wrote: ...
2. Can someone summarise the *other* arguments for the proposal? I'm genuinely struggling to recall what they are (assuming they exist).
My own argument was uniformity: allowing starred expression in other places, and I claim that the None-aware operator "come out" naturally from this uniformity. I understand that uniformity is not held high as far as decision making goes, and I was also kindly asked not to divert this thread, so I did not repeat it, but the argument is there. Elazar
On 17.10.2016 22:12, Paul Moore wrote: that's okay.
If it doesn't meet your standards of real-world code, okay. I meets mine. there were trainers who routinely reported this, that would be a strong argument for it. However, the absence of this signal, is not an argument against it IMHO.
Same here. The discussion is inconclusive. I think it's best to drop it for the time being. Best, Sven
On 17 October 2016 at 21:33, Sven R. Kunze <srkunze@mail.de> wrote:
Thank you. You're correct that was mentioned. I infer from the responses that it isn't sufficient, but I should have noted it explicitly. Elazar also mentioned consistency, which I had also forgotten. He noted in his comment (and I agree) that consistency isn't a compelling argument in itself. I'd generalise that point and say that theoretical arguments are typically considered secondary to real-world requirements.
Apologies. I had completely missed that example. Personally, I'd be inclined to argue that you shouldn't try so hard to build the list you want to return in a single statement. You can build the return value using a loop. Or maybe even write a function designed to filter Postgres result sets, which might even be reusable in other parts of your program. It's not my place to tell you how to redesign your code, or to insist that you have to use a particular style, but if I were writing that code, I wouldn't look for an unpacking syntax.
If this proposal had been described as "a syntax to replace chain.from_iterable", then it might have been received differently. I doubt it would have succeeded even so, but people would have understood the use case better. For my part, I find the name chain.from_iterable to be non-obvious. But if I needed to use it a lot (I don't!) I'd be more likely to simply come up with a better name, and rename it. Naming is *hard*, but it's worthwhile. One problem (IMO) of the "propose some syntax" approach is that it avoids the issue of thinking up a good name, by throwing symbols at the problem. (And you can't google for a string of symbols...)
No-one is looking for arguments *against* the proposal. Like it or not "status quo wins" is the reality. People need to look for arguments in favour of the proposal.
Thanks for the reasoned response. Paul
On Mon, Oct 17, 2016 at 10:33:32PM +0200, Sven R. Kunze wrote:
Sorry? You know, I am all for real-world code and I also delivered: https://mail.python.org/pipermail/python-ideas/2016-October/043030.html
Your example shows the proposed: [*(language, text) for language, text in fulltext_tuples if language == 'english'] which can be written as: [x for language, text in fulltext_tuples for x in (language, text) if language == 'english'] which is only ten characters longer. To me, though, there's simply no nice way of writing this: the repetition of "language, text" reads poorly regardless of whether there is a star or no star. If I were doing this more than once, I'd be strongly inclined to invest in a simple helper function to make this more readable: def filter_and_flatten(language, fulltext): for lang, text in fulltext: if lang == language: yield lang yield text filter_and_flatten('english', fulltext_tuples) In some ways, list comprehensions are a trap: their convenience and ease of use for the easy cases lure us into using them when we ought to be using a generator. But that's just my opinion. -- Steve
Steven D'Aprano wrote:
I think the ugliness of this particular example has roots in the fact that a tuple rather than an object with named fields is being used, which is going to make *any* piece of code that touches it a bit awkward. If it were a namedtuple, for example, you could write [*t for t in fulltext_tuples if t.language == 'english'] or [x for t in fulltext_tuples if t.language == 'english' for x in t] The latter is a bit unsatisfying, because we are having to make up an arbitrary name 'x' to stand for an element of t. Even though the two elements of t have quite different roles, we can't use names that reflect those roles. Because of that, to my eyes the version with * makes it easier to see what is going on. -- Greg
On 18.10.2016 08:23, Greg Ewing wrote:
It's an intriguing idea to use namedtuples but in this case one should not over-engineer. What I dislike most are the names of "fulltext_tuple", "x", "t". If I were to use it, I think my coworkers would tar and feather me. ;) This is one of the cases where it makes absolutely no sense to invent artificial names for the sake of naming. I can name a lot of (internal) examples where we tried really hard at inventing named concepts which make absolutely no sense half a year later even to those who invented them. Repeatedly, in the same part of the code. Each newly named concept introduces another indirection. Thus, we always need to find a middle ground between naming and using language features, so I (personally) would be grateful for this particular feature. :)
Because of that, to my eyes the version with * makes it easier to see what is going on.
That's a very nice phrase: "makes it easier to see what is going on". I need to remember that. Cheers, Sven
Steven D'Aprano wrote:
You don't have to believe that, because thinking about it as a for-loop works equally well. Without the star, it means "insert each of these things into a list". With the star, it means "unpack each of these things into a list".
In a list comprehension, we expect the invariant that the number of items produced will equal the number of loops performed.
There's a corresponding invariant for list displays: the number of items produced is equal to the number of expressions in the display. But that doesn't hold when the display includes unpacking, for obvious reasons. For the same reasons, we shouldn't expect it to hold for comprehensions with unpacking. -- Greg
On Mon, Oct 17, 2016 at 9:11 AM, Random832 <random832@fastmail.com> wrote:
What you're saying is EXACTLY 180 deg reversed from the truth. It's *precisely* because it doesn't need the extra complication that `flatten()` is more flexible and powerful. I have no idea what your example is meant to do, but the actual correspondence is: [f(x) for x in flatten(it)] Under my proposed "more flexible recursion levels" idea, it could even be: [f(x) for x in flatten(it, levels=3)] There would simply be NO WAY to get that out of the * comprehension syntax at all. But a decent flatten() function gets all the flexibility.
I am absolutely sincere in disliking and finding hard-to-teach this novel use of * in comprehensions. Yours, David... P.S. It's very artificial to assume user are unable to use 'from itertools import chain' to try to make chain() seem more cumbersome than it is. Likewise, I would like flatten() in itertools, but I assume the usual pattern would be importing the function itself. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 17.10.2016 20:38, David Mertz wrote:
I see what you are trying to do here and I appreciate it. Just one thought from my practical experience: I haven't had a single usage for levels > 1. levels==1 is basically * which I have at least one example for. Maybe, that relates to the fact that we asked our devs to use names (as in attributes or dicts) instead of deeply nested list/tuple structures. Do you think it would make sense to start a new thread just for the sake of readability?
You are consistent at least. You don't teach * in list displays, no matter if regular lists or comprehensions. +1
I am sorry but it is cumbersome. Regards, Sven
On 17 October 2016 at 20:35, Sven R. Kunze <srkunze@mail.de> wrote:
Imports are a fundamental part of Python. How are they "cumbersome"? Is it cumbersome to have to import sys to get access to argv? To import re to use regular expressions? To import subprocess to run an external program? Importing the features you use (and having an extensive standard library of tools you might want, but which don't warrant being built into the language) is, to me, a basic feature of Python. Certainly having to add an import statement is extra typing. But terseness was *never* a feature of Python. In many ways, a resistance to overly terse (I could say "Perl-like") constructs is one of the defining features of the language - and certainly, it's one that drew me to Python, and one that I value. Paul
On 17.10.2016 22:26, Paul Moore wrote:
The statement about "cumbersomeness" was specific to this whole issue. Of course, importing feature-rich pieces from the stdlib is really cool. It was more the missed ability to do the same with list comprehensions of what is possible with list displays today. List displays feature * without importing anything fancy from the stdlib. Nevermind, it seems we need to wait longer for this issue to come up again and maybe again to solve it eventually. Best, Sven
On 17 October 2016 at 21:43, Sven R. Kunze <srkunze@mail.de> wrote:
In your other post you specifically mentioned itertools.chain.from_iterable. I'd have to agree with you that this specific name feels clumsy to me as well. But I'd argue for finding a better name, not replacing the function with syntax :-) Cheers, Paul
On Mon, Oct 17, 2016, at 14:38, David Mertz wrote:
No, it's not. For a more concrete example: [*range(x) for x in range(4)] [*(),*(0,),*(0,1),*(0,1,2)] [0, 0, 1, 0, 1, 2] There is simply no way to get there by using flatten(range(4)). The only way flatten *without* a generator expression can serve the same use cases as this proposal is for comprehensions of the *exact* form [*x for x in y]. For all other cases you'd need list(flatten(...generator expression without star...)).
As Paul or someone pointed out, that's a fairly odd thing to do. It's the first time that use case has been mentioned in this thread. It's true you've managed to construct something that isn't done by flatten(). I would have had to think a while to see what you meant by the original if you haven't provided the intermediate interpretations. Of course, it's *really simple* to spell that in a natural way with existing syntax that isn't confusing like yours: [x for end in range(4) for x in range(end)] There is no possible way to construct something that would use the proposed syntax that can't be expressed more naturally with a nested loop... because it's just confusing syntax sugar for exactly that. Your example looks like some sort of interview quiz question to see if someone knows obscure and unusual syntax.
On Mon, Oct 17, 2016, at 22:17, David Mertz wrote:
[*range(x) for x in range(4)]
As Paul or someone pointed out, that's a fairly odd thing to do.
I agree with the specific example of it being an odd thing to do with range, it was just an attempt to illustrate with a concrete example.
It's the first time that use case has been mentioned in this thread.
I think that in general the "body involves a subexpression returning an iterable other than the bare loop variable" has been covered before, though it might not have been clear at all times that that was what was being discussed. Frankly, I think it's rare that something of the form "x for x ..." is best written with a comprehension in the first place, and the same would be true for "*x for x..." so I didn't like that some of the translations being discussed only work well for that case.
I feel like I should be honest about something else - I'm always a little bit confused by the ordering for comprehensions involving multiple clauses. For me, it's the fact that: [[a for a in b] for b in ['uvw', 'xyz']] == [['u', 'v', 'w'], ['x', 'y', 'z']] which makes me want to write: [a for a in b for b in ['uvw', 'xyz']] but that's an error, and it actually needs to be [a for b in ['uvw', 'xyz'] for a in b] == ['u', 'v', 'w', 'x', 'y', 'z'] So when this talk of readability issues comes up and the recommended alternative is something that I don't really find readable, it's frustrating. To me this proposal is something that would allow for more things to be expressed without resorting to multi-loop comprehensions.
On Mon, Oct 17, 2016 at 7:50 PM, Random832 <random832@fastmail.com> wrote:
It's also easy to construct examples where the hypothetical * syntax can't handle a requirement. E.g. flatten() with levels>1 (yes, of course you can find some way to nest more loops or more comprehensions-within-comprehensions to make it work in some way that still uses the * by force). I feel like I should be honest about something else - I'm always a
little bit confused by the ordering for comprehensions involving multiple clauses.
Me too! I get the order of nested loops in comprehensions wrong about 25% of the time. Then it's a NameError, and I fix it. This is a lot of why I like a utility function like `flatten()` that is pretty much self-documenting. Perhaps a couple other itertools helpers would be nice. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 18 October 2016 at 13:32, David Mertz <mertz@gnosis.cx> wrote:
This is also one of the main reasons that named generator expression pipelines can sometimes be easier to read than nested comprehensions: incrementally_increasing_ranges = (range(end) for end in itertools.count()) flatten = itertools.chain.from_iterable incrementally_increasing_cycles = flatten(incrementally_increasing_ranges()) Forcing ourselves to come up with a name for the series of values produced by the outer iteration then makes that name available as documentation of our intent for future readers of the code. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 18 October 2016 at 07:31, Nick Coghlan <ncoghlan@gmail.com> wrote:
This is a key point, that is frequently missed when people propose new "shorter" syntax, or constructs that "reduce indentation levels". Make no mistake, coming up with good names is *hard* (and I have enormous respect for the (often unrecognised) people who come up with intuitive APIs and names for functions). So it's very easy to be tempted by "concise" constructs that offer the option of not naming an operation. But as someone who spends 99% of his time doing maintenance programming, be sure that the people who support your code will thank you for spending time naming your abstractions (and avoiding using constructs like the one proposed here). Paul
Random832 wrote:
You're not alone! Lately I've been becoming convinced that this is the way we should have done it right back at the beginning. But it's far too late to change it now, sadly. Our only hope would be to introduce a new syntax, maybe [a for a in b; for b in ['uvw', 'xyz']] Inserting the semicolons mightn't be such a bad idea, because when doing the reversal it would be necessary to keep any 'if' clauses together with their preceding 'for' clause. So if we wrote [a for a in b if cond(a); for b in things] we could say the rule is that you split at the semicolons and then reverse the clauses. -- Greg
On 18/10/2016 07:40, Greg Ewing wrote:
If the syntax were [ x for x in alist for alist in list-of-lists ] there is a smooth (or rather, monotonic) gradation from the smallest object (x) to the next biggest object (alist) to the biggest object (list-of-lists), which IMHO is easier to follow. Each object is conceptually zero or one steps from its neighbour. But with the actual syntax [ x for alist in list-of-lists for x in alist ] there is a conceptual hiatus after "x" ("what on earth are alist and list-of-lists, and what have they got do to with x?"). This would be even more marked with more than 2 loops: we jump from the lowest level object to the two highest level objects, and it all seems like a disorienting non-sequitur until the very last loop "joins the dots". You have to identify the "x" at the beginning with the "x" near (but not at!) the end. Instead of (ideally, if not always in practice) reading the expression from left-to-right in one go, your eyes are forced to jump around in order for your brain to assimilate it. A possible alternative syntax might be to follow more closely the for-loop syntax, e.g. [ for alist in list-of-lists: for x in alist: x ] Here the "conceptual jump" between each object and the next is either 1 or 2, which for me makes this a "second best" option. But at least the "conceptual jump" is bounded (regardless of the number of loops), and this syntax has the advantage of familiarity.
Best wishes Rob Cliffe
For me the current behaviour does not seem unreasonable as it resembles the order in which you write out loops outside a comprehension except that the expression for generated values is provided first. On 21 October 2016 at 05:03, Sven R. Kunze <srkunze@mail.de> wrote:
Alexander Heger wrote:
For me the current behaviour does not seem unreasonable as it resembles the order in which you write out loops outside a comprehension
That's true, but the main reason for having comprehensions syntax in the first place is so that it can be read declaratively -- as a description of the list you want, rather than a step-by-step sequence of instructions for building it up. If you have to stop and mentally transform it into nested for-statements, that very purpose is undermined. -- Greg
Well, an argument that was often brought up on this forum is that Python should do things consistently, and not in one way in one place and in another way in another place, for the same thing. Here it is about the order of loop execution. The current behaviour in comprehension is that is ts being done the same way as in nested for loops. Which is easy enough to remember. Same way, everywhere. -Alexander
On 22.10.2016 09:50, Alexander Heger wrote:
Like * in list displays? ;-)
It still would. Just read it from right to left. The order stays the same.
Which is easy enough to remember. Same way, everywhere.
I am sorry but many disagree with you on this thread. I still don't understand why the order needs to be one-way anyway. Comprehensions are a declarative construct, so it should be possible to mix those "for .. in .."s up in an arbitrary order. Cheers, Sven
Thinking about it, though, I ended up exactly where you are now, except that I then thought about where an item would be known, and it seemed to me, yes, an item would be more likely known *after* looping over it rather than before: [bi for bi in before for before in iterable] # why should "before" exist before it is looped over? correctly: [bi for before in iterable for bi in before] it doesn't, it should be declared to the right hand side and only the result is kept over at the left hand edge. On a same note, with if expressions the picture might look different: [bi for bi in before for before in iterable if before[0] < 3] # ... is bi filtered now or not? correctly: [bi for before in iterable if before[0] < 3 for bi in before] it is filtered very clearly this way. cheers! mar77i
On 17 October 2016 at 21:22, Random832 <random832@fastmail.com> wrote:
the equivalent flatten for that is: flatten(range(x) for x in range(4)) ; flatten has no magic so will not replace a piece of code with two range calls (like your example) for code with one. I see some mention that flatten does not cover all cases; but correct me if I'm wrong with this statement: Any case of [*<exp1> for <vars> in <exp2>] could be replaced with flatten(<exp1> for <vars> in <exp2>). Where flatten is defined as def flatten(it): return [x for for subit in it for x in subit] (there is a slight difference where I'm making flatten iterable instead of a list) What perhaps was confusing is that in the case where <exp1> and <vars> are the same, you can also write flatten(<exp2>). So, for me, this feature is something that could be covered with a (new) function with no new syntax required. All you have to learn is that instead of [*...] you use flatten(...) Am I wrong? I keep reading people on both sides saying "flatten is not enough in all cases", and I can find a counterexample (even for 1 level flatten which is what I used here) PS: or alternatively, flatten = lambda it: list(itertools.chain(it)) # :) PPS: or if you prefer to work with iterators, flatten = itertools.chain -- Daniel F. Moisset - UK Country Manager www.machinalis.com Skype: @dmoisset
On Tue, Oct 18, 2016, at 04:01, Daniel Moisset wrote:
That is correct - though flatten as previously discussed did not return a list, so list(flatten(...)) was required, though I suppose you could use [*flatten(...)] - my point was that [especially with the list constructor] this is significantly more verbose than the proposed syntax.
On 18.10.2016 10:01, Daniel Moisset wrote:
The main motivation is not "hey we need yet another way for xyz" but "could we remove that inconsistency?". You wrote "[*...]"; which already works. It just that it does not work for all kinds of "..." . I hope that makes the motivation clearer. I for one don't need yet another function or function cascade to make things work. "list(itertools.chain.from_iterable(...))" just works fine for all kinds of "...". Cheers, Sven
On Sun, Oct 16, 2016 at 9:06 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I don't think I'd actually propose a builtin for this. For me, including the recipe that is in the itertools documentation into a function in the module would be plenty. Writing "from itertools import flatten" is not hard.
Actually, I think that if we added `flatten()` to itertools, I'd like a more complex implementation that had a signature like: def flatten(it, levels=1): # Not sure the best implementation for clever use of other itertools ... I'm not quite sure how one would specify "flatten all the way down." In practice, `sys.maxsize` is surely large enough; but semantically it feels weird to use an arbitrary large number to mean "infinitely if necessary." -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Sat, Oct 15, 2016 at 9:42 PM Steven D'Aprano <steve@pearwood.info> wrote:
Whoa, hang on a second there. It is intentionally prohibited because Joshua Landau (who helped a lot with writing and implementing the PEP) and I felt like there was going to be a long debate and we wanted to get PEP 448 checked in. If it "didn't make sense" as you say, then we would have said so in the PEP. Instead, Josh wrote: This was met with a mix of strong concerns about readability and mild support. In order not to disadvantage the less controversial aspects of the PEP, this was not accepted with the rest of the proposal. I don't remember who it was who had those strong concerns (maybe you?) But that's why we didn't include it. Best, Neil
On Sun, Oct 16, 2016 at 05:14:54AM +0000, Neil Girdhar wrote: [Steven (me), refering to Greg]
Okay, interesting, and thanks for the correction.
I don't remember who it was who had those strong concerns (maybe you?) But that's why we didn't include it.
I'm pretty sure it wasn't me. I don't recall being involved at all with any discussions about PEP 448, and a quick search has failed to come up with anything relevant. I think I sat that one out. -- Steve
participants (21)
-
Alexander Heger -
Brendan Barnwell -
Brett Cannon -
Chris Angelico -
Daniel Moisset -
David Mertz -
Ethan Furman -
Greg Ewing -
Ivan Levkivskyi -
Mariatta Wijaya -
Mark Lawrence -
Martti Kühne -
Ned Batchelder -
Neil Girdhar -
Nick Coghlan -
Paul Moore -
Random832 -
Rob Cliffe -
Steven D'Aprano -
Sven R. Kunze -
אלעזר