Map-then-filter in comprehensions
tl;dr What is support like for adding an 'as' clause to comprehension syntax? In order to allow map-then-filter, it might look like something this: [y for x in numbers if abs(x) as y > 5] I wish to propose an extension to Python comprehension syntax in an attempt to make it applicable in more areas. I'll first describe the deficiency I perceive in the current comprehension syntax and then propose my extension. I'll then note some drawbacks. For the purposes of illustration I'll use list comprehension syntax but I believe everything I say is equally applicable to set and dictionary comprehensions. I'll also talk about lists but again (more or less) everything applies to the more general concept of iterables. Comprehensions are essentially a filter followed by a map operation. So if you need to perform a filter operation followed by a map operation over a list, this is pretty convenient in Python. Here we are going to take the absolute value of all even numbers within the range -10 to 10. numbers = range(-10, 10) [abs(x) for x in numbers if x % 2 == 0] However, if you wish to perform a map operation and *then* a filter operation this is not so convenient, so suppose we wish to obtain the absolute value of all numbers that have an absolute value larger than 5, we can do this by calling the mapped method twice: abs(x) for x in numbers if abs(x) > 5] This is a bit unsatisfying and even impossible in the case that the mapped method has some side-effect (although arguably if you find yourself in that situation you have taken a mis-step somewhere). An alternative is to apply the mapping first: [y for y in (abs(x) for x in numbers) if y > 5] I have to say I quite like this, as it is pretty explicit, but it is a bit unsatisfying that you require only one comprehension for a filter-then-map but two for a map-then-filter. What would be nice is if we could give a name to the mapped expression and then use that in the filter. [abs(x) as y for x in numbers if y > 5] I don't like this as it means the order of execution is dependent on whether there is an 'as' clause, in particular the 'if' clause itself may do some computation such as in `if f(y) > 5`. An altenative is to allow 'as' expressions in the condition, something like: [y for x in numbers if abs(x) as y > 5] Note that it would be possible to map to an intermediate result, such as: [y**2 for x in numbers if abs(x) as y > 5] Alternatively we could put the 'as' in the pattern: [y**2 for abs(x) as y in numbers if y > 5] I did not like this as it is obscures the fact that 'x' is being set to each element of 'numbers'. Additionally, we might later wish to adopt a functional programming idiom in which we use 'as' for deconstructive assignment whilst giving a name to the entire matched value, for example: [p for (x,y) as p if x > y] Or more generally: (x,y) as a = f(z) But that is getting somewhat off-topic. I promised some drawbacks: * I am confident there are some implementation gotchas nestling in here somewhere. * I could imagine how abuse of such a mechanism to lead to pretty unreadable code. * I'm still not *that* upset by the explicit map first: `[y for y in (abs(x) for x in numbers) if y > 5]` * I could see how it is not immediately obvious what the code does. * It would need to be decided whether you allowed multiple 'as' expression in the condition, particularly using 'and' or 'or' as in 'if f(a) as x > 5 and f(b) as y > 5' To summarise: * It's a touch annoying that comprehensions allow filter-then-map but not map-then-filter * Three proposed syntaxes are: * [abs(x) as y for x in numbers if y > 5] * [y for x in numbers if abs(x) as y > 5] * [y**2 for abs(x) as y in numbers if y > 5] * My favourite is the middle one. Finally these seem to currently be syntax errors so we should not break any existing code.
On 8 March 2016 at 14:17, Allan Clark <allan.clark@gmail.com> wrote:
I understand your motivation here, but Python isn't really a functional programming language and isn't intended to be. So comprehensions, while powerful, are only appropriate for relatively simple cases - at the point where they are no longer sufficiently powerful, it's generally better to use an explicit for loop or generator function. (In actual fact, I'm finding more and more these days that I only use comprehensions for very simple cases, and switch to explicit loops quite early). So what seems to me to be missing from your proposal is an explanation of how extending the comprehension syntax is an improvement over not using a comprehension at all. You suggest [y for x in numbers if abs(x) as y > 5] as a simpler alternative to [y for y in (abs(x) for x in numbers) if y > 5] but I'd be much more likely to write results = [] for x in numbers: y = abs(x) if y > 5: results.append(y) or def bounded(it, lower): for val in it: absval = abs(val) if absval > lower: yield absval list(bounded(numbers, 5)) Obviously real world examples would be better than artificial ones, as artificially simple examples make terse notation look better... And in the example using a generator, you'd be able to give it a far more meaningful name with a bit of real-life domain terminology. Paul
On 8 March 2016 at 16:02, Paul Moore <p.f.moore@gmail.com> wrote:
Yeah that's a pretty fair observation.
Again, fair. Not sure I can quite articulate why I would prefer a comprehension here. A very weak argument would be that such code tends to change, and when it does it may get morphed back into something that *can* be done with a comprehension, but you might end up leaving it as a for-loop with append.
Yeah agreed. I would note that in a real-world example you are giving a name to something that is forced to calculate *and* filter. So your name is going to end-up being something like, say "is_registered_and_voting_for", or "is_highest_tax_bracket_and_is_taxed". Which you might not actually write, and instead opt for something like "tax_amount". Even in your code for my artificial example your generator is named "bounded" but that does not sound like it is doing any filtering at all, it sounds like it is simply bounding all values. Of course to be fair, I didn't give a name for that at all, and probably I want to give a name for the resulting list. Although of course we both need to do that, but at least with a comprehension you only have to come up with a name for the result, not for the result and the associated generator.
On 8 March 2016 at 16:20, Allan Clark <allan.clark@gmail.com> wrote:
That's actually a very good point - there's quite a distinct "break point" where you have to rewrite code as a loop rather than a comprehension, and for maintainability purposes that switch tends to be permanent. So delaying the point where you need to switch (assuming the comprehension form is readable and maintainable) is a fair goal. There's also the point that much code is actually one-off scripts, and the break point can be very different in such code - "how I think of it" has a much higher weight in such a situation (often even greater than "is it maintainable" for completely throwaway code). The problem then becomes one of finding a syntax that is natural and not forced. That's hard, particularly with things like the high bar on introducing new keywords meaning you're trying to reuse words that "sort of" suit the situation.
Again, a good point. Compound clauses make for bad names typically, so if you're thinking in terms of "calculate and filter" it'll be hard to think of a really good name. Maybe a better approach is to look at chaining of individual building blocks. The [y for y in (abs(x) for x in numbers) if y > 5] approach takes that form, but the nesting hides the fact. In some code I wrote today, I used the form data = [abs(x) for x in numbers] data = [x for x in data if x > 5] You can even use generators data = (abs(x) for x in numbers) data = (x for x in data if x > 5) list(data) to make the calculations lazy. To me, that makes the step by step "calculate, then filter" pipeline explicit, and is actually really readable. Of course, once again this is the sort of thing that's very much about personal opinion, and you may hate that style (particularly using the generic variable name "data" over and over). So, overall I'd say I'm not against the idea, but there are some reasonably good alternatives already available, and so coming up with something that's compellingly better than the status quo is going to be a hard job. I'm glad we're having the discussion, though - we need to (re-) explore questions like this to avoid the language stagnating. And who knows when the inspiration will strike? :-) Paul
On Wed, Mar 9, 2016 at 7:06 AM, Pavol Lisy <pavol.lisy@gmail.com> wrote:
Enforcing that the order of operations is as you're implying: [y+z for x in numbers if (abs(x) as y) > 5 or (x**2 as z) > 100] If the first condition is true, the second will not be evaluated. It'll be equivalent to: def <listcomp>(): result = [] for x in numbers: y = abs(x) if y > 5: result.append(y+z) else: z = x**2 if z > 100: result.append(y+z) return result So, yeah, that could surprise a *lot* of people. Either an UnboundLocalError, or retaining the value from the previous iteration. Recommendation: If you use name bindings in the second half, use bitwise operators instead: [y+z for x in numbers if (abs(x) as y) > 5 | (x**2 as z) > 100] That'll force both sides to be evaluated. (Downside: It's reliable only if you actually have True and False.) ChrisA
On Tue, Mar 08, 2016 at 02:17:23PM +0000, Allan Clark wrote:
Seeing the many replies, I'm a bit lost as where to best comment. After thinking about it for a while, I think that currently it's not impossible to do what you want with comprehensions, just a bit convoluted. [y for x in numbers for y in [abs(x)] if y > 5] The tricky part being the `for y in [abs(x)]` basically doing what you want: bind the value `abs(x)` to a name (`y`). Benefits: - It requires no new syntax, no new semantics. - You can easily define multiple items at once. [y + z for x in numbers for y, z in [(abs(x), sgn(x)] if y > 5] - You can still use the original value, in contrast to [y for y in (abs(x) for x in numbers) if y > 5] Downside: - Very unreadable, and probably a long way off from being idiomatic Python. As for yet another syntax suggestion (if we want to introduce something). [y for x in numbers with abs(x) as y if y > 5] The benefit is that it reads quite natural: "with expr as name". Another benefit is that keywords get reused. However, that already has semantics outside a comprehension for context-managers. TL;DR: It's possible with [y for x in numbers for y in [abs(x)] if y > 5] But the syntax is ugly enough that it does warrant some extra syntactic sugar (or something with the same semantics but better performance characteristics).
On 8 March 2016 at 22:21, Sjoerd Job Postmus <sjoerdjob@sjec.nl> wrote:
Agreed, this is a plausible suggestion. But rather than just providing an example usage, it would be helpful to see the full syntax of comprehension with the proposed addition. See https://docs.python.org/3/reference/expressions.html#grammar-token-comprehen... - note that a comprehension can have arbitrarily many for and if clauses, interspersed in any order as long as the first one is a "for". I'm guessing you'd add "with <some_sort_of_expression> as <target_list>" as simply a third option. But what would the semantics be? I'm guessing that "with xxx as yyy" translates basically as a statement "yyy = xxx" in the notional expansion described in that section ("considering each of the for or if clauses a block...") Assuming that is the proposed definition, a few questions arise: 1. Is the behaviour this would assign to "with <foo> as x with <bar> as x" (i.e., repeated bindings of the same name) what we'd want? Is it likely to cause confusion in practice? 2. Do we need to restrict the <target_list>? Consider that "with something as global_var[0]" is allowed by the definition - that would leak values out of the comprehension. This isn't new - the "for" variable works like this already - but is it something that's more likely to be abused than currently? Should we care? (Python doesn't tend to prohibit people from writing bad code).
However, that already has semantics outside a comprehension for context-managers.
This is indeed a concern, albeit not a huge one - the definition has to point out that in the expansion "with" should be read as an assignment (written backwards) not as a with statement. It's not an impossible burden, but it is mildly inconsistent. This is probably the syntax I prefer of the ones suggested so far. But I still haven't seen any *really* convincing arguments that we need anything new in the first place. Paul
On Tue, Mar 08, 2016 at 10:58:16PM +0000, Paul Moore wrote:
It took me a while to figure this out (battling with the compiler over this), but I settled on the following grammar. comprehension ::= expression comp_for comp_for ::= "for" target_list "in" or_test [comp_iter] comp_iter ::= comp_for | comp_if | comp_with comp_if ::= "if" expression_nocond [comp_iter] comp_with ::= "with" or_test "as" target (For comp_with you probably want target_list instead of a single target. However, I felt like double-checking that I got my actual assumptions right by first implementing it (I've got it working now), and seeing as I'm not that experienced with modifying the Python parse/ast/symtable/compile phases, I decided to cop out and take the easy route. It should be fairly trivial to extend it to a target_list instead of a target.)
Trying to word it in such a way: "... considering each of the `for` or `if` clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. The `with expr as target` should be considered equivalent to `target = expr`. (And here we already see the downside of this idea). Normally a comprehension of the form (expr1 for target1 in expr2 for target2 in expr3 if expr4) boils down to for target1 in expr2: for target2 in expr3: if expr4: yield expr1 The natural extension would be for (expr1 for target1 in expr2 with expr3 as target2 if expr4) to reduce as follows. for target1 in expr2: with expr3 as target2: if expr4: yield expr1 Instead, it becomes for target1 in expr2: target2 = expr3: if expr4: yield expr1 But of course we're not going to have context managers in comprehensions, are we? So this inconsistency is somewhat forgiveable.
You mean similar to >>> [x for x in range(5) for x in range(x)] [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] I've never been confused by that in practice, as most people use good names for stuff. So I would see no ultra-important reason to prevent blocking it for this case and not in others.
I'd suggest not to bother with resolving naming conflicts or assignments to things other than names. [__ for foo.bar in range(10)] is already valid Python. No need to limit new features in ways which their 'friends' are not.
I agree on not having seen a really convincing argument yet. Especially since `for target in [expr]` works as well as `with expr as target`. The one thing I can think of is common subexpression elimination. [cheap_process(expensive_thingy(x)) for x in items if expensive_thingy(x) > 0] or [cheap_process(y) for x in items with expensive_thingy(x) as y if y > 0] But in Python3, you could just as well write [cheap_process(x) for x in map(expensive_thingy, items) if x > 0]
Paul
2016-03-09 6:16 GMT+01:00, Sjoerd Job Postmus <sjoerdjob@sjec.nl>: [...]
If we want variable assignment we have already "for var in [expr]" syntax. We could discuss if "with expr as var" syntax is more beautiful. (or if it not against There should be one-- and preferably only one --obvious way to do it.) But why omit context manager semantics in "with expr as var" assignment syntax? I personally don't like idea that semantics could be context sensitive in this way. (and we could already do pretty complex things in comprehension)
Some time ago I was trying to solve the same issue but using a new keyword "where", and I thought that new keyword is too much for just list comprehension filtering, so I've made it something like assignments in expresion, eg. (x+y)**2 + (x-y)**2 where x=1, y=2 So for list comprehension I can write: [stripped for line in lines if stripped where stripped=line.strip()] or: result = map(f, objs) where f=lambda x: x.return_something() or: it = iter(lines) while len(line) > 4 where line=next(it, '').strip(): print(line) or: lambda x, y: ( 0 if z == 0 else 1 if z > 0 else -1) where z = x + y or even: lambda something: d where (d, _)=something, d['a']=1 I even implemented it: https://github.com/thektulu/cpython/commit/9e669d63d292a639eb6ba2ecea3ed2c0c... and it works nicely. I was thinking to reuse "with [expr] as [var]" but I also don't like idea of context sensitive semantics, and I even thought that maybe someone, someday would want to write "content = fp.read() with open('foo.txt') as fp"... The "where" keyword is from guards pattern in Haskell :) 2016-03-10 17:53 GMT+01:00 Pavol Lisy <pavol.lisy@gmail.com>:
On Thu, Mar 10, 2016 at 5:20 PM, Michał Żukowski <thektulu.pp@gmail.com> wrote:
This is my understanding of `with Expr() as Name: Block()`, in pseudocode (ignoring exception handling). _context = Expr() Name = _context.__enter__(...) exec(Block, {"Name": Name}) _context.__exit__(...) Strawman proposal: Extend `with` to allow non-context managers. _context = Expr() if hasattr(_context, '__enter__'): # Context manager. Name = _context.__enter__(...) exec(Block, {"Name": Name}) _context.__exit__(...) else: # Normal object: treat as name bind Name = _context exec(Block, {"Name": Name}) Then you're allowed to write with 5 as x: print(x*2) And if your comprehension `with` receives context managers, they will be treated as such. # Closes each file after reading the second byte. second_bytes = [f.read(1) for fname in fnames with open(fname) as f if f.read(1) != '\0'] # Expanded to a loop. second_bytes = [] for fname in fnames: with open(fname) as f: if f.read(1) != '\0': second_bytes.append(f.read(1)) This breaks TOOWTDI by giving another way to assign. But it might not violate the spirit: I think very few people would trade a simple assignment for a backward assignment and an extra indentation level (!), especially newbies from other languages. TOOWTDI doesn't want competing ways of doing things, and this way isn't competitive. But that means it's a feature not intended for use, and only added for consistency with a new feature. The comprehension `with` will be perfectly aligned and translates naturally to the current `with`. It introduces an inconsistency between context managers and other objects, though. Any code that used to work will still work, but now any code that accidentally uses a non-context manager will not fail fast. It might also make the real use of normal `with` harder to learn. You have a gotcha for people trying to process and store a list of context managers. (Are locks used like this?) I believe that use will be both rare and advanced, so the gotcha is worse than one that is common and beginner-level.
On 11 March 2016 at 12:58, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
with doesn't create a new scope - it's more akin to a try/finally statement with a particular form (PEP 343 defines the full expansion, and that expansion is still broadly accurate, although some the specific details are different these days) The only compound statements in Python that create new scopes are "def" and "class". It's also worth reading PEP 343 for Guido's explanation for why the context management assignment syntax isn't "with VAR = EXPR": it's because EXPR *isn't* the thing being assigned, but rather a stepping stone towards obtaining that thing. This characteristic is true of all of the places where "as" is currently used instead of "=": with CM_EXPR as VAR: # VAR = CM_EXPR.__enter__(), not CM_EXPR except EXC_TYPE_EXPR as VAR: # VAR = the caught exception, not the exception type constraint # There's an implied "del VAR" at the end of the suite, but still not a full implicit scope import MODULE_REFERENCE as VAR from MODULE_REFERENCE import NAME as VAR # Here, the LHS isn't even a normal expression - it's a module name, or a module name plus an attribute name Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mar 10, 2016 10:51 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 11 March 2016 at 12:58, Franklin? Lee <leewangzhong+python@gmail.com>
wrote: thought that
Sorry, that was just sloppiness with `exec`. I wanted to convey the block using the new `Name`, and "function call" popped up first. I erased a comment about how, after running the block, `Name` would still hold the value it received. I forgot to mention that people might choose the proposed `with` for assignment over `=` if they wrongly thought that the `with` form deleted `Name` afterward.
Then would it be a problem if it defaulted to the value of EXPR directly when that value isn't of a specially-handled type (such as a context manager, for `with`)? Other than that learning `with` in comprehensions might make it harder to learn context managers.
2016-03-11 6:38 GMT+01:00, Franklin? Lee <leewangzhong+python@gmail.com>:
On Mar 10, 2016 10:51 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
[...]
Or we could think about syntax "with VAR = EXPR" where it is plain assignment. If we wanted in PEP343 (and still we want) to have different syntax for different semantics then it is better, I think. PS. from my point of view I still don't see why is "with" so much better than existing syntax "for VAR in [EXPR]" for example why is this not enough? -> import unicodedata [(c, i, cat, name, norm) for i in range(1, 0x110000) for c in [chr(i)] for cat in [unicodedata.category(chr(i))] for name in [unicodedata.name(chr(i), 'noname')] for uname in [name.upper()] # name is defined in previous row for norm in [unicodedata.normalize('NFKC', chr(i))] if 'OPERATOR' in uname if norm != c ] PPS. if we want to change syntax the "for VAR = EXPR" is also possibility
On Fri, Mar 11, 2016 at 2:42 PM, João Bernardo <jbvsmo@gmail.com> wrote:
On Thu, Mar 10, 2016 at 7:20 PM, Michał Żukowski <thektulu.pp@gmail.com> wrote:
[...]
One cool thing about this 'where' thing is that, if it made a new scope on every iteration, you could do [lambda x: x * j for i in range(10) where j = i] and it would actually create ten *different* functions (as opposed to without the 'where j = i'). - Koos
On Thu, Mar 10, 2016 at 11:20:21PM +0100, Michał Żukowski wrote:
But in Haskell, the `where` keyword also considers scoping. That is, outside the statement/expression with the `where`, you can't access the variables introduced by the where. Even though the `where` looks kind-of-nice, it (at least to me) is also a bit confusing with respect to evaluation order. Consider [ stripped for idx, line in enumerate(lines) if idx >= 5 or stripped where stripped=line.strip() ] (intended semantics: give me all lines (stripped), but ignore any lines that are whitespace-only in the first 5 lines) retval = [] for idx, line in enumerate(lines): stripped = line.strip() if idx >= 5 or stripped: retval.append(stripped) now I'm not very sure, but I expect what actually happens is: retval = [] for idx, line in enumerate(lines): if idx < 5: stripped = line.strip() if idx >= 5 or stripped: retval.append(stripped) that is, should I read it as (if idx >= 5 or stripped) where stripped=line.strip() or if idx >= 5 or (stripped where stripped=line.strip()) For comprehensions, I'd think the 'let' statement might make more sense. Abusing Haskell's notation: [ stripped | (idx, line) <- zip [0..] lines, let stripped = strip line, idx >= 5 || length stripped > 0 ] Porting this to something Python-ish, it'd be [ stripped for idx, line in enumerate(lines) let stripped = line.strip() if idx >= 5 or stripped ] where `let` is a keyword (possibly only applicable in a compexpr). In Haskell it's a keyword everywhere, but it has somewhat different semantics.
Yes, but I wanted it to be simple and powerful, not necessarily right in every context. Even though the `where` looks kind-of-nice, it (at least to me) is also
I've implemented it as "or_test [where_expr]" so the default order is the same as in: (if idx >= 5 or stripped) where stripped=line.strip() I wanted it to always "scope" left as much as possible - in precedence order between "lambda" and "... if ... else ..." - but left recursion forced me to place it after "... if ... else ..." which can't be used without brackets in list comprehension "if" filtering. But one can always control order with brackets, and make it work like in second example. For comprehensions, I'd think the 'let' statement might make more sense.
I was thinking about "let", but as I said, I think that new keyword should be more powerful than just filtering in list comprehensions, and in regular expresions it would look like this: if let x=foo() in x: print(x) Which does not look great, and there is problem with "in" keyword that make this expresion ambiguous.
On Mar 11, 2016, at 07:29, Michał Żukowski <thektulu.pp@gmail.com> wrote:
This makes where look like another top-level comprehension clause like for and if, but it doesn't follow the same nesting rules as the other clauses. Which means that whenever something isn't trivial enough to just read holistically, trying to apply the dead-simple rule of "write each clause as a nested statement, in the same order" will just lead to confusion, instead of immediately telling you the interpretation. Where that line is depends on how experienced you are with Python, how much time you've spent recently in a language with similar but not identical comprehension syntax, how tired you are, etc., but that doesn't matter--any comprehension with a where clause has at least three clauses, and is likely to be complicated enough to be over that line for some people. So, if you're going to recommend that this is only used when it's simple enough that it doesn't matter that the translation is confusing, that would mean recommending never using it, which implies that we shouldn't have it. A new clause that nests in order, like some of your other variations, doesn't have this problem.
I think you're overcomplicating this. You want let to be a statement, but also have a value--but you only need one or the other. As a statement, there's no need for a value; you're just creating or rebinding a variable for use in the controlled suite: let x = foo(): if x: print(x) And in a comprehension, it's just another clause that converts to the identical compound statement and nests the same as other clauses: [stripped for line in file let stripped=line.strip() if stripped] Hopefully you don't want scoping, but if you do, it's pretty obvious that the scope is the controlled suite; still no need for an in clause. If you really want a let expression instead of a statement, you still don't need an in clause unless you want scoping. Just make it an assignment expression whose value is the assigned value, just like in C and friends: if let x = foo(): print(x) Which extends easily to: if (let x = foo()) > 5: print(x) [stripped for line in file if let stripped=line.strip()] The only reason you'd need an in clause is if you wanted scoped expressions. But those would be almost useless in Python because, by design, you can't do too much inside an expression. But if you really want that, the design is again obvious: just as in functional languages, it's equivalent to a lambda call: let x=foo() in (print(x) if x > 5 else None) (lambda x: print(x) if x > 5 else None)(foo()) ... which is obviously unpythonic enough and useless enough that I think scoped let expressions are immediately dismissible as an idea.
Let statement is pointless - all things you can achievie by simply assign variables. Let expresion in form "let x=1" cannot "return" value when you want to define more than one variable - with "in" keyword you can define (like in Haskell) more than one variable, and then "return" a value, or you would need to assume that the last defined variable is returned, which is not intuitive :) 11-03-2016 21:25, "Andrew Barnert" <abarnert@yahoo.com> napisał(a):
every context. the same as in: people. So, if you're going to recommend that this is only used when it's simple enough that it doesn't matter that the translation is confusing, that would mean recommending never using it, which implies that we shouldn't have it.
A new clause that nests in order, like some of your other variations,
doesn't have this problem.
I wanted it to always "scope" left as much as possible - in precedence
Porting this to something Python-ish, it'd be
[ stripped for idx, line in enumerate(lines) let stripped =
order between "lambda" and "... if ... else ..." - but left recursion forced me to place it after "... if ... else ..." which can't be used without brackets in list comprehension "if" filtering. line, idx >= 5 || length stripped > 0 ] line.strip() if idx >= 5 or stripped ] the scope is the controlled suite; still no need for an in clause.
If you really want a let expression instead of a statement, you still
don't need an in clause unless you want scoping. Just make it an assignment expression whose value is the assigned value, just like in C and friends:
expressions. But those would be almost useless in Python because, by design, you can't do too much inside an expression. But if you really want that, the design is again obvious: just as in functional languages, it's equivalent to a lambda call:
scoped let expressions are immediately dismissible as an idea.
On Mar 11, 2016, at 15:59, Michał Żukowski <thektulu.pp@gmail.com> wrote:
Let statement is pointless - all things you can achievie by simply assign variables.
Except that a let statement starts with a keyword, meaning it can be turned into a comprehension clause, which is the entire point of this thread, and of the (unattributed) message you were replying to. Look at the example you replied to:
[ stripped for idx, line in enumerate(lines) let stripped = line.strip() if idx >= 5 or stripped ]
That's using let as a clause that maps to a statement with the exact same semantics as an assignment statement, and it nests the same way as other comprehension clauses. That's exactly what people are looking for here. Anything you add on top of that, unless it has some independent rationale, is just complication for no benefit.
Let expresion in form "let x=1" cannot "return" value when you want to define more than one variable- with "in" keyword you can define (like in Haskell) more than one variable, and then "return" a value, or you would need to assume that the last defined variable is returned, which is not intuitive :)
Assignments in Python always have a single value. In "x = y = 3", the value is 3. In "x, y = 3, 4" the value is (3, 4). So the value of "let x = y = 3" is 3, and the value of "let x, y = 3, 4" is (3, 4). There's no need for any extra assumptions. The reason Haskell needs a more structured let is scoping (and the fact that Haskell doesn't do sequencing).
I agree, the second suggestion was also my first though when I read the initial post. Seems very natural to me. If a syntax extension is not desired, the first suggestion looks like a cookbook example on how to do this to me. Could use tuple instead of list. +1 On 9 March 2016 at 09:21, Sjoerd Job Postmus <sjoerdjob@sjec.nl> wrote:
On 3/8/2016 9:17 AM, Allan Clark wrote:
-1 Comprehensions abbreviate a particular pattern of collection initialization, nested for and if statements, and collection augmentation innermost, with the only binding being the loop names. They are easily translated back to the original pattern, although move than one level of nesting can challenge comprehension in the normal meaning of the work. I strongly feel that the current correspondence between conprehensions and statements should be maintained. -- Terry Jan Reedy
So, as for the "post filter" - one can currently just express that in Python as: [y for y in (abs(x) for x in numbers) if y > 5] It may have one "internal generator" - but save for the little performance saving, I don't see this as less readable than the proposals above. As for the reuse of a value that might come in an expensive computation, that is something I always missed in Python - so, a couple years ago I put together a small package that can do just that: duplicate a computed value in a seamlesway, without requiring an explicit variable assignment. To do that, I use the mechanics of stack-based languages such as Postscript and Forth - as of now, one can do: ` from stackfull import push, popclear, clear [popclear() for x in numbers if push(expensive_calculation(x)) > 5] ` The functions pergorm stack operation in a list that is created in a transparent way on the current Python execution frame local variables dictionary. Works fine in Python, Python2 and pypy. https://pypi.python.org/pypi/stackfull/0.9 (Yes, I've just updated it a little bit, due to this on-going discussion) --- And no, I am not suggesting this should go into the stdlib, I am just pointing it as a helper to people who would like that right now. js -><- On 8 March 2016 at 22:55, Terry Reedy <tjreedy@udel.edu> wrote:
On 8 March 2016 at 14:17, Allan Clark <allan.clark@gmail.com> wrote:
I understand your motivation here, but Python isn't really a functional programming language and isn't intended to be. So comprehensions, while powerful, are only appropriate for relatively simple cases - at the point where they are no longer sufficiently powerful, it's generally better to use an explicit for loop or generator function. (In actual fact, I'm finding more and more these days that I only use comprehensions for very simple cases, and switch to explicit loops quite early). So what seems to me to be missing from your proposal is an explanation of how extending the comprehension syntax is an improvement over not using a comprehension at all. You suggest [y for x in numbers if abs(x) as y > 5] as a simpler alternative to [y for y in (abs(x) for x in numbers) if y > 5] but I'd be much more likely to write results = [] for x in numbers: y = abs(x) if y > 5: results.append(y) or def bounded(it, lower): for val in it: absval = abs(val) if absval > lower: yield absval list(bounded(numbers, 5)) Obviously real world examples would be better than artificial ones, as artificially simple examples make terse notation look better... And in the example using a generator, you'd be able to give it a far more meaningful name with a bit of real-life domain terminology. Paul
On 8 March 2016 at 16:02, Paul Moore <p.f.moore@gmail.com> wrote:
Yeah that's a pretty fair observation.
Again, fair. Not sure I can quite articulate why I would prefer a comprehension here. A very weak argument would be that such code tends to change, and when it does it may get morphed back into something that *can* be done with a comprehension, but you might end up leaving it as a for-loop with append.
Yeah agreed. I would note that in a real-world example you are giving a name to something that is forced to calculate *and* filter. So your name is going to end-up being something like, say "is_registered_and_voting_for", or "is_highest_tax_bracket_and_is_taxed". Which you might not actually write, and instead opt for something like "tax_amount". Even in your code for my artificial example your generator is named "bounded" but that does not sound like it is doing any filtering at all, it sounds like it is simply bounding all values. Of course to be fair, I didn't give a name for that at all, and probably I want to give a name for the resulting list. Although of course we both need to do that, but at least with a comprehension you only have to come up with a name for the result, not for the result and the associated generator.
On 8 March 2016 at 16:20, Allan Clark <allan.clark@gmail.com> wrote:
That's actually a very good point - there's quite a distinct "break point" where you have to rewrite code as a loop rather than a comprehension, and for maintainability purposes that switch tends to be permanent. So delaying the point where you need to switch (assuming the comprehension form is readable and maintainable) is a fair goal. There's also the point that much code is actually one-off scripts, and the break point can be very different in such code - "how I think of it" has a much higher weight in such a situation (often even greater than "is it maintainable" for completely throwaway code). The problem then becomes one of finding a syntax that is natural and not forced. That's hard, particularly with things like the high bar on introducing new keywords meaning you're trying to reuse words that "sort of" suit the situation.
Again, a good point. Compound clauses make for bad names typically, so if you're thinking in terms of "calculate and filter" it'll be hard to think of a really good name. Maybe a better approach is to look at chaining of individual building blocks. The [y for y in (abs(x) for x in numbers) if y > 5] approach takes that form, but the nesting hides the fact. In some code I wrote today, I used the form data = [abs(x) for x in numbers] data = [x for x in data if x > 5] You can even use generators data = (abs(x) for x in numbers) data = (x for x in data if x > 5) list(data) to make the calculations lazy. To me, that makes the step by step "calculate, then filter" pipeline explicit, and is actually really readable. Of course, once again this is the sort of thing that's very much about personal opinion, and you may hate that style (particularly using the generic variable name "data" over and over). So, overall I'd say I'm not against the idea, but there are some reasonably good alternatives already available, and so coming up with something that's compellingly better than the status quo is going to be a hard job. I'm glad we're having the discussion, though - we need to (re-) explore questions like this to avoid the language stagnating. And who knows when the inspiration will strike? :-) Paul
On Wed, Mar 9, 2016 at 7:06 AM, Pavol Lisy <pavol.lisy@gmail.com> wrote:
Enforcing that the order of operations is as you're implying: [y+z for x in numbers if (abs(x) as y) > 5 or (x**2 as z) > 100] If the first condition is true, the second will not be evaluated. It'll be equivalent to: def <listcomp>(): result = [] for x in numbers: y = abs(x) if y > 5: result.append(y+z) else: z = x**2 if z > 100: result.append(y+z) return result So, yeah, that could surprise a *lot* of people. Either an UnboundLocalError, or retaining the value from the previous iteration. Recommendation: If you use name bindings in the second half, use bitwise operators instead: [y+z for x in numbers if (abs(x) as y) > 5 | (x**2 as z) > 100] That'll force both sides to be evaluated. (Downside: It's reliable only if you actually have True and False.) ChrisA
On Tue, Mar 08, 2016 at 02:17:23PM +0000, Allan Clark wrote:
Seeing the many replies, I'm a bit lost as where to best comment. After thinking about it for a while, I think that currently it's not impossible to do what you want with comprehensions, just a bit convoluted. [y for x in numbers for y in [abs(x)] if y > 5] The tricky part being the `for y in [abs(x)]` basically doing what you want: bind the value `abs(x)` to a name (`y`). Benefits: - It requires no new syntax, no new semantics. - You can easily define multiple items at once. [y + z for x in numbers for y, z in [(abs(x), sgn(x)] if y > 5] - You can still use the original value, in contrast to [y for y in (abs(x) for x in numbers) if y > 5] Downside: - Very unreadable, and probably a long way off from being idiomatic Python. As for yet another syntax suggestion (if we want to introduce something). [y for x in numbers with abs(x) as y if y > 5] The benefit is that it reads quite natural: "with expr as name". Another benefit is that keywords get reused. However, that already has semantics outside a comprehension for context-managers. TL;DR: It's possible with [y for x in numbers for y in [abs(x)] if y > 5] But the syntax is ugly enough that it does warrant some extra syntactic sugar (or something with the same semantics but better performance characteristics).
On 8 March 2016 at 22:21, Sjoerd Job Postmus <sjoerdjob@sjec.nl> wrote:
Agreed, this is a plausible suggestion. But rather than just providing an example usage, it would be helpful to see the full syntax of comprehension with the proposed addition. See https://docs.python.org/3/reference/expressions.html#grammar-token-comprehen... - note that a comprehension can have arbitrarily many for and if clauses, interspersed in any order as long as the first one is a "for". I'm guessing you'd add "with <some_sort_of_expression> as <target_list>" as simply a third option. But what would the semantics be? I'm guessing that "with xxx as yyy" translates basically as a statement "yyy = xxx" in the notional expansion described in that section ("considering each of the for or if clauses a block...") Assuming that is the proposed definition, a few questions arise: 1. Is the behaviour this would assign to "with <foo> as x with <bar> as x" (i.e., repeated bindings of the same name) what we'd want? Is it likely to cause confusion in practice? 2. Do we need to restrict the <target_list>? Consider that "with something as global_var[0]" is allowed by the definition - that would leak values out of the comprehension. This isn't new - the "for" variable works like this already - but is it something that's more likely to be abused than currently? Should we care? (Python doesn't tend to prohibit people from writing bad code).
However, that already has semantics outside a comprehension for context-managers.
This is indeed a concern, albeit not a huge one - the definition has to point out that in the expansion "with" should be read as an assignment (written backwards) not as a with statement. It's not an impossible burden, but it is mildly inconsistent. This is probably the syntax I prefer of the ones suggested so far. But I still haven't seen any *really* convincing arguments that we need anything new in the first place. Paul
On Tue, Mar 08, 2016 at 10:58:16PM +0000, Paul Moore wrote:
It took me a while to figure this out (battling with the compiler over this), but I settled on the following grammar. comprehension ::= expression comp_for comp_for ::= "for" target_list "in" or_test [comp_iter] comp_iter ::= comp_for | comp_if | comp_with comp_if ::= "if" expression_nocond [comp_iter] comp_with ::= "with" or_test "as" target (For comp_with you probably want target_list instead of a single target. However, I felt like double-checking that I got my actual assumptions right by first implementing it (I've got it working now), and seeing as I'm not that experienced with modifying the Python parse/ast/symtable/compile phases, I decided to cop out and take the easy route. It should be fairly trivial to extend it to a target_list instead of a target.)
Trying to word it in such a way: "... considering each of the `for` or `if` clauses a block, nesting from left to right, and evaluating the expression to produce an element each time the innermost block is reached. The `with expr as target` should be considered equivalent to `target = expr`. (And here we already see the downside of this idea). Normally a comprehension of the form (expr1 for target1 in expr2 for target2 in expr3 if expr4) boils down to for target1 in expr2: for target2 in expr3: if expr4: yield expr1 The natural extension would be for (expr1 for target1 in expr2 with expr3 as target2 if expr4) to reduce as follows. for target1 in expr2: with expr3 as target2: if expr4: yield expr1 Instead, it becomes for target1 in expr2: target2 = expr3: if expr4: yield expr1 But of course we're not going to have context managers in comprehensions, are we? So this inconsistency is somewhat forgiveable.
You mean similar to >>> [x for x in range(5) for x in range(x)] [0, 0, 1, 0, 1, 2, 0, 1, 2, 3] I've never been confused by that in practice, as most people use good names for stuff. So I would see no ultra-important reason to prevent blocking it for this case and not in others.
I'd suggest not to bother with resolving naming conflicts or assignments to things other than names. [__ for foo.bar in range(10)] is already valid Python. No need to limit new features in ways which their 'friends' are not.
I agree on not having seen a really convincing argument yet. Especially since `for target in [expr]` works as well as `with expr as target`. The one thing I can think of is common subexpression elimination. [cheap_process(expensive_thingy(x)) for x in items if expensive_thingy(x) > 0] or [cheap_process(y) for x in items with expensive_thingy(x) as y if y > 0] But in Python3, you could just as well write [cheap_process(x) for x in map(expensive_thingy, items) if x > 0]
Paul
2016-03-09 6:16 GMT+01:00, Sjoerd Job Postmus <sjoerdjob@sjec.nl>: [...]
If we want variable assignment we have already "for var in [expr]" syntax. We could discuss if "with expr as var" syntax is more beautiful. (or if it not against There should be one-- and preferably only one --obvious way to do it.) But why omit context manager semantics in "with expr as var" assignment syntax? I personally don't like idea that semantics could be context sensitive in this way. (and we could already do pretty complex things in comprehension)
Some time ago I was trying to solve the same issue but using a new keyword "where", and I thought that new keyword is too much for just list comprehension filtering, so I've made it something like assignments in expresion, eg. (x+y)**2 + (x-y)**2 where x=1, y=2 So for list comprehension I can write: [stripped for line in lines if stripped where stripped=line.strip()] or: result = map(f, objs) where f=lambda x: x.return_something() or: it = iter(lines) while len(line) > 4 where line=next(it, '').strip(): print(line) or: lambda x, y: ( 0 if z == 0 else 1 if z > 0 else -1) where z = x + y or even: lambda something: d where (d, _)=something, d['a']=1 I even implemented it: https://github.com/thektulu/cpython/commit/9e669d63d292a639eb6ba2ecea3ed2c0c... and it works nicely. I was thinking to reuse "with [expr] as [var]" but I also don't like idea of context sensitive semantics, and I even thought that maybe someone, someday would want to write "content = fp.read() with open('foo.txt') as fp"... The "where" keyword is from guards pattern in Haskell :) 2016-03-10 17:53 GMT+01:00 Pavol Lisy <pavol.lisy@gmail.com>:
On Thu, Mar 10, 2016 at 5:20 PM, Michał Żukowski <thektulu.pp@gmail.com> wrote:
This is my understanding of `with Expr() as Name: Block()`, in pseudocode (ignoring exception handling). _context = Expr() Name = _context.__enter__(...) exec(Block, {"Name": Name}) _context.__exit__(...) Strawman proposal: Extend `with` to allow non-context managers. _context = Expr() if hasattr(_context, '__enter__'): # Context manager. Name = _context.__enter__(...) exec(Block, {"Name": Name}) _context.__exit__(...) else: # Normal object: treat as name bind Name = _context exec(Block, {"Name": Name}) Then you're allowed to write with 5 as x: print(x*2) And if your comprehension `with` receives context managers, they will be treated as such. # Closes each file after reading the second byte. second_bytes = [f.read(1) for fname in fnames with open(fname) as f if f.read(1) != '\0'] # Expanded to a loop. second_bytes = [] for fname in fnames: with open(fname) as f: if f.read(1) != '\0': second_bytes.append(f.read(1)) This breaks TOOWTDI by giving another way to assign. But it might not violate the spirit: I think very few people would trade a simple assignment for a backward assignment and an extra indentation level (!), especially newbies from other languages. TOOWTDI doesn't want competing ways of doing things, and this way isn't competitive. But that means it's a feature not intended for use, and only added for consistency with a new feature. The comprehension `with` will be perfectly aligned and translates naturally to the current `with`. It introduces an inconsistency between context managers and other objects, though. Any code that used to work will still work, but now any code that accidentally uses a non-context manager will not fail fast. It might also make the real use of normal `with` harder to learn. You have a gotcha for people trying to process and store a list of context managers. (Are locks used like this?) I believe that use will be both rare and advanced, so the gotcha is worse than one that is common and beginner-level.
On 11 March 2016 at 12:58, Franklin? Lee <leewangzhong+python@gmail.com> wrote:
with doesn't create a new scope - it's more akin to a try/finally statement with a particular form (PEP 343 defines the full expansion, and that expansion is still broadly accurate, although some the specific details are different these days) The only compound statements in Python that create new scopes are "def" and "class". It's also worth reading PEP 343 for Guido's explanation for why the context management assignment syntax isn't "with VAR = EXPR": it's because EXPR *isn't* the thing being assigned, but rather a stepping stone towards obtaining that thing. This characteristic is true of all of the places where "as" is currently used instead of "=": with CM_EXPR as VAR: # VAR = CM_EXPR.__enter__(), not CM_EXPR except EXC_TYPE_EXPR as VAR: # VAR = the caught exception, not the exception type constraint # There's an implied "del VAR" at the end of the suite, but still not a full implicit scope import MODULE_REFERENCE as VAR from MODULE_REFERENCE import NAME as VAR # Here, the LHS isn't even a normal expression - it's a module name, or a module name plus an attribute name Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mar 10, 2016 10:51 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
On 11 March 2016 at 12:58, Franklin? Lee <leewangzhong+python@gmail.com>
wrote: thought that
Sorry, that was just sloppiness with `exec`. I wanted to convey the block using the new `Name`, and "function call" popped up first. I erased a comment about how, after running the block, `Name` would still hold the value it received. I forgot to mention that people might choose the proposed `with` for assignment over `=` if they wrongly thought that the `with` form deleted `Name` afterward.
Then would it be a problem if it defaulted to the value of EXPR directly when that value isn't of a specially-handled type (such as a context manager, for `with`)? Other than that learning `with` in comprehensions might make it harder to learn context managers.
2016-03-11 6:38 GMT+01:00, Franklin? Lee <leewangzhong+python@gmail.com>:
On Mar 10, 2016 10:51 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:
[...]
Or we could think about syntax "with VAR = EXPR" where it is plain assignment. If we wanted in PEP343 (and still we want) to have different syntax for different semantics then it is better, I think. PS. from my point of view I still don't see why is "with" so much better than existing syntax "for VAR in [EXPR]" for example why is this not enough? -> import unicodedata [(c, i, cat, name, norm) for i in range(1, 0x110000) for c in [chr(i)] for cat in [unicodedata.category(chr(i))] for name in [unicodedata.name(chr(i), 'noname')] for uname in [name.upper()] # name is defined in previous row for norm in [unicodedata.normalize('NFKC', chr(i))] if 'OPERATOR' in uname if norm != c ] PPS. if we want to change syntax the "for VAR = EXPR" is also possibility
On Fri, Mar 11, 2016 at 2:42 PM, João Bernardo <jbvsmo@gmail.com> wrote:
On Thu, Mar 10, 2016 at 7:20 PM, Michał Żukowski <thektulu.pp@gmail.com> wrote:
[...]
One cool thing about this 'where' thing is that, if it made a new scope on every iteration, you could do [lambda x: x * j for i in range(10) where j = i] and it would actually create ten *different* functions (as opposed to without the 'where j = i'). - Koos
On Thu, Mar 10, 2016 at 11:20:21PM +0100, Michał Żukowski wrote:
But in Haskell, the `where` keyword also considers scoping. That is, outside the statement/expression with the `where`, you can't access the variables introduced by the where. Even though the `where` looks kind-of-nice, it (at least to me) is also a bit confusing with respect to evaluation order. Consider [ stripped for idx, line in enumerate(lines) if idx >= 5 or stripped where stripped=line.strip() ] (intended semantics: give me all lines (stripped), but ignore any lines that are whitespace-only in the first 5 lines) retval = [] for idx, line in enumerate(lines): stripped = line.strip() if idx >= 5 or stripped: retval.append(stripped) now I'm not very sure, but I expect what actually happens is: retval = [] for idx, line in enumerate(lines): if idx < 5: stripped = line.strip() if idx >= 5 or stripped: retval.append(stripped) that is, should I read it as (if idx >= 5 or stripped) where stripped=line.strip() or if idx >= 5 or (stripped where stripped=line.strip()) For comprehensions, I'd think the 'let' statement might make more sense. Abusing Haskell's notation: [ stripped | (idx, line) <- zip [0..] lines, let stripped = strip line, idx >= 5 || length stripped > 0 ] Porting this to something Python-ish, it'd be [ stripped for idx, line in enumerate(lines) let stripped = line.strip() if idx >= 5 or stripped ] where `let` is a keyword (possibly only applicable in a compexpr). In Haskell it's a keyword everywhere, but it has somewhat different semantics.
Yes, but I wanted it to be simple and powerful, not necessarily right in every context. Even though the `where` looks kind-of-nice, it (at least to me) is also
I've implemented it as "or_test [where_expr]" so the default order is the same as in: (if idx >= 5 or stripped) where stripped=line.strip() I wanted it to always "scope" left as much as possible - in precedence order between "lambda" and "... if ... else ..." - but left recursion forced me to place it after "... if ... else ..." which can't be used without brackets in list comprehension "if" filtering. But one can always control order with brackets, and make it work like in second example. For comprehensions, I'd think the 'let' statement might make more sense.
I was thinking about "let", but as I said, I think that new keyword should be more powerful than just filtering in list comprehensions, and in regular expresions it would look like this: if let x=foo() in x: print(x) Which does not look great, and there is problem with "in" keyword that make this expresion ambiguous.
On Mar 11, 2016, at 07:29, Michał Żukowski <thektulu.pp@gmail.com> wrote:
This makes where look like another top-level comprehension clause like for and if, but it doesn't follow the same nesting rules as the other clauses. Which means that whenever something isn't trivial enough to just read holistically, trying to apply the dead-simple rule of "write each clause as a nested statement, in the same order" will just lead to confusion, instead of immediately telling you the interpretation. Where that line is depends on how experienced you are with Python, how much time you've spent recently in a language with similar but not identical comprehension syntax, how tired you are, etc., but that doesn't matter--any comprehension with a where clause has at least three clauses, and is likely to be complicated enough to be over that line for some people. So, if you're going to recommend that this is only used when it's simple enough that it doesn't matter that the translation is confusing, that would mean recommending never using it, which implies that we shouldn't have it. A new clause that nests in order, like some of your other variations, doesn't have this problem.
I think you're overcomplicating this. You want let to be a statement, but also have a value--but you only need one or the other. As a statement, there's no need for a value; you're just creating or rebinding a variable for use in the controlled suite: let x = foo(): if x: print(x) And in a comprehension, it's just another clause that converts to the identical compound statement and nests the same as other clauses: [stripped for line in file let stripped=line.strip() if stripped] Hopefully you don't want scoping, but if you do, it's pretty obvious that the scope is the controlled suite; still no need for an in clause. If you really want a let expression instead of a statement, you still don't need an in clause unless you want scoping. Just make it an assignment expression whose value is the assigned value, just like in C and friends: if let x = foo(): print(x) Which extends easily to: if (let x = foo()) > 5: print(x) [stripped for line in file if let stripped=line.strip()] The only reason you'd need an in clause is if you wanted scoped expressions. But those would be almost useless in Python because, by design, you can't do too much inside an expression. But if you really want that, the design is again obvious: just as in functional languages, it's equivalent to a lambda call: let x=foo() in (print(x) if x > 5 else None) (lambda x: print(x) if x > 5 else None)(foo()) ... which is obviously unpythonic enough and useless enough that I think scoped let expressions are immediately dismissible as an idea.
Let statement is pointless - all things you can achievie by simply assign variables. Let expresion in form "let x=1" cannot "return" value when you want to define more than one variable - with "in" keyword you can define (like in Haskell) more than one variable, and then "return" a value, or you would need to assume that the last defined variable is returned, which is not intuitive :) 11-03-2016 21:25, "Andrew Barnert" <abarnert@yahoo.com> napisał(a):
every context. the same as in: people. So, if you're going to recommend that this is only used when it's simple enough that it doesn't matter that the translation is confusing, that would mean recommending never using it, which implies that we shouldn't have it.
A new clause that nests in order, like some of your other variations,
doesn't have this problem.
I wanted it to always "scope" left as much as possible - in precedence
Porting this to something Python-ish, it'd be
[ stripped for idx, line in enumerate(lines) let stripped =
order between "lambda" and "... if ... else ..." - but left recursion forced me to place it after "... if ... else ..." which can't be used without brackets in list comprehension "if" filtering. line, idx >= 5 || length stripped > 0 ] line.strip() if idx >= 5 or stripped ] the scope is the controlled suite; still no need for an in clause.
If you really want a let expression instead of a statement, you still
don't need an in clause unless you want scoping. Just make it an assignment expression whose value is the assigned value, just like in C and friends:
expressions. But those would be almost useless in Python because, by design, you can't do too much inside an expression. But if you really want that, the design is again obvious: just as in functional languages, it's equivalent to a lambda call:
scoped let expressions are immediately dismissible as an idea.
On Mar 11, 2016, at 15:59, Michał Żukowski <thektulu.pp@gmail.com> wrote:
Let statement is pointless - all things you can achievie by simply assign variables.
Except that a let statement starts with a keyword, meaning it can be turned into a comprehension clause, which is the entire point of this thread, and of the (unattributed) message you were replying to. Look at the example you replied to:
[ stripped for idx, line in enumerate(lines) let stripped = line.strip() if idx >= 5 or stripped ]
That's using let as a clause that maps to a statement with the exact same semantics as an assignment statement, and it nests the same way as other comprehension clauses. That's exactly what people are looking for here. Anything you add on top of that, unless it has some independent rationale, is just complication for no benefit.
Let expresion in form "let x=1" cannot "return" value when you want to define more than one variable- with "in" keyword you can define (like in Haskell) more than one variable, and then "return" a value, or you would need to assume that the last defined variable is returned, which is not intuitive :)
Assignments in Python always have a single value. In "x = y = 3", the value is 3. In "x, y = 3, 4" the value is (3, 4). So the value of "let x = y = 3" is 3, and the value of "let x, y = 3, 4" is (3, 4). There's no need for any extra assumptions. The reason Haskell needs a more structured let is scoping (and the fact that Haskell doesn't do sequencing).
I agree, the second suggestion was also my first though when I read the initial post. Seems very natural to me. If a syntax extension is not desired, the first suggestion looks like a cookbook example on how to do this to me. Could use tuple instead of list. +1 On 9 March 2016 at 09:21, Sjoerd Job Postmus <sjoerdjob@sjec.nl> wrote:
On 3/8/2016 9:17 AM, Allan Clark wrote:
-1 Comprehensions abbreviate a particular pattern of collection initialization, nested for and if statements, and collection augmentation innermost, with the only binding being the loop names. They are easily translated back to the original pattern, although move than one level of nesting can challenge comprehension in the normal meaning of the work. I strongly feel that the current correspondence between conprehensions and statements should be maintained. -- Terry Jan Reedy
So, as for the "post filter" - one can currently just express that in Python as: [y for y in (abs(x) for x in numbers) if y > 5] It may have one "internal generator" - but save for the little performance saving, I don't see this as less readable than the proposals above. As for the reuse of a value that might come in an expensive computation, that is something I always missed in Python - so, a couple years ago I put together a small package that can do just that: duplicate a computed value in a seamlesway, without requiring an explicit variable assignment. To do that, I use the mechanics of stack-based languages such as Postscript and Forth - as of now, one can do: ` from stackfull import push, popclear, clear [popclear() for x in numbers if push(expensive_calculation(x)) > 5] ` The functions pergorm stack operation in a list that is created in a transparent way on the current Python execution frame local variables dictionary. Works fine in Python, Python2 and pypy. https://pypi.python.org/pypi/stackfull/0.9 (Yes, I've just updated it a little bit, due to this on-going discussion) --- And no, I am not suggesting this should go into the stdlib, I am just pointing it as a helper to people who would like that right now. js -><- On 8 March 2016 at 22:55, Terry Reedy <tjreedy@udel.edu> wrote:
participants (14)
-
Alexander Heger -
Allan Clark -
Andrew Barnert -
Chris Angelico -
Franklin? Lee -
Joao S. O. Bueno -
João Bernardo -
Koos Zevenhoven -
Michał Żukowski -
Nick Coghlan -
Paul Moore -
Pavol Lisy -
Sjoerd Job Postmus -
Terry Reedy