Assignments in list/generator expressions
I often have things like this:
ys = [f(x) for x in xs if f(x)]
Obviously there is a redundant function call. You could rephrase that code to the following in order to avoid it:
ys = [y for y in (f(x) for x in xs) if y]
But I think this is cumbersome and the extra generator overhead is unnecessary. So I propose this syntax:
ys = [f(x) as y for x in xs if y]
It could be transformed to this:
It's 6:18am here so I might not think straight and there is something fundamentally wrong with this. So please flame away. -panzi
Before I go to bead another thing I miss in Python. In generator expressions you can unpack elements:
ys = [f(x1) for x1, x2 in xs]
But sometimes you need not just the unpacked elements but also the original tuple:
If you need x[0] often in your expression it would be nice if you could write something like this:
ys = [f(x1, x) for (x1, x2) as x in xs]
So the "as" keyword here would be like the "@" in Haskell. You don't need repetitive __getitem__ calls and also don't need to rebuild the tuple. Of course if Python would know that the type of whatever is unpacked is a tuple (which is immutalbe) optimizations could create the same bytecode from both old variants then from the new one. Good night, panzi
On Fri, Apr 8, 2011 at 9:22 PM, Mathias Panzenböck <grosser.meister.morti@gmx.net> wrote:
This has been proposed previously: http://mail.python.org/pipermail/python-ideas/2009-June/004946.html (The syntax variant using "as" comes up part of the way through the thread.) Cheers, Chris -- http://blog.rebertia.com
Survey of list comprehension features and syntax: 1) Structure Haskell: [expr | generators], generators = generator (, generator)* Python: [expr generators], generators = generator+ 2) Take Haskell: x <- xs Python: for x in xs 3) Filter Haskell: cond Python: if cond 4) Local definition Haskell: let name = expr Python: strangely missing I'd rather add syntax equivalent to Haskell's let, than inventing completely new syntax. E.g. with name = expr. Original from the first post becomes ys = [y for x in xs with y = f(x) if y] but it's more flexible in general. Eugene
Uh, that's hardly strange considering that Python doesn't have local assignments in expressions anywhere.
Well, depending on your definition of "assignments in expression" you can say that Haskell doesn't have them either. Or you can say that Python has them since list comprehension was introduced -- for x in xs is an assignment to x. In fact, I'm reminded that local assignment in list comprehension is easily emulated with for: ys = [y for x in xs for y in [f(x)] if y] Eugene
The lack of local assignment is a feature. It keeps people from trying to write overly complex one-liners. It is better to separate each step out into a generator expression on a different line of the final program, so that things don't become so dense you can't tell at a glance what's going on. Or heck, you could just use a for-loop if what you're trying to do is complicated enough.
On Fri, Apr 8, 2011 at 9:22 PM, Mathias Panzenböck < grosser.meister.morti@gmx.net> wrote:
So I propose this syntax:
ys = *[f(x) as y for x in xs if y]*
On Sat, Apr 9, 2011 at 10:55 AM, MRAB <python@mrabarnett.plus.com> wrote:
Or possibly: ys =* [y for x in xs with f(x) as y if y]*
I find both of these hard to read, just like a sentence with multiple non-parallel dependent phrases. The meaning of y as x is reversed from x in y. As noted, there are two ways to do this that work already. I find the first one quite readable: On Fri, Apr 8, 2011 at 9:22 PM, Mathias Panzenböck < grosser.meister.morti@gmx.net> wrote:
On Sat, Apr 9, 2011 at 12:04 PM, Eugene Toder <eltoder@gmail.com> wrote:
local assignment in list comprehension is easily emulated with for:
ys = *[y for x in xs for y in [f(x)] if y]* I wonder if the "solution" to this is to publish one of these as the answer to the common question of how to do this. Timing tests indicate that the first version is faster. --- Bruce *New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April Fools! *New!** *Blog post: http://www.vroospeak.com/2011/04/march-gets-more-madness-next-year.html April Fools!
On Sun, Apr 10, 2011 at 2:27 AM, Eugene Toder <eltoder@gmail.com> wrote:
It isn't strangely missing at all, it's just hard: http://www.python.org/dev/peps/pep-3150/ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 10 Apr 2011, at 08:41, Nick Coghlan wrote:
I have just read the proposal and I would like to suggest an idea for implementing it. It may not be a good idea but at least it's simple! Given the "torture test", at the end of the proposal: b = {} a = b[f(a)] = x given: x = 42 def f(x): return x assert "x" not in locals() assert "f" not in locals() assert a == 42 assert d[42] == 42 given: d = b assert "d" not in locals()b = {} Without "given", one would write it as below. Obviously it succeeds at module, local and class scope. b = {} x = 42 def f(x): return x a = b[f(a)] = x del x del f assert "x" not in locals() assert "f" not in locals() assert a == 42 d = b assert d[42] == 42 del d assert "d" not in locals() The only (but big!) problem is that this deletes any previously defined variable "x", "f", and "d" in the current scope. So why not just rename the variables "x", "f", and "d" at compile time to something that is guaranteed not to appear elsewhere in the scope, e.g. "@1", "@2", "@3" or some other naming scheme, in the style of the old "gensym" function in Lisp? So we get: b = {} @1 = 42 def @2(x): return x a = b[@2(a)] = @1 del @1 del @2 assert "x" not in locals() assert "f" not in locals() assert a == 42 @3 = b assert @3[42] == 42 del @3 assert "d" not in locals() -- Arnaud
On Sun, Apr 10, 2011 at 8:37 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:
The only (but big!) problem is that this deletes any previously defined variable "x", "f", and "d" in the current scope. So why not just rename the variables "x", "f", and "d" at compile time to something that is guaranteed not to appear elsewhere in the scope, e.g. "@1", "@2", "@3" or some other naming scheme, in the style of the old "gensym" function in Lisp?
I considered a strategy along those lines when eliminating the leakage of the scope variables for list comprehensions in Py3k and it turns out to be rather painful from a nested scopes point of view. It's not *impossible*, of course, but you end up having to manage this parallel pseudo-namespace inside the real one and it rapidly becomes an absolute mess (as you have to implement a second copy of most of the nested scopes logic in order to handle closures correctly and you will curse the existence of the locals() function). The compiler can't see inside the strings used to look up values via "locals()", so there's no way to adjust them to handle the automatic renaming. If you "lift" a class or function out of a statement local namespace, there's also the question of what the value of the __name__ attribute ends up being. I have updated the torture test accordingly, it now includes a third example that renaming strategies will have serious problems handling: y = y given: x = 42 def f(): pass y = locals("x"), f.__name__ assert "x" not in locals() assert "f" not in locals() assert y == (42, "f") Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 10 Apr 2011, at 13:10, Nick Coghlan wrote:
OK. It's clear I don't know enough about the compiling process to be able to understand all the problems. The f.__name__ issue doesn't seem insurmountable, but I can't see how to get around the locals()["x"] issue in a sane manner (I see what you mean about cursing the locals() function!). BTW, I think there is a typo in the example above: shouldn't y = locals("x"), ... be y = locals()["x"], ... -- Arnaud
On Mon, Apr 11, 2011 at 8:48 AM, Guido van Rossum <guido@python.org> wrote:
While supporting locals() correctly is a bit of a pain, I find "Does read access to locals() still work?" to be a useful proxy for "Will proposing this feature have authors of Python debugging tools massing outside my front door with pitchforks and torches?" :) I don't think it's a trade-off that has to be made in the specific case of PEP 3150 - the copy-in/copy-out idea should be viable and reasonably intuitive given Python's existing scoping rules. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Apr 10, 2011 at 7:33 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Maybe, but debugger may be given access to things that regular code does not, and if it has to work a little harder to get at the values of local variables that is okay in my book. So I'd rather see the actual question asked ("how can a debugger make sense of this") than a proxy for that question. (And sure, if it's easy to make it work with locals(), I am not objecting. :-) -- --Guido van Rossum (python.org/~guido)
It isn't strangely missing at all, it's just hard: http://www.python.org/dev/peps/pep-3150/
Local definition in list comprehension is significantly simpler that 'given'. Semantically, 'with name = expr' can be treated as a more readable form of 'for name in [expr]'. Implementation can use a simple assignment. Eugene
On Sun, Apr 10, 2011 at 11:31 PM, Eugene Toder <eltoder@gmail.com> wrote:
Part of the point of PEP 3150 is that such requests don't *stay* simple. If one subexpression, why not two? If in a list comprehension, why not in a conditional expression? The "you can't do arbitrary embedded assignments in expressions" rule is at least a simple one, even if some folks don't like it (most likely due to a mathematics/functional programming background, where the idea of persistent state isn't a basic assumption the way it is in normal imperative programming). While naming the iteration variables is a fundamental part of writing comprehensions, naming other subexpressions is not, so it doesn't make sense to provide comprehension specific syntax for it. If you want to name subexpressions, the "one obvious way" is to use an explicit loop, typically as part of a generator: def transform(xs): """Meaningful description of whatever the transform does""" for x in xs: y = f(x) if y: yield y ys = transform(xs) The problem of "I want to use this value as part of a conditional expression and in its own right" actually arises in more locations than just filtering comprehensions - it also comes up for it statements, while loops and conditional expressions. In all cases, you run up against the limits of what expressions allow and have to devote a bunch of vertical whitespace to switch to using expressions instead. The issue you're really up against is the fact that Guido made a choice 20 years ago to enforce a statement/expression dichotomy and to keep name binding entirely the purview of statements. Comprehension iteration variables are an exception that were added later on, but they're used in a very constrained way that matches the embedded assignment of a specific statement type (i.e. the header line on for loops). Embedded assignment proposals for conditions suffer badly from the fact that there is no statement level equivalent for the kind of assignment they propose. The obvious solution (allowing the conditional to be named) is too limiting, but solutions that are sufficiently flexible to be worthwhile end up allowing naming of arbitrary subexpressions, effectively throwing away a fundamental design principle of the language. And so the discussion dies, until the next time someone decides that their personal use case for embedded assignment is worth altering the language to allow. Mathias's proposal in this case is a slight improvement over past suggestions, since it allows the conditional expression to differ from the value used in the rest of the expression. However, I'm not sure it would be feasible to implement it in a way that constrained it to comprehensions only. Even if that was possible, fixing only 1 out of the 4 parts of the language where the problem arises isn't much of a solution from a broader point of view. And it still only works in simple cases, being utterly unhelpful for cases like: {f(k):g(v) for k, v in d.items() if f(k) and g(v)} As past discussions have shown, the only thing really powerful enough to cover an acceptably large number of use cases is full-blown embedded assignment for arbitrary expressions: ys = [y for x in xs if not (f(x) as y)] {k2:v2 for k, v in d.items() if (f(k) as k2) and (g(v) as v2)} That's an awfully big sledgehammer for a rather small nail :P Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, 2011-04-11 at 01:18 +1000, Nick Coghlan wrote:
I don't quite understand the obsession of having an entire function on one line. In programming, one line is usually associated with one action. Python is all about readability, and it's probably best to put this kind of stuff in a function on multiple lines, not only will it be easier to read, it'll be reusable and easier to modify.
On Sun, Apr 10, 2011 at 6:31 AM, Eugene Toder <eltoder@gmail.com> wrote:
Eww. Since we already have "with expr as name" I don't think we should add "with name = expr", even if it's in a different context. OTOH I am personally a big fan of PEP 3150; now that the moratorium is lifted I hope that someone will attempt a proper implementation of it so we can find out all the odd corner cases (which I'm sure will be plentiful given that this is quite a novel thing for Python). BTW also +1 on Laura's mini-rant on readability. A lot of folks have ideas that would add a minor shortcut to the language, without making a big difference in true expressivity, but with a big potential cost in readability. (Remember Perl, the ultimate write-only language.) In defense of 'given' I think that it adds something truly new (variables that go out of scope before the function containing them returns) in a very readable way. And the "definition follows use" thing has worked out pretty well in list comprehensions and generator expressions, if you ask me. -- --Guido van Rossum (python.org/~guido)
On Mon, Apr 11, 2011 at 3:42 AM, Guido van Rossum <guido@python.org> wrote:
Not *quite* as novel as one might first think - the specific proposal I currently have in there has its roots in the way list/set/dict comprehensions work under the hood (in CPython, anyway). So we know the core principle is sound, although I'm sure you're correct that there are more corner cases still to be found in generalising the comprehension-style copy-in/copy-out semantics appropriately. I also thought of a somewhat decent use case for the feature, too: "builder" code in module and class namespaces, where you need some temporary functions, variables and loops to create a named variable, but then have to make sure to clean up your temporary storage locations to avoid polluting the public namespace. In the past, I'd mostly been thinking in terms of function namespaces, and keeping those "clean" is nowhere near as important as the situation for classes and modules (it's quite likely someone else has brought this up in the past, . To lift some examples from https://bitbucket.org/pypy/pypy/src/default/lib_pypy/disassembler.py (which is what got me thinking along these lines) from opcode import __all__ as _opcodes_all __all__ = ["dis","disassemble","distb","disco"] + _opcodes_all del _opcodes_all Could become: __all__ = ["dis","disassemble","distb","disco"] + opcode.__all__ given: import opcode And: def _setup(): for opcode in opname: if not opcode.startswith('<'): class O(Opcode): pass opcode = opcode.replace('+', '_') O.__name__ = opcode globals()[opcode] = O _setup() Could become: pass given: # I'll add "pass" to the list of supported statements in PEP 3150 for opcode in opname: if not opcode.startswith('<'): class O(Opcode): pass opcode = opcode.replace('+', '_') O.__name__ = opcode globals()[opcode] = O There's also a performance hack in there: "given:" drops you down into a function scope, so you get the benefits of function local performance. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Apr 10, 2011 at 9:20 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I like this one, but I'd like to see some more local examples too.
Can't say I like this one. "pass given" sounds icky (and I think earlier in this thread someone flagged it as undesirable). I think that "given" will particularly shine in code written in a "top-down" programming style, where one first presents the high-level outcome and pushes details to the back. This can currently be done quite well by putting the "main()" function definition first and then developing it from there, but there is a certain elegance in having the helper functions not exposed at the module level. (Although I can also see that people who really like this style will overuse it and put everything in a big sprawling deeply-nested structure, as opposed to making the helpers all siblings. And it won't increase testability.) -- --Guido van Rossum (python.org/~guido)
On Tue, Apr 12, 2011 at 12:57 AM, Guido van Rossum <guido@python.org> wrote:
It isn't on the list currently in the PEP, as the only ones I included there were the simple statements that permitted the use of subexpressions. However, if "pass" is left *off* the list, then the moral equivalent of "pass given:" could still be written in a variety of other ways, such as: 0 given: # Somewhat counterintuitive due to "if 0:"! pass 1 given: pass "pass" given: pass ... given: pass I figure I may as well bite the bullet and include "pass" as an obvious way of doing inline code in a private scope.
Yeah, being able to reach in and explicitly test "_" prefixed helpers is very handy in writing good white box test suites. My own feelings on PEP 3150 still sit around the level of finding it interesting and potentially valuable as a concept (otherwise I wouldn't have written the PEP in the first place!), but I'm not yet convinced that its net impact on the language would be positive. My main stumbling block is the fact I still can't characterise suitably obvious criteria as to when it should be used over normal imperative code. I'd be a lot happier with the PEP if I could include a specific proposal for PEP 8 additions regarding its applicability. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
I also thought of a somewhat decent use case for the feature, too: "builder" code in module and class namespaces,
It would also give you a really nice way to define properties: foo = property(get, set) given: def get(self): ... def set(self, x): ... and to pass thunks to functions: do_something(blarg, body) given: def body(stuff): ... -- Greg
On Sat, Apr 9, 2011 at 1:22 AM, Mathias Panzenböck <grosser.meister.morti@gmx.net> wrote:
ys = [f(x) as y for x in xs if y] I supose the above example could be expressed like this:
ys = filter(None, (f(x) for x in xs)) (or ys = list( filter(None, (f(x) for x in xs))) if you happen to really need a list ) In nearly any python version. And in doing so, I join the chorus of -1 for such assignment possibilities. And I am one that likes doing big things in one liners from time to time, just for the fun of it - while in-generator assignment would turn these one liners much more flexible, it would, as several others have pointed, imply in the possibility of having rather unreadable code inside programs. Regards, js -><-
-panzi
Before I go to bead another thing I miss in Python. In generator expressions you can unpack elements:
ys = [f(x1) for x1, x2 in xs]
But sometimes you need not just the unpacked elements but also the original tuple:
If you need x[0] often in your expression it would be nice if you could write something like this:
ys = [f(x1, x) for (x1, x2) as x in xs]
So the "as" keyword here would be like the "@" in Haskell. You don't need repetitive __getitem__ calls and also don't need to rebuild the tuple. Of course if Python would know that the type of whatever is unpacked is a tuple (which is immutalbe) optimizations could create the same bytecode from both old variants then from the new one. Good night, panzi
On Fri, Apr 8, 2011 at 9:22 PM, Mathias Panzenböck <grosser.meister.morti@gmx.net> wrote:
This has been proposed previously: http://mail.python.org/pipermail/python-ideas/2009-June/004946.html (The syntax variant using "as" comes up part of the way through the thread.) Cheers, Chris -- http://blog.rebertia.com
Survey of list comprehension features and syntax: 1) Structure Haskell: [expr | generators], generators = generator (, generator)* Python: [expr generators], generators = generator+ 2) Take Haskell: x <- xs Python: for x in xs 3) Filter Haskell: cond Python: if cond 4) Local definition Haskell: let name = expr Python: strangely missing I'd rather add syntax equivalent to Haskell's let, than inventing completely new syntax. E.g. with name = expr. Original from the first post becomes ys = [y for x in xs with y = f(x) if y] but it's more flexible in general. Eugene
Uh, that's hardly strange considering that Python doesn't have local assignments in expressions anywhere.
Well, depending on your definition of "assignments in expression" you can say that Haskell doesn't have them either. Or you can say that Python has them since list comprehension was introduced -- for x in xs is an assignment to x. In fact, I'm reminded that local assignment in list comprehension is easily emulated with for: ys = [y for x in xs for y in [f(x)] if y] Eugene
The lack of local assignment is a feature. It keeps people from trying to write overly complex one-liners. It is better to separate each step out into a generator expression on a different line of the final program, so that things don't become so dense you can't tell at a glance what's going on. Or heck, you could just use a for-loop if what you're trying to do is complicated enough.
On Fri, Apr 8, 2011 at 9:22 PM, Mathias Panzenböck < grosser.meister.morti@gmx.net> wrote:
So I propose this syntax:
ys = *[f(x) as y for x in xs if y]*
On Sat, Apr 9, 2011 at 10:55 AM, MRAB <python@mrabarnett.plus.com> wrote:
Or possibly: ys =* [y for x in xs with f(x) as y if y]*
I find both of these hard to read, just like a sentence with multiple non-parallel dependent phrases. The meaning of y as x is reversed from x in y. As noted, there are two ways to do this that work already. I find the first one quite readable: On Fri, Apr 8, 2011 at 9:22 PM, Mathias Panzenböck < grosser.meister.morti@gmx.net> wrote:
On Sat, Apr 9, 2011 at 12:04 PM, Eugene Toder <eltoder@gmail.com> wrote:
local assignment in list comprehension is easily emulated with for:
ys = *[y for x in xs for y in [f(x)] if y]* I wonder if the "solution" to this is to publish one of these as the answer to the common question of how to do this. Timing tests indicate that the first version is faster. --- Bruce *New! *Puzzazz newsletter: http://j.mp/puzzazz-news-2011-04 including April Fools! *New!** *Blog post: http://www.vroospeak.com/2011/04/march-gets-more-madness-next-year.html April Fools!
On Sun, Apr 10, 2011 at 2:27 AM, Eugene Toder <eltoder@gmail.com> wrote:
It isn't strangely missing at all, it's just hard: http://www.python.org/dev/peps/pep-3150/ Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 10 Apr 2011, at 08:41, Nick Coghlan wrote:
I have just read the proposal and I would like to suggest an idea for implementing it. It may not be a good idea but at least it's simple! Given the "torture test", at the end of the proposal: b = {} a = b[f(a)] = x given: x = 42 def f(x): return x assert "x" not in locals() assert "f" not in locals() assert a == 42 assert d[42] == 42 given: d = b assert "d" not in locals()b = {} Without "given", one would write it as below. Obviously it succeeds at module, local and class scope. b = {} x = 42 def f(x): return x a = b[f(a)] = x del x del f assert "x" not in locals() assert "f" not in locals() assert a == 42 d = b assert d[42] == 42 del d assert "d" not in locals() The only (but big!) problem is that this deletes any previously defined variable "x", "f", and "d" in the current scope. So why not just rename the variables "x", "f", and "d" at compile time to something that is guaranteed not to appear elsewhere in the scope, e.g. "@1", "@2", "@3" or some other naming scheme, in the style of the old "gensym" function in Lisp? So we get: b = {} @1 = 42 def @2(x): return x a = b[@2(a)] = @1 del @1 del @2 assert "x" not in locals() assert "f" not in locals() assert a == 42 @3 = b assert @3[42] == 42 del @3 assert "d" not in locals() -- Arnaud
On Sun, Apr 10, 2011 at 8:37 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:
The only (but big!) problem is that this deletes any previously defined variable "x", "f", and "d" in the current scope. So why not just rename the variables "x", "f", and "d" at compile time to something that is guaranteed not to appear elsewhere in the scope, e.g. "@1", "@2", "@3" or some other naming scheme, in the style of the old "gensym" function in Lisp?
I considered a strategy along those lines when eliminating the leakage of the scope variables for list comprehensions in Py3k and it turns out to be rather painful from a nested scopes point of view. It's not *impossible*, of course, but you end up having to manage this parallel pseudo-namespace inside the real one and it rapidly becomes an absolute mess (as you have to implement a second copy of most of the nested scopes logic in order to handle closures correctly and you will curse the existence of the locals() function). The compiler can't see inside the strings used to look up values via "locals()", so there's no way to adjust them to handle the automatic renaming. If you "lift" a class or function out of a statement local namespace, there's also the question of what the value of the __name__ attribute ends up being. I have updated the torture test accordingly, it now includes a third example that renaming strategies will have serious problems handling: y = y given: x = 42 def f(): pass y = locals("x"), f.__name__ assert "x" not in locals() assert "f" not in locals() assert y == (42, "f") Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 10 Apr 2011, at 13:10, Nick Coghlan wrote:
OK. It's clear I don't know enough about the compiling process to be able to understand all the problems. The f.__name__ issue doesn't seem insurmountable, but I can't see how to get around the locals()["x"] issue in a sane manner (I see what you mean about cursing the locals() function!). BTW, I think there is a typo in the example above: shouldn't y = locals("x"), ... be y = locals()["x"], ... -- Arnaud
On Mon, Apr 11, 2011 at 8:48 AM, Guido van Rossum <guido@python.org> wrote:
While supporting locals() correctly is a bit of a pain, I find "Does read access to locals() still work?" to be a useful proxy for "Will proposing this feature have authors of Python debugging tools massing outside my front door with pitchforks and torches?" :) I don't think it's a trade-off that has to be made in the specific case of PEP 3150 - the copy-in/copy-out idea should be viable and reasonably intuitive given Python's existing scoping rules. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Apr 10, 2011 at 7:33 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Maybe, but debugger may be given access to things that regular code does not, and if it has to work a little harder to get at the values of local variables that is okay in my book. So I'd rather see the actual question asked ("how can a debugger make sense of this") than a proxy for that question. (And sure, if it's easy to make it work with locals(), I am not objecting. :-) -- --Guido van Rossum (python.org/~guido)
It isn't strangely missing at all, it's just hard: http://www.python.org/dev/peps/pep-3150/
Local definition in list comprehension is significantly simpler that 'given'. Semantically, 'with name = expr' can be treated as a more readable form of 'for name in [expr]'. Implementation can use a simple assignment. Eugene
On Sun, Apr 10, 2011 at 11:31 PM, Eugene Toder <eltoder@gmail.com> wrote:
Part of the point of PEP 3150 is that such requests don't *stay* simple. If one subexpression, why not two? If in a list comprehension, why not in a conditional expression? The "you can't do arbitrary embedded assignments in expressions" rule is at least a simple one, even if some folks don't like it (most likely due to a mathematics/functional programming background, where the idea of persistent state isn't a basic assumption the way it is in normal imperative programming). While naming the iteration variables is a fundamental part of writing comprehensions, naming other subexpressions is not, so it doesn't make sense to provide comprehension specific syntax for it. If you want to name subexpressions, the "one obvious way" is to use an explicit loop, typically as part of a generator: def transform(xs): """Meaningful description of whatever the transform does""" for x in xs: y = f(x) if y: yield y ys = transform(xs) The problem of "I want to use this value as part of a conditional expression and in its own right" actually arises in more locations than just filtering comprehensions - it also comes up for it statements, while loops and conditional expressions. In all cases, you run up against the limits of what expressions allow and have to devote a bunch of vertical whitespace to switch to using expressions instead. The issue you're really up against is the fact that Guido made a choice 20 years ago to enforce a statement/expression dichotomy and to keep name binding entirely the purview of statements. Comprehension iteration variables are an exception that were added later on, but they're used in a very constrained way that matches the embedded assignment of a specific statement type (i.e. the header line on for loops). Embedded assignment proposals for conditions suffer badly from the fact that there is no statement level equivalent for the kind of assignment they propose. The obvious solution (allowing the conditional to be named) is too limiting, but solutions that are sufficiently flexible to be worthwhile end up allowing naming of arbitrary subexpressions, effectively throwing away a fundamental design principle of the language. And so the discussion dies, until the next time someone decides that their personal use case for embedded assignment is worth altering the language to allow. Mathias's proposal in this case is a slight improvement over past suggestions, since it allows the conditional expression to differ from the value used in the rest of the expression. However, I'm not sure it would be feasible to implement it in a way that constrained it to comprehensions only. Even if that was possible, fixing only 1 out of the 4 parts of the language where the problem arises isn't much of a solution from a broader point of view. And it still only works in simple cases, being utterly unhelpful for cases like: {f(k):g(v) for k, v in d.items() if f(k) and g(v)} As past discussions have shown, the only thing really powerful enough to cover an acceptably large number of use cases is full-blown embedded assignment for arbitrary expressions: ys = [y for x in xs if not (f(x) as y)] {k2:v2 for k, v in d.items() if (f(k) as k2) and (g(v) as v2)} That's an awfully big sledgehammer for a rather small nail :P Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, 2011-04-11 at 01:18 +1000, Nick Coghlan wrote:
I don't quite understand the obsession of having an entire function on one line. In programming, one line is usually associated with one action. Python is all about readability, and it's probably best to put this kind of stuff in a function on multiple lines, not only will it be easier to read, it'll be reusable and easier to modify.
On Sun, Apr 10, 2011 at 6:31 AM, Eugene Toder <eltoder@gmail.com> wrote:
Eww. Since we already have "with expr as name" I don't think we should add "with name = expr", even if it's in a different context. OTOH I am personally a big fan of PEP 3150; now that the moratorium is lifted I hope that someone will attempt a proper implementation of it so we can find out all the odd corner cases (which I'm sure will be plentiful given that this is quite a novel thing for Python). BTW also +1 on Laura's mini-rant on readability. A lot of folks have ideas that would add a minor shortcut to the language, without making a big difference in true expressivity, but with a big potential cost in readability. (Remember Perl, the ultimate write-only language.) In defense of 'given' I think that it adds something truly new (variables that go out of scope before the function containing them returns) in a very readable way. And the "definition follows use" thing has worked out pretty well in list comprehensions and generator expressions, if you ask me. -- --Guido van Rossum (python.org/~guido)
On Mon, Apr 11, 2011 at 3:42 AM, Guido van Rossum <guido@python.org> wrote:
Not *quite* as novel as one might first think - the specific proposal I currently have in there has its roots in the way list/set/dict comprehensions work under the hood (in CPython, anyway). So we know the core principle is sound, although I'm sure you're correct that there are more corner cases still to be found in generalising the comprehension-style copy-in/copy-out semantics appropriately. I also thought of a somewhat decent use case for the feature, too: "builder" code in module and class namespaces, where you need some temporary functions, variables and loops to create a named variable, but then have to make sure to clean up your temporary storage locations to avoid polluting the public namespace. In the past, I'd mostly been thinking in terms of function namespaces, and keeping those "clean" is nowhere near as important as the situation for classes and modules (it's quite likely someone else has brought this up in the past, . To lift some examples from https://bitbucket.org/pypy/pypy/src/default/lib_pypy/disassembler.py (which is what got me thinking along these lines) from opcode import __all__ as _opcodes_all __all__ = ["dis","disassemble","distb","disco"] + _opcodes_all del _opcodes_all Could become: __all__ = ["dis","disassemble","distb","disco"] + opcode.__all__ given: import opcode And: def _setup(): for opcode in opname: if not opcode.startswith('<'): class O(Opcode): pass opcode = opcode.replace('+', '_') O.__name__ = opcode globals()[opcode] = O _setup() Could become: pass given: # I'll add "pass" to the list of supported statements in PEP 3150 for opcode in opname: if not opcode.startswith('<'): class O(Opcode): pass opcode = opcode.replace('+', '_') O.__name__ = opcode globals()[opcode] = O There's also a performance hack in there: "given:" drops you down into a function scope, so you get the benefits of function local performance. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Apr 10, 2011 at 9:20 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I like this one, but I'd like to see some more local examples too.
Can't say I like this one. "pass given" sounds icky (and I think earlier in this thread someone flagged it as undesirable). I think that "given" will particularly shine in code written in a "top-down" programming style, where one first presents the high-level outcome and pushes details to the back. This can currently be done quite well by putting the "main()" function definition first and then developing it from there, but there is a certain elegance in having the helper functions not exposed at the module level. (Although I can also see that people who really like this style will overuse it and put everything in a big sprawling deeply-nested structure, as opposed to making the helpers all siblings. And it won't increase testability.) -- --Guido van Rossum (python.org/~guido)
On Tue, Apr 12, 2011 at 12:57 AM, Guido van Rossum <guido@python.org> wrote:
It isn't on the list currently in the PEP, as the only ones I included there were the simple statements that permitted the use of subexpressions. However, if "pass" is left *off* the list, then the moral equivalent of "pass given:" could still be written in a variety of other ways, such as: 0 given: # Somewhat counterintuitive due to "if 0:"! pass 1 given: pass "pass" given: pass ... given: pass I figure I may as well bite the bullet and include "pass" as an obvious way of doing inline code in a private scope.
Yeah, being able to reach in and explicitly test "_" prefixed helpers is very handy in writing good white box test suites. My own feelings on PEP 3150 still sit around the level of finding it interesting and potentially valuable as a concept (otherwise I wouldn't have written the PEP in the first place!), but I'm not yet convinced that its net impact on the language would be positive. My main stumbling block is the fact I still can't characterise suitably obvious criteria as to when it should be used over normal imperative code. I'd be a lot happier with the PEP if I could include a specific proposal for PEP 8 additions regarding its applicability. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
I also thought of a somewhat decent use case for the feature, too: "builder" code in module and class namespaces,
It would also give you a really nice way to define properties: foo = property(get, set) given: def get(self): ... def set(self, x): ... and to pass thunks to functions: do_something(blarg, body) given: def body(stuff): ... -- Greg
On Sat, Apr 9, 2011 at 1:22 AM, Mathias Panzenböck <grosser.meister.morti@gmx.net> wrote:
ys = [f(x) as y for x in xs if y] I supose the above example could be expressed like this:
ys = filter(None, (f(x) for x in xs)) (or ys = list( filter(None, (f(x) for x in xs))) if you happen to really need a list ) In nearly any python version. And in doing so, I join the chorus of -1 for such assignment possibilities. And I am one that likes doing big things in one liners from time to time, just for the fun of it - while in-generator assignment would turn these one liners much more flexible, it would, as several others have pointed, imply in the possibility of having rather unreadable code inside programs. Regards, js -><-
-panzi
participants (13)
-
Arnaud Delobelle -
Bruce Leban -
Carl M. Johnson -
Chris Rebert -
Eugene Toder -
Georg Brandl -
Greg Ewing -
Guido van Rossum -
Joao S. O. Bueno -
Mathias Panzenböck -
MRAB -
Nick Coghlan -
Westley Martínez