[Python-ideas] Allowing breaks in generator expressions by overloading the while keyword

[Andrew Barnert]

From: Nick Coghlan <ncoghlan at gmail.com>

Sent: Friday, February 21, 2014 6:17 AM


> The reason I keep beating my head against this particular wall (cf.
> PEP 403's @in clauses and PEP 3150's given suites) is that my personal
> goal for Python is that it should be a tool that lets people express
> what they are thinking clearly and relatively concisely.
> 
> As far as I have been able to tell, the persistent requests for
> "multi-line lambdas", cleaner lambda syntax, etc, are because Python
> doesn't currently make it easy to express a lot of operations that
> involve higher order manipulation of "one shot" callables  - closures
> or custom functions where you *don't* want to re-use them, but Python
> still forces you to pull them out and name them. I think PEP 403 is my
> best current description of the mental speed bump involved:
> http://www.python.org/dev/peps/pep-0403/


There are actually two different—in fact, nearly opposite—problems that get lumped together. The first problem is that you often want to use _expressions_ as values, and you can't. The second problem is that you sometimes want to build complicated functions in-line. Solving that does nothing to help the first problem (as JavaScript proves), and I'm not sure it's really as big a problem as people think (there's a reason experienced JavaScript programmers start using Deferreds/Promises—borrowed from Python—to escape the "callback hell" their language allows), but I won't get into that here. The fact that both of these manifest as "problems with lambda" is really just a coincidence—the first is a problem with lambda because the only way we have to simulate passing around expressions is by wrapping them in functions; the second is a problem with lambda because without statements inside expressions it's the only way to define a function inline.

Anyway, the first problem is why we have comprehensions, and why people want more out of them. What can comprehensions do that map and filter can't? Maybe they're a little faster, maybe they let you write long sequences of map and filter in order instead of mutated into prefix order, but nobody cares about either of those enough to add new syntax. The real benefit is that they let you map an expression over an iterable, or filter an iterable with an expression, without wrapping it upon a function. Compare:

    [x**2 for x in seq]

    map(lambda x: x**2, seq)

The second one forces the reader to think through an extra abstraction that isn't relevant to the problem.

And people want the same thing more generally. Most novices don't think of callbacks as higher-order functions, which is why they write this:

    b = Button("button 10", callback=onclick(10))



… instead of one of these:

    b = Button("button 10", callback=lambda: onclick(10))

    b = Button("button 10", callback=partial(onlick, 10))



But really, why should they have to? All they want to say is "when I click the button, evaluate the expression onclick(10)". Why is that so hard?

It's not really about improving lambda, but about removing the need for it. Surely this isn't any easier to explain:

    b = Button("button 10", callback=:onclick(10))



Even if it isn't a complete solution, there is of course an obvious practical benefit in readability. If you see a chain of widget declarations, and they all end with something like "callback=:onclick(10)", you only really have to process that ":" once for the whole chain, and then it fades into the background as an unimportant detail in understanding the real point of the code. So I love the idea.

And, so far, Python has been adding practical partial solutions. As you (Nick) pointed out, comprehensions, decorators, with statements, conditional and except expressions, etc. are all ways to provide new contexts where expressions can be used without wrapping them in functions. There are probably a few others worth covering, and at that point maybe we'll have handled 90% of the cases people need, and they can deal with the last 10%. While that may sound clunky, I'm not sure it's really a problem. Python isn't a perfect language, but it's a very good language, and that's why we all use it.

Of course if there's a general solution, that would be great. But I'm not sure there is. Lisp-style quoting and macros, or Haskell-style higher-order-functioning, _might_ be general solutions—but even if they are, I'm not sure they're solutions that fit into Python. (Still, for the first two, everyone really should play with MacroPy and get a feel for what it can do and how Pythonic it can feel…)

I wrote this up in more detail, with a lot more digressions, as a blog post, but blogger.com seems to be having problems, so the world is spared my rambling. :)