[Python-ideas] Explicit variable capture list

[Andrew Barnert]

On Jan 22, 2016, at 21:06, Chris Angelico <rosuav at gmail.com> wrote:
> 
> On Sat, Jan 23, 2016 at 3:50 PM, Andrew Barnert via Python-ideas
> <python-ideas at python.org> wrote:
>> Finally, Terry suggested a completely different solution to the problem:
>> don't change closures; change for loops. Make them create a new variable
>> each time through the loop, instead of reusing the same variable. When the
>> variable isn't captured, this would make no difference, but when it is,
>> closures from different iterations would capture different variables (and
>> therefore different cells). For backward-compatibility reasons, this might
>> have to be optional, which means new syntax; he proposed "for new i in
>> range(10):".
> 
> Not just for backward compatibility. Python's scoping and assignment
> rules are currently very straight-forward: assignment creates a local
> name unless told otherwise by a global/nonlocal declaration, and *all*
> name binding follows the same rules as assignment. Off the top of my
> head, I can think of two special cases, neither of which is truly a
> change to the binding semantics: "except X as Y:" triggers an
> unbinding at the end of the block, and comprehensions have a hidden
> function boundary that means their iteration variables are more local
> than you might think. Making for loops behave differently by default
> would be a stark break from that tidiness.

As a side note, notice that if you don't capture the variable, there is no observable difference (which means CPython would be well within its rights to optimize it by reusing the same variable unless it's a cellvar).

Anyway, yes, it's still something that you have to learn--but the unexpected-on-first-encounter interaction between loop variables and closures is also something that everybody has to learn. And, even after you understand it, it still doesn't become obvious until you've been bitten by it enough times (and if you're going back and forth between Python and a language that's solved the problem, one way or the other, you may keep relearning it). So, theoretically, the status quo is certainly simpler, but in practice, I'm not sure it is.

> It seems odd to change this on the loop, though. Is there any reason
> to use "for new i in range(10):" if you're not making a series of
> nested functions?

Rarely if ever. But is there any reason to "def spam(x; i):" or "def [i](x):" or whatever syntax people like if you're not overwriting i with a different and unwanted value? And is there any reason to reuse a variable you've bound in that way if a loop isn't forcing you to do so?

This problem comes up all the time, in all kinds of languages, when loops and closures intersect. It almost never comes up with loops alone or closures alone.

> Seems most logical to make this a special way of
> creating functions, not of looping.

There are also some good theoretical motivations for changing loops, but I'm really hoping someone else (maybe the Swift or C# dev team blogs) has already written it up, so I can just post a link and a short "... and here's why it also applies to Python" (complicated by the fact that one of the motivations _doesn't_ apply to Python...).

Also, the idea of a closure "capturing by value" is pretty strange on the surface; you have to think through why that doesn't just mean "not capturing" in a language like Python. Nick Coghlan suggests calling it "capture at definition" vs. "capture at call", which helps, but it's still weird. Weirder than loops creating a new binding that has the same name as the old one in a let-less language? I don't know. They're both weird. And so is the existing behavior, despite the fact that it makes perfect sense once you work it through.

Anyway, for now, I'll just repeat that Ruby, Swift, C#, etc. all solved this by changing for loops, while only C++, which already needed to change closures because of its lifetime rules, solved it by changing closures. On the other hand, JavaScript and Java both explicitly rejected any change to fix the problem, and Python has lived with it for a long time, so...