[Python-ideas] Explicit variable capture list

[Andrew Barnert]

On Jan 21, 2016, at 00:48, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> Hi,
> 
> Sorry but I'm lost in this long thread. 


I think the whole issue of const optimization is taking this discussion way off track, so let me try to summarize the actual issue.

What the thread is ultimately looking for is a solution to the "closures capturing loop variables" problem. This problem has been in the official programming FAQ[1] for decades, as "Why do lambdas defined in a loop with different values all return the same result"?

    powers = [lambda x: x**i for i in range(10)]

This gives you ten functions that all return x**9, which is probably not what you wanted.

The reason this is a problem is that Python uses "late binding", which in this context means that each of those functions is a closure that captures the variable i in a way that looks up the value of i at call time. All ten functions capture the same variable, and when you later call them, that variable's value is 9.

Almost every language with real closures and for-each loops has the same problem, but someone who's coming to Python as a first language, or coming from a language like C that doesn't have those features, is almost guaranteed to be confused by this when he first sees it. (Presumably, that's why it's in the FAQ.)

The OP proposed that we should add some syntax, borrowed from C++, to function definitions that specifies that some things get captured by value. You could instead describe this as early binding the specified names, or as not capturing at all, but however you describe it, the idea is pretty simple. The obvious way to implement it is to copy the values into the function object at function-creation time, then copy them into locals at call time--exactly like default parameter values. (Not too surprising, because default parameter values are the idiomatic workaround today.)

A few alternatives to the parameter-like syntax borrowed from C++ were proposed, including "def power(x; i):" (notice the semicolon) and "def power(x)(i):". A few people also proposed a new declaration statement similar to "global" and "nonlocal"--which opens the question of what to call it; suggested names included "shared", "sharedlocal", and "capture".

People also suggested an optimization: store them like constants, instead of like default values, so they don't need to be copied into locals. (This is similar to the global->const optimizations being discussed in the FAT threads, but here it's optimizing the equivalent of default parameter values, not globals. Which means it's much less important of an optimization, since defaults are only fetched once per call, after which they're looked up the same as locals, which are just as fast as consts. It _could_ potentially feed into further FAT-type optimizations, but that's getting pretty speculative.) The obvious downside here is that constants are stored in the code object, so instead of 10 (small) function objects all sharing the same (big) code object, you'd have 10 function objects with 10 separate (big) code objects

Another alternative, which I don't think anyone seriously considered, is to flag the specified freevars so that, at function creation time, we copy the cell and bind that copy, instead of binding the original cell. (This alternative can't really be called "early binding" or "capture by value", but it has the same net effect.)

Finally, Terry suggested a completely different solution to the problem: don't change closures; change for loops. Make them create a new variable each time through the loop, instead of reusing the same variable. When the variable isn't captured, this would make no difference, but when it is, closures from different iterations would capture different variables (and therefore different cells). For backward-compatibility reasons, this might have to be optional, which means new syntax; he proposed "for new i in range(10):".

I don't know of any languages that use the C++-style solution that don't have lvalues to worry about. It's actually necessary for other reasons in C++ (capturing a variable doesn't extend its lifetime, so you need to be able to explicitly copy things or you end up with dangling references), but those reasons don't apply to Python (or C#, Swift, JavaScript, etc.). Still, it is a well-known solution to the problem.

Terry's solution, on the other hand, is used by Swift (from the start, even though it _does_ have lvalues), C# (since 5.0), and Ruby (since 1.9), among other languages. C#, in particular, decided to add it as a breaking change to a mature language, rather than adding new syntax, because Eric Lippert believed that almost any code that's relying on the old behavior is probably a bug rather than intentional.


  [1]: https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result


> Do you want to extend the
> Python language to declare constant in a function? Maybe I'm completly
> off-topic, sorry.
> 
> 
> 2016-01-21 1:10 GMT+01:00 Steven D'Aprano <steve at pearwood.info>:
>> (2) If we limit this to only capturing the same name, then we can only
>> write (say) "static x", and that does look like a declaration. But maybe
>> we want to allow the local name to differ from the global name:
>> 
>>    static x = y
> 
> 3 months ago, Serhiy Storchaka proposed a "const var = expr" syntax:
> https://mail.python.org/pipermail/python-ideas/2015-October/037028.html
> 
> With a shortcut "const len" which is like "const len = len".
> 
> In the meanwhile, I implemented an optimization in my FAT Python
> project: "Copy builtins to constant". It's quite simple: replace the
> "LOAD_GLOBAL builtin" instruction with a "LOAD_CONST builtin"
> transation and "patch" co_consts constants of a code object at
> runtime:
> 
>   def hello(): print("hello world")
> 
> is replaced with:
> 
>   def hello(): "LOAD_GLOBAL print"("hello world")
>   hello.__code__ = fat.replace_consts(hello.__code__, {'LOAD_GLOBAL
> print': print})
> 
> Where fat.replace_consts() is an helper to create a new code object
> replacing constants with the specified mapping:
> http://fatoptimizer.readthedocs.org/en/latest/fat.html#replace_consts
> 
> Replacing print(...) with "LOAD_GLOBAL"(...) is done in the
> fatoptimizer (an AST optimpizer):
> http://fatoptimizer.readthedocs.org/en/latest/optimizations.html#copy-builtin-functions-to-constants
> 
> We have to inject the builtin function at runtime. It cannot be done
> when the code object is created by "def ..." because a code object can
> only contain objects serializable by marshal (to be able to compile a
> .py file to a .pyc file).
> 
> 
>> I acknowledge that this goes beyond what the OP asked for, and I think
>> that YAGNI is a reasonable response to the static block idea. I'm not
>> going to champion it any further unless there's a bunch of interest from
>> others. (I'm saving my energy for Eiffel-like require/ensure blocks
>> *wink*).
> 
> The difference between "def hello(print=print): ..." and Serhiy's
> const idea (or my optimization) is that "def hello(print=print): ..."
> changes the signature of the function which can be a serious issue in
> an API.
> 
> Note: The other optimization "local_print = print" in the function is
> only useful for loops (when the builtin is loaded multiple times) and
> it still loads the builtin once per function call, whereas my
> optimization uses a constant and so no lookup is required anymore.
> 
> Then guards are used to disable the optimization if builtins are
> modified. See the PEP 510 for an explanation on that part.
> 
> Victor
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20160122/e8438862/attachment-0001.html>