Tweaking closures and lexical scoping to include the function being defined

The problem of the 'default argument hack' and it's use for early binding and shared state in function definitions is one that has been bugging me for years. You could rightly say that the degree to which it irritates me is all out of proportion to the significance of the use case and frequency with which it arises, and I'd agree with you. That, at least in part, is what has made it such an interesting problem for me: the status quo is suboptimal and confusing (there's a reason the term 'default argument hack' gets thrown around, including by me), but most proposed solutions have involved fairly significant changes to the semantics and syntax of the language that cannot be justified by such a niche use case. The proposed cures (including my own suggestions) have all been worse than the disease. I finally have a possible answer I actually *like* ("nonlocal VAR from EXPR"), but it involves a somewhat novel way of thinking about closures and lexical scopes for it to make sense. This post is an attempt to explain that thought process. The history outlined here will be familiar to many folks on the list, but I hope to make the case that this approach actually simplifies and unifies a few aspects of the language rather than adding anything fundamentally new. The novel aspect lies in recognising and exposing to developers as a coherent feature something that is already implicit in the operation of the language as a whole. == Default Arguments == Default arguments have been a part of Python function definitions for a very long time (since the beginning, even?), so it makes sense to start with those. At function call time, if the relevant parameters are not supplied as arguments, they're populated on the frame object based on the values stored on the function object. Their behaviour is actually quite like a closure: they define shared state that is common to all invocations of the function. == Lexical Scoping == The second step in this journey is the original introduction of lexical scoping by PEP 227 back in Python 2.1 (or 2.2 without a __future__ statement). This changed Python from its original locals->globals->builtins lookup mechanism (still used in class scope to this day), to the closure semantics for nested functions that we're familiar with. However, at this stage, there was no ability to rebind names in outer scopes - they were read-only, so you needed to use other techniques (like 'boxing' in a list) to update immutable values. == Writing to Outer Scopes == PEP 3104 added the ability to write to outer scopes by using the 'nonlocal' statement to declare that a particular variable was not a local in the current frame, but rather a local in an outer frame which is alive at the time the inner function definition statement is executed. It expects the variable to already exist in an outer lexically nested scope and complains if it can't find one. == The "__class__" cell reference == The final entrant in this game, the "__class__" cell reference was added to the language by PEP 335 in order to implement the 3.x super() shorthand. For functions defined within a class body, this effectively lets the class definition play a role in lexical scoping, as the compiler and eval loop cooperate to give the function an indirect reference to the class being defined, even though the function definition completes first. == The Status Quo == If you go look up the definition of 'closure', you'll find that it doesn't actually say anything about nested functions. Instead, it will talk about 'free variables' in the algorithm definition without placing any restrictions on how those variables are later hooked up to the appropriate values. In current Python, ordinary named references can refer to one of 4 namespaces: - locals (stored on the currently executing frame object) - closure reference (stored in a cell object by the function that defined it, kept alive after the frame is recycled by references from still living inner functions that need it) - globals (stored on the module object) - builtins (also stored on a module object, specifically the one for the builtin namespace) PEP 335 also creates a closure reference for "__class__", but in a slightly unusual way. Whereas most targets for closure references are created by the code in the outer function when it runs [1], this closure reference is populated implicitly by the type machinery [2]. The important aspect from my point of view is that this implementation technique starts to break down Python's historical correlation between "function closure" and "lexically nested scope". == Conceptual Unification == The moment of clarity for me came when I realised that default arguments, lexically nested scopes and the new super() implementation can all be seen as just special cases of the broader concept of free variables and function closures. Lexically nested scopes have always been talked about in those terms, so that aspect shouldn't surprise anyone. The new super() implementation is also fairly obviously a closure, since it uses the closure machinery to work its magic. The only difference is in the way the value gets populated in the first place (i.e. by the type machinery rather than by the execution of an outer function). Due to history, default argument *values* aren't often thought of as closure references, but they really are anonymous closures. Instead of using cells, the references are stored in dedicated attributes that are known to the argument parsing machinery, but you could quite easily dispense with that and store everything as cells in the function closure (you wouldn't, since it would be a waste of time and energy, I'm just pointing out the conceptual equivalence. A *new* Python implementation, though, could choose to go down that path). After I had that realisation, the natural follow-up question seemed to be: if I wanted to explicitly declare a closure variable, and provide it with an initial value, without introducing a nested function purely for that purpose, how should I spell that? Well, I think PEP 3104 has already given us the answer: by declaring the variable name as explicitly 'nonlocal', but also providing an initial value so the compiler knows it is a *new* closure variable, rather than one from an outer lexically nested scope. This is a far more useful and meaningful addition than the trivial syntactic sugar mentioned in the PEP (but ultimately not implemented). The other question is what scope the initialisation operation should be executed in, and I think there, default arguments have the answer: in the containing scope, before the function has been defined. == Precise Syntax == By reusing 'nonlocal', we would make it clear that we're not adding a new concept to the language, but rather generalising an existing one (i.e. closure references) to provide additional flexibility in the way they're used. So I *really* want to use that keyword rather than adding a new one just for this task. However, I'm less certain about the spelling of the rest of the statement. There are at least a few possible alternative spellings: nonlocal VAR = EXPR # My initial suggestion nonlocal VAR from EXPR # Strongly indicates there's more than a simple assignment going on here nonlocal EXPR as VAR # Parser may struggle with this one Of the three, 'nonlocal VAR from EXPR' may be the best bet - it's easy for the compiler to parse, PEP 380 set the precedent for the 'from EXPR' clause to introduce a subexpression and 'nonlocal VAR = EXPR' may be too close to 'nonlocal VAR; VAR = EXPR'. Regards, Nick. [1] Some dis module details regarding the different kinds of name reference. Most notable for my point is the correspondence between the 'cell variable' in the outer function and the 'free variable' in the inner function:
[2] Some dis module output to show that there's no corresponding '__class__' cell variable anywhere when the implicit closure entry is created by the new super() machinery.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
I think you're mixing up concepts and implementation here a bit. Cells are an implementation detail. What's important is which *namespace* the name is referring to: - A 'global' declaration makes it refer to the module-level namespace. - A 'nonlocal' declaration makes it refer to some namespace in between local and module-level (there may be more than one of these, so I wouldn't say that there are only 4 namespaces). Now, while the *value* of your proposed new kind of variable would be stored as part of the function's closure, its *name* would be part of the function's *local* namespace. Consider this: i = 88 def f(): nonlocal i = 17 print i def g(): nonlocal i = 42 print i f() g() print i I'm assuming you intend that the two i's here would have nothing to do with each other, or with the i in the enclosing scope, so that the output from this would be 17 42 88 rather than 42 42 42 So, your proposed use of 'nonlocal' would actually be declaring a name to be *local*. That strikes me as weird and perverse. NOBODY-expects-the-spanish-nonlocal-declaration!-ly, Greg

On 26 September 2011 07:56, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
So, your proposed use of 'nonlocal' would actually be declaring a name to be *local*. That strikes me as weird and perverse.
Aha! That's precisely the concern I had with the suggestion of "nonlocal" for this, although I had failed to understand *why* it bothered me, and so hadn't commented... Paul

On Mon, Sep 26, 2011 at 2:56 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Ah, but it *wouldn't* be local, that's the point - it would be stored on the function rather than on the frame, and hence be shared across invocations. Change your example function to this so it actually modifies the name binding, and it becomes clear that this is *not* a local variable declaration: i = 88 def f(): nonlocal i from 17 print(i) i += 1
It would work exactly as if we had introduced a containing scope as a closure: def outer(): i = 17 def f(): nonlocal i print(i) i += 1 return f
The *only* thing that would change is the way the closure reference would be initialised - it would happen as part of the function definition (just like default arguments) rather than needing the developer to write the outer scope explicitly just to initialise the closure references. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 26 September 2011 12:34, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hmm, its lifetime is non-local, but the visibility is still local. My instincts associate the word "(non-)local" with visibility rather than lifetime. If you want a bikeshed to colour in, maybe "persistent" is a better keyword for this: def counter(): persistent n as 1 print(n) n += 1 Paul.

On Mon, Sep 26, 2011 at 7:45 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Adding new keywords is a big, big step that demands a compelling justification. Now that I have come up with a way to make this syntactic sugar for existing usage of nonlocal rather than a new concept, I flatly oppose introduction of a new name for something which is, at a fundamental level, just a variation on existing functionality. I'd rather continue with the status quo indefinitely if people truly find this variant intolerable. It may be easier to turn things around and specifically look at it from the "syntactic sugar" point of view: # Current syntax def _outer(): # Boilerplate VAR = EXPR def FUNC(): # Real function name is hidden nonlocal VAR # VAR repeated # Do stuff with VAR, including rebinding it return f # Boilerplate FUNC = _outer() # Boilerplate and FUNC repeated Most of that code is noise: the interesting aspects are that: 1. There is a function called FUNC() available for use 2. VAR survives across invocations of FUNC() 3. At the first invocation of FUNC(), the initial value of VAR will be EXPR So, let's offer a syntax that just includes those 3 pieces of interesting information without the boilerplate: def FUNC(): # Function of interest is not hidden in a nested scope nonlocal VAR from EXPR # Shared variable and initial value # Do stuff with VAR Is anyone going to *guess* what that means without looking it up? Probably not. But are they going to *forget* what it means once they learn it? Also probably not. "I can guess what this means without reading the docs or having someone explain it to me" is setting the bar too high for what a single keyword can possibly hope to convey. The bar needs to be whether or not it serves as a useful mnemonic to recall the functionality once a user already know what means. For me, 'nonlocal' fits that bill, especially when the feature is described as syntactic sugar for a somewhat common use case for the existing lexical scoping functionality. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 26 September 2011 14:47, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree entirely. My point here wasn't to suggest that this needs a new keyword, but rather that the proposal uses an unnatural keyword to avoid needing a new keyword. Your argument that this is a simple extension of the semantics of "nonlocal" is reasonable when viewing nonlocal in terms of lifetimes. My contention is that most people view nonlocal in terms of visibility (and in that view, the two uses of nonlocal are jarringly dissimilar).
I have no problem with the contention that this would be useful. Minor, as you concede at the start of the thread, but certainly useful. I'm certainly bikeshedding here over the name of the keyword. But I think I'm arguing that green is the wrong colour for this stop sign, because people will misinterpret it, whereas you are arguing it's a great colour because we have this tin of green paint here, and the paint shop's closed. (End of overwrought analogy :-))
But readability matters, and I worry that this isn't "readable". Maybe the fact that neither of us is Dutch is relevant here, too :-) Paul.

On Mon, Sep 26, 2011 at 10:24 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I agree it's certainly a *new* way of looking at the problem, but I don't agree that that necessarily makes it a *wrong* way to look at it. Should we invent a new keyword, or abandon the concept of (more) elegant syntax for shared internal function state, just to avoid teaching people that reference locality involves elements of both visibility *and* lifecycle? Instead, I contend that it is precisely this aspect of the language that makes mutable default arguments such a point of confusion. Since we *don't* separate out the explanations of visibility vs lifetime when discussing closure references, default arguments become a special snowflake that people need to learn about without being able to slot it into a broader framework. I believe we can actually make the concepts involved in both default argument lifecycles and the 3.x super() implementation more semantically coherent by *embracing* nonlocal as influencing both visibility *and* lifetime and clearly articulating that as a core language concept in relation to closures. We *already* have the following possibilities: local visibility, local lifetime: 'ordinary' local variable referenced solely from the executing frame local visibility, nonlocal lifetime: default argument expressions, local variable referenced from inner function that survives current invocation nonlocal visibility, nonlocal lifetime: 3.x __class__ reference, closure reference to outer lexically containing scope, global references My proposed syntax is just a way to explicitly create new entries in that middle category - variables with local visibility and nonlocal lifetime. So 'global VAR' and 'nonlocal VAR' would continue to declare that a NAME belongs in the third category and is initialised somewhere else, while 'nonlocal VAR from EXPR' would say "this has nonlocal lifetime, but local visibility, so here's the initial value since it won't be provided anywhere else". Regards, Nick. P.S. Really nice point about the visibility vs lifecycle distinction. I think it actually *strengthens* my argument about the current lack of a coherent semantic framework in this space, though. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
I don't think that's the right way to characterise default argument values. While the *values* have nonlocal lifetime, the *name* being declared is a parameter, so both its scope and lifetime are local. Your proposal would be creating a new category of name that doesn't currently exist. I don't see how your proposal would do anything to clarify the distinction between visibility and lifetime. Currently, 'nonlocal' always refers to visibility. Your way, it would sometimes mean visibility and sometimes lifetime, with only a very obtuse clue as to the difference. -- Greg

Interesting discussion! Very enlightening. But after all that I'm still unconvinced that the new concept should be tied to the nonlocal keyword. Nick's right that the *mechanism* for nonlocals and __class__ is the same, and the mechanism for the new concept might well be the same. And possibly a hypothetical other Python implementation could use the same mechanism to keep track of default argument values. Or not. But at a different level, nonlocal and the new concept are quite different, and the difference has to do with the scope of the name, not its lifetime (this was a very important insight in the discussion!). I think it is actually very telling that the syntax for nonlocal, __class__ and default argument values is so different: to most users, the lifetime is much less of an issue than the scope. Now, let's pretend the new feature is called "own". We'd write: def counter(): own x = 0 x += 1 return x If we had another function (say, defined in the same scope): def count_by_twos(): own x = 0 x += 2 return x the x in counter() and count_by_twos() are unrelated. This is different from the way nonlocal works (I don't have to type the example I hope :-). Also, nonlocal is illegal in a global function -- syntactically, it must be in a function that is nested inside another function. For all these reasons, sorry Nick, but I really don't think any syntax based on nonlocal will do it, even "nonlocal <var> from <expr>". I see two options open at this point. 1) Bikeshed until we've come up with the right keyword. I think that apart from the keyword the optimal syntax is entirely fixed; "<keyword> <variable> = <expression>" is the way to go. (I suppose we could argue about how to declare multiple variables this way.) I would be okay with using either "static" or "own" for the keyword. Personally, I don't really mind all the negative baggage that static has, but a Google code search for \bown\s= shows way fewer hits than for \bstatic\s= (both with lang:^python$ case:yes). 2) Give up, and keep using the default argument hack. You can protect your arguments from accidentally being overridden by using a "lone star" in the argument list (PEP 3102) and a __private naming convention. The lone star prevents accidental overriding when too many positional arguments are given. The __private provides sufficient protection against misguided attempts to provide a value for such "arguments", even if for functions outside a class it is purely psychological. (Note that someone who *really* wanted to mess with a cell's variable could also do it.) -- --Guido van Rossum (python.org/~guido)

On Mon, Sep 26, 2011 at 9:32 PM, Guido van Rossum <guido@python.org> wrote:
Interesting discussion! Very enlightening.
Indeed! It has certainly clarified many aspects for me.
But after all that I'm still unconvinced that the new concept should be tied to the nonlocal keyword.
<snip>
For all these reasons, sorry Nick, but I really don't think any syntax based on nonlocal will do it, even "nonlocal <var> from <expr>".
Well, the semantics of a 'nonlocal-from' initialised nonlocal variable would be different from a bare nonlocal declaration (i.e. the new form would create a new name binding only visible in local scope, so it wouldn't get shared across functions within the same outer function and is legal in top-level functions), just as they would be with a new keyword. So I'm not prepared to completely give up on this one yet - it took me years of poking at the issue to come up with the idea of reusing nonlocal as the keyword, so I hope people will take some time to consider the idea of using the term to refer to variations in both visibility and lifecycle rather than strictly limiting it to refer only to the visibility aspect.
There have certainly been plenty of proposals over the years. The ones I recall off the top of my head are: once static persistent shared closure own atdef deftime The baggage something like 'static' brings with it would be even more misleading than that associated with 'nonlocal', and the others are either seriously ugly or else aren't particularly suggestive as mnemonics. At least 'nonlocal' indicates something related to function closures is in play, even if it isn't the exact same thing as is currently meant by a bare declaration. Really, the mechanism suffers from the same problem that led to the choice of nonlocal for PEP 3104 in the first place: the important part of the declaration lies in making it clear to the compiler and the reader that the specified name is *not* a normal local variable. Exactly what it *is* instead will depend on the specifics of the code - it may be shared state, it may be something else, the only way to find out for sure is to look at the code and see how it is used. Choosing a keyword based on any given use case (such as 'shared') makes the construct seem odd when used for another use case (such as early binding). I agree the two concepts (bare nonlocal and initialised nonlocal) are not the same. However, I also believe they're close enough that a slight broadening of the definition of 'nonlocal' is preferable to trying to come up with a completely new term that is both readable, usefully mnemonic and not laden down with unrelated baggage either from Python itself or from other languages.
Despite what I wrote earlier in the thread, I don't think we actually need to declare the idea *dead* if you don't currently like 'nonlocal' as the keyword. Perhaps the 'nonlocal' idea will grow on you (it certainly did on me, the longer I considered the idea), or perhaps you or someone else will come up with an appropriate keyword that everyone considers reasonable. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Sep 26, 2011 at 7:43 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Sorry, from reading the responses, the objection that the other use of nonlocal refers to scope, not lifetime, seems pretty common, and I don't think it will go away. You might as well use "def var = expr" as the syntax and justify it by saying that it is clearly defining something and syntactically different from function definitions... OTOH I am happy to let you all bikeshed on a better name. -- --Guido van Rossum (python.org/~guido)

On 9/26/2011 9:48 PM, Guido van Rossum wrote:
I was actually considering asking whether using 'def.static' or 'def.initial' would be a possible way to avoid making a new keyword. I honestly prefer def to nonlocal for this. I like that "our" is short, but while it's less confusing, I don't see is as clear. Questions on options: Could there be an importable "function" that, when used, declares a variable to be one of these? Could the keyword be toggled in or out of namespaces based on, say, an imported module? If so, the keyword wouldn't be terribly burdensome. The following could also add to Nick's list of keyword options: init (or Initial or even initialize) retain hold held persist preserve keep kept lasting stash survive ark reshiyth Given the frequency of the use case, clarity probably trumps concerns about keyword length. To my eye, those all seem more congruous than "nonlocal." (Nick's list: nonlocal once static persistent shared closure own atdef deftime) -Nate

On Mon, Sep 26, 2011 at 8:53 PM, Spectral One <ghostwriter402@gmail.com> wrote:
"our" is something in Perl, right? My proposal was "own", which is much older (Algol-60). Probably nobody else here remembers it.
The compiler would have to recognize that function as special, which is not really possible, since the Python compiler has no access to the runtime environment, so it can't know what is imported (__future__ is the exception, and that's why it's a __xxx__ word).
Could the keyword be toggled in or out of namespaces based on, say, an imported module? If so, the keyword wouldn't be terribly burdensome.
We do that regularly with "from __future__ import <some_syntactic_feature>" and when the parser sees it, it modifies itself (slightly). Modifying the list of reserved words is easy. But it's not a long-term solution. Somehow you triggered a thought in my head: maybe se should stop focusing on the syntax for a while and focus on the code we want to be generated for it. If we can design the bytecode, perhaps that would help us come up with the right keyword.
-- --Guido van Rossum (python.org/~guido)

On 27 September 2011 05:12, Guido van Rossum <guido@python.org> wrote:
"our" is something in Perl, right? My proposal was "own", which is much older (Algol-60). Probably nobody else here remembers it.
I do :-) Not sure that Algol-60 using it is a good enough reason for recommending it, though. The name didn't make sense to me even back then... (Actually, after this discussion, the logic is clearer - so congratulations, Nick, for clarifying a 40-year old puzzle for me!) Call by name, anyone? Paul.

On Tue, Sep 27, 2011 at 12:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
In the days of Algol-60 they were pretty bad at explaining things. I was puzzled by "own" as well. (However, the biggest mystery for me, for a long time, were pointers in Pascal. It didn't clear up until I learned assembly.)
Call by name, anyone?
ABC, Python's predecessor, had it, IIRC. -- --Guido van Rossum (python.org/~guido)

With regard to "own" or "static", my feeling is that this kind of feature (i.e. mutable shared state visible only to one function) is an anti-pattern in language design. We have much better ways nowadays of sharing state without making it completely global, such as modules and classes. There was a discussion a while back about a "const" statement, which was essentially what we are talking about here except that the name would be read-only. This seems like both a better name and better semantics to me. As I remember, the idea foundered because it was thought too confusing to have an expression that was written inside the function but evaluated outside of it. If we're seriously considering the current proposal, presumably we've gotten over that? Or is it still a valid objection? -- Greg

On 27 September 2011 21:38, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote: [...]
I have been reading this thread from the start and FWIW, it's something that I have been feeling uncomfortable about. I value the guarantee that no code in the body of a def statement is executed when the def statement itself is executed. This proposal would break this guarantee. -- Arnaud

On Tue, Sep 27, 2011 at 2:04 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:
That seems a guarantee made up specifically to prevent this proposal. What benefit do you see from it? -- --Guido van Rossum (python.org/~guido)

On 27 September 2011 22:23, Guido van Rossum <guido@python.org> wrote:
I wouldn't do this :) I like new things.
What benefit do you see from it?
It keeps things simple. To be specific, when I read code I can "fold" (mentally or with the help of a code editor) the body of a def statement, knowing that I won't miss anything until the function is called. So the following won't raise: [some code] x = 42 def foo(): [I don't need to read the body] assert x == 42 But now def statements can have non-obvious side-effects. For example: def spam_x(): global x x = "spam" x = 42 def foo(): # Some body nonlocal y from spam_x() # More body assert x == 42 I may be unduly worrying about this, but it feels to me that it lessens the overall tidiness of the language. I would much rather there was a way to initialise these "own" variables outside the body of the function. -- Arnaud

On Tue, Sep 27, 2011 at 3:24 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:
Yes, you have a point. Hiding side effects this way could be nasty, and this is actually the first point in favor of the default argument hack I've heard in a while. :-) C++ has special syntax to force the programmer to write certain things after the argument list but before the function body (I think only for constructors). I have seen too many constructors with a short argument list, a very short body, and a ton of code cramped in between, to like the idea very much: even though it is semantically something totally different, syntactically it would have the same awkwardness. -- --Guido van Rossum (python.org/~guido)

On 9/27/2011 6:32 PM, Guido van Rossum wrote:
On Tue, Sep 27, 2011 at 3:24 PM, Arnaud Delobelle<arnodel@gmail.com> wrote:
I have several times written on python-list that Python has a simple rule: header expressions (for default args) are evaluated one-time when the function is defined; body expressions are evaluated (perhaps) each time when the function is called. If people understand this and do not fight it, they are not surprised that mutating a once-defined default arg mutates it, nor that defining a function inside a loop magically causes define-time binding of names in the body. I would hate for Python to lose this simplicity. I consider it one of its positive features. -- Terry Jan Reedy

Terry Reedy wrote:
Simplicity is good. I don't remember who first suggested it, and I don't remember any discussion on it, but what if decorator functions grew a __prepare__ method similar to metaclasses? If the method exists it is called /before/ the function is evaluated, it returns a namespace that is then used as the basis for the function's (the interpreter can take everything it finds in there and add it as closures to the function), and folks can call it whatever they want. ;) If the method does not exist, nothing is different from what we have now. Seeing-if-we-can-have-more-exploding-heads'ly yours, ~Ethan~

On 9/27/2011 9:26 PM, Ethan Furman wrote:
At least three people, including Guido, have noted than a keyword-only _private parameter solves most of the problems with using default args for constants. Since the _underscore convention is already partly built into the language (import *, help(module), __mangling, __reserved__), we can have other tools like signature introspection also respect the convention. This solution *also* works for the semi-private accumulator parameter of most tail-recursive functions. Example: sum the values in a linked list (without iterators or explicit loops*: def llsum(ll, *, _total = 0): if ll: return llsum(ll[1], _total = _total + ll[0]) else: return _total The private parameter _total is constant for outside calls but gets a new value with each recursive call. It should not appear in the public signature but is used internally. The non-parameter constants proposed as a substitute for default args would not work for this case. * I am QUITE aware that this can be rewritten with while, and indeed I write the two if branches in the order I do to make it easy. def llsum(ll) total = 0): while ll: ll, total = ll[1], total + ll[0] else: return total But this is a different issue. -- Terry Jan Reedy

On Wed, Sep 28, 2011 at 12:12 AM, Terry Reedy <tjreedy@udel.edu> wrote:
Note that this also matches indentation, since decorators are indented with the header, rather than with the body.
Though they may still be surprised in the opposite direction; that you have to write an explicit and redundant i=i to capture the current value when binding.
I would hate for Python to lose this simplicity.
Agreed.
This solution *also* works for the semi-private accumulator parameter of most tail-recursive functions.
Additional advantages: (a) Because it is still a parameter, it *can* be altered for test code; it is just obvious that you're doing something unsupported. (b) The only patch required is to documentation. Instead of saying "Don't do that", the documentation should say "If you just want to save state between calls, make it a keyword-only parameter and indicate that it is private by prefixing the name with an underscore." That is pretty hard to beat from a backwards-compatibility standpoint. -jJ

On Wed, Sep 28, 2011 at 5:41 PM, Jim Jewett <jimjjewett@gmail.com> wrote:
Um, but you can't save state between calls in a default argument value, except by the hack of making it a list (or some other mutable object) and mutating that. -- --Guido van Rossum (python.org/~guido)

So has Jan's post-** argument list proposal from June ( http://mail.python.org/pipermail/python-ideas/2011-June/010479.html) been definitively nixed? It seemed the perfect answer given that its similarity to the default argument hack makes it intuitive when the evaluation is taking place. On Wed, Sep 28, 2011 at 10:54 PM, Guido van Rossum <guido@python.org> wrote:

On Thu, Sep 29, 2011 at 2:27 AM, Matthew J Tretter <matthew@exanimo.com> wrote:
It has in my mind. As Guido noted, default argument *names* refer to ordinary locals, and hence exhibit the same rebinding behaviour as pre-nonlocal closure references (i.e. reassignments don't persist across calls, only mutations do). Since we expressly *don't* want that behaviour, anything directly in the argument list (even after **) would be inappropriate. I'm still mulling over the idea of '@state' either as a keyword in its own right or else as a 'known to the compiler' builtin like super so that the following would work (as several people have suggested): @state(n=0, lock=threading.Lock()) def global_counter(): n += 1 return n The special casing of super() is something that annoys me though, and making 'state' a keyword would be extraordinarily disruptive, so I'm also intrigued by the suggestion of reusing 'nonlocal' for the same purpose: @nonlocal(n=0, lock=threading.Lock()) def global_counter(): n += 1 return n With that spelling, the nonlocal statement would declare references to pre-existing closure variables in an outer scope, while the nonlocal decorator would declare new function state variables, implicitly creating the outer scope without all the boilerplate. I must admit, the 'some kind of decorator' syntax is growing on me, especially the nonlocal variant. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 9/29/2011 8:00 AM, Nick Coghlan wrote:
/'we'/'I and some of the discussants'/ It has occurred to me that one reason this threads is not getting anywhere very fast is that there are two quite different proposals being intermixed. People interested in and discussing the two different proposals have been talking past each other. One is to reproduce the behavior of defaulted parameters without having the name appear in the signature to produce *read-only* private locals, which amount to private define-time constants. This includes the int=int type speed hack. I think '*, _int=int', with adjustment of introspection, is sufficient. This make '_int' pragmatically constant, while still externally over-rideable for exceptional purposes such as testing or tracing. The other is to introduce something new: private *read-write* closure (persistent) name-value pairs. I agree that such a new thing should not be in the param list between '(' and ')'. -- Terry Jan Reedy

On Thu, Sep 29, 2011 at 3:47 PM, Terry Reedy <tjreedy@udel.edu> wrote:
Well, the references would be constant. The state itself could still be mutable, depending on the kinds of objects referred to. I agree this use case could be addressed by blessing the status quo and a couple of conventions.
It's not actually new as far as overall language capabilities go - since PEP 3104, you can get the functionality through appropriate use of an ordinary closure. It's really just about providing a slightly shorter syntax for a particular way of using closures. Well noted that not all proposals are addressing the same functionality, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Are you sure we don't? The ability to persistently re-assign the name is not needed to address the main use cases under consideration, as I understand them to be. In fact, disallowing assignment to the name at all would be fine by me. Hence I would be +0 on using 'const' as the keyword. -- Greg

On Thu, Sep 29, 2011 at 5:24 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Fixed references to mutable objects makes 'const' a rather problematic choice. Besides, if we set the bar at that level of functionality then documenting some conventions for the status quo (i.e. 'private' keyword only arguments) is an adequate response. As Terry noted, there are actually two different features being discussed in the thread, and they differ according to whether or not rebinding is supported. By casting the proposal as syntactic sugar for a particular usage of closures, I'm firmly in the camp that if we change anything here the updated syntax should support rebinding of the shared names. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Sep 29, 2011 at 5:24 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
What do you think the main use cases are? If you can't rebind, it obviously doesn't solve the Counter case. As best I can tell, const only works for speed hacks. And there are other ways to solve that, if you're willing to use a different VM, like pypy does. -jJ

Terry Reedy dixit (2011-09-27, 21:17):
No, it does not cause such a binding. That is one of the cases the proposition of this or that early-binding syntax comes back repeatedly: def strangish_factorission(): chairs = [] for spam in (1,2,3,4,5): def comfy_chair(fmt): # we naively try to make use of early binding here # but there is no such binding here result = fmt % spam return result chairs.append(comfy_chair) return chairs for comfy_chair in strangish_factorission(): print comfy_chair('%d'), -- will print "5 5 5 5 5", not "1 2 3 4 5". To obtain the latter you need to use eigher default argument hack (which is ugly and unsafe in some situations) or a closure (which for now needs another nested scope, which is even worse in terms of readability). Cheers, *j

On 9/28/2011 5:57 PM, Jan Kaliszewski wrote:
Terry Reedy dixit (2011-09-27, 21:17):
defining a function inside a loop
Insert 'does not', which somehow got omitted or deleted.
magically causes define-time binding of names in the body.
No, it does not cause such a binding.
Of course not, as I have said many times over the last decade plus, most recently just 4 hours earlier (at 17:10), when I said "People are assuming [wrongly, when using a local name that matches an outer enclosing loop name] that 'i' is immediately bound to its 'current' value, just like default args." Sorry for the confusing omission. My intention was to list this as a delusion, not as a fact. -- Terry Jan Reedy

I never liked the nontestability of nested functions, and I also don't like the nontestability of these non-global semiglobalnonlocal variables. Real globals have the benefit that a test suite can check what happens to that variable as functions are called fairly easily. Here you have to test against my_func.__closure__[i].cell_contents for some i whose value depends on the order in which you created the variables and the order in which they're added to existing nonlocals (provided that isn't an implementation detail).
+1. Yes.
There is. Have this "nonlocal assignment" as a magical decorator, or as syntax as part of the def statement. I'm not sure to what degree Nick's proposal is about the nonlocal syntax versus the nonlocal semantics. But you can keep the semantics (fast access to variables that are shared as global state) while changing the syntax fairly radically. Devin On Tue, Sep 27, 2011 at 6:24 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:

On Tue, Sep 27, 2011 at 4:38 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
In explaining to someone why I think this issue of function state variables is worth exploring, I described it as being important for the same reason as closures are important. That's also why I've been describing it as syntactic sugar for a particular *kind* of closure usage. The reason I think closures themselves are important is that gets back into the 'algorithm with state' vs 'data with behaviour' contrasting views of a problem. Sometimes one view is more productive, sometimes the other, and there's blurry ground in the middle where either can work. As programmers, our best bet is to learn both modelling tools and then judge which is most appropriate for the problem at hand. One thing I definitely got out of this thread is that I think we need better tools for introspecting closure and generator internal state, so I'd be interested in seeing proposals for additions to the inspect module on that front. However, getting back to the original topic, the @nonlocal suggestion is definitely growing on me, since it seems to combine all the elements of the current equivalent that I'd like to shorten into one concise syntax: def _outer(): n = 0 lock = threading.Lock() def global_counter(): nonlocal n with lock: n += 1 return n return global_counter global_counter = _outer() would become: @nonlocal(n=0, lock=threading.Lock()) def global_counter(): with lock: n += 1 return n It's presence in the decorator list hints that this is something happening outside the functions ordinary local scope, as well as indicating when the initialisation happens. The 'nonlocal' then ties in with how they behave from the point of view of the code in the function body. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 29 September 2011 14:30, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree, this is a nice option. My biggest discomfort with it is the idea of a magic decorator name handled specially by the compiler (and worse still, it's a keyword so it's syntactically very weird indeed). I had similar discomfort over the new super, but I could get over that by simply assuming that super was a normal function that was just more magical than I understood. "@keyword" decorators don't even fit into my model of valid syntax :-( That said, I like it otherwise. The above says to me "the following values are nonlocal to this function", which I can read exactly the way it actually works. Whether that's a result of a week's immersion in Nick's propaganda, I can't say, though :-) Paul.

On 29 September 2011 15:05, Paul Moore <p.f.moore@gmail.com> wrote:
Actually, while we're using keywords as decorator names, @with(n=0, lock=threading.Lock()) def global_counter(): with lock: n += 1 return n reads well to me, and avoids the technical/jargon word "nonlocal". Let the @keyword floodgates open :-) Paul.

On 2011-09-29, at 17:15 , Paul Moore wrote:
The problem with that one is that it's going to be read in light of the corresponding keyword, so it'd make people think the purpose is to __enter__/__exit__ all objects provided around the function call somehow (pretty nice for the lock, nonsensical for the integer, of course).

Paul Moore wrote:
I think 'nonlocal' is the better choice as it mirrors what nonlocal does inside the function. Because of its current usage 'with' will generate questions about __enter__ and __exit__. ~Ethan~

On 29/09/2011 17:21, Ethan Furman wrote:
But what if the decorator is used inside an outer function? Wouldn't that be confusing? What about @func_attrs(...) ? - it would give a hint where the objects are actually located.

On Thu, Sep 29, 2011 at 8:05 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I feel the same way. The alternative is to leave nonlocal as just a simple statement, but change its behavior when the name is not found inside a containing function scope. Currently that is a syntax error. Instead it could imply that the name should fall back to the definition-time "scope" of the current function. In other words, the compiler would treat the name like a closed variable anyway, but with an empty cell (or some default initial value, like None)[1]. A decorator would still be used to explicitly initialize the value: @init_nonlocal(n=0, lock=threading.Lock()) def global_counter(): nonlocal n with lock: n += 1 return n This approach has some advantages: * doesn't require the compiler to do any special handling of decorators, * fewer changes to the compiler (and less substantial), * keeps the advantages of using a decorator for initialization, * the nonlocal nature of the variable is clear in the function body (where it is used), * allows for "uninitialized" def-time names (see [1]), * more intuitively consistent use of the keyword (says my gut), * decorator could be applied to any function at any time (not just def-time); ...and some disadvantages: * the statement may be redundant for def-time nonlocals (if it will always be accompanied by the decorator), * implicit initialization (if used) is...implicit (EIBTI), * implicit default def-time scope is as well, * (maybe) makes it too easy to modify closure cells, if that's a bad thing, * decorator could be applied to any function at any time (not just def-time). The init_nonlocal decorator [factory] would change the value of the appropriate cell in __closure__. Either the decorator would be the function that exposed the ability to modify cells or it would leverage another function that did so. I would think the latter, so it could look like this: def init_nonlocal(**kwargs): def decorator(f): freevars = f.__code__.co_freevars indices = {} for name in kwargs: try: index = freevars.index(name) except ValueError: raise NameError(...) indices[name] = index for name, index in indices.items(): #f.__closure__[index].cell_contents = kwargs[name] functools.update_cell(f, index, kwargs[name] return f return decorator or def init_nonlocal(**kwargs): def decorator(f): functools.update_freevars(**kwargs) return f return decorator The cell-modifying capabilty would have to apply to any cell in __closure__ and not just the ones associated with def-time nonlocals. If the nonlocal statement gets associated with a name in an enclosing function scope, then init_nonlocal would be used to explicitly force the name to be a def-time nonlocal. Maybe that's an argument for syntax to explicitly identify a def-time nonlocal rather than relying on any implicit behavior of the nonlocal keyword. -eric [1] explicitly uninitialized variables in some languages (like C) get implicitly initialized to some default value. I'm not sure what the use case would be for an uninitialized nonlocal (or even if it makes sense for a cell to be uninitialized). In that situation I would expect a NameError when the name is accessed.

On Thu, Sep 29, 2011 at 3:03 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Yeah, I'd thought of that possibility and have a few problems with it: 1. Replacing the current SyntaxError when a nonlocal statement refers to a variable that doesn't exist in an outer scope with a call time "UnboundNonLocalError" or an implicit None reference doesn't thrill me. Either choice would make UnboundLocalError look positively intuitive. 2. Adding a general ability to edit closure references from pure Python code outside the function body is a bridge I'm highly reluctant to cross. That kind of thing is what classes are for. 3. The syntax is repetitive and noisy enough that I think people could be legitimately excused for continuing to use the default argument hack instead. It's important to keep in mind that for folks that are familiar with how it works, the default argument hack is actually pretty easy to read and write. It's quite clearly exploiting a side effect of a feature that was added for other reasons, though, which is why blessing and documenting it as the preferred mechanism for function state variables (if you choose to use them) also doesn't feel right. All that said, the @nonlocal idea certainly has its own problems. In particular, it looks superficially like an ordinary call to a decorator factory, but is actually quite restricted syntactically (keyword arguments only, no * or ** expansion), has significant effects on the resolution of names in the function's namespace and can't be used for post-decoration via an ordinary function call with the just defined function as an argument. It would end up in the same category of "special case that doesn't actually look all that special" as the 3.x version of super(). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Sep 29, 2011 at 12:03 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
For a reason. It would be too easy for a typo to produce the wrong interpretation. -- --Guido van Rossum (python.org/~guido)

On Thu, 2011-09-29 at 13:01 -0700, Guido van Rossum wrote:
But this is exactly how it works now! The behavior of nonlocal doesn't need to be changed. It already behaves that way with closures. :-) Cheers, Ron (PS... Pleas repost if this doesn't show up on the python-ideas. My emails aren't getting there. My apologies for duplicates.) Python 3.2rc2+ (py3k:88358, Feb 6 2011, 08:56:00) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. Closure as defaults.
Closures with nonlocal.
Closure won't allow changes to cells without nonlocal.

On Thu, Sep 29, 2011 at 9:07 PM, Ron Adam <ron3200@gmail.com> wrote:
That isn't what Guido and Eric are talking about here. They're talking about this syntax error:
We had the opportunity in PEP 3104 to make 'nonlocal x' a synonym for 'global x' and chose not to do so. After deliberately passing up that more obvious interpretation, we aren't likely to now decide to use it to denote function state variables. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, 2011-09-29 at 21:32 -0400, Nick Coghlan wrote:
Ah, ok, There are so many suggestions it's hard to keep up. I think, all that is needed is a way to rebind a function inside a decorator. If we can do that, then everything will pretty much just work. And nonlocal should do the right thing as it is. Could a method be added to the Function class, that will copy a function and capture the current scope and any additional cells? class function(object) | function(code, globals[, name[, argdefs[, closure]]]) | | Create a function object from a code object and a dictionary. | The optional name string overrides the name from the code object. | The optional argdefs tuple specifies the default argument values. | The optional closure tuple supplies the bindings for free variables. Looking at this, it may already be able to do that. But it could be easier. How about if we add the ability for it to copy a function, so that we can do... func = type(func)(func) # get a copy func = type(func)(func, closure={'x':0}) # with closures Then we can do... def set_closure(**kwds): def wrapper(f): f = type(f)(f, closure=kwds) return f return wrapper @set_closure(x=0) def adder(y): nonlocal x x += y return x That looks fine to me. def set_this(f): kwds = {'__this__':f} f = type(f)(f, closure={'__this__':f) return f This decorator won't work as the function '__this__' points to is the old function, the new one hasn't been created yet. Hmmm. Cheers, Ron

On Thu, 2011-09-29 at 22:35 -0500, Ron Adam wrote:
Correct that too... def set_this(f): f = type(f)(f, closure={'__this__':f}) return f It still won't work, for the reasons stated.
Cheers, Ron

On Sep 29, 2011, at 5:35 PM, Ron Adam wrote:
Won't work as written because the nonlocal will cause an error before the decorator is applied. The other day, I suggested we add the ability for decorators to have a __prepare__ method that creates the namespace the function being created will use, but it was shot down as impractical. It would however do what you want to do here.

On Thu, 2011-09-29 at 17:45 -1000, Carl Matthew Johnson wrote:
You are right! I didn't see that. But, first lets understand what nonlocal actually does. Doing a lot of comparisons, it tells the compiler to generate STORE_DEREF byte codes instead of STORE_FAST byte codes. In otherwords the nonlocal feature can be moved into the Function class constructor. It doesn't need to actually be in the function it self. So the Function class could take a list of "nonlocal" names, along with the a dict of closure names and values. Then, the function the decorator is on, will work as if it had nonlocal statement in it on those names. def set_closure(**kwds): def wrapper(f): f = type(f)(f, closure=kwds, nonlocals=kwds.keys()) return f return wrapper @set_closure(x=0) def adder(y): x += y return x That would get around the nonlocal error.
I think it would have the same problems, because the namespace isn't associated to the function at the time it's created. If the above is out, then were back to syntax. How about anonymous wrappers? @@(x=0) def adder(y): nonlocal x x += y return x which would be the same as... def _(x=0): def adder(y): nonlocal x x += y return x return adder adder = _() And of course decorators could be stacked above that. Cheers, Ron

On Fri, Sep 30, 2011 at 1:25 AM, Ron Adam <ron3200@gmail.com> wrote:
In otherwords the nonlocal feature can be moved into the Function class constructor. It doesn't need to actually be in the function it self.
It's *really* important to remember the difference between compilation time, definition time and call time in this discussion, since name binding affects all of them. Compilation time (usually at module import): Compiler decides between STORE_FAST (function local), STORE_DEREF (function closure/nonlocal), STORE_GLOBAL (global declaration) and STORE_NAME (class/module scope). Definition time (i.e. when the def statement itself is executed to create the function object): Compiler creates any decorators, any default arguments, loads references to any outer cells on to the stack, loads the code object for the function body, creates the function (passing in the default arguments, cell references and code object), and only then applies the decorators. Call time (i.e. when the function is called): Initialises the execution frame with space for the local variables and references to the cells for variables that may persist beyond this call, binds the arguments to the appropriate locals (potentially filling some in from the default arguments) and then passes the code object to the main eval loop to be executed. The reason nonlocal and global declarations need to exist in the first place is precisely to override the compiler's default assumption that all name bindings in function scope are references to local variables. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Sep 30, 2011 at 5:07 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
+1
Good point. And it's important to reiterate that compilation only results in code objects. When a module is compiled, the bodies of all function defs (and class defs and comprehensions) therein are compiled separately (but relative to the module's code). The resulting code objects are put into the co_consts of the module's code object. You'll find this if you look at a pyc file[1]. Likewise, if you have a nested function, the inner function's code object will be stored in the co_consts of the outer function's code object. All this was a mystery to me until recently.
Exactly! That's a critical point to the discussion that isn't so obvious. Compilation pre-computes a bunch of things that are used at call-time and these are stored on the code object. Two examples are the "space for the local variables" and the size of the frame's stack, which are partially dependent on the local/nonlocal nature of the various names used in the function body. Also, all the names in the signature or used in the body were mapped to indices at compile-time (by virtue of being stored in various tuples). The actual compiled code (in co_code) refers to these indices, which the eval loop uses when pulling values from the appropriate appropriate tuples. Those tuples were pulled from function attributes like __closure__ and __defaults__. So, any solution here must reflect that the compiler needs a way to calculate these things. On the flip side, any solution that relies on def-time hacks of the function object must take into account the subsequent changes that will be necessary on the function's code object. Particularly important is the relationship between __closure__ and co_freevars[2]. It's part of why turning a local into a closed variable is not a trivial thing. <disclaimer> I'll admit that I may have gotten some of this wrong, not having years of experience to back me up here. So I definitely defer to knowledge of Nick, et al. I have been pretty immersed in this stuff for several months and the above is my resultant perspective, that of someone who has dived in without a lot of background in it. :) </disclaimer> -eric [1] See http://code.activestate.com/recipes/577880/ [2] http://hg.python.org/cpython/file/default/Include/funcobject.h#l34 http://hg.python.org/cpython/file/default/Objects/funcobject.c#l454

On Fri, Sep 30, 2011 at 3:01 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Heh, it's not like all this is something that comes up every day - the compiler and eval loop mostly sit in the corner chugging away quietly without bothering anyone. The main advantage of experience is knowing what all the moving parts are and where in the code base to look to refresh my recollection of any details I've forgotten :) The only comment I'd make about your explanation is that a lot of those details aren't actually part of the language spec - they're implementation details of CPython that other implementations may happen to follow because it's a reasonable way to do things. They can still be a useful intuition pump in helping to figure out what is and isn't feasible in the language design, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Sep 30, 2011 at 5:07 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
And something that I missed the first time -- even if you know a name shows up in the closure, that doesn't prevent it from being shadowed locally inside the function. So the function responsible for adjusting another function's bytecode (or other function representation) really does have to know both the contents of the closure AND which names are being removed from the locals. -jJ

Ron Adam dixit (2011-09-30, 00:25):
+1, though I'd rather prefer simply: @(x=0) def adder(y): nonlocal x x += y return x And, of course, if you don't need to rebind the variable, `nonlocal` would not be needed: @(lock=threading.RLock()) def my_foo(): with lock: "do foo" IMHO it is better than @nonlocal because it uses already settled @-decorator idiom and at the same time it does not pretend to be a normal fuction-based decorator. I like it. :) I think it would be not only clear and useful but also beautiful. Cheers. *j

On Fri, Sep 30, 2011 at 10:45 AM, Jan Kaliszewski <zuo@chopin.edu.pl> wrote:
On its own it actually looks very appealing. And it may still be fine. However, when mixed with decorators, it may be too ambiguous: @(lock=threading.RLock()) @does_something_else def my_foo(): with lock: "do foo" or @does_something @(lock=threading.RLock()) @does_something_else def my_foo(): with lock: "do foo" What does "@(lock=threading.RLock())" do here? It would be easy to miss that it affects "my_foo" (at compile-time) and not the result of "@does_something_else" (at def-time). This is the problem with mixing the syntax for a compile-time directive with that of a def-time directive, particularly when they can show up right next to each other. This is a possible solution: a syntax requirement that no real decorators come after this new syntax. I'm still cautious about the idea of sharing syntax between compile-time and def-time directives. However, default arguments sort of do this already. -eric

On Fri, 2011-09-30 at 11:39 -0600, Eric Snow wrote:
I was thinking it would only be allowed just before def statement, (Thats why I used '@@' instead of just '@') If it's out of place then it would be treated just like any other decorator.
And of course cause an error. ;-) (Just maybe not that exact one.) Cheers, Ron

On Fri, Sep 30, 2011 at 1:39 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
All name binding operations have both compile time and runtime semantics, so that's OK. 'nonlocal' and 'global' are actually the odd ones out since they *only* affect compile time and don't actually correspond directly to anything in the generated bytecode (instead influencing the way the names they specify get treated in other parts of the code). If a "function state decorator" approach is used, then yeah, I agree it should come immediately before the main part of the function header. However, I'm not seeing a lot to recommend that kind of syntax over the post-arguments '[]' approach. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan dixit (2011-09-30, 14:14):
Yes, it seems to be a necessary requirement.
However, I'm not seeing a lot to recommend that kind of syntax over the post-arguments '[]' approach.
IMHO @(...) has two advantages over '[]' approach: 1. It keeps all that 'additional scope'-stuff in a separate line, making the code probably more clear visualy, and not making the crowd of elements in the '...):' line even more dense (especially if we did use annotations, e.g.: '...) -> "my annotation":'). 2. It is more consistent with the existing syntax (not introducing '='-based syntax within []; [] are already used for two different things: list literals and item lookup -- both somehow leading your thoughts to sequence/container-related stuff). Cheers. *j

Jan Kaliszewski dixit (2011-09-30, 23:32):
And also -- what maybe is even more important -- 3. Placing it before (and beyond) the whole def statement makes it easier to *explain* the syntax: @(x=1, lock=Lock()) def do_foo(y): nonlocal x with lock: x += y return x as being equivalent to: def _closure_provider(): x = 1 lock = Lock() def do_foo(y): nonlocal x with lock: x += y return x return do_foo do_foo = _closure_provider() del _closure_provider Cheers. *j

On Sat, 2011-10-01 at 13:44 +0200, Jan Kaliszewski wrote:
Supposedly the @ decorator syntax is supposed to be like a pre-compile substitution where.. @decorator def func(x):x Is equivalent to... def func(x):x func = decorator(func) IF that is true, it doesn't make the python core more complex. It would just be a source rewrite of the effected block in memory just before it's compiled. But it seems it's not implemented exactly that way. def deco(func): def _(f): return func(f) return _ @deco(foo) def foo(f): return f print(foo('python')) Results with ... NameError: name 'foo' is not defined def foo(f): return f foo = deco(foo) print(foo('python')) Prints "Python" and doesn't cause a name error. Is this a bug, or by design? I thought, if the @@ wrapper idea could be put in terms of a simple substitution template like the @ design, then it may be more doable. Cheers, Ron

Ron Adam writes:
It is.
This is different syntax, whose interpretation is not obvious from the simpler case. For example, by analogy to method syntax it could mean def foo(f): return f foo = deco(foo, foo) (where the first argument is implicit in the decorator syntax = the function being decorated, and the second is the explicit argument), in which case you would have gotten a "too few arguments" error instead, but it's doesn't. In fact, it is defined to be equivalent to bar = deco(foo) @bar def foo(f): return f Ie, bar = deco(foo) def foo(f): return foo foo = bar(foo) (which makes the reason for the error obvious), not
I guess in theory you could think of moving the evaluation of deco(foo) after the def foo, but then the correct equivalent expression would be def foo(f): return f foo = deco(foo)(foo) which I suspect is likely to produce surprising behavior in many cases.

On 1 October 2011 19:57, Stephen J. Turnbull <stephen@xemacs.org> wrote:
[skip explanation that decorator is evaluated before function definition]
Even this wouldn't work, because in reality (in CPython at least) the name 'foo' is only bound *once*, that is after the application of the decorator - as the example below illustrates:
Anyway, I've now lost track of how this relates to the subject of this thread :) -- Arnaud

On Sat, 2011-10-01 at 20:27 +0100, Arnaud Delobelle wrote:
Anyway, I've now lost track of how this relates to the subject of this thread :)
If a decorator can take the function name it is decorating, then Nicks example of using a private name for recursion becomes easier to do. According to pep316, it should be possible, but it's not implemented the way the pep describes. Also, if decorators are, or can be, implemented as a before compile time, template translation, then these 'sugar' features wont make the underlying core more complicated or complex. And finally, decorator type 'template' solution may also work to create closures. (Without making the core more complex.) Cheers, Ron

On Sun, 2011-10-02 at 03:57 +0900, Stephen J. Turnbull wrote:
Actually, this is what I though it was suppose to be. My example wasn't very good as it returned valid results in more than one path depending on how it was called. (not intentionally) Lets try a clearer example. def supply_y(y): def _(func): def wrapper(x): return func(x, y) return wrapper return _ @supply_y(2) def add(x, y): return x + y Pep 318 says it should translate to ... def add(x, y): return x + y add = supply_y(2)(add) add(3) ---> 5 # call wrapper with x, which calls add with (x, y) But the behavior is as you suggest... _temp = suppy_y(2) def add(x, y): return x + y add = _temp(add) add(3) ---> 5 This isn't what Pep 318 says it should be. Is this intentional, and does Pep 318 need updating? Or an unintended implementation detail? If it wasn't intended, then what? Cheers, Ron

On 1 October 2011 21:36, Ron Adam <ron3200@gmail.com> wrote: [...]
I don't know if it is intentional, but it is probably a consequence of the fact that the CALL_FUNCTION opcode expects the function below its arguments in the stack. So the most concise way of implementing @<decorator_expression> def f(x): .... is to: 1. evaluate <decorator_expression> an push the result on the stack 2. create function f and push it on the stack 3. execute CALL_FUNCTION 4. store the result in 'f' If you swap steps 1 and 2, then you need to add a new step: 2.5 execute ROT_TWO (swap TOS and TOS1) -- Arnaud

Ron Adam writes:
That's what I get, too: the same result using decorator syntax and using an explicit call to supply_y. What's the problem? As far of the subject of the thread goes, though, I don't see how this can help. The problem isn't that there's no way to refer to the decorated function, because there is a reference to it (the argument to the actual decorator, eg, the third foo in "foo = deco(foo)(foo)" which is an "anonymous reference" in the decorator implementation in CPython). The problem isn't referring to the name of the function, either; we don't care what the name is here.

On Sat, Oct 1, 2011 at 2:57 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
It isn't quite - the name binding doesn't happen until *after* the decorator chain has been invoked, so the function is anonymous while the decorators are executing. In addition, the decorator expressions themselves are evaluated before the function is defined. That's not a problem in practice, since each decorator gets passed the result of the previous one (or the original function for the innermost decorator). The real equivalent code is more like: <anon1> = decorator def <anon2>(x): return x func = <anon1>(<anon2>) (IIRC, the anonymous references happen to be stored on the frame stack in CPython, but that's an implementation detail) As far as the proposed semantics for any new syntax to eliminate the desire to use the default argument hack goes, I haven't actually heard any complaints about any addition being syntactic sugar for the following closure idiom: def <anon1>(): NAME = EXPR def FUNC(ARGLIST): """DOC""" nonlocal NAME BODY return FUNC FUNC = <anon1>() The debate focuses on whether or not there is any possible shorthand spelling for those semantics that successfully negotiates the Zen of Python: Beautiful is better than ugly. - the default argument hack is actually quite neat and tidy if you know what it means. Whatever we do should be at least as attractive as that approach. Explicit is better than implicit. - the line between the default argument hack and normal default arguments is blurry. New syntax would fix that. Simple is better than complex. - lexical scoping left simple behind years ago ;) Complex is better than complicated. - IMO, the default argument hack is complicated, since it abuses a tool meant for something else, whereas function state variables would be just another tier in the already complex scoping spectrum from locals through lexical scoping to module globals and builtins (with function state variables slotting in neatly between ordinary locals and lexically scoped nonlocals). Flat is better than nested. - There's a lot of visual nesting going on if you spell out these semantics as a closure or as a class. The appeal of the default argument hack largely lies in its ability to flatten that out into state storage on the function object itself Sparse is better than dense. - This would be the main argument for having something before the header line (decorator style) rather than cramming yet more information into the header line itself. However, it's also an argument against decorator-style syntax, since that is quite heavy on the page (due to the business of the '@' symbol in most fonts) Readability counts. - The class and closure solutions are not readable - that's the big reason people opt for the default argument hack when it applies. It remains to be seen if we can come up with dedicated syntax that is at least as readable as the default argument hack itself. Special cases aren't special enough to break the rules. - I think this is the heart of what killed the "inside the function scope" variants for me. They're *too* magical and different from the way other code at function scope works to be a good fit. Although practicality beats purity. - Using the default argument hack in the first place is the epitome of this :) Errors should never pass silently. Unless explicitly silenced. - This is why 'nonlocal x', where x is not defined in a lexical scope, is, and will remain, a Syntax Error and why nonlocal and global declarations that conflict with the parameter list are also errors. Similar constraints would be placed on any new syntax dedicated to function state variables. In the face of ambiguity, refuse the temptation to guess. - In the case of name bindings, the compiler doesn't actually *guess* anything - name bindings create local variables, unless overridden by some other piece of syntax (i.e. a nonlocal or global declaration). This may, of course, look like guessing to developers that don't understand the scoping rules yet. The challenge for function state variables is coming up with a similarly unambiguous syntax that still allows them to be given an initial state at function definition time. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. - For me, these two are about coming up with a syntax that is easy to *remember* once you know it, even if you have to look up what it means the first time you encounter. Others set the bar higher and want developers to have a reasonable chance of *guessing* what it means without actually reading the documentation for the new feature. I think the latter goal is unattainable and hence not a useful standard. However, I'll also note that the default argument hack itself does meet *my* interpretation of this guideline (if you know it, recognising it and remembering it aren't particularly difficult) Now is better than never. Although never is often better than *right* now. - The status quo has served us well for a long time. If someone can come up with an elegant syntax, great, let's pursue it. Otherwise, this whole issue really isn't that important in the grand scheme of things (although a PEP to capture the current 'state of the art' thinking on the topic would still be nice - I believe Jan and Eric still plan to get to that once the discussion dies down again) If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. - this is where I think the specific proposal to just add syntactic sugar for a particular usage of an ordinary closure is a significant improvement on past attempts. Anyone that understands how closures work will understand the meaning of the new syntax, just as anyone that fully understands 'yield' can understand PEP 380's 'yield from'. Namespaces are one honking great idea -- let's do more of those! - closures are just another form of namespace, even though people typically think of classes, modules and packages when contemplating this precept. "Function state variables" would be formalising the namespace where default argument values live (albeit anonymously) and making it available for programmatic use. Despite its flaws, the simple brackets enclosed list after the function parameter list is still my current favourite: def global_counter(x) [n=0, lock=Lock()]: with lock: n += 1 yield n It just composes more nicely with decorators than the main alternative still standing and is less prone to overwhelming the function name with extraneous implementation details: @contextmanager def counted() [active=collections.Counter(), lock=threading.RLock()]: with lock: active[threading.current_thread().ident] += 1 yield active with lock: active[threading.current_thread().ident] -= 1 far more clearly conveys "this defines a context manager named 'counted'" than the following does: @contextmanager @(active=collections.Counter(), lock=threading.RLock()) def counted(): with lock: active[threading.current_thread().ident] += 1 yield active with lock: active[threading.current_thread().ident] -= 1 Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, 2011-10-01 at 22:11 -0400, Nick Coghlan wrote: +1 on all of the zen statements of course. I think you made a fine case for being careful and mindful about this stuff. :-)
One way to think of this is, Private, Shared, and Public, name spaces. Private and Public are locals and globals, and are pretty well supported, but Shared names spaces, (closures or otherwise) are not well supported. I think the whole concept of explicit shared name spaces, separate from globals and locals is quite important and should be done carefully. I don't think it is just about one or two use-cases that a small tweak will cover.
This syntax to me looks a bit too close to a list literal. How about a name space literal? ie.. a dictionary. def global_counter(x) {n:0, lock=lock}: with lock: n += 1 yield n I think that looks better than dict(n=0, lock=lock). And when used as a repr for name spaces, it is more readable. A literal would cover the default values use case quite nicely. A reference to a pre-defined dictionary would cover values shared between different functions independent of scope.
far more clearly conveys "this defines a context manager named 'counted'" than the following does:
Putting them after the function signature will result in more wrapped function signatures. While its very interesting to try to find a solution, I am also concerned about what this might mean in the long term. Particularly we will see more meta programming. Being able to initiate an object from one or more other objects can be very nice. Python does that sort of thing all over the place. Cheer, Ron

Carl Matthew Johnson wrote:
Is that an attempt to be funny? Because braces are already used for dictionary and set literals. The __future__ braces refers to braces as BEGIN ... END delimiters of code blocks, not any use of braces at all. While we're throwing around colours for the bike-shed, the colour which seems to look best to me is: def global_counter(x, [n=0, lock=lock]): # inside the parameter list ... rather than def global_counter(x) [n=0, lock=lock]: # outside the parameter list ... but Ron's suggestion that we use dictionary syntax does seem intriguing. -- Steven

On Sun, Oct 2, 2011 at 10:30 AM, Steven D'Aprano <steve@pearwood.info> wrote:
The main problem I have with 'inside the parameter' list is the way it looks when there's only state and no arguments: def f([state=State()]): ... One could be forgiven for thinking that f() accepts a single optional positional argument in that case. Separating the two, on the other hand, would keep things clear: def f() [state=State()]: ... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Oct 2, 2011 at 2:29 AM, Ron Adam <ron3200@gmail.com> wrote:
Heh, even if nothing else comes out of these threads, I can be happy with helping others to learn how to look at this kind of question from multiple angles without getting too locked in to one point of view (and getting more practice at doing so, myself, of course!)
"not well supported" seems a little too harsh in the post PEP 3104 'nonlocal' declaration era. If we look at the full suite of typical namespaces in Python, we currently have the following (note that read/write and read-only refer to the name bindings themselves - mutable objects can obviously still be modified for a reference that can't be rebound): Locals: naturally read/write Function state variables (aka default argument values): naturally read-only, very hard to rebind since this namespace is completely anonymous in normal usage Lexically scoped non-locals: naturally read-only, writable with nonlocal declaration Module globals: within functions in module, naturally read-only, writable with global declaration. At module level, naturally read/write. From outside the module, naturally read/write via module object Process builtins: naturally read-only, writable via "import builtins" and attribute assignment Instance variables: in methods, naturally read/write via 'self' object Class variables: in instance methods, naturally read-only, writable via 'type(self)' or 'self.__class__'. Naturally read/write in class methods via 'cls', 'klass' or 'class_' object. Of those, I would put lexical scoping, function state variables and class variables in the 'shared' category - they aren't as contained as locals and instance variables, but they aren't as easy to access as module globals and process builtins, either. The current discussion is about finding a syntax to bring function state variables on par with lexical scoping, such that default argument values are no longer such a unique case.
The "but it looks like a list" argument doesn't really hold any water for me. Parameter and argument lists look like tuples, too, but people figure out from context that they mean something different and permit different content. I have some specific objections to the braces syntax, too: - I believe the assocation with func.__dict__ would be too strong (since it's actually unrelated) - braces and colons are a PITA to type compared to brackets and equals signs - the LHS in a dictionary is an ordinary expression, here it's an unquoted name [NAME=EXPR, NAME2=EXPR2] is clearly illegal as a list, so it must mean something else, perhaps something akin to what (NAME=EXPR, NAME2=EXPR2) would have meant in the immediately preceding parameter list (this intuition would be correct, since the two are closely related, differing only in the scope of any rebindings of the names in the function body). {NAME=EXPR, NAME2=EXPR}, on the other hand, looks an awful lot like {NAME:EXPR, NAME2:EXPR2}, which would be an ordinary dict literal, and *not* particularly related to what the new syntax would mean.
No, that can never work (it's akin to the old "from module import *" at function level, which used to disable fast locals but is now simply not allowed). The names for any shared state *must* be explicit in the syntax so that the compiler knows what they are. When that isn't adequate it's a sign that it's time to upgrade to a full class or closure.
Agreed, but even there I think I prefer that outcome, since the more important information (name and signature) precedes the less important (the state variable initialisation). Worst case, someone can put their state in a named tuple or class instance to reduce the noise in the header line - state variables are about approaching a problem in a different way (i.e. algorithm more prominent than state) rather than about avoiding the use of structured data altogether.
I'm not sure I understand what you mean in your use of the term 'meta-programming' here. The biggest danger to my mind is that we'll see more true process-level globals as state on top-level functions, and those genuinely *can* be problematic (but also very useful, which is why C has them). It's really no worse than class variables, though. The other objection to further enhancing the power of functions to maintain state is that functions aren't naturally decomposable the way classes are - if an algorithm is written cleanly as methods on a class, then you can override just the pieces you need to modify while leaving the overall structure intact. For functions, it's much harder to do the same thing (hence generators, coroutines and things like the visitor pattern when walking data structures). My main counters to those objections are that: 1. Any feature of this new proposal can already be done with explicit closures or the default argument hack. While usage may increase slightly with an officially blessed syntax, I don't expect that to happen to any great extent - I'm more hoping that over time, the default argument hack usages would get replaced 2. When an algorithm inevitably runs up against the practical limits of any new syntax, the full wealth of Python remains available for refactoring (e.g. by upgrading to a full class or closure) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, 2011-10-02 at 09:38 -0400, Nick Coghlan wrote:
Yes, that's a very zen way to look at it. +1 Keeping that larger picture in mind, while sorting though various smaller options is challenging. Hopefully in the end, the best solution, (which may include doing nothing), will be sorted out.
I think the introspection tools you want will help. hmm... what about the vars() function? (I tend to forget that one) vars(...) vars([object]) -> dictionary Without arguments, equivalent to locals(). With an argument, equivalent to object.__dict__. Could we extend that to see closures and scope visible names?
I think it may be easier to classify them in terms of how they are stored. Cell based names spaces: function locals function closures Dictionary based names spaces: class attributes module globals builtins If vars() could get closures, What exactly would it do and how would the output look? Would it indicate which was free variables, from cell variables? [clipped literal parts, for now]
I'm thinking of automated program generation. A programming language that has a lot of hard syntax without a way to do the same things in a dynamic way makes it harder to do that. You pretty much have to resort to exec and eval in those cases. Or avoid those features.
I would like very much for functions to be a bit more decomposable.
The part I like is that it removes a part of functions signatures, which I think are already over extended. Although, only a small part.
I agree, any solution should be compared to an alternative non-syntax way of doing it. All very interesting... Cheers, Ron

Nick Coghlan writes:
As I understand the issue here, as far as the decorators are concerned, the reference passed by the decorator syntax should be enough to do any namespace manipulations that are possible in a (non-magic) decorator. Am I missing something? That is, in
surely the FUNC above ...
... doesn't need to be the *same identifier* as the FUNC below?
FUNC = <anon1>()
Isn't magic needed solely to inject the nonlocal statement(s) into the definition of FUNC inside <anon1> at compile-time?

On Sun, Oct 2, 2011 at 4:16 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Well, having 'FUNC' the same from the compiler's point of view is also necessary to get introspection to work properly (i.e. FUNC.__name__ == 'FUNC'). But yeah, the fundamental challenge lies in telling the compiler to change the way it binds and references certain names (like global and nonlocal declarations) while also being able to initialise them at function definition time (like default arguments). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan writes:
On Sun, Oct 2, 2011 at 4:16 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Not only isn't that magic, but it doesn't currently work anyway, at least not for me in Python 3.2 (borrowing Ron's example):
As expected, but Expedia Per Diem! and
Woops! As for the "not magic" claim:
Should a bug be filed, or is this already part of your "improved introspection for closures" proposal?

On Sun, Oct 2, 2011 at 10:05 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Should a bug be filed, or is this already part of your "improved introspection for closures" proposal?
There's a reason functools.wraps [1] exists (despite the way it works being an egregious hack) :) But that's also the reason I've been careful in my examples of equivalent semantics to make sure the inner function name matches the eventually bound name in the outer scope - without overwriting metadata the way @wraps(f) does, duplicating information in the original source code is the only other way to get informative introspection results. [1] http://docs.python.org/library/functools#functools.wraps Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, 2011-10-02 at 17:16 +0900, Stephen J. Turnbull wrote:
I've managed to do it with a function, but it isn't pretty and isn't complete. It does work for simple cases. But it isn't easy to do... 1. Creating a dummy function with a __closure__, and taking the parts of interst from it. (Requires exec to do it.) 2. Creating a new byte code object with the needed changes. (Hard to get right) 3. Create a new code object with the altered pieces. 4. Make a new function with the new code object and __closure__ attribute. Use the original function to supply all the other parts. What you get is a function that can replace the old one, but it's a lot of work. A compile time solution would be much better, (and faster), and that is why it boils down to either special syntax, or a precompile decorator type solution. If we could make the co_code object less dependent of the cell reference objects, then a dynamic run time solution becomes realistic. But that would take rethinking the byte code to cell relationships. I don't think that is a near term option. Cheers, Ron

On 2 October 2011 21:24, Ron Adam <ron3200@gmail.com> wrote:
And you've got to take care of nested functions. That is, look for code objects in the co_consts attribute of the function's code objects and apply the same transformation (recursively). Moreover, you have to modify (recursively) all the code objects so that any MAKE_FUNCTION is changed to MAKE_CLOSURE, which involves inserting into the bytecode a code sequence to build the tuple of free variables of the closure on the stack. At this point it may be almost easier to write a general purpose code object to source code translator, stick a nonlocal declaration at the start of the source of the function, wrap it in an outer def and recompile the whole thing! A simple example to illustrate: def foo(): def bar(): return x + y return bar This compiles to: 0 LOAD_CONST 1 (<code object bar>) 3 MAKE_FUNCTION 0 6 STORE_FAST 0 (bar) 9 LOAD_FAST 0 (bar) 12 RETURN_VALUE Now imagine we have the non-magic "nonlocal" decorator: @nonlocal(x=27, y=15) def foo(): def bar(): return x + y return bar That should compile to something like this: 0 LOAD_CLOSURE 0 (y) 3 LOAD_CLOSURE 1 (x) 6 BUILD_TUPLE 2 9 LOAD_CONST 3 (<code object bar>) 12 MAKE_CLOSURE 0 15 STORE_FAST 0 (bar) 18 LOAD_CONST 0 (None) 21 RETURN_VALUE And obvioulsy the "bar" code object need to be adjusted (LOAD_GLOBAL -> LOAD_DEREF). -- Arnaud

On Sun, 2011-10-02 at 22:32 +0100, Arnaud Delobelle wrote:
On 2 October 2011 21:24, Ron Adam <ron3200@gmail.com> wrote:
...
I got this much to work.. def foo() @nonlocal(x=27, y=15) def bar(): return x + y return bar But nothing with nested functions. It sort of makes me want pure functions with no scope or closures by default. Cheers, Ron

Nick Coghlan dixit (2011-10-01, 22:11):
I believe that both syntax propositions: def spam(x, some, arguments, foo_bar=997) [variable=1, lock=Lock()]: nonlocal variable with lock: variable += x return variable + foo_bar and @(variable=1, lock=Lock()) def spam(x, some, arguments, foo_bar=997): nonlocal variable with lock: variable += x return variable + foo_bar -- are quite elegant and each of them would be nice and useful equivalent of: def _temp(): variable = 1 lock = Lock() def spam(x, some, arguments, foo_bar=997): nonlocal variable with lock: variable += x return variable + foo_bar return spam spam = _temp() del _temp
Yes, I already started creating such a summary, though this thread changed a lot (at least in my mind -- about the subject). As I had reserved I cannot promise to do it quickly (because of other activities). Cheers. *j

On Fri, Sep 30, 2011 at 12:14 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
However, I'm not seeing a lot to recommend that kind of syntax over the post-arguments '[]' approach.
Agreed, though Jan made some good points about this. But my concern with the decorator approach is that it isn't visually distinct enough from normal decorators. The @(...) form does help though. How about another approach: given: lock=threading.RLock()) def my_foo(): with lock: "do foo" or def my_foo(): with lock: "do foo" given: lock=threading.RLock()) It's an idea that just popped into my head (*cough* PEP 3150 *cough*). But seriously, I don't think statement local namespaces have come up at all in this super-mega thread, which surprises me (or I simply missed it). Doesn't it need a "killer app". Maybe this isn't "killer" enough, but it's something! :) -eric

On Thu, Sep 29, 2011 at 3:03 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Until this discussion started, I had not realized that nonlocal was limited to names from containing *function* scopes; I had thought that it would end up using the module-level globals if need be. Having it indicate a scope *closer* than all the scopes where it already looked but failed to find the name seems wrong. That said, my intuition may be suspect; outside examples, I have never used nonlocal, and work fairly hard to avoid global. I don't *like* boxing up my variables, but it still seems less offensive than a global statement and the resulting side effects. -jJ

me:
On Thu, Sep 29, 2011 at 4:26 PM, Devin Jeanpierre <jeanpierreda@gmail.com> asked:
How is that?
Normally, things at the module level can only be modified by other lines at the module level, or by monkey-patching. The global statement changes that. >>> x=5 # Safe ... no other assignments on the far left ... >>> class T: x=4 # OK, so there is a class level variable def __init__(self): global x # which, alas, isn't what gets used x=3 >>> x 5 >>> T() <__main__.T object at 0x0137A270> >>> x 3 Wait -- how did x get changed? *I* didn't change it, did I? -jJ

On 29 September 2011 22:36, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
I see what you mean. But I think with this version, the fact that the variable is visible in the "decorator" call (i.e., just slightly outside the function body) is enough to make me feel less uncomfortable than the purely-within-the-function "nonlocal VAR from EXPR" did... Paul.

On Tue, Sep 27, 2011 at 12:12 AM, Guido van Rossum <guido@python.org> wrote:
Heh, doing exactly that (i.e. thinking in terms of how it would *work* and how I would *explain* those semantics to someone in terms of existing language concepts) is where my nonlocal suggestion came from in the first place :) As I see it, there are 3 possible ways to approach this feature, and they vary mainly in terms of when the initialisation expressions get evaluated: 1. Prior to function definition (i.e. the same as default argument values) 2. After function definition but before decorators are applied 3. After decorator application I don't like number 3, since it means the closure references haven't been populated yet when the decorators are executed, so decorators that needed to call the function wouldn't work any more and I think number 2 would just be plain confusing. Hence, I prefer number 1, with a decorator based idiom if anyone wants to play recursion tricks. The semantics for a function with a function scoped variable declared would then be along the lines of: @example def global_accumulator(x): nonlocal tally from 0 # Insert preferred spelling here tally += x return tally meaning: def _outer(): tally = 0 @example def global_accumulator(x): tally += x return tally return global_accumulator global_accumulator = _outer() Of course, the compiler wouldn't have to generate the bytecode like that - it could easily do the expression evaluations, stash them on the stack, and then create the inner function. The MAKE_CLOSURE opcode could be adjusted to look at the code object and pop the initialisation values for the function scoped variables and create the additional cell references in the __closure__ attribute. However, one advantage of actually going the implicit outer scope route is that the function scoped variables would naturally be able to refer to each other, so you could easily cache an initial value and then some further derived values that depended on the first one. For the recursion use cases, I'd suggest actually offering a 'functools.rebind_closure' function that provided an officially supported way to modify a closure reference in a function. You could then write a function with an early binding reference to itself as follows: def recursive(f): f.rebind_closure('recurse', f) @recursive def factorial(x): nonlocal recurse from recursive # This value gets replaced by the decorator if x = 0: return 1 return x * recurse(x-1) Yes, it would be rather clumsy, but it is also incredibly uncommon for the simple 'recurse by name' approach to actually fail in practice. If we did decide to support recursive references directly, as in: def factorial(x): nonlocal recurse from factorial # Early binding - and nonlocal suggesting evaluation in outer scope ;) if x = 0: return 1 return x * recurse(x-1) Then the way to go would be the second option above where the function scope variables can see the undecorated version of the function: def _outer(): # First we define our function def global_accumulator(x): tally += x return tally # Then evaluate the function scope variables tally = 0 # Then we apply our decorators return example(global_accumulator) global_accumulator = _outer() Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, 2011-09-27 at 06:08 -0400, Nick Coghlan wrote:
I keep going back to the point I was when I wanted to unpack **kwds into locals(). But that is only part of the problem. The other half is making the function use the decorator as it's closure, or to supply a new one.
+1, I think this is the right approach. But it needs to accept a dictionary because we can't unpack an argument list into locals, and we don't want to specialize (rewrite) the decorator for each individual case! We want to be able to write a *generalized* decorator.
# A generalized decorator for adding closure values. def add_closure(f): """ A decorator to add explicit closure values. """ def _(**kwds): re_define(f, closure=kwds) #As if it was defined here! return f return _ @add_closure(tally=0): def global_accumulator(x): nonlocal tally #Nothing new here. ;-) tally += x return tally A re_define function could be extended to make other adjustments as well. For example a 'defaults=" keyword, that add to locals instead of closures. Those would be set back to their default value on each call. It may be better for a rebind or redefine to return a new function. If we add a way to update locals from kwds, Then the rebind or redefine function can be simpler. def add_closure(f): """ A decorator to add explicit closure values. """ def _(**kwds): replace_locals(kwds) return redefine(f) # as if it was defined here. return _ # Recursive example that works even if the original function is renamed. def set_this(f): """ A decorator that gives a function a visible name "this". """ f.rebind_closure(this=f) @set_this def factorial(x): if x = 0: return 1 return x * this(x-1) Nonlocal isn't needed as we don't write to "this". The rebind_closure function would be able to take '**kwds' as it's argument, so it could be used in the accumulator example as well. No changes to the nonlocal statement are needed if we can rebind, or re_define functions. These complement nonlocal and allow doing things that are not possible (or easy) at this time. Mainly a way to write nice *GENERALIZED* decorators to address the issues we've been discussing. Cheers, Ron

On Sep 27, 2011, at 6:12 , Guido van Rossum wrote:
"our" is something in Perl, right?
That's right, and "our" is similar to Python's "global". Perl went with "state" as the keyword the concept described in this thread. state $foo = 1; --Gisle

On Tue, Sep 27, 2011 at 3:50 PM, Gisle Aas <gisle@activestate.com> wrote:
I actually like that as a term for the concept (I've certainly used it in my own descriptions in this very thread), but I shudder to think how much code would have to change if we made 'state' a keyword :P Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Hmm... 'local', 'statelocal', 'nonlocal', 'global', 'builtin'? -- Nick Coghlan (via Gmail on Android, so likely to be more terse than usual) On Sep 27, 2011 3:55 PM, "Nick Coghlan" <ncoghlan@gmail.com> wrote:

On Tue, Sep 27, 2011 at 3:23 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'll just toss the following variants out there. They tend to read nicely if one assumes the syntax "<keyword> <expr> as <identifier>", though "<keyword> <identifier> = <expr>" would also work for some of them: * Emphasizing that the eval happens in the outer scope: take, grab, seize, capture * Emphasizing that the var/val persists between calls: hold, keep, stow, retain Cheers, Chris -- I <3 Thesauri.

Guido van Rossum dixit (2011-09-26, 19:48):
OTOH I am happy to let you all bikeshed on a better name.
`deflocal`? (definition-time-bound local-scoped var...) `ownlocal`? (function's own local-scoped var...) `boundlocal`? -- that would be nice to have it somehow similar to the `nonlocal` keyword; suggesting *something different from* though *somehow similar or complementary* to the `nonlocal` mechanism. Cheers. *j

Carl Matthew Johnson wrote:
I think ``rebind expression as name`` reads fine. Not sure if "rebind" exactly gives the right impression though.
Why "rebind"? The expression hasn't been bound once yet, so you can't REbind it. Not that I like ``bind expression as name`` any better. -- Steven

Carl Matthew Johnson wrote:
I have to retract my support for 'boundlocal', as well as phrase including bind or bound -- nearly every assignment is a binding, so why single out this particular one with the name? ~Ethan~

On Mon, 2011-09-26 at 22:43 -0400, Nick Coghlan wrote:
I think I agree. It's close, but not close enough to what you want.
Don't give up just yet, I think you may just need an out of the box view point to see it from a different angle... Besides keywords, there is also syntax and or symbols. One of the things I like with python objects, is a symbol or operator, is really just a nicer way to call a method. Maybe this is option #3. ;-) Which reminds me... the '@' used for decorators doesn't follow that pattern. Too bad. If it did, it could be used for other things. The assignment nature of the '@' sort of fits this problem. def foo(x): @own {"y":12} #--> foo.__at__("own", **kwds): ... ... Ok, it's weird, I admit. @validate Anyway, this is just an idea example not to be taken too seriously. Maybe something could work in a similar way that doesn't require a keyword. Cheers, Ron

On Tue, Sep 27, 2011 at 3:36 AM, Ron Adam <ron3200@gmail.com> wrote:
Don't give up just yet, I think you may just need an out of the box view point to see it from a different angle...
As far as I can see from the discussion so far, proposing 'nonlocal' for this use case *is* the out of the box viewpoint seeing it from a different angle ;) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Sep 27, 2011 at 3:14 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Which reminds me of a Calvin and Hobbes cartoon. Calvin has a bad report card and says "You know how Einstein's grades were bad? Well mine are even *worse*!" IOW, out of the box thinking doesn't automatically lead to the right conclusion. :-) You can argue that nonlocal is about lifetime instead of visibility until you're blue in the face, but I still disagree. It is first and foremost about visibility. Lifetime is secondary. Note how a nonlocal variable may actually have a lifetime longer than that of the function using it that declares it as nonlocal -- that function object may be deleted from the containing scope, but the variable of course still exists in the containing scope. -- --Guido van Rossum (python.org/~guido)

How about "eustace" (from Eric Frank Russell's "Next of Kin")? You're declaring the variable to be a eustace for this function. "bopamagilvie" would be better, but is a bit long. <mike

On Tue, Sep 27, 2011 at 1:10 PM, Guido van Rossum <guido@python.org> wrote:
I actually changed my mind on that front when replying to Greg - I now agree that visibility is the important aspect, but I also think there are two levels of visibility within a function: local scope, which is distinct for every call to the function, and "function scope", which is shared across all invocations to the function. This difference is reflected at the implementation level in where the references are stored (i.e. as locals on the frame in the first case, as references from the function object in the latter). Currently the only references which live in that second scope without also being visible outside the function's own lexical scope are default arguments, and those get bound to ordinary local references when it comes to actually accessing them from the body of the function. Accordingly, even after conceding the visibility vs lifetime point, I'm still in favour of extending nonlocal downwards to include "function scope" in addition to its current usage for lexical scoping. Finding a different word for "not as local as a local variable, but not as nonlocal as a lexically scoped variable" has proven to be more than a little challenging and the flat negation of "nonlocal" suggests that any addition would be at least as ill fitting as "global" is at the module end of the scale. Unfortunately, what I see as a minor tweak to the meaning of nonlocal, yourself, Paul and Greg seem to see as referring to a profoundly different concept. While that remains the case, I doubt I'm going to have much luck persuading any of you as to the reasonableness of the syntax. I do find it interesting that at least some of the possible keywords that have been suggested for the function scope use case were *also* put forward as possibilities in the context of PEP 3104 (which eventually gave us 'nonlocal') [1]. I'm not sure how much significance to ascribe to that fact, but I do think it provides at least some level of support for my thesis that the line between the two concepts of function scoping and lexical scoping is blurrier than it first appears. Regards, Nick. [1] http://www.python.org/dev/peps/pep-3104/#id14 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 27 September 2011 20:53, Nick Coghlan <ncoghlan@gmail.com> wrote:
That description just makes my head hurt. To me, visibility is simply which lines of code, when using a name, refer to that particular variable. So a variable x at module level has "global scope" because it is visible from all lines (in the module, ignoring shadowing concerns). If x is defined within a function, it has local (to the function) scope, because only lines within the function can see that variable. In other words scope is essentially a lexical concept tied to lexical visibility. To that extent, these new closed-over values have local scope because they are visible from the same lines of code as a local variable. But their lifetime differs. The ideas here are more commonly seen in the history of Lisp, which originally had "dynamic" scope (which is more lifetime-oriented) but moved to lexical (visibility-based) scope. There was a time when Lisps were described as Lisp-1 or Lisp-2 depending on their position on scope. I need to go and do some research into old Lisp concepts to see how they might apply here. I may be some time :-) Paul.

On Tue, Sep 27, 2011 at 2:15 PM, Paul Moore <p.f.moore@gmail.com> wrote:
Perhaps that is why people sometimes miss that default argument values persist across calls. The names are locals, but the values live in an often unconsidered (and relatively artificial) "function definition scope". The variables in the proposed nonlocal-from statement are in that same function definition scope implicitly. However, unlike other names, they're bound to their scope by a statement in a different scope (the local one). Also, since they are variables they would be stored as cells (like __closure__ does). This is another contrast to default arguments, which are stored only as values (in __defaults__ and __kwdefaults__). The function definition scope isn't the only source of confusion here. In one context "nonlocal" would mean "not local (and not global)". In another context it would mean "in function definition scope". Here's a different angle to show what I mean (I'm not proposing this). Currently, if a nonlocal statement refers to a name that is not bound in an enclosing function scope, it gives a syntax error:
If that situation instead implied "function definition scope", you would get a similar effect to nonlocal-from: persistent state for select variables across calls. However, the behavior would be consistent across all uses of nonlocal. "nonlocal x=5" (if allowed) would be the same as "nonlocal x; x=5". You'd lose the ability to set a one-time initial value, but the point is that nonlocal would become the means to interact with variables in the function definition scope. Ultimately, it seems like we are looking for a way to do three things here: 1. bless a concrete function definition scope (persist across calls), 2. allow one-time initialization of those persistent variables, 3. (relatedly) have some expressions evaluated at definition time but available at execution time. Default arguments cover some of this. Closures cover some of it. The nonlocal-from statement covers a bunch. -eric

On Tue, Sep 27, 2011 at 4:24 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
As do function attributes... However, the big motivator here is execution performance where it matters: in functions.
-eric

I started to write this mail saying that we could use spellings like nonlocal.scoped, nonlocal.own, nonlocal.global and nonlocal.const but then read the message about declarations in the middle of a function having side effects at definition time. That seems like a big problem. And it made me realize it would be weird if (to use one possible bikeshed color): own x = y was not the same as own x x = y but I don't see how these can possibly be the same. You need the first case to do one time initialization and you must be able to change the value as in the second case or the feature is useless. I proposed a special decorator-like syntax ($inject) which would pull the declaration out of the function entirely (putting it at the scope where it evaluates). There wasn't much discussion other than that it wasn't feasible with current implementation. --- Bruce

On 28 September 2011 01:56, Bruce Leban <bruce@leapyear.org> wrote: [...]
A hybrid approach would be possible, taking from both approaches (decorator and keyword): * a keyword to declare in the body of the function that a variable is of the "own" kind. * function objects would grow an __own__ attribute which is a dict-like object: - f.__own__["i"] would return the contents of the cell that the own variable "i" refers to - f.__own__["i"] = 42 would set the cell contents of the own variable "i" to 42 (or some other equivalent mechanism) * then one could create a kind of "inject" decorator. Here's an example: def setown(**kwargs): def decorator(f): for k, v in kwargs.items(): f.__own__[k] = v return decorator @setown(i=42) def foo(x): own i i += 1 return i -- Arnaud

On 2011-09-28 07:23, Arnaud Delobelle wrote:
I like this idea, but I think we should at least consider using "nonlocal" instead of "own" as the keyword (and __nonlocal__ instead of __own__ for the dict-like object). Pros: - No new keyword needed, just a change in how "nonlocal x" is treated when there is no 'x' variable in the defining scope. (Today this is a SyntaxError). Cons: - No way to define an "own" variable with the same name as a variable in the defining scope. - Ambiguity as to exactly what the lifetime of the nonlocal is. (If an outer function with no "x" variable in scope defines two inner functions that both use "nonlocal x", do they see the same "x"?) Ok, so I don't actually like repurposing nonlocal for this. +1 to your "own" suggestion. - Jacob

On Wed, Sep 28, 2011 at 1:23 AM, Arnaud Delobelle <arnodel@gmail.com> wrote:
Any reason not to just use the function itself, as opposed to a designated dictionary? Right now, we don't have a good way to refer to a function either before it defined (decorator time) or while it is being defined, but that seems like a subset of what you would need for the above proposal. (Of course, I did like the functionality of PEP 3130.) -jJ

On 28 September 2011 01:56, Bruce Leban <bruce@leapyear.org> wrote:
Equally to the point, if a function has own x = 1 x = 1 it is *not* valid to remove the second line, as it runs at a different time (runtime rather than define time). That is very weird. Between these points and Arnaud Delobelle's point that code inside a function should do nothing when the def itself is executed, I'm getting more convinced that objects with persistent local scope should be introduced *outside* the function body. That either validates the default-argument hack as a valid response to a specific requirement, or suggests some syntax added to the function definition line. Or of course, just go with a normal (nested function) closure. Paul.

On Wed, Sep 28, 2011 at 3:10 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I'll point out that there's one thing the default argument hack *doesn't* do that is relevant to the discussion: because it doesn't actually move the name into a separate scope (only the value), it has no effect on rebinding. It's a lot like closures in the days before nonlocal declarations: def accumulator(): def incr(x, *, _tally=0): _tally += x return _tally return incr That doesn't actually work, since _tally is an ordinary local that gets reinitialised to zero on each invocation. The only syntax suggested so far that meets the criteria of being both in the function header and creating a separate namespace for the namebindings is the one I put forward in Ron's previous thread (I've excluded the 'after-**' syntax here, since this thread has convinced me it would be crazy to use something like that when the name binding semantics are so different from those of a default argument): def accumulator(): def incr(x) [tally=0]: tally += x return tally return incr As the rough equivalent of today's: def accumulator(): tally = 0 def incr(x): nonlocal tally tally += x return tally return incr The objections to that proposal that I recall were: 1. Can't look up [] in the help index (Agreed, but you can look up 'def' which is where these function state variables would be documented) 2. Noise in the header line (Agreed, but no worse than the default argument hack) 3. The behaviour is very different from a list (Agreed, but function parameters and arguments are very different from tuples and people can cope with that) 4. It should be in the function body (I think Arnaud and Paul have raised some very good points in this thread for why it *shouldn't* be in the function body) As far as the 'hidden state' problem goes, I'd prefer to address that by providing better utilities in functools for accessing closure state (and perhaps even the current internal state of suspended generators). As I said in the previous thread, the "algorithm with shared state" model represented by closures (and, with a slightly different spin on the concept, generators) and the "data with behaviour" model represented by object-oriented programming are very different ways of framing a problem and forcing people to switch models because of internal language issues unrelated to the real world problem they're trying to describe isn't a good thing. I agree closures have testability problems, but I don't think telling people "avoid closures because they have testability problems" is the right answer to that. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Sep 28, 2011 at 5:59 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Oops, s/functools/inspect/. (The original version of that sentence suggested read/write access to the closure state, so functools made more sense, but inspect is a better choice for read-only access) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sep 27, 2011, at 11:59 PM, Nick Coghlan wrote:
Brackets feel too much like line noise… A keyword? def incr(x) and rebind 0 as tally: ? def incr(x) with static tally = 0: ? Just tossing out ideas…

On Wed, Sep 28, 2011 at 6:14 AM, Carl Matthew Johnson <cmjohnson.mailinglist@gmail.com> wrote:
Hah, def statements are already line noise if you use the full syntax :) @what def theheck(do:all, these=1, *different, symbols, **mean) -> seriously: """And this is special, too!""" I think the combination of position with the different bracket style is enough to separate them out from the parameter list and their position in the function header eliminates much of the confusion associated with the various syntax proposals for state variables inside the function body. I actually think it also serves as a mild deterrent to abuse - code that overuses function state variables is likely reaching the point where it needs to be rewritten as a class, and this would be reflected in the unwieldiness of the function definition. Cheers, Nick. [1] http://docs.python.org/py3k/reference/compound_stmts.html#grammar-token-func... -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sep 28, 2011, at 12:27 AM, Nick Coghlan wrote:
Slightly OT, but does anyone else feel like function annotations are a bit of a failed experiment? Or have you guys seen them in a real project, not just a demonstration? I find the idea reasonable enough, but I've never thought to use them for anything serious. I wonder if we need to learn any lessons from that experience… The counterpoint here is that the defaults hack is something one often runs across, so a replacement is likely to have decent pickup.

Other than making my pocket vibrate every 5 mins at work, where is this discussion headed? is there a PEP I can get up to speed with?

Nick Coghlan dixit (2011-09-28, 05:59):
IMHO such a syntax (or a special or non-special decorator) should simply allow to add free variable(s) to a function. Then, if we want to modify such a variable in the function body we should use nonlocal, as with today's closures: def accumulator(): def incr(x) [tally=0]: nonlocal tally tally += x return tally return incr But we would not need to use nonlocal for read-only access, as with today's closures: def my_func() [lock=Lock()]: with lock: "foo" Cheers. *j

On 28 September 2011 08:10, Paul Moore <p.f.moore@gmail.com> wrote:
Taking a step back from all this, using an explicit closure to implement the counter example being used throughout this thread looks something like this:
To be honest, that doesn't actually look that bad. Explicit is better than implicit and all that, and the boilerplate doesn't really bother me that much. The real annoying boilerplate here is the "def dummy_name()...return dummy_name" bit. But fixing that leads us directly back to the perennial discussion on anonymous multiline functions... Paul.

On Wed, Sep 28, 2011 at 6:26 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Actually, there are some additional aspects that annoy me: - the repetition of the variable name 'n' - the scope of the variable name is wrong (since we only need it in the inner function) - indentation matters (cf try/except/finally) - hidden signature for the actual function - hoops to jump through to get decent introspection values (e.g. __name__) I find the following 5 line toy example: from threading import Lock def global_counter() [n=1, lock=Lock()]: with lock: print(n) n += 1 Far more readable than the 10-line closure equivalent: from threading import Lock @apply # I sometimes wonder if we should bring this back in functools... def global_counter(): n = 1 local = Lock() def global_counter(): with lock: print(n) n += 1 return global_counter Or the 10-line 7-self class equivalent: from threading import Lock @apply class global_counter: def __init__(self): self.n = 1 self.lock = Lock() def __call__(self): with self.lock: print(self.n) self.n += 1 Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 September 2011 11:45, Nick Coghlan <ncoghlan@gmail.com> wrote:
Fair points, all (except the scope one, the fact that the scope extends to cover a couple of lines of boilerplate doesn't really bother me, but I concede that you're a purist in these matters :-))
Agreed entirely. If we're back to talking about line-noise syntax rather than using "nonlocal" I'm with you all the way (including the original bit about it still being a large change for a relatively small benefit...) Paul.

On Wed, Sep 28, 2011 at 3:45 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hm, this syntax is *too* concise. It's lack of keywords makes it open to misinterpretation.
True, the outer function completely obscures what's going on.
Yeah, anything involving __call__ is hard to figure out. So after reading some previous messages, I almost wish we could implement the "decorator + nonlocal statement" proposal: @init(n=1, lock=Lock()) def global_counter(): nonlocal n, lock with lock: print(n) n += 1 Alas, the problem is that you can't actually implement such a decorator, not even using an extension module, because the compiler, upon seeing the "nonlocal n, lock", ignoring the decorator (since it should be applied at run time, not at compile time), will complain that there isn't actually an intermediary outer scope defining either n or lock. Some ways out of this: - let the compiler special-case a certain built-in decorator name at compile-time (unorthodox, not impossible) - different syntax, e.g. @[n=1, lock=Lock()] or @in(n=1, lock=Lock()) or even in n=1, lock=Lock() (all followed by "def global_counter() etc.") Of course once there's different syntax, the nonlocal declaration in the function is redundant. And clearly I'm back-peddling. :-) -- --Guido van Rossum (python.org/~guido)

On 28 September 2011 16:38, Guido van Rossum <guido@python.org> wrote:
Of course once there's different syntax, the nonlocal declaration in the function is redundant. And clearly I'm back-peddling. :-)
If we're back to syntax proposals on the def statement, how about def fn() with i=1, lock=Lock(): whatever ? This is basically another bikeshed to paint, though... Paul.

On 28/09/2011 16:58, Paul Moore wrote:
I'd already arrived at the same solution! :-) The complaint about putting "static" or "own" in the body of the function is that it's not clear that it's evaluated at definition time. The complaint about putting it outside the function is that it's not clear that the name is visible only inside the function (and it's also harder for the compiler to recognise it and optimise for it). The only place where the expression can be evaluated at definition time and bound to a name which is visible only inside the function is in the function's header. We don't want it in the parameter list, therefore it can go after the parameter list.

On Sep 28, 2011 8:58 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:
I also had the same idea with: def fn() with i=1, lock=Lock(): Whatever So I guess it's not unobvious. Though "with" is used for something different unless we disallow def fn() with NAME = EXPR: and instead use def fn() with CONTEXT as NAME: And the existing enter/exit mechanism for initialization. Though I agree with the protesters that this construct can be an attractive nuisance or an "anti pattern" as Greg Ewing said. Though I haven't yet been able to articulate in which cases. --Yuval On Sep 28, 2011 8:58 AM, "Paul Moore" <p.f.moore@gmail.com> wrote:

On Wed, Sep 28, 2011 at 11:58 AM, Paul Moore <p.f.moore@gmail.com> wrote:
The PEP 3150 discussion on keywords is relevant to that kind of keyword-based proposal (there's a reason the statement local namespaces proposal suggests 'given' rather than 'with' or 'where') Cheers, Nick -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

This use-case is no good. Consider: _n = 1; _lock = Lock() def global_counter(): global _n with _lock: print(n) n += 1 Devin On Wed, Sep 28, 2011 at 11:38 AM, Guido van Rossum <guido@python.org> wrote:

On Wed, Sep 28, 2011 at 12:39 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
And consider what happens as soon as you have two module level functions that you want to assign a lock to - you end up have to use clumsy name hacks to avoid collisions in the module namespace. I'll stick with this as my toy example (although I forgot to actually acquire the lock in my last post) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Surely the solution is a new namespace (that's still publicly accessible)? It's absolutely pannoying that you have to prefix the globals like in C, but deftime-assigned nonlocals are less convenient in other respects too. e.g. try sharing a lock among multiple functions, or try testing the usage of a lock, etc. Devin On Thu, Sep 29, 2011 at 8:03 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On Thu, Sep 29, 2011 at 1:10 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Sharing state amongst multiple functions is a solved problem - you use either a closure or a class instance to create a shared namespace, depending on which approach makes more sense for the problem at hand. However, both of those solutions feel very heavy when all you want to do is share state across multiple invocations of the *same* function. Hence the current discussion (and the numerous ones that have preceded it over the years). It's a tricky problem precisely because it doesn't take much to tip any given use case over the threshold of complexity to where it's a better idea to use a full-fledged closure or class. And, as I have said several times, I agree closures currently impose testability problems, but I think the answer there lies in providing better introspection tools to lessen those problems rather than advising people not to use closures specifically for those reasons. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

OK, that's fair. In order to support testing you'd need to be able to assign to closure variables (for e.g. mocking) and read them (to test alterations to global state). At that point it's a short leap to defining a decorator that creates these nonlocals. Is there a compelling reason to use a new syntax? Devin On Thu, Sep 29, 2011 at 1:26 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On Thu, Sep 29, 2011 at 1:41 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
I don't think supporting monkey-patching is necessary - the code under test will presumably provide some mechanism to control how the hidden state is initialised or modified. The missing part from a testing point of view is the ability to get that information back out in a clean, portable way so you can make assertions about it for whitebox testing. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I don't think supporting monkey-patching is necessary
Nothing is necessary. I maintain it is extremely useful for testing. There are entire libraries built around monkeypatching code for the purpose of testing it. For example, recently I saw Exocet, which allows you to create a new module object but replacing the imports with whatever you want.(The analogue here would be copying a function, but with new closure variables). Without this you can't really _unit_ test a closure, at least as I understand it. It becomes inextricably linked with its scope, which is not what you want to test. Devin On Thu, Sep 29, 2011 at 2:03 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On Thu, Sep 29, 2011 at 2:25 PM, Devin Jeanpierre <jeanpierreda@gmail.com> wrote:
Improving introspection of closures doesn't magically remove the need to design for testability. Some ways of using closures are fairly easy to test (e.g. you just call the outer function again with monkey-patched globals and get a monkey-patched closure). Others are not (you define excessively complex state in the closure without provide a way to replace it for testing purposes). If the amount of state in a closure is causing serious testing problems, it's a hint that the code is attempting to handle a "data with behaviour" situation more suited to an OOP style rather than an "algorithm with state" that is particularly amenable to modelling with closures. Providing a nicer API for features the language already supports to simplify whitebox testing is one thing, providing a completely new feature (i.e. modifying closure references from outside the function) is something else entirely. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Wed, Sep 28, 2011 at 9:38 AM, Guido van Rossum <guido@python.org> wrote:
I like the idea of a decorator approach too, though just normal decorators. Here's another alternative. It could require exposing function internals a little (impacting the language definition?). There are two distinct outcomes we could pursue: def-time values (like default arguments), and def-time variables (static-like). The Def-time Value Approach ------------------------------------ For def-time values, the values would have to be stored in a function attribute similar to __defaults__ and __kwdefaults__. Let's call it __spam__. A keyword (not "nonlocal" <wink>) in the function body would indicate that the name should be initialized with each call to the appropriate value in __spam__. Then we would have a (builtin?) decorator factory that would update values in __spam__: #<example> def init_deftime(**kwargs): def decorator(f): for name in kwargs: index = figure_out_index(f, name) f.__spam__[index] = kwargs[name] #functools.update_spam(f, name, kwargs[name]) return f return decorator @init_deftime(x=5): def f(): some_keyword x print(x) # prints 5 # Currently using default arguments: def f(x=5): print(x) #</example> As suggested already, the keyword simply indicates to the compiler that x should be a local and it should be initialized from __spam__. Also, if the decorator is not used to initialize a name marked by some_keyword, then it would either just use some default value or raise some exception (just after all decorators have been applied or at call time). The Def-time Variable Approach --------------------------------------- Perhaps we want static-like variables in functions instead. Unless I misread the more recent posts, such variables are a no-go. Just in case, I'll summarize a decorator solution. Basically, all we do is indicate through a keyword that a name will be found in the "function definition scope". Then closure happens like normal. The trick is to reuse nonlocal and add the function definition scope as the implicit default scope. Currently you get a syntax error when nonlocal refers to a name that is not found by the compiler in any enclosing function scope. This would change that. The decorator would be the mechanism for initializing the variable: #<example> def init_deftime(**kwargs): def decorator(f): for name in kwargs: functools.update_closure(f.__closure__, name, kwargs[name]) return f return decorator @init_deftime(x=5): def f(): nonlocal x print(x) # prints 5 #</example> Again, if the nonlocal name is never initialized (and not bound-to in the body), either a default would have to be used or an exception raised. Wrap-up ---------- So there's a decorator approach. For me it's a toss-up between decorators and the bracket syntax in the signature that Nick suggested. I find the deftime-value-in-the-body proposals not nearly as clear. I haven't had much use for static-like variables so I prefer def-time value approaches. Of course, the simplest solution is to do like Terry (et al.) suggested and just encourage the use of "private" default args. -eric

On 28 September 2011 19:08, Eric Snow <ericsnowcurrently@gmail.com> wrote: [...]
Mmh. This looks exactly like one I posted earlier on: http://mail.python.org/pipermail/python-ideas/2011-September/011892.html -- Arnaud

On Wed, Sep 28, 2011 at 1:38 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:
Yeah, very similar. Somehow I missed your post earlier. It seems like a decent approach. It is essentially a variation on how default arguments work right now, using the decorator instead of the function signature. -eric
-- Arnaud

Guido van Rossum dixit (2011-09-28, 08:38):
And what about: @nonlocal n=1, lock=Lock() def global_counter(): "we don't need to repeat nonlocal in the body" or @nonlocal(n=1, lock=Lock()) def global_counter(): "we don't need to repeat nonlocal in the body" ? Cheers. *j

Paul Moore wrote:
def f(x=1): x = 1 has the same behaviour. I don't think either case is weird.
I don't think having code inside a function execute is any worse than having code inside a class execute. class K: print("this is executed at definition time") def f(x=print("this is also executed at definition time")): own y=print("and so is this") To say nothing of decorators. -- Steven

On 28 September 2011 11:42, Steven D'Aprano <steve@pearwood.info> wrote:
You can't compare the two. When a class statement is executed, the *whole* of its body is executed; when a def statement is executed, *none* of its body is. -- Arnaud

Paul Moore wrote:
Actually that distinction is whether there is a separate namespace for functions or not, see http://en.wikipedia.org/wiki/Lisp-1_vs._Lisp-2#The_function_namespace -- Greg

On 28 September 2011 12:21, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Pah. Fading memory. Dynamic vs lexical scoping was in there somewhere, though, just not by the names I thought :-) Paul

On Mon, Sep 26, 2011 at 8:03 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Yes, I did state that for default arguments it referred to the anonymous values. The only name currently in this category is the magic __class__ reference in the new super() implementation. Part of the idea here is to make it so that that name is less of a special case.
Actually, with my proposal it would always refer to lifetime, and only sometimes to visibility. The latter case would be implied by the fact that it isn't initialised inside the current function definition (currently you *can't* initialise it locally, so the term always refers to both lifetime *and* visibility). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 9/26/2011 10:24 AM, Paul Moore wrote:
In the template below, the visibility of VAR in _outer is both ephemeral (lasting only for one quick call) and non-essential (in that VAR is never *used* within _outer. Today's default value objects *are* visible non-locally:
def f(a=3): pass
f.__defaults__[0] 3
They optionally get bound to local names but they are *not* local values any more than cell values are. Lifetime and visibility are actually coupled. You cannot see something that does not exist; invisible objects tend to be garbage collected and cease to exist.
-- Terry Jan Reedy

On Mon, Sep 26, 2011 at 4:01 PM, Terry Reedy <tjreedy@udel.edu> wrote:
Indeed, it could be said that the defining feature of a nonlocal reference is that it is accessible via the function object, whether that's through __defaults__, __kwdefaults__ or __closure__. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Sep 26, 2011 at 2:17 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
First of all, I like the idea. I do agree that nonlocal is not the optimal name (neither is static). However, the concept matches well enough and it's not like nonlocal is used so much that everyone would have to rewire their brains. I do have some questions. Would the value from the nonlocal-from be stored in a cell in __closure__ or in a new special attribute? Would the name be stored in co_freevars or somewhere else? I realize that's an implementation detail and perhaps it's premature to worry about. Still, I'm curious about if you'll be able to distinguish this kind of value from existing ones when inspecting function and code objects. How does this impact the other Python implementations? I would think not much, since it's not that different from how closures are already handled (if I understood the idea correctly). As Steven already pointed out, this may allow the actual function object to be referenced in the function body, but not by default. Sounds good. However, this hinges on the point at which the cell is created relative to different results of the compilation process (like the decorator calls). From what I understand, this should not be hard to take care of. Would it be a big deal to make sure a function's self-referencing nonlocal-from would work? As Terry noted, default argument values are sort of read-only. A function's __defaults__ (and __kwdefaults__) is writable and bound to a tuple. The local name that maps to each value in __defaults__ is re-initialized for each call. The proposed nonlocal-from syntax would not reflect these characteristics. Instead it is more of a persistent/static mechanism, just like closures. So is the nonlocal-from an alternate closure syntax inspired by the default argument hack, or should it actually behave more like default arguments do? -eric

On 9/26/2011 6:23 PM, Eric Snow wrote:
Oh, right, I keep forgetting that. The tuple of defaults in immutable but in current Python, at least, the tuple is replaceable. So a function can indirectly rewrite its defaults as long as it is still bound to its definition name.
For a default that is intended to be permanent and never overshadowed, there is no need to rebind with every call. -- Terry Jan Reedy

Paul Moore wrote:
Same here. What's more, when I think about what 'nonlocal' means at a conceptual level, I'm not thinking about cells attached to function objects. I'm thinking about reaching out to something pre-existing outside of the function. In contrast, the proposed feature would be creating something new, and putting it somewhere that the programmer can't even see.
But readability matters, and I worry that this isn't "readable". Maybe the fact that neither of us is Dutch is relevant here, too :-)
But Guido doesn't appear to like it either, so Dutchness does not seem to be a sufficient condition for appreciating this brand of logic. :-( -- Greg

On Mon, Sep 26, 2011 at 6:47 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Adding new keywords is a big, big step that demands a compelling justification.
Agree.
You can always abuse an existing keyword to avoid adding a new one. For example, this could instead be pass x in from expression for x from expression with x from expression etc. These would surely be rejected because the keywords are used with an entirely different meaning and that makes the language harder to read and write. The keyword nonlocal means that this binding is not local to this scope but can be found up the call stack. In contrast, your usage means the binding is local to this function, created before the function is called the first time and shared with all calls to this function. Those are orthogonal scopes. --- Bruce

The keyword nonlocal means that this binding is not local to this scope but can be found up the call stack.
Lexical stack. (The call stack would be dynamic scope. [I suspect you already know this; I am pedant
Not really. Created differently, yes. But after creation it works identically, by definition. However, this is not what "nonlocal" means to me, and from what others have said, it's not what it means to them either. Devin On Mon, Sep 26, 2011 at 12:27 PM, Bruce Leban <bruce@leapyear.org> wrote:

On Mon, Sep 26, 2011 at 10:16 AM, Devin Jeanpierre <jeanpierreda@gmail.com>wrote:
Yeah, I know that but a perfect example of how easy it is to get confused. :-)
To a python developer, sure. To a python programmer, I don't think so. --- Bruce

Bruce Leban wrote:
Another way of looking at this is that the keyword nonlocal means the name is bound in a surrounding scope; which is still true with Nick's proposal. Nick's proposal has the added benefit of fast lookup. nonlocal also carries with it the implication of persistence, which Nick's proposal also has (indeed, it's one of the main points). ~Ethan~

On Mon, Sep 26, 2011 at 12:27 PM, Bruce Leban <bruce@leapyear.org> wrote:
See my reply to Paul - while I agree that visibility and lifetime are different aspects of named references, it isn't like they're independent. A nonlocal reference from an inner function has a direct effect on the lifecycle of an otherwise local variable in the outer function. Consider the following two functions: def f(): x = 0 return x Here, 'x' is an ordinary local variable - it only exists while the frame is executing. def f(): x = 0 def inner(): nonlocal x x += 1 return x return inner Now, the use of 'nonlocal' in the inner function has *changed the lifecycle* of x. It is now a nonlocal variable - it survives beyond the execution of the function that defines it. Paul was quite right to point out that most developers only think about the visibility aspect of 'nonlocal', since they're looking at the impact from the point of view of the inner function, but that doesn't mean the lifecycle impact on the outer function that I want to highlight is arbitrary or irrelevant. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Yes, but that's just a logical consequence of the fact that it's *visible* from the inner function, together with the fact that Python never throws anything away while it's still accessible. Visibility is still the primary concept attached to 'nonlocal'. -- Greg

On Tue, Sep 27, 2011 at 1:15 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Hmm, I think you just summarised the key insight that led to me suggesting nonlocal in the first place (even though I wasn't thinking of it in exactly these terms at the time). A local name binding is visible only from the current *invocation* of a function, hence it only lasts until the call returns. On the other hand, an explicitly 'nonlocal' name binding is visible from: - the current invocation of the outer function where the name is defined - all invocations of any functions defined within that outer function As you note, the changes in lifecycle then follow from the change in visibility - as long as any of those inner functions survive, so does the name binding. Non-local name bindings can of course also happen implicitly if there's no local assignment to override. My proposal is merely that we add a *new* meaning to 'nonlocal' that supplements the existing meaning, to declare that a name is visible from all invocations of the function currently being defined rather than being local to each invocation. As with existing nonlocal usage, the change in visibility (all invocations vs current invocation) then leads directly to a change in lifecycle (linked to the lifetime of the function object rather than until an individual call returns). The 'nonlocal' terminology could also be seen as referring to the evaluation scope for the initialisation expression, since it would run in the surrounding scope, just like default argument values. Since the new semantics slots in neatly between actual locals and the current usage of 'nonlocal', and because 'nonlocal' itself is such a generic term, it just seems easier to me to teach people about the link between visibility and lifetime and two possible meanings for nonlocal than it would be to: 1. find a new term (we've been trying and failing at this for years now) 2. teach people what it means Step 2 doesn't get any easier just because we use a different name - if anything it gets harder, since people are already used to nonlocal as a modifier on name visibility, and the new term will need to indicate a closely related concept. Is giving 'nonlocal' a second related-but-not-identical meaning a perfect answer? No, I don't think so. However, when generators were added, the 'def' keyword went from unambiguously declaring a function to declaring either an ordinary function or a generator-iterator factory and people seemed to cope. 'yield' and 'yield from' don't mean exactly the same thing, but I expect to be able to cope with that as well. If someone can understand the concept of function scoped variables at all, then they'll be able to understand the difference between 'nonlocal VAR' and 'nonlocal VAR from EXPR'. Given past history, I seriously doubt our ability to come up with a keyword for function scoped variables that is any more intuitive than a variation on nonlocal. We would then have the following hierarchy of visibility: Local scope (VAR = EXPR): visible from current function invocation Function scope (nonlocal VAR from EXPR): visible from all invocations of this function Lexical scope (nonlocal VAR; VAR = EXPR): visible from outer function where VAR is a local and all invocations of this and any peer functions Module scope (global VAR; VAR = EXPR): visible from all code in the current module Process scope (import builtins; builtins.VAR = EXPR): visible from all code in the current process For name *lookup*, lexical scoping, module scoping and process scoping can all happen implicitly when a name doesn't refer to a local. Function scope lookup would only happen with an explicit declaration (regardless of how it was spelt). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 27 September 2011 10:35, Nick Coghlan <ncoghlan@gmail.com> wrote:
I think I'm starting to get o the point where I understand "nonlocal VAR from EXPR" when I see it. But that doesn't mean it's becoming clearer, rather it means that I'm learning to mentally translate it to "static VAR = EXPR" - or own, or whatever - the concept doesn't have a clear name to me, which is why choosing a keyword isn't easy, but whatever it is, it's not really internalised as "nonlocal". On the other hand, I do (sort of) understand "nonlocal VAR" as meaning "look outside for VAR" directly. Only somewhat, because I don't think "nonlocal" is a particularly good name even for that concept, but that ship's sailed.
Nice summary, but it does emphasise that function scope is (still) an odd one out, as it's not available for lookup without a declaration. So we're close, but things still aren't completely unified. And yes, I know that lookup semantics are so baked into the language that they aren't going to change - that's the point, think of it the other way round, that declaring variables as function-scope is odd, because it's the only one that doesn't have an implicit form for lookup semantics of code later in the function. Meh. I tried to give an example of what I mean, to make it clearer, but I can't quite get one to work right now, and I need to go. I'll let it stew for a while and try to clarify a bit more later. Paul. Paul.

On Tue, Sep 27, 2011 at 12:03 PM, Paul Moore <p.f.moore@gmail.com> wrote:
It's actually (kind of) the same in C. Names come from the current or surrounding scope, or are retrieved from another module by the linker, with static used as a modifier to alter lifetime or visibility. At the module level in C, "static VAR = EXPR;" doesn't change the lifecycle of VAR, but it does change the visibility by hiding it from the linker. Inside a function, it doesn't really change the visibility (you can still only access it from code inside the function), but it does alter the lifecycle by sharing the attribute across all invocations of the function. (Since C doesn't have nested functions, such variables effectively become process global). That's why I really don't like 'static' as the keyword - it has two different meanings in C, and C++ added a third by using it to denote class attributes. Python already adopted that third meaning when Guido created 'staticmethod', so I'm not a fan of also using it for 'static' in the "function state shared across invocations" sense. (Although, if we *did* go that way, I'd probably describe it as "a static method remains the same regardless of which instance you retrieve it from and a static reference remains the same across all invocations of the function") If we're going to settle for a term with multiple meanings, then I still think 'nonlocal' is the one that makes the most sense: - it's already a keyword - the scope to be described lies between 'local' and the existing meaning of 'nonlocal' on the visibility scale - it is suggestive of the fact that the initialisation expression is not evaluated in the local scope - the two usages would relate to different aspects of function closures and use the same underlying implementation at call time All that 'nonlocal' really says is 'not a local'. Since a function scoped variable *isn't* a local variable, I don't understand the preference for the even more vague 'static'. I know which one I'd prefer to try to explain to someone learning Python that didn't already know C :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Before I start this, please note that I am not in favour of "static" as the keyword to use here. I agree it has the wrong implications. Although my C-based instincts make it feel right, which implies to me that there's a lot more to this whole debate than purely logical arguments over what is "right". People's tastes and instincts are crucial here, for better or worse. On 27 September 2011 18:03, Nick Coghlan <ncoghlan@gmail.com> wrote:
To be honest, I think you need to stop raising this. You're right, and it's certainly a lot easier to accept reusing an existing keyword than adding a new one. But you need to convince people that "nonlocal" is right, not that it's easier. You're at risk of giving a distinct "I've got a hammer so it must be a nail" impression here...
- the scope to be described lies between 'local' and the existing meaning of 'nonlocal' on the visibility scale
Or, to put that more negatively, "it's different from the existing meaning of nonlocal". I see your point, but the argument's a bit weak.
- it is suggestive of the fact that the initialisation expression is not evaluated in the local scope
Not really (at least not to me). Nonlocal is associated (by its position) with the VAR, not with the EXPR. If anything, it says "assign this local EXPR to the nonlocal VAR" which implies it's equivalent to "nonlocal VAR; VAR = EXPR".
- the two usages would relate to different aspects of function closures and use the same underlying implementation at call time
I don't really follow this - it's too obscure to be compelling, and on an initial reading, the first thing I noticed was "different aspects". Anyway, I'd rather similarity of end user experience dictate similarity of keyword than similarity of implementation...
Agreed. Personally, I'm not trying to argue that "static" is better than "nonlocal". To be precise over my position: 1. I like the functionality, it's a minor improvement, but it seems to fill a gap. (Although it is minor enough that I doubt I'd use it much in practice, which makes me wonder if I'm qualified to have an opinion anyway...) 2. I don't think it's important enough to warrant a new keyword. If there was an "obvious" existing keyword, I'd support it like a shot. But otherwise, unless Guido suddenly says he'd agree to a new keyword for this, I agree we're stuck trying to make existing keywords work if we want the functionality. 3. Intellectually, I can see your arguments as to why "nonlocal" is logical. But I never really liked the term in the first place, and my instinct is that it doesn't work for this new usage, regardless of logic. I keep trying to like it, though (see points 1 and 2 :-)) 4. I'm getting more and more able to simply read "nonlocal VAR from EXPR" as you intend it. But that's just familiarity - I still don't have a proper term to describe the concept (hence my use of the phrase "as you intend it" - I couldn't think of anything natural in the way that "as a global variable" feels). Oddly enough, now that Guido referred me back to the Algol-60 "own" variable concept, I can see why that name fits now (it's one of the function's "own" variables, not belonging to any other scope) and I'm starting to think of these as "own variables" in my mind. That's only appropriate for old fuddy-duddies like me (or experienced language designers like Guido!) though :-) Overall, I'm probably -0 on using "nonlocal". I'm +1 on the functionality, but if it needs a new keyword, that trumps my +1. Paul.

On Tue, Sep 27, 2011 at 2:46 PM, Paul Moore <p.f.moore@gmail.com> wrote:
That comment gets us back to the "nonlocal EXPR as VAR" syntax. That one looks really odd when used with constants though (what's a nonlocal zero?), and I'm not sure the parser could handle it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Sep 27, 2011 at 10:18 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Indeed, although there was also a teensy bit of hyperbole involved :) Arnaud and Paul's examples have persuaded me that embedding this inside the function isn't a good idea, though: x = 42 def f(): # Folded by editor (or even just mentally) assert x == 42 # May fail! def f(): statelocal x = 1 x =1 We'd been considering the latter as arguably tolerable all the way through, but the combination with making the function body relevant when understanding the effect of def statements on the scope that contains them was enough to tip the balance for me. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Paul Moore wrote:
Something that might be slightly more suggestive of Nick's meaning is i = nonlocal i but then there's no indication that i isn't just an ordinary local being assigned at call time. -- Greg

Nick Coghlan <ncoghlan@...> writes:
You had me, you really did. Right up until you showed the current equivalent. This strikes me as a few things. Most importantly, as you noted yourself, a pretty rare case, even in C static variables are probably the rarest scope of variable. This strikes me as a) not saving very much code, it's like crappy HFS instead of real sugar ;), and b) not adding fundamental value, I think both blocks of code are equally readable. Other examples of syntatic sugar, such as decorators, have code motion properties that let you think about code in the places that makes sense, and I don't think this has that. Alex

On 2011-09-26, at 14:26 , Alex Gaynor wrote:
An other thing which strikes me as weird is that the proposal is basically the creation of private instance attribute on functions. Could you not get the same by actually setting an attribute on the function (this can not be used in lambdas in any case)? def f(): print(f.i) f.i += 1 f.i = 17 and some of the verbosity (but mostly reverse-reading) could be sucrosed-away with a decorator: @setattribute(i=17) def f(): print(f.i) f.i += 1 because as far as I can tell, if this can make it there is little justification for keeping an explicit `self` (among other things). This proposal also does not help with the "reverse argument hack" in lambdas, since it's using a statement.

On 26 September 2011 14:03, Masklinn <masklinn@masklinn.net> wrote:
This has been talked about before (in this thread or a related one). It would be slower (requiring two dictionary lookups for each access, rather than a single LOAD_DEREF / STORE_DEREF) and would break if the name 'f' is bound to another object. -- Arnaud

On Mon, Sep 26, 2011 at 9:03 AM, Masklinn <masklinn@masklinn.net> wrote:
No, because this fails if 'f' is rebound in the outer scope.
This proposal also does not help with the "reverse argument hack" in lambdas, since it's using a statement.
Correct, but the same can be said for 'nonlocal' itself. Besides, Guido has already nixed the lambda-friendly expression based suggestions. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

I liked the setattribute decorator. Also, static variables in C are just another way to do it (as opposed to TOOWTDI). The minor namespacing improvement isn't worth the unobvious "nonlocal .. as .." syntax in my mind. Pardon my android top-posting, --Yuval On Sep 26, 2011 9:04 AM, "Masklinn" <masklinn@masklinn.net> wrote: the

Masklinn wrote:
The advantages of the attribute solution are: - you can do it right now (functions have supported public writable attributes since v2.1); - the attribute is writable by the caller. If you want a public, writable attribute on a function, and I frequently do, this works fine. But: - the attribute is exposed to the caller even if it shouldn't be; - it's slow compared to local lookup; - lookup is by name, so if the function is renamed, it breaks; - initial value for the attribute is assigned *after* the function is created -- this is the same problem with decorators that the @ syntax was designed to fix. You can "fix" that last issue by moving the assignment inside the function, only this is even worse: def f(): try: f.i except AttributeError: f.i = 17 print(f.i) f.i += 1 or with a decorator, as you suggested. But still, it's pretty crappy to have slow public attribute access for something which should be fast and private. I have gradually warmed to Nick's suggestion. I'm not completely sold on the "nonlocal var from expr" syntax. Paul Moore's criticism makes a lot of sense to me. At the risk of dooming the proposal, "static" seems to me to be a more sensible keyword than "nonlocal". But for the sake of the argument, I'll stick to nonlocal for now. Some use-cases: (1) Early-binding: you use some constant value in a function, and nowhere else, so you don't want to make it a global, but it's expensive to calculate so you only want to do it once: # old way: _var = some_expensive_calculation() def spam(): do_stuff_with(_var) # or: def spam(_var=some_expensive_calculation()): do_stuff_with(_var) # proposed way: def spam(): nonlocal _var = some_expensive_calculation() do_stuff_with(_var) This puts the calculation inside the function where it belongs and is a win for encapsulation, without the ugly "looks like an argument, quacks like an argument, swims like an argument, but please don't try treating it as an argument" hack. (2) Persistent non-global storage: you have some value which needs to persist between calls to the function, but shouldn't be exposed as a global. A neat example comes from Guido's essay on graphs: def find_path(graph, start, end, path=[]): path = path + [start] if start == end: return path if not graph.has_key(start): return None for node in graph[start]: if node not in path: newpath = find_path(graph, node, end, path) if newpath: return newpath return None http://www.python.org/doc/essays/graphs.html I expect that could be re-written as: def find_path(graph, start, end): nonlocal path from [] path = path + [start] if start == end: return path if not graph.has_key(start): return None for node in graph[start]: if node not in path: newpath = find_path(graph, node, end) if newpath: return newpath return None The downside of this would be that the caller can now no longer seed the path argument with nodes. But for some applications, maybe that's a plus rather than a minus. (3) Micro-optimizations. An example from the random module: def randrange(self, start, stop=None, step=1, int=int): """Choose a random item from ... Do not supply the 'int' argument. """ If we're not supposed to supply the int argument, why is it an argument? Even uglier: def _randbelow(self, n, int=int, maxsize=1<<BPF, type=type, Method=_MethodType, BuiltinMethod=_BuiltinMethodType): These would become: def randrange(self, start, stop=None, step=1): nonlocal int from int ... etc. Much nicer and less confusing for the reader. (4) Recursion in Python is implemented by name lookup, so if you rename a function, or if it is anonymous, you're out of luck. But: def recurse(x): nonlocal recurse from recurse if x > 0: return recurse(x-1)+1 return 1 func = recurse del recurse Note: this use-case implies that the initial binding can't happen until *after* the function exists, otherwise recurse won't exist. -- Steven

I just want to chime in with how static variables can be bad for multi threaded code, yet they aren't considered as bad as global variables. E.g. the find_path guido example will explode if called by 2 threads in tandem. I believe this is more apparent and frowned upon when caused by global variables. Hasn't python given enough temptations to avoid multithreading already? Allowing these fast, local, persistent static variables is like allowing braces instead of whitespace - the good coders will manage either way but the bad will do bad things with it. --Yuval

Yuval Greenfield wrote:
Are you saying that Guido's find_path example is thread-safe when written with the default argument hack, but will fail with the new proposal?
Hasn't python given enough temptations to avoid multithreading already?
One can never have enough reasons to avoid multi-threading! <wink> But seriously, unless this proposal makes things *worse*, I don't see why this is an objection.
That's an analogy that makes no sense to me. -- Steven

On Mon, Sep 26, 2011 at 11:55 AM, Steven D'Aprano <steve@pearwood.info> wrote:
I actually think the 'attractive nuisance' aspect wrt multi-threading is a potentially valid objection. OTOH, it also makes synchronising parts of process global algorithms *much* easier, since you can just do "nonlocal lock from threading.RLock()" to get a shared lock variable. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Sep 26, 2011 at 8:26 AM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
To my mind, a 4:1 reduction in boilerplate lines, moving the function name out to the top level, making it clear that there's only one function (the inner one) that is kept around, avoiding repetition of the variable name and the function name all count as fairly substantial wins on the 'sugary goodness' front :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Sep 26, 2011 at 9:54 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Oops, forget to mention that the toy examples I've been using for the sake of brevity fail to convey one of the benefits of the syntax (i.e. the elimination of all trailing boilerplate following the function definition), since the body of the inner function is so short. To be fair, a more accurate comparison would be to a '@closure' decorator along the lines of something I posted in the previous thread: def closure(f): # Not clear if this is actually the right thing to do # It depends on how you decide to handle annotations # and whether decorators are applied to the inner or the # outer function return functools.wraps(f)(f()) @closure def FUNC(): """Real function is the inner one""" VAR = EXPR def _inner(real, params, here): # Hidden signature! nonlocal VAR # Still have to repeat VAR # Arbitrarily long complex code here return _inner # Still have this trailing boilerplate Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, 2011-09-26 at 10:08 -0400, Nick Coghlan wrote:
Could this help in cleaning up the lru_cash decorator in functools? That seems like it may be a good test case as it has both defaults and wrappers in it. Cheers, Ron (Looking in Lib for things this will help with.)

On Mon, Sep 26, 2011 at 10:31 AM, Ron Adam <ron3200@gmail.com> wrote:
Not really - lru_cache actually *needs* the outer function because it is a decorator factory. Once you have the outer function as a factory anyway, then the new shorthand syntax becomes somewhat less significant. lru_cache also shares the state amongst multiple functions (there are a couple of query operations that are provided as attributes on the function object). (It's written in a somewhat opaque way both to minimise runtime overhead and also to discourage people from becoming dependent on its internal implementation details) The only thing you could use it to clear up is the manual micro-optimisation in the decorating function's default arguments, and that could already be handled by doing the bindings in the outer function. That's one of the reasons this is such a niche feature regardless of how we spell it - it only works when you don't want to share the closure values across multiple inner functions. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

When this conversation started discussing namespaces, it occurred to me that we've had a number of suggestions for "statement-local" namespaces shot down. It seems that they'd solve this case as well as the intended case. I don't expect this to be acceptable, but since it solves part of this problem as well as dealing with the issues for which it was originally created, I thought I'd point it out. I'm talking about the requests that we add the let/where statement found in functional languages. Both add one new keyword. The syntax is: let: assignments in: statements or statement where: assignments which creates a namespace containing names bound in "assignments" for the duration of statement (or statements, as the case may be). I'm going to go with the let version, because it's not clear how what the syntax should be for where on a def statement. The original examples were things like: let: sumsq = sum(a * a for a in mylist) in: value = sumsq * 3 - sumsq So you can deal with the case of wanting to preserve values during a binding with something like: res = [] for i in range(20): let: i = i in: res.append(lambda x: x + i) Allowing a def in the statements means you can write the counter example with something like: let: counter = 0 in: def inc(change=1): nonlocal counter counter += change return counter We have to declare counter nonlocal in order to rebind it in this case. The odd thing here is that bindings in statements happen in the namespace that the let occurs in, but lookups of nonlocal variables include the namespace created by the let. <mike

On Mon, Sep 26, 2011 at 11:45 AM, Mike Meyer <mwm@mired.org> wrote:
See PEP 3150 - I've written fairly extensively on the topic of statement local namespaces, including their potential application to this use case :) There are some fairly significant problems with the idea, which are articulated in the PEP. If you'd like to discuss it further, please start a new thread so this one can stay focused on the more limited 'nonlocal' suggestion. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Since the expression will not be evaluated in the body of the function, would it make sense to define it outside the function body: def FUNC() given VAR1 [= EXPR1], VAR2 [= EXPR2]: # Do stuff with VAR1, VAR2 VAR1, VAR2 etc. will be evaluated in order so that EXPR2 can refer to VAR1. Values of VAR1, VAR2 etc. are preserved across function calls. This will be similar to scheme's let: (define func (let ((var1 expr1) (var2 expr2)) (lambda () <body> ))) Regards, Krishnan

On 9/26/2011 7:34 AM, Nick Coghlan wrote:
Today's default arg values are readonly (because stored in an immutable structure), quasi-anonymous* closure values used to initialize local names. If people understood that better, there would be less puzzlement over their behavior. Some of this is tied up with the confusion over name and value versus 'variable'. * They are anonymous in that they are access by index rather than by name. On the other hand, the call mechanism deterministically associates them with local names. Nick, I think you are on to something, but I don't know how to get from here to there. One of the explicit reasons for introducing closures defined by nesting was to reduce default value use. Personally, though, I think writing an outer function just for that reason (when only one function object is every going to be created, is a cure as bad or worse than the disease. When multiple functions are to be created, the new closures are an improvement. -- Terry Jan Reedy

Terry Reedy wrote:
One of the explicit reasons for introducing closures defined by nesting was to reduce default value use.
Indeed, and the question that needs to be asked is: Why does the default argument hack persist, when we now have a lexical scope system that's just as good as other languages such as Scheme and Haskell have, yet no such kludge seems to be needed in those languages? I don't believe it's because there's anything wrong with our functions or closures. Rather, it's a symptom of lingering deficiencies in *other* parts of the language. For example, when someone writes for i in things: def f(): dosomethingwith(i) squirrelaway(f) and get surprised by the result, he's effectively assuming that the for-loop creates a new binding for i each time around. He may not *realise* he's assuming that, but he is. Given that people seem to unconsciously make that assumption, perhaps we should consider making the for-loop actually behave that way. There's a precedent for this -- list comprehensions used to leak their loop variable into the surrounding scope, but that was eventually changed. Another point I'd like to make is that I don't think Nick's proposal, or any spelling variation of it, helps all that much in this case. Having to do *anything* special in or around the function to work around this problem is something of a kludge, especially if it involves writing anything as tautological-looking as "i = i". -- Greg

On Sep 26, 2011, at 7:05 PM, Greg Ewing wrote:
For the record, I remember being surprised when I learned that for doesn't create a new scope. This seems like a kind of Python 4000 feature though… At the very least you'd need to preserve the ability to use "old-style" for-loops by writing something like, i = None for i in things: ... if i is not None: print("Processed one or more things...") -- Carl Johnson

Carl Matthew Johnson wrote:
There's actually a way of implementing this so that the above code continues to work. If the loop variable is referenced by a nested function, it will be in a cell. All you need to do is create a *new* cell each time round the loop instead of replacing the contents of the existing cell. Code after the loop will then see whatever was last assigned to the loop variable, as it does currently. -- Greg

On Tue, Sep 27, 2011 at 12:03 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Greg, can you start a separate thread for this? It seems to be a much bigger change to the language, and it deserves its own thread, not mixed in with Nick's idea for nonlocal (even if the mechanism for both will end up using cells :-). (And yes, agreed that that can work.) -- --Guido van Rossum (python.org/~guido)

Guido has asked me to start a new thread for discussing this idea. To recap, instead of trying to come up with some new sugar to make the default-argument hack taste slightly less bitter, I suggest making a small change to the semantics of for-loops: If the loop variable is referenced from an inner scope, instead of replacing the contents of its cell, create a *new* cell on each iteration. Code following the loop would then continue to see the last value bound to the loop variable, as now, but inner functions would capture different versions of it. -- Greg

On Thu, Sep 29, 2011 at 5:31 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
That's potentially really nasty from an eval loop point of view. However, I think there's a way to make it work fairly naturally. Currently, the compiler takes all of the following names and effectively mashes them into a flat list: ordinary locals, cell vars (i.e. locals referenced from inner scopes) and free vars (i.e. names from outer lexical scopes). While technically 'free vars' in the normal usage of the term, global and builtin references are handled separately. The bytecode for the function then includes the following kinds of operations: LOAD/STORE_FAST: work on ordinary locals via a numeric index LOAD/STORE_DEREF: work on cells (i.e. cell vars and free vars) via a numeric index LOAD/STORE_GLOBAL: dynamic lookup in the module globals and then builtins by name The key point is that the compiler has enough information to figure all this out at compile time, so any change to loop semantics would need to work in with that. To make the loop rebinding work, it would probably be enough to change for loops and comprehensions to emit a new REPLACE_CELL opcode such that instead of replacing the contents of the existing cell they created a *new* cell and replaced the entire cell. Nested scopes from previous iterations would still have a reference to the old cell but all future references would see the new cell. This wouldn't have any impact on ordinary loops, since the new STORE_CLOSURE would only be used where STORE_DEREF is used currently. If there aren't any nested scopes involved, then STORE_FAST (or STORE_NAME at module or class level) would still get used. However, I'm not sure how we could handle the following pathological case: def outer(): i = 0 def loop(): nonlocal i for i in range(10): def inner(): return i yield inner def shared(): return i return loop, shared
If that inner loop was modified to replace the cells in the STORE_DEREF case then that final call would return 0 rather than 9. If we did this, I think we'd have to make reusing a nonlocal reference as a loop variable a SyntaxError since the two would flatly contradict each other (one says "share via this existing cell" the other says "create a new cell on each iteration"). Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Yeow. That's nasty indeed. Really handling it properly would require cells containing cells, which could get quite tricky. I'm inclined to punt and say that you deserve whatever you get if you do that. In any of the intended use cases, the loop variable will be local.
If we did this, I think we'd have to make reusing a nonlocal reference as a loop variable a SyntaxError
That might be a bit harsh, since it would make currently valid code illegal. Another possibility would be to say that the new semantics only apply when the loop variable is local; if it's declared nonlocal or global, the old semantics apply. -- Greg

On Fri, Sep 30, 2011 at 6:03 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Yeah, I think if we decide how to handle the global case, then the same answer can be applied in the nonlocal case. The local case would be straightforward: def func_gen(): for i in range(3): def inner(): return i yield i, inner
[i, f() for i, f in list(func_gen())] # Today reports [(0, 2), (1, 2), (2, 2)] [(0, 0), (1, 1), (2, 2)]
To maintain semantic equivalence between for loops and while loops, the new semantics would need to affect *all* name binding inside loops for names referenced from inner scopes, not just the iteration variable in for loops (FWIW, this would likely make implementation easier rather than harder): def func_gen(): i = 0 while i < 3: def inner(): return i yield i, inner i += 1
[i, f() for i, f in list(func_gen())] # Today reports [(0, 2), (1, 2), (2, 2)] [(0, 0), (1, 1), (2, 2)]
Explicit nonlocal and global declarations would then override the new semantics completely, just as they override ordinary local semantics today. While locals would gain early binding semantics, declared globals and nonlocals would retain late binding semantics: def func_gen(): global i for i in range(3): def inner(): return i yield i, inner
Code that used the default argument hack would continue to work under the new regime. Code that deliberately exploited the current late binding semantics would break. I think I'm still overall -1 on the idea, since it creates some rather subtle weirdness with the hidden switch to early binding semantics for locals, but the retention of late binding semantics for nonlocals and globals. While late binding for locals has its problems, it at least has the virtues of consistency and familiarity. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 9/29/2011 5:31 PM, Greg Ewing wrote:
Since loop variables do not normally have cells, I really do not understand from this what you are proposing. What I do understand is that you would have the content of the body of a loop change the behavior of the loop. This is a radical change with what to me is little practical justification and not a 'small change'. I have used about 20 languages and in all of them, a 'variable' refers to a specific memory block or or name. You must have a very different background to think that doing something else is normal.
-- Terry Jan Reedy

On Fri, Sep 30, 2011 at 1:06 AM, Terry Reedy <tjreedy@udel.edu> wrote:
On 9/29/2011 5:31 PM, Greg Ewing wrote:
Not really. If the loop (or other) variable is not used in creating closures, then it doesn't really matter whether the variable is stored as a local or a cell -- it happens to be stored as a local for speed reasons. If a variable (including the loop variable) is used inside a closure, then it already creates a cell. This means that loop content already affects the way the loop is compiled, though again, it affects only efficiency, not semantics. The difference is that now the loop will create N separate cells instead of just one, so that each closure will see its own private variable, instead of all sharing the same one. That is a semantic difference, but if the closed-over variable is also a loop variable, it will normally be a bugfix.
I have used about 20 languages and in all of them, a 'variable' refers to a specific memory block or or name.
There are languages (Erlang is the best known) that make "variables" write-once; in those cases, loops will always create N separate "variables". (Expensive on memory; good for concurrency.) This proposal says that python should do so when (and perhaps only when) those separate variable instances are used in different closures. -jJ

On 30 Sep, 2011, at 16:17, Jim Jewett wrote:
What worries me with this proposal is that it only affects the loop variable, and not other variables. This makes it easy to introduce subtle bugs when you forget this. I often have code like this when using closures in a loop: for val in sequence: other = lookup(val) result.append(lambda val=val, other=other): doit(val, other)) Greg's proposal means that 'val=val' would no longer be needed, but you'd still need to use the default argument trick for other variables. The difference between the behavior for the loop variable and other variables is also relatively hard to explain. Ronald

On Tue, Oct 4, 2011 at 7:22 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
What worries me with this proposal is that it only affects the loop variable, and not other variables.
Right. Another similar example: def f1(seq): for i in seq: yield lambda: i.name # each lambda refers to a different i def f2(seq): for i in seq: x = i.name yield lambda: x # each lambda refers to the same local x Think of f2 as a refactoring of f1. I think the programmer has a right to expect that to work, but it would change behavior in a really surprising way. Also: What if the loop variable is also used elsewhere in the function? def f(seq): i = None for i in seq: pass return i Does the loop variable get a cell in this case? If so, I guess it must be a different variable from the local i. So would f([1]) return None? That would be a rather astonishing change in behavior! So perhaps the loop variable cell would be kept in sync with the local variable during the loop body? It all seems too weird. -j

Jason Orendorff wrote:
No, because i is not referenced from an inner function. There is no change from the current semantics in this case. If it *were* referenced from an inner function, then cell replacement would occur, notwithstanding the fact that it's used elsewhere in the function. -- Greg

The proposal actually wasn't clear whether it affected all name bindings in loops or just the iteration variables in for loops. Either way, the fact that unrolling the loops (even partially to implement loop-and-a-half without using break) fundamentally changed the name binding semantics pretty much killed the idea from my point of view. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Ronald Oussoren wrote:
This is a good point, and I have an alternative version of the idea that addresses it. Instead of the cell-replacement behaviour being automatic, we provide a way to explicity request it, such as for new i in stuff: ... The advantage is that can be applied to anything that binds a name, e.g. for x in stuff: new i = 2 * x def f(): print i store_away(f) The disadvantage is that you need to be aware of it and remember to use it when required. However that's no worse than the status quo, and I think it would provide a nicer solution than the default argument hack or any of its proposed variations. -- Greg

On 9/27/2011 1:05 AM, Greg Ewing wrote:
People who post their 'surprise' on python-list have nearly always used lambda, not def, to define the function. This is so much the case that I have concluded that 'lambda' has a hypnotic effect that 'def' does not.
he's effectively assuming that the for-loop creates a new binding for i each time around.
No, for loops *do* rebind loop variables each time around. People are assuming that 'i' is immediately bound to its 'current' value, just like default args. (This is the opposite of people who mis-assume that default arg expressions are re-evaluated with each call.) This assumption is tied much more to 'lambda' than 'def'.
*Some* people -- those who come from a language that does whatever it is that python does not do that you want it to do.
This has *nothing* to do with people assuming early binding of names in lambda expressions. The posted 'surprise code' typically uses list comps. So changing for loops to be like list comps would have no effect on that mis-assumptions and the resulting surprise. -- Terry Jan Reedy

Terry Reedy wrote:
No, for loops *do* rebind loop variables each time around.
I probably should have avoided the word "bind", because it can be ambiguous. What I mean is that it assigns a different value to the same variable instead of creating a new variable. This is different from what equivalent constructs in some other languages do.
People are assuming that 'i' is immediately bound to its 'current' value,
Yes, that's another possible misinterpretation they may be making -- if they're thinking that deeply about the issue at all. However, changing the language to make *that* true would break a lot of other things, such as mutual recursion, and isn't really an option.
Agreed, because list comps in Python have the same problem -- even if the name is local to the list comp, it's still just one variable being reassigned. I only mentioned list comps as a precedent for making some kind of change to the way a loop variable is treated. I didn't mean to suggest that the way they currently work would solve the closure problem. -- Greg

On 2011-09-29, at 12:52 , Greg Ewing wrote: the construct closes over a new name in a different lexical scope instead of (as in Python, or Javascript) the same name in the same lexical. It's not about rebinding so much as about the breadth of lexical scopes.

Nick Coghlan dixit (2011-09-26, 11:01):
nonlocal VAR from EXPR # Strongly indicates there's more than a
nonlocal EXPR as VAR # Parser may struggle with this one
The construct itself appeals to me, but I share reservations against using `nonlocal` for something used within the local lexical scope only. I'd like to use `deflocal` keyword -- it'd be suggest something local (in terms of scope), something constant (definition-time-related) and something complementary to `nonlocal`. deflocal VAR from EXPR -- would be perfect for me. But indeed -- that's a new keyword. Regards. *j

Nick Coghlan wrote:
I think you're mixing up concepts and implementation here a bit. Cells are an implementation detail. What's important is which *namespace* the name is referring to: - A 'global' declaration makes it refer to the module-level namespace. - A 'nonlocal' declaration makes it refer to some namespace in between local and module-level (there may be more than one of these, so I wouldn't say that there are only 4 namespaces). Now, while the *value* of your proposed new kind of variable would be stored as part of the function's closure, its *name* would be part of the function's *local* namespace. Consider this: i = 88 def f(): nonlocal i = 17 print i def g(): nonlocal i = 42 print i f() g() print i I'm assuming you intend that the two i's here would have nothing to do with each other, or with the i in the enclosing scope, so that the output from this would be 17 42 88 rather than 42 42 42 So, your proposed use of 'nonlocal' would actually be declaring a name to be *local*. That strikes me as weird and perverse. NOBODY-expects-the-spanish-nonlocal-declaration!-ly, Greg

On 26 September 2011 07:56, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
So, your proposed use of 'nonlocal' would actually be declaring a name to be *local*. That strikes me as weird and perverse.
Aha! That's precisely the concern I had with the suggestion of "nonlocal" for this, although I had failed to understand *why* it bothered me, and so hadn't commented... Paul

On Mon, Sep 26, 2011 at 2:56 AM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Ah, but it *wouldn't* be local, that's the point - it would be stored on the function rather than on the frame, and hence be shared across invocations. Change your example function to this so it actually modifies the name binding, and it becomes clear that this is *not* a local variable declaration: i = 88 def f(): nonlocal i from 17 print(i) i += 1
It would work exactly as if we had introduced a containing scope as a closure: def outer(): i = 17 def f(): nonlocal i print(i) i += 1 return f
The *only* thing that would change is the way the closure reference would be initialised - it would happen as part of the function definition (just like default arguments) rather than needing the developer to write the outer scope explicitly just to initialise the closure references. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 26 September 2011 12:34, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hmm, its lifetime is non-local, but the visibility is still local. My instincts associate the word "(non-)local" with visibility rather than lifetime. If you want a bikeshed to colour in, maybe "persistent" is a better keyword for this: def counter(): persistent n as 1 print(n) n += 1 Paul.

On Mon, Sep 26, 2011 at 7:45 AM, Paul Moore <p.f.moore@gmail.com> wrote:
Adding new keywords is a big, big step that demands a compelling justification. Now that I have come up with a way to make this syntactic sugar for existing usage of nonlocal rather than a new concept, I flatly oppose introduction of a new name for something which is, at a fundamental level, just a variation on existing functionality. I'd rather continue with the status quo indefinitely if people truly find this variant intolerable. It may be easier to turn things around and specifically look at it from the "syntactic sugar" point of view: # Current syntax def _outer(): # Boilerplate VAR = EXPR def FUNC(): # Real function name is hidden nonlocal VAR # VAR repeated # Do stuff with VAR, including rebinding it return f # Boilerplate FUNC = _outer() # Boilerplate and FUNC repeated Most of that code is noise: the interesting aspects are that: 1. There is a function called FUNC() available for use 2. VAR survives across invocations of FUNC() 3. At the first invocation of FUNC(), the initial value of VAR will be EXPR So, let's offer a syntax that just includes those 3 pieces of interesting information without the boilerplate: def FUNC(): # Function of interest is not hidden in a nested scope nonlocal VAR from EXPR # Shared variable and initial value # Do stuff with VAR Is anyone going to *guess* what that means without looking it up? Probably not. But are they going to *forget* what it means once they learn it? Also probably not. "I can guess what this means without reading the docs or having someone explain it to me" is setting the bar too high for what a single keyword can possibly hope to convey. The bar needs to be whether or not it serves as a useful mnemonic to recall the functionality once a user already know what means. For me, 'nonlocal' fits that bill, especially when the feature is described as syntactic sugar for a somewhat common use case for the existing lexical scoping functionality. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 26 September 2011 14:47, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree entirely. My point here wasn't to suggest that this needs a new keyword, but rather that the proposal uses an unnatural keyword to avoid needing a new keyword. Your argument that this is a simple extension of the semantics of "nonlocal" is reasonable when viewing nonlocal in terms of lifetimes. My contention is that most people view nonlocal in terms of visibility (and in that view, the two uses of nonlocal are jarringly dissimilar).
I have no problem with the contention that this would be useful. Minor, as you concede at the start of the thread, but certainly useful. I'm certainly bikeshedding here over the name of the keyword. But I think I'm arguing that green is the wrong colour for this stop sign, because people will misinterpret it, whereas you are arguing it's a great colour because we have this tin of green paint here, and the paint shop's closed. (End of overwrought analogy :-))
But readability matters, and I worry that this isn't "readable". Maybe the fact that neither of us is Dutch is relevant here, too :-) Paul.

On Mon, Sep 26, 2011 at 10:24 AM, Paul Moore <p.f.moore@gmail.com> wrote:
I agree it's certainly a *new* way of looking at the problem, but I don't agree that that necessarily makes it a *wrong* way to look at it. Should we invent a new keyword, or abandon the concept of (more) elegant syntax for shared internal function state, just to avoid teaching people that reference locality involves elements of both visibility *and* lifecycle? Instead, I contend that it is precisely this aspect of the language that makes mutable default arguments such a point of confusion. Since we *don't* separate out the explanations of visibility vs lifetime when discussing closure references, default arguments become a special snowflake that people need to learn about without being able to slot it into a broader framework. I believe we can actually make the concepts involved in both default argument lifecycles and the 3.x super() implementation more semantically coherent by *embracing* nonlocal as influencing both visibility *and* lifetime and clearly articulating that as a core language concept in relation to closures. We *already* have the following possibilities: local visibility, local lifetime: 'ordinary' local variable referenced solely from the executing frame local visibility, nonlocal lifetime: default argument expressions, local variable referenced from inner function that survives current invocation nonlocal visibility, nonlocal lifetime: 3.x __class__ reference, closure reference to outer lexically containing scope, global references My proposed syntax is just a way to explicitly create new entries in that middle category - variables with local visibility and nonlocal lifetime. So 'global VAR' and 'nonlocal VAR' would continue to declare that a NAME belongs in the third category and is initialised somewhere else, while 'nonlocal VAR from EXPR' would say "this has nonlocal lifetime, but local visibility, so here's the initial value since it won't be provided anywhere else". Regards, Nick. P.S. Really nice point about the visibility vs lifecycle distinction. I think it actually *strengthens* my argument about the current lack of a coherent semantic framework in this space, though. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
I don't think that's the right way to characterise default argument values. While the *values* have nonlocal lifetime, the *name* being declared is a parameter, so both its scope and lifetime are local. Your proposal would be creating a new category of name that doesn't currently exist. I don't see how your proposal would do anything to clarify the distinction between visibility and lifetime. Currently, 'nonlocal' always refers to visibility. Your way, it would sometimes mean visibility and sometimes lifetime, with only a very obtuse clue as to the difference. -- Greg

Interesting discussion! Very enlightening. But after all that I'm still unconvinced that the new concept should be tied to the nonlocal keyword. Nick's right that the *mechanism* for nonlocals and __class__ is the same, and the mechanism for the new concept might well be the same. And possibly a hypothetical other Python implementation could use the same mechanism to keep track of default argument values. Or not. But at a different level, nonlocal and the new concept are quite different, and the difference has to do with the scope of the name, not its lifetime (this was a very important insight in the discussion!). I think it is actually very telling that the syntax for nonlocal, __class__ and default argument values is so different: to most users, the lifetime is much less of an issue than the scope. Now, let's pretend the new feature is called "own". We'd write: def counter(): own x = 0 x += 1 return x If we had another function (say, defined in the same scope): def count_by_twos(): own x = 0 x += 2 return x the x in counter() and count_by_twos() are unrelated. This is different from the way nonlocal works (I don't have to type the example I hope :-). Also, nonlocal is illegal in a global function -- syntactically, it must be in a function that is nested inside another function. For all these reasons, sorry Nick, but I really don't think any syntax based on nonlocal will do it, even "nonlocal <var> from <expr>". I see two options open at this point. 1) Bikeshed until we've come up with the right keyword. I think that apart from the keyword the optimal syntax is entirely fixed; "<keyword> <variable> = <expression>" is the way to go. (I suppose we could argue about how to declare multiple variables this way.) I would be okay with using either "static" or "own" for the keyword. Personally, I don't really mind all the negative baggage that static has, but a Google code search for \bown\s= shows way fewer hits than for \bstatic\s= (both with lang:^python$ case:yes). 2) Give up, and keep using the default argument hack. You can protect your arguments from accidentally being overridden by using a "lone star" in the argument list (PEP 3102) and a __private naming convention. The lone star prevents accidental overriding when too many positional arguments are given. The __private provides sufficient protection against misguided attempts to provide a value for such "arguments", even if for functions outside a class it is purely psychological. (Note that someone who *really* wanted to mess with a cell's variable could also do it.) -- --Guido van Rossum (python.org/~guido)

On Mon, Sep 26, 2011 at 9:32 PM, Guido van Rossum <guido@python.org> wrote:
Interesting discussion! Very enlightening.
Indeed! It has certainly clarified many aspects for me.
But after all that I'm still unconvinced that the new concept should be tied to the nonlocal keyword.
<snip>
For all these reasons, sorry Nick, but I really don't think any syntax based on nonlocal will do it, even "nonlocal <var> from <expr>".
Well, the semantics of a 'nonlocal-from' initialised nonlocal variable would be different from a bare nonlocal declaration (i.e. the new form would create a new name binding only visible in local scope, so it wouldn't get shared across functions within the same outer function and is legal in top-level functions), just as they would be with a new keyword. So I'm not prepared to completely give up on this one yet - it took me years of poking at the issue to come up with the idea of reusing nonlocal as the keyword, so I hope people will take some time to consider the idea of using the term to refer to variations in both visibility and lifecycle rather than strictly limiting it to refer only to the visibility aspect.
There have certainly been plenty of proposals over the years. The ones I recall off the top of my head are: once static persistent shared closure own atdef deftime The baggage something like 'static' brings with it would be even more misleading than that associated with 'nonlocal', and the others are either seriously ugly or else aren't particularly suggestive as mnemonics. At least 'nonlocal' indicates something related to function closures is in play, even if it isn't the exact same thing as is currently meant by a bare declaration. Really, the mechanism suffers from the same problem that led to the choice of nonlocal for PEP 3104 in the first place: the important part of the declaration lies in making it clear to the compiler and the reader that the specified name is *not* a normal local variable. Exactly what it *is* instead will depend on the specifics of the code - it may be shared state, it may be something else, the only way to find out for sure is to look at the code and see how it is used. Choosing a keyword based on any given use case (such as 'shared') makes the construct seem odd when used for another use case (such as early binding). I agree the two concepts (bare nonlocal and initialised nonlocal) are not the same. However, I also believe they're close enough that a slight broadening of the definition of 'nonlocal' is preferable to trying to come up with a completely new term that is both readable, usefully mnemonic and not laden down with unrelated baggage either from Python itself or from other languages.
Despite what I wrote earlier in the thread, I don't think we actually need to declare the idea *dead* if you don't currently like 'nonlocal' as the keyword. Perhaps the 'nonlocal' idea will grow on you (it certainly did on me, the longer I considered the idea), or perhaps you or someone else will come up with an appropriate keyword that everyone considers reasonable. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Sep 26, 2011 at 7:43 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Sorry, from reading the responses, the objection that the other use of nonlocal refers to scope, not lifetime, seems pretty common, and I don't think it will go away. You might as well use "def var = expr" as the syntax and justify it by saying that it is clearly defining something and syntactically different from function definitions... OTOH I am happy to let you all bikeshed on a better name. -- --Guido van Rossum (python.org/~guido)

On 9/26/2011 9:48 PM, Guido van Rossum wrote:
I was actually considering asking whether using 'def.static' or 'def.initial' would be a possible way to avoid making a new keyword. I honestly prefer def to nonlocal for this. I like that "our" is short, but while it's less confusing, I don't see is as clear. Questions on options: Could there be an importable "function" that, when used, declares a variable to be one of these? Could the keyword be toggled in or out of namespaces based on, say, an imported module? If so, the keyword wouldn't be terribly burdensome. The following could also add to Nick's list of keyword options: init (or Initial or even initialize) retain hold held persist preserve keep kept lasting stash survive ark reshiyth Given the frequency of the use case, clarity probably trumps concerns about keyword length. To my eye, those all seem more congruous than "nonlocal." (Nick's list: nonlocal once static persistent shared closure own atdef deftime) -Nate

On Mon, Sep 26, 2011 at 8:53 PM, Spectral One <ghostwriter402@gmail.com> wrote:
"our" is something in Perl, right? My proposal was "own", which is much older (Algol-60). Probably nobody else here remembers it.
The compiler would have to recognize that function as special, which is not really possible, since the Python compiler has no access to the runtime environment, so it can't know what is imported (__future__ is the exception, and that's why it's a __xxx__ word).
Could the keyword be toggled in or out of namespaces based on, say, an imported module? If so, the keyword wouldn't be terribly burdensome.
We do that regularly with "from __future__ import <some_syntactic_feature>" and when the parser sees it, it modifies itself (slightly). Modifying the list of reserved words is easy. But it's not a long-term solution. Somehow you triggered a thought in my head: maybe se should stop focusing on the syntax for a while and focus on the code we want to be generated for it. If we can design the bytecode, perhaps that would help us come up with the right keyword.
-- --Guido van Rossum (python.org/~guido)

On 27 September 2011 05:12, Guido van Rossum <guido@python.org> wrote:
"our" is something in Perl, right? My proposal was "own", which is much older (Algol-60). Probably nobody else here remembers it.
I do :-) Not sure that Algol-60 using it is a good enough reason for recommending it, though. The name didn't make sense to me even back then... (Actually, after this discussion, the logic is clearer - so congratulations, Nick, for clarifying a 40-year old puzzle for me!) Call by name, anyone? Paul.

On Tue, Sep 27, 2011 at 12:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
In the days of Algol-60 they were pretty bad at explaining things. I was puzzled by "own" as well. (However, the biggest mystery for me, for a long time, were pointers in Pascal. It didn't clear up until I learned assembly.)
Call by name, anyone?
ABC, Python's predecessor, had it, IIRC. -- --Guido van Rossum (python.org/~guido)

With regard to "own" or "static", my feeling is that this kind of feature (i.e. mutable shared state visible only to one function) is an anti-pattern in language design. We have much better ways nowadays of sharing state without making it completely global, such as modules and classes. There was a discussion a while back about a "const" statement, which was essentially what we are talking about here except that the name would be read-only. This seems like both a better name and better semantics to me. As I remember, the idea foundered because it was thought too confusing to have an expression that was written inside the function but evaluated outside of it. If we're seriously considering the current proposal, presumably we've gotten over that? Or is it still a valid objection? -- Greg

On 27 September 2011 21:38, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote: [...]
I have been reading this thread from the start and FWIW, it's something that I have been feeling uncomfortable about. I value the guarantee that no code in the body of a def statement is executed when the def statement itself is executed. This proposal would break this guarantee. -- Arnaud

On Tue, Sep 27, 2011 at 2:04 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:
That seems a guarantee made up specifically to prevent this proposal. What benefit do you see from it? -- --Guido van Rossum (python.org/~guido)

On 27 September 2011 22:23, Guido van Rossum <guido@python.org> wrote:
I wouldn't do this :) I like new things.
What benefit do you see from it?
It keeps things simple. To be specific, when I read code I can "fold" (mentally or with the help of a code editor) the body of a def statement, knowing that I won't miss anything until the function is called. So the following won't raise: [some code] x = 42 def foo(): [I don't need to read the body] assert x == 42 But now def statements can have non-obvious side-effects. For example: def spam_x(): global x x = "spam" x = 42 def foo(): # Some body nonlocal y from spam_x() # More body assert x == 42 I may be unduly worrying about this, but it feels to me that it lessens the overall tidiness of the language. I would much rather there was a way to initialise these "own" variables outside the body of the function. -- Arnaud

On Tue, Sep 27, 2011 at 3:24 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:
Yes, you have a point. Hiding side effects this way could be nasty, and this is actually the first point in favor of the default argument hack I've heard in a while. :-) C++ has special syntax to force the programmer to write certain things after the argument list but before the function body (I think only for constructors). I have seen too many constructors with a short argument list, a very short body, and a ton of code cramped in between, to like the idea very much: even though it is semantically something totally different, syntactically it would have the same awkwardness. -- --Guido van Rossum (python.org/~guido)

On 9/27/2011 6:32 PM, Guido van Rossum wrote:
On Tue, Sep 27, 2011 at 3:24 PM, Arnaud Delobelle<arnodel@gmail.com> wrote:
I have several times written on python-list that Python has a simple rule: header expressions (for default args) are evaluated one-time when the function is defined; body expressions are evaluated (perhaps) each time when the function is called. If people understand this and do not fight it, they are not surprised that mutating a once-defined default arg mutates it, nor that defining a function inside a loop magically causes define-time binding of names in the body. I would hate for Python to lose this simplicity. I consider it one of its positive features. -- Terry Jan Reedy

Terry Reedy wrote:
Simplicity is good. I don't remember who first suggested it, and I don't remember any discussion on it, but what if decorator functions grew a __prepare__ method similar to metaclasses? If the method exists it is called /before/ the function is evaluated, it returns a namespace that is then used as the basis for the function's (the interpreter can take everything it finds in there and add it as closures to the function), and folks can call it whatever they want. ;) If the method does not exist, nothing is different from what we have now. Seeing-if-we-can-have-more-exploding-heads'ly yours, ~Ethan~

On 9/27/2011 9:26 PM, Ethan Furman wrote:
At least three people, including Guido, have noted than a keyword-only _private parameter solves most of the problems with using default args for constants. Since the _underscore convention is already partly built into the language (import *, help(module), __mangling, __reserved__), we can have other tools like signature introspection also respect the convention. This solution *also* works for the semi-private accumulator parameter of most tail-recursive functions. Example: sum the values in a linked list (without iterators or explicit loops*: def llsum(ll, *, _total = 0): if ll: return llsum(ll[1], _total = _total + ll[0]) else: return _total The private parameter _total is constant for outside calls but gets a new value with each recursive call. It should not appear in the public signature but is used internally. The non-parameter constants proposed as a substitute for default args would not work for this case. * I am QUITE aware that this can be rewritten with while, and indeed I write the two if branches in the order I do to make it easy. def llsum(ll) total = 0): while ll: ll, total = ll[1], total + ll[0] else: return total But this is a different issue. -- Terry Jan Reedy

On Wed, Sep 28, 2011 at 12:12 AM, Terry Reedy <tjreedy@udel.edu> wrote:
Note that this also matches indentation, since decorators are indented with the header, rather than with the body.
Though they may still be surprised in the opposite direction; that you have to write an explicit and redundant i=i to capture the current value when binding.
I would hate for Python to lose this simplicity.
Agreed.
This solution *also* works for the semi-private accumulator parameter of most tail-recursive functions.
Additional advantages: (a) Because it is still a parameter, it *can* be altered for test code; it is just obvious that you're doing something unsupported. (b) The only patch required is to documentation. Instead of saying "Don't do that", the documentation should say "If you just want to save state between calls, make it a keyword-only parameter and indicate that it is private by prefixing the name with an underscore." That is pretty hard to beat from a backwards-compatibility standpoint. -jJ

On Wed, Sep 28, 2011 at 5:41 PM, Jim Jewett <jimjjewett@gmail.com> wrote:
Um, but you can't save state between calls in a default argument value, except by the hack of making it a list (or some other mutable object) and mutating that. -- --Guido van Rossum (python.org/~guido)

So has Jan's post-** argument list proposal from June ( http://mail.python.org/pipermail/python-ideas/2011-June/010479.html) been definitively nixed? It seemed the perfect answer given that its similarity to the default argument hack makes it intuitive when the evaluation is taking place. On Wed, Sep 28, 2011 at 10:54 PM, Guido van Rossum <guido@python.org> wrote:

On Thu, Sep 29, 2011 at 2:27 AM, Matthew J Tretter <matthew@exanimo.com> wrote:
It has in my mind. As Guido noted, default argument *names* refer to ordinary locals, and hence exhibit the same rebinding behaviour as pre-nonlocal closure references (i.e. reassignments don't persist across calls, only mutations do). Since we expressly *don't* want that behaviour, anything directly in the argument list (even after **) would be inappropriate. I'm still mulling over the idea of '@state' either as a keyword in its own right or else as a 'known to the compiler' builtin like super so that the following would work (as several people have suggested): @state(n=0, lock=threading.Lock()) def global_counter(): n += 1 return n The special casing of super() is something that annoys me though, and making 'state' a keyword would be extraordinarily disruptive, so I'm also intrigued by the suggestion of reusing 'nonlocal' for the same purpose: @nonlocal(n=0, lock=threading.Lock()) def global_counter(): n += 1 return n With that spelling, the nonlocal statement would declare references to pre-existing closure variables in an outer scope, while the nonlocal decorator would declare new function state variables, implicitly creating the outer scope without all the boilerplate. I must admit, the 'some kind of decorator' syntax is growing on me, especially the nonlocal variant. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 9/29/2011 8:00 AM, Nick Coghlan wrote:
/'we'/'I and some of the discussants'/ It has occurred to me that one reason this threads is not getting anywhere very fast is that there are two quite different proposals being intermixed. People interested in and discussing the two different proposals have been talking past each other. One is to reproduce the behavior of defaulted parameters without having the name appear in the signature to produce *read-only* private locals, which amount to private define-time constants. This includes the int=int type speed hack. I think '*, _int=int', with adjustment of introspection, is sufficient. This make '_int' pragmatically constant, while still externally over-rideable for exceptional purposes such as testing or tracing. The other is to introduce something new: private *read-write* closure (persistent) name-value pairs. I agree that such a new thing should not be in the param list between '(' and ')'. -- Terry Jan Reedy

On Thu, Sep 29, 2011 at 3:47 PM, Terry Reedy <tjreedy@udel.edu> wrote:
Well, the references would be constant. The state itself could still be mutable, depending on the kinds of objects referred to. I agree this use case could be addressed by blessing the status quo and a couple of conventions.
It's not actually new as far as overall language capabilities go - since PEP 3104, you can get the functionality through appropriate use of an ordinary closure. It's really just about providing a slightly shorter syntax for a particular way of using closures. Well noted that not all proposals are addressing the same functionality, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Are you sure we don't? The ability to persistently re-assign the name is not needed to address the main use cases under consideration, as I understand them to be. In fact, disallowing assignment to the name at all would be fine by me. Hence I would be +0 on using 'const' as the keyword. -- Greg

On Thu, Sep 29, 2011 at 5:24 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Fixed references to mutable objects makes 'const' a rather problematic choice. Besides, if we set the bar at that level of functionality then documenting some conventions for the status quo (i.e. 'private' keyword only arguments) is an adequate response. As Terry noted, there are actually two different features being discussed in the thread, and they differ according to whether or not rebinding is supported. By casting the proposal as syntactic sugar for a particular usage of closures, I'm firmly in the camp that if we change anything here the updated syntax should support rebinding of the shared names. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Sep 29, 2011 at 5:24 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Nick Coghlan wrote:
What do you think the main use cases are? If you can't rebind, it obviously doesn't solve the Counter case. As best I can tell, const only works for speed hacks. And there are other ways to solve that, if you're willing to use a different VM, like pypy does. -jJ

Terry Reedy dixit (2011-09-27, 21:17):
No, it does not cause such a binding. That is one of the cases the proposition of this or that early-binding syntax comes back repeatedly: def strangish_factorission(): chairs = [] for spam in (1,2,3,4,5): def comfy_chair(fmt): # we naively try to make use of early binding here # but there is no such binding here result = fmt % spam return result chairs.append(comfy_chair) return chairs for comfy_chair in strangish_factorission(): print comfy_chair('%d'), -- will print "5 5 5 5 5", not "1 2 3 4 5". To obtain the latter you need to use eigher default argument hack (which is ugly and unsafe in some situations) or a closure (which for now needs another nested scope, which is even worse in terms of readability). Cheers, *j

On 9/28/2011 5:57 PM, Jan Kaliszewski wrote:
Terry Reedy dixit (2011-09-27, 21:17):
defining a function inside a loop
Insert 'does not', which somehow got omitted or deleted.
magically causes define-time binding of names in the body.
No, it does not cause such a binding.
Of course not, as I have said many times over the last decade plus, most recently just 4 hours earlier (at 17:10), when I said "People are assuming [wrongly, when using a local name that matches an outer enclosing loop name] that 'i' is immediately bound to its 'current' value, just like default args." Sorry for the confusing omission. My intention was to list this as a delusion, not as a fact. -- Terry Jan Reedy

I never liked the nontestability of nested functions, and I also don't like the nontestability of these non-global semiglobalnonlocal variables. Real globals have the benefit that a test suite can check what happens to that variable as functions are called fairly easily. Here you have to test against my_func.__closure__[i].cell_contents for some i whose value depends on the order in which you created the variables and the order in which they're added to existing nonlocals (provided that isn't an implementation detail).
+1. Yes.
There is. Have this "nonlocal assignment" as a magical decorator, or as syntax as part of the def statement. I'm not sure to what degree Nick's proposal is about the nonlocal syntax versus the nonlocal semantics. But you can keep the semantics (fast access to variables that are shared as global state) while changing the syntax fairly radically. Devin On Tue, Sep 27, 2011 at 6:24 PM, Arnaud Delobelle <arnodel@gmail.com> wrote:

On Tue, Sep 27, 2011 at 4:38 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
In explaining to someone why I think this issue of function state variables is worth exploring, I described it as being important for the same reason as closures are important. That's also why I've been describing it as syntactic sugar for a particular *kind* of closure usage. The reason I think closures themselves are important is that gets back into the 'algorithm with state' vs 'data with behaviour' contrasting views of a problem. Sometimes one view is more productive, sometimes the other, and there's blurry ground in the middle where either can work. As programmers, our best bet is to learn both modelling tools and then judge which is most appropriate for the problem at hand. One thing I definitely got out of this thread is that I think we need better tools for introspecting closure and generator internal state, so I'd be interested in seeing proposals for additions to the inspect module on that front. However, getting back to the original topic, the @nonlocal suggestion is definitely growing on me, since it seems to combine all the elements of the current equivalent that I'd like to shorten into one concise syntax: def _outer(): n = 0 lock = threading.Lock() def global_counter(): nonlocal n with lock: n += 1 return n return global_counter global_counter = _outer() would become: @nonlocal(n=0, lock=threading.Lock()) def global_counter(): with lock: n += 1 return n It's presence in the decorator list hints that this is something happening outside the functions ordinary local scope, as well as indicating when the initialisation happens. The 'nonlocal' then ties in with how they behave from the point of view of the code in the function body. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 29 September 2011 14:30, Nick Coghlan <ncoghlan@gmail.com> wrote:
I agree, this is a nice option. My biggest discomfort with it is the idea of a magic decorator name handled specially by the compiler (and worse still, it's a keyword so it's syntactically very weird indeed). I had similar discomfort over the new super, but I could get over that by simply assuming that super was a normal function that was just more magical than I understood. "@keyword" decorators don't even fit into my model of valid syntax :-( That said, I like it otherwise. The above says to me "the following values are nonlocal to this function", which I can read exactly the way it actually works. Whether that's a result of a week's immersion in Nick's propaganda, I can't say, though :-) Paul.

On 29 September 2011 15:05, Paul Moore <p.f.moore@gmail.com> wrote:
Actually, while we're using keywords as decorator names, @with(n=0, lock=threading.Lock()) def global_counter(): with lock: n += 1 return n reads well to me, and avoids the technical/jargon word "nonlocal". Let the @keyword floodgates open :-) Paul.

On 2011-09-29, at 17:15 , Paul Moore wrote:
The problem with that one is that it's going to be read in light of the corresponding keyword, so it'd make people think the purpose is to __enter__/__exit__ all objects provided around the function call somehow (pretty nice for the lock, nonsensical for the integer, of course).

Paul Moore wrote:
I think 'nonlocal' is the better choice as it mirrors what nonlocal does inside the function. Because of its current usage 'with' will generate questions about __enter__ and __exit__. ~Ethan~

On 29/09/2011 17:21, Ethan Furman wrote:
But what if the decorator is used inside an outer function? Wouldn't that be confusing? What about @func_attrs(...) ? - it would give a hint where the objects are actually located.
participants (33)
-
Alex Gaynor
-
alex23
-
Arnaud Delobelle
-
Bruce Leban
-
Carl Matthew Johnson
-
Chris Rebert
-
Devin Jeanpierre
-
Eric Snow
-
Ethan Furman
-
Gisle Aas
-
Greg Ewing
-
Guido van Rossum
-
H. Krishnan
-
Jacob Holm
-
Jan Kaliszewski
-
Jason Orendorff
-
Jim Jewett
-
Joachim König
-
Masklinn
-
Matt Joiner
-
Matthew J Tretter
-
Mike Meyer
-
MRAB
-
Nick Coghlan
-
Paul Moore
-
ron adam
-
Ron Adam
-
Ronald Oussoren
-
Spectral One
-
Stephen J. Turnbull
-
Steven D'Aprano
-
Terry Reedy
-
Yuval Greenfield