Mechanisms which rely on manipulating variables within closures or nested scopes to function properly can be elegant, but I've not yet seen one that *really* is.

This really isn't a case for or against what I'm proposing since we can already do this in today's Python with mutable variables in an enclosing scope (see below). I am proposing a language change to help make closures more orthogonal to the scoping constructs that are already in place for the global scope.

You state that using classes can be "heavy handed", but one of the major uses of classes is as a *namespace*. Many desired uses of closures (including the various uses you have outlined) is to hide a *namespace*, and combining both closures with classes can offer that to you, without requiring a language change.

Closures are also used in more functional styles of programming for defining customized control structures (those Ruby folks like them for this purpose). Granted you can do this with classes/objects and defining interfaces the end result can be somewhat un-natural for some problems--but I don't want to get into an argument between closures vs. objects since that is not what my proposal is aimed at and Python already has both.

Of course using classes directly with a bit of work can offer you everything you want from a closure, with all of the explcitness that you could ever want.

Really, the easiest way to emulate what I want in today's Python is to create a mutable object (like a dict or list) in the enclosing scope to work around the semantic that the first assignment in a local scope binds a new name. Doing this seems rather un-natural and forcing the use of classes doesn't seem more natural def incgen( inc = 1 ) : env = [ 6 ] def incrementor() : env[ 0 ] += inc return env[ 0 ] return incrementor This is a work around for something a developer cannot do more naturally today. I do not think using some combination of classes and closures makes things clearer--it is still working around what I would construe as the non-orthogonal nature of nested lexical scopes in Python since the language provides a construct to deal with the problem for global variables. a = 6 def incgen( inc = 1 ) : def incrementor() : global a a += inc return a return incrementor Granted this is a somewhat trivial example, but I think it demonstrates my point about how nested lexical scopes are second class (since the language has no equivalent construct for them) and don't behave like the global scope.

As an aside, you mention both 'use' and 'scope' as possible keyword additions for various uses of nested scopes. In my experience, when one goes beyond 3 or so levels of nested scopes (methods of a class defined within a class namespace, or perhaps methods of a class defined within a method of a class), it starts getting to the point where the programmer is trying to be too clever.

Even though I may agree with you on this, your argument is more of an argument against PEP 227 than what I am proposing. Again, today's Python already allows a developer to have deep nested scopes. -Almann -- Almann T. Goo almann.goo@gmail.com

Josiah Carlson wrote:

Introducing these two new keywords is equivalent to encouraging nested scope use. Right now nested scope use is "limited" or "fraught with gotchas".

What you seem to be saying here is: Nested scope use is Inherently Bad. Therefore we will keep them Limited and Fraught With Gotchas, so people will be discouraged from using them. Sounds a bit like the attitude of certain religious groups to condoms. (Might encourage people to have sex -- can't have that -- look at all the nasty diseases you can get!) Greg

Greg Ewing wrote:

def my_func(): namespace foo foo.x = 42

def inc_x(): foo.x += 1

The idea here is that foo wouldn't be an object in its own right, but just a collection of names that would be implemented as local variables of my_func.

But why is that better than class namespace(object): pass def my_func(): foo = namespace() (...) ? Georg

Greg Ewing wrote:

Josiah Carlson wrote:

...

Mechanisms which rely on manipulating variables within closures or nested scopes to function properly can be elegant, but I've not yet seen one that *really* is.

It seems a bit inconsistent to say on the one hand that direct assignment to a name in an outer scope is not sufficiently useful to be worth supporting, while at the same time providing a way to do it for one particular scope, i.e. 'global'. Would you advocate doing away with it?

I didn't conceive of the idea or implementation of 'global', it was before my time. I have found that *using* global can be convenient (and sometimes even directly manipulating globals() can be even more convenient). However, I believe global was and is necessary for the same reasons for globals in any other language. Are accessors for lexically nested scopes necessary? Obviously no. The arguments for their inclusion are: easier access to parent scopes and potentially faster execution. The question which still remains in my mind, which I previously asked, is whether the use cases are compelling enough to warrant the feature addition.

...

Of course using classes directly with a bit of work can offer you everything you want from a closure, with all of the explcitness that you could ever want.

There are cases where the overhead (in terms of amount of code) of defining a class and creating an instance of it swamps the code which does the actual work, and, I feel, actually obscures what is being done rather than clarifies it. These cases benefit from the ability to refer to names in enclosing scopes, and I believe they would benefit further from the ability to assign to such names.

class namespace: pass def fcn(...): foo = namespace() ... Overwhelms the user?

Certainly the feature could be abused, as can the existing nested scope facilities, or any other language feature for that matter. Mere potential for abuse is not sufficient reason to reject a feature, or the language would have no features at all.

Indeed, but as I have asked, I would like to see some potential nontrivial *uses*. No one has responded to this particular request. When I am confronted with a lack of uses, and the potential for abuses, I'm going to have to side on "no thanks, the potential abuse outweighs the nonexistant nontrivial use".

Another consideration is efficiency. CPython currently implements access to local variables (both in the current scope and all outer ones except the module scope) in an extremely efficient way. There's always the worry that using attribute access in place of local variable access is greatly increasing the runtime overhead for no corresponding benefit.

Indeed, the only benefit to using classes is that you gain explicitness. To gain speed in current Python, one may need to do a bit more work (slots, call frame hacking, perhaps an AST manipulation with the new AST branch, etc.).

You mention the idea of namespaces. Maybe an answer is to provide some lightweight way of defining a temporary, singe-use namespace for use within nested scopes -- lightweight in terms of both code volume and runtime overhead. Perhaps something like

def my_func(): namespace foo foo.x = 42

def inc_x(): foo.x += 1

Because this discussion is not about "how do I create a counter in Python", let's see some examples which are not counters and which are improved through the use of this "namespace", or "use", "scope", etc.

...

Introducing these two new keywords is equivalent to encouraging nested scope use. Right now nested scope use is "limited" or "fraught with gotchas".

What you seem to be saying here is: Nested scope use is Inherently Bad. Therefore we will keep them Limited and Fraught With Gotchas, so people will be discouraged from using them.

Sounds a bit like the attitude of certain religious groups to condoms. (Might encourage people to have sex -- can't have that -- look at all the nasty diseases you can get!)

If you take that statement within the context of the other things I had been saying in regards to closures and nested scopes, namely that I find their use rarely, if ever, truely elegant, it becomes less like "condom use" as purported by some organizations, and more like kicking a puppy for barking: it is of my opinion that there are usually better ways of dealing with the problem (don't kick puppies for barking and don't use closures). - Josiah

Phillip J. Eby wrote:

def incrementer(val): def inc(): .val += 1 return .val return inc

-1, too obscure. -- Greg

Steven Bethard wrote:

And, as you mention, it's consistent with the relative import feature.

Only rather vaguely -- it's really somewhat different. With imports, .foo is an abbreviation for myself.foo, where myself is the absolute name for the current module, and you could replace all instances of .foo with that. But in the suggested scheme, .foo wouldn't have any such interpretation -- there would be no other way of spelling it. Also, with imports, the dot refers to a single well- defined point in the module-name hierarchy, but here it would imply a search upwards throught the scope hierarchy. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing@canterbury.ac.nz +--------------------------------------+

Steven Bethard wrote:

And, as you mention, it's consistent with the relative import feature.

Greg Ewing wrote:

With imports, .foo is an abbreviation for myself.foo, where myself is the absolute name for the current module, and you could replace all instances of .foo with that.

Phillip J. Eby wrote:

Actually, "import .foo" is an abbreviation for "import myparent.foo", not "import myparent.myself.foo".

If we wanted to be fully consistent with the relative import mechanism, we would require as many dots as nested scopes. So: def incrementer(val): def inc(): .val += 1 return .val return inc but also: def incrementer_getter(val): def incrementer(): def inc(): ..val += 1 return ..val return inc return incrementer (Yes, I know the example is silly. It's not meant as a use case, just to demonstrate the usage of dots.) I actually don't care which way it goes here, but if you want to make the semantics as close to the relative import semantics as possible, then this is the way to go. STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy

If we wanted to be fully consistent with the relative import mechanism, we would require as many dots as nested scopes.

At first I was a bit taken a back with the syntax, but after reading PEP 328 (re: Relative Import) I think I can stomach the syntax a bit better ; ). That said, -1 because I believe it adds more problems than the one it is designed to fix. Part of me can appreciate using the prefixing "dot" as a way to spell "my parent's scope" since it does not add a new keyword and in this regard would appear to be equally as backwards compatible as the ":=" proposal (to which I am not a particularly big fan of either but could probably get used to it). Since the current semantics allow *evaluation* to an enclosing scope's name by an "un-punctuated" name, "var" is a synonym to ".var" (if "var" is bound in the immediately enclosing scope). However for *re-binding* to an enclosing scope's name, the "punctuated" name is the only one we can use, so the semantic becomes more cluttered. This can make a problem that I would say is akin to the "dangling else problem." def incrementer_getter(val): def incrementer(): val = 5 def inc(): ..val += 1 return val return inc return incrementer Building on an example that Steve wrote to demonstrate the syntax proposed, you can see that if a user inadvertently uses the enclosing scope for the return instead of what would presumably be the outer most bound parameter. Now remove the binding in the incrementer function and it works the way the user probably thought. Because of this, I think by adding the "dot" to allow resolving a name in an explicit way hurts the language by adding a new "gotcha" with existing name binding semantics. I would be okay with this if all name access for enclosing scopes (binding and evaluation) required the "dot" syntax (as I believe Steve suggests for Python 3K)--thus keeping the semantics cleaner--but that would be incredibly backwards incompatible for what I would guess is *a lot* of code. This is where the case for the re-bind operator (i.e. ":=") or an "outer" type keyword is stronger--the semantics in the language today are not adversely affected. -Almann -- Almann T. Goo almann.goo@gmail.com

On 2/23/06, Steven Bethard wrote:

On 2/22/06, Almann T. Goo wrote:

...

def incrementer_getter(val): def incrementer(): val = 5 def inc(): ..val += 1 return val return inc return incrementer

Sorry, what way did the user think? I'm not sure what you think was supposed to happen.

My apologies ... I shouldn't use vague terms like what the "user thinks." My problem, as is demonstrated in the above example, is that the implicit nature of evaluating a name in Python conflicts with the explicit nature of the proposed "dot" notation. It makes it easier for a user to write obscure code (until Python 3K when we force users to use "dot" notation for all enclosing scope access ;-) ). This sort of thing can be done today with code using attribute access on its module object to evaluate and rebind global names. With the "global" keyword however, users don't have to resort to this sort of trick. Because of Python's name binding semantics and the semantic for the "global" keyword, I think the case for an "outer"-type keyword is stronger and we could deprecate "global" going forward in Python 3K. One of the biggest points of contention to this is of course the backwards incompatibility with a new keyword ... Python has already recently added "yield" and we're about to get "with" and "as" in 2.5. As far as the "user-interface" of the language getting bloated, I personally think trading "global" for an "outer" mitigates that some. -Almann -- Almann T. Goo almann.goo@gmail.com

On 2/26/06, Greg Ewing wrote:

Alternatively, 'global' could be redefined to mean what we're thinking of for 'outer'. Then there would be no change in keywordage.

There would be potential for breaking code, but I suspect the actual amount of breakage would be small, since there would have to be 3 scopes involved, with something in the middle one shadowing a global that was referenced in the inner one with a global statement.

Given the rarity of global statement usage to begin with, I'd say that narrows things down to something well within the range of acceptable breakage in 3.0.

You read my mind--I made a reply similar to this on another branch of this thread just minutes ago :). I am curious to see what the community thinks about this. -Almann -- Almann T. Goo almann.goo@gmail.com

Terry Reedy wrote:

"Almann T. Goo" wrote in message news:7e9b97090602252315mf6d4686ud86dd5163ea76b37@mail.gmail.com...

...

...

Alternatively, 'global' could be redefined to mean what we're thinking of for 'outer'. Then there would be no change in keywordage. Given the rarity of global statement usage to begin with, I'd say that narrows things down to something well within the range of acceptable breakage in 3.0. You read my mind--I made a reply similar to this on another branch of

On 2/26/06, Greg Ewing wrote: this thread just minutes ago :).

I am curious to see what the community thinks about this.

I *think* I like this better than more complicated proposals. I don't think I would ever have a problem with the intermediate scope masking the module scope. After all, if I really meant to access the current global scope from a nested function, I simply would not use that name in the intermediate scope.

tjr

Would this apply to reading intermediate scopes without the global keyword? How would you know you aren't in inadvertently masking a name in a function you call? In most cases it will probably break something in an obvious way, but I suppose in some cases it won't be so obvious. Ron

On 2/26/06, Alex Martelli wrote:

For the record: I detest the existing 'global' (could I change but ONE thing in Python, that would be the one -- move from hated 'global' to a decent namespace use, e.g. glob.x=23 rather than global x;x=23), and I'd detest a similar 'outer' just as intensely (again, what I'd like instead is a decent namespace) -- so I might well be sympathetic to your POV, if I could but understand it;-).

I would prefer a more explicit means to accomplish this too (I sort of like the prefix dot in this regard), however the fundamental problem with allowing this lies in how accessing and binding names works in Python today (sorry if I sound like a broken record in this regard). Unless we change how names can be accessed/re-bound (very bad for backwards compatibility), any proposal that forces explicit name spaces would have to allow for both accessing "simple names" (like just "var") and names via attribute access (name spaces) like "glob.var"--I think this adds the problem of introducing obscurity to the language. -Almann -- Almann T. Goo almann.goo@gmail.com

On Feb 26, 2006, at 4:20 PM, Ron Adam wrote: ...

(sigh of relief) Ok, so the following example will still be true.

Yep, no danger of dynamic scoping, be certain of that.

Maybe something explicit like:

...

...

...

import __main__ as glob

Sure, or the more general ''glob=__import__(__name__)''.

I'm -1 on adding the intermediate (outer) scopes to functions. I'd even like to see closures gone completely, but there's probably a reason they are there. What I like about functions is they are fast, clean up behind themselves, and act *exactly* the same on consecutive calls.

Except that the latter assertion is just untrue in Python -- we already have a bazilion ways to perform side effects, and, since there is no procedure/function distinction, side effects in functions are an extremely common thing. If you're truly keen on having the "exactly the same" property, you may want to look into functional languages, such as Haskell -- there, all data is immutable, so the property does hold (any *indispensable* side effects, e.g. I/O, are packed into 'monads' -- but that's another story). Closures in Python are often extremely handy, as long as you use them much as you would in Haskell -- treating data as immutable (and in particular outer names as unrebindable). You'd think that functional programming fans wouldn't gripe so much about Python closures being meant for use like Haskell ones, hm?-) But, of course, they do want to have their closure and rebind names too... Alex

On Feb 26, 2006, at 5:43 PM, Ron Adam wrote: ...

So far everywhere I've seen closures used, a class would work. But maybe not as conveniently or as fast?

Yep. In this, closures are like generators: much more convenient than purpose-built classes, but not as general.

Haskel sounds interesting, maybe I'll try a bit of it sometime. But I like Python. ;-)

So do I, so do many others: the first EuroHaskell was held the day right after a EuroPython, in the same venue (a Swedish University, Chalmers) -- that was convenient because so many delegates were interested in both languages, see. We stole list comprehensions and genexps from Haskell (the idea and most of the semantics, not the syntax, which was Pythonized relentlessly) -- and the two languages share the concept of indentation being significant for grouping, with some minor differences in details since they developed these concepts independently. Hey, what more do you need?-) Alex

[Alex Martelli]

...

We stole list comprehensions and genexps from Haskell

[Greg Ewing]

The idea predates Haskell, I think. I first saw it in Miranda, and it may have come from something even earlier -- SETL, maybe?

Haskell indeed took list comprehensions from SETL. SETL in turn took them from much earlier standard notation in set theory, related to the oddly named (but not universally so named) "axiom of comprehension". genexps were more directly taken from Icon, but tying them into Python's iteration protocol is a powerful twist not directly present in Icon.

Tim Peters wrote:

[Alex Martelli]

...

...

We stole list comprehensions and genexps from Haskell

[Greg Ewing]

...

The idea predates Haskell, I think. I first saw it in Miranda, and it may have come from something even earlier -- SETL, maybe?

Haskell indeed took list comprehensions from SETL. SETL in turn adopted them from pre-computer standard notation in set theory, related to the oddly named (but not universally so named) "axiom of comprehension".

genexps were more directly taken from Icon (because of the "generator" part).

SETL and Icon, of course, both have their roots in SNOBOL4, which was designed by Griswold when he worked at (IIRC) Bell Labs. Robert Dewar produced the machine-independent SPITBOL implementation (which I ported to DecSystem 10 as an undergraduate project at Leeds University). Griswold later went to the University of Arizona and developed Icon, Dewar went to Rutgers (I think) and developed SETL. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd www.holdenweb.com Love me, love my blog holdenweb.blogspot.com

Would this apply to reading intermediate scopes without the global keyword?

Using a name from an enclosing scope without re-binding to it would not require the "global" keyword. This actually is the case today with "global" and accessing a name from the global scope versus re-binding to it--this would make "global" more general than explicitly overriding to the global scope.

How would you know you aren't in inadvertently masking a name in a function you call?

I think is really an issue with the name binding semantics in Python. There are benefits to not having variable declarations, but with assignment meaning bind locally, you can already shadow a name in a nested scope inadvertently today.

In most cases it will probably break something in an obvious way, but I suppose in some cases it won't be so obvious.

Having the "global" keyword semantics changed to be "lexically global" would break in the cases that "global" is used on a name within a nested scope that has an enclosing scope with the same name. I would suppose that actual instances in real code of this would be rare. Consider:

...

...

x = 1 def f() : ... x = 2 ... def inner() : ... global x ... print x ... inner() ... f() 1

Under the proposed rules:

...

...

f() 2

PEP 227 also had backwards incompatibilities that were similar and I suggest handling them the same way by issuing a warning in these cases when the new semantics are not being used (i.e. no "from __future__"). -Almann -- Almann T. Goo almann.goo@gmail.com

On 2/26/06, Steven Bethard wrote:

Then do you also dislike the original proposal: that only a single dot be allowed, and that the '.' would mean "this name, but in the nearest outer scope that defines it"? Then:

def incrementer_getter(val): def incrementer(): val = 5 def inc(): .val += 1 return val return inc return incrementer

would do what I think you want it to[1]. Note that I only suggested extending the dot-notation to allow multiple dots because of Greg Ewing's complaint that it wasn't enough like the relative import notation. Personally I find PJE's original proposal more intuitive, and based on your example, I suspect so do you.

[1] That is, increment the ``val`` in incrementer(), return the same ``val``, and never modify the ``val`` in incrementer_getter().

I'm not sure if I find this more intuitive, but I think it is more convenient than the "explicit dots" for each scope. However my biggest issue is still there. I am not a big fan of letting users have synonyms for names. Notice how ".var" means the same as "var" in some contexts in the example above--that troubles me. PEP 227 addresses this concern with regard to the class scope: Names in class scope are not accessible. Names are resolved in the innermost enclosing function scope. If a class definition occurs in a chain of nested scopes, the resolution process skips class definitions. This rule prevents odd interactions between class attributes and local variable access. As the PEP further states: An alternative would have been to allow name binding in class scope to behave exactly like name binding in function scope. This rule would allow class attributes to be referenced either via attribute reference or simple name. This option was ruled out because it would have been inconsistent with all other forms of class and instance attribute access, which always use attribute references. Code that used simple names would have been obscure. I especially don't want to add an issue that is similar to one that PEP 227 went out of its way to avoid. -Almann -- Almann T. Goo almann.goo@gmail.com

On 2/21/06, Josiah Carlson wrote:

The question which still remains in my mind, which I previously asked, is whether the use cases are compelling enough to warrant the feature addition.

I don't know whether I support the proposal or not, but in reading Mark Russel's email, I realized that I just recently ran into a use case: ---------------------------------------------------------------------- # group tokens into chunks by their chunk labels token_groups = [] curr_suffix = '' curr_tokens = [] for token in document.IterAnnotations('token', percent=80): label = token[attr_name] # determine the prefix and suffix of the label prefix, suffix = label[0], label[2:] # B labels start a new chunk if prefix == 'B': curr_suffix = suffix curr_tokens = [token] token_groups.append((curr_suffix, curr_tokens)) # I labels continue the previous chunk elif prefix == 'I': if curr_suffix == suffix: curr_tokens.append(token) # error: change in suffix - this should be a B label else: # log the error message = '%r followed by %r' last_label = curr_tokens[-1][attr_name] self._logger.info(message % (last_label, label)) # start a new chunk curr_suffix = suffix curr_tokens = [token] token_groups.append((curr_suffix, curr_tokens)) # O labels end any previous chunks elif prefix == 'O': curr_suffix = suffix curr_tokens = [token] ---------------------------------------------------------------------- You can see that the code:: curr_suffix = suffix curr_tokens = [token] token_groups.append((curr_suffix, curr_tokens)) is repeated in two places. I would have liked to factor this out into a function, but since the code requires rebinding curr_suffix and curr_tokens, I can't. I'm not sure I care that much -- it's only three lines of code and only duplicated once -- but using something like ``curr_suffix :=`` or Phillip J. Eby's suggestion of ``.curr_suffix =`` would allow this code to be factored out into a function. STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy

Josiah Carlson wrote:

In this particular example, there is no net reduction in line use. The execution speed of your algorithm would be reduced due to function calling overhead.

If there were more uses of the function, the line count reduction would be greater. In any case, line count and execution speed aren't the only issues -- there is DRY to consider. -- Greg

Oddly, in Python, 'global' isn't actually necessary, since the module can always import itself and use attribute access.

Clearly, though, Guido must have thought at the time that it was worth providing an alternative way.

I believe that use cases for rebinding globals (module attributes) from within a module are more numerous than rebinding in an enclosing lexical scope (although rebinding a name in the global scope from a local scope is really just a specific case of that). I would think this was probably a motivator for the 'global' key word to avoid clumsier workarounds. Since there were no nested lexical scopes back then, there was no need to have a construct for arbitrary enclosing scopes. -Almann -- Almann T. Goo almann.goo@gmail.com

On 2/22/06, Greg Ewing wrote:

That's what rankles people about this, I think -- there doesn't seem to be a good reason for treating the global scope so specially, given that all scopes could be treated uniformly if only there were an 'outer' statement. All the arguments I've seen in favour of the status quo seem like rationalisations after the fact.

I agree, hence my initial pre-PEP feeler on the topic ;).

...

Since there were no nested lexical scopes back then, there was no need to have a construct for arbitrary enclosing scopes.

However, if nested scopes *had* existed back then, I rather suspect we would have had an 'outer' statement from the beginning, or else 'global' would have been given the semantics we are now considering for 'outer'.

Would it not be so horrible to make "global" be the "outer"-type keyword--basically meaning "lexically global" versus "the global scope"? It would make the semantics for Python's nested lexical scopes to be more in line with other languages with this feature and fix my orthogonality gripes. As far as backwards compatibility, I doubt there would be too much impact in this regard, as places that would break would be where "global" was used in a closure where the name was shadowed in an enclosing scope. A "from __future__ import lexical_global" (which we'd have for adding the "outer"-like keyword anyway) could help diminish the growing pains. -Almann -- Almann T. Goo almann.goo@gmail.com

Jeremy, I definitely agree that option one is more in line with the semantics in place within Python today.

The names of naming statements are quite hard to get right, I fear. I don't particularly like "use." It's too generic. (I don't particularly like "scope" for option 2, either, for similar reasons. It doesn't indicate what kind of scope issue is being declared.) The most specifc thing I can think of is "free" to indicate that the variable is free in the current scope. It may be too specialized a term to be familiar to most people.

I am not married to any particular keyword for sure--I would be happy for the most part if the language was fixed regardless of the keyword chosen. "free" gives me the sense that I am de-allocating memory (my C background talking), I don't think most people would get the mathematical reference for "free". I certainly hope that an initiative like this doesn't get stymied by the lack of a good name for such a keyword. Maybe something like "outer"?

I think free == global in the absence of other bindings.

I actually like this, would sort of make "global" obsolete (and thus making the global scope behave like other lexical scopes with regard to to re-binding, which is probably a good thing) -Almann -- Almann T. Goo almann.goo@gmail.com

On 2/21/06, Terry Reedy wrote:

"Almann T. Goo" wrote in message news:7e9b97090602210516o5d1a823apedcea66846a271b5@mail.gmail.com...

...

I certainly hope that an initiative like this doesn't get stymied by the lack of a good name for such a keyword. Maybe something like "outer"?

Adding a keyword has a cost that you have so far ignored. Guido is rightfully very cautious about additions, especially for esthetic reasons.

The issue of rebinding enclosed names was partly discussed in PEP 227. Sometime after the implementation of the PEP in 2.1, it was thoroughly discussed again (100+ posts?) in this forum. There were perhaps 10 different proposals, including, I believe, 'outer'. Guido rejected them all as having costs greater than the benefits. Perhaps you can find this discussion in the archives. I remember it as a Jan-Feb discussion but might be wrong.

If I recall the discussion correctly, Guido said he was open to a version of nested scopes that allowed rebinding. Not sure that the specifics of the previous discussion are necessary, but I recall being surprised by the change in opinion since 2.1 :-). Jeremy

This thread so far seems like a rehash of parts of the earlier discussion. In the absence of indication from Guido that he is ready to reopen the issue, perhaps it would better go to comp.lang.python. In and case, reconsideration is more likely to be stimulated by new experience with problems in real code than by repeats of 'orthogonality' desires and rejected changes.

---

In another post, you rejected the use of class instances by opining:

...

Because I think that this is a workaround for a concept that the language doesn't support elegantly with its lexically nested scopes.

...

IMO, you are emulating name rebinding in a closure by creating an object to encapsulate the name you want to rebind

Guido, on the other hand, views classes and instances as Python's method of doing what other (functional) languages do with closures. From the PEP: "Given that this would encourage the use of local variables to hold state that is better stored in a class instance, it's not worth adding new syntax to make this possible (in Guido's opinion)." He reiterated this viewpoint in the post-PEP discussion mentioned above. I think he would specificly reject the view that Python's alternative is a 'workaround' and 'emulation' of what you must consider to be the real thing.

Terry Jan Reedy

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu

Mark Russell wrote:

PEP 227 mentions using := as a rebinding operator, but rejects the idea as it would encourage the use of closures. But to me it seems more elegant than some special keyword, especially is it could also replace the "global" keyword. It doesn't handle things like "x += y" but I think you could deal with that by just writing "x := x + y".

Actually, it could handle += just fine, since that operator has written "rebinding" all over it... I'd be +1 on := (and augmented assignment being rebinding), but the argument against it (if I recall correctly) was that rebinding should be a property of the name, not of the operator. Yet "declaring" a name local is also done trough an operator: a = 1 means a is local (unless it was declared global). It can definitely be argued either way. Btw, PJE's "crazy" idea (.name, to rebind an outer name) was proposed before, but Guido wanted to reserve .name for a (Pascal-like) 'with' statement. Hmm, http://mail.python.org/pipermail/python-dev/2004-March/043545.html confirms that, although it wasn't in response to a rebinding syntax. So maybe it wasn't proposed before after all... Just

Mark Russell wrote:

On 21 Feb 2006, at 19:25, Jeremy Hylton wrote:

...

If I recall the discussion correctly, Guido said he was open to a version of nested scopes that allowed rebinding.

PEP 227 mentions using := as a rebinding operator, but rejects the idea as it would encourage the use of closures. But to me it seems more elegant than some special keyword, especially is it could also replace the "global" keyword. It doesn't handle things like "x += y" but I think you could deal with that by just writing "x := x + y".

By rebinding operator, does that mean it is actually an operator? I.e.: # Required assignment to declare?: chunk = None while chunk := f.read(1000): ... -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org

Mark Russell wrote:

PEP 227 mentions using := as a rebinding operator, but rejects the idea as it would encourage the use of closures.

Well, anything that facilitates rebinding in outer scopes is going to encourage the use of closures, so I can't see that as being a reason to reject a particular means of rebinding. You either think such rebinding is a good idea or not -- and that seems to be a matter of highly individual taste. On this particular idea, I tend to think it's too obscure as well. Python generally avoids attaching randomly-chosen semantics to punctuation, and I'd like to see it stay that way. -- Greg

"Jeremy Hylton" wrote in message news:e8bf7a530602211125k28fc64bcx62430c375d060a8b@mail.gmail.com...

If I recall the discussion correctly, Guido said he was open to a version of nested scopes that allowed rebinding.

Yes. Among other places, he said in http://article.gmane.org/gmane.comp.python.devel/25153/match=nested+scopes ''' Your PEP wonders why I am against allowing assignment to intermediate levels. Here's my answer: all the syntaxes that have been proposed to spell this have problems. So let's not provide a way to spell it. I predict that it won't be a problem. If it becomes a problem, we can add a way to spell it later. '' tjr

As far as I remember, Guido wasn't particularly opposed to the idea, but the discussion fizzled out after having failed to reach a consensus on an obviously right way to go about it.

My apologies for bringing this debated topic again to the front-lines--that said, I think there has been good, constructive things said again and sometimes it doesn't hurt to kick up an old topic. After pouring through some of the list archive threads and reading through this thread, it seems clear to me that the community doesn't seem all that keen on fixing issue--which was my goal to ferret out. For me this is one of those things where the Pythonic thing to do is not so clear--and that mysterious, enigmatic definition of what it means to be Pythonic can be quite individual so I definitely don't want to waste my time arguing what that means. The most compelling argument for not doing anything about it is that the use cases are probably not that many--that in itself makes me less apt to push much harder--especially since my pragmatic side agrees with a lot of what has been said to this regard. IMO, Having properly nested scopes in Python in a sense made having closures a natural idiom to the language and part of its "user interface." By not allowing the name re-binding it almost seems like that "user interface" has a rough edge that is almost too easy to get cut on. This in-elegance seems very un-Pythonic to me. Anyhow, good discussion. Cheers, Almann -- Almann T. Goo almann.goo@gmail.com

Almann T. Goo wrote:

...

As far as I remember, Guido wasn't particularly opposed to the idea, but the discussion fizzled out after having failed to reach a consensus on an obviously right way to go about it.

My apologies for bringing this debated topic again to the front-lines--that said, I think there has been good, constructive things said again and sometimes it doesn't hurt to kick up an old topic. After pouring through some of the list archive threads and reading through this thread, it seems clear to me that the community doesn't seem all that keen on fixing issue--which was my goal to ferret out.

For me this is one of those things where the Pythonic thing to do is not so clear--and that mysterious, enigmatic definition of what it means to be Pythonic can be quite individual so I definitely don't want to waste my time arguing what that means.

The most compelling argument for not doing anything about it is that the use cases are probably not that many--that in itself makes me less apt to push much harder--especially since my pragmatic side agrees with a lot of what has been said to this regard.

IMO, Having properly nested scopes in Python in a sense made having closures a natural idiom to the language and part of its "user interface." By not allowing the name re-binding it almost seems like that "user interface" has a rough edge that is almost too easy to get cut on. This in-elegance seems very un-Pythonic to me.

If you are looking for rough edges about nested scopes in Python this is probably worse:

...

...

x = [] for i in range(10): ... x.append(lambda : i) ... [y() for y in x] [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

although experienced people can live with it. The fact is that importing nested scope from the like of Scheme it was not considered that in Scheme for example, looping constructs introduce new scopes. So this work more as expected there. There were long threads about this at some point too. Idioms and features mostly never port straightforwardly from language to language. For example Python has nothing with the explicit context introduction and grouping of a Scheme 'let', so is arguable that nested scope code, especially with rebindings, would be less clear, readable than in Scheme (tastes in parenthesis kept aside).

Anyhow, good discussion.

Cheers, Almann

-- Almann T. Goo almann.goo@gmail.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/pedronis%40strakt.com

On Thu, Feb 23, 2006 at 05:25:30PM +1300, Greg Ewing wrote:

Samuele Pedroni wrote:

...

If you are looking for rough edges about nested scopes in Python this is probably worse:

...

...

...

x = [] for i in range(10): ... x.append(lambda : i) ... [y() for y in x] [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

As an aside, is there any chance that this could be changed in 3.0? I.e. have the for-loop create a new binding for the loop variable on each iteration.

You can't do that without introducing a whole new scope for the body of the 'for' loop, and that means (in the current rules) you can't assign to any function-local names in the for loop. The nested scope in that 'lambda' refers to the 'slot' for the variable 'i' in the outer namespace (in this case, the global one.) You can't 'remove' the binding, either; 'del' will not allow you to. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Feb 24, 2006, at 1:54 AM, Greg Ewing wrote:

Thomas Wouters wrote:

...

On Thu, Feb 23, 2006 at 05:25:30PM +1300, Greg Ewing wrote:

...

As an aside, is there any chance that this could be changed in 3.0? I.e. have the for-loop create a new binding for the loop variable on each iteration.

You can't do that without introducing a whole new scope for the body of the 'for' loop,

There's no need for that. The new scope need only include the loop variable -- everything else could still refer to the function's main scope.

No, that would be insane. You get the exact same problem, now even more confusing: l=[] for x in range(10): y = x l.append(lambda: (x, y)) print l[0]() With your suggestion, that would print (0, 9). Unless python grows a distinction between creating a binding and assigning to one as most other languages have, this problem is here to stay. James

Jeremy Hylton wrote:

The more practical complaint is that list comprehensions use the same namespace as the block that contains them. ... but I suspect we're stuck with the current behavior for backwards compatibility reasons.

There will be no backwards compatibility in 3.0, so perhaps this could be fixed then? Greg

On Tue, Feb 21, 2006 at 08:02:08AM -0500, Jeremy Hylton wrote:

The lack of support for rebinding names in enclosing scopes is certainly a wart. I think option one is a better fit for Python, because it more closely matches the existing naming semantics. Namely that assignment in a block creates a new name unless a global statement indicates otherwise. The revised rules would be that assignment creates a new name unless a global or XXX statement indicates otherwise.

I agree with Jeremy on this. I've been thinking about doing something like this myself, but never got 'round to it. It doesn't make working with closures much easier, and I doubt it'll encourage using closures much, but it does remove the wart of needing to use mutable objects to make them read-write.

The names of naming statements are quite hard to get right, I fear. I don't particularly like "use." It's too generic. (I don't particularly like "scope" for option 2, either, for similar reasons. It doesn't indicate what kind of scope issue is being declared.) The most specifc thing I can think of is "free" to indicate that the variable is free in the current scope. It may be too specialized a term to be familiar to most people.

I was contemplating 'enclosed' as a declaration, myself. Maybe, if there's enough of a consent on any name before Python 2.5a1 is released, and the feature isn't going to make it into 2.5, we could ease the introduction of a new keyword by issuing warning about the keyword in 2.5 already. (Rather than a future-import to enable it in 2.6.) Maybe, and only if there's no doubt about how it's going in, of course. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

I had to lookup top-post :-). On 2/21/06, Bengt Richter wrote:

On Tue, 21 Feb 2006 08:02:08 -0500, "Jeremy Hylton" wrote:

...

Jeremy Hey, only Guido is allowed to top-post. He said so ;-)

The Gmail UI makes it really easy to forget where the q

But to the topic, it just occurred to me that any outer scopes could be given names (including global namespace, but that would have the name global by default, so global.x would essentially mean what globals()['x'] means now, except it would be a name error if x didn't pre-exist when accessed via namespace_name.name_in_space notation.

namespace g_alias # g_alias.x becomes alternate spelling of global.x def outer(): namespace mezzanine a = 123 print a # => 123 print mezzanine.a # => 123 (the name space name is visible and functional locally) def inner(): print mezzanine.a => 123 mezznine.a =456 inner() print a # = 456 global.x = re-binds global x, name error if not preexisting.

This would allow creating mezzanine like an attribute view of the slots in that local namespace, as well as making namespace itself visible there, so the access to mezzanine would look like a read access to an ordinary object named mezzanine that happened to have attribute slots matching outer's local name space.

Efficiency might make it desirable not to extend named namespaces with new names, function locals being slotted in a fixed space tied into the frame (I think). But there are tricks I guess. Anyway, I hadn't seen this idea before. Seems

Regards, Bengt Richter

...

On 2/20/06, Almann T. Goo wrote:

...

I am considering developing a PEP for enabling a mechanism to assign to free variables in a closure (nested function). My rationale is that with the advent of PEP 227 , Python has proper nested lexical scopes, but can have undesirable behavior (especially with new developers) when a user makes wants to make an assignment to a free variable within a nested function. Furthermore, after seeing numerous kludges to "solve" the problem with a mutable object, like a list, as the free variable do not seem "Pythonic." I have also seen mention that the use of classes can mitigate this, but that seems, IMHO, heavy handed in cases when an elegant solution using a closure would suffice and be more appropriate--especially when Python already has nested lexical scopes.

I propose two possible approaches to solve this issue:

1. Adding a keyword such as "use" that would follow similar semantics as "global" does today. A nested scope could declare names with this keyword to enable assignment to such names to change the closest parent's binding. The semantic would be to keep the behavior we experience today but tell the compiler/interpreter that a name declared with the "use" keyword would explicitly use an enclosing scope. I personally like this approach the most since it would seem to be in keeping with the current way the language works and would probably be the most backwards compatible. The semantics for how this interacts with the global scope would also need to be defined (should "use" be equivalent to a global when no name exists all parent scopes, etc.)

def incgen( inc = 1 ) : a = 6 def incrementer() : use a #use a, inc <-- list of names okay too a += inc return a return incrementer

Of course, this approach suffers from a downside that every nested scope that wanted to assign to a parent scope's name would need to have the "use" keyword for those names--but one could argue that this is in keeping with one of Python's philosophies that "Explicit is better than implicit" (PEP 20). This approach also has to deal with a user declaring a name with " use" that is a named parameter--this would be a semantic error that could be handled like "global " does today with a SyntaxError.

2. Adding a keyword such as "scope" that would behave similarly to JavaScript's " var" keyword. A name could be declared with such a keyword optionally and all nested scopes would use the declaring scope's binding when accessing or assigning to a particular name. This approach has similar benefits to my first approach, but is clearly more top-down than the first approach. Subsequent "scope" declarations would create a new binding at the declaring scope for the declaring and child scopes to use. This could potentially be a gotcha for users expecting the binding semantics in place today. Also the scope keyword would have to be allowed to be used on parameters to allow such parameter names to be used in a similar fashion in a child scope.

def incgen( inc = 1 ) : #scope inc <-- allow scope declaration for bound parameters (not a big fan of this) scope a = 6 def incrementer() : a += inc return a return incrementer

This approach would be similar to languages like JavaScript that allow for explicit scope binding with the use of "var" or more static languages that allow re-declaring names at lower scopes. I am less in favor of this, because I don't think it feels very "Pythonic".

As a point of reference, some languages such as Ruby will only bind a new name to a scope on assignment when an enclosing scope does not have the name bound. I do believe the Python name binding semantics have issues (for which the "global" keyword was born), but I feel that the "fixing" the Python semantic to a more "Ruby-like" one adds as many problems as it solves since the "Ruby-like" one is just as implicit in nature. Not to mention the backwards compatibility impact is probably much larger.

I would like the community's opinion if there is enough out there that think this would be a worthwile endevour--or if there is already an initiative that I missed. Please let me know your questions, comments.

Best Regards, Almann

-- Almann T. Goo almann.goo@gmail.com _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gman...

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu

Jeremy Hylton wrote:

On 2/21/06, Jeremy Hylton wrote:

...

I had to lookup top-post :-).

On 2/21/06, Bengt Richter wrote:

...

On Tue, 21 Feb 2006 08:02:08 -0500, "Jeremy Hylton" wrote:

...

Jeremy Hey, only Guido is allowed to top-post. He said so ;-) The Gmail UI makes it really easy to forget where the q

Sorry about that. Hit the send key by mistake.

The Gmail UI makes it really easy to forget where the quoted text is in relation to your own text.

...

...

But to the topic, it just occurred to me that any outer scopes could be given names (including global namespace, but that would have the name global by default, so global.x would essentially mean what globals()['x'] means now, except it would be a name error if x didn't pre-exist when accessed via namespace_name.name_in_space notation.

Isn't this suggestion that same as Greg Ewing's?

...

...

namespace g_alias # g_alias.x becomes alternate spelling of global.x def outer(): namespace mezzanine a = 123 print a # => 123 print mezzanine.a # => 123 (the name space name is visible and functional locally) def inner(): print mezzanine.a => 123 mezznine.a =456 inner() print a # = 456 global.x = re-binds global x, name error if not preexisting.

This would allow creating mezzanine like an attribute view of the slots in that local namespace, as well as making namespace itself visible there, so the access to mezzanine would look like a read access to an ordinary object named mezzanine that happened to have attribute slots matching outer's local name space.

Why not just use a class? def incgen(start=0, inc=1) : class incrementer(object): a = start - inc def __call__(self): self.a += inc return self.a return incrementer() a = incgen(7, 5) for n in range(10): print a(), 7 12 17 22 27 32 37 42 47 52 Cheers, Ronald Adam

Jeremy Hylton wrote:

On 2/21/06, Jeremy Hylton wrote:

...

On 2/21/06, Bengt Richter wrote:

...

...

But to the topic, it just occurred to me that any outer scopes could be given names (including global namespace, but that would have the name global by default, so global.x would essentially mean what globals()['x'] means now, except it would be a name error if x didn't pre-exist when accessed via namespace_name.name_in_space notation.

Isn't this suggestion that same as Greg Ewing's?

It's not quite the same, because in my scheme the namespace statement creates a new namespace embedded in the scope where it appears, whereas Bengt's one seems to just give a name to the scope itself. I'm not really in favour of either of these -- I'd be just as happy with a simple 'outer' statement. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing@canterbury.ac.nz +--------------------------------------+

Jeremy Hylton wrote:

The names of naming statements are quite hard to get right, I fear.

My vote goes for 'outer'. And if this gets accepted, remove 'global' in 3.0. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing@canterbury.ac.nz +--------------------------------------+