[Paul Prescod]

... Along the same lines, might a new rule make Python code more robust? We could say that a local can only shadow a global if the local is formally declared. It's pretty rare that there is a good reason to shadow a global and Python makes it too easy to do accidentally.

I've rarely seen problems due to shadowing a global, but have often seen problems due to shadowing a builtin. Alas, if this rule were extended to builtins too-- where it would do the most good --then the names of builtins would effectively become reserved words (any code shadowing them today would be broken until declarations were added, and any code working today may break tomorrow if a new builtin were introduced that happened to have the same name as a local).

[Tim]

...

I've rarely seen problems due to shadowing a global, but have often seen problems due to shadowing a builtin.

[Paul Prescod]

Really?

Yes.

I think that there are two different issues here. One is consciously choosing to create a new variable but not understanding that there already exists a variable by that name. (i.e. str, list).

Yes, and that's what I've often seen, typically long after the original code is written: someone sticks in some debugging output, or makes a small change to the implementation, and introduces e.g. str = some_preexisting_var + ":" yadda(str) "Suddenly" the program misbehaves in baffling ways. They're "baffling" because the errors do not occur on the lines where the changes were made, and are almost never related to the programmer's intent when making the changes.

Another is trying to assign to a global but actually shadowing it.

I've rarely seen that.

There is no way that anyone coming from another language is going to consider this transcript reasonable:

True, but I don't really care: everyone gets burned once, the better ones eventually learn to use classes instead of mutating globals, and even the dull get over it. It is not, in my experience, an on-going problem for anyone. But I still get burned regularly by shadowing builtins. The burns are not fatal, however, and I can't think of an ointment less painful than the blisters.

...

...

...

a=5 def show(): ... print a ... def set(val): ... a=val ... a 5 show() 5 set(10) show() 5

It doesn't seem to make any sense. My solution is to make the assignment in "set" illegal unless you add a declaration that says: "No, really. I mean it. Override that sucker." As the PEP points out, overriding is seldom a good idea so the requirement to declare would be rarely invoked.

I expect it would do less harm to introduce a compile-time warning for locals that are never referenced (such as the "a" in "set").

... The "right answer" in terms of namespace theory is to consistently refer to builtins with a prefix (whether "__builtins__" or "$") but that's pretty unpalatable from an aesthetic point of view.

Right, that's one of the ointments I won't apply to my own code, so wouldn't think of asking others to either. WRT mutable globals, people who feel they have to use them would be well served to adopt a naming convention. For example, begin each name with "g" and capitalize the second letter. This can make global-rich code much easier to follow (I've done-- and very happily --similar things in Javascript and C++).

[Tim]

I expect it would do less harm to introduce a compile-time warning for locals that are never referenced (such as the "a" in "set").

[Guido]

Another warning that would be quite useful (and trap similar cases) would be "local variable used before set".

Java elevated that last one to a compile-time error, via its "definite assignment" rules: you not only have to make sure a local is bound before reference, you have to make it *obvious* to the compiler that it's bound before reference. I think this is a Good Thing, because with intense training, people can learn to think like a compiler too <wink>. Seriously, in several of the cases where gcc warned about "maybe used before set" in the Python implementation, the warnings were bogus but it was non-trivial to deduce that. Such code is very brittle under modification, and the definite assignment rules make that path to error a non-starter. Example: def f(N): if N > 0: for i in range(N): if i == 0: j = 42 else: f2(i) elif N <= 0: j = 24 return j It's a Crime Against Humanity to make the code reader *deduce* that j is always bound by the time "return" is executed.

Greg Ewing wrote:

Paul Prescod <paulp@ActiveState.com>:

...

We could say that a local can only shadow a global if the local is formally declared.

How do you intend to enforce that? Seems like it would require a test on every assignment to a local, to make sure nobody has snuck in a new global since the function was compiled.

I would expect that all of the checks would be at compile-time. Except for __dict__ hackery, I think it is doable. Python already keeps track of all assignments to locals and all assignments to globals in a function scope. The only addition is keeping track of assignments at a global scope.

...

Actually, one could argue that there is no good reason to even *allow* the shadowing of globals.

If shadowing were completely disallowed, it would make it impossible to write a completely self-contained function whose source could be moved from one environment to another without danger of it breaking. I wouldn't like the language to have a characteristic like that.

That seems like a very esoteric requirement. How often do you have functions that do not rely *at all* on their environment (other functions, import statements, global variables). When you move code you have to do some rewriting or customizing of the environment in 94% of the cases. How much effort do you want to spend on the other 6%? Also, there are tools that are designed to help you move code without breaking programs (refactoring editors). They can just as easily handle renaming local variables as adding import statements and fixing up function calls. Paul Prescod

Guido van Rossum wrote:

...

Can we cut this out please? Paul is misguided. There's no reason to forbid a local shadowing a global. All languages with nested scopes allow this.

Python is the only one I know of that implicitly shadows without requiring some form of declaration. JavaScript has it right: reading and writing of globals are symmetrical. In the rare case that you explicitly want to shadow, you need a declaration. Python's rule is confusing, implicit and error causing. In my opinion, of course. If you are dead-set against explicit declarations then I would say that disallowing the ambiguous construct is better than silently treating it as a declaration. Paul Prescod

Jeremy Hylton wrote:

...

...

...

...

...

"MAL" == M -A Lemburg <mal@lemburg.com> writes:

MAL> Jeremy Hylton wrote:

...

...

I've got a new draft of PEP 227. The terminology and wording are more convoluted than they need to be. I'll do at least one revision just to say things more clearly, but I'd appreciate comments on the proposed spec if you can read the current draft.

MAL> The PEP doesn't mention the problems I pointed out about MAL> breaking the lookup schemes w/r to symbols in methods, classes MAL> and globals.

I believe it does. There was some discussion on python-dev and with others in private email about how classes should be handled.

The relevant section of the specification is:

If a name is used within a code block, but it is not bound there and is not declared global, the use is treated as a reference to the nearest enclosing function region. (Note: If a region is contained within a class definition, the name bindings that occur in the class block are not visible to enclosed functions.)

Well hidden ;-) Honestly, I think that you should either make this specific case more visible to readers of the PEP since this single detail would produce most of the problems with nested scopes. BTW, what about nested classes ? AFAIR, the PEP only talks about nested functions.

MAL> Please add a comment about this to the PEP + maybe the example MAL> I gave in one the posts to python-dev about it. I consider the MAL> problem serious enough to limit the nested scoping to lambda MAL> functions (or functions in general) only if that's possible.

If there was some other concern you had, then I don't know what it was. I recall that you had a longish example that raised a NameError immediately :-).

The idea behind the example should have been clear, though. x = 1 class C: x = 2 def test(self): print x -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

[Paul Prescod]

... Along the same lines, might a new rule make Python code more robust? We could say that a local can only shadow a global if the local is formally declared. It's pretty rare that there is a good reason to shadow a global and Python makes it too easy to do accidentally.

I've rarely seen problems due to shadowing a global, but have often seen problems due to shadowing a builtin. Alas, if this rule were extended to builtins too-- where it would do the most good --then the names of builtins would effectively become reserved words (any code shadowing them today would be broken until declarations were added, and any code working today may break tomorrow if a new builtin were introduced that happened to have the same name as a local).

[Tim]

...

I've rarely seen problems due to shadowing a global, but have often seen problems due to shadowing a builtin.

[Paul Prescod]

Really?

Yes.

I think that there are two different issues here. One is consciously choosing to create a new variable but not understanding that there already exists a variable by that name. (i.e. str, list).

Yes, and that's what I've often seen, typically long after the original code is written: someone sticks in some debugging output, or makes a small change to the implementation, and introduces e.g. str = some_preexisting_var + ":" yadda(str) "Suddenly" the program misbehaves in baffling ways. They're "baffling" because the errors do not occur on the lines where the changes were made, and are almost never related to the programmer's intent when making the changes.

Another is trying to assign to a global but actually shadowing it.

I've rarely seen that.

There is no way that anyone coming from another language is going to consider this transcript reasonable:

True, but I don't really care: everyone gets burned once, the better ones eventually learn to use classes instead of mutating globals, and even the dull get over it. It is not, in my experience, an on-going problem for anyone. But I still get burned regularly by shadowing builtins. The burns are not fatal, however, and I can't think of an ointment less painful than the blisters.

...

...

...

a=5 def show(): ... print a ... def set(val): ... a=val ... a 5 show() 5 set(10) show() 5

It doesn't seem to make any sense. My solution is to make the assignment in "set" illegal unless you add a declaration that says: "No, really. I mean it. Override that sucker." As the PEP points out, overriding is seldom a good idea so the requirement to declare would be rarely invoked.

I expect it would do less harm to introduce a compile-time warning for locals that are never referenced (such as the "a" in "set").

... The "right answer" in terms of namespace theory is to consistently refer to builtins with a prefix (whether "__builtins__" or "$") but that's pretty unpalatable from an aesthetic point of view.

Right, that's one of the ointments I won't apply to my own code, so wouldn't think of asking others to either. WRT mutable globals, people who feel they have to use them would be well served to adopt a naming convention. For example, begin each name with "g" and capitalize the second letter. This can make global-rich code much easier to follow (I've done-- and very happily --similar things in Javascript and C++).

[Tim]

I expect it would do less harm to introduce a compile-time warning for locals that are never referenced (such as the "a" in "set").

[Guido]

Another warning that would be quite useful (and trap similar cases) would be "local variable used before set".

Java elevated that last one to a compile-time error, via its "definite assignment" rules: you not only have to make sure a local is bound before reference, you have to make it *obvious* to the compiler that it's bound before reference. I think this is a Good Thing, because with intense training, people can learn to think like a compiler too <wink>. Seriously, in several of the cases where gcc warned about "maybe used before set" in the Python implementation, the warnings were bogus but it was non-trivial to deduce that. Such code is very brittle under modification, and the definite assignment rules make that path to error a non-starter. Example: def f(N): if N > 0: for i in range(N): if i == 0: j = 42 else: f2(i) elif N <= 0: j = 24 return j It's a Crime Against Humanity to make the code reader *deduce* that j is always bound by the time "return" is executed.

Greg Ewing wrote:

Paul Prescod <paulp@ActiveState.com>:

...

We could say that a local can only shadow a global if the local is formally declared.

How do you intend to enforce that? Seems like it would require a test on every assignment to a local, to make sure nobody has snuck in a new global since the function was compiled.

I would expect that all of the checks would be at compile-time. Except for __dict__ hackery, I think it is doable. Python already keeps track of all assignments to locals and all assignments to globals in a function scope. The only addition is keeping track of assignments at a global scope.

...

Actually, one could argue that there is no good reason to even *allow* the shadowing of globals.

If shadowing were completely disallowed, it would make it impossible to write a completely self-contained function whose source could be moved from one environment to another without danger of it breaking. I wouldn't like the language to have a characteristic like that.

That seems like a very esoteric requirement. How often do you have functions that do not rely *at all* on their environment (other functions, import statements, global variables). When you move code you have to do some rewriting or customizing of the environment in 94% of the cases. How much effort do you want to spend on the other 6%? Also, there are tools that are designed to help you move code without breaking programs (refactoring editors). They can just as easily handle renaming local variables as adding import statements and fixing up function calls. Paul Prescod

Guido van Rossum wrote:

...

Can we cut this out please? Paul is misguided. There's no reason to forbid a local shadowing a global. All languages with nested scopes allow this.

Python is the only one I know of that implicitly shadows without requiring some form of declaration. JavaScript has it right: reading and writing of globals are symmetrical. In the rare case that you explicitly want to shadow, you need a declaration. Python's rule is confusing, implicit and error causing. In my opinion, of course. If you are dead-set against explicit declarations then I would say that disallowing the ambiguous construct is better than silently treating it as a declaration. Paul Prescod

Jeremy Hylton wrote:

...

...

...

...

...

"MAL" == M -A Lemburg <mal@lemburg.com> writes:

MAL> Jeremy Hylton wrote:

...

...

I've got a new draft of PEP 227. The terminology and wording are more convoluted than they need to be. I'll do at least one revision just to say things more clearly, but I'd appreciate comments on the proposed spec if you can read the current draft.

MAL> The PEP doesn't mention the problems I pointed out about MAL> breaking the lookup schemes w/r to symbols in methods, classes MAL> and globals.

I believe it does. There was some discussion on python-dev and with others in private email about how classes should be handled.

The relevant section of the specification is:

If a name is used within a code block, but it is not bound there and is not declared global, the use is treated as a reference to the nearest enclosing function region. (Note: If a region is contained within a class definition, the name bindings that occur in the class block are not visible to enclosed functions.)

Well hidden ;-) Honestly, I think that you should either make this specific case more visible to readers of the PEP since this single detail would produce most of the problems with nested scopes. BTW, what about nested classes ? AFAIR, the PEP only talks about nested functions.

MAL> Please add a comment about this to the PEP + maybe the example MAL> I gave in one the posts to python-dev about it. I consider the MAL> problem serious enough to limit the nested scoping to lambda MAL> functions (or functions in general) only if that's possible.

If there was some other concern you had, then I don't know what it was. I recall that you had a longish example that raised a NameError immediately :-).

The idea behind the example should have been clear, though. x = 1 class C: x = 2 def test(self): print x -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/