Mailman 3 LOAD_NAME & classes - Python-Dev

LOAD_NAME & classes

Jeremy Hylton

22 Apr 2002 22 Apr '02

5:30 p.m.

There's a bug report on SF that notes there is a difference between: def f(x): class Foo: x = x and x = 1 class Foo: x = x The latter works because the class namespace uses LOAD_NAME and finds a global binding for x. The fact that x is also a local is invisible to LOAD_NAME. The former fails because x is none of locals, globals, or builtins, although it is bound in an enclosing scope. LOAD_NAME knows nothing about free variables, however, so it just blows up. Do we want to do anything about this apparent inconsistency? LOAD_NAME is obviously necessary to make stuff like exec work at all, and after a recent bug fix, it evens works as documented for nested scopes. But the docs for a class namespace (there aren't any, right?) don't suggest there is anything special going on. I imagine it would be possible to stop using LOAD_NAME for classes, but I'm not sure if that's a good thing. It could presumably break code that relies on LOAD_NAME's old fashioned search. It also seems like a non trivial amount of work because we'd need a new LOAD/STORE combo that only searched a locals dict. (Maybe that's not so hard.) I think module namespaces also use LOAD_NAME, but it's not clear why. Isn't a module's locals the same as its globals? If so, LOAD_GLOBAL could be used for all names. Any opinion on whether this is worth fixing? And is it a bug fix or a new feature? Jeremy

Show replies by thread

Guido van Rossum

22 Apr 22 Apr

6:15 p.m.

...

There's a bug report on SF that notes there is a difference between:

def f(x): class Foo: x = x

and

x = 1 class Foo: x = x

The latter works because the class namespace uses LOAD_NAME and finds a global binding for x. The fact that x is also a local is invisible to LOAD_NAME.

The former fails because x is none of locals, globals, or builtins, although it is bound in an enclosing scope. LOAD_NAME knows nothing about free variables, however, so it just blows up.

Do we want to do anything about this apparent inconsistency?

LOAD_NAME is obviously necessary to make stuff like exec work at all, and after a recent bug fix, it evens works as documented for nested scopes. But the docs for a class namespace (there aren't any, right?) don't suggest there is anything special going on.

I imagine it would be possible to stop using LOAD_NAME for classes, but I'm not sure if that's a good thing. It could presumably break code that relies on LOAD_NAME's old fashioned search. It also seems like a non trivial amount of work because we'd need a new LOAD/STORE combo that only searched a locals dict. (Maybe that's not so hard.)

I think module namespaces also use LOAD_NAME, but it's not clear why. Isn't a module's locals the same as its globals? If so, LOAD_GLOBAL could be used for all names.

Any opinion on whether this is worth fixing? And is it a bug fix or a new feature?

I just tried this in 2.1 (without nested scopes enabled) and there the first example fails too. While it's slightly confusing, it's consistent with the rule that class bodies don't play the nested scopes game, and I think it shouldn't be fixed. Otherwise you'd have the confusing issue that a function inside a class can't see the class scope, but a class inside a function can see the function scope. Better if neither can see the other. --Guido van Rossum (home page: http://www.python.org/~guido/)

jeremy＠zope.com

6:21 p.m.

...

...
...
...
...
"GvR" == Guido van Rossum writes:

...

def f(x): class Foo: x = x

and

x = 1 class Foo: x = x

GvR> I just tried this in 2.1 (without nested scopes enabled) and GvR> there the first example fails too. I think you misunderstood my original explanation. With the current implementations, the first example fails and the second example works. I think the bug is that the second example works. GvR> While it's slightly confusing, it's consistent with the rule GvR> that class bodies don't play the nested scopes game, and I GvR> think it shouldn't be fixed. Otherwise you'd have the GvR> confusing issue that a function inside a class can't see the GvR> class scope, but a class inside a function can see the function GvR> scope. Better if neither can see the other. None of the documentation suggests that class and functions don't see each other. Rather, it says the free variables are resolved in the nearest enclosing function scope. The current implementation supports resolution of free variables in class scope, e.g. def f(x): class Foo: print x

...

...
...
f(3) 3

I think the problem is with x = x, which ought to be an error if x is a local and x is unbound. The code will succeed, nested or otherwise, if x happens to be a module global. The confusion is that the code will not succeed if x is defined in an enclosing, non-top-level scope. Jeremy

jeremy＠zope.com

6:37 p.m.

...

...
...
...
...
"GvR" == Guido van Rossum writes:

...

...
I think you misunderstood my original explanation. With the current implementations, the first example fails and the second example works. I think the bug is that the second example works.

GvR> But it has always worked, and I definitely don't want to break GvR> this now. I don't want to break this now either (although I don't think either of my messages betrayed that intent). I think it's a wart we live with, but I'd love to fix it when we get to Py3K. (Will we ever?) GvR> I don't understand why you want to break it? That would GvR> definitely break working code! And there's the rub.

...

...
I think the problem is with x = x, which ought to be an error if x is a local and x is unbound. The code will succeed, nested or otherwise, if x happens to be a module global. The confusion is that the code will not succeed if x is defined in an enclosing, non-top-level scope.

GvR> Nested scopes in Python will never be completely intuitive, GvR> because they break the illusion that "there's only runtime" (as GvR> Samuele summarized it a while ago). Wasn't that about macros? GvR> That said, I can't decide whether it's better to make the first GvR> example work, or reject the bug report. I'm tempted to say GvR> "the ice is thin here, don't play with fire" or something GvR> cryptic like that, and leave it. In the bug tracker (532860), I suggested "The result of a reference to an unbound local name in a class block is undefined." With a note that says this is for backwards compatibility and another that explains how CPython is currently implemented. I have a separate bug report about ref man documentation for all this. The ref man documents that world the way I'd like it to be, nicely glossing over all the messy details like LOAD_NAME. I think we need an extra section (4.1.1) that explains some of the messy details. Jeremy

Guido van Rossum

6:39 p.m.

...

...
...
...
...
...
"GvR" == Guido van Rossum writes:

...
def f(x): class Foo: x = x

and

x = 1 class Foo: x = x

GvR> I just tried this in 2.1 (without nested scopes enabled) and GvR> there the first example fails too.

I think you misunderstood my original explanation. With the current implementations, the first example fails and the second example works. I think the bug is that the second example works.

But it has always worked, and I definitely don't want to break this now. I don't understand why you want to break it? That would definitely break working code!

...

GvR> While it's slightly confusing, it's consistent with the rule GvR> that class bodies don't play the nested scopes game, and I GvR> think it shouldn't be fixed. Otherwise you'd have the GvR> confusing issue that a function inside a class can't see the GvR> class scope, but a class inside a function can see the function GvR> scope. Better if neither can see the other.

None of the documentation suggests that class and functions don't see each other. Rather, it says the free variables are resolved in the nearest enclosing function scope.

Oops, I misremembered how this works.

...

The current implementation supports resolution of free variables in class scope, e.g.

def f(x): class Foo: print x

...
...
...
f(3) 3

I think the problem is with x = x, which ought to be an error if x is a local and x is unbound. The code will succeed, nested or otherwise, if x happens to be a module global. The confusion is that the code will not succeed if x is defined in an enclosing, non-top-level scope.

Nested scopes in Python will never be completely intuitive, because they break the illusion that "there's only runtime" (as Samuele summarized it a while ago). That said, I can't decide whether it's better to make the first example work, or reject the bug report. I'm tempted to say "the ice is thin here, don't play with fire" or something cryptic like that, and leave it. --Guido van Rossum (home page: http://www.python.org/~guido/)

Aahz

6:44 p.m.

On Mon, Apr 22, 2002, Guido van Rossum wrote:

...

That said, I can't decide whether it's better to make the first example work, or reject the bug report. I'm tempted to say "the ice is thin here, don't play with fire" or something cryptic like that, and leave it.

"On thin ice, play with fire you should not." -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ What if there were no rhetorical questions?

Jeremy Hylton

6:59 p.m.

...

...
...
...
...
"A" == aahz writes:

A> "On thin ice, play with fire you should not." A> What if there were no rhetorical questions? I think I should read this in a Frank Oz voice, right? Somehow, though, it makes me think: "but sometimes my arms bend back." can-guido-pronounce-his-own-name-ly y'rs, Jeremy

Guido van Rossum

7:04 p.m.

...

...
...
...
...
...
"GvR" == Guido van Rossum writes:

...
...
I think you misunderstood my original explanation. With the current implementations, the first example fails and the second example works. I think the bug is that the second example works.

GvR> But it has always worked, and I definitely don't want to break GvR> this now.

I don't want to break this now either (although I don't think either of my messages betrayed that intent).

The fact that you called it a bug (rather than a problem) made me think this.

...

I think it's a wart we live with, but I'd love to fix it when we get to Py3K. (Will we ever?)

There probably won't ever be an opportunity to start over and be truly incompatible. I don't think I'm up for a herculean effort like the Perl 6 folks are making.

...

GvR> I don't understand why you want to break it? That would GvR> definitely break working code!

And there's the rub.

...
...
I think the problem is with x = x, which ought to be an error if x is a local and x is unbound. The code will succeed, nested or otherwise, if x happens to be a module global. The confusion is that the code will not succeed if x is defined in an enclosing, non-top-level scope.

GvR> Nested scopes in Python will never be completely intuitive, GvR> because they break the illusion that "there's only runtime" (as GvR> Samuele summarized it a while ago).

Wasn't that about macros?

That was the context, but it nicely summarizes a lot about Python, and about scripting languages in general. It may even be an explanation of the success of scripting languages: typical programmers probably have a much better mental model of runtime than of compile time. Variable scope already violates the "only runtime" rule, and this has caused endless complaints and misunderstandings. Before nested scopes, I've toyed with the idea of making LOAD_FAST fall through (conceptually) to LOAD_GLOBAL when the local slot is NULL, just to shut up the complaints. Then this example (whose failure always surprises newcomers) would work: x = 10 def f(): print x x = 12 f() But now that we have nested scopes, I don't think this would be feasible -- the fallback would have to dynamically inspect all surrounding scopes.

...

GvR> That said, I can't decide whether it's better to make the first GvR> example work, or reject the bug report. I'm tempted to say GvR> "the ice is thin here, don't play with fire" or something GvR> cryptic like that, and leave it.

In the bug tracker (532860), I suggested "The result of a reference to an unbound local name in a class block is undefined." With a note that says this is for backwards compatibility and another that explains how CPython is currently implemented.

At least this leaves the door open to fixing it my way. :-)

...

I have a separate bug report about ref man documentation for all this. The ref man documents that world the way I'd like it to be, nicely glossing over all the messy details like LOAD_NAME. I think we need an extra section (4.1.1) that explains some of the messy details.

You can never go wrong by documenting something. (Famous last words, I expect. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

Tim Peters

7:48 p.m.

[Guido]

...

... Variable scope already violates the "only runtime" rule, and this has caused endless complaints and misunderstandings.

Any scope rules do: you can't win here. You can only choose which specific rules cause endless complaints and misunderstandings. Pure dynamic scoping is the easiest for newcomers to understand, and it's a Bad Idea despite that.

...

Before nested scopes, I've toyed with the idea of making LOAD_FAST fall through (conceptually) to LOAD_GLOBAL when the local slot is NULL, just to shut up the complaints. Then this example (whose failure always surprises newcomers)

It surprises some newcomers, plus Andrew Kuchling <wink>.

...

would work:

x = 10 def f(): print x x = 12 f()

But this still wouldn't: x = 10 def f(): print x def g(): x = 12 f() g() You can't explain why that doesn't print 12 without insisting that different textual regions have different scopes. Newbie complaints and misunderstandings follow. Under an "only runtime" conception, the only obvious behavior is that x is bound to 12 at the time f() is called, so of course it should print 12. It takes a certain sophistication to understand that the name "x" means different things in different parts of the file. Once that's understood, the behavior of your example is no longer a mystery; but before it's understood, your example is only one of countless confusions. Indeed, the single most frequent newbie confusion I've heard over the last year is variations on "OK, I did 'import math' like you said, but 'sin(x)' *still* gives a NameError!".

...

But now that we have nested scopes, I don't think this would be feasible -- the fallback would have to dynamically inspect all surrounding scopes.

As above, dynamic scoping is what newbies expect -- but it would be a great disservice to give it to them. pons-asinorum-ly y'rs - tim

Guido van Rossum

8:11 p.m.

[Guido]

...

...
would work:

x = 10 def f(): print x x = 12 f()

[Tim]

...

But this still wouldn't:

x = 10 def f(): print x

def g(): x = 12 f()

g()

You can't explain why that doesn't print 12 without insisting that different textual regions have different scopes. Newbie complaints and misunderstandings follow. Under an "only runtime" conception, the only obvious behavior is that x is bound to 12 at the time f() is called, so of course it should print 12. It takes a certain sophistication to understand that the name "x" means different things in different parts of the file. Once that's understood, the behavior of your example is no longer a mystery; but before it's understood, your example is only one of countless confusions.

I think this is a red herring, and not an argument against what I proposed. The "only runtime" rules doesn't require dynamic scopes (I agree that dynamic scopes would be bad). Dynamic scopes, and your example, mix up the call context with the definition context. My example takes the definition context, and applies the "is x defined in this scope?" test at runtime instead of at compile time. Very different! --Guido van Rossum (home page: http://www.python.org/~guido/)

Tim Peters

9:23 p.m.

[Guido]

...

I think this is a red herring, and not an argument against what I proposed.

Sure it does <wink>. It's one example of the larger point that *any* scope rules confuse newbies. Lexical scoping doesn't come naturally except to born Schemers, and indeed didn't come naturally to language designers either (e.g., early LISPs all had dynamic scoping). So it goes.

...

The "only runtime" rules doesn't require dynamic scopes (I agree that dynamic scopes would be bad). Dynamic scopes, and your example, mix up the call context with the definition context. My example takes the definition context, and applies the "is x defined in this scope?" test at runtime instead of at compile time. Very different!

Rewrite it trivially: def f(): print x x = 12 x = 10 # moved the line down f() I don't know what you mean by "definition context" (and neither will a newbie), but any guess I'm likely to come up with wouldn't include the idea that "x = 10" is now in f's definition context. You would have to spell out that there are three namespaces at work here, and that what "x" means in f depends on the dynamic state of two of them, and simply can't be known in general without running the program. If that's not dynamic scoping, it's too close for me to believe a distinction is worth making. BTW, I consider Python's treatment of global vs builtin namespaces dynamic scoping too, and it's nothing but trouble that globals can mask and unmask builtins dynamically. I'd rather make globals and builtins act more like locals now than make locals act more like they're dynamically scoped. BTW2, I see plenty of UnboundLocalErrors in my own code, and some of those have occurred when the same name is also in use as a global. It's always been a logic error due to forgetting to initialize a local, and usually due to "moving code up" in an editor. It sure wouldn't be doing me any favor to let such code silently pick up whatever crap happened to be bound to the same-named global; UnboundLocalError is a fine bug-catcher.

Guido van Rossum

10 p.m.

...

...
The "only runtime" rules doesn't require dynamic scopes (I agree that dynamic scopes would be bad). Dynamic scopes, and your example, mix up the call context with the definition context. My example takes the definition context, and applies the "is x defined in this scope?" test at runtime instead of at compile time. Very different!

Rewrite it trivially:

def f(): print x x = 12

x = 10 # moved the line down f()

I don't know what you mean by "definition context" (and neither will a newbie), but any guess I'm likely to come up with wouldn't include the idea that "x = 10" is now in f's definition context. You would have to spell out that there are three namespaces at work here, and that what "x" means in f depends on the dynamic state of two of them, and simply can't be known in general without running the program. If that's not dynamic scoping, it's too close for me to believe a distinction is worth making.

I seem to have trouble explaining what I meant. Long ago, before I introduced LOAD_FAST and friends, Python had something that for want of a better term I'll call "lexical scoping with dynamic lookup". It did a dynamic lookup in a (max 3 deep: local / global / builtin) stack of namespaces, but the set of namespaces was determined by the compiler. This does not have the problems of dynamic scoping (the caller's stack frame can't cause trouble). But it also doesn't have the problem of the current strict static scoping. I like the older model better than the current model (apart from nested scopes) and I believe that the "only runtime" rule explains why the old model is more attractive: it doesn't require you to think of the compiler scanning all the code of your function looking for definitions of names. You can think of the interpreter pretty much executing code as it sees it. You have to have a model for name lookup that requires a chaining of namespaces based on where a function is defined, but that's all still purely runtime (it involves executing the def statement). This requires some sophistication for a newbie to understand, but it's been explained successfully for years, and the explanation would be easier without UnboundLocalError. Note that it explains your example above completely: the namespace where f is defined contains a definition of x when f is called, and thus the search stops there.

...

BTW, I consider Python's treatment of global vs builtin namespaces dynamic scoping too, and it's nothing but trouble that globals can mask and unmask builtins dynamically. I'd rather make globals and builtins act more like locals now than make locals act more like they're dynamically scoped.

Um, that's not what I'd call dynamic scoping. It's dynamic lookup. It's trouble for a compiler that wants to optimize builtins, but the semantic model is nice and simple and easy to explain with the "only runtime" rule.

...

BTW2, I see plenty of UnboundLocalErrors in my own code, and some of those have occurred when the same name is also in use as a global. It's always been a logic error due to forgetting to initialize a local, and usually due to "moving code up" in an editor. It sure wouldn't be doing me any favor to let such code silently pick up whatever crap happened to be bound to the same-named global; UnboundLocalError is a fine bug-catcher.

You definitely have a point there -- like with most irritating errors, the coin has two sides. I don't know which side would waste more time. (When UnboundLocalError was still spelled as NameError, I'd bet on the latter.) --Guido van Rossum (home page: http://www.python.org/~guido/)

Tim Peters

10:48 p.m.

[Guido]

...

I seem to have trouble explaining what I meant.

I know, and I confess I'm giving you a hard time. There's a point to that too: uniqueness also imposes costs on newbies and/or newcomers. Across the world of programming languages now, dynamic scoping and lexical scoping are "almost entirely *it*". For example, the Perl spelling of the running example here does work the way you intend, but the explanation in Perl is full-blown dynamic scoping: sub g { print "$x\n"; # prints 12 -- "full-blown dynamic scoping" } sub f { print "$x\n"; # prints 10 local($x) = 12; &g(); } $x = 10; &f(); print "$x\n"; # prints 10 Once you make f print 10, you're on that path as far as anyone coming from any other language can tell at first glance (or even second and third). If you go on to make g print 10 too, it's inexplicable via reference to how any other language works. If there were a huge payback for "being different" here, cool, but the only real payback I see is letting newbies avoid learning how lexical scoping works, and only for a little while.

...

Long ago, before I introduced LOAD_FAST and friends, Python had something that for want of a better term I'll call "lexical scoping with dynamic lookup".

I'm old enough to remember this <wink>.

...

It did a dynamic lookup in a (max 3 deep: local / global / builtin) stack of namespaces, but the set of namespaces was determined by the compiler. This does not have the problems of dynamic scoping (the caller's stack frame can't cause trouble). But it also doesn't have the problem of the current strict static scoping.

Nor its advantages, including better error detection, and ease of transferring hard-won knowledge among other lexically scoped languages.

...

I like the older model better than the current model (apart from nested scopes) and I believe that the "only runtime" rule explains why the old model is more attractive: it doesn't require you to think of the compiler scanning all the code of your function looking for definitions of names. You can think of the interpreter pretty much executing code as it sees it. You have to have a model for name lookup that requires a chaining of namespaces based on where a function is defined, but that's all still purely runtime (it involves executing the def statement).

This requires some sophistication for a newbie to understand, but it's been explained successfully for years, and the explanation would be easier without UnboundLocalError.

Note that it explains your example above completely: the namespace where f is defined contains a definition of x when f is called, and thus the search stops there.

Does it scale? x = 0 def f(i): if i & 4: x = 10 def g(i): if i & 2: x = 20 def h(i): if i & 1: x = 30 print x h(i) g(i) f(3) I can look at that today and predict with confidence that h() will either print 30 (if and only if i is odd), or raise an exception. This is from purely local analysis of h's body -- it doesn't matter that it's nested, and it's irrelvant what the enclosing functions look like or do. That's a great aid to writing correct code. If the value of x h sees *may* come from h, or from g, or from f, or from the module scope instead, depending on i's specific value at the time f is called, there's a lot more to think about. I could keep local+global straight in pre-1.0 Python, although I never got used to the inability to write nested functions that could refer to each other (perhaps you've forgotten how many times you had to explain that one, and how difficult it was to get across?). Now that Python has full-blown nested scopes, the namespace interactions are potentially much more convoluted, and the "purely local analysis" shortcut made possible by everyone else's <wink> notion of lexical scoping becomes correspondingly more valuable.

...

... Um, that's not what I'd call dynamic scoping. It's dynamic lookup.

I know -- the problem is that you're the only one in the world making this distinction, and that makes it hard to maintain over time. If it had some killer advantage ... but it doesn't seem to. When Python switched to "strict local" names before 1.0, I don't recall anyone complaining -- if there was a real advantage to dynamic lookup at the local scope, it appeared to have escaped Python's users <wink>. I'll grant that it did make exec and "import *" more predictable in corner cases.

...

It's trouble for a compiler that wants to optimize builtins, but the semantic model is nice and simple and easy to explain with the "only runtime" rule.

Dynamic scoping is also easy to explain, but it doesn't scale. I'm afraid dynamic lookup doesn't scale either. You should have stuck with Python's original two-level namespace, you know <0.9 wink>. the-builtins-didn't-count-ly y'rs - tim

Greg Ewing

23 Apr 23 Apr

12:57 a.m.

Tim Peters :

...

Lexical scoping doesn't come naturally except to born Schemers, and indeed didn't come naturally to language designers either (e.g., early LISPs all had dynamic scoping).

I doubt whether the early designers of Lisp explicitly designed it to have dynamic scoping. Coming from lambda calculus, they were probably *thinking* of lexical scoping; it's just that the implementation they chose had some unfortunate side effects. So, I conjecture that the *idea* of lexical scoping comes naturally enough, but a correct implementation of it doesn't. :-) By the way, even in dynamically scoped Lisp, there's no equivalent of an UnboundLocalError -- to get a local variable at all, you have to bind some initial value to it. So, using Guido's terminology, early Lisp had dynamic scoping, but not dynamic lookup. Confused enough yet?-) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Tim Peters

3:21 a.m.

[Greg Ewing]

...

I doubt whether the early designers of Lisp explicitly designed it to have dynamic scoping. Coming from lambda calculus, they were probably *thinking* of lexical scoping; it's just that the implementation they chose had some unfortunate side effects.

I think McCarthy may be a good source <wink>: http://www-formal.stanford.edu/jmc/history/lisp/lisp.html The distinction between dynamic and lexical scoping is clear-cut only in hindsight. As McCarthy relates, Algol 60 later bumped into much the same scoping surprises they did. The treatment of free variables in formal logic is also much clearer in hindsight, and was the source of many difficulties before a proper definition of substitution was crafted. Cute: Curry and Schoenfinkel (independently) invented combinatory logic to get rid of free variables (and the difficulties that come with them) entirely, and decades later combinatory logic overthrew the lambda calculus as the basis for high-performance implementations of pure functional languages. Applying a contemporary understanding of these things retroactively just doesn't fly; whatever clarity we have now was a result of their struggles.

...

... By the way, even in dynamically scoped Lisp, there's no equivalent of an UnboundLocalError -- to get a local variable at all, you have to bind some initial value to it.

There were certainly NameErrors. All you needed to do was evaluate any old function that referenced a free variable, at a time when no binding for that variable was on the stack. You can't do this in Scheme, but it was dead easy in LISP; it's harder in Common Lisp, but dynamic scoping can still be gotten at there.

...

So, using Guido's terminology, early Lisp had dynamic scoping, but not dynamic lookup. Confused enough yet?-)

Not as confused as McCarthy turned out to have been <wink>.

Greg Ewing

5:26 a.m.

Tim Peters :

...

I think McCarthy may be a good source <wink>:

http://www-formal.stanford.edu/jmc/history/lisp/lisp.html

Tim Peters

6:15 a.m.

[Greg Ewing]

...

...
http://www-formal.stanford.edu/jmc/history/lisp/lisp.html

...

From there:

The difficulty was that when an inner recursion occurred, the value of car[x] wanted was the outer value, but the inner value was actually used. In modern terminology, lexical scoping was wanted, and dynamic scoping was obtained. ... I must confess that I regarded this difficulty as just a bug and expressed confidence that Steve Russell would soon fix it.

...

So it seems I was more or less right -- the semantics they wanted was lexical scoping, effectively, even if they didn't explicitly think about it.

They didn't even have names for it, Greg -- note the "in modern terminology" there. This is McCarthy reflecting on things that happened more than 20 years before he wrote the paper. It was so muddy at the time that this paper was reduced to Unfortunately, time did not permit writing an appendix giving the history of the problem, and the interested reader is referred to (Moses 1970) as a place to start. (David Park tells me that Patrick Fischer also had a hand in developing the FUNARG device). Part of the problem is that the Lisp interpreter was written in Lisp, and contrary to hopes it didn't fully specify the semantics: it turned out that what the Lisp implementation just happened to do for a thing sometimes reflected directly in how the Lisp interpreter emulated that thing, and so studying the interpreter code sheds no light on intent in such cases -- it's a kind of "self-fulfilling algorithm" then. "Metacircular interpreters" fell out of favor for specifying language semantics as a result. Modern attempts to rehabilitate the idea are still stumbling over how to nail the intended scoping semantics; see, e.g. (skip to Conclusions): http://linux.rice.edu/~rahul/hbaker/MetaCircular.html

Guido van Rossum

12:35 p.m.

[Tim]

...

There's a point to that too: uniqueness also imposes costs on newbies and/or newcomers. Across the world of programming languages now, dynamic scoping and lexical scoping are "almost entirely *it*". [...] Nor its advantages, including better error detection, and ease of transferring hard-won knowledge among other lexically scoped languages.

But Python *is* unique in that it doesn't require declarations. (I've got to admit that the Perl example saddened me. But then in Perl, local variables are a recent invention. :-) We've found before that this can go against what's common knowledge for other language (e.g. integer division). [...]

...

Does it scale?

x = 0

def f(i): if i & 4: x = 10 def g(i): if i & 2: x = 20 def h(i): if i & 1: x = 30 print x h(i) g(i)

f(3)

I can look at that today and predict with confidence that h() will either print 30 (if and only if i is odd), or raise an exception. This is from purely local analysis of h's body -- it doesn't matter that it's nested, and it's irrelvant what the enclosing functions look like or do. That's a great aid to writing correct code. If the value of x h sees *may* come from h, or from g, or from f, or from the module scope instead, depending on i's specific value at the time f is called, there's a lot more to think about.

Yup. But it also requires intricate knowledge of Python's rules, which are different than any other language's rules. You simply can't have a variable declaration inside an if statement in other languages that extends to the entire function body -- either the scope would be limited to the block it's in, or the syntax wouldn't allow it. Python's original semantic model on the other hand, and the model that's still used for globals at the global level, gives a clear explanation: a namespace is implemented as a dictionary, and name lookup searches a pre-set sequence of namespaces until it finds a hit. The lexical scoping rule determines how namespaces are combined. Doing the lookup at runtime is easier to understand than doing it at compile time -- even if the compile version might catch more bugs. But I'm repeating myself; I already said that in my previous message.

...

I could keep local+global straight in pre-1.0 Python, although I never got used to the inability to write nested functions that could refer to each other (perhaps you've forgotten how many times you had to explain that one, and how difficult it was to get across?).

No; apart from you, most people were happy with the rule "nested functions don't work".

...

Now that Python has full-blown nested scopes, the namespace interactions are potentially much more convoluted, and the "purely local analysis" shortcut made possible by everyone else's <wink> notion of lexical scoping becomes correspondingly more valuable.

I don't know. Full-blown nested scopes make namespace interactions more convoluted no matter *what* set of rules we pick. An alternative implementation model (with associated subtly different semantics semantics) would have been to create an explicit list of the dicts involved in the name resolution for a particular function invocation; we rejected that model because we wanted this to be (nearly) as fast as locals, so we moved more of the analysis to compile time. But by doing so, we introduced more of a dependency on the programmer's ability to understand what happens at compile time, and that breaks the "only runtime exists" illusion. In PEP 267 Jeremy is exploring how to optimize access to globals *without* changing the rules. The change to LOAD_FAST that I considered before would have optimized access to locals without changing the rules, and I still regret that I didn't think of that when I created LOAD_FAST (even though you disagree): the "only runtime" rule is helpful for a large class of programmers, not only newbies, and I'm not sure that adding more and more cruft from truly compiled languages to Python's *semantics* is a good idea. (Adding compiler technology that doesn't change the rules is fine, of course, if it helps optimizations or better diagnostics.)

...

...
... Um, that's not what I'd call dynamic scoping. It's dynamic lookup.

I know -- the problem is that you're the only one in the world making this distinction, and that makes it hard to maintain over time.

You can say that, but that doesn't make it so, and it doesn't convince me. The three-scope was gospel in the Python world, and many people actively disliked adding nested scopes (some still do).

...

If it had some killer advantage ... but it doesn't seem to. When Python switched to "strict local" names before 1.0, I don't recall anyone complaining -- if there was a real advantage to dynamic lookup at the local scope, it appeared to have escaped Python's users <wink>. I'll grant that it did make exec and "import *" more predictable in corner cases.

Well, we gave them a big reason not to complain: this was the singlemost biggest speedup in Python's history. But the rules were definitely harder to explain, because for the first time we had to explain a second compiler pass.

...

...
It's trouble for a compiler that wants to optimize builtins, but the semantic model is nice and simple and easy to explain with the "only runtime" rule.

Dynamic scoping is also easy to explain, but it doesn't scale. I'm afraid dynamic lookup doesn't scale either. You should have stuck with Python's original two-level namespace, you know <0.9 wink>.

We need more than a single example to decide which rules bites worse for large programs. Deep nesting is not common; long functions are. And there the common annoyance is that a change in line 150 can break the code in line 2 of the function. --Guido van Rossum (home page: http://www.python.org/~guido/)

Patrick K. O'Brien

1:51 p.m.

[Guido van Rossum]

...

We need more than a single example to decide which rules bites worse for large programs. Deep nesting is not common; long functions are. And there the common annoyance is that a change in line 150 can break the code in line 2 of the function.

I'm not exactly sure what you mean by this. Can you share an example? (Not necessarily 150+ lines long, of course.) Thanks. --- Patrick K. O'Brien Orbtech

Guido van Rossum

2:16 p.m.

...

[Guido van Rossum]

...
We need more than a single example to decide which rules bites worse for large programs. Deep nesting is not common; long functions are. And there the common annoyance is that a change in line 150 can break the code in line 2 of the function.

I'm not exactly sure what you mean by this. Can you share an example? (Not necessarily 150+ lines long, of course.) Thanks.

It's a classic. Before we had UnboundLocalError (i.e. in 1.5.2 and before) this was a common problem on c.l.py: x = "a global" def f(): print x # user thinks this should print the global # 2000 lines of unrelated code for x in "some sequence": # doesn't realize this overrides x do_something_with(x) Calling f() would raise NameError: x, which caused lots of confusion. We added UnboundLocalError th make it clearer what's going on (so at least the experienced c.l.py users would know right away where the problem was :-), but still requires you to know about something obscure that's going on at compile time (the compiler scanning your entire function for variable definitions). --Guido van Rossum (home page: http://www.python.org/~guido/)

Patrick K. O'Brien

3:22 p.m.

[Guido van Rossum]

...

...
...
And there the common annoyance is that a change in line 150 can break the code in line 2 of the function.

I'm not exactly sure what you mean by this. Can you share an example? (Not necessarily 150+ lines long, of course.) Thanks.

It's a classic. Before we had UnboundLocalError (i.e. in 1.5.2 and before) this was a common problem on c.l.py:

x = "a global"

def f(): print x # user thinks this should print the global # 2000 lines of unrelated code for x in "some sequence": # doesn't realize this overrides x do_something_with(x)

Calling f() would raise NameError: x, which caused lots of confusion.

We added UnboundLocalError th make it clearer what's going on (so at least the experienced c.l.py users would know right away where the problem was :-), but still requires you to know about something obscure that's going on at compile time (the compiler scanning your entire function for variable definitions).

Okay. I think I'm following you, but I want to be certain about the statement that "a change in line 150 can break the code in line 2 of the function." Using your example, the function f() works, but only because of a "fortunate" side effect of sorts. So if the code was later changed to "for y in ..." then f() no longer works. But the example is fundamentally flawed to begin with. Proper code shouldn't have to worry that "a change in line 150 can break the code in line 2 of the function." Right? Or am I still missing something? I've never felt that I needed to know about something obscure going on at compile time in order to write decent Python code. Maybe I'm just being paranoid, but this whole discussion just struck me as odd because I can't recall ever having any problem like this. For the most part Python does exactly what I think it should do. And when it doesn't, I'm usually wrong. --- Patrick K. O'Brien Orbtech

Steve Holden

3:30 p.m.

...

...
[Guido van Rossum]

...
We need more than a single example to decide which rules bites worse for large programs. Deep nesting is not common; long functions are. And there the common annoyance is that a change in line 150 can break the code in line 2 of the function.

[Patrick]

...
I'm not exactly sure what you mean by this. Can you share an example? (Not necessarily 150+ lines long, of course.) Thanks.

[Guido]

...

It's a classic. Before we had UnboundLocalError (i.e. in 1.5.2 and before) this was a common problem on c.l.py:

x = "a global"

def f(): print x # user thinks this should print the global # 2000 lines of unrelated code for x in "some sequence": # doesn't realize this overrides x do_something_with(x)

Calling f() would raise NameError: x, which caused lots of confusion.

We added UnboundLocalError th make it clearer what's going on (so at least the experienced c.l.py users would know right away where the problem was :-), but still requires you to know about something obscure that's going on at compile time (the compiler scanning your entire function for variable definitions).

So, the problem is the implicit local declaration that assumed when the compiler detects a binding lower down the function's code than the first reference to it, coupled with their function-wide scope. It is compounded by the fact that although the analysis is performed at compile time the error is only reported at run time. Might it make more sense to issue a warning at compile time to the effect that a variable is being used before it's assigned? How completely are (re)bindings detected by the static analysis? Can you necessarily guarantee that a LOAD_FAST is always looking at a local name? [You'll understand I'm not familiar with the details of code generation]. Seems to me the real problem here is explaining (by the interpreter's behavior, rather than in the documentation ;-) the scope of locals and how a name is determined to be local. It might help beginners if there was some really easy way to get a fuller explanation of an error message. Not sure how best that could be retrofitted, but it's better than code breakage. Unfortunately the scoping rules are now "in the wild", so it's not possible to change things too radically without mucho pain for those who have already relied on them. regards Steve -- home: http://www.holdenweb.com/ Python Web Programming: http://pydish.holdenweb.com/pwp/

Guido van Rossum

3:33 p.m.

...

Okay. I think I'm following you, but I want to be certain about the statement that "a change in line 150 can break the code in line 2 of the function." Using your example, the function f() works, but only because of a "fortunate" side effect of sorts.

Why is the fact that this works: x = 12 def f(): print x a "fortunate" side effect? That's how the language works!

...

So if the code was later changed to "for y in ..." then f() no longer works. But the example is fundamentally flawed to begin with. Proper code shouldn't have to worry that "a change in line 150 can break the code in line 2 of the function." Right? Or am I still missing something?

I think you've got it backwards. My complaint is that if f() above eventually grew 150 lines of unrelated code ending with an unrelated assignment to a local variable x, the breakage would show up at an unexpected point. Except for this one, it's hard to make a change at the *tail* of a function that breaks something at the beginning!

...

I've never felt that I needed to know about something obscure going on at compile time in order to write decent Python code. Maybe I'm just being paranoid, but this whole discussion just struck me as odd because I can't recall ever having any problem like this. For the most part Python does exactly what I think it should do. And when it doesn't, I'm usually wrong.

This particular form of breakage was a common error reported on c.l.py and to help at python.org until we added UnboundLocalError to make the diagnostic cleaner. Maybe that's all that's needed; getting a NameError when you see this: x = 12 def f(): print x # <--- NameError raised here! ...150 line of code you didn't think could cause the problem... was very disturbing, causing people to look for places where x was deleted from the global namespace later in the program. --Guido van Rossum (home page: http://www.python.org/~guido/)

Alex Martelli

3:47 p.m.

On Tuesday 23 April 2002 05:30 pm, Steve Holden wrote: ...

...

Might it make more sense to issue a warning at compile time to the effect that a variable is being used before it's assigned? How completely are

Hard to make sure, so, it would always be "MIGHT be used before assignment". E.g., def f(): for i in range(6): if i>0: print x, x = str(i) this is OK... no UnboundNameError. Now, change the if's guard to if SolveTuringHaltingProblem(i): print x, to see why it's hard to make sure about this at compile time. A warning might still be OK of course, even though it may need to be worded in wishy-washy "might" terms -- this IS peculiar usage. Alex

Guido van Rossum

3:57 p.m.

[Steve Holden]

...

So, the problem is the implicit local declaration that assumed when the compiler detects a binding lower down the function's code than the first reference to it, coupled with their function-wide scope. It is compounded by the fact that although the analysis is performed at compile time the error is only reported at run time.

Well put.

...

Might it make more sense to issue a warning at compile time to the effect that a variable is being used before it's assigned?

Yes. Historically (when LOAD_FAST was added) we didn't have a framework for compile-time warnings, but now we do.

...

How completely are (re)bindings detected by the static analysis? Can you necessarily guarantee that a LOAD_FAST is always looking at a local name? [You'll understand I'm not familiar with the details of code generation].

Yes, LOAD_FAST is only emitted if the compiler has decided that it's a local name, and then there's no exception (though that is exactly what I contemplated changing earlier in this thread).

...

Seems to me the real problem here is explaining (by the interpreter's behavior, rather than in the documentation ;-) the scope of locals and how a name is determined to be local. It might help beginners if there was some really easy way to get a fuller explanation of an error message. Not sure how best that could be retrofitted, but it's better than code breakage.

I think if they look up UnboundLocalError in the docs they find the right clues. Of course it's not necessarily easy to find that bit of documentation (though Google's first hit is very helpful).

...

Unfortunately the scoping rules are now "in the wild", so it's not possible to change things too radically without mucho pain for those who have already relied on them.

Changing LOAD_FAST so that x = 1 def f(): print x x = 2 print x prints 1 followed by 2 instead of raising UnboundLocalError would not break existing code (except code relying on specific exceptions to be raised -- but such code is always exempt from non-breakage guarantees). --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum

4:02 p.m.

...

...
Might it make more sense to issue a warning at compile time to the effect that a variable is being used before it's assigned? How completely are

Hard to make sure, so, it would always be "MIGHT be used before assignment". E.g.,

def f(): for i in range(6): if i>0: print x, x = str(i)

this is OK... no UnboundNameError. Now, change the if's guard to if SolveTuringHaltingProblem(i): print x, to see why it's hard to make sure about this at compile time.

A warning might still be OK of course, even though it may need to be worded in wishy-washy "might" terms -- this IS peculiar usage.

GCC warns about this stuff as soon as there's a path through the code that could leave a variable unassigned, even if the path may not be reachable depending on what other functions do, and while it occasionally caused my to insert a useless initialization, I've usually found that it was hard enough to prove to myself that the initialization was unneeded to make me happy to insert it. --Guido van Rossum (home page: http://www.python.org/~guido/)

Tim Peters

4:04 p.m.

[Guido]

...

... No; apart from you, most people were happy with the rule "nested functions don't work".

Most, yes, but most people are happy with UnboundLocalError too. The business about nested functions not being able to see each other was a FAQ regardless, and Aaron Watters wrote the FAQ Wizard blurb about it. This FAQ entry was distinct from the FAQ entry about nested functions not being able to see the locals in enclosing functions, despite that they were really the same issue, because people who understood the latter just didn't see the connection to the former without additional words. So the answer to my original question: (perhaps you've forgotten how many times you had to explain that one, and how difficult it was to get across?). is either "yes", or "no, but I eventually left such explaining to others, and so have nothing to forget" <wink>.

Jeremy Hylton

4:06 p.m.

...

...
...
...
...
"GvR" == Guido van Rossum writes:

GvR> Changing LOAD_FAST so that GvR> x = 1 GvR> def f(): GvR> print x GvR> x = 2 GvR> print x GvR> prints 1 followed by 2 instead of raising UnboundLocalError GvR> would not break existing code (except code relying on specific GvR> exceptions to be raised -- but such code is always exempt from GvR> non-breakage guarantees). It wouldn't break existing code, but it would encourage the existence of broken code. It's asking for trouble to write a function that uses a local and a global with the same name. I don't think we should bend the rules to accommodate this kind of confusion. Jeremy

Patrick K. O'Brien

4:18 p.m.

[Guido van Rossum]

...

Why is the fact that this works:

x = 12 def f(): print x

a "fortunate" side effect? That's how the language works!

Yes. I got a fresh cup of coffee and smashed my head against a wall and now my mental processes appear to be working again. At least, there's more squeaking and rattling. My previous choice of words now appears to be rather... unfortunate. <wink>

...

I think you've got it backwards. My complaint is that if f() above eventually grew 150 lines of unrelated code ending with an unrelated assignment to a local variable x, the breakage would show up at an unexpected point. Except for this one, it's hard to make a change at the *tail* of a function that breaks something at the beginning!

Yes, I don't know what I was thinking. Maybe I need a donut. Thanks. --- Patrick K. O'Brien Orbtech

Tim Peters

4:43 p.m.

[Guido]

...

... But Python *is* unique in that it doesn't require declarations. (I've got to admit that the Perl example saddened me. But then in Perl, local variables are a recent invention. :-)

Just FYI, "local()" in Perl is very old, and has always used dynamic scoping rules. If you don't wrap local() around a vrbl decl, you get a global vrbl instead, and that's also very old. What's relatively new is the "my()" vrbl wrapper, which asks for a lexically scoped vrbl. What's very new is "our()", which I still haven't figured out: An "our" declares the listed variables to be valid globals within the enclosing block, file, or eval. That is, it has the same scoping rules as a "my" declaration, but does not create a local variable. If more than one value is listed, the list must be placed in parentheses. The our declaration has no semantic effect unless "use strict vars" is in effect, in which case it lets you use the declared global variable without qualifying it with a package name. (But only within the lexical scope of the our declaration. In this it differs from "use vars", which is package scoped.) Anyone want to make a motion that we not ape Perl's scoping rules <wink>?

Aahz

5:04 p.m.

On Tue, Apr 23, 2002, Guido van Rossum wrote:

...

Changing LOAD_FAST so that

x = 1 def f(): print x x = 2 print x

prints 1 followed by 2 instead of raising UnboundLocalError would not break existing code (except code relying on specific exceptions to be raised -- but such code is always exempt from non-breakage guarantees).

-1 I agree with Jeremy that such code is by definition broken because it mixes global and local usage of the same variable within the same scope. That's a nightmare for visual inspection. BTW, the indentation came out bizarrely because you used a TAB char for indentation. Naughty, naughty. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ What if there were no rhetorical questions?

Michael Gilfix

5:05 p.m.

On Tue, Apr 23 @ 12:43, Tim Peters wrote:

...

Just FYI, "local()" in Perl is very old, and has always used dynamic scoping rules. If you don't wrap local() around a vrbl decl, you get a global vrbl instead, and that's also very old. What's relatively new is the "my()" vrbl wrapper, which asks for a lexically scoped vrbl. What's very new is "our()", which I still haven't figured out:

An "our" declares the listed variables to be valid globals within the enclosing block, file, or eval. That is, it has the same scoping rules as a "my" declaration, but does not create a local variable. If more than one value is listed, the list must be placed in parentheses. The our declaration has no semantic effect unless "use strict vars" is in effect, in which case it lets you use the declared global variable without qualifying it with a package name. (But only within the lexical scope of the our declaration. In this it differs from "use vars", which is package scoped.)

Anyone want to make a motion that we not ape Perl's scoping rules <wink>?

The our is akin to declaring something static in C. Except in Perl, it can apply to an enclosure as well since you can build funcs dynamically. Yay. I think python's scoping rules work just fine :) Has there ever been a discussion about some easy or straight-forward way of sharing a global instance across modules? For example, in a gui app, you might want to structure the program such that there's a global instance app (main.app) of the application that other modules might want to query. I was never to happy with importing main and then using main.app... I felt like I wanted to qualify it as a global, like: from main import global app -- Mike -- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html

pinard＠iro.umontreal.ca

5:27 p.m.

[Guido van Rossum]

...

But Python *is* unique in that it doesn't require declarations. (I've got to admit that the Perl example saddened me. But then in Perl, local variables are a recent invention. :-)

Guido, I miss the point of the second sentence. Why were you saddened? Don't you prefer _not_ having to declare locals, as Python does now? -- François Pinard http://www.iro.umontreal.ca/~pinard

Aahz

6:13 p.m.

On Tue, Apr 23, 2002, Michael Gilfix wrote:

...

Has there ever been a discussion about some easy or straight-forward way of sharing a global instance across modules? For example, in a gui app, you might want to structure the program such that there's a global instance app (main.app) of the application that other modules might want to query. I was never to happy with importing main and then using main.app... I felt like I wanted to qualify it as a global, like:

from main import global app

The simple way to do it, IMO, is to import a joint module, called something like cfg. I just did a little hack in a script with cfg = imp.new_module('cfg') but I don't know how well that works across modules. I think you'd need to poke it into sys.modules in order for it to be shared, but I haven't tested it. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ What if there were no rhetorical questions?

Guido van Rossum

6:37 p.m.

...

[Guido van Rossum]

...
But Python *is* unique in that it doesn't require declarations. (I've got to admit that the Perl example saddened me. But then in Perl, local variables are a recent invention. :-)

Guido, I miss the point of the second sentence. Why were you saddened? Don't you prefer _not_ having to declare locals, as Python does now?

François Pinard http://www.iro.umontreal.ca/~pinard

François, I was sad because Perl seemed to use dynamic scoping when you declare a local variable. Tim has since explained that Perl's local is in fact ancient. I'm also sad that Perl doesn't default to locals, but that's not my problem. --Guido van Rossum (home page: http://www.python.org/~guido/)

Michael Gilfix

7:04 p.m.

On Tue, Apr 23 @ 14:13, Aahz wrote:

...

On Tue, Apr 23, 2002, Michael Gilfix wrote:

...
Has there ever been a discussion about some easy or straight-forward way of sharing a global instance across modules? For example, in a gui app, you might want to structure the program such that there's a global instance app (main.app) of the application that other modules might want to query. I was never to happy with importing main and then using main.app... I felt like I wanted to qualify it as a global, like:

from main import global app

The simple way to do it, IMO, is to import a joint module, called something like cfg. I just did a little hack in a script with

cfg = imp.new_module('cfg')

but I don't know how well that works across modules. I think you'd need to poke it into sys.modules in order for it to be shared, but I haven't tested it.

Yeah, that's another good way to do it. Have some sort of configuration object you could query. It would be nice if python offered a facility for sharing pooled data: maybe a global configuration object? That way it's more explicit. It could also remove some redundant code that I tend to put in many applications. import config my_opt = config.get_opt ('mymodule') # Set a new global opt config.set_opt ('attrib') = blah Maybe even provide some hooks for populating the config object. -- Mike -- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html

pinard＠iro.umontreal.ca

7:27 p.m.

[Guido van Rossum]

...

I was sad because Perl seemed to use dynamic scoping when you declare a local variable. Tim has since explained that Perl's local is in fact ancient. I'm also sad that Perl doesn't default to locals, but that's not my problem.

Agreed. I've been using Perl since Perl 1, all the way to Perl 5. When the `local()' declaration appeared, probably around Perl 2 or Perl 3 (I do not remember), I started using it everywhere in an effort of protecting each procedure against each other. It was also a convenient way to give names to formal arguments, through "local(name1, name2, ...) = @_" as the first statement in a `sub'. When "my(LIST)" appeared, one or two major releases later, I progressively converted all my programs to use it, as it is more in the spirit of what I wanted. The overall impression that was left to me is that a lot of declarations are needed, anyway, if you consistently aim clean Perl scripts. So, there is now an "our(...)" declaration in Perl? Hopefully, it will not miss me much! :-) And who remembers the "own" specifier of Algol-60? :-) -- François Pinard http://www.iro.umontreal.ca/~pinard

pinard＠iro.umontreal.ca

7:49 p.m.

[Michael Gilfix]

...

It would be nice if python offered a facility for sharing pooled data: maybe a global configuration object? That way it's more explicit. It could also remove some redundant code that I tend to put in many applications.

...

import config

...

my_opt = config.get_opt ('mymodule') # Set a new global opt config.set_opt ('attrib') = blah

...

Maybe even provide some hooks for populating the config object.

I already use `config' (that precise identifier) a lot in my things, so please do not even _think_ using it for something else! :-) :-) For small projects, such a thing might be overkill. For big projects, there are a few (maybe many) avenues and ways how to achieve this sharing, and I'm glad having the freedom of installing subtle nuances between projects, as needed. That is, I would fear a bit being pulled in an arbitrary direction: it would have to be convincingly damned good. :-) Better, maybe, would be to merely document some of the possible avenues. Most of these require only a tiny investment in lines of code anyway. -- François Pinard http://www.iro.umontreal.ca/~pinard

Michael Gilfix

7:59 p.m.

Upon futher reflection, I would agree with you. It's something I find slightly confusing when I first started with python - what the best method sharing a global instance is. I agree any such mechanism needs to be well thought out though. Perhaps some good documentation is the best solution. -- Mike On Tue, Apr 23 @ 15:49, François Pinard wrote:

...

I already use `config' (that precise identifier) a lot in my things, so please do not even _think_ using it for something else! :-) :-)

For small projects, such a thing might be overkill. For big projects, there are a few (maybe many) avenues and ways how to achieve this sharing, and I'm glad having the freedom of installing subtle nuances between projects, as needed. That is, I would fear a bit being pulled in an arbitrary direction: it would have to be convincingly damned good. :-)

Better, maybe, would be to merely document some of the possible avenues. Most of these require only a tiny investment in lines of code anyway.

-- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html

Fredrik Lundh

8:17 p.m.

michael wrote:

...

Upon futher reflection, I would agree with you. It's something I find slightly confusing when I first started with python - what the best method sharing a global instance is. I agree any such mechanism needs to be well thought out though. Perhaps some good documentation is the best solution.

http://www.python.org/doc/current/tut/node8.html#SECTION00810000000000000000... </F>

Aahz

8:57 p.m.

On Tue, Apr 23, 2002, Fredrik Lundh wrote:

...

michael wrote:

...
Upon futher reflection, I would agree with you. It's something I find slightly confusing when I first started with python - what the best method sharing a global instance is. I agree any such mechanism needs to be well thought out though. Perhaps some good documentation is the best solution.

http://www.python.org/doc/current/tut/node8.html#SECTION00810000000000000000...

That addresses the technical mechanism, but not the use case at hand. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ What if there were no rhetorical questions?

Fredrik Lundh

9:24 p.m.

Aahz wrote:

...

...
http://www.python.org/doc/current/tut/node8.html#SECTION00810000000000000000...

That addresses the technical mechanism, but not the use case at hand.

the use case was "some easy or straight-forward way of sharing a global instance across modules". the answer is "put it in a module". end of story. followups to comp.lang.python, thank you. </F>

Aahz

9:39 p.m.

On Tue, Apr 23, 2002, Fredrik Lundh wrote:

...

Aahz wrote:

...
...
http://www.python.org/doc/current/tut/node8.html#SECTION00810000000000000000...

That addresses the technical mechanism, but not the use case at hand.

the use case was "some easy or straight-forward way of sharing a global instance across modules".

the answer is "put it in a module". end of story.

That's what I said earlier. But where is this documented? Never mind, I just submitted a FAQ entry, 4.105. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ What if there were no rhetorical questions?

Guido van Rossum

9:45 p.m.

...

That's what I said earlier. But where is this documented?

I'm not sure that every possible handy coding tip deserves to be in the documentation. There ought to be (and are!) plenty of other introductory texts that can serve this purpose. Also, in this particular case, the question ("how do I share data between modules") and the answer ("put it in a module and import it") seems pretty obvious once you have a decent grip of module semantics. --Guido van Rossum (home page: http://www.python.org/~guido/)

Aahz

9:55 p.m.

On Tue, Apr 23, 2002, Guido van Rossum wrote:

...

Aahz:

...
That's what I said earlier. But where is this documented?

Also, in this particular case, the question ("how do I share data between modules") and the answer ("put it in a module and import it") seems pretty obvious once you have a decent grip of module semantics.

Maybe. I personally agree. But I've seen too many people on c.l.py who appear to be definitely past the newbie level miss the boat on this topic. We'll see whether the FAQ entry helps; it should be sufficient. I noted in the FAQ entry that importing a module is also the Pythonic answer to the Singleton design pattern, so people searching for Singleton should find that. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ What if there were no rhetorical questions?

Patrick K. O'Brien

10:04 p.m.

[Aahz]

...

Maybe. I personally agree. But I've seen too many people on c.l.py who appear to be definitely past the newbie level miss the boat on this topic. We'll see whether the FAQ entry helps; it should be sufficient. I noted in the FAQ entry that importing a module is also the Pythonic answer to the Singleton design pattern, so people searching for Singleton should find that.

Send them to Alex's article: http://www.aleax.it/5ep.html --- Patrick K. O'Brien Orbtech

Alex Martelli

10:09 p.m.

On Wednesday 24 April 2002 00:04, Patrick K. O'Brien wrote:

...

[Aahz]

...
I noted in the FAQ entry that importing a module is also the Pythonic answer to the Singleton design pattern, so people searching for Singleton should find that.

Send them to Alex's article: http://www.aleax.it/5ep.html

Using a module is a better answer over 90% of the time. Most people think of Singleton "by reflex" when they believe "there's gonna be only one of these", rather than when they really need to solve the forces covered in the Gof4's Singleton DP, first and foremost subclassability. Alex

Greg Ewing

24 Apr 24 Apr

2:13 a.m.

Guido:

...

def f(): print x # user thinks this should print the global # 2000 lines of unrelated code for x in "some sequence": # doesn't realize this overrides x do_something_with(x)

Perhaps a warning could be issued if the first reference to a local textually precedes the first assignment to it? Would that help catch things like this, without complaining about too much existing code? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Neal Norwitz

2:34 a.m.

Greg Ewing wrote:

...

Guido:

...
def f(): print x # user thinks this should print the global # 2000 lines of unrelated code for x in "some sequence": # doesn't realize this overrides x do_something_with(x)

Perhaps a warning could be issued if the first reference to a local textually precedes the first assignment to it? Would that help catch things like this, without complaining about too much existing code?

The standard library has 4 such occurrences: ftplib.py:678: Variable (macro_lines) used before being set ftplib.py:681: Variable (macro_name) used before being set tokenize.py:148: Variable (strstart) used before being set tokenize.py:149: Variable (endprog) used before being set The line numbers are from CVS version subtract 77/20 for 2.2.1. Neal

Michael Gilfix

2:37 a.m.

On Tue, Apr 23 @ 17:45, Guido van Rossum wrote:

...

I'm not sure that every possible handy coding tip deserves to be in the documentation. There ought to be (and are!) plenty of other introductory texts that can serve this purpose.

Also, in this particular case, the question ("how do I share data between modules") and the answer ("put it in a module and import it") seems pretty obvious once you have a decent grip of module semantics.

I agree now as well. I wouldn't have when I started though. At the time I wondered if I had committed a stylistic sin because it didn't feel "right". -- Mike -- Michael Gilfix mgilfix@eecs.tufts.edu For my gpg public key: http://www.eecs.tufts.edu/~mgilfix/contact.html

Greg Ewing

2:40 a.m.

Michael Gilfix :

...

The our is akin to declaring something static in C.

Except that it sounds like if two functions declare "our" variables with the same name, they get the same variable, whereas in C they would be different variables. (I think -- it just occurred to me that I'm not really sure about that!) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Tim Peters

3:53 a.m.

[Guido]

...

... This particular form of breakage was a common error reported on c.l.py and to help at python.org until we added UnboundLocalError to make the diagnostic cleaner.

It was indeed, and adding UnboundLocalError did cut the number of questions.

...

Maybe that's all that's needed;

It's hard to know what could really help more. If Greg Wilson is still running newcomer experiments, I'd like to see what newcomers have to say about this: x = 2 def f(): print x # A x = 3 f() print x # B A: What do you think should happen when the print at A executes? B: " " " " " " " " " " B " ? What I suspect, but don't know, is that a majority of newcomers who expect A to print 2 *also* expect B to print 3. That is, that they're thinking x is a global variable, and have no conception of local variables in mind. This is actually what happens in Icon, which also lacks declarations (in the same sense Python lacks them: it doesn't lack them <wink>). The difference is that all variables are global by default in Icon, and you need to explicitly say "local x" if you want a local var instead. That's error-prone for sure, by not quite as much so as Perl (where x is also global by default, but "local $x" sucks you into dynamic scoping; it does not in Icon).

Guido van Rossum

4:28 a.m.

...

...
def f(): print x # user thinks this should print the global # 2000 lines of unrelated code for x in "some sequence": # doesn't realize this overrides x do_something_with(x)

Perhaps a warning could be issued if the first reference to a local textually precedes the first assignment to it? Would that help catch things like this, without complaining about too much existing code?

Absolutely, and I believe that's what PyChecker is doing. Getting it added to Python's own compiler is tricky though -- maybe we should put off all such enhancements until Jeremy's new AST-based compiler is finished. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum

4:49 a.m.

[Tim]

...

It's hard to know what could really help more. If Greg Wilson is still running newcomer experiments, I'd like to see what newcomers have to say about this:

x = 2 def f(): print x # A x = 3

f() print x # B

A: What do you think should happen when the print at A executes? B: " " " " " " " " " " B " ?

What I suspect, but don't know, is that a majority of newcomers who expect A to print 2 *also* expect B to print 3. That is, that they're thinking x is a global variable, and have no conception of local variables in mind.

That's not a fair experiment until after you've given them a good concept of local and global variables without name conflicts. On the one hand, the importance of having local variables at all isn't clear if there are no name conflicts (until you introduce recursion, which I would *definitely* introduce much later, despite Matthias Felleisen's teachings :-). But on the other hand, it's easy to show that after >>> def f(): ... greeting = "hello world" ... print greeting ... >>> f() hello world >>> print greeting NameError: greeting >>> there's no global variable 'greeting', which can then be used to explain that variable assignments create local variables. You don't need to explain the *reason* for this feature at this point; that'll be clear by the time you've explained the rest. Next you can explain the global statement. This appears irrelevant to the proposed experiment, but it's useful to take away fears that you can't change globals at all: people coming from other languages will know that that is important, and worry how Python deals with this. After that you can show how name conflicts are handled in the "normal" case, where you shadow a global in a function by assigning to it and then using it, without a global statement. At some point you should also point out that if you don't *change* a global, you can use in a function it without the global statement. This comes so natural that it's easy to gloss over, especially when the "global" in question is a function or an imported module; but it *is* an important feature. *THEN* you are ready for the experiment Tim proposes above. --Guido van Rossum (home page: http://www.python.org/~guido/)

Scott Gilbert

7:37 a.m.

--- Tim Peters wrote:

...

This is actually what happens in Icon, which also lacks declarations (in the same sense Python lacks them: it doesn't lack them <wink>). The difference is that all variables are global by default in Icon, and you need to explicitly say "local x" if you want a local var instead. That's error-prone for sure, by not quite as much so as Perl (where x is also global by default, but "local $x" sucks you into dynamic scoping; it does not in Icon).

A few corrections. New variable names were local by default. The only way to get a global variable was to make a "global varname" statement at the global scope. Icon gave you a None-like value, &null, for your variables if you hadn't already assigned to them somewhere. So strictly speaking, you could read a variable before writing to it, and a declaration (first initialization) wasn't necessary the same way it is in Python. Using "local x" was strongly recommended because (a) you could use the -u switch of icont to warn you when you didn't declare your locals (usually from typos), and (b) in case you forgot that you used the same name as a global somewhere else, or more likely you "link"ed with a file that had global declarations you weren't aware of. So it really wasn't so error prone. (Icon's not even close to Perl in this regard. :-) Cheers, -Scott __________________________________________________ Do You Yahoo!? Yahoo! Games - play chess, backgammon, pool and more http://games.yahoo.com/

Alex Martelli

7:38 a.m.

On Wednesday 24 April 2002 04:40, Greg Ewing wrote:

...

Michael Gilfix :

...
The our is akin to declaring something static in C.

Except that it sounds like if two functions declare "our" variables with the same name, they get the same variable, whereas in C they would be different variables. (I think -- it just occurred to me that I'm not really sure about that!)

You think correctly: two C functions declaring static function-scope variables with the same name get different variables. static variables at FILE scope are 'shared' throughout the file, but not between files. Alex

Tim Peters

25 Apr 25 Apr

5:08 a.m.

[Guido, on

...

x = 2 def f(): print x # A x = 3

f() print x # B

A: What do you think should happen when the print at A executes? B: " " " " " " " " " " B " ? ]

...

That's not a fair experiment until after you've given them a good concept of local and global variables without name conflicts.

I think my point is getting lost twice: 1. My suspicion is that the majority of people I've explained this to, over the years and years, would have answered the equivalent to "well, B should print 3, of course!", when translated to the specific context in which their confusion arose. That is, I suspect they didn't understand local variables at all, not they that had some vision of "well, it's the global name unless and until I dynamically override it with a local name, at which point the global name with the same spelling is no longer relevant". 2. If we're talking about people willing to be taught "a good concept of local and global variables" before going postal, there's no "problem" to be solved here! I suspect even Mr. Kuchling understands how local variables work today <wink>.

...

... But on the other hand, it's easy to show that after ... which can then be used to explain that variable assignments create local variables. You don't need to explain the *reason* for this feature at this point .. Next you can explain the global statement ... After that you can show how name conflicts are handled ... At some point you should also point out that ... *THEN* you are ready for the experiment Tim proposes above.

Anyone willing to sit through that much explanation will have no problem understanding how locals work today. Besides, if you get to tell them beforehand that "variable assigments *create* local variables" dynamically, then I get to tell them beforehand that variable assignments don't create local variables dynamically. At that point we'd only be testing how faithfully they can regurgitate what they've just been told. I'm not a sure a difference between 12% comprehension rate and 13% comprehension rate is statistically significant anyway <0.9 wink>.

Tim Peters

6:06 a.m.

[Scott Gilbert, on Icon]

...

A few corrections. New variable names were local by default.

Ack! You're right! Years of stuffing in "local" decls made me forget why they were recommended to begin with.

...

The only way to get a global variable was to make a "global varname" statement at the global scope.

The difficulty being-- why explicit "local" was really recommended --that a new global magically changes every *implicit* local vrbl of the same name into a global. This bit too when an implicit local happened to have the same name as a new function added later, because function names are automatically global. Like: procedure main() f() g() end procedure f() count := 1 while count < 10 do count +:= 1 write(count) end procedure g() write(len([2, 3, 4]) | "oops!") end procedure len(x) return *x end That prints 10 3 If you just happen to change the name of the "len" function to "count" later: procedure g() write(count([2, 3, 4]) | "oops!") end procedure count(x) return *x end it suddenly prints 10 oops! instead (the expression count([2, 3, 4]) | "oops!" becomes 10([2, 3, 4]) | "oops!" because the once-local "count" in f is suddenly global, and retains its value of 10; then 10([2, 3, 4]) fails, and the alternative "oops!" is tried and succeeds). In real life these could be an absolute bitch to track down, so I always used local. Insert local count at the top of f and everything's fine again.

...

Icon gave you a None-like value, &null, for your variables if you hadn't already assigned to them somewhere. So strictly speaking, you could read a variable before writing to it, and a declaration (first initialization) wasn't necessary the same way it is in Python.

I prefer Python distinguishing between None and unbound too; e.g., I'd much rather have x := y blow up at once if y hasn't been bound than to silently propagate &null (or a Python None). &null could travel *all over the place* before you finally hit an expression where it didn't just silently fail (very few things in Icon raise errors, as the 10([2, 3, 4]) example should have suggested to onlookers <wink>).

...

Using "local x" was strongly recommended because (a) you could use the -u switch of icont to warn you when you didn't declare your locals (usually from typos),

Unfortunately, -u doesn't complain about the "10 oops!" example above: as far as it's concerned, all variables were declared in that example. Fortunately, it does complain about the original way of writing it, because "count" is undeclared then.

...

and (b) in case you forgot that you used the same name as a global somewhere else, or more likely you "link"ed with a file that had global declarations you weren't aware of.

Like an implicit "import *" in Python -- we're well aware of the dangers of *that* around here <wink>.

...

So it really wasn't so error prone. (Icon's not even close to Perl in this regard. :-)

Not for scope rules alone, no, although Perl sticking to explicit "my" everywhere isn't much worse than Icon sticking to explicit "local" everywere.

Scott Gilbert

8:30 a.m.

--- Tim Peters wrote:

...

instead (the expression

count([2, 3, 4]) | "oops!"

becomes

10([2, 3, 4]) | "oops!"

because the once-local "count" in f is suddenly global, and retains its value of 10; then 10([2, 3, 4]) fails, and the alternative "oops!" is tried and succeeds).

Yep that's ugly. I never wrote any programs in Icon larger than a couple thousand lines, and I never used Icon in a team setting, so I didn't get bit by things like this.

...

I prefer Python distinguishing between None and unbound too; e.g., I'd much rather have

x := y

blow up at once if y hasn't been bound than to silently propagate &null

I have mixed feelings on this one. I remember using the / and \ operators all over the place to work around it, so in that regard I guess I didn't like it. On the other hand it nicely parallels how tables would return &null (or other default) for keys that weren't already in there, and that I liked. In Python, I use the d.get('key', default) construct quite a bit. Probably because I learned to like getting a default value from Icon. I could use a DefaultDict class derived from dict, but that's less convenient and doesn't help me with **kwds arguments. Plus it's a hindrance to someone reading my code who knows what Pythonic behavior is supposed to be. No one says that key lookup in dicts/tables and variable lookup in the namespace have to follow the same rules though. It just seems that they do in both Icon and Python. Each with a different choice for rules.

...

(very few things in Icon raise errors, as the 10([2, 3, 4]) example should have suggested to onlookers <wink>).

I took two Icon courses from Ralph, and several times while sitting in class, I got the impression that he regretted a lot of things like this. Having integers usable in the place of functions was odd and rarely useful. He also didn't seem to have much respect for co-expressions. He was really excited about the graphics features though. Now *strings* as functions, that I used. :-) I think part of the charm of Icon was* that it felt like an experiment to see what cool new concepts come out of letting anything do everything. When it comes time to teach it or use it in a large project, having the language force a little more bondage is probably a good thing. * - I should say "charm of Icon is". It looks like it's still got something of a pulse in academia... In fact the latest version just built under Cygwin without major headaches.

...

Unfortunately, -u doesn't complain about the "10 oops!" example above: as

Yeah, a -U-and-I-really-mean-it flag would fix this. :-) Maybe if procedures were "const" the problem would go away. (While probably creating some new problem that I'm not thinking of...) Trying to draw this back as relevant to Python and this thread: I think Icon has an easier time with scoping rules than Python because execution starts in "procedure main()", not the beginning of the file. Since code isn't floating out there at the topmost level, one can't create variables there without an explicit global declaration. There really are only two scopes for global and local. (And some builtins of course...) Following this thread and reading some of the examples, I don't think I have a deep grasp of what the rules are in Python. I just have a set of conventions that I follow which seem to keep me out of trouble. Your example of: x = 2 def f(): print x #x = 3 ### uncomment to go BOOM kind of surprised me. I presented it to another co-worker who wasn't very Python savvy. It caught me off guard that he expected the exception at the print statement, but he didn't realize it only occurs when you have the "local declaration" later.

...

Like an implicit "import *" in Python -- we're well aware of the dangers of *that* around here <wink>.

Yep, and I litter my libraries with underscores to protect from it. A number of users at my company gripe about having to use parens and commas for function calls, especially in interactive mode, so convincing them to use module.function(arg, arg, arg) is going to be a really tough sell... (Someone suggested the LazyPython hack as a sort of solution to the first problem, but they would prefer "import module" defaulted to the bad behavior) The underscores also make it easy to see when something is global or not, avoiding the BOOM in your example above.

...

...
So it really wasn't so error prone. (Icon's not even close to Perl in this regard. :-)

Not for scope rules alone, no, although Perl sticking to explicit "my" everywhere isn't much worse than Icon sticking to explicit "local" everywhere.

My nostalgia for Icon makes me forget about any of the bad things. I don't have much nostalgia for Perl, so it's faults I remember. Cheers, -Scott __________________________________________________ Do You Yahoo!? Yahoo! Games - play chess, backgammon, pool and more http://games.yahoo.com/

8034

Age (days ago)

8037

Last active (days ago)

List overview

Download

58 comments

14 participants

participants (14)

Aahz
Alex Martelli
Fredrik Lundh
Greg Ewing
Guido van Rossum
Jeremy Hylton
jeremy＠zope.com
Michael Gilfix
Neal Norwitz
Patrick K. O'Brien
pinard＠iro.umontreal.ca
Scott Gilbert
Steve Holden
Tim Peters

LOAD_NAME & classes

Jeremy Hylton

jeremy＠zope.com

jeremy＠zope.com

Jeremy Hylton

Steve Holden

Jeremy Hylton

Michael Gilfix

pinard＠iro.umontreal.ca

Michael Gilfix

pinard＠iro.umontreal.ca

pinard＠iro.umontreal.ca

Michael Gilfix

Neal Norwitz

Michael Gilfix

Scott Gilbert

Scott Gilbert

tags

participants (14)