
I'm glad to see Anthony ratcheting down. At this point, we need to be fixing bugs and improving doc. Maybe Anthony and I should have a contest to see who can revert the most changes. :-) There are at least 6 bugs that really, really need to be fixed before release. Several of these are AST bugs. Jeremy knows about them and plans to fix them once he's back from vacation. Anyone else wanna help out? One is for a socket problem and another is for doc. The current list of serious bugs are in the PEP: http://www.python.org/dev/peps/pep-0356/ If there are any bugs you think should be considered show stoppers, mail them to the list and I will update the PEP. If you are a committer, just update the PEP yourself. We really need everyone to help. There were a lot of changes that didn't have tests and/or NEWS entries. I tried to reply to the checkin messages for those I noticed. I have tons of messages in my inbox where I don't know if the issue was addressed or not. Can everyone try to find the holes? And new ones keep popping up! Please let the author know they need to fix the problem. It's really tempting to just revert these changes... We also need to fix the test suite. This warning needs to be addressed! Lib/struct.py:63: DeprecationWarning: struct integer overflow masking is deprecated return o.pack(*args) Since we are in feature freeze, now seems like a good time to make a PEP for 2.6: http://www.python.org/dev/peps/pep-0361/ It's pretty empty right now. The plan is to make the schedule in a year from now. Start adding your new features to the PEP, not the code. n

On Fri, 30 Jun 2006, Neal Norwitz wrote:
The current list of serious bugs are in the PEP:
Among them is this one: Incorrect LOAD/STORE_GLOBAL generation http://python.org/sf/1501934 The question is, what behaviour is preferable for this code: g = 1 def f(): g += 1 f() Should this raise an UnboundLocalError or should it increment g? (Or, in other words, should augmented assignment be considered a local binding like regular assignment, or not?) -- ?!ng

[Ka-Ping Yee, on http://www.python.org/dev/peps/pep-0356/ ]
Of course it should, since that's the way it _is_ treated in all released Pythons, and there was no intent to change the semantics (let alone a FutureWarning in 2.4 to alert users that the meaning was going to change in 2.5). The Reference Manual also makes no distinction between assignment statements and augmented assignment statements in any relevant respect. The change in behavior here in 2.5 is plainly a bug (although someone may want to argue "it should" be different in P3K).

Ka-Ping Yee <python-dev@zesty.ca> writes:
I didn't think there was any question: this change in behaviour from 2.4 is just an accidental change, a bug that should be fixed. If you want to elevate it to feature status, I think we've missed the freeze :) (and also, I oppose the idea). Cheers, mwh -- If I didn't have my part-time performance art income to help pay the bills, I could never afford to support my programming lifestyle. -- Jeff Bauer, 21 Apr 2000

Ping> The question is, what behaviour is preferable for this code: Ping> g = 1 Ping> def f(): Ping> g += 1 Ping> f() If you treat "g += 1" as "g = g + 1" then it should create a local variable with a value of 2. There being no global statement in f() it must not modify the global variable. Skip

On Fri, 30 Jun 2006, Andrew Koenig wrote:
I saw messages out of sequence and did not realize that this would be a change in behavior from 2.4. Sigh.
Yes, this is not a good time to change it.
I hope Py3000 has lexical scoping a la Scheme...
Me too -- that would be really nice. -- ?!ng

On 6/30/06, Ka-Ping Yee <python-dev@zesty.ca> wrote:
That's not a very constructive proposal (especially since I don't know Scheme). Perhaps you could elaborate on what needs to change? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, 30 Jun 2006, Guido van Rossum wrote:
Sorry! I should have been more clear. Right now, Python supports the existence of nested scopes: a = 3 def f(): a = 4 def g(): a = 5 print a # prints 5 g() print a # prints 4 f() print a # prints 3 The above example shows that there are three distinct scopes, and that each one has a distinct binding named 'a' -- assigning to one doesn't affect the others. a = 3 def f(): b = 4 def g(): c = 5 print a, b, c # i can see all three g() f() The above example shows that all of the scopes can be *read*. But in today's Python, not all of the scopes can be *written*. a = 3 def f(): b = 4 def g(): c = 5 a, b, c = 0, 1, 2 # changes local c, not outer a and b g() f() The code in g() can affect its own local, 'c', and it can affect the global variable 'a' if it declares 'global a', but no matter what you write in g(), it cannot assign to 'b' (or to any other intermediate scope). This is a strange limitation and it would be nice to remove it. The asymmetry comes from Python having one foot in the new paradigm of nested lexical scopes and one foot still in the older paradigm of only two scopes, local and global. Most other languages that support lexical scoping (including Scheme, JavaScript, Ruby, Perl, E, Java, Smalltalk) provide a uniform way to read and write to scopes at all levels. This is done by letting programmers specify the scope in which they want a variable bound (usually with a keyword like "var" in JavaScript, "my" in Perl, or "define" in E). So here are some thoughts on how Python might be adjusted to support this. I'm not saying these would be the only ways, but at least they're some ideas to start with. In JavaScript, the "var" keyword is required whenever you want to declare a local variable. Anything without "var" is assumed to be a global name. The cleanest and most consistent solution that comes to mind would be to adopt exactly this for Python. Without "var": a = 3 # global def f(): b = 4 # global def g(): c = 5 # global a, b, c = 0, 1, 2 # changes all three globals g() f() print a, b, c # prints 0, 1, 2 With "var": var a = 3 # global def f(): var b = 4 # local to f def g(): var c = 5 # local to g a, b, c = 0, 1, 2 # changes outer a, outer b, and c print c # prints 2 g() print b # prints 1 f() print a # prints 0 print b # no such name print c # no such name But that is a big change. Perhaps it would be too unpythonic to suddenly require declarations for all local variables. So an alternative would be to retain the default assumption that undeclared variables are local. Here's what we'd get: Without "var": a = 3 def f(): b = 4 def g(): c = 5 a, b, c = 0, 1, 2 # changes local c, not outer a and b g() f() With "var": var a = 3 def f(): var b = 4 def g(): var c = 5 a, b, c = 0, 1, 2 # changes outer a, outer b, and c g() f() Now i think this is a little bit weird, because the statement "var b = 4" in an outer scope changes the meaning of "b" in an inner scope. But it does have the virtue of retaining behaviour compatible with today's Python, while offering a way to get proper lexical scopes for those who want to use them. Thoughts? Other ideas? -- ?!ng

Ka-Ping Yee <python-dev@zesty.ca> wrote: [snip lexical scoping option]
Using a keyword in an outer scope to state that a variable could be used in a nested scope is counter to the current method for accessing a parent scope with 'global'. Using 'var' as the equivalent of 'global', only for nested scopes, would be a more reasonable approach. However, I'm -1 on the feature now, for the same reasons I've been -1 on the feature for the last 2 times it has come up. In many of the cases where lexically nested scopes have been used to solve problems in Python, and programmers have run into a 'limitation' where not being able to modify a value in a parent scope has hindered them, the problem could have been better solved with another method that was more readable, more extensible, etc. What I asked before, and what I'd like to ask again, is if there are any _nontrivial uses_ of lexically nested scopes which are made cumbersome by our inability to write to parent scopes. If there aren't, then I'm going to again have to argue against new syntax, keywords, and their use. If there are, then we'll see how compelling such uses are. - Josiah

Josiah Carlson wrote:
The trouble with taking that position is that the very cases which would benefit are very *simple* ones, where it would be cumbersome to refactor it to use a class, or mutable object in the outer scope, etc. So you've effectively set up your acceptance criteria to be unmeetable.
If there aren't, then I'm going to again have to argue against new syntax, keywords, and their use.
There's one very simple way we could do this in Py3k without requiring any new syntax or keywords: just redefine the meaning of "global" to mean "not local". -- Greg

Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
If the only code that benefits from such changes are "very *simple*", then I think that says something about its necessity. That is, if anything more complicated than those that are "very *simple*" generally don't benefit, then I don't believe that such a modification would be beneficial to the language overall. Further, a simple namespace factory can handle much of the current issues, without needing to create or change keywords. def namespace(**kwds): class namespace(object): __slots__ = kwds.keys() def __init__(self): for i,j in kwds.iteritems(): setattr(self, i,j) return namespace() def trivial_counter(start): ns = namespace(current=start-1) def next(): ns.current += 1 return ns.current return next Maybe a variant of the above namespace factory should make it into the collections module.
I would probably be a solid -0 on such a proposal; I still don't think it's really necessary, but I've never used (or really seen) global more than one level deep, so would guess its impact would be low. - Josiah

On 7/1/06, Josiah Carlson <jcarlson@uci.edu> wrote:
This has been discussed at length in the following thread that I started in February and at least one time before that. http://mail.python.org/pipermail/python-dev/2006-February/061568.html I think using the "global" keyword is probably the lowest impact form and has the least amount of backwards incompatibility. Below is the part of the last thread that I talked about changing the meaning of "global." http://mail.python.org/pipermail/python-dev/2006-February/061852.html Having the "global" keyword semantics changed to be "lexically global" would break in the cases that "global" is used on a name within a nested scope that has an enclosing scope with the same name. I would suppose that actual instances in real code of this would be rare. Consider:
Under the proposed rules:
* f() *2
PEP 227 also had backwards incompatibilities that were similar and I suggest handling them the same way by issuing a warning in these cases when the new semantics are not being used (i.e. no "from __future__"). Most people probably think that this is a low impact "wart" on the Python language that not really worth fixing as there are workarounds (i.e. mutable objects) or other ways to express (i.e. use classes) such things, but it does trip people up from time to time as warts typically do--I guess that's why this gets brought up now and again. Best regards, Almann -- Almann T. Goo almann.goo@gmail.com

What about doing something similar to how import was changed? .a = 5 # this scope (self might be too magical ..a = 3 # up one scope ...a # up three Of course, this looks ... perhaps a bit strange. Also, counting is a bother. //Simon

I'd rather see a simpler rule: = never defines a variable in a surrounding scope. If you want to affect the binding of such a variable, you have to define it explicitly in the scope in which you want it. Example: x = 42 def f(): x = 123 # rebinds x as defined above y = 123 # defines local variable f() print x # prints 123 print y # error -- y not defined Yes, I know that rule is too simplistic. But I think I'd still prefer it to the way things are now.

On Sat, 1 Jul 2006, Andrew Koenig wrote:
I agree with you that this is a nicer and more consistent rule. What do you think of the proposal for a keyword to say "don't rebind"? It would achieve the same distinction you're aiming for above, but without the drastic incompatibility with today's Python. This has been previously discussed as "change the meaning of 'global' to mean 'not local'": http://mail.python.org/pipermail/python-dev/2006-February/061568.html http://mail.python.org/pipermail/python-dev/2006-July/066908.html I support this proposal, though i would prefer a clearer keyword such as "outer x" or "nonlocal x". If we can't agree on another keyword (or can't afford to spend one more keyword), i'm willing to support "global" for this purpose. -- ?!ng

Josiah Carlson wrote:
If the only code that benefits from such changes are "very *simple*", then I think that says something about its necessity.
The point is that they're only "very simple" if you can write them using access to an outer scope. Without that ability, they become less simple, less efficient, more convoluted, harder to follow, etc. Also I don't buy the argument that something has to be useful for big, complicated things in order to be worth having in the language. -- Greg

Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
As is known and has been stated, assigning to a parent scope can be emulated in various ways, either through an explicit namespace object, or through a namespace list.
I never claimed that something needed to be useful for "big, complicated things" in order to be worth having in the language. To be explicit, if nontrivial code isn't improved, that doesn't necessarily mean that the feature is useless. However, if the feature is really only useful for generally trivial cases *without* the feature, then making them even more trivial, I think, is a bit of over optimization. - Josiah

On 7/3/06, Josiah Carlson <jcarlson@uci.edu> wrote:
And the fact that this desire and need remains, even amongst people who should know better, suggests that it may be worth supporting it more directly, as the current work-arounds ain't pretty. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

"Guido van Rossum" <guido@python.org> wrote:
Perhaps not pretty, but not wholly ugly either. Or expressed another way, it's a wart, but the wart isn't 1" across on a forehead, it's fairly small and tucked away on an elbow. I had hoped that there would be a response to my second (and I believe more applicable statement); "if the feature is really only useful for generally trivial cases *without* the feature, then making them even more trivial, I think, is a bit of over optimization." As for a solution, I find the "global means 'not local'" proposition is the least undesireable of the possibilities. It suffers from a change in semantics and potential name masking issues, but I don't believe these are any more serious than normal global masking for the former, and the latter is solvable with a __future__ (at least for 2.6). I'm a solid -0 on this particular proposition, which is far better than the -1 I am on all of the other recent lexical scoping propositions. - Josiah

Josiah Carlson wrote:
It really depends on how common the trivial case is. In other words, multiply the savings for each occurance times the number of occurances. (Unfortunately, I don't know what units to measure said savings in - is there a unit of 'mental disconnect' or unintuitiveness? :) In an idealy world, the language would allow everything to be said in the most comprehensible way possible. Longer and more verbose ways of stating something are at an inherent disadvantage in this, simply because of the time it takes to scan and absorb the information by the human brain. However, losing excess syntax has to be done in a way that doesn't also lose information. Highly compressed representations of a concept may require such a level of abstraction that it is as much work to puzzle out their meaning as it would be to read the longer version and more. To put it another way - I am an advocate of applying Claude Shannon's theory of information to language design. The highest level of compression should be used for expressions that occur the most frequently.
I'd say that the more common case is where you want global to really mean global - that is, you want to be able to write to some module-level variable, regardless of how deeply nested your function scope is. While being able to access the 'next outer scope' is occasionally useful, it's not all that common. So changing the behavior of 'global' in this case would be both confusing (since it no longer means 'global'), and less useful (because it doesn't match the most common case.) (This assumes that I haven't completely understood the meaning of the phrase 'not local' - I assumed that it means 'not defined in this scope') Of course, the reason why it's not all that common may be because of the fact that it's not as easy to do, and so people tend to (consciously or otherwise) avoid that pattern in their designs. That being said, I don't think that's necessarily such a bad thing. Python isn't Scheme, and the scoping rules of Python are IMHO more oriented towards practicality and common sense than theoretical purity. This is why I'm not bothered by the fact that Python doesn't create a new scope for loop statements and such. Most of the time, this is what you want. It does mean that you need to name all of your variables uniquely, but that's good programming style in any case. The same is true for local variables not needing to be specially declared as 'my' or 'var' - most of the time, a local variable is what you want. On the other hand, the thing about theoretical purity is that it can be so mouth-wateringly powerful at times. For example, a language that supports closures is, IMHO, at least twice as powerful as a language that doesn't -- because you can use them in so many different and interesting ways. OK, so about the lexical scoping issue - let me brainstorm a moment: One idea would be to introduce the keyword 'local' which would have the effect of capturing any 'global' statements in any enclosing scope. So for example: f = 1 def a(): local f f = 2 def b(): global f f = 3 So in this case, the 'global' statement, which would normally associate 'f' with the outermost (module-level) scope, would instead associate 'f' with the innermost 'local' declaration of that variable. So in the above example, assigning 3 to f assigns it to the middle scope, but does not affect the module-level definition. Admittedly, that's a bit confusing and also verbose, considering that you are not only adding an extra keyword, but also using two statements to specify the home of one variable. Another alternative would be a way to declare an explicitly scoped variable. Lets use the keyword 'my' to indicate this: f = 1 def a(): my f = 2 def b(): f = 3 In this case, what the 'my' statement is doing is indicating that this scope 'owns' the definition of 'f' -- in other words, the definition is hoisted out of any enclosed scopes. So again, in the above example, the innermost assignment will be to the definition of 'f' in the middle scope. What's interesting about this is that you can use the same method with globals: my f = 1 def a(): f = 2 a() print f # prints '2' So again, you are indicating that the global scope 'owns' the definition of 'f', and any enclosed scopes should use that definition, and not create their own. Of course, if you really *do* need to have your own version, you can always override the 'my' statement with another 'my' statement: my f = 1 def a(): my f = 2 a() print f # prints '1' The 'my' statement essentially changes the scoping rules for all variables of that name, within the defining scope and all enclosed scopes. Of course, you can also override this behavior using the 'global' statement, which does exactly what it does now - makes the reference global (i.e. module-level): my f = 1 def a(): global f f = 2 a() print f # prints '2' All right, I'm pretty happy with that. Brainstorming done. :) -- Talin

Talin wrote:
I believe the proposal in question would cause no net worsening in this information content, and may actually improve it slightly, due to allowing a few things to be written in a shorter and clearer way, while allowing the vast majority of existing things to be written in exactly the same way.
It would still mean that, except in the (expected to be *extremely* rare) case where you happened to have a variable with the same name assigned in some intermediate scope. Such cases would be easily fixed by renaming the intermediate variable -- using a name of shorter or equal length, if you like, to keep the information content up. :-)
So changing the behavior of 'global' in this case would be both confusing (since it no longer means 'global'),
An alternative would be to change the keyword as well, to something like 'outer', which would better match its semantics. But if that were done, I would argue for the *removal* of the existing 'global' keyword, which would then be almost completely redundant. This would break large amounts of existing code, however, and it's highly dubious whether that would be worth the small increase in pendantic accuracy, even in Py3k. We're not supposed to be *gratuitously* breaking things in Py3k, after all.
(This assumes that I haven't completely understood the meaning of the phrase 'not local' - I assumed that it means 'not defined in this scope')
Yes, the new meaning would be "in the next outermost scope where there is an assignment to this name, or the module scope if you get that far".
Yes, but I find it hard to regard being *forbidden* from assigning to intermediate scopes as something driven by practical need rather than just being a historical accident. Back when there were strictly two scopes, many people argued themselves blue in the face that this was actually a *good* thing, even if you didn't realise it, and that Python was doing you a favour by enforcing it. Eventually a measure of sanity prevailed, and we got something a lot more like traditional lexical scoping. But one remnant of the old system remained, like a vestigial organ -- the 'global' statement that reaches all the way out to the module scope, regardless of what exists in between. To someone used to lexical scoping in almost any other language that has it, this is a *very* strange and unintuitive thing. Looking back, I think the meaning of 'global' should have been redefined right then at the same time. That would have been the logical and consistent thing to do, and in my opinion would have resulted in a scoping model that was simpler, more useful and no less practical. The most theoretically pure thing would have been to change it to 'outer' at the same time, but that would have broken too much code, and would therefore not have been practical. See, I'm not immune to practicality arguments. :-)
One idea would be to introduce the keyword 'local' which would have the effect of capturing any 'global' statements in any enclosing scope.
That seems unnecessary to me. Or at least not necessary enough to be worth the extra complexity in the scoping model. -- Greg

Josiah Carlson wrote:
I don't think "trivial" is the right word to use here, since it implies something that's of so little importance that it can be ignored. But the simple cases are precisely the ones where this wart hurts the most, so we can't ignore them. Arguments that a feature is undesirable because this or that workaround exists seem like post-hoc justifications to me. Think about it the other way around -- if writing to outer scopes had been straightforward from the beginning, would you be arguing for the *removal* of that ability? Would it even have occurred to anyone to do such a thing? -- Greg

I'd like to inject an example that might help make this discussion more concrete. Consider the following function: def for_each(seq, f): for i in seq: f(i) I'm sure I've seen more than one instance of someone on comp.lang.python trying to do the equivalent of using a function such as this one to compute the sum of the elements of a sequence as follows: def sum(seq): result = 0 def accum(i): result += i for_each(seq, accum) return result and wonder why it doesn't work. Still odder, why it doesn't work and the following does: def sum(seq): result = [0] def accum(i): result[0] += i for_each(seq, accum) return result[0] Transforming the first definition of sum above into the second may be trivial, but only if you've encountered the technique before. Moreover, the first version of sum uses a technique that is more than 45 years old (!), as it was available to Algol 60 programmers.

On 7/3/06, Andrew Koenig <ark@acm.org> wrote:
Much though the Algol 60 tickles my nostalgia (it was my first programming language!) I don't think that it's a particularly strong argument. I like to think that we have better ways these days. I think you need to come up with a better motivating example; the above is particular un-idiomatic Python. It starts by defining a higher-order function for_each that has little to offer over writing an explicit for loop, and then uses this to motivate writing a simple operation (result += i) as a function instead so that it fits in the inconvenient for_each() API. I understand that both for_each() and accum() are just examples of more complicated functions, but I can't help thinking that the problem here only occurs for very *simple* functions in the place of accum(); a more complicated form of accum would likely be a bound method of a class instance which carries the state. A better way to decompose these kinds of problems is probably by using generators. The equivalent of for_each() would not take a function parameter but *yield* the successive values instead of calling f() with successive values; e.g.: def for_each(seq): for i in seq: yield i Then the sum() function could be written like this: def sum(seq): result = 0 for i in for_each(seq): result += i return result -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Even if so, that's not the point I was trying to make. The point is that there is a programming technique that is widely used, works in many languages, and has been around for 45 years; and when you try to use it in Python, it fails. I believe that such failures, even if there are alternative ways of solving the problems that engender them, are barriers to learning that should be removed if it is possible to do so without substantial cost.

On 7/3/06, Andrew Koenig <ark@acm.org> wrote:
That's true for lots of things that have been around for a long time. Can you provide a better example? (The use of += is not particularly relevant to the example; it could just as well have said "result = result + i".)
And that is of course the crucial question. Probably the only proposal that has any chance of succeeding is to extend the 'global' statement so that it also applies to variables in intermediate outer scopes; or perhaps a new keyword (since "global" is not a very good name for the extended semantics). We would have to decide what this example would do: def outer(): def inner1(x): global a a = x def inner2(): return a return inner1, inner2 f1, f2 = outer() g1, g2 = outer() f1(42) g1(0) print f2() # Does it print 0 or 42 ??? In current Python this prints 0: there's only one (global) variable a, and the call to g1(0) overrides the value that was stored by f1(42). If global were changed to mean "nonlocal" what should it do? The question the example poses is that a is not initialized except in inner1() -- we somehow have to decide whether this is an error, or whether it chooses some well-defined outer scope, with the choices being the nearest enclosing scope or the outermost (truly global) scope. We have one guideline: if there is exactly one outer scope that defines a variable named a, we would like it to be referenced (by the 'global a') statement and the variable references governed by it automatically. Also, of there's more than one such scope, we'd like it to reference the innermost one. But this doesn't have a natural extension to what should happen if there's no such scope! Perhaps the best solution would be to make it an error if there wasn't a visible variable named a in an outer scope. That would suit me fine because I'd like to migrate towards more static analysis of variables anyway. If that means equipping modues with traps for attempts to store arbitrary variables into their namespaces, that's fine with me (as long as there's some escape -- and of course instancs and classes remain fully dynamic). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Josiah> As for a solution, I find the "global means 'not local'" Josiah> proposition is the least undesireable of the possibilities. It Josiah> suffers from a change in semantics and potential name masking Josiah> issues... Pychecker and PyLint both already identify cases where builtins are masked by locals or module globals (and may identify cases where locals mask module globals - I don't recall). I suspect both could be generalized in this regard without a huge effort. That's probably the best place for this sort of warning. Skip

Maybe an object, like self, for referring to enclosing scopes? a = 3 def f(): b = 4 def g(): c = 5 outer.outer.a, outer.b, c = 0, 1, 2 # changes outer a, outer b, and c g() f() Chaining the keyword looks a little weird, but it is not often that you have to refer to variables in the enclosing scope of the enclosing scope. I have often wanted something similar to that for global variables, instead of the global declaration: cache = None def init(): if not global.cache: global.cache = init_cache() -- mvh Björn

Ka-Ping Yee wrote:
while offering a way to get proper lexical scopes for those who want to use them.
I don't disagree with anything you said, but I think it would be a good idea to avoid using phrases like "proper lexical scopes", which is likely to set people off on a tangent. The issue isn't lexicality, it's writeability. -- Greg

Ka-Ping Yee <python-dev@zesty.ca> wrote:
That's not the Python way, IMO. I think the right way (assuming we actually want to allow it) is to introduce a pure assignment statement in addition to the assignment/declaration statement that we already have. For example: a = 1 def f(): b = 2 a := 2 def g(): b := 3 print a, b, c g() f() would print "2 3 4". The := would assign but not declare a variable in the current scope. Neil

Neil Schemenauer wrote:
The := would assign but not declare a variable in the current scope.
There are other benefits to such a statement, too, since we can make it similar to other augmented assignments by letting the object being assigned to interfere with the process. a := 2 could translate to something like: a = a.__assign__(2) with the default behaviour of __assign__ simply being: def __assign__(rhs) return rhs This gives you: - runtime checking for typos (you can't accidentally declare a new variable with := when you really meant to assign to an existing one) - if/when control flow analysis is added to the AST compiler, it will be picked up as an error at compile time along with the other augmented assignments - the object being assigned to can validate/modify its replacement (e.g. automatically wrapping it in a weakref proxy, or checking that it has the correct type) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

That's not a very constructive proposal (especially since I don't know Scheme). Perhaps you could elaborate on what needs to change?
The fundamental principle is that the binding of every name is determined during compilation, not during execution. This property does not quite apply to Python at present. For example: x = 42 def f(): y = x x = 123 return y f() This example fails with "local variable 'x' referenced before assignment" because the compiler sees that f contains an assignment to x, so it makes x a local variable, and then when you try to assign x to y during execution, it fails. This behavior is consistent with the notion of lexical scoping. However, if I write def g(): return x x = 42 g() the result is 42. With lexical scoping, I believe it should be undefined. The reason is that when the compiler encounters the definition of g, variable x is not yet bound, and there is nothing in the body of g that would bind it. Therefore, g requires a binding to exist at the time it is compiled; because no such binding exists, this example would be an error (at compile time) under lexical scoping.

The trouble is that you don't necessarily know in what scope they will be defined, so I think that forcing you to be explicit about it is useful. Can you show me an example of where you think it isn't? Incidentally, I think that lexical scoping would also deal with the problem that people often encounter in which they have to write things like "lambda x=x:" where one would think "lambda x:" would suffice.

[Andrew Koenig]
They _shouldn't_ encounter that at all anymore. For example,
works fine in modern Pythons. Earlier Python's had a no-exceptions 3-scope implementation (local, global, builtin), and in those the "x" in the lambda body was "not local" (was either global or builtin, although the compiler couldn't tell which of those two it was). In _those_ Pythons people had to write "lambda x=x: x+1" instead, to suck the binding of the outer x into the lambda body, but if people are still doing that they're confused. Modern Pythons do have lexical scoping + global + builtin, although there's no way to spell "rebind a name local to an outer scope from within an inner scope".

[Giovanni Bajo]
So stay away from excruciating abuses of lexical scoping you don't understand ;-) What do you _expect_ `i` to refer to? "Oh, it should guess that I didn't really mean to defer evaluation of the lambda body at all, but instead evaluate the lambda body at the time I define the lambda and then synthesize some other function that captures the specific outer bindings in effect at lambda-definition time" doesn't really cut it. Try spelling what you think you want here in Scheme. Before it works, you'll probably end up with some equally "atrocious" (let ((i i)) ...) gimmick to force capturing each binding for `i` as it flies by. Else Scheme will also use the outer binding for `i` in effect at the time the lambdas are _executed_. This isn't typical use for lambda, and I don't think it's what Andrew had in mind.

[Giovanni Bajo]
[Tim Peters]
I think I understand what happens, I just don't know whether this can be "fixed" or not. Unless you are saying that the above behaviour is not only a complex side-effect the way things are, but the way things should be. Do you agree that it would be ideal if the above code generated range(10) instead of [9]*10, or you believe that the current behaviour is more sound (and if so, why)? As for actual implementing this change of semantic, the fact that `i` is a local variable in the outer scope (assuming it's all within a function), doesn't make it possible for Python to early-bound it, by realizing that, since `i` is not an argument of the lambda, and it's a local of the outer scope? At worse, couldn't Python do the "i=i" trick by itself when it sees that `i` is a local in the outer scope? Right now I can't think off-hand of a case in which this would break things. [Tim Peters]
This isn't typical use for lambda,
Yes, maybe it's not the most used idiom and Andrew wasn't referring to this, but it happens quite often to me (where 'often' means 'many times' among my rare usages of lambda). For instance, in GUI code, it's common to do things like: for b in self.buttons: self.setEventCallback(b, "clicked", lambda: self.label.setText("I pressed button %r" % b)) ... which of course won't work, as written above. Giovanni Bajo

"Giovanni Bajo" <rasky@develer.com> wrote in message news:027f01c69caf$b1c20450$d1b12997@bagio...
. Do you agree that it would be ideal if the above code generated range(10) instead of [9]*10,
No. You are trying to reify an optical illusion resulting from putting a constant function definition inside a loop. Making the meaning of 'def f(): return i' depend on the definition-time context by partially and variably evaluating the body would make code much harder to read and understand. Consider: if a: i=666 <intervening code> def f(): return i
It would make code more fragile. for i in range(666): print name[i] ... <intervening code> ... def total(num): return cost[item]*num Now someone decides first loop should have more expressive loop var name and changes the first line to for item in range(666): print name[item] and the meaning of total is completely changed. Adding such long-range coupling between language statements strikes me as a poor idea. Terry Jan Reedy

For Common Lispers and probably Schemers, Python has some surprising semantics around scope and lifetime extent of variables. Three that leap out at me are: * function parameters with default values are NOT new bindings for each invocation, so a default value of [] changes if you destructively modify this list object in the function * loop variables are NOT distinct lexical variables. The binding gloms on to a variable in the function's scope, both changing that lexical binding and not creating a new one for the loop (that goes away when the loop's scope ends) * loop variables are NOT distinct bindings per iteration, leading to the surprising results below Bill -----Original Message----- From: python-dev-bounces+billchi=microsoft.com@python.org [mailto:python-dev-bounces+billchi=microsoft.com@python.org] On Behalf Of Bill Janssen Sent: Friday, June 30, 2006 6:31 PM To: Giovanni Bajo Cc: Phillip J. Eby; Ka-Ping Yee; Guido van Rossum; Tim Peters; python-dev@python.org Subject: Re: [Python-Dev] 2.5 and beyond
Isn't this exactly what you'd expect? Maybe I've been writing Python for too long... :-). Bill _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/billchi%40microsoft.co m

"Bill Chiles" <billchi@microsoft.com> wrote:
One thing to remember is that Python is not Scheme/Lisp. It borrows some ideas from Scheme/Lisp, but that borrowing does not necessitate a it also use a completely equivalent scoping mechanism. From what I have been hearing about Python 2.6, and 3.0, the three "surprises" you describe are not going to be "fixed" (with respect to expected Scheme/Lisp semantics). Feel free to argue as to why they should be "fixed" in Py3k (unless Guido says, "you're dreaming"), but please do so in the py3k list. - Josiah

On 7/4/06, Josiah Carlson <jcarlson@uci.edu> wrote:
One thing to remember is that Python is not Scheme/Lisp. It borrows some ideas from Scheme/Lisp,
I can say it stronger. Any resemblance between Python and Scheme or Lisp is purely a coincidence. Neither language is in Python's ancestry, at least not explicitly; I'd never used or tried to learn Scheme when I started Python (still haven't) and my Lisp experience was limited to copying Emacs startup code from friends (still is). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

"Giovanni Bajo" <rasky@develer.com> wrote in message news:020c01c69ca1$754c7310$d1b12997@bagio...
The 'subtle sematic' had nothing to do with lambda but with Python functions. The above is exactly equivalent (except the different .funcname) to a = [] for i in range(10): def f(): return i a.append(f) del f That should be equally confusing (or not), and equally requires the 'i=i' trick (or not). As is, either function definitiion is a constant and the loop makes useless duplicates. Either form would have the same effect is hoisted out of the loop. Terry Jan Reedy

Giovanni Bajo wrote:
This has *nothing* to do with the semantics of lambda! It's because Python's for-loop doesn't put its control variable in a new scope, the way Scheme's equivalent construct does. *That's* what needs to be addressed to fix this problem. I've made a suggestion about that before, but Guido rejected it, so I won't repeat it here. -- Greg

[Giovanni Bajo]
[Greg Ewing]
I don't think I follow that. Scheme has no loops in Python's sense -- things like "do" are shorthand for expressing stylized recursion, where each conceptual iteration gets a fresh set of "loop variables". When people talk about giving a Python for-loop vrbl its own scope, they generally don't mean a new scope on _each iteration_, they just mean that, e.g., i = 5 for i in range(10): # do stuff print i prints 5 intead of 9, about the same as creating a nested block with its own autos in C. The Scheme way is more like: i = 5 def step(i): # do stuff if i < 9: step(i+1) step(0) print i except with tail-recursion elimination. That also prints 5, but does a hell of a lot more than _just_ arrange for that.
Don't recall what that was, but creating a new scope on each iteration sounds hard to explain in Python. If Giovanni wants the Scheme way ;-), it's available: """ a = [] def step(i): a.append(lambda: i) if i < 9: step(i+1) step(0) print [x() for x in a] """ prints [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], although it's more sanely written in Python with a loop: """ def make_lambda(i): return lambda: i a = [] for i in range(10): a.append(make_lambda(i)) print [x() for x in a] """ Abusing the default-argument machinery to capture current bindings is never necessary, and _is_ abuse. Of course I do it too -- but rarely :-)

Don't recall what that was, but creating a new scope on each iteration sounds hard to explain in Python.
I don't think it's particularly hard to explain. For example, one way to explain it is to say that for i in <<stuff>>: body is equivalent to for <<hiddenvar>> in <<stuff>>: local i = <<hiddenvar>> body This explanation doesn't need to rest on recursion.

[Tim]
Don't recall what that was, but creating a new scope on each iteration sounds hard to explain in Python.
[Andrew Koenig]
Sorry, but as a Python programmer that explanation makes little sense to me. In effect, it pushes the mystery into what "local" is supposed to mean, but there's nothing _already_ in Python that acts the way you need "local" to act. Scope in Python is defined wrt "blocks", so you need to phrase this in terms of blocks, and there are very few kinds of blocks in Python's execution model: A block is a piece of Python program text that is executed as a unit. The following are blocks: a module, a function body, and a class definition. Each command typed interactively is a block. A script file (a file given as standard input to the interpreter or specified on the interpreter command line the first argument) is a code block. A script command (a command specified on the interpreter command line with the `-c' option) is a code block. The file read by the built-in function execfile() is a code block. The string argument passed to the built-in function eval() and to the exec statement is a code block. The expression read and evaluated by the built-in function input() is a code block. That's from section "Naming and binding" of the Python Reference Manual. I expect most Python programmers have "module, function, class ... plus some weird stuff I don't much care about" in mind. Python's execution model also has a one-to-one correspondence between active blocks and execution frames (see the rest of that section), which would need to be snapped to consider a finer-grained notion of block that didn't have its own execution frame. In short, it's only easy to define this in Python, without invoking nested functions, if you don't have Python's execution model in mind to begin with.

Tim Peters wrote:
Scheme has no loops in Python's sense -- things like "do" are shorthand for expressing stylized recursion
But it does have foreach and map, which are the moral equivalent of Python's for-loops and list comprehensions. The body is a lambda which takes the loop variable as a parameter, thus providing the extra level of scope. Recursion isn't needed.
When people talk about giving a Python for-loop vrbl its own scope, they generally don't mean a new scope on _each iteration_,
But that's exactly what you *do* need in order for a for-loop with a lambda in it to behave intuitively. If that's not what people mean, it's because they don't fully understand what they really mean to mean. :-) BTW, I'm not suggesting that a new stack frame gets allocated for every iteration -- all you need is a cell.
about the same as creating a nested block with its own autos in C.
Analogies with C aren't very helpful here, because it doesn't have closures, so it's only a matter of visibility, not lifetime.
creating a new scope on each iteration sounds hard to explain in Python.
But is it harder to explain than the reason someone's loop-with-a-lambda doesn't do what they expect? BTW, I wouldn't explain it by saying it creates a new scope, I'd say it creates a new binding on each iteration, or something like that. In my earlier proposal, you would actually say that explicitly, with something like for new i in range(10): ...
Abusing the default-argument machinery to capture current bindings is never necessary, and _is_ abuse.
But it's abuse that still happens, because although scoping has been fixed, other parts of the story are still missing. -- Greg

Aha! -- Thank you for jogging my memory. You seem to be right -- the problem is not that Python is lexically scoped, but that when you define a variable with =, it leaks out into the surrounding function scope. Here's an example: If True: y = 123 print y It may be obvious that this should print 123, but that's only because = combines properties of assignment and definition. In particular, if we were to write y = 42 if True: y = 123 print y it would be very surprising if this example were to print anything but 123. Here is a corresponding fragment in C++: int y = 42; if (true) { y = 123; } std::cout << y << "\n"; The "int" in the first line means that the variable y is being defined. Its lack in the third line means that y refers to a variable defined in an outer scope. So both instances of y here refer to the same variable, as they do in Python. But because definition and assignment are separated in C++, we can also write int y = 42; if (true) { int y = 123; } std::cout << y << "\n"; and the fragment will print 42. In this example, there are two distinct variables, both named y. So the problem, as I see it, is indeed that in Python there are suites that look to me as if they should define scopes, but don't. Indeed, if I write if (foo): y = 123 I can't even determine by inspecting the program whether y is defined at all. I might argue that y is always defined, by virtue of appearing before = somewhere in this scope, but the compiler tells me "name 'y' is not defined" if I try it, so I guess that's the right way to treat it. So here's how I understand what Greg was saying. Suppose I write x = [] for i in range(10): x.append(lambda:i) print [f() for f in x] This example will print [9, 9, 9, 9, 9, 9, 9, 9, 9, 9], which I think is wildly unintuitive. My intuition in this matter is partly formed by C++, but it is also formed by other languages going as far back as Algol 68. That intuition says that because the suite controlled by a "for" statement is executed any number of times, potentially including zero, it should be considered as its own scope, and any variables defined in that scope should stay there. In particular, the variable "i" should be defined in the scope of the "for", which implies that each time through the loop, the name "i" should be (re)bound to a different object. What surprises me even more is that if I try to define such a variable explicitly, it still doesn't work: x = [] for i in range(10): j = i x.append(lambda:j) print [f() for f in x] This example still prints [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]. If I understand the reason correctly, it is because even though j is defined only in the body of the loop, loop bodies are not scopes, so the variable's definition is hoisted out into the surrounding function scope. To convince myself of this behavior, I defined an extra function scope, the purpose of which is to localize j: x = [] for i in range(10): def foo(): j = i return lambda:j x.append(foo()) print [f() for f in x] Indeed, this example prints [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. The example also points up the fact that x.append(lambda:i) and def foo(): j = i return lambda:j x.append(foo()) behave differently, where my intuition (and, I suspect, many other people's as well) would be that they would be equivalent. Finally, I observe that this second example above is also equivalent to x.append(lambda i=i: i) which is what explains the fairly common idiom x = [] for i in range(10): x.append(lambda i=i:i) print [f() for f in x] So maybe what I meant when I asked for lexical scopes was two things: 1) Every indentation level should be a scope; 2) In general, variable definitions should not leak into surrounding scopes. I realize that (2) is too simplistic. Someone who writes if x < 0: y = -x else: y = x will expect y to be defined in the scope surrounding the "if" even if it was not already defined there. On the other hand, I think that the subtle pitfalls that come from allowing "for" variables to leak into the surrounding scopes are much harder to deal with and understand than would be the consequences of restricting their scopes as outlined above.

Andrew Koenig wrote:
That is my point: to me, it's counter-intuitive just like the infamous "except NameError, TypeError". I believe that names in lambdas/nested-functions referring to local names in the outer scope should really be bound at function definition time (much like default arguments are).
Yes. And by itself, I like this fact because it's very handy in many cases. And it's also handy that the iteration variable of the for loop is accessible after the for loop is terminated (in fact, this specific behaviour is already listed among the wont-change for Py3k).
As I said, to me there's nothing wrong with the way Python variables leak out of the suites; or, in other words, with the fact that Python has only two namespaces, the function-local and the global namespace. What I don't like is that the lookup of lambda's names are fully deferred at execution time. This behaviour is already not fully followed for local variables in functions, since:
which means that Python users *already* know that a variable is not really looked up only at run-time, but there's "something" going on even at function definition time. I don't see anything wrong if lambdas (or nested scopes) did the same for names provably coming from the outer scope. -- Giovanni Bajo

Giovanni Bajo wrote:
No, you don't want that, because it would make functions that call each other very awkward to arrange.
I'd just like to point out that the create-a-new-cell behaviour that I have proposed for loop variables *preserves* this ability! for new i in range(10): ... print i will still print 9. -- Greg

Giovanni Bajo wrote:
If you'd like each function instance to have a separate closure scope, then *give* each function a separate closure scope, instead of making them all share the same one the way you have above:
Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On Fri, 30 Jun 2006, Andrew Koenig wrote:
I think this property does apply. In your example:
It is already known at compile time that the "return x" in g() refers to an 'x' in the outer scope. 'x' cannot be a local variable to g() because there are no statements in g() that bind 'x'. Regardless of whether the binding itself exists yet, you (the reader) and the compiler can know which scope to look in for the binding at runtime. Have i understood your desired property correctly? -- ?!ng

Guido van Rossum wrote:
I believe the essence of their request for lexical scope boils down to allowing rebinding. Such code like the following is legal in Scheme: def f(x): def incr(): x = x + 1 return x def decr(): x = x - 1 return x return (incr, decr) (incr, decr) = f(1) print incr() # 2 print incr() # 3 print decr() # 2 print decr() # 1 -- FWIW, the Scheme equivalent would be something like: (define f (lambda (x) (list (lambda () (set! x (+ x 1)) x) (lambda () (set! x (- x 1)) x)))) (let ([fs (f 1)]) (let ([incr (car fs)] [decr (cadr fs)]) (display (incr)) (newline) ; 2 (display (incr)) (newline) ; 3 (display (decr)) (newline) ; 2 (display (decr)) (newline))) ; 1 As a more personal aside, I can't imagine where I would use this in any python program I have ever wrote. I actually never noticed that rebinding wasn't allowed until recently. -- Scott Dial scott@scottdial.com scodial@indiana.edu

[Andrew Koenig]
I saw messages out of sequence and did not realize that this would be a change in behavior from 2.4. Sigh.
[Ka-Ping Yee]
Yes, this is not a good time to change it.
I hope Py3000 has lexical scoping a la Scheme...
Me too -- that would be really nice.
[Guido]
That's not a very constructive proposal (especially since I don't know Scheme). Perhaps you could elaborate on what needs to change?
It's effectively the opposite of Python <0.1 wink>: a name is local to a scope in Scheme if and only if a declaration says it is. For example, the "let" family of forms is often used for this purpose, and (let ((x 2) (y 3)) # declares x and y as local to this `let`, and gives initial values (let ((x 7) (z (+ x y))) # x comes from outer `let` so is 2, and z is 2+3=5 (* z x))) # x comes from inner `let`, so this is 5*7=35 If you use `let*` instead of `let` in the inner one, z picks up the inner x=7, so that z is 7+3=10, and the result is 7*10 = 70 instead. The bindings in a `let` "happen" in an undefined order. In `let*`, a binding is visible "to its right" within the `let*`. Then there's `letrec`, which allows establishing mutually recursive bindings. While the `let` family is entirely about declaration, there are lots of other forms that mix in some declaration as part of their purpose (for example, the names in a lambda's argument list are local to the lambda's body), but they're all explicit in Scheme. I read "a la Scheme" here as "actually nothing like Scheme, except I want a non-tricky way to rebind a name from an enclosing scope within an enclosed scope". In Scheme, the scope a name x belongs to is found by searching enclosing scopes until you hit the first with an explicit "x belongs to me" declaration (OK, there's a hokey fallback to "top level" definitions too). Searching "textually up" always suffices (there's no form of delayed declaration -- a name must be declared before use). Scheme's assignment of assignment: (set! variable expression) has nothing to do with establishing the scope of `variable`.

On 6/30/06, Andrew Koenig <ark@acm.org> wrote:
Then your example def f(): return x x = 42 print f() is entirely well-defined -- x is a global and the compiler in fact generates code that benefits from knowing that it's not a local. Python knows which locals there are; also which locals there are in surrounding function scopes. It *could* also know which globals and builtins there are, except the language currently allows dynamic rebinding of module-level variables so that they replace builtins. E.g. def f(): return len([]) print f() # prints 0 def len(x): return "booh" print f() # prints "booh" del len print f() # prints 0 again Worse, instead if explicitly overriding len in the module, it could have been an assignment to __main__.len in some other module. We've been thinking on how to deal with this for years, since nobody really likes it in all its freedom. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

[Andrew Koenig]
Local names are always determined at compile-time in Python. What you can't always determine is whether a non-local (to any enclosing scope) name will end up getting resolved from the module globals or from __builtin__. The runtime error in: def f(): y = x x = 1 f() doesn't occur because Python doesn't know "x" is local to "f" at compile-time (it does know that), it's because Python's compiler doesn't do any flow analysis to detect potential use-before-definition. Instead the runtime initalizes locals to a special "not bound yet" value that the LOAD_FAST (really "load local") opcode special-cases. Note that this is quite unlike Scheme, in which declaration must appear before use (ignoring fancy letrec cases), and declaration must also supply an initial binding (Scheme has no "unbound local" problem because there's no way to create an uninitialized local).

Tim Peters wrote:
Note that this is quite unlike Scheme, in which declaration must appear before use (ignoring fancy letrec cases),
I think that's overstating things a bit -- mutually recursive functions are quite easy to write in Scheme and don't look at all "fancy" (unless you object for some reason to using (define ...)).
That much is true. -- Greg

[Tim Peters]
Note that this is quite unlike Scheme, in which declaration must appear before use (ignoring fancy letrec cases),
[Greg Ewing]
I think that's overstating things a bit --
So do I :-), but I don't really care about Scheme here.
In this context, yes, I object to using "define", because the semantics of internal definitions are defined in terms of an equivalent (letrec ...) form. The "fancy" gimmick is that letrec views all its bindings as occurring simultaneously, so strains a natural, linear understanding of "no use before declaration". But none of this appears to have any relevance to Python, so I'm happiest _here_ just calling that "fancy" and ignoring the details. Ditto "top level" definitions, which have unique rules of their own.

On Fri, 30 Jun 2006 00:05:10 -0700, Neal Norwitz <nnorwitz@gmail.com> wrote:
Please add #1494314 to the list. http://sourceforge.net/tracker/index.php?func=detail&aid=1494314&group_id=5470&atid=105470 Jean-Paul

On Jun 30, 2006, at 3:05 AM, Neal Norwitz wrote:
If there are any bugs you think should be considered show stoppers, mail them to the list and I will update the PEP.
I just submitted http://python.org/sf/1515169 for the ImportWarning issue previously discussed here. IMO it's important. James

James Y Knight wrote:
At the moment (i.e. without an acceptable alternative implementation) it's primarily a policy issue. There really isn't any bug here; (to speak with Microsoft's words): This behavior is by design. Only the release manager or the BDFL could revert the feature, and Guido already stated that the warning stays until Python 3, and probably even after that. I personally believe the only chance to get this changed now is a well-designed alternative implementation (although this is no promise that such an alternative would actually be accepted). Regards, Martin

On Saturday 01 July 2006 05:19, Martin v. Löwis wrote:
given the number of people and ways that this can emit a spurious warning, I think it should be reverted for 2.5. At _best_ we could maybe have a new -W switch to make it be generated, but this should be off by default. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter wrote:
Last line of warnings.py Copy, paste, s/PendingDeprecationWarning/ImportWarning -Wd to enable it again :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Neal Norwitz schrieb:
Neal (and/or Anthony), I would like to ask about the possibility to add some improvements to ctypes in Python 2.5, although the feature freeze is now in effect. Hopefully former third-party libraries can have the freeze relaxed somewhat;-). I intend to do these changes, the first is a small and trivial one, but allows a lot of flexibility: - Remove the restriction that the argtypes attribute of foreign functions must be ctypes types. Instead they are only required to implement a .from_param class method. The advantage is that custom objects can be used as function parameters. One usecase is to allow numpy arrays as function parameters without any conversion - this change at least allows to code this in Python. The patch is attached as from_param.patch. The second one is more involved, and not yet complete. I can post the patch or a link to it for review when it is implemented completely: - Implement the __array_struct__ attribute as describes by the numpy pep at http://numpy.scipy.org/array_interface.html. The properties needed to implement the __array_struct__ attribute could be calculated from a given ctypes array type, however, it would be more efficient to calculate them at type creation time. This requires the StgDSictObject that holds information about the ctypes type to grow a few fields: 'int nd' - contains the number of dimensions, 'char typekind' - a struct-like character for the item type, and 'Py_intptr_t *shape' - an array of size 'nd' containing shape information. Thanks for investigating this, Thomas Index: _ctypes.c =================================================================== RCS file: /cvsroot/ctypes/ctypes/source/_ctypes.c,v retrieving revision 1.340 diff -u -r1.340 _ctypes.c --- _ctypes.c 22 Jun 2006 19:21:28 -0000 1.340 +++ _ctypes.c 4 Jul 2006 09:56:06 -0000 @@ -1633,9 +1633,8 @@ for (i = 0; i < nArgs; ++i) { PyObject *tp = PyTuple_GET_ITEM(ob, i); - StgDictObject *dict = PyType_stgdict(tp); PyObject *cnv = PyObject_GetAttrString(tp, "from_param"); - if (!dict || !cnv) + if (!cnv) goto argtypes_error_1; PyTuple_SET_ITEM(converters, i, cnv); }

Thomas Heller wrote:
[...] I'd just like to provide a bit of context for Thomas' request (disclaimer: he did NOT ask me to write this, nor did anyone else). I understand the release managers' need to be strict with the freeze, but perhaps knowing what's behind this particular change will help them make a more informed decision. Numpy (http://numpy.scipy.org/) is the new python array package for numerical computing, which has been developed at enormous effort by Travis Oliphant (with community help) over the last year, as a way to unify the old Numeric package (written by Jim Hugunin, of Jython and IronPython fame) and Numarray (written by the Hubble telescope team). The effect of numpy in the community, even in its current pre-1.0 form, has been tremendous. There is a real, pressing need in the scientific world for open source and technically superior replacements to Matlab and IDL, the propietary 800-lb gorillas of the field. Many major research institutions across the world are seriously looking at python as fulfilling this role, but the previous situation of a divided library (Numeric/numarray) was keeping a lot of people on the fence. With Travis' effort and numpy maturing towards a 1.0 release right around the time of python 2.5, a LOT of people have come out of the woodwork to contribute code, ideas, documentation, etc. There is a real sense that the combination of python2.5 (with better 64-bit and __index__ support) and numpy will provide a significant advancement for scientific computing with modern, high-level tools. In this particular community, the need to interface with low-level existing libraries is probably much more common than in other fields. There are literally millions of lines of C/C++ code for scientific work which we have to use efficiently, and this is an everyday need for us. While there are a number of tools for this (SWIG, Boost::Python, pyrex, scipy.weave,...), very recently people have discovered how useful ctypes can be for this task. One of the Google SoC projects (http://2006.planet-soc.com/blog/140) started trying to wrap libsvm with SWIG and a week of frustrated efforts led nowhere. Albert then discovered ctypes and in a few hours was up and running. This has generated a lot of interest in the numpy crowd for ctypes, and people would really, really like to see python2.5 come 'out of the box' with as solid a support as possible from ctypes for numpy array interfacing. Ultimately the decision is up to the release team, I know that. But at least with this info, I hope you can understand: 1. why this is important to this community 2. why the timing isn't ideal: it is only /very/ recently that the numpy team 'discovered' how much ctypes could truly help with a necessary (and often very unpleasant) task in the numerical/python world. Thanks for reading, f

On 7/4/06, Thomas Heller <theller@python.net> wrote:
Ok, former third-party libraries get a 1e-308 reprieve. :-) I think the general answer is to plan some way that you could make an external release to support the other communities like numpy. I don't know the details, but I believe Barry does this with email. This would allow Python to be more stable, but allow additional features for the communities that are interested in installing an additional ctypes release.
This patch is ok. I will add comments to the patch on SF.
This is too much to add to a beta. Perhaps you could work on a branch to add these features and integrate the branch back in after 2.5 has been released. You could make an external release from the branch. n

Neal Norwitz wrote:
http://www.python.org/sf/1519018 I believe this regression in syntax checking needs to be addressed. -- Scott Dial scott@scottdial.com scodial@indiana.edu

On Fri, 30 Jun 2006, Neal Norwitz wrote:
The current list of serious bugs are in the PEP:
Among them is this one: Incorrect LOAD/STORE_GLOBAL generation http://python.org/sf/1501934 The question is, what behaviour is preferable for this code: g = 1 def f(): g += 1 f() Should this raise an UnboundLocalError or should it increment g? (Or, in other words, should augmented assignment be considered a local binding like regular assignment, or not?) -- ?!ng

[Ka-Ping Yee, on http://www.python.org/dev/peps/pep-0356/ ]
Of course it should, since that's the way it _is_ treated in all released Pythons, and there was no intent to change the semantics (let alone a FutureWarning in 2.4 to alert users that the meaning was going to change in 2.5). The Reference Manual also makes no distinction between assignment statements and augmented assignment statements in any relevant respect. The change in behavior here in 2.5 is plainly a bug (although someone may want to argue "it should" be different in P3K).

Ka-Ping Yee <python-dev@zesty.ca> writes:
I didn't think there was any question: this change in behaviour from 2.4 is just an accidental change, a bug that should be fixed. If you want to elevate it to feature status, I think we've missed the freeze :) (and also, I oppose the idea). Cheers, mwh -- If I didn't have my part-time performance art income to help pay the bills, I could never afford to support my programming lifestyle. -- Jeff Bauer, 21 Apr 2000

Ping> The question is, what behaviour is preferable for this code: Ping> g = 1 Ping> def f(): Ping> g += 1 Ping> f() If you treat "g += 1" as "g = g + 1" then it should create a local variable with a value of 2. There being no global statement in f() it must not modify the global variable. Skip

On Fri, 30 Jun 2006, Andrew Koenig wrote:
I saw messages out of sequence and did not realize that this would be a change in behavior from 2.4. Sigh.
Yes, this is not a good time to change it.
I hope Py3000 has lexical scoping a la Scheme...
Me too -- that would be really nice. -- ?!ng

On 6/30/06, Ka-Ping Yee <python-dev@zesty.ca> wrote:
That's not a very constructive proposal (especially since I don't know Scheme). Perhaps you could elaborate on what needs to change? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On Fri, 30 Jun 2006, Guido van Rossum wrote:
Sorry! I should have been more clear. Right now, Python supports the existence of nested scopes: a = 3 def f(): a = 4 def g(): a = 5 print a # prints 5 g() print a # prints 4 f() print a # prints 3 The above example shows that there are three distinct scopes, and that each one has a distinct binding named 'a' -- assigning to one doesn't affect the others. a = 3 def f(): b = 4 def g(): c = 5 print a, b, c # i can see all three g() f() The above example shows that all of the scopes can be *read*. But in today's Python, not all of the scopes can be *written*. a = 3 def f(): b = 4 def g(): c = 5 a, b, c = 0, 1, 2 # changes local c, not outer a and b g() f() The code in g() can affect its own local, 'c', and it can affect the global variable 'a' if it declares 'global a', but no matter what you write in g(), it cannot assign to 'b' (or to any other intermediate scope). This is a strange limitation and it would be nice to remove it. The asymmetry comes from Python having one foot in the new paradigm of nested lexical scopes and one foot still in the older paradigm of only two scopes, local and global. Most other languages that support lexical scoping (including Scheme, JavaScript, Ruby, Perl, E, Java, Smalltalk) provide a uniform way to read and write to scopes at all levels. This is done by letting programmers specify the scope in which they want a variable bound (usually with a keyword like "var" in JavaScript, "my" in Perl, or "define" in E). So here are some thoughts on how Python might be adjusted to support this. I'm not saying these would be the only ways, but at least they're some ideas to start with. In JavaScript, the "var" keyword is required whenever you want to declare a local variable. Anything without "var" is assumed to be a global name. The cleanest and most consistent solution that comes to mind would be to adopt exactly this for Python. Without "var": a = 3 # global def f(): b = 4 # global def g(): c = 5 # global a, b, c = 0, 1, 2 # changes all three globals g() f() print a, b, c # prints 0, 1, 2 With "var": var a = 3 # global def f(): var b = 4 # local to f def g(): var c = 5 # local to g a, b, c = 0, 1, 2 # changes outer a, outer b, and c print c # prints 2 g() print b # prints 1 f() print a # prints 0 print b # no such name print c # no such name But that is a big change. Perhaps it would be too unpythonic to suddenly require declarations for all local variables. So an alternative would be to retain the default assumption that undeclared variables are local. Here's what we'd get: Without "var": a = 3 def f(): b = 4 def g(): c = 5 a, b, c = 0, 1, 2 # changes local c, not outer a and b g() f() With "var": var a = 3 def f(): var b = 4 def g(): var c = 5 a, b, c = 0, 1, 2 # changes outer a, outer b, and c g() f() Now i think this is a little bit weird, because the statement "var b = 4" in an outer scope changes the meaning of "b" in an inner scope. But it does have the virtue of retaining behaviour compatible with today's Python, while offering a way to get proper lexical scopes for those who want to use them. Thoughts? Other ideas? -- ?!ng

Ka-Ping Yee <python-dev@zesty.ca> wrote: [snip lexical scoping option]
Using a keyword in an outer scope to state that a variable could be used in a nested scope is counter to the current method for accessing a parent scope with 'global'. Using 'var' as the equivalent of 'global', only for nested scopes, would be a more reasonable approach. However, I'm -1 on the feature now, for the same reasons I've been -1 on the feature for the last 2 times it has come up. In many of the cases where lexically nested scopes have been used to solve problems in Python, and programmers have run into a 'limitation' where not being able to modify a value in a parent scope has hindered them, the problem could have been better solved with another method that was more readable, more extensible, etc. What I asked before, and what I'd like to ask again, is if there are any _nontrivial uses_ of lexically nested scopes which are made cumbersome by our inability to write to parent scopes. If there aren't, then I'm going to again have to argue against new syntax, keywords, and their use. If there are, then we'll see how compelling such uses are. - Josiah

Josiah Carlson wrote:
The trouble with taking that position is that the very cases which would benefit are very *simple* ones, where it would be cumbersome to refactor it to use a class, or mutable object in the outer scope, etc. So you've effectively set up your acceptance criteria to be unmeetable.
If there aren't, then I'm going to again have to argue against new syntax, keywords, and their use.
There's one very simple way we could do this in Py3k without requiring any new syntax or keywords: just redefine the meaning of "global" to mean "not local". -- Greg

Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
If the only code that benefits from such changes are "very *simple*", then I think that says something about its necessity. That is, if anything more complicated than those that are "very *simple*" generally don't benefit, then I don't believe that such a modification would be beneficial to the language overall. Further, a simple namespace factory can handle much of the current issues, without needing to create or change keywords. def namespace(**kwds): class namespace(object): __slots__ = kwds.keys() def __init__(self): for i,j in kwds.iteritems(): setattr(self, i,j) return namespace() def trivial_counter(start): ns = namespace(current=start-1) def next(): ns.current += 1 return ns.current return next Maybe a variant of the above namespace factory should make it into the collections module.
I would probably be a solid -0 on such a proposal; I still don't think it's really necessary, but I've never used (or really seen) global more than one level deep, so would guess its impact would be low. - Josiah

On 7/1/06, Josiah Carlson <jcarlson@uci.edu> wrote:
This has been discussed at length in the following thread that I started in February and at least one time before that. http://mail.python.org/pipermail/python-dev/2006-February/061568.html I think using the "global" keyword is probably the lowest impact form and has the least amount of backwards incompatibility. Below is the part of the last thread that I talked about changing the meaning of "global." http://mail.python.org/pipermail/python-dev/2006-February/061852.html Having the "global" keyword semantics changed to be "lexically global" would break in the cases that "global" is used on a name within a nested scope that has an enclosing scope with the same name. I would suppose that actual instances in real code of this would be rare. Consider:
Under the proposed rules:
* f() *2
PEP 227 also had backwards incompatibilities that were similar and I suggest handling them the same way by issuing a warning in these cases when the new semantics are not being used (i.e. no "from __future__"). Most people probably think that this is a low impact "wart" on the Python language that not really worth fixing as there are workarounds (i.e. mutable objects) or other ways to express (i.e. use classes) such things, but it does trip people up from time to time as warts typically do--I guess that's why this gets brought up now and again. Best regards, Almann -- Almann T. Goo almann.goo@gmail.com

What about doing something similar to how import was changed? .a = 5 # this scope (self might be too magical ..a = 3 # up one scope ...a # up three Of course, this looks ... perhaps a bit strange. Also, counting is a bother. //Simon

I'd rather see a simpler rule: = never defines a variable in a surrounding scope. If you want to affect the binding of such a variable, you have to define it explicitly in the scope in which you want it. Example: x = 42 def f(): x = 123 # rebinds x as defined above y = 123 # defines local variable f() print x # prints 123 print y # error -- y not defined Yes, I know that rule is too simplistic. But I think I'd still prefer it to the way things are now.

On Sat, 1 Jul 2006, Andrew Koenig wrote:
I agree with you that this is a nicer and more consistent rule. What do you think of the proposal for a keyword to say "don't rebind"? It would achieve the same distinction you're aiming for above, but without the drastic incompatibility with today's Python. This has been previously discussed as "change the meaning of 'global' to mean 'not local'": http://mail.python.org/pipermail/python-dev/2006-February/061568.html http://mail.python.org/pipermail/python-dev/2006-July/066908.html I support this proposal, though i would prefer a clearer keyword such as "outer x" or "nonlocal x". If we can't agree on another keyword (or can't afford to spend one more keyword), i'm willing to support "global" for this purpose. -- ?!ng

Josiah Carlson wrote:
If the only code that benefits from such changes are "very *simple*", then I think that says something about its necessity.
The point is that they're only "very simple" if you can write them using access to an outer scope. Without that ability, they become less simple, less efficient, more convoluted, harder to follow, etc. Also I don't buy the argument that something has to be useful for big, complicated things in order to be worth having in the language. -- Greg

Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
As is known and has been stated, assigning to a parent scope can be emulated in various ways, either through an explicit namespace object, or through a namespace list.
I never claimed that something needed to be useful for "big, complicated things" in order to be worth having in the language. To be explicit, if nontrivial code isn't improved, that doesn't necessarily mean that the feature is useless. However, if the feature is really only useful for generally trivial cases *without* the feature, then making them even more trivial, I think, is a bit of over optimization. - Josiah

On 7/3/06, Josiah Carlson <jcarlson@uci.edu> wrote:
And the fact that this desire and need remains, even amongst people who should know better, suggests that it may be worth supporting it more directly, as the current work-arounds ain't pretty. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

"Guido van Rossum" <guido@python.org> wrote:
Perhaps not pretty, but not wholly ugly either. Or expressed another way, it's a wart, but the wart isn't 1" across on a forehead, it's fairly small and tucked away on an elbow. I had hoped that there would be a response to my second (and I believe more applicable statement); "if the feature is really only useful for generally trivial cases *without* the feature, then making them even more trivial, I think, is a bit of over optimization." As for a solution, I find the "global means 'not local'" proposition is the least undesireable of the possibilities. It suffers from a change in semantics and potential name masking issues, but I don't believe these are any more serious than normal global masking for the former, and the latter is solvable with a __future__ (at least for 2.6). I'm a solid -0 on this particular proposition, which is far better than the -1 I am on all of the other recent lexical scoping propositions. - Josiah

Josiah Carlson wrote:
It really depends on how common the trivial case is. In other words, multiply the savings for each occurance times the number of occurances. (Unfortunately, I don't know what units to measure said savings in - is there a unit of 'mental disconnect' or unintuitiveness? :) In an idealy world, the language would allow everything to be said in the most comprehensible way possible. Longer and more verbose ways of stating something are at an inherent disadvantage in this, simply because of the time it takes to scan and absorb the information by the human brain. However, losing excess syntax has to be done in a way that doesn't also lose information. Highly compressed representations of a concept may require such a level of abstraction that it is as much work to puzzle out their meaning as it would be to read the longer version and more. To put it another way - I am an advocate of applying Claude Shannon's theory of information to language design. The highest level of compression should be used for expressions that occur the most frequently.
I'd say that the more common case is where you want global to really mean global - that is, you want to be able to write to some module-level variable, regardless of how deeply nested your function scope is. While being able to access the 'next outer scope' is occasionally useful, it's not all that common. So changing the behavior of 'global' in this case would be both confusing (since it no longer means 'global'), and less useful (because it doesn't match the most common case.) (This assumes that I haven't completely understood the meaning of the phrase 'not local' - I assumed that it means 'not defined in this scope') Of course, the reason why it's not all that common may be because of the fact that it's not as easy to do, and so people tend to (consciously or otherwise) avoid that pattern in their designs. That being said, I don't think that's necessarily such a bad thing. Python isn't Scheme, and the scoping rules of Python are IMHO more oriented towards practicality and common sense than theoretical purity. This is why I'm not bothered by the fact that Python doesn't create a new scope for loop statements and such. Most of the time, this is what you want. It does mean that you need to name all of your variables uniquely, but that's good programming style in any case. The same is true for local variables not needing to be specially declared as 'my' or 'var' - most of the time, a local variable is what you want. On the other hand, the thing about theoretical purity is that it can be so mouth-wateringly powerful at times. For example, a language that supports closures is, IMHO, at least twice as powerful as a language that doesn't -- because you can use them in so many different and interesting ways. OK, so about the lexical scoping issue - let me brainstorm a moment: One idea would be to introduce the keyword 'local' which would have the effect of capturing any 'global' statements in any enclosing scope. So for example: f = 1 def a(): local f f = 2 def b(): global f f = 3 So in this case, the 'global' statement, which would normally associate 'f' with the outermost (module-level) scope, would instead associate 'f' with the innermost 'local' declaration of that variable. So in the above example, assigning 3 to f assigns it to the middle scope, but does not affect the module-level definition. Admittedly, that's a bit confusing and also verbose, considering that you are not only adding an extra keyword, but also using two statements to specify the home of one variable. Another alternative would be a way to declare an explicitly scoped variable. Lets use the keyword 'my' to indicate this: f = 1 def a(): my f = 2 def b(): f = 3 In this case, what the 'my' statement is doing is indicating that this scope 'owns' the definition of 'f' -- in other words, the definition is hoisted out of any enclosed scopes. So again, in the above example, the innermost assignment will be to the definition of 'f' in the middle scope. What's interesting about this is that you can use the same method with globals: my f = 1 def a(): f = 2 a() print f # prints '2' So again, you are indicating that the global scope 'owns' the definition of 'f', and any enclosed scopes should use that definition, and not create their own. Of course, if you really *do* need to have your own version, you can always override the 'my' statement with another 'my' statement: my f = 1 def a(): my f = 2 a() print f # prints '1' The 'my' statement essentially changes the scoping rules for all variables of that name, within the defining scope and all enclosed scopes. Of course, you can also override this behavior using the 'global' statement, which does exactly what it does now - makes the reference global (i.e. module-level): my f = 1 def a(): global f f = 2 a() print f # prints '2' All right, I'm pretty happy with that. Brainstorming done. :) -- Talin

Talin wrote:
I believe the proposal in question would cause no net worsening in this information content, and may actually improve it slightly, due to allowing a few things to be written in a shorter and clearer way, while allowing the vast majority of existing things to be written in exactly the same way.
It would still mean that, except in the (expected to be *extremely* rare) case where you happened to have a variable with the same name assigned in some intermediate scope. Such cases would be easily fixed by renaming the intermediate variable -- using a name of shorter or equal length, if you like, to keep the information content up. :-)
So changing the behavior of 'global' in this case would be both confusing (since it no longer means 'global'),
An alternative would be to change the keyword as well, to something like 'outer', which would better match its semantics. But if that were done, I would argue for the *removal* of the existing 'global' keyword, which would then be almost completely redundant. This would break large amounts of existing code, however, and it's highly dubious whether that would be worth the small increase in pendantic accuracy, even in Py3k. We're not supposed to be *gratuitously* breaking things in Py3k, after all.
(This assumes that I haven't completely understood the meaning of the phrase 'not local' - I assumed that it means 'not defined in this scope')
Yes, the new meaning would be "in the next outermost scope where there is an assignment to this name, or the module scope if you get that far".
Yes, but I find it hard to regard being *forbidden* from assigning to intermediate scopes as something driven by practical need rather than just being a historical accident. Back when there were strictly two scopes, many people argued themselves blue in the face that this was actually a *good* thing, even if you didn't realise it, and that Python was doing you a favour by enforcing it. Eventually a measure of sanity prevailed, and we got something a lot more like traditional lexical scoping. But one remnant of the old system remained, like a vestigial organ -- the 'global' statement that reaches all the way out to the module scope, regardless of what exists in between. To someone used to lexical scoping in almost any other language that has it, this is a *very* strange and unintuitive thing. Looking back, I think the meaning of 'global' should have been redefined right then at the same time. That would have been the logical and consistent thing to do, and in my opinion would have resulted in a scoping model that was simpler, more useful and no less practical. The most theoretically pure thing would have been to change it to 'outer' at the same time, but that would have broken too much code, and would therefore not have been practical. See, I'm not immune to practicality arguments. :-)
One idea would be to introduce the keyword 'local' which would have the effect of capturing any 'global' statements in any enclosing scope.
That seems unnecessary to me. Or at least not necessary enough to be worth the extra complexity in the scoping model. -- Greg

Josiah Carlson wrote:
I don't think "trivial" is the right word to use here, since it implies something that's of so little importance that it can be ignored. But the simple cases are precisely the ones where this wart hurts the most, so we can't ignore them. Arguments that a feature is undesirable because this or that workaround exists seem like post-hoc justifications to me. Think about it the other way around -- if writing to outer scopes had been straightforward from the beginning, would you be arguing for the *removal* of that ability? Would it even have occurred to anyone to do such a thing? -- Greg

I'd like to inject an example that might help make this discussion more concrete. Consider the following function: def for_each(seq, f): for i in seq: f(i) I'm sure I've seen more than one instance of someone on comp.lang.python trying to do the equivalent of using a function such as this one to compute the sum of the elements of a sequence as follows: def sum(seq): result = 0 def accum(i): result += i for_each(seq, accum) return result and wonder why it doesn't work. Still odder, why it doesn't work and the following does: def sum(seq): result = [0] def accum(i): result[0] += i for_each(seq, accum) return result[0] Transforming the first definition of sum above into the second may be trivial, but only if you've encountered the technique before. Moreover, the first version of sum uses a technique that is more than 45 years old (!), as it was available to Algol 60 programmers.

On 7/3/06, Andrew Koenig <ark@acm.org> wrote:
Much though the Algol 60 tickles my nostalgia (it was my first programming language!) I don't think that it's a particularly strong argument. I like to think that we have better ways these days. I think you need to come up with a better motivating example; the above is particular un-idiomatic Python. It starts by defining a higher-order function for_each that has little to offer over writing an explicit for loop, and then uses this to motivate writing a simple operation (result += i) as a function instead so that it fits in the inconvenient for_each() API. I understand that both for_each() and accum() are just examples of more complicated functions, but I can't help thinking that the problem here only occurs for very *simple* functions in the place of accum(); a more complicated form of accum would likely be a bound method of a class instance which carries the state. A better way to decompose these kinds of problems is probably by using generators. The equivalent of for_each() would not take a function parameter but *yield* the successive values instead of calling f() with successive values; e.g.: def for_each(seq): for i in seq: yield i Then the sum() function could be written like this: def sum(seq): result = 0 for i in for_each(seq): result += i return result -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Even if so, that's not the point I was trying to make. The point is that there is a programming technique that is widely used, works in many languages, and has been around for 45 years; and when you try to use it in Python, it fails. I believe that such failures, even if there are alternative ways of solving the problems that engender them, are barriers to learning that should be removed if it is possible to do so without substantial cost.

On 7/3/06, Andrew Koenig <ark@acm.org> wrote:
That's true for lots of things that have been around for a long time. Can you provide a better example? (The use of += is not particularly relevant to the example; it could just as well have said "result = result + i".)
And that is of course the crucial question. Probably the only proposal that has any chance of succeeding is to extend the 'global' statement so that it also applies to variables in intermediate outer scopes; or perhaps a new keyword (since "global" is not a very good name for the extended semantics). We would have to decide what this example would do: def outer(): def inner1(x): global a a = x def inner2(): return a return inner1, inner2 f1, f2 = outer() g1, g2 = outer() f1(42) g1(0) print f2() # Does it print 0 or 42 ??? In current Python this prints 0: there's only one (global) variable a, and the call to g1(0) overrides the value that was stored by f1(42). If global were changed to mean "nonlocal" what should it do? The question the example poses is that a is not initialized except in inner1() -- we somehow have to decide whether this is an error, or whether it chooses some well-defined outer scope, with the choices being the nearest enclosing scope or the outermost (truly global) scope. We have one guideline: if there is exactly one outer scope that defines a variable named a, we would like it to be referenced (by the 'global a') statement and the variable references governed by it automatically. Also, of there's more than one such scope, we'd like it to reference the innermost one. But this doesn't have a natural extension to what should happen if there's no such scope! Perhaps the best solution would be to make it an error if there wasn't a visible variable named a in an outer scope. That would suit me fine because I'd like to migrate towards more static analysis of variables anyway. If that means equipping modues with traps for attempts to store arbitrary variables into their namespaces, that's fine with me (as long as there's some escape -- and of course instancs and classes remain fully dynamic). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Josiah> As for a solution, I find the "global means 'not local'" Josiah> proposition is the least undesireable of the possibilities. It Josiah> suffers from a change in semantics and potential name masking Josiah> issues... Pychecker and PyLint both already identify cases where builtins are masked by locals or module globals (and may identify cases where locals mask module globals - I don't recall). I suspect both could be generalized in this regard without a huge effort. That's probably the best place for this sort of warning. Skip

Maybe an object, like self, for referring to enclosing scopes? a = 3 def f(): b = 4 def g(): c = 5 outer.outer.a, outer.b, c = 0, 1, 2 # changes outer a, outer b, and c g() f() Chaining the keyword looks a little weird, but it is not often that you have to refer to variables in the enclosing scope of the enclosing scope. I have often wanted something similar to that for global variables, instead of the global declaration: cache = None def init(): if not global.cache: global.cache = init_cache() -- mvh Björn

Ka-Ping Yee wrote:
while offering a way to get proper lexical scopes for those who want to use them.
I don't disagree with anything you said, but I think it would be a good idea to avoid using phrases like "proper lexical scopes", which is likely to set people off on a tangent. The issue isn't lexicality, it's writeability. -- Greg

Ka-Ping Yee <python-dev@zesty.ca> wrote:
That's not the Python way, IMO. I think the right way (assuming we actually want to allow it) is to introduce a pure assignment statement in addition to the assignment/declaration statement that we already have. For example: a = 1 def f(): b = 2 a := 2 def g(): b := 3 print a, b, c g() f() would print "2 3 4". The := would assign but not declare a variable in the current scope. Neil

Neil Schemenauer wrote:
The := would assign but not declare a variable in the current scope.
There are other benefits to such a statement, too, since we can make it similar to other augmented assignments by letting the object being assigned to interfere with the process. a := 2 could translate to something like: a = a.__assign__(2) with the default behaviour of __assign__ simply being: def __assign__(rhs) return rhs This gives you: - runtime checking for typos (you can't accidentally declare a new variable with := when you really meant to assign to an existing one) - if/when control flow analysis is added to the AST compiler, it will be picked up as an error at compile time along with the other augmented assignments - the object being assigned to can validate/modify its replacement (e.g. automatically wrapping it in a weakref proxy, or checking that it has the correct type) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

That's not a very constructive proposal (especially since I don't know Scheme). Perhaps you could elaborate on what needs to change?
The fundamental principle is that the binding of every name is determined during compilation, not during execution. This property does not quite apply to Python at present. For example: x = 42 def f(): y = x x = 123 return y f() This example fails with "local variable 'x' referenced before assignment" because the compiler sees that f contains an assignment to x, so it makes x a local variable, and then when you try to assign x to y during execution, it fails. This behavior is consistent with the notion of lexical scoping. However, if I write def g(): return x x = 42 g() the result is 42. With lexical scoping, I believe it should be undefined. The reason is that when the compiler encounters the definition of g, variable x is not yet bound, and there is nothing in the body of g that would bind it. Therefore, g requires a binding to exist at the time it is compiled; because no such binding exists, this example would be an error (at compile time) under lexical scoping.

The trouble is that you don't necessarily know in what scope they will be defined, so I think that forcing you to be explicit about it is useful. Can you show me an example of where you think it isn't? Incidentally, I think that lexical scoping would also deal with the problem that people often encounter in which they have to write things like "lambda x=x:" where one would think "lambda x:" would suffice.

[Andrew Koenig]
They _shouldn't_ encounter that at all anymore. For example,
works fine in modern Pythons. Earlier Python's had a no-exceptions 3-scope implementation (local, global, builtin), and in those the "x" in the lambda body was "not local" (was either global or builtin, although the compiler couldn't tell which of those two it was). In _those_ Pythons people had to write "lambda x=x: x+1" instead, to suck the binding of the outer x into the lambda body, but if people are still doing that they're confused. Modern Pythons do have lexical scoping + global + builtin, although there's no way to spell "rebind a name local to an outer scope from within an inner scope".

[Giovanni Bajo]
So stay away from excruciating abuses of lexical scoping you don't understand ;-) What do you _expect_ `i` to refer to? "Oh, it should guess that I didn't really mean to defer evaluation of the lambda body at all, but instead evaluate the lambda body at the time I define the lambda and then synthesize some other function that captures the specific outer bindings in effect at lambda-definition time" doesn't really cut it. Try spelling what you think you want here in Scheme. Before it works, you'll probably end up with some equally "atrocious" (let ((i i)) ...) gimmick to force capturing each binding for `i` as it flies by. Else Scheme will also use the outer binding for `i` in effect at the time the lambdas are _executed_. This isn't typical use for lambda, and I don't think it's what Andrew had in mind.

[Giovanni Bajo]
[Tim Peters]
I think I understand what happens, I just don't know whether this can be "fixed" or not. Unless you are saying that the above behaviour is not only a complex side-effect the way things are, but the way things should be. Do you agree that it would be ideal if the above code generated range(10) instead of [9]*10, or you believe that the current behaviour is more sound (and if so, why)? As for actual implementing this change of semantic, the fact that `i` is a local variable in the outer scope (assuming it's all within a function), doesn't make it possible for Python to early-bound it, by realizing that, since `i` is not an argument of the lambda, and it's a local of the outer scope? At worse, couldn't Python do the "i=i" trick by itself when it sees that `i` is a local in the outer scope? Right now I can't think off-hand of a case in which this would break things. [Tim Peters]
This isn't typical use for lambda,
Yes, maybe it's not the most used idiom and Andrew wasn't referring to this, but it happens quite often to me (where 'often' means 'many times' among my rare usages of lambda). For instance, in GUI code, it's common to do things like: for b in self.buttons: self.setEventCallback(b, "clicked", lambda: self.label.setText("I pressed button %r" % b)) ... which of course won't work, as written above. Giovanni Bajo

"Giovanni Bajo" <rasky@develer.com> wrote in message news:027f01c69caf$b1c20450$d1b12997@bagio...
. Do you agree that it would be ideal if the above code generated range(10) instead of [9]*10,
No. You are trying to reify an optical illusion resulting from putting a constant function definition inside a loop. Making the meaning of 'def f(): return i' depend on the definition-time context by partially and variably evaluating the body would make code much harder to read and understand. Consider: if a: i=666 <intervening code> def f(): return i
It would make code more fragile. for i in range(666): print name[i] ... <intervening code> ... def total(num): return cost[item]*num Now someone decides first loop should have more expressive loop var name and changes the first line to for item in range(666): print name[item] and the meaning of total is completely changed. Adding such long-range coupling between language statements strikes me as a poor idea. Terry Jan Reedy

For Common Lispers and probably Schemers, Python has some surprising semantics around scope and lifetime extent of variables. Three that leap out at me are: * function parameters with default values are NOT new bindings for each invocation, so a default value of [] changes if you destructively modify this list object in the function * loop variables are NOT distinct lexical variables. The binding gloms on to a variable in the function's scope, both changing that lexical binding and not creating a new one for the loop (that goes away when the loop's scope ends) * loop variables are NOT distinct bindings per iteration, leading to the surprising results below Bill -----Original Message----- From: python-dev-bounces+billchi=microsoft.com@python.org [mailto:python-dev-bounces+billchi=microsoft.com@python.org] On Behalf Of Bill Janssen Sent: Friday, June 30, 2006 6:31 PM To: Giovanni Bajo Cc: Phillip J. Eby; Ka-Ping Yee; Guido van Rossum; Tim Peters; python-dev@python.org Subject: Re: [Python-Dev] 2.5 and beyond
Isn't this exactly what you'd expect? Maybe I've been writing Python for too long... :-). Bill _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/billchi%40microsoft.co m

"Bill Chiles" <billchi@microsoft.com> wrote:
One thing to remember is that Python is not Scheme/Lisp. It borrows some ideas from Scheme/Lisp, but that borrowing does not necessitate a it also use a completely equivalent scoping mechanism. From what I have been hearing about Python 2.6, and 3.0, the three "surprises" you describe are not going to be "fixed" (with respect to expected Scheme/Lisp semantics). Feel free to argue as to why they should be "fixed" in Py3k (unless Guido says, "you're dreaming"), but please do so in the py3k list. - Josiah

On 7/4/06, Josiah Carlson <jcarlson@uci.edu> wrote:
One thing to remember is that Python is not Scheme/Lisp. It borrows some ideas from Scheme/Lisp,
I can say it stronger. Any resemblance between Python and Scheme or Lisp is purely a coincidence. Neither language is in Python's ancestry, at least not explicitly; I'd never used or tried to learn Scheme when I started Python (still haven't) and my Lisp experience was limited to copying Emacs startup code from friends (still is). -- --Guido van Rossum (home page: http://www.python.org/~guido/)

"Giovanni Bajo" <rasky@develer.com> wrote in message news:020c01c69ca1$754c7310$d1b12997@bagio...
The 'subtle sematic' had nothing to do with lambda but with Python functions. The above is exactly equivalent (except the different .funcname) to a = [] for i in range(10): def f(): return i a.append(f) del f That should be equally confusing (or not), and equally requires the 'i=i' trick (or not). As is, either function definitiion is a constant and the loop makes useless duplicates. Either form would have the same effect is hoisted out of the loop. Terry Jan Reedy

Giovanni Bajo wrote:
This has *nothing* to do with the semantics of lambda! It's because Python's for-loop doesn't put its control variable in a new scope, the way Scheme's equivalent construct does. *That's* what needs to be addressed to fix this problem. I've made a suggestion about that before, but Guido rejected it, so I won't repeat it here. -- Greg

[Giovanni Bajo]
[Greg Ewing]
I don't think I follow that. Scheme has no loops in Python's sense -- things like "do" are shorthand for expressing stylized recursion, where each conceptual iteration gets a fresh set of "loop variables". When people talk about giving a Python for-loop vrbl its own scope, they generally don't mean a new scope on _each iteration_, they just mean that, e.g., i = 5 for i in range(10): # do stuff print i prints 5 intead of 9, about the same as creating a nested block with its own autos in C. The Scheme way is more like: i = 5 def step(i): # do stuff if i < 9: step(i+1) step(0) print i except with tail-recursion elimination. That also prints 5, but does a hell of a lot more than _just_ arrange for that.
Don't recall what that was, but creating a new scope on each iteration sounds hard to explain in Python. If Giovanni wants the Scheme way ;-), it's available: """ a = [] def step(i): a.append(lambda: i) if i < 9: step(i+1) step(0) print [x() for x in a] """ prints [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], although it's more sanely written in Python with a loop: """ def make_lambda(i): return lambda: i a = [] for i in range(10): a.append(make_lambda(i)) print [x() for x in a] """ Abusing the default-argument machinery to capture current bindings is never necessary, and _is_ abuse. Of course I do it too -- but rarely :-)

Don't recall what that was, but creating a new scope on each iteration sounds hard to explain in Python.
I don't think it's particularly hard to explain. For example, one way to explain it is to say that for i in <<stuff>>: body is equivalent to for <<hiddenvar>> in <<stuff>>: local i = <<hiddenvar>> body This explanation doesn't need to rest on recursion.

[Tim]
Don't recall what that was, but creating a new scope on each iteration sounds hard to explain in Python.
[Andrew Koenig]
Sorry, but as a Python programmer that explanation makes little sense to me. In effect, it pushes the mystery into what "local" is supposed to mean, but there's nothing _already_ in Python that acts the way you need "local" to act. Scope in Python is defined wrt "blocks", so you need to phrase this in terms of blocks, and there are very few kinds of blocks in Python's execution model: A block is a piece of Python program text that is executed as a unit. The following are blocks: a module, a function body, and a class definition. Each command typed interactively is a block. A script file (a file given as standard input to the interpreter or specified on the interpreter command line the first argument) is a code block. A script command (a command specified on the interpreter command line with the `-c' option) is a code block. The file read by the built-in function execfile() is a code block. The string argument passed to the built-in function eval() and to the exec statement is a code block. The expression read and evaluated by the built-in function input() is a code block. That's from section "Naming and binding" of the Python Reference Manual. I expect most Python programmers have "module, function, class ... plus some weird stuff I don't much care about" in mind. Python's execution model also has a one-to-one correspondence between active blocks and execution frames (see the rest of that section), which would need to be snapped to consider a finer-grained notion of block that didn't have its own execution frame. In short, it's only easy to define this in Python, without invoking nested functions, if you don't have Python's execution model in mind to begin with.

Tim Peters wrote:
Scheme has no loops in Python's sense -- things like "do" are shorthand for expressing stylized recursion
But it does have foreach and map, which are the moral equivalent of Python's for-loops and list comprehensions. The body is a lambda which takes the loop variable as a parameter, thus providing the extra level of scope. Recursion isn't needed.
When people talk about giving a Python for-loop vrbl its own scope, they generally don't mean a new scope on _each iteration_,
But that's exactly what you *do* need in order for a for-loop with a lambda in it to behave intuitively. If that's not what people mean, it's because they don't fully understand what they really mean to mean. :-) BTW, I'm not suggesting that a new stack frame gets allocated for every iteration -- all you need is a cell.
about the same as creating a nested block with its own autos in C.
Analogies with C aren't very helpful here, because it doesn't have closures, so it's only a matter of visibility, not lifetime.
creating a new scope on each iteration sounds hard to explain in Python.
But is it harder to explain than the reason someone's loop-with-a-lambda doesn't do what they expect? BTW, I wouldn't explain it by saying it creates a new scope, I'd say it creates a new binding on each iteration, or something like that. In my earlier proposal, you would actually say that explicitly, with something like for new i in range(10): ...
Abusing the default-argument machinery to capture current bindings is never necessary, and _is_ abuse.
But it's abuse that still happens, because although scoping has been fixed, other parts of the story are still missing. -- Greg

Aha! -- Thank you for jogging my memory. You seem to be right -- the problem is not that Python is lexically scoped, but that when you define a variable with =, it leaks out into the surrounding function scope. Here's an example: If True: y = 123 print y It may be obvious that this should print 123, but that's only because = combines properties of assignment and definition. In particular, if we were to write y = 42 if True: y = 123 print y it would be very surprising if this example were to print anything but 123. Here is a corresponding fragment in C++: int y = 42; if (true) { y = 123; } std::cout << y << "\n"; The "int" in the first line means that the variable y is being defined. Its lack in the third line means that y refers to a variable defined in an outer scope. So both instances of y here refer to the same variable, as they do in Python. But because definition and assignment are separated in C++, we can also write int y = 42; if (true) { int y = 123; } std::cout << y << "\n"; and the fragment will print 42. In this example, there are two distinct variables, both named y. So the problem, as I see it, is indeed that in Python there are suites that look to me as if they should define scopes, but don't. Indeed, if I write if (foo): y = 123 I can't even determine by inspecting the program whether y is defined at all. I might argue that y is always defined, by virtue of appearing before = somewhere in this scope, but the compiler tells me "name 'y' is not defined" if I try it, so I guess that's the right way to treat it. So here's how I understand what Greg was saying. Suppose I write x = [] for i in range(10): x.append(lambda:i) print [f() for f in x] This example will print [9, 9, 9, 9, 9, 9, 9, 9, 9, 9], which I think is wildly unintuitive. My intuition in this matter is partly formed by C++, but it is also formed by other languages going as far back as Algol 68. That intuition says that because the suite controlled by a "for" statement is executed any number of times, potentially including zero, it should be considered as its own scope, and any variables defined in that scope should stay there. In particular, the variable "i" should be defined in the scope of the "for", which implies that each time through the loop, the name "i" should be (re)bound to a different object. What surprises me even more is that if I try to define such a variable explicitly, it still doesn't work: x = [] for i in range(10): j = i x.append(lambda:j) print [f() for f in x] This example still prints [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]. If I understand the reason correctly, it is because even though j is defined only in the body of the loop, loop bodies are not scopes, so the variable's definition is hoisted out into the surrounding function scope. To convince myself of this behavior, I defined an extra function scope, the purpose of which is to localize j: x = [] for i in range(10): def foo(): j = i return lambda:j x.append(foo()) print [f() for f in x] Indeed, this example prints [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. The example also points up the fact that x.append(lambda:i) and def foo(): j = i return lambda:j x.append(foo()) behave differently, where my intuition (and, I suspect, many other people's as well) would be that they would be equivalent. Finally, I observe that this second example above is also equivalent to x.append(lambda i=i: i) which is what explains the fairly common idiom x = [] for i in range(10): x.append(lambda i=i:i) print [f() for f in x] So maybe what I meant when I asked for lexical scopes was two things: 1) Every indentation level should be a scope; 2) In general, variable definitions should not leak into surrounding scopes. I realize that (2) is too simplistic. Someone who writes if x < 0: y = -x else: y = x will expect y to be defined in the scope surrounding the "if" even if it was not already defined there. On the other hand, I think that the subtle pitfalls that come from allowing "for" variables to leak into the surrounding scopes are much harder to deal with and understand than would be the consequences of restricting their scopes as outlined above.

Andrew Koenig wrote:
That is my point: to me, it's counter-intuitive just like the infamous "except NameError, TypeError". I believe that names in lambdas/nested-functions referring to local names in the outer scope should really be bound at function definition time (much like default arguments are).
Yes. And by itself, I like this fact because it's very handy in many cases. And it's also handy that the iteration variable of the for loop is accessible after the for loop is terminated (in fact, this specific behaviour is already listed among the wont-change for Py3k).
As I said, to me there's nothing wrong with the way Python variables leak out of the suites; or, in other words, with the fact that Python has only two namespaces, the function-local and the global namespace. What I don't like is that the lookup of lambda's names are fully deferred at execution time. This behaviour is already not fully followed for local variables in functions, since:
which means that Python users *already* know that a variable is not really looked up only at run-time, but there's "something" going on even at function definition time. I don't see anything wrong if lambdas (or nested scopes) did the same for names provably coming from the outer scope. -- Giovanni Bajo

Giovanni Bajo wrote:
No, you don't want that, because it would make functions that call each other very awkward to arrange.
I'd just like to point out that the create-a-new-cell behaviour that I have proposed for loop variables *preserves* this ability! for new i in range(10): ... print i will still print 9. -- Greg

Giovanni Bajo wrote:
If you'd like each function instance to have a separate closure scope, then *give* each function a separate closure scope, instead of making them all share the same one the way you have above:
Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

On Fri, 30 Jun 2006, Andrew Koenig wrote:
I think this property does apply. In your example:
It is already known at compile time that the "return x" in g() refers to an 'x' in the outer scope. 'x' cannot be a local variable to g() because there are no statements in g() that bind 'x'. Regardless of whether the binding itself exists yet, you (the reader) and the compiler can know which scope to look in for the binding at runtime. Have i understood your desired property correctly? -- ?!ng

Guido van Rossum wrote:
I believe the essence of their request for lexical scope boils down to allowing rebinding. Such code like the following is legal in Scheme: def f(x): def incr(): x = x + 1 return x def decr(): x = x - 1 return x return (incr, decr) (incr, decr) = f(1) print incr() # 2 print incr() # 3 print decr() # 2 print decr() # 1 -- FWIW, the Scheme equivalent would be something like: (define f (lambda (x) (list (lambda () (set! x (+ x 1)) x) (lambda () (set! x (- x 1)) x)))) (let ([fs (f 1)]) (let ([incr (car fs)] [decr (cadr fs)]) (display (incr)) (newline) ; 2 (display (incr)) (newline) ; 3 (display (decr)) (newline) ; 2 (display (decr)) (newline))) ; 1 As a more personal aside, I can't imagine where I would use this in any python program I have ever wrote. I actually never noticed that rebinding wasn't allowed until recently. -- Scott Dial scott@scottdial.com scodial@indiana.edu

[Andrew Koenig]
I saw messages out of sequence and did not realize that this would be a change in behavior from 2.4. Sigh.
[Ka-Ping Yee]
Yes, this is not a good time to change it.
I hope Py3000 has lexical scoping a la Scheme...
Me too -- that would be really nice.
[Guido]
That's not a very constructive proposal (especially since I don't know Scheme). Perhaps you could elaborate on what needs to change?
It's effectively the opposite of Python <0.1 wink>: a name is local to a scope in Scheme if and only if a declaration says it is. For example, the "let" family of forms is often used for this purpose, and (let ((x 2) (y 3)) # declares x and y as local to this `let`, and gives initial values (let ((x 7) (z (+ x y))) # x comes from outer `let` so is 2, and z is 2+3=5 (* z x))) # x comes from inner `let`, so this is 5*7=35 If you use `let*` instead of `let` in the inner one, z picks up the inner x=7, so that z is 7+3=10, and the result is 7*10 = 70 instead. The bindings in a `let` "happen" in an undefined order. In `let*`, a binding is visible "to its right" within the `let*`. Then there's `letrec`, which allows establishing mutually recursive bindings. While the `let` family is entirely about declaration, there are lots of other forms that mix in some declaration as part of their purpose (for example, the names in a lambda's argument list are local to the lambda's body), but they're all explicit in Scheme. I read "a la Scheme" here as "actually nothing like Scheme, except I want a non-tricky way to rebind a name from an enclosing scope within an enclosed scope". In Scheme, the scope a name x belongs to is found by searching enclosing scopes until you hit the first with an explicit "x belongs to me" declaration (OK, there's a hokey fallback to "top level" definitions too). Searching "textually up" always suffices (there's no form of delayed declaration -- a name must be declared before use). Scheme's assignment of assignment: (set! variable expression) has nothing to do with establishing the scope of `variable`.

On 6/30/06, Andrew Koenig <ark@acm.org> wrote:
Then your example def f(): return x x = 42 print f() is entirely well-defined -- x is a global and the compiler in fact generates code that benefits from knowing that it's not a local. Python knows which locals there are; also which locals there are in surrounding function scopes. It *could* also know which globals and builtins there are, except the language currently allows dynamic rebinding of module-level variables so that they replace builtins. E.g. def f(): return len([]) print f() # prints 0 def len(x): return "booh" print f() # prints "booh" del len print f() # prints 0 again Worse, instead if explicitly overriding len in the module, it could have been an assignment to __main__.len in some other module. We've been thinking on how to deal with this for years, since nobody really likes it in all its freedom. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

[Andrew Koenig]
Local names are always determined at compile-time in Python. What you can't always determine is whether a non-local (to any enclosing scope) name will end up getting resolved from the module globals or from __builtin__. The runtime error in: def f(): y = x x = 1 f() doesn't occur because Python doesn't know "x" is local to "f" at compile-time (it does know that), it's because Python's compiler doesn't do any flow analysis to detect potential use-before-definition. Instead the runtime initalizes locals to a special "not bound yet" value that the LOAD_FAST (really "load local") opcode special-cases. Note that this is quite unlike Scheme, in which declaration must appear before use (ignoring fancy letrec cases), and declaration must also supply an initial binding (Scheme has no "unbound local" problem because there's no way to create an uninitialized local).

Tim Peters wrote:
Note that this is quite unlike Scheme, in which declaration must appear before use (ignoring fancy letrec cases),
I think that's overstating things a bit -- mutually recursive functions are quite easy to write in Scheme and don't look at all "fancy" (unless you object for some reason to using (define ...)).
That much is true. -- Greg

[Tim Peters]
Note that this is quite unlike Scheme, in which declaration must appear before use (ignoring fancy letrec cases),
[Greg Ewing]
I think that's overstating things a bit --
So do I :-), but I don't really care about Scheme here.
In this context, yes, I object to using "define", because the semantics of internal definitions are defined in terms of an equivalent (letrec ...) form. The "fancy" gimmick is that letrec views all its bindings as occurring simultaneously, so strains a natural, linear understanding of "no use before declaration". But none of this appears to have any relevance to Python, so I'm happiest _here_ just calling that "fancy" and ignoring the details. Ditto "top level" definitions, which have unique rules of their own.

On Fri, 30 Jun 2006 00:05:10 -0700, Neal Norwitz <nnorwitz@gmail.com> wrote:
Please add #1494314 to the list. http://sourceforge.net/tracker/index.php?func=detail&aid=1494314&group_id=5470&atid=105470 Jean-Paul

On Jun 30, 2006, at 3:05 AM, Neal Norwitz wrote:
If there are any bugs you think should be considered show stoppers, mail them to the list and I will update the PEP.
I just submitted http://python.org/sf/1515169 for the ImportWarning issue previously discussed here. IMO it's important. James

James Y Knight wrote:
At the moment (i.e. without an acceptable alternative implementation) it's primarily a policy issue. There really isn't any bug here; (to speak with Microsoft's words): This behavior is by design. Only the release manager or the BDFL could revert the feature, and Guido already stated that the warning stays until Python 3, and probably even after that. I personally believe the only chance to get this changed now is a well-designed alternative implementation (although this is no promise that such an alternative would actually be accepted). Regards, Martin

On Saturday 01 July 2006 05:19, Martin v. Löwis wrote:
given the number of people and ways that this can emit a spurious warning, I think it should be reverted for 2.5. At _best_ we could maybe have a new -W switch to make it be generated, but this should be off by default. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.

Anthony Baxter wrote:
Last line of warnings.py Copy, paste, s/PendingDeprecationWarning/ImportWarning -Wd to enable it again :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Neal Norwitz schrieb:
Neal (and/or Anthony), I would like to ask about the possibility to add some improvements to ctypes in Python 2.5, although the feature freeze is now in effect. Hopefully former third-party libraries can have the freeze relaxed somewhat;-). I intend to do these changes, the first is a small and trivial one, but allows a lot of flexibility: - Remove the restriction that the argtypes attribute of foreign functions must be ctypes types. Instead they are only required to implement a .from_param class method. The advantage is that custom objects can be used as function parameters. One usecase is to allow numpy arrays as function parameters without any conversion - this change at least allows to code this in Python. The patch is attached as from_param.patch. The second one is more involved, and not yet complete. I can post the patch or a link to it for review when it is implemented completely: - Implement the __array_struct__ attribute as describes by the numpy pep at http://numpy.scipy.org/array_interface.html. The properties needed to implement the __array_struct__ attribute could be calculated from a given ctypes array type, however, it would be more efficient to calculate them at type creation time. This requires the StgDSictObject that holds information about the ctypes type to grow a few fields: 'int nd' - contains the number of dimensions, 'char typekind' - a struct-like character for the item type, and 'Py_intptr_t *shape' - an array of size 'nd' containing shape information. Thanks for investigating this, Thomas Index: _ctypes.c =================================================================== RCS file: /cvsroot/ctypes/ctypes/source/_ctypes.c,v retrieving revision 1.340 diff -u -r1.340 _ctypes.c --- _ctypes.c 22 Jun 2006 19:21:28 -0000 1.340 +++ _ctypes.c 4 Jul 2006 09:56:06 -0000 @@ -1633,9 +1633,8 @@ for (i = 0; i < nArgs; ++i) { PyObject *tp = PyTuple_GET_ITEM(ob, i); - StgDictObject *dict = PyType_stgdict(tp); PyObject *cnv = PyObject_GetAttrString(tp, "from_param"); - if (!dict || !cnv) + if (!cnv) goto argtypes_error_1; PyTuple_SET_ITEM(converters, i, cnv); }

Thomas Heller wrote:
[...] I'd just like to provide a bit of context for Thomas' request (disclaimer: he did NOT ask me to write this, nor did anyone else). I understand the release managers' need to be strict with the freeze, but perhaps knowing what's behind this particular change will help them make a more informed decision. Numpy (http://numpy.scipy.org/) is the new python array package for numerical computing, which has been developed at enormous effort by Travis Oliphant (with community help) over the last year, as a way to unify the old Numeric package (written by Jim Hugunin, of Jython and IronPython fame) and Numarray (written by the Hubble telescope team). The effect of numpy in the community, even in its current pre-1.0 form, has been tremendous. There is a real, pressing need in the scientific world for open source and technically superior replacements to Matlab and IDL, the propietary 800-lb gorillas of the field. Many major research institutions across the world are seriously looking at python as fulfilling this role, but the previous situation of a divided library (Numeric/numarray) was keeping a lot of people on the fence. With Travis' effort and numpy maturing towards a 1.0 release right around the time of python 2.5, a LOT of people have come out of the woodwork to contribute code, ideas, documentation, etc. There is a real sense that the combination of python2.5 (with better 64-bit and __index__ support) and numpy will provide a significant advancement for scientific computing with modern, high-level tools. In this particular community, the need to interface with low-level existing libraries is probably much more common than in other fields. There are literally millions of lines of C/C++ code for scientific work which we have to use efficiently, and this is an everyday need for us. While there are a number of tools for this (SWIG, Boost::Python, pyrex, scipy.weave,...), very recently people have discovered how useful ctypes can be for this task. One of the Google SoC projects (http://2006.planet-soc.com/blog/140) started trying to wrap libsvm with SWIG and a week of frustrated efforts led nowhere. Albert then discovered ctypes and in a few hours was up and running. This has generated a lot of interest in the numpy crowd for ctypes, and people would really, really like to see python2.5 come 'out of the box' with as solid a support as possible from ctypes for numpy array interfacing. Ultimately the decision is up to the release team, I know that. But at least with this info, I hope you can understand: 1. why this is important to this community 2. why the timing isn't ideal: it is only /very/ recently that the numpy team 'discovered' how much ctypes could truly help with a necessary (and often very unpleasant) task in the numerical/python world. Thanks for reading, f

On 7/4/06, Thomas Heller <theller@python.net> wrote:
Ok, former third-party libraries get a 1e-308 reprieve. :-) I think the general answer is to plan some way that you could make an external release to support the other communities like numpy. I don't know the details, but I believe Barry does this with email. This would allow Python to be more stable, but allow additional features for the communities that are interested in installing an additional ctypes release.
This patch is ok. I will add comments to the patch on SF.
This is too much to add to a beta. Perhaps you could work on a branch to add these features and integrate the branch back in after 2.5 has been released. You could make an external release from the branch. n

Neal Norwitz wrote:
http://www.python.org/sf/1519018 I believe this regression in syntax checking needs to be addressed. -- Scott Dial scott@scottdial.com scodial@indiana.edu
participants (28)
-
"Martin v. Löwis"
-
Almann T. Goo
-
Andrew Koenig
-
Anthony Baxter
-
Bill Chiles
-
Bill Janssen
-
BJörn Lindqvist
-
Fernando Perez
-
Gareth McCaughan
-
Giovanni Bajo
-
Greg Ewing
-
Guido van Rossum
-
James Y Knight
-
Jean-Paul Calderone
-
Josiah Carlson
-
Ka-Ping Yee
-
Michael Hudson
-
Neal Norwitz
-
Neil Schemenauer
-
Nick Coghlan
-
Phillip J. Eby
-
Scott Dial
-
Simon Percivall
-
skip@pobox.com
-
Talin
-
Terry Reedy
-
Thomas Heller
-
Tim Peters