On Wed, Feb 21, 2001 at 03:46:40PM +0100, Samuele Pedroni wrote:

IMO breaking code would be ok if we issue warnings today and implement nested scopes issuing errors tomorrow. But this is simply a statement about principles and raised impression.

Agreed. So maybe that's the best solution: pull nested scopes from 2.1 and add a warning for from...import (and exec?) inside a function using nested scopes, and only add nested scopes in 2.2, after everyone has had 6 months or a year to fix their code. --amk

I don't recall seeing any substanital discussion of this PEP on python-dev or python-list, nor do I recall a BDFL decision on the PEP. There has been lots of discussion about backwards compatibility, but not much consensus. Jeremy

On Wed, 21 Feb 2001 11:42:18 -0500, "Eric S. Raymond" <esr@thyrsus.com> wrote: [re: disabling nested scopes]

Aaargghh! I'm already using them.

That's not a valid excuse. The official position of Python-Dev regarding alphas is "a feature is not in until it's a release candidate -- we reserve the right to pull features before" Whatever we do, ifdefing is not the answer -- two incompat. versions of Python with the same number? Are we insane? -- "I'll be ex-DPL soon anyway so I'm |LUKE: Is Perl better than Python? looking for someplace else to grab power."|YODA: No...no... no. Quicker, -- Wichert Akkerman (on debian-private)| easier, more seductive. For public key, finger moshez@debian.org |http://www.{python,debian,gnu}.org

On Wed, Feb 21, 2001 at 07:01:05PM +0100, Fredrik Lundh wrote:

Is it time to shut down python-dev? (yes, I'm serious)

Just in case it might not be obvious, I concur with Fredrik, and I usually try to have a bit less of a temper than him. I have to warn, though, I just came from a meeting with Ministry of Justice lawyers, so I'm not in that good a mood, though my mood does force me to drop my politeness and just say what I really mean: I keep running into the ugly sides of the principle of nested scopes in python, and the implementation in particular. Most of them could be fixed, but not *all* of them, and the impact of those that can't be fixed is entirely unclear. Will it break a lot of code ? Possibly. Will it annoy a lot of people ? Quite certainly, it already did. Will it force people to turn away in disgust ? Definately possibly, since it's nearly doing that for *me*. I'm not sure if I'd want to admit to people that I'm a Python developper if that means they'll ask me why in hell 2.1 was released with that deficiency. I have been able to argue my way out of the gripes I currently get, but I'm not sure if I can do that for 2.1. I think adding nested scopes like this is a very bad idea. Patching up the problems by adding more special cases in which the old syntax would work is not the right solution, even though I did initially think so. And I'd like to note that none of these issues were addressed in the PEP. The PEP doesn't even mention them, though 'from Tkinter import *' is used as an example code snippet. And it seems most people are either indifferent or against the whole thing. I personally think the old 'hack' is *way* clearer, and more obvious, than the nested scopes patch. But maybe my perception is flawed. Maybe all the pro-nested-scopes, pro-breakage people are keeping quiet, in which case I'll quietly sulk away in a corner ;P Mr.-Conservatively-Grumpy-ly y'rs, -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Wed, Feb 21, 2001 at 07:13:17PM +0100, Thomas Wouters wrote:

But maybe my perception is flawed. Maybe all the pro-nested-scopes, pro-breakage people are keeping quiet, in which case I'll quietly sulk away in a corner ;P

The scoping rules are, IMHO, the most serious problem listed on the Python Warts page, and adding nested scopes fixes them. So it's nice that this flaw could be cleaned up, though people will naturally differ in their perceptions of how serious the problem is, and how much pain it's worth to fix it.

On Wed, Feb 21, 2001 at 07:01:05PM +0100, Fredrik Lundh wrote:

...

Is it time to shut down python-dev? (yes, I'm serious)

I've previously stated my intention to unsubscribe from python-dev after 2.1 ships, mostly because hacking on the Python core has ceased to be fun any more, and because my non-core projects have suffered. Once that happens, the incentive to try out new Python versions will really ebb; if I wasn't on python-dev, I don't think upgrading to 2.1 would be a big priority because none of its new features solve any burning problems for me. It's hard to say what compelling new features would make me enthuastically adopt 2.2 as soon as it comes out, and I can't really think of any -- perhaps interfaces would be such a feature. You can take that as lukewarm agreement with Fredrik's rhetorical suggestion. --amk

On Wed, Feb 21, 2001 at 02:35:02PM -0500, Jeremy Hylton wrote:

Do you think Python development has changed in ways that make it no longer fun? Or do you think that you've changed in ways that make you no longer enjoy Python development? I'm sure that it's not as simple

Mostly me; I'm trying to decrease my CPU load and have dropped a number of activities. I've mostly lost my taste for language hackery, and find that the discussions are getting more trivial and less interesting. Adding Unicode support, for example, was a lengthy and at times bloody discussion, but it resulted in a significant new capability. Debate about whether 'A in dict' is the same as 'A in dict.keys()' or 'A in dict.values()' is IMHO quite dull. Twhe unit testing debate was the last one I cared about to any significant degree. --amk

Jeremy Hylton wrote:

I don't recall seeing any substanital discussion of this PEP on python-dev or python-list, nor do I recall a BDFL decision on the PEP. There has been lots of discussion about backwards compatibility, but not much consensus.

We can have the discussion now, then. In my opinion it is irresponsible to knowingly unleash backwards incompatibilities on the world with no warning. If people think Python is unstable it will negatively impact its growth much more than the delay of some esoteric features. Let me put the ball back in your court: Is the benefit provided by having nested scopes this year rather than next year worth the pain of howls of outrage in Python-land. If we give people a year to upgrade (with warning messages) they will (rightly) grumble but not scream. -- Vote for Your Favorite Python & Perl Programming Accomplishments in the first Active Awards! http://www.ActiveState.com/Awards

On Wed, Feb 21, 2001 at 04:53:21PM -0500, Jeremy Hylton wrote:

When we unleashed Python 2.1a1, there was a fair amount of discussion about nested scopes on python-dev and on python-list.

Nested scopes weren't in 2.1a1, they were added between 2.1a1 and 2.1a2.

The fact that code would break has been documented in the PEP since December, before the BDFL pronounced on it.

The PEP only mentions one type of breakage, a local vrbl in an upper scope shadowing a global. It doesn't mention exec or from-module-import-*. I don't recall seeing a BDFL pronouncement on this issue, though I did whine about the whole thing from the start ;-P

I've heard plenty of hypothetical howls and one real one, from Mark.

Don't forget that the std. library itself had to be fixed in several places, because it violated the reference manual. Doesn't that hint that there is much more code out there that uses it ? I found two instances myself in old first-attempt GUI scripts of mine, which I never finished and thus aren't worth much more than the hypothetical howls. This is like spanking the dog/kid for doing something bad he had no way of knowing was bad. You can't expect the dog or the kid to read up on federal law to make sure he isn't doing anything bad by accident. Besides from any real problems we'll see, the added wartiness (which is what the hypothetical howls are all about) does really matter. What are we trying to solve with nested scopes ? Anything other than the default-argument hack wart ? Aren't we adding more warts to fix that one wart ? -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Wed, Feb 21, 2001 at 05:41:41PM -0500, Andrew Kuchling wrote:

Compatibility... ay, there's the rub!

If you include 'ways of thinking' in 'compatibility', I'll agree. Many people are used to being able to use exec/from-foo-import-*, and consider it part of Python's wonderful flexibility and straightforwardness (I know I do, and all my python-proficient and python-learning colleagues do.) -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Is it time to shut down python-dev? (yes, I'm serious)

I've been out in meetings all day, and just now checking my email. I'm a bit surprised by this sudden uprising. From the complaints so far, I don't really believe it's so bad. The embargo on not breaking code has never been absolute in my view. I do want to minimize breakage, but in the end my goal is to make people happy -- trying not to break code is only a means to that goal. It so happens that nested scopes will make many people happy too (if only because it allows references to surrounding locals from nested lambdas). I also don't mind as much breaking code that I consider ugly. I find import * inside a function very ugly (because I happen to know how much time it wastes). I find exec (without the ``in dict1, dict2'' clause) also pretty ugly, and usually being misused. I don't want to roll back nested scopes unless there's a lot more evidence that they are evil. Go through the PythonWare code base and look for code that would break -- and report back in the same style that Jeremy used. (Jeremy, it would help if you provided the tool you used for this analysis.) I remember you complained loudly about requiring list.append((x, y)) and socket.connect((host, port)) too -- but once you had fixed your code I didn't hear from you again, and I haven't had much feedback that this is a problem for the general population either. --Guido van Rossum (home page: http://www.python.org/~guido/)

I would consider the type/class split, making something like ExtensionClass neccessary, much more annoying for the advanced programmer. IMHO more efforts should go into this issue _even before_ p3000.

Yes, indeed. This will be on the agenda for Python 2.2. Digital Creations really wants PythonLabs to work on this issue! --Guido van Rossum (home page: http://www.python.org/~guido/)

On Wed, Feb 21, 2001 at 05:41:41PM -0500, Andrew Kuchling wrote:

...

Compatibility... ay, there's the rub!

If you include 'ways of thinking' in 'compatibility', I'll agree. Many people are used to being able to use exec/from-foo-import-*, and consider it part of Python's wonderful flexibility and straightforwardness (I know I do, and all my python-proficient and python-learning colleagues do.)

Actually, I've always considered 'exec' mostly one of those must-have- because-the-competition-has-it features. Language theorists love it. In practice, bare exec not that useful; a more restricted form (e.g. one that always requires the caller to explicitly pass in an environment) makes much more sense. As for import *, we all know that it's an abomination... --Guido van Rossum (home page: http://www.python.org/~guido/)

Tim wrote:

[/F]

...

Is it time to shut down python-dev? (yes, I'm serious)

I can't imagine that it would be possible to have such a vigorous and focused debate about Python development in the absence of Python-Dev.

If a debate doesn't lead anywhere, it's just a waste of time. Code monkey contributions can be handled via sourceforge, and general whining works just as well on comp.lang.python. ::: Donning my devil's advocate suite, here are some recent observations: - Important decisions are made on internal PythonLabs meetings (unit testing, the scope issue, etc), not by an organized python- dev process. Does anyone care about -1 and +1's anymore? - The PEP process isn't working ("I updated the PEP and checked in the code", "but *that* PEP doesn't apply to *me*", etc). - Impressive hacks are more important than concerns from people who make their living selling Python technology (rather than a specific application). Codewise, nested scopes are amazing. From a marketing perspective, it's a disaster. (even more absurd allegations snipped) Am I entirely wrong? Cheers /F

Fredrik Lundh wrote:

Tim wrote:

...

[/F]

...

Is it time to shut down python-dev? (yes, I'm serious)

I can't imagine that it would be possible to have such a vigorous and focused debate about Python development in the absence of Python-Dev.

If a debate doesn't lead anywhere, it's just a waste of time.

Code monkey contributions can be handled via sourceforge, and general whining works just as well on comp.lang.python.

Na, Fredrik, we wouldn't want to lose our nice little chat room -- it's way too much fun around here :-)

:::

Donning my devil's advocate suite, here are some recent observations:

- Important decisions are made on internal PythonLabs meetings (unit testing, the scope issue, etc), not by an organized python- dev process. Does anyone care about -1 and +1's anymore?

Well, being one of the first opponents of nested scopes (nobody else seemed to care back then...) and seeing how many of those other obscure PEPs made their way into the core, I have similar feelings. Still, I see the voting system as being a democratic method of reaching consensus: if there only one -1 and half a dozen +1s then I am overruled.

- The PEP process isn't working ("I updated the PEP and checked in the code", "but *that* PEP doesn't apply to *me*", etc).

Aren't PEPs meant to store information gathered in ongoing discussions rather than being an official statement of consent ?

- Impressive hacks are more important than concerns from people who make their living selling Python technology (rather than a specific application). Codewise, nested scopes are amazing. From a marketing perspective, it's a disaster.

Agreed and I have never understood why getting lambdas to work without keyword hacks is motivation enough to break code in all kinds of places. The nested scopes thingie started out as simple idea, but has in time grown so many special cases that I think the idea has already proven all by itself that it is the wrong approach to the problem (if there ever was a problem -- lambdas are certainly not newbie style gadgets). -- Marc-Andre Lemburg ______________________________________________________________________ Company & Consulting: http://www.egenix.com/ Python Pages: http://www.lemburg.com/python/

Donning my devil's advocate suite, here are some recent observations:

- Important decisions are made on internal PythonLabs meetings (unit testing, the scope issue, etc), not by an organized python- dev process. Does anyone care about -1 and +1's anymore?

Python-dev is as organized as its participants want it to be. It appeared that very few people (apart from you) were interested in unit testing, so we looked elsewhere. We found that others inside Digital Creations had lots of experience with PyUnit and really liked it. Without arguments, +1 and -1's indeed don't have that much weight. With the right argument, a single +1 or -1 can be sufficient. Python is (still) not a democracy.

- The PEP process isn't working ("I updated the PEP and checked in the code", "but *that* PEP doesn't apply to *me*", etc).

I wouldn't say it isn't working. I believe it's very helpful to have a working document checked in somewhere to augment the discussion, and the PEPs make progress possible where in the past we went around in circles in the list without ever coming to a conclusion. Forcing the proposer of a new feature to write a PEP is a good way to think through more of the consequences of a new idea. Referring to a PEP when arguments are repeated can cut short discussion. Note that the PEP work flow document (PEP 1) explicitly states that the BDFL has the final word. But of course sometimes the realities of software development catch up with us -- we can't possibly hope to do all design ahead of all implementation, and during testing we may discover important new facts that must affect the design.

- Impressive hacks are more important than concerns from people who make their living selling Python technology (rather than a specific application). Codewise, nested scopes are amazing. From a marketing perspective, it's a disaster.

Aha, now we're talking. Python is growing up, and more and more people are making money by supporting it. Obviously, businesspeople have to be more conservative than software developers. But do you *really* think that breaking the occasional exec-without-in-clause or from-import-* will affect a large enough portion of the user population to make a difference? People with a lot at stake tend to be slow in upgrading anyway. So we're releasing 2.1 mostly for the bleeding edge consumers -- e.g. Paul Barret recently announced that his institute is upgrading to 2.0 and doesn't plan to switch to 2.1 any time soon. That's fine with me. Hey, here's an idea. We could add the warning API to 2.0.1 (it's backwards compatible AFAIK), and you can release PY201 with warnings added for things that your customers will need to change before they switch to PY21. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thu, Feb 22, 2001 at 08:51:03AM -0500, Guido van Rossum wrote:

Hey, here's an idea. We could add the warning API to 2.0.1 (it's backwards compatible AFAIK), and you can release PY201 with warnings added for things that your customers will need to change before they switch to PY21.

Definately +1 on that. While on the subject: will all of 'from module import *' be deprecated, even at module level ? How should code like Mailman's mm_cfg.py/Defaults.py construct be rewritten to provide similar functionality ? Much as I dislike 'from module import *', it really does have its uses. -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

...

...

...

...

"GvR" == Guido van Rossum <guido@digicool.com> writes:

>> How should code like Mailman's mm_cfg.py/Defaults.py construct >> be rewritten to provide similar functionality ? Much as I >> dislike 'from module import *', it really does have its uses. GvR> I have no idea what mm_cfg.py/Defaults.py is, but yes, import GvR> * has its uses! Not that it's really that important to the discussion, but the way Mailman lets users override its defaults is by putting all the (autoconf and hardcoded) system defaults in Defaults.py, which the user is never supposed to touch. Then mm_cfg.py does a "from Defaults import *" -- at module level of course -- and users put any overridden values in mm_cfg.py. All Mailman modules that have to reference a system default do so by importing and using mm_cfg. This was Ken's idea, and a darn good one! It's got a wart or two, but they are quite minor. -Barry

[/F]

If a debate doesn't lead anywhere, it's just a waste of time.

If you end up being on the winning side, is it still a waste of time? If you end up being on the losing side of a debate, perhaps, sometimes. But I can't predict the future well enough to know the outcome in advance.

Donning my devil's advocate suite, here are some recent observations:

- Important decisions are made on internal PythonLabs meetings (unit testing, the scope issue, etc), not by an organized python- dev process.

Decisions are-- and were --made in Guido's head. Python-Dev was established to give him easier access to higher-quality input than was possible on c.l.py at the time, and to give Python developers a higher S/N place to hang out when discussing Python development. Internal PythonLabs meetings are really much the same, just on a smaller scale and with a higher-still S/N ratio. Both work for those purposes. It isn't-- and wasn't --the purpose of either to strip Guido of the last word.

Does anyone care about -1 and +1's anymore?

Did anyone ever <0.5 wink>? A scattering of two-character arguments is interesting to get a quick feel, but even I wouldn't *decide* anything on that basis. If this were an ANSI/ISO committee, a single -1 would have absolute power -- and then we'd still be using Python 0.9.6 (ANSI/ISO committees need soul-crushingly boring and budget-bustingly expensive meetings regularly else consensus would never be reached on anything -- if people get to veto in their spare time while sitting at home, and without opponents blowing spit right in their face for the 18th time in 6 years, there's insufficient pressure *to* compromise).

- The PEP process isn't working ("I updated the PEP and checked in the code", "but *that* PEP doesn't apply to *me*", etc).

Need to define "working". I don't think it's what it should be yet, but is making progress.

- Impressive hacks are more important than concerns from people who make their living selling Python technology (rather than a specific application). Codewise, nested scopes are amazing. From a marketing perspective, it's a disaster.

Any marketing droid would believe that Python's current market is a fraction of its potential market, and so welcome any "new feature" that makes new sales easier. c.l.py is a microcosm of this battlefield, and the cry for nested scopes has continued unabated since the day lambda was introduced. I've never met a marketing type (and I've met more than my share ...) who wouldn't seize this as an opportunity to *expand* market share. Sales droids servicing existing accounts *may* grumble -- or the more inventive may take it as an opportunity to drive home the importance of their relationship to their customers ("it's us against them, and boy aren't you glad you've got Amalgamated Pythonistries on your side!").

(even more absurd allegations snipped)

With gratitude, and I'll skip even more absurd rationalizations <wink>.

Am I entirely wrong?

Of course not. The world isn't that simple. indeed-the-world-is-heavily-nested<wink>-ly y'rs - tim PS: At the internal PythonLabs mtg today, I voted against nested scopes. But also for them. Leaving that to Jeremy to explain.

Guido:

...

Language theorists love [exec].

Really? I'd have thought language theorists would be the ones who hate it, given all the problems it causes...

Depends on where they're coming from. Or maybe I should have said Lisp folks... --Guido van Rossum (home page: http://www.python.org/~guido/)

We (PythonLabs) have received a lot of flak over our plan to introduce nested scopes despite the fact that it appears to break a small but significant amount of working code. We discussed this at an PythonLabs group meeting today. After the meeting, Tim posted this teaser:

PS: At the internal PythonLabs mtg today, I voted against nested scopes. But also for them. Leaving that to Jeremy to explain.

After the meeting Jeremy had a four hour commute home due to bad weather, so let me do the honors for him. (Jeremy will update the PEP, implement the feature, and update the documentation, in that order.) We have clearly underestimated how much code the nested scopes would break, but more importantly we have underestimated how much value our community places on stability. At the same time we really like nested scopes, and we would like to see the feature introduced at some point. So here's the deal: we'll make nested scopes an optional feature in 2.1, default off, selectable on a per-module basis using a mechanism that's slightly hackish but is guaranteed to be safe. (See below.) At the same time, we'll augment the compiler to detect all situations that will break when nested scopes are introduced in the future, and issue warnings for those situations. The idea here is that warnings don't break code, but encourage folks to fix their code so we can introduce nested scopes in 2.2. Given our current pace of releases that should be about 6 months warning. These warnings are *not* optional -- they are issued regardless of whether you select to use nested scopes. However there is a command line option (crudest form: -Wi) to disable warnings; there are also ways to disable them programmatically. If you want to make sure that you don't ignore the warnings, there's also a way to turn warnings into errors (-We from the command line). How do you select nested scopes? Tim suggested a mechanism that is used by the ANSI C committee to enable language features that are backwards incompatible: they trigger on the import of a specific previously non-existant header file. (E.g. after #include <imaginary.h>, "imaginary" becomes a reserved word.) The Python equivalent of this is a magical import that is recognized by the compiler; this was also proposed by David Scherer for making integer division yield a float. (See http://mail.python.org/pipermail/edu-sig/2000-May/000499.html) You could say that Perl's "use" statement is similar. We haven't decided yet which magical import; two proposals are: import __nested_scopes__ from __future__ import nested_scopes The magical import only affects the source file in which it occurs. It is recognized by the compiler as it is scanning the source code. It must appear at the top-level (no "if" or "try" or "def" or anything else around it) and before any code that could be affected. We realize that PEP 5 specifies a one-year transition period. We believe that that is excessive in this case, and would like to change the PEP to be more flexible. (The PEP has questionable status -- it was never formally discussed.) We also believe that the magical import mechanism is useful enough to be reused for other situations like this; Tim will draft a PEP to describe in excruciating detail. I thank everybody who gave feedback on this issue. And thanks to Jeremy for implementing nested scopes! --Guido van Rossum (home page: http://www.python.org/~guido/)

[Guido]

We also believe that the magical import mechanism is useful enough to be reused for other situations like this; Tim will draft a PEP to describe in excruciating detail.

[Jeremy Hylton]

... I'm happy to hear that Tim will draft this PEP. He didn't mention it at lunch today or I would have given him a big hug (or bought him a Coke).

Guido's msg was the first I heard of it too. I think this is the same process by which I got assigned to change Windows imports: the issue came up, and I opened my mouth <-0.9 wink>.

As Tim knows, I think the PEP needs to say something about whether these magic imports create name bindings and what objects are bound to the names.

Will we need an __nested_scopes__.py in the Lib directory?

Offhand, I suggest to create a real Lib/__future__.py, and let import code get generated as always. The purpose of __future__.py is to record release info in an *obvious* place to look for it (BTW, best I can tell, sys.version isn't documented anywhere, so this serves that purpose too <wink>): ------------------------------------------------------------------ """__future__: Record of phased-in incompatible language changes. Each line is of the form: FeatureName = ReleaseInfo ReleaseInfo is a pair of the form: (OptionalRelease, MandatoryRelease) where, normally, OptionalRelease <= MandatoryRelease, and both are 5-tuples of the same form as sys.version_info: (PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int PY_MINOR_VERSION, # the 1; an int PY_MICRO_VERSION, # the 0; an int PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string PY_RELEASE_SERIAL # the 3; an int ) In the case of MandatoryReleases that have not yet occurred, MandatoryRelease predicts the release in which the feature will become a permanent part of the language. Else MandatoryRelease records when the feature became a permanent part of the language; in releases at or after that, modules no longer need from __future__ import FeatureName to use the feature in question, but may continue to use such imports. In releases before OptionalRelease, an import from __future__ of FeatureName will raise an exception. MandatoryRelease may also be None, meaning that a planned feature got dropped. No line is ever to be deleted from this file. """ nested_scopes = (2, 1, 0, "beta", 1), (2, 2, 0, "final", 0) ----------------------------------------------------------------- While this is 100% intended to serve a documentation purpose, I also intend to use it in my own code, like so (none of which is special to the compiler except for the first line): from __future__ import nested_scopes import sys assert sys.version_info < nested_scopes[1], "delete this section!" # Note that the assert above also triggers if MandatoryRelease is None, # i.e. if the feature got dropped (under 2.1 rules, None is smaller than # anything else <wink>). del sys, nested_scopes Other rules: # Legal only at module scope, before all non-comment occurrences of # name, and only when name is known to the compiler. from __future__ import name # Ditto. name2 has no special meaning. from __future__ import name as name2 The purpose of the next two is to allow programmatic manipulation of the info in __future__.py (generate help msgs, build a histogram of adoption dates for incompatible changes by decade over the previous two centuries, whatever). # Legal anywhere, but no special meaning. import __future__ import __future__ as name

[TIm]

(BTW, best I can tell, sys.version isn't documented anywhere, so this serves that purpose too <wink>).

Wow. Averaging two errors per line! I meant sys.version_info, and it's documented in the obvious place. error-free-at-laat!-ly y'rs - itm

[Peter Funk]

I believe __future__ is a bad name. What appears today as the bright shining future will be the distant dusty past of tomorrow. But the name of the module is not going to change anytime soon. right?

The name of what module? Any statement of the form from __future__ import shiny becomes unnecessary as soon as shiny's future arrives, at which point the statement can be removed. The statement is necessary only so long as shiny *is* in the future. So the name is thoroughly appropriate.

Please call it __progress__ or __history__ or even __python_history__ or come up with some other name.

Sorry, but none of those make any sense given the intended use. It's not a part of Python 2.1 "history" that nested scopes won't be the default before 2.2!

What about __python_bloat__ ? <duck ;-)>.

*That* one makes some sense.

In my experience of computing it is a really bad idea to call anything 'new', 'old', 'future', '2000' or some such because those names last much longer than you would have believed at the time the name was choosen.

The purpose of __future__ is to supply a means to try out future incompatible extensions before they become the default. The set of future extensions will change from release to release, but that they *are* in the future remains invariant even if Python goes on until universal heat death. Given the rules I already posted, it will be very easy to write a Python tool to identify obsolete __future__ imports and remove them (if you want).

On Fri, 23 Feb 2001, Mikael Olofsson <mikael@isy.liu.se> wrote:

There really is a time machine. So I guess I can get the full Python 3k functionality by doing

from __future__ import *

In Py3K from import * will be illegal, so this will unfortunately blow up the minute the "import_star_bad" is imported. You'll just have to try them one by one... -- "I'll be ex-DPL soon anyway so I'm |LUKE: Is Perl better than Python? looking for someplace else to grab power."|YODA: No...no... no. Quicker, -- Wichert Akkerman (on debian-private)| easier, more seductive. For public key, finger moshez@debian.org |http://www.{python,debian,gnu}.org

[Guido]

...

We also believe that the magical import mechanism is useful enough to be reused for other situations like this; Tim will draft a PEP to describe in excruciating detail.

[Jeremy Hylton]

...

... I'm happy to hear that Tim will draft this PEP. He didn't mention it at lunch today or I would have given him a big hug (or bought him a Coke).

Guido's msg was the first I heard of it too. I think this is the same process by which I got assigned to change Windows imports: the issue came up, and I opened my mouth <-0.9 wink>.

Oops. I swear I heard you offer to write it. I guess all you said was that it should be written. Oh well. Somebody will write it. :-) Looks like Tim's proposed __future__.py is in good shape already. --Guido van Rossum (home page: http://www.python.org/~guido/)

...

from __future__ import nested_scopes

I this this alternative better since there is only one "reserved" module name.

Noted.

I still think releasing 2.0.1 with warnings is a good idea. OTOH, maybe its hard for that compiler to detect questionable code.

The problem is that in order to do a decent job of compile-time warnings, not only the warnings module and API would have to be retrofitted in 2.0.1, but also the entire new compiler, which has the symbol table needed to be able to detect the situations we want to warn about. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Thu, Feb 22, 2001 at 07:14:50PM -0500, Eric S. Raymond wrote:

practice than it is in theory. In fact, Python has rather forced me to question whether "No separation between code and data" was as important a component of LISP's supernal wonderfulness as I believed when I was a fully fervent member of the cult.

I think it is. Currently I'm reading Steven Tanimoto's introductory AI book in a doomed-from-the-start attempt to learn about rule-based systems, and along the way am thinking about how I'd do similar tasks in Python. The problem is that, for applying pattern matching to data structures, Python has no good equivalent of Lisp's (pattern-match data '((? X) 1 2)). [1] Perhaps this is more a benefit of Lisp's simple syntax than the "no separation between code and data" priniciple. In Python you could write some sort of specialized parser, of course, but that's really a distraction from the primary AI task of writing a really bitchin' Eliza program (or whatever). --amk [1] Which would match any list whose 2nd and 3rd elements are (1 2), and bind the first element to X somehow.

[Guido]

Oops. I swear I heard you offer to write it. I guess all you said was that it should be written. Oh well. Somebody will write it. :-)

Na, I'll write it! I didn't volunteer, but since I've already thought about it more than anyone on Earth, I'm the natural vic^H^H^Hauthor. cementing-my-monopoly-on-retroactive-peps-ly y'rs - tim

[Jeremy]

1 Should this work?

if x: from __future__ import nested_scopes

I presume not, but the sketch of the rules you posted earlier presumably allow it.

You have to learn to think more like tabnanny: "module scope" obviously means "indent level 0" if you're obsessed with whitespace <wink>.

2. How should the interactive interpreter be handled?

You're kidding. I thought we agreed to drop the interactive interpreter for 2.1? (Let's *really* give 'em something to carp about ...)

I presume if you type

...

...

...

from __future__ import nested_scopes

That everything thereafter will be compiled with nested scopes.

That's my guess too, of course.

This ends up being a little tricky, because the interpreter has to hang onto this information and tell the compiler about it.

Ditto for python -i some_script.py where some_script.py contains a magical import. OTOH, does exec-compiled (or execfile-ed) code start with a clean slate, or inherent the setting of the module from which it's exec[file]'ed? I think the latter has to be true. Could get messy, so it's a good thing we've got several whole days to work out the kinks ... business-as-usual-ly y'rs - tim

Hi, Tim Peters: [...]

Any statement of the form

from __future__ import shiny

becomes unnecessary as soon as shiny's future arrives, at which point the statement can be removed. The statement is necessary only so long as shiny *is* in the future. So the name is thoroughly appropriate. [...]

Obviously you assume, that software written in Python will be bundled only with one certain version of the Python interpreter. This might be true for Windows, where Python is no integral part of base operating system. Not so for Linux: There application developers have to support a range of versions covering at least 3 years, if they don't want to start fighting against the preinstalled Python. A while ago I decided to drop the support for Python 1.5.1 and earlier in our software. This has bitten me bad: Upgrading the Python 1.5.1 installation to 1.5.2 on SuSE Linux 6.0 machine at a customer site resulted in a nightmare. Obviously I would have saved half of the night, if I had decided to install a development system (GCC, libs ...) there and would have Python recompiled from source instead of trying to incrementally upgrade parts of the system using the precompiled binary RPMs delivered by SuSE). Now I have learned my lessons and I will not drop support for 1.5.2 until 2003. BTW: SuSE will start to ship SuSE Linux 7.1 just now in the US (it is available here since Feb 10th). AFAIK this is the first Linux distribution coming with Python 2.0 as the default Python. Every other commercially used Linux system out there probably has Python 1.5.2 or older.

Given the rules I already posted, it will be very easy to write a Python tool to identify obsolete __future__ imports and remove them (if you want). [...]

Hmmm... If my Python apps have to support for example Python from version 2.1 up to 2.5 or 2.6 in 2003, I certainly have to leave the 'from __future__ import ...'-statements alone and can't remove them without sacrifying backward compatibility to the Python interpreter which made this feature available for the first time. At this time __future__ will contain features, that are 2.5 years old. BTW: We will abstain from using string methods, augmented assignments and list compr. for at least the next two years out of similar reasons. On the other hand I would never bother with IO-Port hacking to get a 200Hz and 1.5 second long "beep" out of the PC builtin speaker... ;-) Have a nice weekend and good night, Peter

On Fri, Feb 23, 2001 at 06:00:59PM -0500, Jeremy Hylton wrote:

Hmmmm... I'm not yet sure how to deduce indent level 0 inside the parser.

Uhm, why are we adding that restriction anyway, if it's hard for the parser/compiler to detect it ? I think I'd like to put them in try/except or if/else clauses, for fully portable code. While on the subject, a way to distinguish between '__future__ not found' and '__future__.feature not found', other than hardcoding the minimal version might be nice. -- Thomas Wouters <thomas@xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

[Tim]

Any statement of the form

from __future__ import shiny

becomes unnecessary as soon as shiny's future arrives, at which point the statement can be removed. The statement is necessary only so long as shiny *is* in the future. So the name is thoroughly appropriate.

[Peter Funk]

Obviously you assume, that software written in Python will be bundled only with one certain version of the Python interpreter.

Not really. I think it's more the case that you're viewing this gimmick through the eyes of your particular problems, and criticizing because it don't solve them. However, it wasn't intended to solve them.

This might be rue for Windows, where Python is no integral part of base operating system. Not so for Linux: There application developers have to support a range of versions covering at least 3 years, if they don't want to start fighting against the preinstalled Python.

It's not true that Windows is devoid of compatibility problems. But Windows Python takes a different approach: we even rename the Windows Python DLLs with each release. That way any number of incompatible Pythons can coexist peacefully (this isn't trivial under Windows, because we have to install the core DLL in a specific magic directory). A serious Python app developed for Windows generally ships with the specific Python it wants, too (not unique to Python, of course, serious apps of all kinds ship with the support softare they need on Windows, up to and sometimes even including the basic MS C runtime libs). How people on other OSes choose to deal with this is up to them. If you find the Linux approach lacking, I believe you, but the "magical import" mechanism is too feeble a base on which to pin your hopes. Get serious about this! Write a PEP that will truly address your problems. This one does not; I don't even see that it's *related* to your problems.

... BTW: SuSE will start to ship SuSE Linux 7.1 just now in the US (it is available here since Feb 10th). AFAIK this is the first Linux distribution coming with Python 2.0 as the default Python. Every other commercially used Linux system out there probably has Python 1.5.2 or older.

Yet another reason to prefer Windows <wink>.

... Hmmm... If my Python apps have to support for example Python from version 2.1 up to 2.5 or 2.6 in 2003, I certainly have to leave the 'from __future__ import ...'-statements alone and can't remove them without sacrifying backward compatibility to the Python interpreter which made this feature available for the first time.

The only way to write a piece of code that runs under all of 2.1 thru 2.6 is to avoid any behavior whatsoever that's specific to some proper subset of those versions. That's hard, and I don't think "from __future__" even *helps* with that. But it wasn't meant to. It was meant to make life easier for people who *do* upgrade in a timely fashion, in accord with at least the spirit of the existing PEPs on the topic.

At this time __future__ will contain features, that are 2.5 years old.

And ...? That is, what of it? In 1000 years, it will contain features that are 1000 years old. So? Else code written now and never purged of obsolete __future__s would break 1000 years from now. You can fault the scheme on many bases, but not on the basis that it creates new incompatibility problems. Leaving the old __future__s in will help a little in the other direction: code that announces it relies on a __future__ F will reliably fail at compile-time if run under a release less than F's OptionalRelease value.

BTW: We will abstain from using string methods, augmented assignments and list compr. for at least the next two years out of similar reasons.

If that's the best you think can you do, so it goes. It would be nice to think of a better way. But this isn't the right gimmick, and that it doesn't solve your problems doesn't mean it fails to solve anyone's problems.

On the other hand I would never bother with IO-Port hacking to get a 200Hz and 1.5 second long "beep" out of the PC builtin speaker... ;-)

That's compatibility: it worked before under NT and 2000, but not under Win9X, and it has high newbie appeal (I dove it into after making excuses about Win9X Beep() for the umpteenth time on the Tutor list). If you want to make Linux attractive to newbies, implementing Beep() for it too would be an excellent step. If you like, I'll reserve from __future__ import MakeLinuxBearableForNewbies right now <wink>.

...

Ditto for

python -i some_script.py

[Samuele Pedroni]

This make sense but I guess people will ask for a way to disable the feature after a while in the session, even trickier.

The purpose is to let interested people use new features "early", not to let people jerk off. That is, they can ask all they want <wink>.

...

[Tim sez exec and execfile should inherit the module's setting]

I disagree, although this reduces the number of places where one has to delete from __future__ import when _future_ is here,

That isn't the intent. The intent is that a module containing from __future__ import f is announcing it *wants* future semantics for f. Therefore the module should act, in *all* respects (incl. exec and execfile), as if the release were already the future one in which f is no longer optional. If exec, eval or execfile continue to act like the older release, the module isn't getting the semantics it specifically asked for, and the user isn't getting a correct test of future functionality.

for some uses of execfile the original program has just little control over what is in the executed file I guess,

Then they may have deeper problems than this gimmick can address, but they're not going to find out whether the files they're execfile'ing *will* have a problem in the future unless the module asking for future semantics gets future semantics.

better having people being explicit there about what they want.

They already are being explicit: they get future semantics when and only when they include a from__future__ thingie.

And this way we don't have to invent a way for forcing disabling the feature (at least not because of the inherited default problems).

There is *no* intent here that a single module be able to pick and choose different behaviors in different contexts. The purpose is to allow early testing and development of code to ensure it will work correctly in a future release. That's all.

... Or we need an even more complicated mechanismus? like your proposed import not.

I doubt core Python will ever support "moving back in time" (a heavily conditionalized code base is much harder to maintain -- ask Jeremy how much fun he's having trying to make this optional *now*). May (or may not) be an interesting idea for repackagers to consider, though.

[Jeremy Hylton]

Hmmmm... I'm not yet sure how to deduce indent level 0 inside the parser.

[Thomas Wouters]

Uhm, why are we adding that restriction anyway, if it's hard for the parser/compiler to detect it ?

I talked with Jeremy, and turns out it's not.

I think I'd like to put them in try/except or if/else clauses, for fully portable code.

And, sorry, but I take back saying that we should allow that. We shouldn't. Despite that it looks like an import statement (and actually *is* one, for that matter), the key info is extracted at compile time. So in stuff like if x: from __future__ import alabaster_weenoblobs whether or not alabaster_weenoblobs is in effect has nothing to do with the runtime value of x. So it's plain Bad to allow it to look as if it did. The only stuff that can textually precede: from __future__ import f is: + The module docstring (if any). + Comments. + Blank lines. + Other instances of from __future__. This also makes clear that one of these things applies to the entire module. Again, the thrust of this is *not* to help in writing portable code. It's to help users upgrade to the next release, in two ways: (1) by not breaking their code before the next release; and, (2) to let them migrate their code to next-release semantics incrementally. Note: "next release" means whatever MandatoryRelease is associated with the feature of interest. "Cross version portable code" is a more pressing problem for some, but is outside the scope of this gimmick. *This* gimmick is something we can actually do <0.5 wink>.

Gimme a PEP number, and I'll post this to the real users too <wink>. PEP: ? Title: Back to the __future__ Version: $Revision: 1.0 $ Author: Tim Peters <tim@digicool.com> Python-Version: 2.1 Status: ? Type: Standards Track Post-History: Motivation From time to time, Python makes an incompatible change to the advertised semantics of core language constructs, or changes their accidental (implementation-dependent) behavior in some way. While this is never done capriciously, and is always done with the aim of improving the language over the long term, over the short term it's contentious and disrupting. The "Guidelines for Language Evolution" PEP [1] suggests ways to ease the pain, and this PEP introduces some machinery in support of that. The "Statically Nested Scopes" PEP [2] is the first application, and will be used as an example here. Intent When an incompatible change to core language syntax or semantics is being made: 1. The release C that introduces the change does not change the syntax or semantics by default. 2. A future release R is identified in which the new syntax or semantics will be enforced. 3. The mechanisms described in the "Warning Framework" PEP [3] are used to generate warnings, whenever possible, about constructs or operations whose meaning may[4] change in release R. 4. The new future_statement (see below) can be explicitly included in a module M to request that the code in module M use the new syntax or semantics in the current release C. So old code continues to work by default, for at least one release, although it may start to generate new warning messages. Migration to the new syntax or semantics can proceed during that time, using the future_statement to make modules containing it act as if the new syntax or semantics were already being enforced. Syntax A future_statement is simply a from/import statement using the reserved module name __future__: future_statement: "from" "__future__" "import" feature ["as" name] ("," feature ["as" name])* feature: identifier In addition, all future_statments must appear near the top of the module. The only lines that can appear before a future_statement are: + The module docstring (if any). + Comments. + Blank lines. + Other future_statements. Example: """This is a module docstring.""" # This is a comment, preceded by a blank line and followed by # a future_statement. from __future__ import nested_scopes from math import sin from __future__ import alabaster_weenoblobs # compile-time error! # That was an error because preceded by a non-future_statement. Semantics A future_statement is recognized and treated specially at compile time: changes to the semantics of core constructs are often implemented by generating different code. It may even be the case that a new feature introduces new incompatible syntax (such as a new reserved word), in which case the compiler may need to parse the module differently. Such decisions cannot be pushed off until runtime. For any given release, the compiler knows which feature names have been defined, and raises a compile-time error if a future_statement contains a feature not known to it[5]. The direct runtime semantics are the same as for any import statement: there is a standard module __future__.py, described later, and it will be imported in the usual way at the time the future_statement is executed. The *interesting* runtime semantics depend on the feature(s) "imported" by the future_statement(s) appearing in the module. Since a module M containing a future_statement naming feature F explicitly requests that the current release act like a future release with respect to F, any code interpreted dynamically from an eval, exec or execfile executed by M will also use the new syntax or semantics associated with F. A future_statement appearing "near the top" (see Syntax above) of code interpreted dynamically by an exec or execfile applies to the code block executed by the exec or execfile, but has no further effect on the module that executed the exec or execfile. Note that there is nothing special about the statement: import __future__ [as name] That is not a future_statement; it's an ordinary import statement, with no special syntax restrictions or special semantics. Interactive shells may pose special problems. The intent is that a future_statement typed at an interactive shell prompt affect all code typed to that shell for the remaining life of the shell session. It's not clear how to achieve that. Example Consider this code, in file scope.py: x = 42 def f(): x = 666 def g(): print "x is", x g() f() Under 2.0, it prints: x is 42 Nested scopes[2] are being introduced in 2.1. But under 2.1, it still prints x is 42 and also generates a warning. In 2.2, and also in 2.1 *if* "from __future__ import nested_scopes" is included at the top of scope.py, it prints x is 666 Standard Module __future__.py Lib/__future__.py is a real module, and serves three purposes: 1. To avoid confusing existing tools that analyze import statements and expect to find the modules they're importing. 2. To ensure that future_statements run under releases prior to 2.1 at least yield runtime exceptions (the import of __future__ will fail, because there was no module of that name prior to 2.1). 3. To document when incompatible changes were introduced, and when they will be-- or were --made mandatory. This is a form of executable documentation, and can be inspected programatically via importing __future__ and examining its contents. Each statment in __future__.py is of the form: FeatureName = ReleaseInfo ReleaseInfo is a pair of the form: (OptionalRelease, MandatoryRelease) where, normally, OptionalRelease < MandatoryRelease, and both are 5-tuples of the same form as sys.version_info: (PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int PY_MINOR_VERSION, # the 1; an int PY_MICRO_VERSION, # the 0; an int PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string PY_RELEASE_SERIAL # the 3; an int ) OptionalRelease records the first release in which from __future__ import FeatureName was accepted. In the case of MandatoryReleases that have not yet occurred, MandatoryRelease predicts the release in which the feature will become part of the language. Else MandatoryRelease records when the feature became part of the language; in releases at or after that, modules no longer need from __future__ import FeatureName to use the feature in question, but may continue to use such imports. MandatoryRelease may also be None, meaning that a planned feature got dropped. No line will ever be deleted from __future__.py. Example line: nested_scopes = (2, 1, 0, "beta", 1), (2, 2, 0, "final", 0) This means that from __future__ import nested_scopes will work in all releases at or after 2.1b1, and that nested_scopes are intended to be enforced starting in release 2.2. Questions and Answers Q: What about a "from __past__" version, to get back *old* behavior? A: Outside the scope of this PEP. Seems unlikely to the author, though. Write a PEP if you want to pursue it. Q: What about incompatibilites due to changes in the Python virtual machine? A: Outside the scope of this PEP, although PEP 5[1] suggests a grace period there too, and the future_statement may also have a role to play there. Q: What about incompatibilites due to changes in Python's C API? A: Outside the scope of this PEP. Q: I want to wrap future_statements in try/except blocks, so I can use different code depending on which version of Python I'm running. Why can't I? A: Sorry! try/except is a runtime feature; future_statements are primarily compile-time gimmicks, and your try/except happens long after the compiler is done. That is, by the time you do try/except, the semantics in effect for the module are already a done deal. Since the try/except wouldn't accomplish what it *looks* like it should accomplish, it's simply not allowed. We also want to keep these special statements very easy to find and to recognize. Note that you *can* import __future__ directly, and use the information in it, along with sys.version_info, to figure out where the release you're running under stands in relation to a given feature's status. Q: Going back to the nested_scopes example, what if release 2.2 comes along and I still haven't changed my code? How can I keep the 2.1 behavior then? A: By continuing to use 2.1, and not moving to 2.2 until you do change your code. The purpose of future_statement is to make life easier for people who keep keep current with the latest release in a timely fashion. We don't hate you if you don't, but your problems are much harder to solve, and somebody with those problems will need to write a PEP addressing them. future_statement is aimed at a different audience. Copyright This document has been placed in the public domain. References and Footnotes [1] http://python.sourceforge.net/peps/pep-0005.html [2] http://python.sourceforge.net/peps/pep-0227.html [3] http://python.sourceforge.net/peps/pep-0230.html [4] Note that this is "may" and not "will": better safe than sorry. Of course spurious warnings won't be generated when avoidable with reasonable cost. [5] This ensures that a future_statement run under a release prior to the first one in which a given feature is known (but >= 2.1) will raise a compile-time error rather than silently do a wrong thing. If transported to a release prior to 2.1, a runtime error will be raised because of the failure to import __future__ (no such module existed in the standard distribution before the 2.1 release, and the double underscores make it a reserved name).  Local Variables: mode: indented-text indent-tabs-mode: nil End:

[Tim]

... A compile-time exception: when you're asking for semantics the compiler can't give you, the presumption has to favor that you're in big trouble. You can't catch such an exception directly in the same module (because it occurs at compile time), but can catch it if you import the module from elsewhere.

Relatedly, you could do: try: compile("from __future__ import whatever", "", "exec") except whatever2: whatever3 else: whatever4 Then the future_stmt's compile-time is your module's runtime. still-looks-pretty-useless-to-me-though-ly y'rs - tim

Since a module M containing a future_statement naming feature F explicitly requests that the current release act like a future release with respect to F, any code interpreted dynamically from an eval, exec or execfile executed by M will also use the new syntax or semantics associated with F.

This means that a run-time flag must be available for inspection by eval() and execfile(), at least. I'm not sure that I agree with this for execfile() though -- that's often used by mechanisms that emulate module imports, and there it would be better if it started off with all features reset to their default. I'm also not sure about exec and eval() -- it all depends on the reason why exec is being invoked. Plus, exec and eval() also take a compiled code object, and there it's too late to change the future. Which leads to the question: should compile() also inherit the future settings? It's certainly a lot easier to implement if exec c.s. are *not* affected by the future selection of the invoking environment. And if you *want* it, at least for exec, you can insert the future_statement in front of the executed code string.

Interactive shells may pose special problems. The intent is that a future_statement typed at an interactive shell prompt affect all code typed to that shell for the remaining life of the shell session. It's not clear how to achieve that.

The same flag that I mentioned above can be used here -- basically, we can treat each interactive command as an "exec". Except that this time, the command that is the future_statement *does* export its flag to the invoking environment. Plus, I've made a good case against the flag. :-( --Guido van Rossum (home page: http://www.python.org/~guido/)

Code emulating module imports is rare. People writing such mechanisms had better be experts! I don't want to warp the normal case to cater to a handful of deep-magic propeller-heads (they can take care of themselves).

OK. I'm not completely convinced, but at least 60%, and that's enough. --Guido van Rossum (home page: http://www.python.org/~guido/)

Last week Guido sent a message about our decisions to make nested scopes an optional feature for 2.1 in advance of their mandatory introduction in Python 2.2. I've included an ndiff of the PEP for reference. The beta release on Friday will contain the features as described in the PEP. Jeremy -: old-pep-0227.txt +: pep-0227.txt PEP: 227 Title: Statically Nested Scopes - Version: $Revision: 1.6 $ ? ^ + Version: $Revision: 1.7 $ ? ^ Author: jeremy@digicool.com (Jeremy Hylton) Status: Draft Type: Standards Track Python-Version: 2.1 Created: 01-Nov-2000 Post-History: Abstract This PEP proposes the addition of statically nested scoping (lexical scoping) for Python 2.1. The current language definition defines exactly three namespaces that are used to resolve names -- the local, global, and built-in namespaces. The addition of nested scopes would allow resolution of unbound local names in enclosing functions' namespaces. One consequence of this change that will be most visible to Python programs is that lambda statements could reference variables in the namespaces where the lambda is defined. Currently, a lambda statement uses default arguments to explicitly creating bindings in the lambda's namespace. Introduction This proposal changes the rules for resolving free variables in - Python functions. The Python 2.0 definition specifies exactly - three namespaces to check for each name -- the local namespace, - the global namespace, and the builtin namespace. According to - this defintion, if a function A is defined within a function B, - the names bound in B are not visible in A. The proposal changes - the rules so that names bound in B are visible in A (unless A + Python functions. The new name resolution semantics will take + effect with Python 2.2. These semantics will also be available in + Python 2.1 by adding "from __future__ import nested_scopes" to the + top of a module. + + The Python 2.0 definition specifies exactly three namespaces to + check for each name -- the local namespace, the global namespace, + and the builtin namespace. According to this definition, if a + function A is defined within a function B, the names bound in B + are not visible in A. The proposal changes the rules so that + names bound in B are visible in A (unless A contains a name - contains a name binding that hides the binding in B). ? ---------------- + binding that hides the binding in B). The specification introduces rules for lexical scoping that are common in Algol-like languages. The combination of lexical scoping and existing support for first-class functions is reminiscent of Scheme. The changed scoping rules address two problems -- the limited - utility of lambda statements and the frequent confusion of new + utility of lagmbda statements and the frequent confusion of new ? + users familiar with other languages that support lexical scoping, e.g. the inability to define recursive functions except at the module level. + + XXX Konrad Hinsen suggests that this section be expanded The lambda statement introduces an unnamed function that contains a single statement. It is often used for callback functions. In the example below (written using the Python 2.0 rules), any name used in the body of the lambda must be explicitly passed as a default argument to the lambda. from Tkinter import * root = Tk() Button(root, text="Click here", command=lambda root=root: root.test.configure(text="...")) This approach is cumbersome, particularly when there are several names used in the body of the lambda. The long list of default arguments obscure the purpose of the code. The proposed solution, in crude terms, implements the default argument approach automatically. The "root=root" argument can be omitted. + The new name resolution semantics will cause some programs to + behave differently than they did under Python 2.0. In some cases, + programs will fail to compile. In other cases, names that were + previously resolved using the global namespace will be resolved + using the local namespace of an enclosing function. In Python + 2.1, warnings will be issued for all program statement that will + behave differently. + Specification Python is a statically scoped language with block structure, in the traditional of Algol. A code block or region, such as a - module, class defintion, or function body, is the basic unit of a + module, class definition, or function body, is the basic unit of a ? + program. Names refer to objects. Names are introduced by name binding operations. Each occurrence of a name in the program text refers to the binding of that name established in the innermost function block containing the use. The name binding operations are assignment, class and function definition, and import statements. Each assignment or import statement occurs within a block defined by a class or function definition or at the module level (the top-level code block). If a name binding operation occurs anywhere within a code block, all uses of the name within the block are treated as references to the current block. (Note: This can lead to errors when a name is used within a block before it is bound.) If the global statement occurs within a block, all uses of the name specified in the statement refer to the binding of that name in the top-level namespace. Names are resolved in the top-level namespace by searching the global namespace, the namespace of the module containing the code block, and the builtin namespace, the namespace of the module __builtin__. The global namespace is searched first. If the name is not found there, the builtin - namespace is searched. + namespace is searched. The global statement must precede all uses + of the name. If a name is used within a code block, but it is not bound there and is not declared global, the use is treated as a reference to the nearest enclosing function region. (Note: If a region is contained within a class definition, the name bindings that occur in the class block are not visible to enclosed functions.) A class definition is an executable statement that may uses and definitions of names. These references follow the normal rules for name resolution. The namespace of the class definition becomes the attribute dictionary of the class. The following operations are name binding operations. If they occur within a block, they introduce new local names in the current block unless there is also a global declaration. - Function defintion: def name ... + Function definition: def name ... ? + Class definition: class name ... Assignment statement: name = ... Import statement: import name, import module as name, from module import name Implicit assignment: names are bound by for statements and except clauses The arguments of a function are also local. There are several cases where Python statements are illegal when used in conjunction with nested scopes that contain free variables. If a variable is referenced in an enclosing scope, it is an error to delete the name. The compiler will raise a SyntaxError for 'del name'. - If the wildcard form of import (import *) is used in a function + If the wild card form of import (import *) is used in a function ? + and the function contains a nested block with free variables, the compiler will raise a SyntaxError. If exec is used in a function and the function contains a nested block with free variables, the compiler will raise a SyntaxError - unless the exec explicit specifies the local namespace for the + unless the exec explicitly specifies the local namespace for the ? ++ exec. (In other words, "exec obj" would be illegal, but "exec obj in ns" would be legal.) + If a name bound in a function scope is also the name of a module + global name or a standard builtin name and the function contains a + nested function scope that references the name, the compiler will + issue a warning. The name resolution rules will result in + different bindings under Python 2.0 than under Python 2.2. The + warning indicates that the program may not run correctly with all + versions of Python. + Discussion The specified rules allow names defined in a function to be referenced in any nested function defined with that function. The name resolution rules are typical for statically scoped languages, with three primary exceptions: - Names in class scope are not accessible. - The global statement short-circuits the normal rules. - Variables are not declared. Names in class scope are not accessible. Names are resolved in - the innermost enclosing function scope. If a class defintion + the innermost enclosing function scope. If a class definition ? + occurs in a chain of nested scopes, the resolution process skips class definitions. This rule prevents odd interactions between class attributes and local variable access. If a name binding - operation occurs in a class defintion, it creates an attribute on + operation occurs in a class definition, it creates an attribute on ? + the resulting class object. To access this variable in a method, or in a function nested within a method, an attribute reference must be used, either via self or via the class name. An alternative would have been to allow name binding in class scope to behave exactly like name binding in function scope. This rule would allow class attributes to be referenced either via attribute reference or simple name. This option was ruled out because it would have been inconsistent with all other forms of class and instance attribute access, which always use attribute references. Code that used simple names would have been obscure. The global statement short-circuits the normal rules. Under the proposal, the global statement has exactly the same effect that it - does for Python 2.0. It's behavior is preserved for backwards ? - + does for Python 2.0. Its behavior is preserved for backwards compatibility. It is also noteworthy because it allows name binding operations performed in one block to change bindings in another block (the module). Variables are not declared. If a name binding operation occurs anywhere in a function, then that name is treated as local to the function and all references refer to the local binding. If a reference occurs before the name is bound, a NameError is raised. The only kind of declaration is the global statement, which allows programs to be written using mutable global variables. As a consequence, it is not possible to rebind a name defined in an enclosing scope. An assignment operation can only bind a name in the current scope or in the global scope. The lack of declarations and the inability to rebind names in enclosing scopes are unusual for lexically scoped languages; there is typically a mechanism to create name bindings (e.g. lambda and let in Scheme) and a mechanism to change the bindings (set! in Scheme). XXX Alex Martelli suggests comparison with Java, which does not allow name bindings to hide earlier bindings. Examples A few examples are included to illustrate the way the rules work. XXX Explain the examples >>> def make_adder(base): ... def adder(x): ... return base + x ... return adder >>> add5 = make_adder(5) >>> add5(6) 11 >>> def make_fact(): ... def fact(n): ... if n == 1: ... return 1L ... else: ... return n * fact(n - 1) ... return fact >>> fact = make_fact() >>> fact(7) 5040L >>> def make_wrapper(obj): ... class Wrapper: ... def __getattr__(self, attr): ... if attr[0] != '_': ... return getattr(obj, attr) ... else: ... raise AttributeError, attr ... return Wrapper() >>> class Test: ... public = 2 ... _private = 3 >>> w = make_wrapper(Test()) >>> w.public 2 >>> w._private Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: _private - An example from Tim Peters of the potential pitfalls of nested scopes ? ^ -------------- + An example from Tim Peters demonstrates the potential pitfalls of ? +++ ^^^^^^^^ - in the absence of declarations: + nested scopes in the absence of declarations: ? ++++++++++++++ i = 6 def f(x): def g(): print i # ... # skip to the next page # ... for i in x: # ah, i *is* local to f, so this is what g sees pass g() The call to g() will refer to the variable i bound in f() by the for loop. If g() is called before the loop is executed, a NameError will be raised. XXX need some counterexamples Backwards compatibility There are two kinds of compatibility problems caused by nested scopes. In one case, code that behaved one way in earlier - versions, behaves differently because of nested scopes. In the ? - + versions behaves differently because of nested scopes. In the other cases, certain constructs interact badly with nested scopes and will trigger SyntaxErrors at compile time. The following example from Skip Montanaro illustrates the first kind of problem: x = 1 def f1(): x = 2 def inner(): print x inner() Under the Python 2.0 rules, the print statement inside inner() refers to the global variable x and will print 1 if f1() is called. Under the new rules, it refers to the f1()'s namespace, the nearest enclosing scope with a binding. The problem occurs only when a global variable and a local variable share the same name and a nested function uses that name to refer to the global variable. This is poor programming practice, because readers will easily confuse the two different variables. One example of this problem was found in the Python standard library during the implementation of nested scopes. To address this problem, which is unlikely to occur often, a static analysis tool that detects affected code will be written. - The detection problem is straightfoward. + The detection problem is straightforward. ? + - The other compatibility problem is casued by the use of 'import *' ? - + The other compatibility problem is caused by the use of 'import *' ? + and 'exec' in a function body, when that function contains a nested scope and the contained scope has free variables. For example: y = 1 def f(): exec "y = 'gotcha'" # or from module import * def g(): return y ... At compile-time, the compiler cannot tell whether an exec that - operators on the local namespace or an import * will introduce ? ^^ + operates on the local namespace or an import * will introduce ? ^ name bindings that shadow the global y. Thus, it is not possible to tell whether the reference to y in g() should refer to the global or to a local name in f(). In discussion of the python-list, people argued for both possible interpretations. On the one hand, some thought that the reference in g() should be bound to a local y if one exists. One problem with this interpretation is that it is impossible for a human reader of the code to determine the binding of y by local inspection. It seems likely to introduce subtle bugs. The other interpretation is to treat exec and import * as dynamic features that do not effect static scoping. Under this interpretation, the exec and import * would introduce local names, but those names would never be visible to nested scopes. In the specific example above, the code would behave exactly as it did in earlier versions of Python. - Since each interpretation is problemtatic and the exact meaning ? - + Since each interpretation is problematic and the exact meaning ambiguous, the compiler raises an exception. A brief review of three Python projects (the standard library, Zope, and a beta version of PyXPCOM) found four backwards compatibility issues in approximately 200,000 lines of code. There was one example of case #1 (subtle behavior change) and two examples of import * problems in the standard library. (The interpretation of the import * and exec restriction that was implemented in Python 2.1a2 was much more restrictive, based on language that in the reference manual that had never been enforced. These restrictions were relaxed following the release.) + Compatibility of C API + + The implementation causes several Python C API functions to + change, including PyCode_New(). As a result, C extensions may + need to be updated to work correctly with Python 2.1. + locals() / vars() These functions return a dictionary containing the current scope's local variables. Modifications to the dictionary do not affect the values of variables. Under the current rules, the use of locals() and globals() allows the program to gain access to all the namespaces in which names are resolved. An analogous function will not be provided for nested scopes. Under this proposal, it will not be possible to gain dictionary-style access to all visible scopes. + Warnings and Errors + + The compiler will issue warnings in Python 2.1 to help identify + programs that may not compile or run correctly under future + versions of Python. Under Python 2.2 or Python 2.1 if the + nested_scopes future statement is used, which are collectively + referred to as "future semantics" in this section, the compiler + will issue SyntaxErrors in some cases. + + The warnings typically apply when a function that contains a + nested function that has free variables. For example, if function + F contains a function G and G uses the builtin len(), then F is a + function that contains a nested function (G) with a free variable + (len). The label "free-in-nested" will be used to describe these + functions. + + import * used in function scope + + The language reference specifies that import * may only occur + in a module scope. (Sec. 6.11) The implementation of C + Python has supported import * at the function scope. + + If import * is used in the body of a free-in-nested function, + the compiler will issue a warning. Under future semantics, + the compiler will raise a SyntaxError. + + bare exec in function scope + + The exec statement allows two optional expressions following + the keyword "in" that specify the namespaces used for locals + and globals. An exec statement that omits both of these + namespaces is a bare exec. + + If a bare exec is used in the body of a free-in-nested + function, the compiler will issue a warning. Under future + semantics, the compiler will raise a SyntaxError. + + local shadows global + + If a free-in-nested function has a binding for a local + variable that (1) is used in a nested function and (2) is the + same as a global variable, the compiler will issue a warning. + Rebinding names in enclosing scopes There are technical issues that make it difficult to support rebinding of names in enclosing scopes, but the primary reason that it is not allowed in the current proposal is that Guido is opposed to it. It is difficult to support, because it would require a new mechanism that would allow the programmer to specify that an assignment in a block is supposed to rebind the name in an enclosing block; presumably a keyword or special syntax (x := 3) would make this possible. The proposed rules allow programmers to achieve the effect of rebinding, albeit awkwardly. The name that will be effectively rebound by enclosed functions is bound to a container object. In place of assignment, the program uses modification of the container to achieve the desired effect: def bank_account(initial_balance): balance = [initial_balance] def deposit(amount): balance[0] = balance[0] + amount return balance def withdraw(amount): balance[0] = balance[0] - amount return balance return deposit, withdraw Support for rebinding in nested scopes would make this code clearer. A class that defines deposit() and withdraw() methods and the balance as an instance variable would be clearer still. Since classes seem to achieve the same effect in a more straightforward manner, they are preferred. Implementation The implementation for C Python uses flat closures [1]. Each def or lambda statement that is executed will create a closure if the body of the function or any contained function has free variables. Using flat closures, the creation of closures is somewhat expensive but lookup is cheap. The implementation adds several new opcodes and two new kinds of names in code objects. A variable can be either a cell variable or a free variable for a particular code object. A cell variable is referenced by containing scopes; as a result, the function where it is defined must allocate separate storage for it on each - invocation. A free variable is reference via a function's closure. ? --------- + invocation. A free variable is referenced via a function's ? + + closure. + + The choice of free closures was made based on three factors. + First, nested functions are presumed to be used infrequently, + deeply nested (several levels of nesting) still less frequently. + Second, lookup of names in a nested scope should be fast. + Third, the use of nested scopes, particularly where a function + that access an enclosing scope is returned, should not prevent + unreferenced objects from being reclaimed by the garbage + collector. XXX Much more to say here References [1] Luca Cardelli. Compiling a functional language. In Proc. of the 1984 ACM Conference on Lisp and Functional Programming, pp. 208-217, Aug. 1984 http://citeseer.nj.nec.com/cardelli84compiling.html