[Python-Dev] Nested scopes resolution -- you can breathe again!

[Guido van Rossum]

We (PythonLabs) have received a lot of flak over our plan to introduce
nested scopes despite the fact that it appears to break a small but
significant amount of working code.  We discussed this at an
PythonLabs group meeting today.  After the meeting, Tim posted this
teaser:

> PS: At the internal PythonLabs mtg today, I voted against nested
> scopes.  But also for them.  Leaving that to Jeremy to explain.

After the meeting Jeremy had a four hour commute home due to bad
weather, so let me do the honors for him.  (Jeremy will update the
PEP, implement the feature, and update the documentation, in that
order.)

We have clearly underestimated how much code the nested scopes would
break, but more importantly we have underestimated how much value our
community places on stability.  At the same time we really like nested
scopes, and we would like to see the feature introduced at some point.

So here's the deal: we'll make nested scopes an optional feature in
2.1, default off, selectable on a per-module basis using a mechanism
that's slightly hackish but is guaranteed to be safe.  (See below.)

At the same time, we'll augment the compiler to detect all situations
that will break when nested scopes are introduced in the future, and
issue warnings for those situations.  The idea here is that warnings
don't break code, but encourage folks to fix their code so we can
introduce nested scopes in 2.2.  Given our current pace of releases
that should be about 6 months warning.

These warnings are *not* optional -- they are issued regardless of
whether you select to use nested scopes.  However there is a command
line option (crudest form: -Wi) to disable warnings; there are also
ways to disable them programmatically.  If you want to make sure that
you don't ignore the warnings, there's also a way to turn warnings
into errors (-We from the command line).

How do you select nested scopes?  Tim suggested a mechanism that is
used by the ANSI C committee to enable language features that are
backwards incompatible: they trigger on the import of a specific
previously non-existant header file.  (E.g. after #include
<imaginary.h>, "imaginary" becomes a reserved word.)

The Python equivalent of this is a magical import that is recognized
by the compiler; this was also proposed by David Scherer for making
integer division yield a float.  (See
http://mail.python.org/pipermail/edu-sig/2000-May/000499.html)  You
could say that Perl's "use" statement is similar.

We haven't decided yet which magical import; two proposals are:

    import __nested_scopes__
    from __future__ import nested_scopes

The magical import only affects the source file in which it occurs.
It is recognized by the compiler as it is scanning the source code.
It must appear at the top-level (no "if" or "try" or "def" or anything
else around it) and before any code that could be affected.

We realize that PEP 5 specifies a one-year transition period.  We
believe that that is excessive in this case, and would like to change
the PEP to be more flexible.  (The PEP has questionable status -- it
was never formally discussed.)

We also believe that the magical import mechanism is useful enough to
be reused for other situations like this; Tim will draft a PEP to
describe in excruciating detail.

I thank everybody who gave feedback on this issue.  And thanks to
Jeremy for implementing nested scopes!

--Guido van Rossum (home page: http://www.python.org/~guido/)