This my last effort for today ;).
> On Wed, Feb 21, 2001 at 05:41:41PM -0500, Andrew Kuchling wrote:
> > Compatibility... ay, there's the rub!
> If you include 'ways of thinking' in 'compatibility', I'll agree. Many
> people are used to being able to use exec/from-foo-import-*, and consider it
> part of Python's wonderful flexibility and straightforwardness (I know I do,
> and all my python-proficient and python-learning colleagues do.)
1) I'm convinced that on the long run that both:
- import *
- exec without in
should be deprecated, so we could start issueing warning with 2.1 or 2.2
and make them errors when people get annoyed by the warnings enough ;)
This has nothing to do with nested scopes. So people have time
to change their mind.
2) The actual implementation of nested scopes (with or without compatibilty
hacks) is based on the assumption that
- one can detect lexically scoped variables as up 2.0 python was able
to detect local vars (without the need of explicit declarations) -,
and this is pythonic and neat, so let's do it.
But this thread and the matter of fact that with the implementation
some old code is not more valid or behave in a different way
shows that maybe (I say maybe) this assumption is not completely valid.
It is clear too that this difference between reality and theory has not
that big predictable consequences, it's just annoying for some among us.
But a survey among users to detect the extent of this has started.
But from the theoretical (and maybe PR?) viewpoint the difference
On the other hand the (potential) solution (wich I'm aware open some other
subtle issues to discuss but keep old code working as it was) of using some kind
of explicit declarations is a no-go, no-story. Yes is not that much pythonic...
Is'nt it possible to be all happy?
I'm wondering if we have not transformed in an holy war a problem
that offer at least some space for a technical discussion.
regards, Samuele Pedroni.
PS: sorry for my abuse of we given that I'm jython devel not a python one,
but it is already difficult so... I feel I'm missing something about
this group dynamics.
I'm running into strange problems with import in frozen Mac programs.
On the Mac a program is frozen in a rather different way from how it happens
on Unix/Windows: basically all .pyc files are stuffed into resources, and if
the import code comes across a file on sys.path it will look for PYC resources
in that file. So, you freeze a program by stuffing all your modules into the
interpreter executable as PYC resources and setting sys.path to contain only
the executable file, basically.
This week I noticed that these resource imports have suddenly become very very
slow. Whereas startup time of my application used to be around 2 seconds
(where the non-frozen version took 6 seconds) it now takes almost 20 times as
long. The non-frozen version still takes 6 seconds.
I suspect this may have something to do with recent mods to the import code,
but attempts to pinpoint the problem have failed so far (somehow the profiler
crashes my app). I've put a breakpoint at import.c:check_case(), and it isn't
hit (as is to be expected), so that isn't the problem.
Does anyone have a hint for where I could start looking?
Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++
Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++
www.oratrix.nl/~jack | ++++ see http://www.xs4all.nl/~tank/ ++++
On 10 Feb, GvR quoted and wrote:
>> Skip Montanaro writes:
>> > MAL> The same could be done for e.g. soundex ...
>> Fred Drake wrote:
>> Given that Skip has published this module and that the C version can
>> always be retrieved from CVS if anyone really wants it, and that
>> soundex has been listed in the "Obsolete Modules" section in the
>> documentation for quite some time, this is probably a good time to
>> remove it from the source distribution.
>Yes, go ahead.
Guido, did you mean go ahead and remove soundex, or the dl module, or
while 1: compile("print 'hello'\n", "<string>", "exec")
current CVS leaks just over 1k per call to compile.
1.5.2 and 2.0 doesn't leak a byte.
make the script a little more complex, and it leaks even
more (4k for a small function, 650k for Tkinter.py, etc).
There has been renewed discussion of backwards compatibility issues
introduced by nested scopes. Following some discussion on python-dev,
I have updated the discussion of these issues in the PEP. Of course,
more comments are welcome. I am particularly interested in reports of
actual compatibility issues with existing code, as opposed to
The particular concerns raised lately have to do with previously legal
code that will fail with a SyntaxError with nested scopes. Early in
the design process, there was discussion of code that will behave
differently with nested scopes. At the time, the subtle behavior
change was considered acceptable because it was believed to occur
rarely in practice and was probably hard to understand to begin with.
A related issue, already discussed on both lists, was the restrictions
added in Python 2.1a2 on the use of import * in functions and exec
with nested scope. The former restriction was always documented in
the reference manual, but never enforced. Subsequently, we decided to
allow import * and exec except in cases where the meaning was
ambiguous with respect to nested scopes.
This probably sounds a bit abstract; I hope the PEP (included below)
spells out the issues more clearly. If you have code that currently
depends on any of the three following behaviors, I'd like to hear
- A function is contained within another function. The outer
function contains a local name that shadows a global name. The
inner function uses the global. The one case of this I have
seen in the wild was caused by a local variable named str in the
outer function and a use of builtin str in the inner function.
- A function that contains a nested function with free variables
and also uses exec that does not specify a namespace, e.g.
"exec foo in ns" should be legal, although the current CVS code
base does not yet allow it.
- A function like the one above, except that is uses import *
instead of exec.
Title: Statically Nested Scopes
Version: $Revision: 1.6 $
Author: jeremy(a)digicool.com (Jeremy Hylton)
Type: Standards Track
Post-History: XXX what goes here?
This PEP proposes the addition of statically nested scoping
(lexical scoping) for Python 2.1. The current language definition
defines exactly three namespaces that are used to resolve names --
the local, global, and built-in namespaces. The addition of
nested scopes would allow resolution of unbound local names in
enclosing functions' namespaces.
One consequence of this change that will be most visible to Python
programs is that lambda statements could reference variables in
the namespaces where the lambda is defined. Currently, a lambda
statement uses default arguments to explicitly creating bindings
in the lambda's namespace.
This proposal changes the rules for resolving free variables in
Python functions. The Python 2.0 definition specifies exactly
three namespaces to check for each name -- the local namespace,
the global namespace, and the builtin namespace. According to
this defintion, if a function A is defined within a function B,
the names bound in B are not visible in A. The proposal changes
the rules so that names bound in B are visible in A (unless A
contains a name binding that hides the binding in B).
The specification introduces rules for lexical scoping that are
common in Algol-like languages. The combination of lexical
scoping and existing support for first-class functions is
reminiscent of Scheme.
The changed scoping rules address two problems -- the limited
utility of lambda statements and the frequent confusion of new
users familiar with other languages that support lexical scoping,
e.g. the inability to define recursive functions except at the
The lambda statement introduces an unnamed function that contains
a single statement. It is often used for callback functions. In
the example below (written using the Python 2.0 rules), any name
used in the body of the lambda must be explicitly passed as a
default argument to the lambda.
from Tkinter import *
root = Tk()
Button(root, text="Click here",
command=lambda root=root: root.test.configure(text="..."))
This approach is cumbersome, particularly when there are several
names used in the body of the lambda. The long list of default
arguments obscure the purpose of the code. The proposed solution,
in crude terms, implements the default argument approach
automatically. The "root=root" argument can be omitted.
Python is a statically scoped language with block structure, in
the traditional of Algol. A code block or region, such as a
module, class defintion, or function body, is the basic unit of a
Names refer to objects. Names are introduced by name binding
operations. Each occurrence of a name in the program text refers
to the binding of that name established in the innermost function
block containing the use.
The name binding operations are assignment, class and function
definition, and import statements. Each assignment or import
statement occurs within a block defined by a class or function
definition or at the module level (the top-level code block).
If a name binding operation occurs anywhere within a code block,
all uses of the name within the block are treated as references to
the current block. (Note: This can lead to errors when a name is
used within a block before it is bound.)
If the global statement occurs within a block, all uses of the
name specified in the statement refer to the binding of that name
in the top-level namespace. Names are resolved in the top-level
namespace by searching the global namespace, the namespace of the
module containing the code block, and the builtin namespace, the
namespace of the module __builtin__. The global namespace is
searched first. If the name is not found there, the builtin
namespace is searched.
If a name is used within a code block, but it is not bound there
and is not declared global, the use is treated as a reference to
the nearest enclosing function region. (Note: If a region is
contained within a class definition, the name bindings that occur
in the class block are not visible to enclosed functions.)
A class definition is an executable statement that may uses and
definitions of names. These references follow the normal rules
for name resolution. The namespace of the class definition
becomes the attribute dictionary of the class.
The following operations are name binding operations. If they
occur within a block, they introduce new local names in the
current block unless there is also a global declaration.
Function defintion: def name ...
Class definition: class name ...
Assignment statement: name = ...
Import statement: import name, import module as name,
from module import name
Implicit assignment: names are bound by for statements and except
The arguments of a function are also local.
There are several cases where Python statements are illegal when
used in conjunction with nested scopes that contain free
If a variable is referenced in an enclosing scope, it is an error
to delete the name. The compiler will raise a SyntaxError for
If the wildcard form of import (import *) is used in a function
and the function contains a nested block with free variables, the
compiler will raise a SyntaxError.
If exec is used in a function and the function contains a nested
block with free variables, the compiler will raise a SyntaxError
unless the exec explicit specifies the local namespace for the
exec. (In other words, "exec obj" would be illegal, but
"exec obj in ns" would be legal.)
The specified rules allow names defined in a function to be
referenced in any nested function defined with that function. The
name resolution rules are typical for statically scoped languages,
with three primary exceptions:
- Names in class scope are not accessible.
- The global statement short-circuits the normal rules.
- Variables are not declared.
Names in class scope are not accessible. Names are resolved in
the innermost enclosing function scope. If a class defintion
occurs in a chain of nested scopes, the resolution process skips
class definitions. This rule prevents odd interactions between
class attributes and local variable access. If a name binding
operation occurs in a class defintion, it creates an attribute on
the resulting class object. To access this variable in a method,
or in a function nested within a method, an attribute reference
must be used, either via self or via the class name.
An alternative would have been to allow name binding in class
scope to behave exactly like name binding in function scope. This
rule would allow class attributes to be referenced either via
attribute reference or simple name. This option was ruled out
because it would have been inconsistent with all other forms of
class and instance attribute access, which always use attribute
references. Code that used simple names would have been obscure.
The global statement short-circuits the normal rules. Under the
proposal, the global statement has exactly the same effect that it
does for Python 2.0. It's behavior is preserved for backwards
compatibility. It is also noteworthy because it allows name
binding operations performed in one block to change bindings in
another block (the module).
Variables are not declared. If a name binding operation occurs
anywhere in a function, then that name is treated as local to the
function and all references refer to the local binding. If a
reference occurs before the name is bound, a NameError is raised.
The only kind of declaration is the global statement, which allows
programs to be written using mutable global variables. As a
consequence, it is not possible to rebind a name defined in an
enclosing scope. An assignment operation can only bind a name in
the current scope or in the global scope. The lack of
declarations and the inability to rebind names in enclosing scopes
are unusual for lexically scoped languages; there is typically a
mechanism to create name bindings (e.g. lambda and let in Scheme)
and a mechanism to change the bindings (set! in Scheme).
XXX Alex Martelli suggests comparison with Java, which does not
allow name bindings to hide earlier bindings.
A few examples are included to illustrate the way the rules work.
XXX Explain the examples
>>> def make_adder(base):
... def adder(x):
... return base + x
... return adder
>>> add5 = make_adder(5)
>>> def make_fact():
... def fact(n):
... if n == 1:
... return 1L
... return n * fact(n - 1)
... return fact
>>> fact = make_fact()
>>> def make_wrapper(obj):
... class Wrapper:
... def __getattr__(self, attr):
... if attr != '_':
... return getattr(obj, attr)
... raise AttributeError, attr
... return Wrapper()
>>> class Test:
... public = 2
... _private = 3
>>> w = make_wrapper(Test())
Traceback (most recent call last):
File "<stdin>", line 1, in ?
An example from Tim Peters of the potential pitfalls of nested scopes
in the absence of declarations:
i = 6
# skip to the next page
for i in x: # ah, i *is* local to f, so this is what g sees
The call to g() will refer to the variable i bound in f() by the for
loop. If g() is called before the loop is executed, a NameError will
XXX need some counterexamples
There are two kinds of compatibility problems caused by nested
scopes. In one case, code that behaved one way in earlier
versions, behaves differently because of nested scopes. In the
other cases, certain constructs interact badly with nested scopes
and will trigger SyntaxErrors at compile time.
The following example from Skip Montanaro illustrates the first
kind of problem:
x = 1
x = 2
Under the Python 2.0 rules, the print statement inside inner()
refers to the global variable x and will print 1 if f1() is
called. Under the new rules, it refers to the f1()'s namespace,
the nearest enclosing scope with a binding.
The problem occurs only when a global variable and a local
variable share the same name and a nested function uses that name
to refer to the global variable. This is poor programming
practice, because readers will easily confuse the two different
variables. One example of this problem was found in the Python
standard library during the implementation of nested scopes.
To address this problem, which is unlikely to occur often, a
static analysis tool that detects affected code will be written.
The detection problem is straightfoward.
The other compatibility problem is casued by the use of 'import *'
and 'exec' in a function body, when that function contains a
nested scope and the contained scope has free variables. For
y = 1
exec "y = 'gotcha'" # or from module import *
At compile-time, the compiler cannot tell whether an exec that
operators on the local namespace or an import * will introduce
name bindings that shadow the global y. Thus, it is not possible
to tell whether the reference to y in g() should refer to the
global or to a local name in f().
In discussion of the python-list, people argued for both possible
interpretations. On the one hand, some thought that the reference
in g() should be bound to a local y if one exists. One problem
with this interpretation is that it is impossible for a human
reader of the code to determine the binding of y by local
inspection. It seems likely to introduce subtle bugs. The other
interpretation is to treat exec and import * as dynamic features
that do not effect static scoping. Under this interpretation, the
exec and import * would introduce local names, but those names
would never be visible to nested scopes. In the specific example
above, the code would behave exactly as it did in earlier versions
Since each interpretation is problemtatic and the exact meaning
ambiguous, the compiler raises an exception.
A brief review of three Python projects (the standard library,
Zope, and a beta version of PyXPCOM) found four backwards
compatibility issues in approximately 200,000 lines of code.
There was one example of case #1 (subtle behavior change) and two
examples of import * problems in the standard library.
(The interpretation of the import * and exec restriction that was
implemented in Python 2.1a2 was much more restrictive, based on
language that in the reference manual that had never been
enforced. These restrictions were relaxed following the release.)
locals() / vars()
These functions return a dictionary containing the current scope's
local variables. Modifications to the dictionary do not affect
the values of variables. Under the current rules, the use of
locals() and globals() allows the program to gain access to all
the namespaces in which names are resolved.
An analogous function will not be provided for nested scopes.
Under this proposal, it will not be possible to gain
dictionary-style access to all visible scopes.
Rebinding names in enclosing scopes
There are technical issues that make it difficult to support
rebinding of names in enclosing scopes, but the primary reason
that it is not allowed in the current proposal is that Guido is
opposed to it. It is difficult to support, because it would
require a new mechanism that would allow the programmer to specify
that an assignment in a block is supposed to rebind the name in an
enclosing block; presumably a keyword or special syntax (x := 3)
would make this possible.
The proposed rules allow programmers to achieve the effect of
rebinding, albeit awkwardly. The name that will be effectively
rebound by enclosed functions is bound to a container object. In
place of assignment, the program uses modification of the
container to achieve the desired effect:
balance = [initial_balance]
balance = balance + amount
balance = balance - amount
return deposit, withdraw
Support for rebinding in nested scopes would make this code
clearer. A class that defines deposit() and withdraw() methods
and the balance as an instance variable would be clearer still.
Since classes seem to achieve the same effect in a more
straightforward manner, they are preferred.
The implementation for C Python uses flat closures . Each def
or lambda statement that is executed will create a closure if the
body of the function or any contained function has free
variables. Using flat closures, the creation of closures is
somewhat expensive but lookup is cheap.
The implementation adds several new opcodes and two new kinds of
names in code objects. A variable can be either a cell variable
or a free variable for a particular code object. A cell variable
is referenced by containing scopes; as a result, the function
where it is defined must allocate separate storage for it on each
invocation. A free variable is reference via a function's closure.
XXX Much more to say here
 Luca Cardelli. Compiling a functional language. In Proc. of
the 1984 ACM Conference on Lisp and Functional Programming,
pp. 208-217, Aug. 1984
I should reformulate:
I think a possible not arbitrary rule for exec is
only exec ... in ... is valid, but this also something ok
only on the long-run (like import * deprecation).
Then it is necessary to agree on the semantic of locals().
What would happen right now mixing lexical scoping and
exec ... in locals()?
regards, Samuele Pedroni.
So few code breakage is nice.
> I also realized that the exec rule is still too string. If the exec
> statement passes an explicit namespace -- "exec in foo" -- then there
> shouldn't be any problem, because the executed code can't affect the
> current namespace. If this form is allowed, the exec errors in xpcom
> and Zope disappear.
My very personal feeling is that *any* rule on exec just sounds
arbitrary (even if motived and acceptable).
regards, Samuele Pedroni.
> Samuele wrote:
> > On the other hand just saying that new feature X make code Y (previously
> > meaningless and so the unique solution is to discard Y as garbage,
> > is something that cannot be sold for cheap. I have the feeling that this
> > is the *point*.
> I don't mind new features if I can chose to ignore them...
Along this line of thought and summarizing:
- import * (in an inner scope) is somehow a problem but on the long run it
should be likely deprecated and become an error anyway.
- mixing of inner defs or lambdas and exec is a real issue
(Mark Hammond original posting was caused but such a situation):
for that there is no clear workaround: I repeat
if we want 2 as return value it's a mess (the problem could end up being
more perfomance than complexity, altough simple impl is a long-run win).
Developing special rules is also not that simple: just put an y = 9
before the exec, what is expected then? This promises lot of confusion.
- I'm not a partisan of this, but if we want to able
to "choose to ignore" lexical scoping, we will need to make
its activation explicit.
but this has been discarded, so no story...
Implicit scoping semantic has been changed and now we just have to convince
ourself that this is a win, and there is no big code breakage (this is very
likely, without irony)
and that transforming working code (I'm referring to code using 'exec's not
import *) in invalid code is just natural language evolution
that users will understand <wink>.
We can make the transition more smooth:
> >IMO breaking code would be ok if we issue warnings today and implement
> >nested scopes issuing errors tomorrow. But this is simply a statement
> >about principles and raised impression.
> Agreed. So maybe that's the best solution: pull nested scopes from
> 2.1 and add a warning for from...import (and exec?) inside a function
> using nested scopes, and only add nested scopes in 2.2, after everyone
> has had 6 months or a year to fix their code.
But the problem with exec will remain.
PS: to be honest the actual impl of nested scope is fine for me from the
viewpoint of the guy that should implement that for jython ;).