[Python-checkins] peps: Updating PEP 3150's statement local namespaces to be substantially simpler and

nick.coghlan python-checkins at python.org
Wed Sep 5 13:15:11 CEST 2012


http://hg.python.org/peps/rev/2f04a362718c
changeset:   4507:2f04a362718c
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Wed Sep 05 21:15:00 2012 +1000
summary:
  Updating PEP 3150's statement local namespaces to be substantially simpler and easier to understand, and make PEP 403 and PEP 3150 direct competitors again

files:
  pep-0403.txt |  149 +++-----
  pep-3150.txt |  650 +++++++++++++++++---------------------
  2 files changed, 361 insertions(+), 438 deletions(-)


diff --git a/pep-0403.txt b/pep-0403.txt
--- a/pep-0403.txt
+++ b/pep-0403.txt
@@ -1,5 +1,5 @@
 PEP: 403
-Title: General purpose decorator clause
+Title: General purpose decorator clause (aka "@in" clause)
 Version: $Revision$
 Last-Modified: $Date$
 Author: Nick Coghlan <ncoghlan at gmail.com>
@@ -209,11 +209,16 @@
 the simple name binding of a standard function definition to be replaced
 with something else (like assigning the result of the function to a value).
 
-This PEP also achieves most of the other effects described in PEP 3150
-without introducing a new brainbending kind of scope. All of the complex
-scoping rules in PEP 3150 are replaced in this PEP with allowing a forward
-reference to the associated function or class definition without creating an
-actual name binding in the current scope.
+Despite having the same author, the two PEPs are in direct competition with
+each other. This PEP represents a minimalist approach that attempts to
+achieve useful functionality with a minimum of change from the status quo.
+PEP 3150 instead aims for a more flexible standalone statement design, which
+requires a larger degree of change to the language.
+
+Note that where this PEP is better suited to explaining the behaviour of
+generator expressions correctly, PEP 3150 is better able to explain the
+behaviour of decorator clauses in general. Both PEPs support adequate
+explanations for the semantics of container comprehensions.
 
 
 Keyword Choice
@@ -284,6 +289,47 @@
 nested suite).
 
 
+Explaining Container Comprehensions and Generator Expressions
+-------------------------------------------------------------
+
+One interesting feature of the proposed construct is that it can be used as
+a primitive to explain the scoping and execution order semantics of
+both generator expressions and container comprehensions::
+
+    seq2 = [x for x in y if q(x) for y in seq if p(y)]
+
+    # would be equivalent to
+
+    @in seq2 = f(seq):
+    def f(seq)
+        result = []
+        for y in seq:
+            if p(y):
+                for x in y:
+                    if q(x):
+                        result.append(x)
+        return result
+
+The important point in this expansion is that it explains why comprehensions
+appear to misbehave at class scope: only the outermost iterator is evaluated
+at class scope, while all predicates, nested iterators and value expressions
+are evaluated inside a nested scope.
+
+An equivalent expansion is possible for generator expressions::
+
+    gen = (x for x in y if q(x) for y in seq if p(y))
+
+    # would be equivalent to
+
+    @in gen = g(seq):
+    def g(seq)
+        for y in seq:
+            if p(y):
+                for x in y:
+                    if q(x):
+                        yield x
+
+
 More Examples
 =============
 
@@ -413,18 +459,19 @@
 
 The problems with using a full nested suite are best described in
 PEP 3150. It's comparatively difficult to implement properly, the scoping
-semantics are hard to explain and it creates quite a few situations where
+semantics are harder to explain and it creates quite a few situations where
 there are two ways to do it without clear guidelines for choosing between
 them (as almost any construct that can be expressed with ordinary imperative
-code could instead be expressed using a given statement). While the PEP did
+code could instead be expressed using a given statement). While the PEP does
 propose some new PEP 8 guidelines to help address that last problem, the
-difficulties in implementation and explanation are not so easily dealt with.
+difficulties in implementation are not so easily dealt with.
 
 By contrast, the decorator inspired syntax in this PEP explicitly limits the
 new feature to cases where it should actually improve readability, rather
 than harming it. As in the case of the original introduction of decorators,
 the idea of this new syntax is that if it *can* be used (i.e. the local name
-binding of the function is completely unnecessary) then it *should* be used.
+binding of the function is completely unnecessary) then it probably *should*
+be used.
 
 Another possible variant of this idea is to keep the decorator based
 *semantics* of this PEP, while adopting the prettier syntax from PEP 3150::
@@ -438,82 +485,14 @@
 one. A secondary concern is that it's not clear how the compiler will know
 which name(s) in the leading expression are forward references (although
 that could potentially be addressed through a suitable definition of the
-suite-that-is-not-a-suite in the languge grammar).
+suite-that-is-not-a-suite in the language grammar).
 
-There's yet another possibility which would require names to be explicitly
-lifted out of the private namespace, eliminating much of the implementation
-complexity discussed in PEP 3150 (as only identifiers would be exported,
-rather than arbitrary subexpressions)::
-
-    x = weakref.ref(target, report_destruction) given report_destruction from:
-        def report_destruction(obj):
-            print("{} is being destroyed".format(obj))
-
-or even::
-
-    x = weakref.ref(target, f) given report_destruction as f from:
-        def report_destruction(obj):
-            print("{} is being destroyed".format(obj))
-
-This approach actually has much to recommend it. It's as powerful as
-PEP 3150 while being substantially simpler both to implement and to
-explain. The main downside is that it requires even *more* repetition
-of names than is needed with the proposed ``@in`` decorator based syntax
-or with the status quo where the function is defined and bound to a local
-name prior to its sole use.
-
-Some more examples using this last variant::
-
-    sorted_list = sorted(original, key=f) given f from:
-        def f(item):
-            try:
-                return item.calc_sort_order()
-            except NotSortableError:
-                return float('inf')
-
-    funcs = [adder(i) for i in range(10)] given adder from:
-        def adder(i):
-            return lambda x: x + i
-    
-    environ = _createenviron() given _createenviron from:
-        def _createenviron():
-            ... # 27 line function
-
-    c = math.sqrt(a*a + b*b) given a, b from:
-        a = calculate_a()
-        b = calculate_b()
-
-    dispatch[MyClass] = f given f from:
-        def f():
-            ...
-
-    f() given f from:
-        def f():
-            ...
-
-One interesting outcome of such an approach is that it allows the semantics
-of function decoration to be represented almost exactly as a specific
-case of the new syntax::
-
-    # @deco2
-    # @deco1
-    # def name():
-    #    ...
-
-    name = f given f from:
-        # Decorator expressions are evaluated first, in code order
-        _d2 = deco2
-        _d1 = deco1
-
-        # The base function object is defined
-        def name():
-            ...
-
-        # Decorators are applied innermost first
-        f = _d2(_d1(name))
-
-Hmm, perhaps PEP 3150 isn't *quite* as dead as I thought, although the
-triple repetition of names is still a rather large downside.
+However, a nested suite has not yet been ruled out completely. The latest
+version of PEP 3150 uses explicit forward reference and name binding
+schemes that greatly simplify the semantics of the statement, and it
+does offer the advantage of allowing the definition of arbitrary
+subexpressions rather than being restricted to a single function or
+class definition.
 
 
 References
diff --git a/pep-3150.txt b/pep-3150.txt
--- a/pep-3150.txt
+++ b/pep-3150.txt
@@ -3,7 +3,7 @@
 Version: $Revision$
 Last-Modified: $Date$
 Author: Nick Coghlan <ncoghlan at gmail.com>
-Status: Withdrawn
+Status: Deferred
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 2010-07-09
@@ -19,7 +19,9 @@
 Python statements that do not currently have an associated code suite. This
 clause will create a statement local namespace for additional names that are
 accessible in the associated statement, but do not become part of the
-containing namespace.
+containing namespace. To permit a sane implementation strategy, forward
+references to names from the ``given`` clause will need to be marked
+explicitly.
 
 The primary motivation is to enable a more declarative style of programming,
 where the operation to be performed is presented to the reader first, and the
@@ -47,23 +49,6 @@
 but has not yet itself been subject to the test of implementation.
 
 
-PEP Withdrawal
-==============
-
-I've had a complicated history with this PEP. For a long time I left it in
-the Deferred state because I wasn't convinced the additional complexity was
-worth the payoff. Then, briefly, I became more enamoured of the idea and
-only left it at Deferred because I didn't really have time to pursue it.
-
-I'm now withdrawing it, as, the longer I reflect on the topic, the more I
-feel this approach is simply far too intrusive and complicated to ever be
-a good idea for Python as a language.
-
-I've also finally found a couple of syntax proposals for PEP 403 that
-read quite nicely and address the same set of use cases as this PEP
-while remaining significantly simpler.
-
-
 Proposal
 ========
 
@@ -87,16 +72,171 @@
 name in the header line, with the actual definitions following in
 the indented clause. As a simple example::
 
-   sorted_data = sorted(data, key=sort_key) given:
+   sorted_data = sorted(data, key=.sort_key) given:
        def sort_key(item):
            return item.attr1, item.attr2
 
+The leading ``.`` on ``.sort_key`` indicates to the compiler that this
+is a forward reference to a name defined in the ``given`` clause.
+
 The ``pass`` statement is included to provide a consistent way to skip
 inclusion of a meaningful expression in the header line. While this is not
 an intended use case, it isn't one that can be prevented as multiple
 alternatives (such as ``...`` and ``()``) remain available even if ``pass``
 itself is disallowed.
 
+The body of the given clause will execute in a new scope, using normal
+function closure semantics. To support early binding of loop variables
+and global references, as well as to allow access to other names defined at
+class scope, the ``given`` clause will also allow explicit
+binding operations in the header line::
+
+   # Explicit early binding via given clause
+   seq = []
+   for i in range(10):
+       seq.append(.f) given i=i:
+           def f():
+               return i
+   assert [f() for f in seq] == list(range(10))
+
+
+Semantics
+---------
+
+The following statement::
+
+   op(.f, .g) given bound_a=a, bound_b=b:
+       def f():
+           return bound_a + bound_b
+       def g():
+           return bound_a - bound_b
+
+Would be roughly equivalent to the following code (``__var`` denotes a
+hidden compiler variable or simply an entry on the interpreter stack)::
+
+   __arg1 = a
+   __arg2 = b
+   def __scope(bound_a, bound_b):
+       def f():
+           return bound_a + bound_b
+       def g():
+           return bound_a - bound_b
+      return f, g
+   __ref1, __ref2 = __scope(__arg1)
+   op(__ref1, __ref2)
+
+A ``given`` clause is essentially a nested function which is created and
+then immediately executed. Unless explicitly passed in, names are looked
+up using normal scoping rules, and thus names defined at class scope will
+not be visible. Names declared as forward references are returned and
+used in the header statement, without being bound locally in the
+surrounding namespace.
+
+
+Syntax Change
+-------------
+
+Current::
+
+   expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
+                ('=' (yield_expr|testlist_star_expr))*)
+   del_stmt: 'del' exprlist
+   pass_stmt: 'pass'
+   return_stmt: 'return' [testlist]
+   yield_stmt: yield_expr
+   raise_stmt: 'raise' [test ['from' test]]
+   assert_stmt: 'assert' test [',' test]
+
+
+New::
+
+   expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
+                ('=' (yield_expr|testlist_star_expr))*) [given_clause]
+   del_stmt: 'del' exprlist [given_clause]
+   pass_stmt: 'pass' [given_clause]
+   return_stmt: 'return' [testlist] [given_clause]
+   yield_stmt: yield_expr [given_clause]
+   raise_stmt: 'raise' [test ['from' test]] [given_clause]
+   assert_stmt: 'assert' test [',' test] [given_clause]
+   given_clause: "given" (NAME '=' test)* ":" suite
+
+(Note that ``expr_stmt`` in the grammar is a slight misnomer, as it covers
+assignment and augmented assignment in addition to simple expression
+statements)
+
+The new clause is added as an optional element of the existing statements
+rather than as a new kind of compound statement in order to avoid creating
+an ambiguity in the grammar. It is applied only to the specific elements
+listed so that nonsense like the following is disallowed::
+
+   break given:
+       a = b = 1
+
+   import sys given:
+       a = b = 1
+
+However, the precise Grammar change described above is inadequate, as it
+creates problems for the definition of simple_stmt (which allows chaining of
+multiple single line statements with ";" rather than "\\n").
+
+So the above syntax change should instead be taken as a statement of intent.
+Any actual proposal would need to resolve the simple_stmt parsing problem
+before it could be seriously considered. This would likely require a
+non-trivial restructuring of the grammar, breaking up small_stmt and
+flow_stmt to separate the statements that potentially contain arbitrary
+subexpressions and then allowing a single one of those statements with
+a ``given`` clause at the simple_stmt level. Something along the lines of::
+
+   stmt: simple_stmt | given_stmt | compound_stmt
+   simple_stmt: small_stmt (';' (small_stmt | subexpr_stmt))* [';'] NEWLINE
+   small_stmt: (pass_stmt | flow_stmt | import_stmt |
+                global_stmt | nonlocal_stmt)
+   flow_stmt: break_stmt | continue_stmt
+   given_stmt: subexpr_stmt (given_clause |
+                 (';' (small_stmt | subexpr_stmt))* [';']) NEWLINE
+   subexpr_stmt: expr_stmt | del_stmt | flow_subexpr_stmt | assert_stmt
+   flow_subexpr_stmt: return_stmt | raise_stmt | yield_stmt
+   given_clause: "given" (NAME '=' test)* ":" suite
+
+For reference, here are the current definitions at that level::
+
+   stmt: simple_stmt | compound_stmt
+   simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
+   small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
+                import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
+   flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt
+
+In addition to the above changes, the definition of ``atom`` would be changed
+to also allow ``"." NAME``. The restriction of this usage to statements with
+an associated ``given`` clause would be handled by a later stage of the
+compilation process (likely AST construction, which already enforces
+other restrictions where the grammar is overly permissive in order to
+simplify the initial parsing step).
+
+
+New PEP 8 Guidelines
+--------------------
+
+As discussed on python-ideas ([7]_, [9]_) new PEP 8 guidelines would also
+need to be developed to provide appropriate direction on when to use the
+``given`` clause over ordinary variable assignments.
+
+Based on the similar guidelines already present for ``try`` statements, this
+PEP proposes the following additions for ``given`` statements to the
+"Programming Conventions" section of PEP 8:
+
+  - for code that could reasonably be factored out into a separate function,
+    but is not currently reused anywhere, consider using a ``given`` clause.
+    This clearly indicates which variables are being used only to define
+    subcomponents of another statement rather than to hold algorithm or
+    application state. This is an especially useful technique when
+    passing multi-line functions to operations which take callable
+    arguments.
+
+  - keep ``given`` clauses concise. If they become unwieldy, either break
+    them up into multiple steps or else move the details into a separate
+    function.
+
 
 Rationale
 =========
@@ -152,11 +292,14 @@
 One potentially useful way to think of the proposed clause is as a middle
 ground between conventional in-line code and separation of an
 operation out into a dedicated function, just as an inline while loop may
-eventually be factored out into a dedicator generator.
+eventually be factored out into a dedicated generator.
 
 
+Design Discussion
+=================
+
 Keyword Choice
-==============
+--------------
 
 This proposal initially used ``where`` based on the name of a similar
 construct in Haskell. However, it has been pointed out that there
@@ -177,12 +320,86 @@
 That way lies C++ and Perl :)
 
 
+Relation to PEP 403
+-------------------
+
+PEP 403 (General Purpose Decorator Clause) attempts to achieve the main
+goals of this PEP using a less radical language change inspired by the
+existing decorator syntax.
+
+Despite having the same author, the two PEPs are in direct competition with
+each other. PEP 403 represents a minimalist approach that attempts to achieve
+useful functionality with a minimum of change from the status quo. This PEP
+instead aims for a more flexible standalone statement design, which requires
+a larger degree of change to the language.
+
+Note that where PEP 403 is better suited to explaining the behaviour of
+generator expressions correctly, this PEP is better able to explain the
+behaviour of decorator clauses in general. Both PEPs support adequate
+explanations for the semantics of container comprehensions.
+
+
+Explaining Container Comprehensions and Generator Expressions
+-------------------------------------------------------------
+
+One interesting feature of the proposed construct is that it can be used as
+a primitive to explain the scoping and execution order semantics of
+container comprehensions::
+
+    seq2 = [x for x in y if q(x) for y in seq if p(y)]
+
+    # would be equivalent to
+
+    seq2 = .result given seq=seq:
+        result = []
+        for y in seq:
+            if p(y):
+                for x in y:
+                    if q(x):
+                        result.append(x)
+
+The important point in this expansion is that it explains why comprehensions
+appear to misbehave at class scope: only the outermost iterator is evaluated
+at class scope, while all predicates, nested iterators and value expressions
+are evaluated inside a nested scope.
+
+Not that, unlike PEP 403, the current version of this PEP *cannot*
+provide a precisely equivalent expansion for a generator expression. The
+closest it can get is to define an additional level of scoping::
+
+    seq2 = .g(seq) given:
+        def g(seq):
+            for y in seq:
+                if p(y):
+                    for x in y:
+                        if q(x):
+                            yield x
+
+Explaining Decorator Clause Evaluation and Application
+------------------------------------------------------
+
+The standard explanation of decorator clause evaluation and application
+has to deal with the idea of hidden compiler variables in order to show
+steps in their order of execution. The given statement allows a decorated
+function definition like::
+
+   @classmethod
+   def classname(cls):
+       return cls.__name__
+
+To instead be explained as roughly equivalent to::
+
+   classname = .d1(classname) given:
+       d1 = classmethod
+       def classname(cls):
+           return cls.__name__
+
 Anticipated Objections
-======================
+----------------------
 
 
 Two Ways To Do It
------------------
+~~~~~~~~~~~~~~~~~
 
 A lot of code may now be written with values defined either before the
 expression where they are used or afterwards in a ``given`` clause, creating
@@ -203,11 +420,11 @@
 Opinions differ, and that's OK.
 
 However, explicit PEP 8 guidance will be needed for CPython and the standard
-library, and that is discussed below.
+library, and that is discussed in the proposal above.
 
 
 Out of Order Execution
-----------------------
+~~~~~~~~~~~~~~~~~~~~~~
 
 The ``given`` clause makes execution jump around a little strangely, as the
 body of the ``given`` clause is executed before the simple statement in the
@@ -231,7 +448,7 @@
    sorted_list = sorted(original, key=(lambda v: v.attr1, v.attr2))
 
 That gets the job done, but it hardly reaches the standard of ``executable
-pseudocode`` that Python aspires to, does it?
+pseudocode`` that fits Python's reputation.
 
 If they don't like ``lambda`` specifically, the ``operator`` module offers an
 alternative that still allows the key function to be defined inline::
@@ -239,7 +456,8 @@
    sorted_list = sorted(original,
                         key=operator.attrgetter(v. 'attr1', 'attr2'))
 
-Again, it gets the job done, but executable pseudocode it ain't.
+Again, it gets the job done, but even the most generous of readers would
+ not consider that to be "executable pseudocode".
 
 If they think both of the above options are ugly and confusing, or they need
 logic in their key function that can't be expressed as an expression (such
@@ -256,47 +474,20 @@
 but it really does represent a break between what the user is thinking and
 what the language allows them to express.
 
-I believe the proposal in this PEP will finally let Python get close to the
+I believe the proposal in this PEP would finally let Python get close to the
 "executable pseudocode" bar for the kind of thought expressed above::
 
-   sorted_list = sorted(original, key=sort_key) given:
+   sorted_list = sorted(original, key=.sort_key) given:
        def sort_key(item):
            return item.attr1, item.attr2
 
 Everything is in the same order as it was in the user's original thought, the
 only addition they have to make is to give the sorting criteria a name so that
 the usage can be linked up to the subsequent definition.
-
-One other useful note on this front, is that this PEP allows existing out of
-order execution constructs to be described as special cases of the more
-general out of order execution syntax (just as comprehensions are now special
-cases of the more general generator expression syntax, even though list
-comprehensions existed first)::
-
-   @classmethod
-   def classname(cls):
-       return cls.__name__
-
-Would be roughly equivalent to::
-
-   classname = f1(classname) given:
-       f1 = classmethod
-       def classname(cls):
-           return cls.__name__
-
-A list comprehension like ``squares = [x*x for x in range(10)]``
-would be equivalent to::
-
-   # Note: this example uses an explicit early binding variant that
-   # isn't yet reflected in the rest of the PEP. It will get there, though.
-   squares = seq given outermost=range(10):
-       seq = []
-       for x in outermost:
-           seq.append(x*x)
    
 
 Harmful to Introspection
-------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~
 
 Poking around in module and class internals is an invaluable tool for
 white-box testing and interactive debugging. The ``given`` clause will be
@@ -314,7 +505,7 @@
 
 
 Lack of Real World Impact Assessment
-------------------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The examples in the current PEP are almost all relatively small "toy"
 examples. The proposal in this PEP needs to be subjected to the test of
@@ -322,261 +513,32 @@
 Twisted application) in a search for examples where the readability of real
 world code is genuinely enhanced.
 
-This is more of a deficiency in the PEP rather than the idea, though.
+This is more of a deficiency in the PEP rather than the idea, though. If
+it wasn't a real world problem, we wouldn't get so many complaints about
+the lack of multi-line lambda support and Ruby's block construct
+probaly wouldn't be quite so popular.
 
 
-New PEP 8 Guidelines
-====================
+Open Questions
+==============
 
-As discussed on python-ideas ([7]_, [9]_) new PEP 8 guidelines would also
-need to be developed to provide appropriate direction on when to use the
-``given`` clause over ordinary variable assignments.
-
-Based on the similar guidelines already present for ``try`` statements, this
-PEP proposes the following additions for ``given`` statements to the
-"Programming Conventions" section of PEP 8:
-
-  - for code that could reasonably be factored out into a separate function,
-    but is not currently reused anywhere, consider using a ``given`` clause.
-    This clearly indicates which variables are being used only to define
-    subcomponents of another statement rather than to hold algorithm or
-    application state.
-
-  - keep ``given`` clauses concise. If they become unwieldy, either break
-    them up into multiple steps or else move the details into a separate
-    function.
-
-
-Syntax Change
-=============
-
-Current::
-
-   expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
-                ('=' (yield_expr|testlist_star_expr))*)
-   del_stmt: 'del' exprlist
-   pass_stmt: 'pass'
-   return_stmt: 'return' [testlist]
-   yield_stmt: yield_expr
-   raise_stmt: 'raise' [test ['from' test]]
-   assert_stmt: 'assert' test [',' test]
-
-
-New::
-
-   expr_stmt: testlist_star_expr (augassign (yield_expr|testlist) |
-                ('=' (yield_expr|testlist_star_expr))*) [given_clause]
-   del_stmt: 'del' exprlist [given_clause]
-   pass_stmt: 'pass' [given_clause]
-   return_stmt: 'return' [testlist] [given_clause]
-   yield_stmt: yield_expr [given_clause]
-   raise_stmt: 'raise' [test ['from' test]] [given_clause]
-   assert_stmt: 'assert' test [',' test] [given_clause]
-   given_clause: "given" ":" suite
-
-(Note that ``expr_stmt`` in the grammar is a slight misnomer, as it covers
-assignment and augmented assignment in addition to simple expression
-statements)
-
-The new clause is added as an optional element of the existing statements
-rather than as a new kind of compound statement in order to avoid creating
-an ambiguity in the grammar. It is applied only to the specific elements
-listed so that nonsense like the following is disallowed::
-
-   break given:
-       a = b = 1
-
-   import sys given:
-       a = b = 1
-
-However, the precise Grammar change described above is inadequate, as it
-creates problems for the definition of simple_stmt (which allows chaining of
-multiple single line statements with ";" rather than "\\n").
-
-So the above syntax change should instead be taken as a statement of intent.
-Any actual proposal would need to resolve the simple_stmt parsing problem
-before it could be seriously considered. This would likely require a
-non-trivial restructuring of the grammar, breaking up small_stmt and
-flow_stmt to separate the statements that potentially contain arbitrary
-subexpressions and then allowing a single one of those statements with
-a ``given`` clause at the simple_stmt level. Something along the lines of::
-
-   stmt: simple_stmt | given_stmt | compound_stmt
-   simple_stmt: small_stmt (';' (small_stmt | subexpr_stmt))* [';'] NEWLINE
-   small_stmt: (pass_stmt | flow_stmt | import_stmt |
-                global_stmt | nonlocal_stmt)
-   flow_stmt: break_stmt | continue_stmt
-   given_stmt: subexpr_stmt (given_clause |
-                 (';' (small_stmt | subexpr_stmt))* [';']) NEWLINE
-   subexpr_stmt: expr_stmt | del_stmt | flow_subexpr_stmt | assert_stmt
-   flow_subexpr_stmt: return_stmt | raise_stmt | yield_stmt
-   given_clause: "given" ":" suite
-
-For reference, here are the current definitions at that level::
-
-   stmt: simple_stmt | compound_stmt
-   simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
-   small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
-                import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
-   flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt
-
-
-Possible Implementation Strategy
-================================
-
-Torture Test
-------------
-
-An implementation of this PEP should support execution of the following
-code at module, class and function scope::
-
-   b = {}
-   a = b[f(a)] = x given:
-       x = 42
-       def f(x):
-           return x
-   assert "x" not in locals()
-   assert "f" not in locals()
-   assert a == 42
-   assert d[42] == 42 given:
-       d = b
-   assert "d" not in locals()
-   y = y given:
-       x = 42
-       def f(): pass
-       y = locals("x"), f.__name__
-   assert "x" not in locals()
-   assert "f" not in locals()
-   assert y == (42, "f")
-
-Most naive implementations will choke on the first complex assignment,
-while less naive but still broken implementations will fail when
-the torture test is executed at class scope. Renaming based strategies
-struggle to support ``locals()`` correctly and also have problems with
-class and function ``__name__`` attributes.
-
-And yes, that's a perfectly well-defined assignment statement. Insane,
-you might rightly say, but legal::
-
-   >>> def f(x): return x
-   ...
-   >>> x = 42
-   >>> b = {}
-   >>> a = b[f(a)] = x
-   >>> a
-   42
-   >>> b
-   {42: 42}
-
-Details of Proposed Semantics
+Syntax for Forward References
 -----------------------------
 
-AKA How Class Scopes Screw You When Attempting To Implement This
+The leading ``.`` arguably fails the "syntax shall not look like grit on
+Uncle Tim's monitor" test. However, it does have the advantages of being
+easy to type and already having an association with namespaces.
 
-The natural idea when setting out to implement this concept is to
-use an ordinary nested function scope. This doesn't work for the
-two reasons mentioned in the Torture Test section above:
 
-* Non-local variables are not your friend because they ignore class scopes
-  and (when writing back to the outer scope) aren't really on speaking
-  terms with module scopes either.
+Handling of ``nonlocal`` and ``global``
+---------------------------------------
 
-* Return-based semantics struggle with complex assignment statements
-  like the one in the torture test
+``nonlocal`` and ``global`` are explicitly disallowed in the ``given`` clause
+suite and will be syntax errors if they occur. They will work normally if
+they appear within a ``def`` statement within that suite.
 
-The second thought is generally some kind of hidden renaming strategy. This
-also creates problems, as Python exposes variables names via the ``locals()``
-dictionary and class and function ``__name__`` attributes.
-
-The most promising approach is one based on symtable analysis and
-copy-in-copy-out referencing semantics to move any required name
-bindings between the inner and outer scopes. The torture test above
-would then translate to something like the following::
-
-   b = {}
-   def _anon1(b): # 'b' reference copied in
-       x = 42
-       def f(x):
-           return x
-       a = b[f(a)] = x
-       return a # 'a' reference copied out
-   a = _anon1(b)
-   assert "x" not in locals()
-   assert "f" not in locals()
-   assert a == 42
-   def _anon2(b) # 'b' reference copied in
-       d = b
-       assert d[42] == 42
-       # Nothing to copy out (not an assignment)
-   _anon2()
-   assert "d" not in locals()
-   def _anon3() # Nothing to copy in (no references to other variables)
-       x = 42
-       def f(): pass
-       y = locals("x"), f.__name__
-       y = y    # Assuming no optimisation of special cases
-       return y # 'y' reference copied out
-   y = _anon3()
-   assert "x" not in locals()
-   assert "f" not in locals()
-   assert y == (42, "f")
-
-However, as noted in the abstract, an actual implementation of
-this idea has never been tried.
-
-
-Detailed Semantics #1: Early Binding of Variable References
------------------------------------------------------------
-
-The copy-in-copy-out semantics mean that all variable references from a
-``given`` clause will exhibit early binding behaviour, in contrast to the
-late binding typically seen with references to closure variables and globals
-in ordinary functions. This behaviour will allow the ``given`` clause to
-be used as a substitute for the default argument hack when early binding
-behaviour is desired::
-
-  # Current Python (late binding)
-  seq = []
-  for i in range(10):
-    def f():
-      return i
-    seq.append(f)
-  assert [f() for f in seq] == [9]*10
-
-  # Current Python (early binding via default argument hack)
-  seq = []
-  for i in range(10):
-    def f(_i=i):
-      return i
-    seq.append(f)
-  assert [f() for f in seq] == list(range(10))
-
-  # Early binding via given clause
-  seq = []
-  for i in range(10):
-    seq.append(f) given:
-      def f():
-        return i
-  assert [f() for f in seq] == list(range(10))
-
-Note that the current intention is for the copy-in/copy-out semantics to
-apply only to names defined in the local scope containing the ``given``
-clause. Name in outer scopes will be referenced as normal.
-
-This intention is subject to revision based on feedback and practicalities
-of implementation.
-
-
-Detailed Semantics #2: Handling of ``nonlocal`` and ``global``
---------------------------------------------------------------
-
-``nonlocal`` and ``global`` will largely operate as if the anonymous
-functions were defined as in the expansion above. However, they will also
-override the default early-binding semantics for names from the containing
-scope.
-
-This intention is subject to revision based on feedback and practicalities
-of implementation.
+Alternatively, they could be defined as operating as if the anonymous
+functions were defined as in the expansion above.
 
 
 Detailed Semantics #3: Handling of ``break`` and ``continue``
@@ -588,53 +550,14 @@
 a ``for`` or ``while`` loop as part of that suite.
 
 
-Detailed Semantics #4: Handling of ``return`` and ``yield``
--------------------------------------------------------------
+Handling of ``return`` and ``yield``
+------------------------------------
 
 ``return`` and ``yield`` are explicitly disallowed in the ``given`` clause
 suite and will be syntax errors if they occur. They will work normally if
 they appear within a ``def`` statement within that suite.
 
 
-Alternative Semantics for Name Binding
---------------------------------------
-
-The "early binding" semantics proposed for the ``given`` clause are driven
-by the desire to have ``given`` clauses work "normally" in class scopes (that
-is, allowing them to see the local variables in the class, even though classes
-do not participate in normal lexical scoping).
-
-There is an alternative, which is to simply declare that the ``given`` clause
-creates an ordinary nested scope, just like comprehensions and generator
-expressions. Thus, the given clause would share the same quirks as those
-constructs: they exhibit surprising behaviour at class scope, since they
-can't see the local variables in the class definition. While this behaviour
-is considered desirable for method definitions (where class variables are
-accessed via the class or instance argument passed to the method), it can be
-surprising and inconvenient for implicit scopes that are designed to hide
-their own name bindings from the containing scope rather than vice-versa.
-
-A third alternative, more analogous to the comprehension case (where the
-outermost iterator expression is evaluated in the current scope and hence can
-see class locals normally), would be to allow *explicit* early binding in the
-``given`` clause, by passing an optional tuple of assignments after the
-``given`` keyword::
-
-  # Explicit early binding via given clause
-  seq = []
-  for i in range(10):
-    seq.append(f) given i=i:
-      def f():
-        return i
-  assert [f() for f in seq] == list(range(10))
-
-(Note: I actually like the explicit early binding idea significantly more
-than I do the implicit early binding - expect a future version of the PEP
-to be updated accordingly. I've already used it above when describing how
-an existing construct like a list comprehension could be expressed as a
-special case of the new syntax)
-
-
 Examples
 ========
 
@@ -654,10 +577,9 @@
   @singleton(*params)
   class public_name():
     ... # However many lines
-  public_name = public_name(*params)
 
   # Becomes:
-  public_name = MeaningfulClassName(*params) given:
+  public_name = .MeaningfulClassName(*params) given:
     class MeaningfulClassName():
       ... # Should trawl the stdlib for an example of doing this
 
@@ -671,7 +593,7 @@
   del _createenviron
 
   # Becomes:
-  environ = _createenviron() given:
+  environ = ._createenviron() given:
       def _createenviron():
         ... # 27 line function
 
@@ -684,7 +606,7 @@
   return decorating_function
 
   # Becomes:
-  return decorating_function given:
+  return .decorating_function given:
     # Cell variables rather than locals, but should give similar speedup
     tuple, sorted, len, KeyError = tuple, sorted, len, KeyError
     def decorating_function(user_function):
@@ -709,7 +631,29 @@
 
 * The "explicit early binding" variant may be applicable to the discussions
   on python-ideas on how to eliminate the default argument hack. A ``given``
-  clause in the header line for functions may be the answer to that question.
+  clause in the header line for functions (after the return type annotation)
+  may be the answer to that question.
+
+
+Rejected Alternatives
+=====================
+
+* An earlier version of this PEP allowed implicit forward references to the
+  names in the trailing suite, and also used implicit early binding
+  semantics. Both of these ideas substantially complicated the proposal
+  without providing a sufficient increase in expressive power. The current
+  proposing with explicit forward references and early binding brings the
+  new construct into line with existing scoping semantics, greatly
+  improving the chances the idea can actually be implemented.
+
+* In addition to the proposals made here, there have also been suggestions
+  of two suite "in-order" variants which provide the limited scoping of
+  names without supporting out-of-order execution. I believe these
+  suggestions largely miss the point of what people are complaining about
+  when they ask for multi-line lambda support - it isn't that coming up
+  with a name for the subexpression is especially difficult, it's that
+  naming the function before the statement that uses it means the code
+  no longer matches the way the developer thinks about the problem at hand.
 
 
 Reference Implementation
@@ -722,10 +666,10 @@
 TO-DO
 =====
 
-* Mention two-suite in-order variants (and explain why they're even more
-  pointless than the specific idea in the PEP)
 * Mention PEP 359 and possible uses for locals() in the ``given`` clause
 
+* Figure out if this can be used internally to make the implementation of
+  zero-argument super() calls less awful
 
 References
 ==========

-- 
Repository URL: http://hg.python.org/peps


More information about the Python-checkins mailing list