[Python-ideas] Statement local functions and classes (aka PEP 3150 is dead, say 'Hi!' to PEP 403)

[Eric Snow]

On Wed, Oct 12, 2011 at 6:22 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> After some interesting conversations at PyCodeConf, I'm killing PEP
> 3150 (Statement Local Namespaces). It's too big, too unwieldy, too
> confusing and too hard to implement to ever be a good idea.
>
> PEP 403 is a far simpler idea, that looks to decorators (and Ruby
> blocks) for inspiration. It's still a far from perfect idea, but it
> has a lot more going for it than PEP 3150 ever did.
>
> The basic question to ask yourself is this: What if we had a syntax
> that allowed us to replace the final "name = obj" step that is
> implicit in function and class definitions with an alternative
> statement, and had a symbol that allowed us to refer to the function
> or class in that statement without having to repeat the name?
>
> The new PEP is included below and is also available online:
> http://www.python.org/dev/peps/pep-0403/
>
> I would *love* for people to dig through their callback based code
> (and any other examples of "single use" functions and classes) to see
> if this idea would help them.
>
> Cheers,
> Nick.
>
> PEP: 403
> Title: Statement local classes and functions
> Version: $Revision$
> Last-Modified: $Date$
> Author: Nick Coghlan <ncoghlan at gmail.com>
> Status: Deferred
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 2011-10-13
> Python-Version: 3.x
> Post-History: 2011-10-13
> Resolution: TBD
>
>
> Abstract
> ========
>
> This PEP proposes the addition of ':' as a new class and function prefix
> syntax (analogous to decorators) that permits a statement local function or
> class definition to be appended to any Python statement that currently does
> not have an associated suite.
>
> In addition, the new syntax would allow the '@' symbol to be used to refer
> to the statement local function or class without needing to repeat the name.
>
> When the ':' prefix syntax is used, the associated statement would be executed
> *instead of* the normal local name binding currently implicit in function
> and class definitions.
>
> This PEP is based heavily on many of the ideas in PEP 3150 (Statement Local
> Namespaces) so some elements of the rationale will be familiar to readers of
> that PEP. That PEP has now been withdrawn in favour of this one.
>
>
> PEP Deferral
> ============
>
> Like PEP 3150, this PEP currently exists in a deferred state. Unlike PEP 3150,
> this isn't because I suspect it might be a terrible idea or see nasty problems
> lurking in the implementation (aside from one potential parsing issue).
>
> Instead, it's because I think fleshing out the concept, exploring syntax
> variants, creating a reference implementation and generally championing
> the idea is going to require more time than I can give it in the 3.3 time
> frame.
>
> So, it's deferred. If anyone wants to step forward to drive the PEP for 3.3,
> let me know and I can add you as co-author and move it to Draft status.
>
>
> Basic Examples
> ==============
>
> Before diving into the long history of this problem and the detailed
> rationale for this specific proposed solution, here are a few simple
> examples of the kind of code it is designed to simplify.
>
> As a trivial example, weakref callbacks could be defined as follows::
>
>    :x = weakref.ref(obj, @)
>    def report_destruction(obj):
>        print("{} is being destroyed".format(obj))
>
> This contrasts with the current repetitive "out of order" syntax for this
> operation::
>
>    def report_destruction(obj):
>        print("{} is being destroyed".format(obj))
>
>    x = weakref.ref(obj, report_destruction)
>
> That structure is OK when you're using the callable multiple times, but
> it's irritating to be forced into it for one-off operations.
>
> Similarly, singleton classes could now be defined as::
>
>  :instance = @()
>  class OnlyOneInstance:
>    pass
>
> Rather than::
>
>  class OnlyOneInstance:
>    pass
>
>  instance = OnlyOneInstance()
>
> And the infamous accumulator example could become::
>
>    def counter():
>        x = 0
>        :return @
>        def increment():
>            nonlocal x
>            x += 1
>            return x
>
> Proposal
> ========
>
> This PEP proposes the addition of an optional block prefix clause to the
> syntax for function and class definitions.
>
> This block prefix would be introduced by a leading ``:`` and would be
> allowed to contain any simple statement (including those that don't
> make any sense in that context - while such code would be legal,
> there wouldn't be any point in writing it).
>
> The decorator symbol ``@`` would be repurposed inside the block prefix
> to refer to the function or class being defined.
>
> When a block prefix is provided, it *replaces* the standard local
> name binding otherwise implicit in a class or function definition.
>
>
> Background
> ==========
>
> The question of "multi-line lambdas" has been a vexing one for many
> Python users for a very long time, and it took an exploration of Ruby's
> block functionality for me to finally understand why this bugs people
> so much: Python's demand that the function be named and introduced
> before the operation that needs it breaks the developer's flow of thought.
> They get to a point where they go "I need a one-shot operation that does
> <X>", and instead of being able to just *say* that, they instead have to back
> up, name a function to do <X>, then call that function from the operation
> they actually wanted to do in the first place. Lambda expressions can help
> sometimes, but they're no substitute for being able to use a full suite.
>
> Ruby's block syntax also heavily inspired the style of the solution in this
> PEP, by making it clear that even when limited to *one* anonymous function per
> statement, anonymous functions could still be incredibly useful. Consider how
> many constructs Python has where one expression is responsible for the bulk of
> the heavy lifting:
>
>  * comprehensions, generator expressions, map(), filter()
>  * key arguments to sorted(), min(), max()
>  * partial function application
>  * provision of callbacks (e.g. for weak references)
>  * array broadcast operations in NumPy
>
> However, adopting Ruby's block syntax directly won't work for Python, since
> the effectiveness of Ruby's blocks relies heavily on various conventions in
> the way functions are *defined* (specifically, Ruby's ``yield`` syntax to
> call blocks directly and the ``&arg`` mechanism to accept a block as a
> functions final argument.
>
> Since Python has relied on named functions for so long, the signatures of
> APIs that accept callbacks are far more diverse, thus requiring a solution
> that allows anonymous functions to be slotted in at the appropriate location.
>
>
> Relation to PEP 3150
> ====================
>
> PEP 3150 (Statement Local Namespaces) described its primary motivation
> as being to elevate ordinary assignment statements to be on par with ``class``
> and ``def`` statements where the name of the item to be defined is presented
> to the reader in advance of the details of how the value of that item is
> calculated. This PEP achieves the same goal in a different way, by allowing
> the simple name binding of a standard function definition to be replaced
> with something else (like assigning the result of the function to a value).
>
> This PEP also achieves most of the other effects described in PEP 3150
> without introducing a new brainbending kind of scope. All of the complex
> scoping rules in PEP 3150 are replaced in this PEP with the simple ``@``
> reference to the statement local function or class definition.
>
>
> Symbol Choice
> ==============
>
> The ':' symbol was chosen due to its existing presence in Python and its
> association with 'functions in expressions' via ``lambda`` expressions. The
> past Simple Implicit Lambda proposal (PEP ???) was also a factor.
>
> The proposal definitely requires *some* kind of prefix to avoid parsing
> ambiguity and backwards compatibility problems and ':' at least has the
> virtue of brevity. There's no obious alternative symbol that offers a
> clear improvement.
>
> Introducing a new keyword is another possibility, but I haven't come up
> with one that really has anything to offer over the leading colon.
>
>
> Syntax Change
> =============
>
> Current::
>
>    atom: ('(' [yield_expr|testlist_comp] ')' |
>           '[' [testlist_comp] ']' |
>           '{' [dictorsetmaker] '}' |
>           NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')
>
> Changed::
>
>    atom: ('(' [yield_expr|testlist_comp] ')' |
>           '[' [testlist_comp] ']' |
>           '{' [dictorsetmaker] '}' |
>           NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False' | '@')
>
> New::
>
>    blockprefix: ':' simple_stmt
>    block: blockprefix (decorated | classdef | funcdef)
>
> The above is the general idea, but I suspect that change to the 'atom'
> definition would cause an ambiguity problem in the parser when it comes to
> detecting decorator lines. So the actual implementation would be more complex
> than that.
>
> Grammar: http://hg.python.org/cpython/file/default/Grammar/Grammar
>
>
> Possible Implementation Strategy
> ================================
>
> This proposal has one titanic advantage over PEP 3150: implementation
> should be relatively straightforward.
>
> Both the class and function definition statements emit code to perform
> the local name binding for their defined name. Implementing this PEP
> should just require intercepting that code generation and replacing
> it with the code in the block prefix.
>
> The one potentially tricky part is working out how to allow the dual
> use of '@' without rewriting half the grammar definition.
>
> More Examples
> =============
>
> Calculating attributes without polluting the local namespace (from os.py)::
>
>  # Current Python (manual namespace cleanup)
>  def _createenviron():
>      ... # 27 line function
>
>  environ = _createenviron()
>  del _createenviron
>
>  # Becomes:
>  :environ = @()
>  def _createenviron():
>      ... # 27 line function
>
> Loop early binding::
>
>  # Current Python (default argument hack)
>  funcs = [(lambda x, i=i: x + i) for i in range(10)]
>
>  # Becomes:
>  :funcs = [@(i) for i in range(10)]
>  def make_incrementor(i):
>    return lambda x: x + i
>
>  # Or even:
>  :funcs = [@(i) for i in range(10)]
>  def make_incrementor(i):
>    :return @
>    def incrementor(x):
>        return x + i
>
>
> Reference Implementation
> ========================
>
> None as yet.
>
>
> TO DO
> =====
>
> Sort out links and references to everything :)
>
>
> Acknowledgements
> ================
>
> Huge thanks to Gary Bernhardt for being blunt in pointing out that I had no
> idea what I was talking about in criticising Ruby's blocks, kicking off a
> rather enlightening process of investigation.
>
>
> References
> ==========
>
> TBD
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
> ..
>   Local Variables:
>   mode: indented-text
>   indent-tabs-mode: nil
>   sentence-end-double-space: t
>   fill-column: 70
>   coding: utf-8
>   End:
>
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> http://mail.python.org/mailman/listinfo/python-ideas
>

This is really clever.  I have some comments:

1. decorators

just to clarify, this is legal:

   :assert @(1) == 1
   @lambda f: return lambda x: return x
   def spam(): pass

but decorators above the block prefix are not (since they wouldn't be
decorating a def/class statement).

2. namespaces

I hate using a class definition as a plain namespace, but the
following is cool (if I'm reading this right):

   :some_func(@.x, @.y)
   class _:
       x = 4
       y = 1

and given a purer namespace (like http://code.activestate.com/recipes/577887):

   :some_func(**@)
   @as_namespace
   class _:
       x = 4
       y = 1

3. does it cover all the default arguments hack use cases? If so, is
it too cumbersome to be a replacement?

4. how do you introspect the statement local function/class?

5. the relationship to statement local namespaces makes me think
def-from and assignment decorators, but this seems like something
else.

Some of these are probably pretty obvious, but it's getting late and
my brain's a little fuzzy.  :)  I'll think about these some more
tomorrow.  All in all, this is a pretty sweet idea!

-eric