[Python-ideas] Statement local functions and classes (aka PEP 3150 is dead, say 'Hi!' to PEP 403)

[Nick Coghlan]

After some interesting conversations at PyCodeConf, I'm killing PEP
3150 (Statement Local Namespaces). It's too big, too unwieldy, too
confusing and too hard to implement to ever be a good idea.

PEP 403 is a far simpler idea, that looks to decorators (and Ruby
blocks) for inspiration. It's still a far from perfect idea, but it
has a lot more going for it than PEP 3150 ever did.

The basic question to ask yourself is this: What if we had a syntax
that allowed us to replace the final "name = obj" step that is
implicit in function and class definitions with an alternative
statement, and had a symbol that allowed us to refer to the function
or class in that statement without having to repeat the name?

The new PEP is included below and is also available online:
http://www.python.org/dev/peps/pep-0403/

I would *love* for people to dig through their callback based code
(and any other examples of "single use" functions and classes) to see
if this idea would help them.

Cheers,
Nick.

PEP: 403
Title: Statement local classes and functions
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan at gmail.com>
Status: Deferred
Type: Standards Track
Content-Type: text/x-rst
Created: 2011-10-13
Python-Version: 3.x
Post-History: 2011-10-13
Resolution: TBD


Abstract
========

This PEP proposes the addition of ':' as a new class and function prefix
syntax (analogous to decorators) that permits a statement local function or
class definition to be appended to any Python statement that currently does
not have an associated suite.

In addition, the new syntax would allow the '@' symbol to be used to refer
to the statement local function or class without needing to repeat the name.

When the ':' prefix syntax is used, the associated statement would be executed
*instead of* the normal local name binding currently implicit in function
and class definitions.

This PEP is based heavily on many of the ideas in PEP 3150 (Statement Local
Namespaces) so some elements of the rationale will be familiar to readers of
that PEP. That PEP has now been withdrawn in favour of this one.


PEP Deferral
============

Like PEP 3150, this PEP currently exists in a deferred state. Unlike PEP 3150,
this isn't because I suspect it might be a terrible idea or see nasty problems
lurking in the implementation (aside from one potential parsing issue).

Instead, it's because I think fleshing out the concept, exploring syntax
variants, creating a reference implementation and generally championing
the idea is going to require more time than I can give it in the 3.3 time
frame.

So, it's deferred. If anyone wants to step forward to drive the PEP for 3.3,
let me know and I can add you as co-author and move it to Draft status.


Basic Examples
==============

Before diving into the long history of this problem and the detailed
rationale for this specific proposed solution, here are a few simple
examples of the kind of code it is designed to simplify.

As a trivial example, weakref callbacks could be defined as follows::

    :x = weakref.ref(obj, @)
    def report_destruction(obj):
        print("{} is being destroyed".format(obj))

This contrasts with the current repetitive "out of order" syntax for this
operation::

    def report_destruction(obj):
        print("{} is being destroyed".format(obj))

    x = weakref.ref(obj, report_destruction)

That structure is OK when you're using the callable multiple times, but
it's irritating to be forced into it for one-off operations.

Similarly, singleton classes could now be defined as::

  :instance = @()
  class OnlyOneInstance:
    pass

Rather than::

  class OnlyOneInstance:
    pass

  instance = OnlyOneInstance()

And the infamous accumulator example could become::

    def counter():
        x = 0
        :return @
        def increment():
            nonlocal x
            x += 1
            return x

Proposal
========

This PEP proposes the addition of an optional block prefix clause to the
syntax for function and class definitions.

This block prefix would be introduced by a leading ``:`` and would be
allowed to contain any simple statement (including those that don't
make any sense in that context - while such code would be legal,
there wouldn't be any point in writing it).

The decorator symbol ``@`` would be repurposed inside the block prefix
to refer to the function or class being defined.

When a block prefix is provided, it *replaces* the standard local
name binding otherwise implicit in a class or function definition.


Background
==========

The question of "multi-line lambdas" has been a vexing one for many
Python users for a very long time, and it took an exploration of Ruby's
block functionality for me to finally understand why this bugs people
so much: Python's demand that the function be named and introduced
before the operation that needs it breaks the developer's flow of thought.
They get to a point where they go "I need a one-shot operation that does
<X>", and instead of being able to just *say* that, they instead have to back
up, name a function to do <X>, then call that function from the operation
they actually wanted to do in the first place. Lambda expressions can help
sometimes, but they're no substitute for being able to use a full suite.

Ruby's block syntax also heavily inspired the style of the solution in this
PEP, by making it clear that even when limited to *one* anonymous function per
statement, anonymous functions could still be incredibly useful. Consider how
many constructs Python has where one expression is responsible for the bulk of
the heavy lifting:

  * comprehensions, generator expressions, map(), filter()
  * key arguments to sorted(), min(), max()
  * partial function application
  * provision of callbacks (e.g. for weak references)
  * array broadcast operations in NumPy

However, adopting Ruby's block syntax directly won't work for Python, since
the effectiveness of Ruby's blocks relies heavily on various conventions in
the way functions are *defined* (specifically, Ruby's ``yield`` syntax to
call blocks directly and the ``&arg`` mechanism to accept a block as a
functions final argument.

Since Python has relied on named functions for so long, the signatures of
APIs that accept callbacks are far more diverse, thus requiring a solution
that allows anonymous functions to be slotted in at the appropriate location.


Relation to PEP 3150
====================

PEP 3150 (Statement Local Namespaces) described its primary motivation
as being to elevate ordinary assignment statements to be on par with ``class``
and ``def`` statements where the name of the item to be defined is presented
to the reader in advance of the details of how the value of that item is
calculated. This PEP achieves the same goal in a different way, by allowing
the simple name binding of a standard function definition to be replaced
with something else (like assigning the result of the function to a value).

This PEP also achieves most of the other effects described in PEP 3150
without introducing a new brainbending kind of scope. All of the complex
scoping rules in PEP 3150 are replaced in this PEP with the simple ``@``
reference to the statement local function or class definition.


Symbol Choice
==============

The ':' symbol was chosen due to its existing presence in Python and its
association with 'functions in expressions' via ``lambda`` expressions. The
past Simple Implicit Lambda proposal (PEP ???) was also a factor.

The proposal definitely requires *some* kind of prefix to avoid parsing
ambiguity and backwards compatibility problems and ':' at least has the
virtue of brevity. There's no obious alternative symbol that offers a
clear improvement.

Introducing a new keyword is another possibility, but I haven't come up
with one that really has anything to offer over the leading colon.


Syntax Change
=============

Current::

    atom: ('(' [yield_expr|testlist_comp] ')' |
           '[' [testlist_comp] ']' |
           '{' [dictorsetmaker] '}' |
           NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False')

Changed::

    atom: ('(' [yield_expr|testlist_comp] ')' |
           '[' [testlist_comp] ']' |
           '{' [dictorsetmaker] '}' |
           NAME | NUMBER | STRING+ | '...' | 'None' | 'True' | 'False' | '@')

New::

    blockprefix: ':' simple_stmt
    block: blockprefix (decorated | classdef | funcdef)

The above is the general idea, but I suspect that change to the 'atom'
definition would cause an ambiguity problem in the parser when it comes to
detecting decorator lines. So the actual implementation would be more complex
than that.

Grammar: http://hg.python.org/cpython/file/default/Grammar/Grammar


Possible Implementation Strategy
================================

This proposal has one titanic advantage over PEP 3150: implementation
should be relatively straightforward.

Both the class and function definition statements emit code to perform
the local name binding for their defined name. Implementing this PEP
should just require intercepting that code generation and replacing
it with the code in the block prefix.

The one potentially tricky part is working out how to allow the dual
use of '@' without rewriting half the grammar definition.

More Examples
=============

Calculating attributes without polluting the local namespace (from os.py)::

  # Current Python (manual namespace cleanup)
  def _createenviron():
      ... # 27 line function

  environ = _createenviron()
  del _createenviron

  # Becomes:
  :environ = @()
  def _createenviron():
      ... # 27 line function

Loop early binding::

  # Current Python (default argument hack)
  funcs = [(lambda x, i=i: x + i) for i in range(10)]

  # Becomes:
  :funcs = [@(i) for i in range(10)]
  def make_incrementor(i):
    return lambda x: x + i

  # Or even:
  :funcs = [@(i) for i in range(10)]
  def make_incrementor(i):
    :return @
    def incrementor(x):
        return x + i


Reference Implementation
========================

None as yet.


TO DO
=====

Sort out links and references to everything :)


Acknowledgements
================

Huge thanks to Gary Bernhardt for being blunt in pointing out that I had no
idea what I was talking about in criticising Ruby's blocks, kicking off a
rather enlightening process of investigation.


References
==========

TBD


Copyright
=========

This document has been placed in the public domain.


..
   Local Variables:
   mode: indented-text
   indent-tabs-mode: nil
   sentence-end-double-space: t
   fill-column: 70
   coding: utf-8
   End:


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia