[Python-ideas] Tweaking closures and lexical scoping to include the function being defined

[Nick Coghlan]

On Sat, Oct 1, 2011 at 2:57 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Ron Adam writes:
>
>  > Supposedly the @ decorator syntax is supposed to be like a pre-compile
>  > substitution where..
>  >
>  >     @decorator
>  >     def func(x):x
>  >
>  > Is equivalent to...
>  >
>  >     def func(x):x
>  >     func = decorator(func)
>  >
>  > IF that is true,
>
> It is.

It isn't quite - the name binding doesn't happen until *after* the
decorator chain has been invoked, so the function is anonymous while
the decorators are executing. In addition, the decorator expressions
themselves are evaluated before the function is defined. That's not a
problem in practice, since each decorator gets passed the result of
the previous one (or the original function for the innermost
decorator).

The real equivalent code is more like:

    <anon1> = decorator
    def <anon2>(x):
        return x
    func = <anon1>(<anon2>)

(IIRC, the anonymous references happen to be stored on the frame stack
in CPython, but that's an implementation detail)

As far as the proposed semantics for any new syntax to eliminate the
desire to use the default argument hack goes, I haven't actually heard
any complaints about any addition being syntactic sugar for the
following closure idiom:

    def <anon1>():
        NAME = EXPR
        def FUNC(ARGLIST):
            """DOC"""
            nonlocal NAME
            BODY
        return FUNC
    FUNC = <anon1>()

The debate focuses on whether or not there is any possible shorthand
spelling for those semantics that successfully negotiates the Zen of
Python:

Beautiful is better than ugly.
    - the default argument hack is actually quite neat and tidy if you
know what it means. Whatever we do should be at least as attractive as
that approach.

Explicit is better than implicit.
    - the line between the default argument hack and normal default
arguments is blurry. New syntax would fix that.

Simple is better than complex.
    - lexical scoping left simple behind years ago ;)

Complex is better than complicated.
    - IMO, the default argument hack is complicated, since it abuses a
tool meant for something else, whereas function state variables would
be just another tier in the already complex scoping spectrum from
locals through lexical scoping to module globals and builtins (with
function state variables slotting in neatly between ordinary locals
and lexically scoped nonlocals).

Flat is better than nested.
   - There's a lot of visual nesting going on if you spell out these
semantics as a closure or as a class. The appeal of the default
argument hack largely lies in its ability to flatten that out into
state storage on the function object itself

Sparse is better than dense.
  - This would be the main argument for having something before the
header line (decorator style) rather than cramming yet more
information into the header line itself. However, it's also an
argument against decorator-style syntax, since that is quite heavy on
the page (due to the business of the '@' symbol in most fonts)

Readability counts.
  - The class and closure solutions are not readable - that's the big
reason people opt for the default argument hack when it applies. It
remains to be seen if we can come up with dedicated syntax that is at
least as readable as the default argument hack itself.

Special cases aren't special enough to break the rules.
  - I think this is the heart of what killed the "inside the function
scope" variants for me. They're *too* magical and different from the
way other code at function scope works to be a good fit.

Although practicality beats purity.
  - Using the default argument hack in the first place is the epitome of this :)

Errors should never pass silently.
Unless explicitly silenced.
  - This is why 'nonlocal x', where x is not defined in a lexical
scope, is, and will remain, a Syntax Error and why nonlocal and global
declarations that conflict with the parameter list are also errors.
Similar constraints would be placed on any new syntax dedicated to
function state variables.

In the face of ambiguity, refuse the temptation to guess.
  - In the case of name bindings, the compiler doesn't actually
*guess* anything - name bindings create local variables, unless
overridden by some other piece of syntax (i.e. a nonlocal or global
declaration). This may, of course, look like guessing to developers
that don't understand the scoping rules yet. The challenge for
function state variables is coming up with a similarly unambiguous
syntax that still allows them to be given an initial state at function
definition time.

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
  - For me, these two are about coming up with a syntax that is easy
to *remember* once you know it, even if you have to look up what it
means the first time you encounter. Others set the bar higher and want
developers to have a reasonable chance of *guessing* what it means
without actually reading the documentation for the new feature. I
think the latter goal is unattainable and hence not a useful standard.
However, I'll also note that the default argument hack itself does
meet *my* interpretation of this guideline (if you know it,
recognising it and remembering it aren't particularly difficult)

Now is better than never.
Although never is often better than *right* now.
  - The status quo has served us well for a long time. If someone can
come up with an elegant syntax, great, let's pursue it. Otherwise,
this whole issue really isn't that important in the grand scheme of
things (although a PEP to capture the current 'state of the art'
thinking on the topic would still be nice - I believe Jan and Eric
still plan to get to that once the discussion dies down again)

If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
  - this is where I think the specific proposal to just add syntactic
sugar for a particular usage of an ordinary closure is a significant
improvement on past attempts. Anyone that understands how closures
work will understand the meaning of the new syntax, just as anyone
that fully understands 'yield' can understand PEP 380's 'yield from'.

Namespaces are one honking great idea -- let's do more of those!
  - closures are just another form of namespace, even though people
typically think of classes, modules and packages when contemplating
this precept. "Function state variables" would be formalising the
namespace where default argument values live (albeit anonymously) and
making it available for programmatic use.

Despite its flaws, the simple brackets enclosed list after the
function parameter list is still my current favourite:

    def global_counter(x) [n=0, lock=Lock()]:
        with lock:
            n += 1
            yield n

It just composes more nicely with decorators than the main alternative
still standing and is less prone to overwhelming the function name
with extraneous implementation details:

    @contextmanager
    def counted()  [active=collections.Counter(), lock=threading.RLock()]:
        with lock:
            active[threading.current_thread().ident] += 1
        yield active
        with lock:
            active[threading.current_thread().ident] -= 1

far more clearly conveys "this defines a context manager named
'counted'" than the following does:

    @contextmanager
    @(active=collections.Counter(), lock=threading.RLock())
    def counted():
        with lock:
            active[threading.current_thread().ident] += 1
        yield active
        with lock:
            active[threading.current_thread().ident] -= 1

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia