[Python-ideas] Statement local functions and classes (aka PEP 3150 is dead, say 'Hi!' to PEP 403)

[Nick Coghlan]

On Mon, Oct 17, 2011 at 5:52 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>>
>> Yeah, that's a large part of why I now think the given clause needs to
>> be built on the same semantics that we already use internally for
>> implicit out of order evaluation (i.e. decorators, comprehensions and
>> generator expressions), such that it merely exposes the unifying
>> mechanic underlying existing constructs rather than creating a
>> completely new way of doing things.
>
> I'm not sure what you mean by that. If you're talking about
> the implementation, all three of those use rather different
> underlying mechanics. What exactly do you see about these
> that unifies them?

Actually, comprehensions and generator expressions are almost
identical in 3.x (they only differ in the details of the inner loop in
the anonymous function).

For comprehensions, the parallel with the proposed given statement
would be almost exact:

    seq = [x*y for x in range(10) for y in range(5)]

would map to:

    seq = _list_comp given _outermost_iter = range(10):
        _list_comp = []
        for x in _outermost_iter:
            for y in range(5):
                _list_comp.append(x*y)

And similarly for set and dict comprehensions:

    # unique = {x*y for x in range(10) for y in range(5)}
    unique = _set_comp given _outermost_iter = range(10):
        _set_comp = set()
        for x in _outermost_iter:
            for y in range(5):
                _set_comp.add(x*y)

    # map = {(x, y):x*y for x in range(10) for y in range(5)}
    map = _dict_comp given _outermost_iter = range(10):
        _anon = {}
        for x in _outermost_iter:
            for y in range(5):
                _anon[x,y] = x*y

Note that this lays bare some of the quirks of comprehension scoping -
at class scope, the outermost iterator expression can sometimes see
names that the inner iterator expressions miss.

For generator expressions, the parallel isn't quite as strong, since
the compiler is able to avoid the redundant anonymous function
involved in the given clause and just emit an anonymous generator
directly. However, the general principle still holds:

    # gen_iter = (x*y for x in range(10) for y in range(5))
    gen_iter = _genexp() given _outermost_iter = range(10):
        def _genexp():
            for x in _outermost_iter:
                for y in range(5):
                    yield x*y

For decorated functions, the parallel is actually almost as weak as it
is for classes, since so many of the expressions involved (decorator
expressions, default arguments, annotations) get evaluated in order in
the current scope and even a given statement can't reproduce the
actual function statement's behaviour of not being bound at *all* in
the current scope while decorators are being applied, even though the
function already knows what it is going to be called:

>>> def call(f):
...     print(f.__name__)
...     return f()
...
>>> @call
... def func():
...     return func.__name__
...
func
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in call
  File "<stdin>", line 3, in func
NameError: global name 'func' is not defined

So it's really only the machinery underlying comprehensions that is
being exposed by the PEP rather than anything more far reaching.

Exposing the generator expression machinery directly would require the
ability to turn the given clause into a generator (via a top level
yield expression) and then a means to reference that from the header
line, which gets us back into cryptic and unintuitive PEP 403
territory. Better to settle for the named alternative.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia