[Python-Dev] PEP 572 semantics

[Tim Peters]

Just a quickie:

[Steve Dower]

> > The PEP uses the phrase "an assignment expression
> occurs in a comprehension" - what does this mean?

It's about static analysis of the source code, at compile-time, to
establish scopes.  So "occurs in" means your eyeballs see an assignment
expression in a comprehension.

> Does it occur when/where it is compiled, instantiated, or executed? This
> is important because where it occurs determines which scope will be
> modified. For sanity sake, I want to assume that it means compiled,

Probably ;-)  I just don't know _exactly_ what those distinctions mean to
you.

> but now what happens when that scope is gone?

Nothing new.  There's no new concept of "scope" here, which is why the PEP
doesn't say a whole lot about scope.

>
>  >>> def f():

> > ...     return (a := i for i in range(5))

> > ...

Same as now, `i` is local to the synthetic nested function created for the
genexp.  The scope of `a` is determined by pretending the assignment
occurred in the block containing the outermost (textually - static
analysis) comprehension.  In this case, `a = anything` before the `return`
would establish that `a` is local to `f`, so that's the answer:  `a` is
local to `f`.  If `a` had been declared global in `f`, then `a` in the
genexp would be the same global `a`.  And similarly if `a` had been
declared nonlocal.in `f`.

In all cases the scope resolution is inherited from the closest containing
non-comprehension/genexp block, with the twist if that if a name is unknown
in that block, the name is established as being local to that block.  So
your example is actually the subtlest case.


> >>> list(f())

> > [0, 1, 2, 3, 4]   # or a new error because the scope has gone?

Function scopes in Python have "indefinite extent", and nothing about that
changes.  So, ya, that's the output - same as today if you changed your
example to delete the "a :=" part.

Internally, f's local `a` was left bound to 4, but there's no way to see
that here because the genexp has been run to exhaustion and
reference-counting has presumably thrown everything away by now.

>>> a

> ???

Same thing typing `a` would result in if you had never typed `list(f())`.

Here's a variation:

    def f():

>        yield (a := i for i in range(5))
       yield a

Then:

>>> g = f()
>>> list(next(g))
[0, 1, 2, 3, 4]
>>> next(g)
4

> I'll push back real hard on doing the assignment in the scope
> where the generator is executed:

> >

> > >>> def do_secure_op(name, numbers):

> > ...     authorised = check_authorised(name)

This instance of `authorized` is local to `do_secure_op`.

> ...     if not all(numbers):

> > ...         raise ValueError()

> > ...     if not authorised:

> > ...         raise SecurityError()

> > ...     print('You made it!')

> > ...

> > >>> do_secure_op('whatever', (authorised := i for i in [1, 2, 3]))

And this instance of `authorized` is a global (because the genexp appears
in top-level code, so its containing block is the module).  The two
instances have nothing to do with each other.

> You made it!

Yup - you did!

> >>> authorised

> > NameError: name 'authorised' is undefined

It would display 3 instead.


> From the any()/all() examples, it seems clear that the target scope for

> > the assignment has to be referenced from the generator scope (but not

> > for other comprehension types, which can simply do one transfer of the

> > assigned name after fully evaluating all the contents).

I don't think that follows.  It _may_ in some cases.  For example,

def f():
    i = None # not necessary, just making i's scope obvious
    def g(ignore):
        return i+1
    return [g(i := j) for j in range(3)]

_While_ the list comprehension is executing, it needs to rebind f's `i` on
each iteration so that the call to `g()` on each iteration can see `i`'s
then-current value.

> Will this reference keep the frame object alive for as
> long as the generator exists? Can it be a weak reference?
> Are assignments just going to be silently ignored when
> the frame they should assign to is gone? I'd like to see these
> clarified in the main text.

Here you're potentially asking questions about how closures work in Python
(in the most-likely case that an embedded assignment-statement target
resolves to an enclosing function-local scope), but the PEP doesn't change
anything about how closures already work.  Closures are implemented via
"cell objects", one per name, which already supply both "read" and "write"
access to both the owning and referencing scopes.

def f():
    a = 42
    return (a+1 for i in range(3))

That works fine today, and a cell object is used in the synthesized genexp
function to get read access to f's local `a`.  But references to `a` in `f`
_also_ use that cell object. - the thing that lives in f's frame isn't
really the binding for `a`, but a reference to the cell object that _holds_
a's current binding.  The cell object doesn't care whether f's frame goes
away (except that the cell object's refcount drops by one when f's frame
vanishes).  Nothing about that changes if the synthesized genexp wants
write access instead.

While a gentle introduction to how closures are implemented in Python would
be a cool thing, this PEP is the last place to include one ;-)

It may help to realize that there's nothing here that can't be done today
by explicitly writing nested functions with appropriate scope declarations,
in a straightforward way.  There's nothing _inherently_ new.

Huh!  Not so much "a quickie" after all :-(  So I'll stop here for now.
Thank you for the critical reading and feedback!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180704/bc74b922/attachment-0001.html>