On Wed, Apr 18, 2018 at 11:17 AM, Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Apr 19, 2018 at 2:18 AM, Guido van Rossum <guido@python.org> wrote:
> On Wed, Apr 18, 2018 at 7:35 AM, Chris Angelico <rosuav@gmail.com> wrote:
>>
>> On Wed, Apr 18, 2018 at 11:58 PM, Guido van Rossum <guido@python.org>
>> wrote:
>> > I can't tell from this what the PEP actually says should happen in that
>> > example. When I first saw it I thought "Gaah! What a horrible piece of
>> > code." But it works today, and people's code *will* break if we change
>> > its
>> > meaning.
>> >
>> > However we won't have to break that. Suppose the code is (perversely)
>> >
>> > t = range(3)
>> > a = [t for t in t if t]
>> >
>> > If we translate this to
>> >
>> > t = range(3)
>> > def listcomp(t=t):
>> >     a = []
>> >     for t in t:
>> >         if t:
>> >             a.append(t)
>> >     return a
>> > a = listcomp()
>> >
>> > Then it will still work. The trick will be to recognize "imported" names
>> > that are also assigned and capture those (as well as other captures as
>> > already described in the PEP).
>>
>> That can be done. However, this form of importing will have one of two
>> consequences:
>>
>> 1) Referencing an unbound name will scan to outer scopes at run time,
>> changing the semantics of Python name lookups
>
>
> I'm not even sure what this would do.

The implicit function of the listcomp would attempt to LOAD_FAST 't',
and upon finding that it doesn't have a value for it, would go and
look for the name 't' in a surrounding scope. (Probably LOAD_CLOSURE.)

We agree that that's too dynamic to be explainable.
 
>> 2) Genexps will eagerly evaluate a lookup if it happens to be the same
>> name as an internal iteration variable.
>
>
> I think we would have to specify this more precisely.
>
> Let's say by "eagerly evaluate a lookup" you mean "include it in the
> function parameters with a default value being the lookup (i.e. starting in
> the outer scope), IOW "t=t" as I showed above.

Yes. To be technically precise, there's no default argument involved,
and the call to the implicit function explicitly passes all the
arguments.

OK, and the idea is the same -- it's explicitly evaluated in the outer scope either way.
 
> The question is *when* we
> would do this. IIUC the PEP already does this if the "outer scope" is a
> class scope for any names that a simple static analysis shows are references
> to variables in the class scope.

Correct.

> (I don't know exactly what this static
> analysis should do but it could be as simple as gathering all names that are
> assigned to in the class, or alternatively all names assigned to before the
> point where the comprehension occurs. We shouldn't be distracted by dynamic
> definitions like `exec()` although we should perhaps be aware of `del`.)

At the moment, it isn't aware of 'del'. The analysis is simple and
100% static: If a name is in the table of names the class uses AND
it's in the table of names the comprehension uses, it gets passed as a
parameter. I don't want to try to be aware of del, because of this:

class X:
    x = 1
    if y: del x
    print(x)
    z = (q for q in x if q)

If y is true, this will eagerly look up x using the same semantics in
both the print and the genexp (on construction, not when you iterate
over the genexp). If y is false, it'll still eagerly look up x, and
it'll still use the same semantics for print and the genexp (and it'll
find an 'x' in a surrounding scope).

(The current implementation actually is a bit different from that. I'm
not sure whether it's possible to do it as simply as given without an
extra compilation pass. But it's close enough.)

Yeah, I threw 'del' in there mostly so we wouldn't get *too* confident. I see a fair amount of this:

  d = {}
  for x, y in blah():
      d[x] = y
  del x, y
 
> My proposal is to extend this static analysis for certain loop control
> variables (any simple name assigned to in a for-clause in the
> comprehension), regardless of what kind of scope the outer scope is. If the
> outer scope is a function we already know how to do this. If it's a class we
> use the analysis referred to above. If the outer scope is the global scope
> we have to do something new. I propose to use the same simple static
> analysis we use for class scopes.
>
> Furthermore I propose to *only* do this for the loop control variable(s) of
> the outermost for-clause, since that's the only place where without all this
> rigmarole we would have a clear difference in behavior with Python 3.7 in
> cases like [t for t in t]. Oh, and probably we only need to do this if that
> loop control variable is also used as an expression in the iterable (so we
> don't waste time doing any of this for e.g. [t for t in q]).

Okay. Here's something that would be doable:

If the name is written to within the comprehension, AND it is read
from in the outermost iterable, it is flagged early-bind.

OK, that's close enough to what I am looking for that I don't think it matters.
 
I'll have to try implementing that to be sure, but it should be
possible I think. It would cover a lot of cases, keeping them the same
as we currently have.

> Since we now have once again introduced an exception for the outermost loop
> control variable and the outermost iterable, we can consider doing this only
> as a temporary measure. We could have a goal to eventually make [t for t in
> t] fail, and in the meantime we would deprecate it -- e.g. in 3.8 a silent
> deprecation, in 3.9 a noisy one, in 3.10 break it. Yes, that's a lot of new
> static analysis for deprecating an edge case, but it seems reasonable to
> want to preserve backward compatibility when breaking this edge case since
> it's likely not all that uncommon. Even if most occurrences are bad style
> written by lazy programmers, we should not break working code, if it is
> reasonable to expect that it's relied upon in real code.

Fair enough. So the outermost iterable remains special for a short
while, with deprecation.

I'll get onto the coding side of it during my Copious Free Time,
hopefully this week some time.

Here's hoping!

Don't get your hopes up too high. A lot of respectable core devs have expressed a -1.

--
--Guido van Rossum (python.org/~guido)