Looking for examples: proof that a list comp is a function
Guido has stated that this parallel is desired and important: result = [f(x) for x in iter if g(x)] result = list(f(x) for x in iter if g(x)) Obviously the genexp has to be implemented with a nested function, since there's no guarantee that it'll be iterated over in this way. With current semantics, you can easily prove that a list comp is implemented with a function by looking at how it interacts with other scopes (mainly class scope), but Tim's proposal may change that. So I'm looking for examples that prove that a list comp is executed inside an implicit function. Ideally, examples that are supported by language guarantees, but something that's "CPython has done it this way since 3.0" is important too. I'm aware of just two: the name lookup interaction that may be changing, and the fact that there's an extra line in a traceback. And the latter, as far as I know, is not guaranteed (and I doubt anyone would care if it changed). Are there any other provable points? ChrisA
On Mon, May 14, 2018, 03:36 Chris Angelico
Guido has stated that this parallel is desired and important:
result = [f(x) for x in iter if g(x)] result = list(f(x) for x in iter if g(x))
Obviously the genexp has to be implemented with a nested function, since there's no guarantee that it'll be iterated over in this way. With current semantics, you can easily prove that a list comp is implemented with a function by looking at how it interacts with other scopes (mainly class scope), but Tim's proposal may change that.
So I'm looking for examples that prove that a list comp is executed inside an implicit function. Ideally, examples that are supported by language guarantees, but something that's "CPython has done it this way since 3.0" is important too.
I'm aware of just two: the name lookup interaction that may be changing, and the fact that there's an extra line in a traceback. And the latter, as far as I know, is not guaranteed (and I doubt anyone would care if it changed). Are there any other provable points?
Related to the traceback one: the extra stack frame shows up in a debugger, and a profiler counts the extra frame separately. The first often confuses me because I don't immediately see which frame I'm in just by seeing the line of code. There are odd interactions between `yield`/`yield from` and comprehensions that was discussed some months ago: "[Python-Dev] Tricky way of of creating a generator via a comprehension expression". Wait, is this a continuation of that discussion?
[Chris Angelico ...
With current semantics, you can easily prove that a list comp is
implemented with a function by looking at how it interacts with other
scopes (mainly class scope), but Tim's proposal may change that. Absolutely not. I haven't considered for a nanosecond that anything
_essential_ would change in the current implementation. In effect, my
proposal to
bind assignment statement targets
that appear in a listcomp or genexp
in the blocks that immediately contain their synthetic functions
"merely" sprinkles in some `nonlocal` and/or `global` declarations to
change the targets' scopes. Indeed, it _relies_ on that they're
implemented as (potentially nested) synthetic functions today. And if
you haven't read my proposed changes to the reference manual, they
explicitly state that they're talking about the synthetic functions
created to implement genexps and listcomps. So I'm looking for examples that prove that a list comp is executed
inside an implicit function. Ideally, examples that are supported by
language guarantees, but something that's "CPython has done it this
way since 3.0" is important too. I don't believe you'll find that - but, of course, may be wrong about that. I'm aware of just two: the name lookup interaction that may be
changing, and the fact that there's an extra line in a traceback. And
the latter, as far as I know, is not guaranteed (and I doubt anyone
would care if it changed). Are there any other provable points? Nick pointed me to these future docs that _will_ pretty much imply it:
https://docs.python.org/dev/reference/expressions.html#displays-for-lists-se...
In part:
"""
However, aside from the iterable expression in the leftmost for
clause, the comprehension is executed in a separate implicitly nested
scope. This ensures that names assigned to in the target list don’t
“leak” into the enclosing scope.
The iterable expression in the leftmost for clause is evaluated
directly in the enclosing scope and then passed as an argument to the
implictly nested scope.
"""
I say "pretty much" because, for whatever reason(s), it seems to be
trying hard _not_ to use the word "function". But I can't guess what
"then passed as an argument to the implicitly nested scope" could
possibly mean otherwise (it doesn't make literal sense to "pass an
argument" to "a scope").
On 14 May 2018 at 04:05, Tim Peters
I say "pretty much" because, for whatever reason(s), it seems to be trying hard _not_ to use the word "function". But I can't guess what "then passed as an argument to the implicitly nested scope" could possibly mean otherwise (it doesn't make literal sense to "pass an argument" to "a scope").
I think my motivation was to avoid promising *exact* equivalence with a regular nested function, since the define-and-call may allow us opportunities for optimization that don't exist when those two are separated (e.g. Guido's point in another thread that we actually avoid calling "iter" twice even though the nominal expansion implies that we should). However, you're right that just calling it a function may be clearer than relying on the ill-defined phrase "implicitly nested scope". For Chris's actual question, this is part of why I think adding "parentlocal" would actually make the scoping proposal easier to explain, as it means the name binding semantics aren't a uniquely magical property of binding expressions (however spelled), they're just a new form of target scope declaration that the compiler understands, and the binding expression form implies. Note: eas*ier*, not easy ;) It also occurs to me that we could do something pretty neat for class scopes: have parent local declarations in methods target the implicit lexical scope where __class__ lives (to support zero-arg super), *not* the class body. That would entail adding a "classlocal" declaration to target that implied scope, though. That would give the following definition for "lexical scopes that parent local scoping can target": - module globals (parentlocal -> global) - function locals, including lambda expression locals (parentlocal -> nonlocal) - implicit class closure, where __class__ lives (parentlocal -> nonlocal in current scope, classlocal in class scope) Most notably, in the synthetic functions created for generator expressions and comprehensions, a parentlocal declaration in a child scope would imply a parentlocal declaration in the synthetic function as well, propagating back up the chain of nested lexical scopes until it terminated in one of the above three permitted targets. Using the explicit forms would then look like: from __future import parent_scopes # Enable the explicit declaration forms class C: classlocal _n # Declares _n as a cell akin to __class__ rather than a class attribute _n = [] @staticmethod def get_count(): return len(_n) assert not hasattr(C, "_n") assert C.get_count() == 0 def _writes_to_parent_scope(): parentlocal outer_name outer_name = 42 assert outer_name == 42 I'm still doubtful the complexity of actually doing that is warranted, but I'm now satisfied the semantics can be well specified in a way that allows us to retain the explanation of generator expressions and comprehensions in terms of their statement level counterparts (with the added bonus of making "__class__" a little less of a magically unique snowflake along the way). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
[ Tim, about the most version of the docs at https://docs.python.org/dev/reference/expressions.html#displays-for-lists-se... ]
I say "pretty much" because, for whatever reason(s), it seems to be trying hard _not_ to use the word "function". But I can't guess what "then passed as an argument to the implicitly nested scope" could possibly mean otherwise (it doesn't make literal sense to "pass an argument" to "a scope").
[Nick Coghlan
I think my motivation was to avoid promising *exact* equivalence with a regular nested function, since the define-and-call may allow us opportunities for optimization that don't exist when those two are separated (e.g. Guido's point in another thread that we actually avoid calling "iter" twice even though the nominal expansion implies that we should). However, you're right that just calling it a function may be clearer than relying on the ill-defined phrase "implicitly nested scope".
Plus that, as noted, what passing an argument "to a scope" means is mysterious. Language standard committees struggle for years with how to phrase things so that no more than is intended appears to be promised. It's hard! For example, if you were to show a workalike function and note that the exact placement - and number - of `iter()` calls is not guaranteed, someone else would point out that you need to explicitly say that by "iter" you mean the builtin function of that name, not one user code may have overridden it with in the current scope. Then someone else will note that it's tedious to say things like that whenever they're needed, and more-general text will be added elsewhere in the docs saying that the _rest_ of the docs always mean the language-supplied versions of such-&-such explicitly named functions/classes/modules/... I'd say "nested function" anyway ;-) And for another reason: not just someone from Mars is prone to misreading 'scope", but just about anyone on Earth coming from another language. The idea that the word "scope" all by itself implies "and in general any name bound to within the top-level code spanned by the scope is implicitly local to the scope unless explicitly declared `global` or `nonlocal` in the scope" may be unique to Python.
For Chris's actual question, this is part of why I think adding "parentlocal" would actually make the scoping proposal easier to explain, as it means the name binding semantics aren't a uniquely magical property of binding expressions (however spelled), they're just a new form of target scope declaration that the compiler understands, and the binding expression form implies. Note: eas*ier*, not easy ;)
Adding an explanation of `parentlocal` to the docs could be a useful pedagogical device, but I don't think I'd support adding that statement to the _language_. It's too weird, and seems to be at a wrong level for plausible future language developments. Let's step waaaaay back for a minute. In many languages with full-blown closures, first-class functions, and nested lexical scopes, it's pretty common to define the meaning of various language constructs in terms of calling derived lexically nested functions. In those languages, any "work variables" needed by the synthetic functions are declared as being local to those functions, and _that's the end of it_. They're done. All other names inside the expansions mean exactly the same as what they mean in whatever chunks of user-supplied code the construct interpolates into the synthesized functions. It doesn't matter one whit in which context(s) they appear. That's the only long-term sane way to go about defining constructs in terms of calling synthesized functions interpolating user-supplied pieces of code. Now _if_ Python had been able to do that, the meaning of genexps and listcomps would have been defined, from the start, in terms of synthesized functions that declared all & only the for-target names "local". And, in fact, the change I'm suggesting wouldn't have required changing the comprehension implementation _at all_ when assignment expressions were added. Instead the implementation would need to change to _add_ assignment expression targets to the things declared local if it was decided that those targets should be _local_ to the derived functions instead. That's why this all seems so bloody obvious to me ;-) It's how virtually every other language in the business of defining constructs in terms of nested synthesized functions works. So if that's something we may ever do again - and possibly even if we don't expect to ever do it again - I suggest a more generally useful approach would be to add a new flavor of _function_ to Python. Namely one wherein the only locals are the formal arguments and those explicitly declared local. Whether or not a name is bound in the body would be irrelevant. To avoid a new keyword, `local` could be spelled `not nonlocal` ;-) Note that the only use for `parentlocal` so far is tediously emulating the _effects_ of what that hypothetical `deflocal` flavor of function would do all the time with names not declared local in it. If we ever do something like this again, it would be far easier and clearer to just say the synthetic functions are of the `deflocal` flavor, and here are the names declared local in this case: x, y, z, ... User-defined functions may well find that useful at times too. Although it would be a large conceptual addition to part of Python, adding `parentlocal` would be too, and all by itself the latter looks like an incoherent pile of bizarre tricks. The _meaning_ of `deflocal` would be immediately clear to people coming from any number of other modern-ish languages.
It also occurs to me that we could do something pretty neat for class scopes: have parent local declarations in methods target the implicit lexical scope where __class__ lives (to support zero-arg super), *not* the class body. That would entail adding a "classlocal" declaration to target that implied scope, though.
That would give the following definition for "lexical scopes that parent local scoping can target":
- module globals (parentlocal -> global) - function locals, including lambda expression locals (parentlocal -> nonlocal)
Except that for top-level functions, parentlocal -> global; and regardless of nesting level, also implies `global` if the name is declared `global` in the parent block. Unless I've wholly lost track of your intent, which is quite possible. === no new content below ===
- implicit class closure, where __class__ lives (parentlocal -> nonlocal in current scope, classlocal in class scope)
Most notably, in the synthetic functions created for generator expressions and comprehensions, a parentlocal declaration in a child scope would imply a parentlocal declaration in the synthetic function as well, propagating back up the chain of nested lexical scopes until it terminated in one of the above three permitted targets.
Using the explicit forms would then look like:
from __future import parent_scopes # Enable the explicit declaration forms
class C: classlocal _n # Declares _n as a cell akin to __class__ rather than a class attribute _n = [] @staticmethod def get_count(): return len(_n)
assert not hasattr(C, "_n") assert C.get_count() == 0
def _writes_to_parent_scope(): parentlocal outer_name outer_name = 42
assert outer_name == 42
I'm still doubtful the complexity of actually doing that is warranted, but I'm now satisfied the semantics can be well specified in a way that allows us to retain the explanation of generator expressions and comprehensions in terms of their statement level counterparts (with the added bonus of making "__class__" a little less of a magically unique snowflake along the way).
participants (4)
-
Chris Angelico
-
Franklin? Lee
-
Nick Coghlan
-
Tim Peters