I agree that compiler complexity by itself shouldn't dissuade us from considering a particular solution. On the other hand, any time a solution starts to feel overly complex, my "engineering spidey sense" kicks in and tells me that I'm probably on the wrong path. And in this case, my spidey sense was tingling. :)
It turns out that complexity alone isn't the only problem with this approach.
Lambda lifting works OK for global and function scopes. However, it has problems when used within a class scope. The problem is that symbols declared within a class scope are no longer visible within the lambda. Class-scoped symbols cannot be captured in an inner scope, so any annotation in a "def" statement or base class or keyword argument in a "class" statement will generate a runtime error if it refers to a class-scoped symbol.
```python class Foo: class A: ... class B[T](A): ... # Error: A is not defined def foo[T](self, a: A): ... # Error: A is not defined ```
We could introduce the idea of a "read-only captured variable" that doesn't require a cell variable. That addresses the problem, but it involves a significant change to the compiler and (depending on the implementation) also to the runtime structures. It also introduces other problems...
Let's assume that we implement "read-only captured variables". Consider the following:
```python class Outer: class A: ... class B[T](A): print(A) ```
Today, the `print` statement would fail because it references `A` which is not visible inside the class `B` body. But if it becomes visible within the implied lambda, it will also be accessible to inner scopes like `B`.
Even weirder is this case:
```python def foo(): A = 0
class Outer: class A: ... class B[T](A): nonlocal A print(A) ```
Today, the `A` referenced by the print statement refers to the `A` in the outer scope `foo`. If we were to introduce an implied lambda, it would reference the `A` in the implied lambda, which happens to be the `A` in `Outer`. Definitely non-intuitive.
There is still another problem here related to forward-reference annotations. This solution depends on the compiler having visibility into which class-scoped symbols are referenced from within the lambda. Unfortunately, there are cases where the compiler doesn't have such visibility — namely, with forward-referenced annotations. Consider the following, which works today.
```python def outer(): def inner[T]() -> "A": return 3
A = int ```
When the forward-referenced annotation "A" is later evaluated through `get_type_hints()`, it will fail with lambda lifting whereas it succeeds today. We could decide that it is OK to break this use case, but I consider this yet another strike against the lambda lifting solution.
**Walrus in the inner scope.**
The issue here is not with default argument values (which can easily be evaluated in the outer scope) or annotations (which should never contain walrus operators, as you point out). The problem is with base classes and keyword argument expressions within a class declaration. Although it's unlikely that someone would use a walrus operator here, they are permitted today.
```python class Foo(x := Base, metaclass=(y := Metaclass)): ... ```
We could disallow walrus operators in these situations when the new syntax is used. Probably not a big deal, but it is an odd special case that would need to be documented and justified to the SC. Yet another strike against this approach.
In addition to the runtime overhead that you've mentioned (which I agree is probably not significant enough to be concerned about), we have to also consider:
1. It will generate additional cell variables and captures. Accesses these variables requires double dereferencing in both the inner scope and outer scopes. 2. If we implement the "read-only captured variable" idea mentioned above, the cost of passing additional arguments to the lambda. That cost will depend on the number of class-scoped variables that are referenced within annotations (for "def" statements) and base classes plus keyword arguments (for "class" statements).
I could easily be convinced that these also don't represent enough runtime overhead to be concerned about, but I mention them for completeness.
3.11 broke compatibility for debuggers and we are addressing this
OK, that's good to know.
Despite several hurdles, I still favor the introduction of an actual new scope
Let me know what you think about my points above. If you're still in favor of adding a new scope, I'm game to give it a try and prototype it.