I agree the problem of the scope of type variables is thorny. I worry that thinking about it too much in terms of "fewest changes to the compiler" might over-constrain the solution space though.

I re-read some earlier messages in this thread. For a while Eric seemed in favor of putting typevars in a new scope, especially after hitting upon the idea of lambda lifting, but eventually soured on it for several reasons: compiler complexity, walrus operators in the inner scope, and worries about runtime overhead and debugger compatibility.

**Compiler complexity.**

This is the least worrisome to me. If it's the right thing for the user, we should do it even if it's complex to implement. (This is often a tenet of Python these days, despite what you may read in the Zen of Python.) The notion that occurrences of T are translated to an expression that at runtime constructs a new instance of some dummy type doesn't seem easy to explain to users, and I worry about 3rd party tools that do runtime introspection of annotations (not just runtime type checkers).

**Walrus in the inner scope.**

There are only two types of places where a walrus could occur: type annotations (for arguments and return value) and default values. I don't think a walrus in a type annotation would be useful (and I believe most static type checkers don't allow them, since annotations are required to follow a simpler syntax), so I assume this is about defaults. Example:
def foo[T](arg: T = (x := f())) -> T: ...
Using the lambda lifting approach we could compute the default in the outer scope and pass it into the helper:
def __foo(T, __arg):
    def foo(arg: T = __arg) -> T: ...
    return foo
foo = __foo(TypeVar("T"), (x := f()))
Now, this means we can't use a walrus in the type annotation, but I don't think that's a useful pattern. Certainly static type checkers don't allow it. It's likely that someone, somewhere is using a walrus in a type annotation that's only for consumption by some dynamic tool, but that (in my expectation highly uncommon) scenario could easily be fixed by moving the walrus out of the annotation into an assignment statement preceding the function definition:
# Old, would not work with lambda lifting:
def foo(arg: X := Y): ...

# New, also works in 3.11 and before:
TypeOfArg = (X := Y)
def foo(arg: TypeOfArg): ...
So I think it's fine to declare any use of a walrus in an annotation illegal. Or perhaps only when occurring in a class -- that's s similar constraint as PEP 572 imposes on using a walrus in a comprehension:
class C:
    a = [x := i for i in range(3)]
This gives the following error:
SyntaxError: assignment expression within a comprehension cannot be used in a class body

**Runtime overhead**

I presume this is a concern about the cost of the extra function definition and call used by lambda lifting. This would be the biggest concern for generic functions (class construction is already relatively slow). It would turn every (generic) function definition into two function definitions plus a call. A highly unscientific experiment using `timeit` tells me that on my computer the simplest definition costs 65 ns and the simplest call costs 35 ns. OTOH, calling `TypeVar("T")` costs 850 ns. So what are we even talking about? At best this concern is premature.

**Debugger compatibility**

I don't want to speculate about this, but I note that 3.11 broke compatibility for debuggers and we are addressing this (for the long term) by adding better APIs for debuggers to do what they want to do.

**Other stuff**

In a private message Eric mentioned to me that his current prototype generates bytecode to import e.g. TypeVar from the typing module. A production-quality implementation would have to reimplement TypeVar, TypeVarTuple, ParamSpec and Generic in C. That's a fair amount of work, but I'm confident that we can do it if the PEP is accepted. As long as it's a proof-of-concept prototype I don't think it will matter.

**Final words**

Despite several hurdles, I still favor the introduction of an actual new scope holding the type variables belonging to a given generic class, function or type alias, over Eric's (indubitably clever) current hack.


PS. Regarding the syntax to indicate subclass constraints, I am fine with ":".

On Fri, Jul 1, 2022 at 10:38 AM Eric Traut <eric@traut.com> wrote:
This is a highly constrained problem, so I think our options are rather limited.

I agree that the solution I've proposed feels somewhat inelegant, but by comparison to everything else I've explored, it's the simplest solution that addresses all of the requirements and constraints. I'm open to other ideas if someone has a better suggestion. Given my current understanding of the way the Python compiler works and the constraints and requirements that we've discussed, I don't see any better options.

Let's take a deep look at the implications of my proposed implementation and see if we can convince ourselves that they're acceptable in terms of future extensibility.

The main problems I see with my proposed approach are:
1. If you "eval" a snippet of code that refers to a type parameter defined by an outer scope, that "eval" operation will fail unless you manually include symbols in the `locals` dictionary that are stand-ins for the active type parameters. This may be problematic for runtime type checkers.
2. At runtime, you receive a different object each time you refer to a type parameter. That's because it generates a proxy object rather than referring to a common object that represents the parameter. It's similar to what happens if you use the expression `[]` multiple times in your code; each time a new list object will be constructed.
3. I'm not sure how existing Python debuggers will handle type parameters. It depends on the introspection mechanism they currently use when evaluating expressions.

If you want to play with the current implementation, I have a fork of cpython available. Here's the link: https://github.com/erictraut/cpython/tree/type_param_syntax

- Eric
Typing-sig mailing list -- typing-sig@python.org
To unsubscribe send an email to typing-sig-leave@python.org
Member address: guido@python.org

--Guido van Rossum (python.org/~guido)