Jella and I met yesterday, and he provided me with additional insights about how runtime type checkers use type information. From that conversation, I concluded that my proposed approach (the one that involves type parameter "proxies") probably wouldn't meet the needs of runtime type checkers. We did some additional brainstorming, and I spent some time today exploring alternatives. I think I have a design that meets all of the requirements.
Like the previous proposal, this new design doesn't require the use of "lambda lifting", which I still consider problematic for a number of reasons.
The new design involves two new fields in the "frame" object at runtime. These fields track which type parameters are "live" at runtime. Both of these fields can be NULL or refer to a tuple of TypeVar-like objects (TypeVar, TypeVarTuple, ParamSpec). The first field is called "f_outer_typeparams" and contains all of the type parameters defined by outer scopes. The second field is called "f_local_typeparams". It contains all of the outer type parameters plus any type parameters that are temporarily needed for a generic "class", "def" or "type" statement in the current scope. This proposal introduces two new opcodes: EXTEND_TYPEPARAMS and LOAD_TYPEPARAM. The EXTEND_TYPEPARAMS op builds a new "f_local_typeparams" tuple from the "f_outer_typeparams" tuple and n new TypeVar-like objects that have been pushed onto the stack. The LOAD_TYPEPARAM op loads a single type parameter from "f_local_typeparams" referenced by numeric index. The compiler tracks the indices of all "live" type parameters
, so it's able to emit the appropriate index as part of the opcode. When a new scope is entered (e.g. during a function or lambda call or the execution of a comprehension), the "f_local_typeparams" tuple is copied into the "f_outer_typeparams" field of the next scope. Through this mechanism, all of the "live" type parameter objects are always available even in inner scopes.
Here's an example:
```python
# At this point, f_outer_typeparams and f_local_typeparams are NULL
# The compiler emits code to construct two TypeVar objects for `A` and `B`, then
# emits a EXTEND_TYPEPARAMS(2) op. This builds a new f_local_typeparams tuple
# that contains (A, B). When evaluating the expression `dict[A, B]`, the reference to
# `A` generates a LOAD_TYPEPARAM(0), and the reference to `B` generates a
# LOAD_TYPEPARAM(1) op.
class Outer[A, B](dict[A, B]):
# At this point, f_outer_typeparams and f_local_typeparams contain (A, B).
# The compiler emits code to construct two TypeVar objects for `C` and `D`, then
# emits a EXTEND_TYPEPARAMS(2) op. This builds a new f_local_typeparams tuple
# that contains (A, B, C, D). When evaluating the expression `dict[B, D]`, the reference to
# `B` generates a LOAD_TYPEPARAM(1), and the reference to `D` generates a
# LOAD_TYPEPARAM(3) op.
class Inner[C, D](dict[B, D]): ...
# At this point, f_outer_typeparams and f_local_typeparams contain (A, B).
# The compiler emits code to construct one TypeVar object for `X`, then
# emits a EXTEND_TYPEPARAMS(1) op. This builds a new f_local_typeparams tuple
# that contains (A, B, X). When evaluating the type annotations for parameters `a`,
# `b` and `x`, the appropriate LOAD_TYPEPARAM ops are generated.
def method[X](self, a: A, b: B, x: X): ...
```
> print(Outer.__parameters__)
(~A, ~B)
>print(Outer.Inner.__parameters__)
(~C, ~D)
>print(Outer.method.__annotations__)
{'a': ~A, 'b': ~B, 'x': ~X}
The f_outer_typeparams value is also stored in a function object in a new internal field called func_typeparams. This allows allows f_outer_typeparams to be restored to the current scope when resuming a coroutine, executing an async function, calling a lambda, etc.
Other than the addition of two new fields to the frame object, one new field in the function object, and two new opcodes, this proposal adds no significant complexity to the compiler or the runtime. Performance impact should be negligible. It works well with the PEP 649 proposal for deferred evaluation of annotations, and it works well with today's (non-deferred) annotation evaluation.
One small complexity is with forward-referenced annotations and the use of `get_type_hints()`, which will need to populate the `locals` dictionary with the "live" type parameters before calling `eval()` on the annotation string. The live type parameters are available on function, class, and type alias objects via a new `__typeparams__` attribute.
>print(Outer.__typeparams__)
(~A, ~B)
>print(Outer.Inner.__typeparams__)
(~A, ~B, ~C, ~D)
>print(Outer.method.__typeparams__)
(~A, ~B, ~X)
Let me know if you see any holes or have concerns with this proposal. If you're interested in looking at the CPython implementation, check out this branch: https://github.com/erictraut/cpython/commits/type_param_syntax2.
Assuming that this design is amenable to the typing community, my next step is to post an update to the PEP that describes the design. At that point, I think we can notify the python-dev community that the PEP is ready for their review.
-Eric
_______________________________________________
Typing-sig mailing list -- typing-sig@python.org
To unsubscribe send an email to typing-sig-leave@python.org
https://mail.python.org/mailman3/lists/typing-sig.python.org/
Member address: guido@python.org