I have drafted an initial PEP for TypeForm, a way to spell the type of typing special forms (like Union[int, str], Literal['foo'], etc). Please see the following commentable document, or the copy at the bottom of this email: https://docs.google.com/document/d/18UF8V00EVU1-h-BtiVFhXoJkvfL4rHp4ORaenMQL... Please leave your comments either on the document itself or by responses to this email on typing-sig. I will integrate feedback periodically. For further background on why this feature is being introduced and prior discussions, please see: (1) the originating thread on typing-sig at: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ... (2) the discussion about TypeForm on the mypy issue tracker at: https://github.com/python/mypy/issues/9773 -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: TypeForm: Type Hints for Typing Special Forms and Regular Types Author: David Foster <david@dafoster.net> Sponsor: TODO Discussions-To: typing-sig@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 647 (TypeGuard) Created: 21-Dec-2020 Python-Version: 3.10 Post-History: 24-Jan-2021 Abstract ======== PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. [^type-c]: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects Motivation ========== The introduction of `TypeForm` allows new kinds of metaprogramming functions that operate on typing special forms to be type-annotated. For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value: ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... ``` And here is another function that checks whether a value is assignable to a variable of a particular type, and if so returns `True`: ``` def isassignable(value: object, form: TypeForm) -> bool: ... ``` With the introduction of `TypeGuard` in PEP 647[^TypeGuardPep] the above function can be enhanced to return a `TypeGuard` instead of a regular `bool`: [^TypeGuardPep]: https://www.python.org/dev/peps/pep-0647/ ``` def isassignable(value: object, form: TypeForm[T]) -> TypeGuard[T]: ... ``` Without `TypeForm` the next-best type that could be given to the `form` parameter in the above examples would be `object` which would improperly allow values like `1` to be passed in. More importantly, there would be no way to express the relationship between the parameter type of `form` and the function's return type. NB: The preceding example functions implement the kinds of enhanced `isinstance` checks that were omitted in PEP 589[^typeddict-no-isinstance] which are very useful for, among other things, [checking whether a value decoded from JSON conforms to a particular structure] of nested `TypedDict`s, `List`s, `Optional`s, `Literal`s, and other types. [^typeddict-no-isinstance]: https://www.python.org/dev/peps/pep-0589/#using-typeddict-types [checking whether a value decoded from JSON conforms to a particular structure]: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ... Specification ============= A type-form type represents a `type` object or a special typing form such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. A type-form type can be written as either `TypeForm[T]` where `T` is a type variable or as `TypeForm` with no argument. The `T` in `TypeForm[T]` is always an invariant type variable with no bound: * Attempting to use a type variable `T` declared with a `bound=...` or marked with `covariant=True` or `contravariant=True` in a `TypeForm[T]` should be rejected by typecheckers. * Attempting to use a literal type-form `*form*` as an argument to `TypeForm[*form*]`, such as `TypeForm[int]`, should be rejected by typecheckers. (See §"TypeForm[*form*] or Literal[*form*]" in Rejected Ideas for rationale.) The syntax `TypeForm` alone, without a type argument, is equivalent to `TypeForm[_T]` where `_T` is a freshly created type variable. ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... def isassignable(value: object, form: TypeForm) -> bool: ... def is_type(form: TypeForm) -> bool: ... ``` Using TypeForm Types -------------------- Type-form types may be used as function parameter types, return types, and variable types: ``` def is_type(form: TypeForm) -> bool: ... # parameter type ``` ``` S = TypeVar('S') T = TypeVar('T') U = TypeVar('U') def meet_types(s: TypeForm[S], t: TypeForm[T]) \ -> Union[TypeForm[S], TypeForm[T], TypeForm[U]]: ... # return types ``` ``` NULL_TYPE: TypeForm # variable type NULL_TYPE = type(None) ``` ``` NULL_TYPE: TypeForm = type(None) # variable type ``` Note however that a typechecker won't automatically infer a TypeForm type for an unannotated variable assignment that contains a special typing form on the right-hand-side because PEP 484 [^type-alias-syntax] reserves that syntax for defining type aliases: [^type-alias-syntax]: https://www.python.org/dev/peps/pep-0484/#type-aliases ``` NULL_TYPE = type(None) # OOPS; treated as a type alias! ``` If you want a typechecker to infer a TypeForm variable type for a bare assignment, use the walrus operator instead of the assignment operator: ``` NULL_TYPE := type(None) # infers NULL_TYPE as having type TypeForm ``` Or explicitly declare the assignment-target as having `TypeForm` type: ``` NULL_TYPE: TypeForm = type(None) ``` ``` NULL_TYPE = type(None) # type: TypeForm # the type comment is significant ``` ``` NULL_TYPE: TypeForm NULL_TYPE = type(None) ``` Values of type TypeForm ----------------------- A particular literal value is said to *inhabit* a type if it is described by that type and can be assigned to a variable of that type. The type `TypeForm` is inhabited by exactly those runtime objects that are valid on the right-hand-side of a variable declaration, ``` value: *form* ``` the right-hand-side of a parameter declaration, ``` def some_func(value: *form*): ``` *and* as the return type of a function: ``` def some_func() -> *form*: ``` Any runtime object that is not valid in one of the above locations is not an inhabitant of `TypeForm`. For example the special forms `Final`, `Final[int]`, `NoReturn`, and `ClassVar[int]` are not inhabitants. Incomplete forms like a bare `Optional` or `Union` are also not inhabitants. Example of inhabitants include: * type objects like `int`, `str`, `object`, and `FooClass` * generic collections like `List`, `List[int]`, `Dict`, or `Dict[K, V]` * callables like `Callable`, `Callable[[Arg1Type, Arg2Type], ReturnType]`, `Callable[..., ReturnType]` * union forms like `Optional[str]` or `Union[int, str]` * literal forms like `Literal['r', 'rb', 'w', 'wb']` * type variables like `T` or `AnyStr` * annotated types like `Annotated[int, ValueRange(-10, 5)]` * type aliases like `Vector` (where `Vector = list[float]`) * the `Any` form * the `Type` and `Type[C]` forms * the `TypeForm` and `TypeForm[T]` forms A few peculiar cases permitted by the above include: ``` STR_FORM: TypeForm = str STR_OR_SUBCLASS_FORM: TypeForm = Type[str] SOME_FORM: TypeForm = TypeForm ``` Forward References '''''''''''''''''' Runtime objects for forms which contain forward references such as `Union['Triangle', 'Shape']` are normalized at runtime to `Union[ForwardRef('Triangle'), ForwardRef('Shape')]`. The latter are inhabitants of `TypeForm`. Additionally the "top-level" form for a particular variable type can itself be a forward reference such as in: ``` class Node: value: object next: 'Node' # 'Node' is normalized to ForwardRef('Node') at runtime ``` Code that manipulates TypeForm values at runtime should be prepared to deal with ForwardRef(...) subcomponents. Type Consistency ---------------- Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the `Any` type. It is defined more formally in PEP 483 [^type-consistency]. This section introduces the new rules needed to support type consistency for TypeForm types: [^type-consistency]: https://www.python.org/dev/peps/pep-0483/#summary-of-gradual-typing * `Type[C]` is consistent with `TypeForm`. `TypeForm` is consistent with `object`. * `TypeForm[S]` is consistent with `TypeForm[T]` iff `S` and `T` are the same type variable. * `Any` is consistent with `TypeForm`. `TypeForm` is consistent with `Any`. Overloading ----------- `TypeForm` can be used in overloads. For example: ``` T = TypeVar('T') F = TypeVar('F') @overload def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... @overload def trycast(form: TypeForm[T], value: object, failure: F) -> Union[T, F]: ... ``` Interactions with Type[C] ------------------------- A `TypeForm` and a `Type` cannot simultaneously constrain the same type variable within a generic function definition: ``` def bad_func(form: TypeForm[T], tp: Type[T]): ... # invalid ``` A group of type variables where some variables are constrained by `TypeForm`s, some are constrained by `Type`s, and others are unconstrained may appear together as arguments to the same type constructor (i.e. `Union[...]`, `Dict[K, V]`, etc): ``` F = TypeVar('F') C = TypeVar('C') V = TypeVar('V') def pick_random(form: TypeForm[F], tp: Type[C], value: V) -> Union[F, C, V]: ... ``` Interactions with cast ---------------------- It is possible to cast a value to a `TypeForm` or `TypeForm[T]` type: ``` OPTIONAL_STR_OBJECT: object = Optional[str] OPTIONAL_STR = cast(TypeForm, OPTIONAL_STR_OBJECT) ``` Type Inference -------------- A runtime object corresponding to a special form should be inferred as having type `TypeForm` unless it can instead be inferred to have type `Type[C]` for some `C`: ``` T = TypeVar('T') def identity(value: T) -> T: return value A_TYPE = identity(str) # inferred type is: Type[str] A_FORM = identity(Optional[str]) # inferred type is: TypeForm ``` Backwards Compatibility ======================= Previously the type `object` would be inferred for runtime objects corresponding to a special form like `Optional[str]` rather than `TypeForm`. But since `TypeForm` is consistent with `object` no unexpected typechecking failures will be introduced by this change. Ongoing Maintenance =================== As new kinds of typing special forms are introduced by upcoming PEPs (usually to the `typing` module), it will be necessary for typecheckers to define whether or not the new forms should be considered inhabitants of `TypeForm` or not, based on the general rules in §"Values of type TypeForm" above. Therefore it is recommended that any new PEP introducing a new typing special form should define whether it inhabits `TypeForm` or not. Reference Implementation ======================== The following will be true when [mypy#9773](https://github.com/python/mypy/issues/9773) is implemented: The mypy type checker supports `TypeForm` types. A reference implementation of the runtime component is provided in the `typing_extensions` module. In addition an implementation of the `trycast` and `isassignable` functions mentioned in various examples above is being implemented in the [trycast PyPI module](https://github.com/davidfstr/trycast). Rejected Ideas ============== TypeForm[*form*] or Literal[*form*] ----------------------------------- This PEP only defines syntax for a bare `TypeForm` and a `TypeForm[T]` parameterized by a type variable. In particular it does not define syntax like that allows referring to a specific known type-form value via a syntax like `TypeForm[*form*]`: ``` NULL_TYPE := type(None) # has type TypeForm[_T] rather than TypeForm[type(None)] UNION_STR_TYPE := Union[Str] # has type TypeForm[_T] rather than TypeForm[Union[str]] ``` In addition similar syntax like `Literal[*form*]` also does not work. Neither of those syntaxes seem to be especially useful; type forms are not typically used as sentinel values. It is possible that a future PEP may revisit such syntaxes. Open Issues =========== [Any points that are still being decided/discussed.] References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<