The fourth draft for the TypeForm PEP is now ready! Major changes: * Improvements to motivation - Why not use Type[C] instead of TypeForm[T]? - Explain more in: §"Rejected Ideas" > §"Widen Type[T] to support all typing special forms" * Add/revise examples of functions that use TypeForm. * Explain more about the T in TypeForm[T]. No longer restrict it to being a TypeVar. * Clarify that a bare `TypeForm` is the same as `TypeForm[Any]`. * Eliminate redundant section on overloads. * Eliminate special rules around TypeVar constraint solving that involves a TypeForm. I have highlighted changed sections by marking them with ⭐. The PEP really seems to be coming together now. Next big steps I believe are: * Find a sponsor * RST-ize the PEP and submit a PR to the PEPs repo -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: TypeForm: Type Hints for Typing Special Forms and Regular Types Author: David Foster <david at dafoster.net> Sponsor: TODO Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 647 (TypeGuard) Created: 21-Dec-2020 Python-Version: 3.10 Post-History: 24-Jan-2021, 28-Jan-2021, 04-Feb-2021, 14-Feb-2021 Abstract ======== PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. [^type-c]: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects Motivation ========== The introduction of `TypeForm` allows new kinds of metaprogramming functions that operate on typing special forms to be type-annotated. For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value: ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... ``` It's very powerful to be able to define a function whose return type can be influenced by a `form` value passed at runtime. Here is another function that checks whether a value is assignable to a variable of a particular type, and if so returns True (as a special TypeGuard bool [^TypeGuardPep]): [^TypeGuardPep]: https://www.python.org/dev/peps/pep-0647/ ``` def isassignable(value: object, form: TypeForm[T]) -> TypeGuard[T]: ... ``` When combining TypeForm and TypeGuard together in this way, typecheckers can understand the relationship between the type provided to parameter `form` and the function's return type, and narrow the return type appropriately depending on what form is passed in: ``` request_json = ... # type: object assert isassignable(request_json, Shape) # request_json is narrowed to type Shape here! ``` NB: The preceding example functions implement the kinds of enhanced `isinstance` checks that were omitted in PEP 589[^typeddict-no-isinstance] which are very useful for, among other things, [checking whether a value decoded from JSON conforms to a particular structure] of nested `TypedDict`s, `List`s, `Optional`s, `Literal`s, and other types. [^typeddict-no-isinstance]: https://www.python.org/dev/peps/pep-0589/#using-typeddict-types [checking whether a value decoded from JSON conforms to a particular structure]: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ... ⭐ One might think you could define the `trycast` or `isassignable` examples above to take a `Type[T]` - which is syntax that already exists - rather than a `TypeForm[T]`. However if you were to do that then certain typing special forms like `Optional[str]` - which are not class objects and therefore not `Type`s at runtime - would be rejected: ``` # uses a Type[T] parameter rather than a TypeForm[T] def trycast_type(form: Type[T], value: object) -> Optional[T]: ... trycast_type(str, 'hi') # ok; str is a Type trycast_type(Optional[str], 'hi') # ERROR; Optional[str] is not a Type trycast_type(Union[str, int], 'hi') # ERROR; Union[str, int] is not a Type trycast_type(MyTypedDict, dict(value='hi')) # questionable usage; but ok in Python 3.9 ``` For a longer explanation of why we don't just widen `Type[T]` to accept all special forms, see §"Widen Type[T] to support all typing special forms" in §"Rejected Ideas". Specification ============= A type-form type represents a `type` object or a special typing form such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. A type-form type can be written as either `TypeForm[T]` where `T` is an type variable or as `TypeForm` with no argument. ⭐ The type parameter `T` in `TypeForm[T]` describes the type of an *instance* of `TypeForm[T]`. For example, `TypeForm[Union[int, str]]` describes a type that, when instantiated, is either an instance of `int` or an instance of `str`. The parameter is invariant, so an instance of `TypeForm[object]` *must* be the value `object` and cannot be, for example, `str` or any other subtype of `object`. ⭐ The syntax `TypeForm` alone, without a type argument, is equivalent to `TypeForm[Any]`. ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... def isassignable(value: object, form: T) -> TypeGuard[T]: ... def is_type(form: TypeForm) -> bool: ... ``` Here are some complete functions that use `TypeForm`: ⭐ ``` from typing import TypeForm, TypeVar TargetType = TypeVar('TargetType', int, float, str) def convert(value: object, form: TypeForm[TargetType]) -> TargetType: if form is str: return str(value) if form is int: return int(value) if form is float: return float(value) raise NotImplementedError('unsupported TargetType') print(convert('1', int) + convert('1.2', float)) ``` ⭐ ``` from typing import cast, get_args, get_origin, List, NoReturn, Sequence, TypeForm def deunionize(form: TypeForm) -> Tuple[TypeForm, ...]: """Decomposes a union or standalone type to its individual type components.""" if form is NoReturn: return () if get_origin(form) is Union: return get_args(form) else: return (form,) def unionize(forms: Sequence[TypeForm]) -> TypeForm: """Composes multiple type components to a single union type.""" components = [] # type: List[TypeForm] for form in forms: components.extend(deunionize(form)) if len(components) == 0: return NoReturn else: return Union.__getitem__(*components) # Union[*components] ``` ⭐ For an even longer example of a function that could use TypeForm, please see the initial implementation of the [trycast] function. [trycast]: https://github.com/davidfstr/trycast/blob/65859341ff9bab13fd5c39c0f5b1721d5d... Using TypeForm Types -------------------- TypeForm types may be used as function parameter types, return types, and variable types: ``` def is_type(form: TypeForm) -> bool: ... # parameter type ``` ``` S = TypeVar('S') T = TypeVar('T') def meet_types(s: TypeForm[S], t: TypeForm[T]) \ -> Union[TypeForm[S], TypeForm[T], TypeForm]: ... # return types ``` ``` NULL_TYPE: TypeForm # variable type NULL_TYPE = type(None) ``` ``` NULL_TYPE: TypeForm = type(None) # variable type ``` Note however that a typechecker won't automatically infer a TypeForm type for an unannotated variable assignment that contains a special typing form on the right-hand-side because PEP 484 [^type-alias-syntax] reserves that syntax for defining type aliases: [^type-alias-syntax]: https://www.python.org/dev/peps/pep-0484/#type-aliases ``` NULL_TYPE = type(None) # OOPS; treated as a type alias! ``` If you want a typechecker to use a TypeForm variable type for a bare assignment you'll need to explicitly declare the assignment-target as having `TypeForm` type: ``` NULL_TYPE: TypeForm = type(None) ``` ``` NULL_TYPE = type(None) # type: TypeForm # the type comment is significant ``` ``` NULL_TYPE: TypeForm NULL_TYPE = type(None) ``` Values of type TypeForm ----------------------- The type `TypeForm` has values corresponding to exactly those runtime objects that are valid on the right-hand-side of a variable declaration, ``` value: *form* ``` the right-hand-side of a parameter declaration, ``` def some_func(value: *form*): ``` or as the return type of a function: ``` def some_func() -> *form*: ``` Any runtime object that is valid in one of the above locations is a value of `TypeForm`. Incomplete forms like a bare `Optional` or `Union` are not values of `TypeForm`. Example of values include: * type objects like `int`, `str`, `object`, and `FooClass` * generic collections like `List`, `List[int]`, `Dict`, or `Dict[K, V]` * callables like `Callable`, `Callable[[Arg1Type, Arg2Type], ReturnType]`, `Callable[..., ReturnType]` * union forms like `Optional[str]`, `Union[int, str]`, or `NoReturn` * literal forms like `Literal['r', 'rb', 'w', 'wb']` * type variables like `T` or `AnyStr` * annotated types like `Annotated[int, ValueRange(-10, 5)]` * type aliases like `Vector` (where `Vector = list[float]`) * the `Any` form * the `Type` and `Type[C]` forms * the `TypeForm` and `TypeForm[T]` forms Forward References '''''''''''''''''' Runtime objects for forms which contain forward references such as `Union['Triangle', 'Shape']` are normalized at runtime to `Union[ForwardRef('Triangle'), ForwardRef('Shape')]`. These `ForwardRef(...)` objects are also values of TypeForm. Code that manipulates TypeForm values at runtime should be prepared to deal with ForwardRef(...) values. Type Consistency ---------------- Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the `Any` type. It is defined more formally in PEP 483 [^type-consistency]. This section introduces the new rules needed to support type consistency for TypeForm types: [^type-consistency]: https://www.python.org/dev/peps/pep-0483/#summary-of-gradual-typing * `type`, `Type`, and `Type[C]` are consistent with `TypeForm`. `TypeForm` is consistent with `object`. ⭐ * `TypeForm[S]` is consistent with `TypeForm[T]` iff `S` and `T` are the same type. * `Any` is consistent with `TypeForm`. `TypeForm` is consistent with `Any`. Notably `TypeForm` is *not* consistent with `type`, `Type`, or `Type[C]`. ⭐ Interactions with Type[C] ------------------------- A group of type variables where some variables are constrained by TypeForms, some are constrained by `Type`s, and others are unconstrained may appear together as arguments to the same type constructor (i.e. `Union[...]`, `Dict[K, V]`, etc): ``` F = TypeVar('F') C = TypeVar('C') V = TypeVar('V') # Union[F, C, V] contains type variables with many kinds of constraints def pick_random(form: TypeForm[F], tp: Type[C], value: V) -> Union[F, C, V]: return random.choice((form, tp, value)) ``` No additional type-checking rules should be need to support this case; the normal rules of TypeVar constraint solving apply here. Backwards Compatibility ======================= No other backward incompatible changes are made by this PEP. Reference Implementation ======================== The following will be true when [mypy#9773](https://github.com/python/mypy/issues/9773) is implemented: The mypy type checker supports `TypeForm` types. A reference implementation of the runtime component is provided in the `typing_extensions` module. In addition an implementation of the `trycast` and `isassignable` functions mentioned in various examples above is being implemented in the [trycast PyPI module](https://github.com/davidfstr/trycast). Rejected Ideas ============== ⭐ Widen Type[T] to support all typing special forms ------------------------------------------------- `Type` was [designed] to only be used to describe class objects. A class object can always be instantiated by calling it and can always be used as the second argument of `isinstance()`. `TypeForm` on the other hand is typically introspected by the user in some way, is not necessarily directly instantiable, and is not necessary directly usable in a regular `isinstance()` check. It would be possible to widen `Type` to include the additional values allowed by `TypeForm` but it would reduce clarity about the user's intentions when working with a `Type`. Different concepts and usage patterns; different spellings. [designed]: https://mail.python.org/archives/list/typing-sig@python.org/message/D5FHORQV... Restricting TypeForm to "universal" forms only ---------------------------------------------- Originally it was considered important to restrict TypeForm such that its values only included forms that could be used as a local variable type, a parameter type, *and* as a parameter type. For example that would rule out `NoReturn` as a value because it can only be used as a return type and not as a local variable type. Functions taking TypeForms as parameters that only wish to accept "universal" forms (that can be used in all 3 of the aforementioned positions) can write code that raises TypeError or a similar exception at runtime when provided a non-universal form. Standardized type inference rules --------------------------------- This PEP intentionally does not specify any rules for implicitly inferring a `TypeForm` type for an expression. No other type-related PEPs prescribe inference rules, and existing type checkers vary in their behavior. So it would be inconsistent to dictate an inference rule for TypeForm. Individual type checkers may define their own inference rules if they wish. Open Issues =========== [Any points that are still being decided/discussed.] References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<