The second draft of the TypeForm PEP is ready for another look! The first draft specified certain syntax and semantics that were more complex than necessary. And that complexity could have outweighed the benefit of providing the ability to write powerful type introspection functions for advanced users. My hope is that this second draft will be considered significantly simpler and cheaper to implement, making the value added by this PEP more competitive. Major changes in this draft: * Tightened up the Motivation section. * Retired `TypeForm[T]` syntax in favor of `TypeVar('T', bound=TypeForm)`, simplifying and eliminating many sections. * Add examples of complete functions that use TypeForm. * Removed the `var := <form>` trick, since it isn't legal syntax. * Removed the "Type Inference" section entirely. This also eliminated the only backward-compatibility concern. * Removed ban on "non-universal typing forms", simplifying value recognition rules and eliminating the "Ongoing Maintenance" section. * Reworded "inhabit" to be either "value of" or "is consistent with" as appropriate. Again, please leave your comments either as responses to this email or as inline comments in the original document. A copy is at the bottom of this email and at the link: * https://docs.google.com/document/d/18UF8V00EVU1-h-BtiVFhXoJkvfL4rHp4ORaenMQL... -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: TypeForm: Type Hints for Typing Special Forms and Regular Types Author: David Foster <david@dafoster.net> Sponsor: TODO Discussions-To: typing-sig@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 647 (TypeGuard) Created: 21-Dec-2020 Python-Version: 3.10 Post-History: 24-Jan-2021, 28-Jan-2021 Abstract ======== PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. [^type-c]: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects Motivation ========== The introduction of `TypeForm` allows new kinds of metaprogramming functions that operate on typing special forms to be type-annotated. For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value: ``` T = TypeVar('T', bound=TypeForm) def trycast(form: T, value: object) -> Optional[T]: ... ``` It's very powerful to be able to define a function whose return type can be influenced by a `form` value passed at runtime. Here is another function that checks whether a value is assignable to a variable of a particular type, and if so returns True (as a special TypeGuard bool [^TypeGuardPep]): [^TypeGuardPep]: https://www.python.org/dev/peps/pep-0647/ ``` T = TypeVar('T', bound=TypeForm) def isassignable(value: object, form: T) -> TypeGuard[T]: ... ``` When combining TypeForm and TypeGuard together in this way, typecheckers can understand the relationship between the type provided to parameter `form` and the function's return type, and narrow the return type appropriately depending on what form is passed in: ``` request_json = ... # type: object assert isassignable(request_json, Shape) # request_json is narrowed to type Shape here! ``` NB: The preceding example functions implement the kinds of enhanced `isinstance` checks that were omitted in PEP 589[^typeddict-no-isinstance] which are very useful for, among other things, [checking whether a value decoded from JSON conforms to a particular structure] of nested `TypedDict`s, `List`s, `Optional`s, `Literal`s, and other types. [^typeddict-no-isinstance]: https://www.python.org/dev/peps/pep-0589/#using-typeddict-types [checking whether a value decoded from JSON conforms to a particular structure]: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ... Specification ============= A type-form type represents a `type` object or a special typing form such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. A type-form type is written as `TypeForm`. `TypeForm` can be used as a standalone type or as a bound for a `TypeVar`: ``` T = TypeVar('T', bound=TypeForm) def trycast(form: T, value: object) -> Optional[T]: ... def isassignable(value: object, form: T) -> TypeGuard[T]: ... def is_type(form: TypeForm) -> bool: ... ``` Here are some complete functions that use `TypeForm`: ``` from typing import cast, get_args, get_origin, List, NoReturn, TypeForm def deunionize(form: TypeForm) -> List[TypeForm]: """Decomposes a union or standalone type to its individual type components.""" if form is NoReturn: return [] if get_origin(form) is Union: return cast(List[TypeForm], get_args(form)) else: return [form] def unionize(forms: List[TypeForm]) -> TypeForm: """Composes multiple type components to a single union type.""" components = [] # type: List[TypeForm] for form in forms: components.extend(deunionize(form)) if len(components) == 0: return NoReturn else: return Union[*components] ``` Using TypeForm Types -------------------- TypeForm types may be used as function parameter types, return types, and variable types: ``` def is_type(form: TypeForm) -> bool: ... # parameter type ``` ``` S = TypeVar('S', bound=TypeForm) T = TypeVar('T', bound=TypeForm) def meet_types(s: S, t: T) -> Union[S, T, TypeForm]: ... # return types ``` ``` NULL_TYPE: TypeForm # variable type NULL_TYPE = type(None) ``` ``` NULL_TYPE: TypeForm = type(None) # variable type ``` Note however that a typechecker won't automatically infer a TypeForm type for an unannotated variable assignment that contains a special typing form on the right-hand-side because PEP 484 [^type-alias-syntax] reserves that syntax for defining type aliases: [^type-alias-syntax]: https://www.python.org/dev/peps/pep-0484/#type-aliases ``` NULL_TYPE = type(None) # OOPS; treated as a type alias! ``` If you want a typechecker to use a TypeForm variable type for a bare assignment you'll need to explicitly declare the assignment-target as having `TypeForm` type: ``` NULL_TYPE: TypeForm = type(None) ``` ``` NULL_TYPE = type(None) # type: TypeForm # the type comment is significant ``` ``` NULL_TYPE: TypeForm NULL_TYPE = type(None) ``` Values of type TypeForm ----------------------- The type `TypeForm` has values corresponding to exactly those runtime objects that are valid on the right-hand-side of a variable declaration, ``` value: *form* ``` the right-hand-side of a parameter declaration, ``` def some_func(value: *form*): ``` or as the return type of a function: ``` def some_func() -> *form*: ``` Any runtime object that is valid in one of the above locations is a value of `TypeForm`. Incomplete forms like a bare `Optional` or `Union` are not values of `TypeForm`. Example of values include: * type objects like `int`, `str`, `object`, and `FooClass` * generic collections like `List`, `List[int]`, `Dict`, or `Dict[K, V]` * callables like `Callable`, `Callable[[Arg1Type, Arg2Type], ReturnType]`, `Callable[..., ReturnType]` * union forms like `Optional[str]`, `Union[int, str]`, or `NoReturn` * literal forms like `Literal['r', 'rb', 'w', 'wb']` * type variables like `T` or `AnyStr` * annotated types like `Annotated[int, ValueRange(-10, 5)]` * type aliases like `Vector` (where `Vector = list[float]`) * the `Any` form * the `Type` and `Type[C]` forms * the `TypeForm` form Forward References '''''''''''''''''' Runtime objects for forms which contain forward references such as `Union['Triangle', 'Shape']` are normalized at runtime to `Union[ForwardRef('Triangle'), ForwardRef('Shape')]`. These `ForwardRef(...)` objects are also values of TypeForm. Code that manipulates TypeForm values at runtime should be prepared to deal with ForwardRef(...) values. Type Consistency ---------------- Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the `Any` type. It is defined more formally in PEP 483 [^type-consistency]. This section introduces the new rules needed to support type consistency for TypeForm types: [^type-consistency]: https://www.python.org/dev/peps/pep-0483/#summary-of-gradual-typing * `type`, `Type`, and `Type[C]` are consistent with `TypeForm`. `TypeForm` is consistent with `object`. * `Any` is consistent with `TypeForm`. `TypeForm` is consistent with `Any`. Notably `TypeForm` is *not* consistent with `type`, `Type`, or `Type[C]`. Interactions with Type[C] ------------------------- When writing `Type[C]` it is necessary that `C` must be consistent with `type`. Since `TypeForm` is not consistent with `type`, it is not valid to write `Type[C]` when `C` is a TypeVar with a `bound` of `TypeForm`: ``` T = TypeVar('T', bound=TypeForm) def bad_func(form: T, tp: Type[T]): ... # Type[T] is invalid! ``` A group of type variables where some variables are constrained by TypeForms, some are constrained by `Type`s, and others are unconstrained may appear together as arguments to the same type constructor (i.e. `Union[...]`, `Dict[K, V]`, etc): ``` F = TypeVar('F', bound=TypeForm) C = TypeVar('C') V = TypeVar('V') # Union[F, C, V] contains type variables with many kinds of constraints def pick_random(form: F, tp: Type[C], value: V) -> Union[F, C, V]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. Reference Implementation ======================== The following will be true when [mypy#9773](https://github.com/python/mypy/issues/9773) is implemented: The mypy type checker supports `TypeForm` types. A reference implementation of the runtime component is provided in the `typing_extensions` module. In addition an implementation of the `trycast` and `isassignable` functions mentioned in various examples above is being implemented in the [trycast PyPI module](https://github.com/davidfstr/trycast). Rejected Ideas ============== TypeForm[T] as syntax for TypeVar('T', bound=TypeForm) ------------------------------------------------------ The original proposed syntax for TypeForm used `TypeForm[T]` to mean what is now written as `TypeVar('T', bound=TypeForm)`. The original syntax was seeking to spell a typing constraint that was propagated through an unbounded invariant TypeVar. However it was observed that using TypeForm as a bound on a TypeVar is a lot more straightforward to understand. It's also significantly easier to implement in typecheckers, as it leverages existing constraint-propagation infrastructure for TypeVar rather than creating its own. Restricting TypeForm to "universal" forms only ---------------------------------------------- Originally it was considered important to restrict TypeForm such that its values only included forms that could be used as a local variable type, a parameter type, *and* as a parameter type. For example that would rule out `NoReturn` as a value because it can only be used as a return type and not as a local variable type. Functions taking TypeForms as parameters that only wish to accept "universal" forms (that can be used in all 3 of the aforementioned positions) can write code that raises TypeError or a similar exception at runtime when provided a non-universal form. Standardized type inference rules --------------------------------- This PEP intentionally does not specify any rules for implicitly inferring a `TypeForm` type for an expression. No other type-related PEPs prescribe inference rules, and existing type checkers vary in their behavior. So it would be inconsistent to dictate an inference rule for TypeForm. Individual type checkers may define their own inference rules if they wish. Open Issues =========== [Any points that are still being decided/discussed.] References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< On 1/24/21 10:06 PM, David Foster wrote:
I have drafted an initial PEP for TypeForm, a way to spell the type of typing special forms (like Union[int, str], Literal['foo'], etc). Please see the following commentable document, or the copy at the bottom of this email: https://docs.google.com/document/d/18UF8V00EVU1-h-BtiVFhXoJkvfL4rHp4ORaenMQL...
Please leave your comments either on the document itself or by responses to this email on typing-sig. I will integrate feedback periodically.
For further background on why this feature is being introduced and prior discussions, please see:
(1) the originating thread on typing-sig at: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ...
(2) the discussion about TypeForm on the mypy issue tracker at: https://github.com/python/mypy/issues/9773