TypeForm: Type Hint for Typing Special Forms and Regular Types
I have drafted an initial PEP for TypeForm, a way to spell the type of typing special forms (like Union[int, str], Literal['foo'], etc). Please see the following commentable document, or the copy at the bottom of this email: https://docs.google.com/document/d/18UF8V00EVU1-h-BtiVFhXoJkvfL4rHp4ORaenMQL... Please leave your comments either on the document itself or by responses to this email on typing-sig. I will integrate feedback periodically. For further background on why this feature is being introduced and prior discussions, please see: (1) the originating thread on typing-sig at: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ... (2) the discussion about TypeForm on the mypy issue tracker at: https://github.com/python/mypy/issues/9773 -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: TypeForm: Type Hints for Typing Special Forms and Regular Types Author: David Foster <david@dafoster.net> Sponsor: TODO Discussions-To: typing-sig@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 647 (TypeGuard) Created: 21-Dec-2020 Python-Version: 3.10 Post-History: 24-Jan-2021 Abstract ======== PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. [^type-c]: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects Motivation ========== The introduction of `TypeForm` allows new kinds of metaprogramming functions that operate on typing special forms to be type-annotated. For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value: ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... ``` And here is another function that checks whether a value is assignable to a variable of a particular type, and if so returns `True`: ``` def isassignable(value: object, form: TypeForm) -> bool: ... ``` With the introduction of `TypeGuard` in PEP 647[^TypeGuardPep] the above function can be enhanced to return a `TypeGuard` instead of a regular `bool`: [^TypeGuardPep]: https://www.python.org/dev/peps/pep-0647/ ``` def isassignable(value: object, form: TypeForm[T]) -> TypeGuard[T]: ... ``` Without `TypeForm` the next-best type that could be given to the `form` parameter in the above examples would be `object` which would improperly allow values like `1` to be passed in. More importantly, there would be no way to express the relationship between the parameter type of `form` and the function's return type. NB: The preceding example functions implement the kinds of enhanced `isinstance` checks that were omitted in PEP 589[^typeddict-no-isinstance] which are very useful for, among other things, [checking whether a value decoded from JSON conforms to a particular structure] of nested `TypedDict`s, `List`s, `Optional`s, `Literal`s, and other types. [^typeddict-no-isinstance]: https://www.python.org/dev/peps/pep-0589/#using-typeddict-types [checking whether a value decoded from JSON conforms to a particular structure]: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ... Specification ============= A type-form type represents a `type` object or a special typing form such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. A type-form type can be written as either `TypeForm[T]` where `T` is a type variable or as `TypeForm` with no argument. The `T` in `TypeForm[T]` is always an invariant type variable with no bound: * Attempting to use a type variable `T` declared with a `bound=...` or marked with `covariant=True` or `contravariant=True` in a `TypeForm[T]` should be rejected by typecheckers. * Attempting to use a literal type-form `*form*` as an argument to `TypeForm[*form*]`, such as `TypeForm[int]`, should be rejected by typecheckers. (See §"TypeForm[*form*] or Literal[*form*]" in Rejected Ideas for rationale.) The syntax `TypeForm` alone, without a type argument, is equivalent to `TypeForm[_T]` where `_T` is a freshly created type variable. ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... def isassignable(value: object, form: TypeForm) -> bool: ... def is_type(form: TypeForm) -> bool: ... ``` Using TypeForm Types -------------------- Type-form types may be used as function parameter types, return types, and variable types: ``` def is_type(form: TypeForm) -> bool: ... # parameter type ``` ``` S = TypeVar('S') T = TypeVar('T') U = TypeVar('U') def meet_types(s: TypeForm[S], t: TypeForm[T]) \ -> Union[TypeForm[S], TypeForm[T], TypeForm[U]]: ... # return types ``` ``` NULL_TYPE: TypeForm # variable type NULL_TYPE = type(None) ``` ``` NULL_TYPE: TypeForm = type(None) # variable type ``` Note however that a typechecker won't automatically infer a TypeForm type for an unannotated variable assignment that contains a special typing form on the right-hand-side because PEP 484 [^type-alias-syntax] reserves that syntax for defining type aliases: [^type-alias-syntax]: https://www.python.org/dev/peps/pep-0484/#type-aliases ``` NULL_TYPE = type(None) # OOPS; treated as a type alias! ``` If you want a typechecker to infer a TypeForm variable type for a bare assignment, use the walrus operator instead of the assignment operator: ``` NULL_TYPE := type(None) # infers NULL_TYPE as having type TypeForm ``` Or explicitly declare the assignment-target as having `TypeForm` type: ``` NULL_TYPE: TypeForm = type(None) ``` ``` NULL_TYPE = type(None) # type: TypeForm # the type comment is significant ``` ``` NULL_TYPE: TypeForm NULL_TYPE = type(None) ``` Values of type TypeForm ----------------------- A particular literal value is said to *inhabit* a type if it is described by that type and can be assigned to a variable of that type. The type `TypeForm` is inhabited by exactly those runtime objects that are valid on the right-hand-side of a variable declaration, ``` value: *form* ``` the right-hand-side of a parameter declaration, ``` def some_func(value: *form*): ``` *and* as the return type of a function: ``` def some_func() -> *form*: ``` Any runtime object that is not valid in one of the above locations is not an inhabitant of `TypeForm`. For example the special forms `Final`, `Final[int]`, `NoReturn`, and `ClassVar[int]` are not inhabitants. Incomplete forms like a bare `Optional` or `Union` are also not inhabitants. Example of inhabitants include: * type objects like `int`, `str`, `object`, and `FooClass` * generic collections like `List`, `List[int]`, `Dict`, or `Dict[K, V]` * callables like `Callable`, `Callable[[Arg1Type, Arg2Type], ReturnType]`, `Callable[..., ReturnType]` * union forms like `Optional[str]` or `Union[int, str]` * literal forms like `Literal['r', 'rb', 'w', 'wb']` * type variables like `T` or `AnyStr` * annotated types like `Annotated[int, ValueRange(-10, 5)]` * type aliases like `Vector` (where `Vector = list[float]`) * the `Any` form * the `Type` and `Type[C]` forms * the `TypeForm` and `TypeForm[T]` forms A few peculiar cases permitted by the above include: ``` STR_FORM: TypeForm = str STR_OR_SUBCLASS_FORM: TypeForm = Type[str] SOME_FORM: TypeForm = TypeForm ``` Forward References '''''''''''''''''' Runtime objects for forms which contain forward references such as `Union['Triangle', 'Shape']` are normalized at runtime to `Union[ForwardRef('Triangle'), ForwardRef('Shape')]`. The latter are inhabitants of `TypeForm`. Additionally the "top-level" form for a particular variable type can itself be a forward reference such as in: ``` class Node: value: object next: 'Node' # 'Node' is normalized to ForwardRef('Node') at runtime ``` Code that manipulates TypeForm values at runtime should be prepared to deal with ForwardRef(...) subcomponents. Type Consistency ---------------- Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the `Any` type. It is defined more formally in PEP 483 [^type-consistency]. This section introduces the new rules needed to support type consistency for TypeForm types: [^type-consistency]: https://www.python.org/dev/peps/pep-0483/#summary-of-gradual-typing * `Type[C]` is consistent with `TypeForm`. `TypeForm` is consistent with `object`. * `TypeForm[S]` is consistent with `TypeForm[T]` iff `S` and `T` are the same type variable. * `Any` is consistent with `TypeForm`. `TypeForm` is consistent with `Any`. Overloading ----------- `TypeForm` can be used in overloads. For example: ``` T = TypeVar('T') F = TypeVar('F') @overload def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... @overload def trycast(form: TypeForm[T], value: object, failure: F) -> Union[T, F]: ... ``` Interactions with Type[C] ------------------------- A `TypeForm` and a `Type` cannot simultaneously constrain the same type variable within a generic function definition: ``` def bad_func(form: TypeForm[T], tp: Type[T]): ... # invalid ``` A group of type variables where some variables are constrained by `TypeForm`s, some are constrained by `Type`s, and others are unconstrained may appear together as arguments to the same type constructor (i.e. `Union[...]`, `Dict[K, V]`, etc): ``` F = TypeVar('F') C = TypeVar('C') V = TypeVar('V') def pick_random(form: TypeForm[F], tp: Type[C], value: V) -> Union[F, C, V]: ... ``` Interactions with cast ---------------------- It is possible to cast a value to a `TypeForm` or `TypeForm[T]` type: ``` OPTIONAL_STR_OBJECT: object = Optional[str] OPTIONAL_STR = cast(TypeForm, OPTIONAL_STR_OBJECT) ``` Type Inference -------------- A runtime object corresponding to a special form should be inferred as having type `TypeForm` unless it can instead be inferred to have type `Type[C]` for some `C`: ``` T = TypeVar('T') def identity(value: T) -> T: return value A_TYPE = identity(str) # inferred type is: Type[str] A_FORM = identity(Optional[str]) # inferred type is: TypeForm ``` Backwards Compatibility ======================= Previously the type `object` would be inferred for runtime objects corresponding to a special form like `Optional[str]` rather than `TypeForm`. But since `TypeForm` is consistent with `object` no unexpected typechecking failures will be introduced by this change. Ongoing Maintenance =================== As new kinds of typing special forms are introduced by upcoming PEPs (usually to the `typing` module), it will be necessary for typecheckers to define whether or not the new forms should be considered inhabitants of `TypeForm` or not, based on the general rules in §"Values of type TypeForm" above. Therefore it is recommended that any new PEP introducing a new typing special form should define whether it inhabits `TypeForm` or not. Reference Implementation ======================== The following will be true when [mypy#9773](https://github.com/python/mypy/issues/9773) is implemented: The mypy type checker supports `TypeForm` types. A reference implementation of the runtime component is provided in the `typing_extensions` module. In addition an implementation of the `trycast` and `isassignable` functions mentioned in various examples above is being implemented in the [trycast PyPI module](https://github.com/davidfstr/trycast). Rejected Ideas ============== TypeForm[*form*] or Literal[*form*] ----------------------------------- This PEP only defines syntax for a bare `TypeForm` and a `TypeForm[T]` parameterized by a type variable. In particular it does not define syntax like that allows referring to a specific known type-form value via a syntax like `TypeForm[*form*]`: ``` NULL_TYPE := type(None) # has type TypeForm[_T] rather than TypeForm[type(None)] UNION_STR_TYPE := Union[Str] # has type TypeForm[_T] rather than TypeForm[Union[str]] ``` In addition similar syntax like `Literal[*form*]` also does not work. Neither of those syntaxes seem to be especially useful; type forms are not typically used as sentinel values. It is possible that a future PEP may revisit such syntaxes. Open Issues =========== [Any points that are still being decided/discussed.] References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
First impressions: I noticed that none of your examples actually show how you would write the body of a function like isassignable(). Are you assuming that those implementations just introspect the runtime representation? I also saw this: If you want a typechecker to infer a TypeForm variable type for a bare
assignment, use the walrus operator instead of the assignment operator:
```
NULL_TYPE := type(None) # infers NULL_TYPE as having type TypeForm
```
The walrus operator syntactically cannot be used at the top level, so this clever hack won't work. All in all I feel pretty lukewarm about this proposal (but then again, static type checking satisfies all my needs). --Guido On Sun, Jan 24, 2021 at 10:07 PM David Foster <davidfstr@gmail.com> wrote:
I have drafted an initial PEP for TypeForm, a way to spell the type of typing special forms (like Union[int, str], Literal['foo'], etc). Please see the following commentable document, or the copy at the bottom of this email:
https://docs.google.com/document/d/18UF8V00EVU1-h-BtiVFhXoJkvfL4rHp4ORaenMQL...
Please leave your comments either on the document itself or by responses to this email on typing-sig. I will integrate feedback periodically.
For further background on why this feature is being introduced and prior discussions, please see:
(1) the originating thread on typing-sig at:
https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ...
(2) the discussion about TypeForm on the mypy issue tracker at: https://github.com/python/mypy/issues/9773
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: TypeForm: Type Hints for Typing Special Forms and Regular Types Author: David Foster <david@dafoster.net> Sponsor: TODO Discussions-To: typing-sig@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 647 (TypeGuard) Created: 21-Dec-2020 Python-Version: 3.10 Post-History: 24-Jan-2021
Abstract ========
PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`.
[^type-c]: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects
Motivation ==========
The introduction of `TypeForm` allows new kinds of metaprogramming functions that operate on typing special forms to be type-annotated.
For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value:
``` T = TypeVar('T')
def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... ```
And here is another function that checks whether a value is assignable to a variable of a particular type, and if so returns `True`:
``` def isassignable(value: object, form: TypeForm) -> bool: ... ```
With the introduction of `TypeGuard` in PEP 647[^TypeGuardPep] the above function can be enhanced to return a `TypeGuard` instead of a regular `bool`:
[^TypeGuardPep]: https://www.python.org/dev/peps/pep-0647/
``` def isassignable(value: object, form: TypeForm[T]) -> TypeGuard[T]: ... ```
Without `TypeForm` the next-best type that could be given to the `form` parameter in the above examples would be `object` which would improperly allow values like `1` to be passed in. More importantly, there would be no way to express the relationship between the parameter type of `form` and the function's return type.
NB: The preceding example functions implement the kinds of enhanced `isinstance` checks that were omitted in PEP 589[^typeddict-no-isinstance] which are very useful for, among other things, [checking whether a value decoded from JSON conforms to a particular structure] of nested `TypedDict`s, `List`s, `Optional`s, `Literal`s, and other types.
[^typeddict-no-isinstance]: https://www.python.org/dev/peps/pep-0589/#using-typeddict-types
[checking whether a value decoded from JSON conforms to a particular structure]:
https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ...
Specification =============
A type-form type represents a `type` object or a special typing form such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. A type-form type can be written as either `TypeForm[T]` where `T` is a type variable or as `TypeForm` with no argument.
The `T` in `TypeForm[T]` is always an invariant type variable with no bound:
* Attempting to use a type variable `T` declared with a `bound=...` or marked with `covariant=True` or `contravariant=True` in a `TypeForm[T]` should be rejected by typecheckers.
* Attempting to use a literal type-form `*form*` as an argument to `TypeForm[*form*]`, such as `TypeForm[int]`, should be rejected by typecheckers. (See §"TypeForm[*form*] or Literal[*form*]" in Rejected Ideas for rationale.)
The syntax `TypeForm` alone, without a type argument, is equivalent to `TypeForm[_T]` where `_T` is a freshly created type variable.
``` T = TypeVar('T')
def trycast(form: TypeForm[T], value: object) -> Optional[T]: ...
def isassignable(value: object, form: TypeForm) -> bool: ...
def is_type(form: TypeForm) -> bool: ... ```
Using TypeForm Types --------------------
Type-form types may be used as function parameter types, return types, and variable types:
``` def is_type(form: TypeForm) -> bool: ... # parameter type ```
``` S = TypeVar('S') T = TypeVar('T') U = TypeVar('U')
def meet_types(s: TypeForm[S], t: TypeForm[T]) \ -> Union[TypeForm[S], TypeForm[T], TypeForm[U]]: ... # return types ```
``` NULL_TYPE: TypeForm # variable type NULL_TYPE = type(None) ```
``` NULL_TYPE: TypeForm = type(None) # variable type ```
Note however that a typechecker won't automatically infer a TypeForm type for an unannotated variable assignment that contains a special typing form on the right-hand-side because PEP 484 [^type-alias-syntax] reserves that syntax for defining type aliases:
[^type-alias-syntax]: https://www.python.org/dev/peps/pep-0484/#type-aliases
``` NULL_TYPE = type(None) # OOPS; treated as a type alias! ```
If you want a typechecker to infer a TypeForm variable type for a bare assignment, use the walrus operator instead of the assignment operator:
``` NULL_TYPE := type(None) # infers NULL_TYPE as having type TypeForm ```
Or explicitly declare the assignment-target as having `TypeForm` type:
``` NULL_TYPE: TypeForm = type(None) ```
``` NULL_TYPE = type(None) # type: TypeForm # the type comment is significant ```
``` NULL_TYPE: TypeForm NULL_TYPE = type(None) ```
Values of type TypeForm -----------------------
A particular literal value is said to *inhabit* a type if it is described by that type and can be assigned to a variable of that type.
The type `TypeForm` is inhabited by exactly those runtime objects that are valid on the right-hand-side of a variable declaration,
``` value: *form* ```
the right-hand-side of a parameter declaration,
``` def some_func(value: *form*): ```
*and* as the return type of a function:
``` def some_func() -> *form*: ```
Any runtime object that is not valid in one of the above locations is not an inhabitant of `TypeForm`. For example the special forms `Final`, `Final[int]`, `NoReturn`, and `ClassVar[int]` are not inhabitants. Incomplete forms like a bare `Optional` or `Union` are also not inhabitants.
Example of inhabitants include:
* type objects like `int`, `str`, `object`, and `FooClass` * generic collections like `List`, `List[int]`, `Dict`, or `Dict[K, V]` * callables like `Callable`, `Callable[[Arg1Type, Arg2Type], ReturnType]`, `Callable[..., ReturnType]` * union forms like `Optional[str]` or `Union[int, str]` * literal forms like `Literal['r', 'rb', 'w', 'wb']` * type variables like `T` or `AnyStr` * annotated types like `Annotated[int, ValueRange(-10, 5)]` * type aliases like `Vector` (where `Vector = list[float]`) * the `Any` form * the `Type` and `Type[C]` forms * the `TypeForm` and `TypeForm[T]` forms
A few peculiar cases permitted by the above include:
``` STR_FORM: TypeForm = str STR_OR_SUBCLASS_FORM: TypeForm = Type[str] SOME_FORM: TypeForm = TypeForm ```
Forward References ''''''''''''''''''
Runtime objects for forms which contain forward references such as `Union['Triangle', 'Shape']` are normalized at runtime to `Union[ForwardRef('Triangle'), ForwardRef('Shape')]`. The latter are inhabitants of `TypeForm`.
Additionally the "top-level" form for a particular variable type can itself be a forward reference such as in:
``` class Node: value: object next: 'Node' # 'Node' is normalized to ForwardRef('Node') at runtime ```
Code that manipulates TypeForm values at runtime should be prepared to deal with ForwardRef(...) subcomponents.
Type Consistency ----------------
Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the `Any` type. It is defined more formally in PEP 483 [^type-consistency]. This section introduces the new rules needed to support type consistency for TypeForm types:
[^type-consistency]: https://www.python.org/dev/peps/pep-0483/#summary-of-gradual-typing
* `Type[C]` is consistent with `TypeForm`. `TypeForm` is consistent with `object`.
* `TypeForm[S]` is consistent with `TypeForm[T]` iff `S` and `T` are the same type variable.
* `Any` is consistent with `TypeForm`. `TypeForm` is consistent with `Any`.
Overloading -----------
`TypeForm` can be used in overloads. For example:
``` T = TypeVar('T') F = TypeVar('F')
@overload def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... @overload def trycast(form: TypeForm[T], value: object, failure: F) -> Union[T, F]: ... ```
Interactions with Type[C] -------------------------
A `TypeForm` and a `Type` cannot simultaneously constrain the same type variable within a generic function definition:
``` def bad_func(form: TypeForm[T], tp: Type[T]): ... # invalid ```
A group of type variables where some variables are constrained by `TypeForm`s, some are constrained by `Type`s, and others are unconstrained may appear together as arguments to the same type constructor (i.e. `Union[...]`, `Dict[K, V]`, etc):
``` F = TypeVar('F') C = TypeVar('C') V = TypeVar('V')
def pick_random(form: TypeForm[F], tp: Type[C], value: V) -> Union[F, C, V]: ... ```
Interactions with cast ----------------------
It is possible to cast a value to a `TypeForm` or `TypeForm[T]` type:
``` OPTIONAL_STR_OBJECT: object = Optional[str] OPTIONAL_STR = cast(TypeForm, OPTIONAL_STR_OBJECT) ```
Type Inference --------------
A runtime object corresponding to a special form should be inferred as having type `TypeForm` unless it can instead be inferred to have type `Type[C]` for some `C`:
``` T = TypeVar('T')
def identity(value: T) -> T: return value
A_TYPE = identity(str) # inferred type is: Type[str] A_FORM = identity(Optional[str]) # inferred type is: TypeForm ```
Backwards Compatibility =======================
Previously the type `object` would be inferred for runtime objects corresponding to a special form like `Optional[str]` rather than `TypeForm`. But since `TypeForm` is consistent with `object` no unexpected typechecking failures will be introduced by this change.
Ongoing Maintenance ===================
As new kinds of typing special forms are introduced by upcoming PEPs (usually to the `typing` module), it will be necessary for typecheckers to define whether or not the new forms should be considered inhabitants of `TypeForm` or not, based on the general rules in §"Values of type TypeForm" above. Therefore it is recommended that any new PEP introducing a new typing special form should define whether it inhabits `TypeForm` or not.
Reference Implementation ========================
The following will be true when [mypy#9773](https://github.com/python/mypy/issues/9773) is implemented:
The mypy type checker supports `TypeForm` types. A reference implementation of the runtime component is provided in the `typing_extensions` module.
In addition an implementation of the `trycast` and `isassignable` functions mentioned in various examples above is being implemented in the [trycast PyPI module](https://github.com/davidfstr/trycast).
Rejected Ideas ==============
TypeForm[*form*] or Literal[*form*] -----------------------------------
This PEP only defines syntax for a bare `TypeForm` and a `TypeForm[T]` parameterized by a type variable. In particular it does not define syntax like that allows referring to a specific known type-form value via a syntax like `TypeForm[*form*]`:
``` NULL_TYPE := type(None) # has type TypeForm[_T] rather than TypeForm[type(None)] UNION_STR_TYPE := Union[Str] # has type TypeForm[_T] rather than TypeForm[Union[str]] ```
In addition similar syntax like `Literal[*form*]` also does not work.
Neither of those syntaxes seem to be especially useful; type forms are not typically used as sentinel values. It is possible that a future PEP may revisit such syntaxes.
Open Issues ===========
[Any points that are still being decided/discussed.]
References ==========
[A collection of URLs used as references through the PEP.]
Copyright =========
This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive.
.. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Hello. I'm interested in this since my library, cattrs, supports this use case and I have users raising issues about this missing functionality. (Case in point, https://github.com/Tinche/cattrs/issues/111) On Mon, Jan 25, 2021 at 10:34 PM Guido van Rossum <guido@python.org> wrote:
First impressions:
I noticed that none of your examples actually show how you would write the body of a function like isassignable(). Are you assuming that those implementations just introspect the runtime representation?
In cattrs it's complicated, using the type passed in and a system of direct matching, singledispatch and predicates to choose how to handle the type. cattrs needs to support arbitrary type forms though, like Tuple[Optional[DateTime], ObjectId] (where (pendulum.)DateTime and (bson.)ObjectId are non-stdlib classes), hence the complexity.
Could you construct some small(ish) examples and paste them here? I am still having a hard time imagining how you implement this at runtime. On Mon, Jan 25, 2021 at 2:54 PM Tin Tvrtković <tinchester@gmail.com> wrote:
Hello. I'm interested in this since my library, cattrs, supports this use case and I have users raising issues about this missing functionality. (Case in point, https://github.com/Tinche/cattrs/issues/111)
On Mon, Jan 25, 2021 at 10:34 PM Guido van Rossum <guido@python.org> wrote:
First impressions:
I noticed that none of your examples actually show how you would write the body of a function like isassignable(). Are you assuming that those implementations just introspect the runtime representation?
In cattrs it's complicated, using the type passed in and a system of direct matching, singledispatch and predicates to choose how to handle the type. cattrs needs to support arbitrary type forms though, like Tuple[Optional[DateTime], ObjectId] (where (pendulum.)DateTime and (bson.)ObjectId are non-stdlib classes), hence the complexity.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Ok, sure. Are you looking for usage examples or me running through how the code works internally? On Tue, Jan 26, 2021 at 12:08 AM Guido van Rossum <guido@python.org> wrote:
Could you construct some small(ish) examples and paste them here? I am still having a hard time imagining how you implement this at runtime.
I really want to understand what you're doing to compute the results at runtime -- the PEP is already full of example signatures. On Mon, Jan 25, 2021 at 4:10 PM Tin Tvrtković <tinchester@gmail.com> wrote:
Ok, sure. Are you looking for usage examples or me running through how the code works internally?
On Tue, Jan 26, 2021 at 12:08 AM Guido van Rossum <guido@python.org> wrote:
Could you construct some small(ish) examples and paste them here? I am still having a hard time imagining how you implement this at runtime.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Alright. Let's assume you have a 3.9 virtualenv with cattrs installed.
from cattr import structure structure(["1", 1.0, True], tuple[int, int, int]) (1, 1, 1)
1. We run the value `tuple[int, int, int]` through the dispatch mechanism. It matches on this function: https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42..., which ends up invoking this handler: https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42... . 2. This handler fishes out the tuple type arguments, and it basically returns `tuple(cattr.structure(obj[0], int), cattr.structure(obj[1], int), cattr.structure(obj[2], int))`. 3. All three of these structure calls end up matching at https://github.com/Tinche/cattrs/blob/master/src/cattr/converters.py#L131, which points to https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42..., which basically does int(x). The top level call basically evaluates to `tuple(int(x[0]), int(x[1]), int(x[2]))`. This same mechanism supports other primitives, lists, dicts, sets, frozensets, optionals, enums, attrs classes, unions of attrs classes (with some restrictions) out of the box. I feel if you'd like a more thorough walkthrough a video call would be better. On Tue, Jan 26, 2021 at 1:52 AM Guido van Rossum <guido@python.org> wrote:
I really want to understand what you're doing to compute the results at runtime -- the PEP is already full of example signatures.
On Mon, Jan 25, 2021 at 4:10 PM Tin Tvrtković <tinchester@gmail.com> wrote:
Ok, sure. Are you looking for usage examples or me running through how the code works internally?
On Tue, Jan 26, 2021 at 12:08 AM Guido van Rossum <guido@python.org> wrote:
Could you construct some small(ish) examples and paste them here? I am still having a hard time imagining how you implement this at runtime.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
No video please! :-) I see what your code does, in introspects tuple[int, int, int] (the dispatch mechanism isn't super important). But what if the type is tuple[T, T, T]? On Mon, Jan 25, 2021 at 5:09 PM Tin Tvrtković <tinchester@gmail.com> wrote:
Alright.
Let's assume you have a 3.9 virtualenv with cattrs installed.
from cattr import structure structure(["1", 1.0, True], tuple[int, int, int]) (1, 1, 1)
1. We run the value `tuple[int, int, int]` through the dispatch mechanism. It matches on this function: https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42..., which ends up invoking this handler: https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42... .
2. This handler fishes out the tuple type arguments, and it basically returns `tuple(cattr.structure(obj[0], int), cattr.structure(obj[1], int), cattr.structure(obj[2], int))`.
3. All three of these structure calls end up matching at https://github.com/Tinche/cattrs/blob/master/src/cattr/converters.py#L131, which points to https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42..., which basically does int(x).
The top level call basically evaluates to `tuple(int(x[0]), int(x[1]), int(x[2]))`. This same mechanism supports other primitives, lists, dicts, sets, frozensets, optionals, enums, attrs classes, unions of attrs classes (with some restrictions) out of the box.
I feel if you'd like a more thorough walkthrough a video call would be better.
On Tue, Jan 26, 2021 at 1:52 AM Guido van Rossum <guido@python.org> wrote:
I really want to understand what you're doing to compute the results at runtime -- the PEP is already full of example signatures.
On Mon, Jan 25, 2021 at 4:10 PM Tin Tvrtković <tinchester@gmail.com> wrote:
Ok, sure. Are you looking for usage examples or me running through how the code works internally?
On Tue, Jan 26, 2021 at 12:08 AM Guido van Rossum <guido@python.org> wrote:
Could you construct some small(ish) examples and paste them here? I am still having a hard time imagining how you implement this at runtime.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
As in TypeVars? It'd blow up. We need concrete types on this layer. On Tue, Jan 26, 2021 at 2:23 AM Guido van Rossum <guido@python.org> wrote:
No video please! :-)
I see what your code does, in introspects tuple[int, int, int] (the dispatch mechanism isn't super important).
But what if the type is tuple[T, T, T]?
On Mon, Jan 25, 2021 at 5:09 PM Tin Tvrtković <tinchester@gmail.com> wrote:
Alright.
Let's assume you have a 3.9 virtualenv with cattrs installed.
from cattr import structure structure(["1", 1.0, True], tuple[int, int, int]) (1, 1, 1)
1. We run the value `tuple[int, int, int]` through the dispatch mechanism. It matches on this function: https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42..., which ends up invoking this handler: https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42... .
2. This handler fishes out the tuple type arguments, and it basically returns `tuple(cattr.structure(obj[0], int), cattr.structure(obj[1], int), cattr.structure(obj[2], int))`.
3. All three of these structure calls end up matching at https://github.com/Tinche/cattrs/blob/master/src/cattr/converters.py#L131, which points to https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42..., which basically does int(x).
The top level call basically evaluates to `tuple(int(x[0]), int(x[1]), int(x[2]))`. This same mechanism supports other primitives, lists, dicts, sets, frozensets, optionals, enums, attrs classes, unions of attrs classes (with some restrictions) out of the box.
I feel if you'd like a more thorough walkthrough a video call would be better.
On Tue, Jan 26, 2021 at 1:52 AM Guido van Rossum <guido@python.org> wrote:
I really want to understand what you're doing to compute the results at runtime -- the PEP is already full of example signatures.
On Mon, Jan 25, 2021 at 4:10 PM Tin Tvrtković <tinchester@gmail.com> wrote:
Ok, sure. Are you looking for usage examples or me running through how the code works internally?
On Tue, Jan 26, 2021 at 12:08 AM Guido van Rossum <guido@python.org> wrote:
Could you construct some small(ish) examples and paste them here? I am still having a hard time imagining how you implement this at runtime.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Okay. Does the PEP call that out in it’s specification? On Mon, Jan 25, 2021 at 17:33 Tin Tvrtković <tinchester@gmail.com> wrote:
As in TypeVars? It'd blow up. We need concrete types on this layer.
On Tue, Jan 26, 2021 at 2:23 AM Guido van Rossum <guido@python.org> wrote:
No video please! :-)
I see what your code does, in introspects tuple[int, int, int] (the dispatch mechanism isn't super important).
But what if the type is tuple[T, T, T]?
On Mon, Jan 25, 2021 at 5:09 PM Tin Tvrtković <tinchester@gmail.com> wrote:
Alright.
Let's assume you have a 3.9 virtualenv with cattrs installed.
from cattr import structure structure(["1", 1.0, True], tuple[int, int, int]) (1, 1, 1)
1. We run the value `tuple[int, int, int]` through the dispatch mechanism. It matches on this function: https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42..., which ends up invoking this handler: https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42... .
2. This handler fishes out the tuple type arguments, and it basically returns `tuple(cattr.structure(obj[0], int), cattr.structure(obj[1], int), cattr.structure(obj[2], int))`.
3. All three of these structure calls end up matching at https://github.com/Tinche/cattrs/blob/master/src/cattr/converters.py#L131, which points to https://github.com/Tinche/cattrs/blob/e54fa1713b39fa3c944415bbe61aabd7e8bb42..., which basically does int(x).
The top level call basically evaluates to `tuple(int(x[0]), int(x[1]), int(x[2]))`. This same mechanism supports other primitives, lists, dicts, sets, frozensets, optionals, enums, attrs classes, unions of attrs classes (with some restrictions) out of the box.
I feel if you'd like a more thorough walkthrough a video call would be better.
On Tue, Jan 26, 2021 at 1:52 AM Guido van Rossum <guido@python.org> wrote:
I really want to understand what you're doing to compute the results at runtime -- the PEP is already full of example signatures.
On Mon, Jan 25, 2021 at 4:10 PM Tin Tvrtković <tinchester@gmail.com> wrote:
Ok, sure. Are you looking for usage examples or me running through how the code works internally?
On Tue, Jan 26, 2021 at 12:08 AM Guido van Rossum <guido@python.org> wrote:
Could you construct some small(ish) examples and paste them here? I am still having a hard time imagining how you implement this at runtime.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
-- --Guido (mobile)
Like Guido, I’m lukewarm on this proposal. It provides some value, but that value seems relatively low and may not be enough to justify the feature. Here are some additional pieces of feedback. I don’t understand why TypeForm is a generic type that accepts an optional type argument. I don’t see any reason why `TypeForm` needs to be parameterized. It’s not clear what meaning the type parameter has in this context. Therefore it’s also not clear whether it should be invariant or covariant. It leads to odd questions like “is `TypeForm[int | str]` the same as `TypeForm[int] | TypeForm[str]`”? The examples in the PEP don’t motivate the need for a generic type either. These examples could all use a bound type variable instead. ```python T = TypeVar("T", bound=TypeForm) def isassignable(value: object, form: T) -> TypeGuard[T]: … def trycast(form: T, value: object) -> Optional[T]: ... ``` I therefore strongly recommend dropping the type parameter from TypeForm. If you do so, many aspects of this design become simpler. You can eliminate most of the explanations and special rules later in the document (including the sections titled “Type Consistency”, “Overloading”, and “Interactions with Type[C]”). Nit: In the example under “Using TypeForm Types”, the `meet_types` function signature is invalid because it uses a lone instance of TypeVar `U`. Perhaps you meant to include another input parameter `u: TypeForm[U]`? As others have pointed out, an assignment expression (walrus operator) cannot be used as a statement. I also think it’s a really bad idea to change the semantics of assignment based on whether `=` or `:=` is used. The semantics should be consistent. In the first sentence under “Values of type TypeForm”, you say “A particular literal value…”. I was confused by this because you used the term “literal”, but I think you just mean “A value…” regardless of whether it’s a literal expression. You define a new term “inhabit”. AFAIK, this term isn’t used in any other type-related PEPs, so I find it to be confusing and out of place. I’d like to suggest instead using the term “assignable” or “compatible with”. You also use the term “inhabitant” a few sentences later, but it has a slightly different connotation here. You mention that `NoReturn` is not a valid TypeForm. Why would that be excluded? It is a valid parameter and return type annotation, so it seems odd to exclude it. I don’t agree with the proposed inference rules in the “Type Inference” section. My recommendation is to delete this section entirely. No other type-related PEPs prescribe inference rules, and existing type checkers vary in their behavior. It would be inconsistent to dictate an inference rule in this particular case. In any event, I don’t think a type checker should ever infer type `TypeForm`, especially in the example you provided. Today, Pyright infers the type of `A_FORM` as `Type[str] | None` which is a much better (more precise) answer than `TypeForm`. -- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.
Thanks for taking a close look Eric! On 1/26/21 1:23 PM, Eric Traut wrote:
Like Guido, I’m lukewarm on this proposal. It provides some value, but that value seems relatively low and may not be enough to justify the feature.
Here are some additional pieces of feedback.
I don’t understand why TypeForm is a generic type that accepts an optional type argument. I don’t see any reason why `TypeForm` needs to be parameterized. It’s not clear what meaning the type parameter has in this context. Therefore it’s also not clear whether it should be invariant or covariant. It leads to odd questions like “is `TypeForm[int | str]` the same as `TypeForm[int] | TypeForm[str]`”? > The examples in the PEP don’t motivate the need for a generic type either. These examples could all use a bound type variable instead.
```python T = TypeVar("T", bound=TypeForm) def isassignable(value: object, form: T) -> TypeGuard[T]: … def trycast(form: T, value: object) -> Optional[T]: ... ```
Interesting. I was going for the idea that TypeForm[T] applied a constraint to a T that appears in multiple places, where T can only be a TypeVar. (So a construction like `TypeForm[int]` would be rejected.) I actually like `TypeVar("T", bound=TypeForm)` a lot better as a way to propagate constraints to other parts of the signature. So I'll drop the type parameter from TypeForm.
I therefore strongly recommend dropping the type parameter from TypeForm. If you do so, many aspects of this design become simpler. You can eliminate most of the explanations and special rules later in the document (including the sections titled “Type Consistency”, “Overloading”, and “Interactions with Type[C]”).
Nit: In the example under “Using TypeForm Types”, the `meet_types` function signature is invalid because it uses a lone instance of TypeVar `U`. Perhaps you meant to include another input parameter `u: TypeForm[U]`?
Ah. I didn't know a lone TypeVar was improper. So I guess the example then becomes: S = TypeVar('S', bound=TypeForm) T = TypeVar('T', bound=TypeForm) def meet_types(s: S, t: T) \ -> Union[S, T, TypeForm]: ...
As others have pointed out, an assignment expression (walrus operator) cannot be used as a statement. I also think it’s a really bad idea to change the semantics of assignment based on whether `=` or `:=` is used. The semantics should be consistent.
In the first sentence under “Values of type TypeForm”, you say “A particular literal value…”. I was confused by this because you used the term “literal”, but I think you just mean “A value…” regardless of whether it’s a literal expression.
Right, was using the English term "literal". Will rephrase.
You define a new term “inhabit”. AFAIK, this term isn’t used in any other type-related PEPs, so I find it to be confusing and out of place. I’d like to suggest instead using the term “assignable” or “compatible with”. You also use the term “inhabitant” a few sentences later, but it has a slightly different connotation here.
Ah I was using the term "inhabit" from type theory literature. I believe it is equivalent to "assignable" or "compatible with", so I'll go with those instead.
You mention that `NoReturn` is not a valid TypeForm. Why would that be excluded? It is a valid parameter and return type annotation, so it seems odd to exclude it.
My understanding is that NoReturn can only be used as a return type and not as either a parameter type or a local variable type. At least PEP 484 gives no such examples in those positions, and it doesn't make logical sense to me to have a parameter of type NoReturn. I was thinking it might be best to restrict TypeForm to only refer to forms that can legally be used in *all* "typelike" locations. Such a restriction would rule out NoReturn, because you can't use that as a local variable type; Final[...], because that can only be used as a variable type and not as a parameter or return type; and some other forms too.
I don’t agree with the proposed inference rules in the “Type Inference” section. My recommendation is to delete this section entirely. No other type-related PEPs prescribe inference rules, and existing type checkers vary in their behavior. It would be inconsistent to dictate an inference rule in this particular case. In any event, I don’t think a type checker should ever infer type `TypeForm`, especially in the example you provided.
I'm okay with not declaring rules to infer a TypeForm. Users will then be required to explicitly annotate where usage is desired, and I think that's okay.
Today, Pyright infers the type of `A_FORM` as `Type[str] | None` which is a much better (more precise) answer than `TypeForm`. > -- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On 1/25/21 1:33 PM, Guido van Rossum wrote:
First impressions:
I noticed that none of your examples actually show how you would write the body of a function like isassignable(). Are you assuming that those implementations just introspect the runtime representation?
Yes. Here's a simpler function than trycast: from typing import cast, get_args, get_origin def ununion(form: TypeForm) -> List[TypeForm]: if get_origin(form) is Union: return cast(List[TypeForm], get_args(form)) else: return [form] For trycast, I have an implementation here: https://github.com/davidfstr/trycast/blob/f59cf5ad18402cc3c156de741670ccac6a...
I also saw this:
If you want a typechecker to infer a TypeForm variable type for a bare assignment, use the walrus operator instead of the assignment operator:
``` NULL_TYPE := type(None) # infers NULL_TYPE as having type TypeForm ```
The walrus operator syntactically cannot be used at the top level, so this clever hack won't work.
Ah ha. I'll take out related examples then. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
The second draft of the TypeForm PEP is ready for another look! The first draft specified certain syntax and semantics that were more complex than necessary. And that complexity could have outweighed the benefit of providing the ability to write powerful type introspection functions for advanced users. My hope is that this second draft will be considered significantly simpler and cheaper to implement, making the value added by this PEP more competitive. Major changes in this draft: * Tightened up the Motivation section. * Retired `TypeForm[T]` syntax in favor of `TypeVar('T', bound=TypeForm)`, simplifying and eliminating many sections. * Add examples of complete functions that use TypeForm. * Removed the `var := <form>` trick, since it isn't legal syntax. * Removed the "Type Inference" section entirely. This also eliminated the only backward-compatibility concern. * Removed ban on "non-universal typing forms", simplifying value recognition rules and eliminating the "Ongoing Maintenance" section. * Reworded "inhabit" to be either "value of" or "is consistent with" as appropriate. Again, please leave your comments either as responses to this email or as inline comments in the original document. A copy is at the bottom of this email and at the link: * https://docs.google.com/document/d/18UF8V00EVU1-h-BtiVFhXoJkvfL4rHp4ORaenMQL... -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: TypeForm: Type Hints for Typing Special Forms and Regular Types Author: David Foster <david@dafoster.net> Sponsor: TODO Discussions-To: typing-sig@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 647 (TypeGuard) Created: 21-Dec-2020 Python-Version: 3.10 Post-History: 24-Jan-2021, 28-Jan-2021 Abstract ======== PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. [^type-c]: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects Motivation ========== The introduction of `TypeForm` allows new kinds of metaprogramming functions that operate on typing special forms to be type-annotated. For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value: ``` T = TypeVar('T', bound=TypeForm) def trycast(form: T, value: object) -> Optional[T]: ... ``` It's very powerful to be able to define a function whose return type can be influenced by a `form` value passed at runtime. Here is another function that checks whether a value is assignable to a variable of a particular type, and if so returns True (as a special TypeGuard bool [^TypeGuardPep]): [^TypeGuardPep]: https://www.python.org/dev/peps/pep-0647/ ``` T = TypeVar('T', bound=TypeForm) def isassignable(value: object, form: T) -> TypeGuard[T]: ... ``` When combining TypeForm and TypeGuard together in this way, typecheckers can understand the relationship between the type provided to parameter `form` and the function's return type, and narrow the return type appropriately depending on what form is passed in: ``` request_json = ... # type: object assert isassignable(request_json, Shape) # request_json is narrowed to type Shape here! ``` NB: The preceding example functions implement the kinds of enhanced `isinstance` checks that were omitted in PEP 589[^typeddict-no-isinstance] which are very useful for, among other things, [checking whether a value decoded from JSON conforms to a particular structure] of nested `TypedDict`s, `List`s, `Optional`s, `Literal`s, and other types. [^typeddict-no-isinstance]: https://www.python.org/dev/peps/pep-0589/#using-typeddict-types [checking whether a value decoded from JSON conforms to a particular structure]: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ... Specification ============= A type-form type represents a `type` object or a special typing form such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. A type-form type is written as `TypeForm`. `TypeForm` can be used as a standalone type or as a bound for a `TypeVar`: ``` T = TypeVar('T', bound=TypeForm) def trycast(form: T, value: object) -> Optional[T]: ... def isassignable(value: object, form: T) -> TypeGuard[T]: ... def is_type(form: TypeForm) -> bool: ... ``` Here are some complete functions that use `TypeForm`: ``` from typing import cast, get_args, get_origin, List, NoReturn, TypeForm def deunionize(form: TypeForm) -> List[TypeForm]: """Decomposes a union or standalone type to its individual type components.""" if form is NoReturn: return [] if get_origin(form) is Union: return cast(List[TypeForm], get_args(form)) else: return [form] def unionize(forms: List[TypeForm]) -> TypeForm: """Composes multiple type components to a single union type.""" components = [] # type: List[TypeForm] for form in forms: components.extend(deunionize(form)) if len(components) == 0: return NoReturn else: return Union[*components] ``` Using TypeForm Types -------------------- TypeForm types may be used as function parameter types, return types, and variable types: ``` def is_type(form: TypeForm) -> bool: ... # parameter type ``` ``` S = TypeVar('S', bound=TypeForm) T = TypeVar('T', bound=TypeForm) def meet_types(s: S, t: T) -> Union[S, T, TypeForm]: ... # return types ``` ``` NULL_TYPE: TypeForm # variable type NULL_TYPE = type(None) ``` ``` NULL_TYPE: TypeForm = type(None) # variable type ``` Note however that a typechecker won't automatically infer a TypeForm type for an unannotated variable assignment that contains a special typing form on the right-hand-side because PEP 484 [^type-alias-syntax] reserves that syntax for defining type aliases: [^type-alias-syntax]: https://www.python.org/dev/peps/pep-0484/#type-aliases ``` NULL_TYPE = type(None) # OOPS; treated as a type alias! ``` If you want a typechecker to use a TypeForm variable type for a bare assignment you'll need to explicitly declare the assignment-target as having `TypeForm` type: ``` NULL_TYPE: TypeForm = type(None) ``` ``` NULL_TYPE = type(None) # type: TypeForm # the type comment is significant ``` ``` NULL_TYPE: TypeForm NULL_TYPE = type(None) ``` Values of type TypeForm ----------------------- The type `TypeForm` has values corresponding to exactly those runtime objects that are valid on the right-hand-side of a variable declaration, ``` value: *form* ``` the right-hand-side of a parameter declaration, ``` def some_func(value: *form*): ``` or as the return type of a function: ``` def some_func() -> *form*: ``` Any runtime object that is valid in one of the above locations is a value of `TypeForm`. Incomplete forms like a bare `Optional` or `Union` are not values of `TypeForm`. Example of values include: * type objects like `int`, `str`, `object`, and `FooClass` * generic collections like `List`, `List[int]`, `Dict`, or `Dict[K, V]` * callables like `Callable`, `Callable[[Arg1Type, Arg2Type], ReturnType]`, `Callable[..., ReturnType]` * union forms like `Optional[str]`, `Union[int, str]`, or `NoReturn` * literal forms like `Literal['r', 'rb', 'w', 'wb']` * type variables like `T` or `AnyStr` * annotated types like `Annotated[int, ValueRange(-10, 5)]` * type aliases like `Vector` (where `Vector = list[float]`) * the `Any` form * the `Type` and `Type[C]` forms * the `TypeForm` form Forward References '''''''''''''''''' Runtime objects for forms which contain forward references such as `Union['Triangle', 'Shape']` are normalized at runtime to `Union[ForwardRef('Triangle'), ForwardRef('Shape')]`. These `ForwardRef(...)` objects are also values of TypeForm. Code that manipulates TypeForm values at runtime should be prepared to deal with ForwardRef(...) values. Type Consistency ---------------- Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the `Any` type. It is defined more formally in PEP 483 [^type-consistency]. This section introduces the new rules needed to support type consistency for TypeForm types: [^type-consistency]: https://www.python.org/dev/peps/pep-0483/#summary-of-gradual-typing * `type`, `Type`, and `Type[C]` are consistent with `TypeForm`. `TypeForm` is consistent with `object`. * `Any` is consistent with `TypeForm`. `TypeForm` is consistent with `Any`. Notably `TypeForm` is *not* consistent with `type`, `Type`, or `Type[C]`. Interactions with Type[C] ------------------------- When writing `Type[C]` it is necessary that `C` must be consistent with `type`. Since `TypeForm` is not consistent with `type`, it is not valid to write `Type[C]` when `C` is a TypeVar with a `bound` of `TypeForm`: ``` T = TypeVar('T', bound=TypeForm) def bad_func(form: T, tp: Type[T]): ... # Type[T] is invalid! ``` A group of type variables where some variables are constrained by TypeForms, some are constrained by `Type`s, and others are unconstrained may appear together as arguments to the same type constructor (i.e. `Union[...]`, `Dict[K, V]`, etc): ``` F = TypeVar('F', bound=TypeForm) C = TypeVar('C') V = TypeVar('V') # Union[F, C, V] contains type variables with many kinds of constraints def pick_random(form: F, tp: Type[C], value: V) -> Union[F, C, V]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. Reference Implementation ======================== The following will be true when [mypy#9773](https://github.com/python/mypy/issues/9773) is implemented: The mypy type checker supports `TypeForm` types. A reference implementation of the runtime component is provided in the `typing_extensions` module. In addition an implementation of the `trycast` and `isassignable` functions mentioned in various examples above is being implemented in the [trycast PyPI module](https://github.com/davidfstr/trycast). Rejected Ideas ============== TypeForm[T] as syntax for TypeVar('T', bound=TypeForm) ------------------------------------------------------ The original proposed syntax for TypeForm used `TypeForm[T]` to mean what is now written as `TypeVar('T', bound=TypeForm)`. The original syntax was seeking to spell a typing constraint that was propagated through an unbounded invariant TypeVar. However it was observed that using TypeForm as a bound on a TypeVar is a lot more straightforward to understand. It's also significantly easier to implement in typecheckers, as it leverages existing constraint-propagation infrastructure for TypeVar rather than creating its own. Restricting TypeForm to "universal" forms only ---------------------------------------------- Originally it was considered important to restrict TypeForm such that its values only included forms that could be used as a local variable type, a parameter type, *and* as a parameter type. For example that would rule out `NoReturn` as a value because it can only be used as a return type and not as a local variable type. Functions taking TypeForms as parameters that only wish to accept "universal" forms (that can be used in all 3 of the aforementioned positions) can write code that raises TypeError or a similar exception at runtime when provided a non-universal form. Standardized type inference rules --------------------------------- This PEP intentionally does not specify any rules for implicitly inferring a `TypeForm` type for an expression. No other type-related PEPs prescribe inference rules, and existing type checkers vary in their behavior. So it would be inconsistent to dictate an inference rule for TypeForm. Individual type checkers may define their own inference rules if they wish. Open Issues =========== [Any points that are still being decided/discussed.] References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< On 1/24/21 10:06 PM, David Foster wrote:
I have drafted an initial PEP for TypeForm, a way to spell the type of typing special forms (like Union[int, str], Literal['foo'], etc). Please see the following commentable document, or the copy at the bottom of this email: https://docs.google.com/document/d/18UF8V00EVU1-h-BtiVFhXoJkvfL4rHp4ORaenMQL...
Please leave your comments either on the document itself or by responses to this email on typing-sig. I will integrate feedback periodically.
For further background on why this feature is being introduced and prior discussions, please see:
(1) the originating thread on typing-sig at: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ...
(2) the discussion about TypeForm on the mypy issue tracker at: https://github.com/python/mypy/issues/9773
On Thu, Jan 28, 2021 at 10:59 PM David Foster <davidfstr@gmail.com> wrote:
The second draft of the TypeForm PEP is ready for another look! [...] For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value:
``` T = TypeVar('T', bound=TypeForm)
def trycast(form: T, value: object) -> Optional[T]: ... ```
This seems wrong. Without `bound=TypeForm`, the signature of trycast() would match things like ``` def trycast(form: int, value: object) -> Optional[int]: ... def trycast(form: List[Union[str, int]], value: object) -> Optional[List[Union[str, int]]]: ... ``` and valid *calls* would be e.g. ``` trycast(3, x) # -> Optional[int] trycast(["a", 42], a) # -> Optional[List[Union[str, int]]] ``` However I presume that your intention for TypeForm is to enable calls like these: ``` trycast(int, x) # -> Optional[int] trycast(List[Union[str, int]], a) # -> Optional[List[Union[str, int]]] ``` That's a very unusual effect of `bound=...`! Normally it restricts acceptable values but doesn't *change* them (here, from values of a type to the spelling of that type). Maybe Eric suggestion to overhaul the design so that you'd write `T` instead of `TypeForm[T]` was just wrong? After all, `TypeForm[T]` feels like an extension of `Type[T]` -- it should accept anything that's valid for `Type[T]`, *plus* e.g. `Union[...]`, `Tuple[...]` or `Any`. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Oh, that's embarrassing. You're right, Guido. My suggested change doesn't allow for the conversion of a TypeForm into an _instance_ of that TypeForm, and that's required for some of the use cases David has in mind. I like the simplification that resulted from this change, but I guess that a parameterized special form is required here. David, sorry that my suggestion led you astray. -- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.
On 1/29/21 10:03 PM, Guido van Rossum wrote:
That's a very unusual effect of `bound=...`! Normally it restricts acceptable values but doesn't *change* them (here, from values of a type to the spelling of that type).
[...] My suggested change doesn't allow for the conversion of a TypeForm into an _instance_ of that TypeForm, and that's required for some of the use cases David has in mind.
I like the simplification that resulted from this change, but I guess
Ah shucks, I see the problem now. On 1/30/21 12:38 AM, Eric Traut wrote: that a parameterized special form is required here.
David, sorry that my suggestion led you astray.
No worries. I was really liking many of the simplifications myself. :) At least I was able to independently eliminate the requirement for "banning non-universal typing forms" which was introducing a fair bit of complexity itself. I'll make a third draft soon. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
And here's the third draft for the TypeForm PEP! Only one change was made: * Revert back to the `TypeForm[T]` syntax (with the [T]), removing the `T = TypeVar('T', bound=TypeForm)` syntax I was pleasantly surprised to see that this reversion barely changed the complexity, especially since the `T` in a `TypeForm[T]` is tightly restricted to only allow a plain invariant TypeVar with no bound. Let me know what you think! -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: TypeForm: Type Hints for Typing Special Forms and Regular Types Author: David Foster <david at dafoster.net> Sponsor: TODO Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 647 (TypeGuard) Created: 21-Dec-2020 Python-Version: 3.10 Post-History: 24-Jan-2021, 28-Jan-2021, 04-Feb-2021 Abstract ======== PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. [^type-c]: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects Motivation ========== The introduction of `TypeForm` allows new kinds of metaprogramming functions that operate on typing special forms to be type-annotated. For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value: ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... ``` It's very powerful to be able to define a function whose return type can be influenced by a `form` value passed at runtime. Here is another function that checks whether a value is assignable to a variable of a particular type, and if so returns True (as a special TypeGuard bool [^TypeGuardPep]): [^TypeGuardPep]: https://www.python.org/dev/peps/pep-0647/ ``` def isassignable(value: object, form: TypeForm[T]) -> TypeGuard[T]: ... ``` When combining TypeForm and TypeGuard together in this way, typecheckers can understand the relationship between the type provided to parameter `form` and the function's return type, and narrow the return type appropriately depending on what form is passed in: ``` request_json = ... # type: object assert isassignable(request_json, Shape) # request_json is narrowed to type Shape here! ``` NB: The preceding example functions implement the kinds of enhanced `isinstance` checks that were omitted in PEP 589[^typeddict-no-isinstance] which are very useful for, among other things, [checking whether a value decoded from JSON conforms to a particular structure] of nested `TypedDict`s, `List`s, `Optional`s, `Literal`s, and other types. [^typeddict-no-isinstance]: https://www.python.org/dev/peps/pep-0589/#using-typeddict-types [checking whether a value decoded from JSON conforms to a particular structure]: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ... Specification ============= A type-form type represents a `type` object or a special typing form such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. A type-form type can be written as either `TypeForm[T]` where `T` is an type variable or as `TypeForm` with no argument. The `T` in `TypeForm[T]` is always an invariant type variable with no bound: * Attempting to use a type variable `T` declared with a `bound=...` or marked with `covariant=True` or `contravariant=True` in a `TypeForm[T]` should be rejected by typecheckers. * Attempting to use a literal type-form `*form*` as an argument to `TypeForm[*form*]`, such as `TypeForm[int]`, should be rejected by typecheckers. (See §"TypeForm[*form*] or Literal[*form*]" in Rejected Ideas for rationale.) The syntax `TypeForm` alone, without a type argument, is equivalent to `TypeForm[_T]` where `_T` is a freshly created type variable. ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... def isassignable(value: object, form: T) -> TypeGuard[T]: ... def is_type(form: TypeForm) -> bool: ... ``` Here are some complete functions that use `TypeForm`: ``` from typing import cast, get_args, get_origin, List, NoReturn, TypeForm def deunionize(form: TypeForm) -> List[TypeForm]: """Decomposes a union or standalone type to its individual type components.""" if form is NoReturn: return [] if get_origin(form) is Union: return cast(List[TypeForm], get_args(form)) else: return [form] def unionize(forms: List[TypeForm]) -> TypeForm: """Composes multiple type components to a single union type.""" components = [] # type: List[TypeForm] for form in forms: components.extend(deunionize(form)) if len(components) == 0: return NoReturn else: return Union[*components] ``` Using TypeForm Types -------------------- TypeForm types may be used as function parameter types, return types, and variable types: ``` def is_type(form: TypeForm) -> bool: ... # parameter type ``` ``` S = TypeVar('S') T = TypeVar('T') def meet_types(s: TypeForm[S], t: TypeForm[T]) \ -> Union[TypeForm[S], TypeForm[T], TypeForm]: ... # return types ``` ``` NULL_TYPE: TypeForm # variable type NULL_TYPE = type(None) ``` ``` NULL_TYPE: TypeForm = type(None) # variable type ``` Note however that a typechecker won't automatically infer a TypeForm type for an unannotated variable assignment that contains a special typing form on the right-hand-side because PEP 484 [^type-alias-syntax] reserves that syntax for defining type aliases: [^type-alias-syntax]: https://www.python.org/dev/peps/pep-0484/#type-aliases ``` NULL_TYPE = type(None) # OOPS; treated as a type alias! ``` If you want a typechecker to use a TypeForm variable type for a bare assignment you'll need to explicitly declare the assignment-target as having `TypeForm` type: ``` NULL_TYPE: TypeForm = type(None) ``` ``` NULL_TYPE = type(None) # type: TypeForm # the type comment is significant ``` ``` NULL_TYPE: TypeForm NULL_TYPE = type(None) ``` Values of type TypeForm ----------------------- The type `TypeForm` has values corresponding to exactly those runtime objects that are valid on the right-hand-side of a variable declaration, ``` value: *form* ``` the right-hand-side of a parameter declaration, ``` def some_func(value: *form*): ``` or as the return type of a function: ``` def some_func() -> *form*: ``` Any runtime object that is valid in one of the above locations is a value of `TypeForm`. Incomplete forms like a bare `Optional` or `Union` are not values of `TypeForm`. Example of values include: * type objects like `int`, `str`, `object`, and `FooClass` * generic collections like `List`, `List[int]`, `Dict`, or `Dict[K, V]` * callables like `Callable`, `Callable[[Arg1Type, Arg2Type], ReturnType]`, `Callable[..., ReturnType]` * union forms like `Optional[str]`, `Union[int, str]`, or `NoReturn` * literal forms like `Literal['r', 'rb', 'w', 'wb']` * type variables like `T` or `AnyStr` * annotated types like `Annotated[int, ValueRange(-10, 5)]` * type aliases like `Vector` (where `Vector = list[float]`) * the `Any` form * the `Type` and `Type[C]` forms * the `TypeForm` and `TypeForm[T]` forms Forward References '''''''''''''''''' Runtime objects for forms which contain forward references such as `Union['Triangle', 'Shape']` are normalized at runtime to `Union[ForwardRef('Triangle'), ForwardRef('Shape')]`. These `ForwardRef(...)` objects are also values of TypeForm. Code that manipulates TypeForm values at runtime should be prepared to deal with ForwardRef(...) values. Type Consistency ---------------- Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the `Any` type. It is defined more formally in PEP 483 [^type-consistency]. This section introduces the new rules needed to support type consistency for TypeForm types: [^type-consistency]: https://www.python.org/dev/peps/pep-0483/#summary-of-gradual-typing * `type`, `Type`, and `Type[C]` are consistent with `TypeForm`. `TypeForm` is consistent with `object`. * `TypeForm[S]` is consistent with `TypeForm[T]` iff `S` and `T` are the same type variable. * `Any` is consistent with `TypeForm`. `TypeForm` is consistent with `Any`. Notably `TypeForm` is *not* consistent with `type`, `Type`, or `Type[C]`. In addition the `T` for `TypeForm[T]` is not consistent with `type`, `Type`, or `Type[C]`. See §"Interactions with Type[C]" below for details. Overloading ----------- `TypeForm` can be used in overloads. For example: ``` T = TypeVar('T') F = TypeVar('F') @overload def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... @overload def trycast(form: TypeForm[T], value: object, failure: F) -> Union[T, F]: ... ``` Interactions with Type[C] ------------------------- When writing `Type[C]` it is necessary that `C` must be consistent with `type`. Since the `T` in `TypeForm[T]` is not consistent with `type`, it is not valid for a `TypeForm` and a `Type` to simultaneously constrain the same type variable within a generic function definition: ``` T = TypeVar('T') def bad_func(form: TypeForm[T], tp: Type[T]): ... # invalid ``` A group of type variables where some variables are constrained by TypeForms, some are constrained by `Type`s, and others are unconstrained may appear together as arguments to the same type constructor (i.e. `Union[...]`, `Dict[K, V]`, etc): ``` F = TypeVar('F') C = TypeVar('C') V = TypeVar('V') # Union[F, C, V] contains type variables with many kinds of constraints def pick_random(form: TypeForm[F], tp: Type[C], value: V) -> Union[F, C, V]: ... ``` Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. Reference Implementation ======================== The following will be true when [mypy#9773](https://github.com/python/mypy/issues/9773) is implemented: The mypy type checker supports `TypeForm` types. A reference implementation of the runtime component is provided in the `typing_extensions` module. In addition an implementation of the `trycast` and `isassignable` functions mentioned in various examples above is being implemented in the [trycast PyPI module](https://github.com/davidfstr/trycast). Rejected Ideas ============== TypeForm[*form*] or Literal[*form*] ----------------------------------- This PEP only defines syntax for a bare `TypeForm` and a `TypeForm[T]` parameterized by a type variable. In particular it does not define syntax like that allows referring to a specific known type-form value via a syntax like `TypeForm[*form*]`: ``` NULL_TYPE = type(None) # type: TypeForm[_T], rather than TypeForm[type(None)] UNION_STR_TYPE = Union[Str] # type: TypeForm[_T], rather than TypeForm[Union[str]] ``` In addition similar syntax like `Literal[*form*]` also does not work. Neither of those syntaxes seem to be especially useful; type forms are not typically used as sentinel values. It is possible that a future PEP may revisit such syntaxes. Restricting TypeForm to "universal" forms only ---------------------------------------------- Originally it was considered important to restrict TypeForm such that its values only included forms that could be used as a local variable type, a parameter type, *and* as a parameter type. For example that would rule out `NoReturn` as a value because it can only be used as a return type and not as a local variable type. Functions taking TypeForms as parameters that only wish to accept "universal" forms (that can be used in all 3 of the aforementioned positions) can write code that raises TypeError or a similar exception at runtime when provided a non-universal form. Standardized type inference rules --------------------------------- This PEP intentionally does not specify any rules for implicitly inferring a `TypeForm` type for an expression. No other type-related PEPs prescribe inference rules, and existing type checkers vary in their behavior. So it would be inconsistent to dictate an inference rule for TypeForm. Individual type checkers may define their own inference rules if they wish. Open Issues =========== [Any points that are still being decided/discussed.] References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Hi David, Thank you for writing up the PEP; that was an interesting read. On Thu, Feb 4, 2021, at 22:27, David Foster wrote:
PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`.
The question raised on the GitHub issue [0], "Why can't we make `Type[T]` also work for other special forms?", will likely come up for others reading this too. Would it be worth adding a sentence to explain?
For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value:
``` T = TypeVar('T')
def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... ```
Would it make sense to have an example that explicitly ties the concern raised in the introductory paragraph ("explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]`") to what this PEP aims to solve? Phrased differently: why can the `trycast` definition not be: ``` def trycast(form: Type[T], value: object) -> Optional[T]: ... ``` That seems to work OK (see attached), although understanding why it isn't adequate would help motivate the document. I don't know who the target audience of the PEP is, but presumably it would not hurt to make it a bit more widely accessible. Crystallizing the advantage over what is currently available with usage examples of, say, `Union[int, str]`, would help. I've never been active on a SIG before, so please accept my apologies if comments from those still learning the inner workings of types are considered unhelpful at this stage or a breach of social protocol. Best regards, Stéfan [0] https://github.com/python/mypy/issues/9773#issuecomment-736810721
Thanks David. Here’s feedback on your latest draft. As Stefan and others have suggested, I think it’s worth having a discussion about why we wouldn't extend `Type[T]` to accommodate this use case. I can think of compelling arguments in favor and against doing so. At a minimum, a section should be added to the “Rejected Ideas” section explaining why this approach was not used. I think it’s important to formally explain what the type parameter for TypeForm means. The parameter describes the type of an _instance_ of the `TypeForm`. For example, `TypeForm[int | str]` describes a type that, when instantiated, is either an instance of `int` or an instance of `str`. This is important when considering signatures like `def foo(TypeForm[T]) -> T`. You state that the type _parameter_ for `TypeForm` is invariant, which is important to specify. However, I don’t think there is a need to say that a covariant or contravariant type variable used as a type _argument_ is illegal. That would be a really odd restriction that no other generic type imposes (including `Type`), and I don’t see any reason why it is necessary here. The variance of the _type parameter_ is what matters. I also don’t see any reason why to preclude the use of “bound” for a type argument. Why make a special case here when it’s not needed? There are even cases where it might be useful. I also don’t see a need to preclude the use of a class as a type argument (e.g. `TypeForm[int]`). This has a legitimate meaning, so why create an odd special case to preclude it? (As you can probably tell, I’m not a fan of special cases. There are too many of them in the Python type system already, IMO.) You specifically mention that `TypeForm` alone is equivalent to "`TypeForm[_T]` where `_T` is a freshly created type variable”. First, that’s an odd phrase, and I’m not sure how to interpret “freshly created type variable". Second, this is another situation where deviating from the normal rules is undesirable and unnecessary. By normal rules established in PEP 484, missing type arguments are assumed to be `Any`. That is, `TypeForm` should imply `TypeForm[Any]`. (Or in the case of Pyright, which distinguishes between explicit and implied `Any`, it means `TypeForm[Unknown]` and will be flagged as an error in strict mode.) I recommend deleting this sentence completely so as not imply that there is a special-case rule. The samples in the current draft use the “bare” form of `TypeForm` with no type arguments. I recommend against using (or encouraging the use of) the bare form. It will be considered an error in strict mode. Therefore, the samples should show an explicitly parameterized form: `TypeForm[Any]`. The `deunionize` sample contains a `cast` operation that masks an error. The `get_args` function returns a tuple, but the sample is casting it directly to a `list`, which will result in runtime errors if a caller tries to use it as a list. I recommend removing the cast entirely and either wrapping the results of `get_args(form)` in a `list()` constructor or (even better) changing the return type from `List[TypeForm]` to `Sequence[TypeForm[Any]]`. The `unionize` example contains the expression `Union[*components]` which is not legal syntax — at least not until PEP 637 is ratified and implemented. Is there another way to dynamically construct a `Union` object? I’m not aware of any way to do so in a way that a type checker would be able to reason about it. I don’t understand where `deunionize` and `unionize` operations (as shown in the examples) would be used. They provide no value for static type checking. Perhaps you have in mind use cases for dynamic (runtime) type checking? If the intent is to demonstrate a good static type checking use case for `TypeForm`, you might want to look for better examples. The section on “overloading” seems unnecessary to me. The fact that `TypeForm` works in an overload is implied by the previous sections of the spec. I guess it doesn’t hurt to say that it works with overloads, but it makes me wonder why overloads are specifically called out here. The section titled “Interactions with Type[C]” state, “it is not valid for a `TypeForm` and a `Type` to simultaneously constrain…”. I disagree with that statement. There are cases where such a constraint can be satisfied and cases where it cannot. For example, if someone were to call `bad_func(int, float)`, there is a valid solution (T => `float`) that satisfies all the constraints. In cases where the constraint cannot be met (e.g. `bad_func(Callable[…, Any], int)`), the constraint solver will detect that there is no valid solution and emit an error. So I don’t see a need to call this out as illegal. The normal rules of TypeVar constraint solving apply here, and no special case is needed. My recommendation is that you delete this entire section. -- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.
On 2/7/21 1:04 PM, Eric Traut wrote:
Thanks David.
Here’s feedback on your latest draft.
As Stefan and others have suggested, I think it’s worth having a discussion about why we wouldn't extend `Type[T]` to accommodate this use case. I can think of compelling arguments in favor and against doing so. At a minimum, a section should be added to the “Rejected Ideas” section explaining why this approach was not used.
+1
I think it’s important to formally explain what the type parameter for TypeForm means. The parameter describes the type of an _instance_ of the `TypeForm`. For example, `TypeForm[int | str]` describes a type that, when instantiated, is either an instance of `int` or an instance of `str`. This is important when considering signatures like `def foo(TypeForm[T]) -> T`.
Will do.
You state that the type _parameter_ for `TypeForm` is invariant, which is important to specify. However, I don’t think there is a need to say that a covariant or contravariant type variable used as a type _argument_ is illegal. That would be a really odd restriction that no other generic type imposes (including `Type`), and I don’t see any reason why it is necessary here. The variance of the _type parameter_ is what matters.
I also don’t see any reason why to preclude the use of “bound” for a type argument. Why make a special case here when it’s not needed? There are even cases where it might be useful.
The idea was that by restricting the kind of the type variable used, the implementation might be simpler. Based on what I'm hearing from you it would actually not be simpler to impose these restrictions. Opening up the use of bound=... in particular feels like it would require some elaboration. I'll think about it.
I also don’t see a need to preclude the use of a class as a type argument (e.g. `TypeForm[int]`). This has a legitimate meaning, so why create an odd special case to preclude it? (As you can probably tell, I’m not a fan of special cases. There are too many of them in the Python type system already, IMO.)
I suppose types like `TypeForm[int]` seemed odd to me. Consider the following function: ``` def i_like_int(form: TypeForm[int]) -> None: pass i_like_int(int) # ok i_like_int(str) # error ``` If you have a TypeForm[...] where ... is not a type variable then you're asking for a particular value (or values in the case of a Union) to be passed, which doesn't seem like it would be used by any normal function I can think of. Again since this usage seemed un-useful, I thought to ban it for simplicity. But again if it's not actually *simplifying* to ban its usage, then I'm happy to un-ban it.
You specifically mention that `TypeForm` alone is equivalent to "`TypeForm[_T]` where `_T` is a freshly created type variable”. First, that’s an odd phrase, and I’m not sure how to interpret “freshly created type variable". Second, this is another situation where deviating from the normal rules is undesirable and unnecessary. By normal rules established in PEP 484, missing type arguments are assumed to be `Any`. That is, `TypeForm` should imply `TypeForm[Any]`. (Or in the case of Pyright, which distinguishes between explicit and implied `Any`, it means `TypeForm[Unknown]` and will be flagged as an error in strict mode.) I recommend deleting this sentence completely so as not imply that there is a special-case rule.
Aye. In my mind `TypeForm[Any]` and "`TypeForm[_T]` where `_T` is a freshly created type variable” are equivalent anyway, so happy to clarify or remove the mention outright.
The samples in the current draft use the “bare” form of `TypeForm` with no type arguments. I recommend against using (or encouraging the use of) the bare form. It will be considered an error in strict mode. Therefore, the samples should show an explicitly parameterized form: `TypeForm[Any]`.
Is it also discouraged to use the bare form of `Type` (or `type`)? Seems like that's a more succinct usage when the specific type (or form) doesn't matter. I'm not sure what you mean by strict mode. I don't recall a "strict mode" being mentioned in any typing-related PEP.
The `deunionize` sample contains a `cast` operation that masks an error. The `get_args` function returns a tuple, but the sample is casting it directly to a `list`, which will result in runtime errors if a caller tries to use it as a list. I recommend removing the cast entirely and either wrapping the results of `get_args(form)` in a `list()` constructor or (even better) changing the return type from `List[TypeForm]` to `Sequence[TypeForm[Any]]`.
Ah ha. Will either alter the function to return a tuple or wrap in list constructor.
The `unionize` example contains the expression `Union[*components]` which is not legal syntax — at least not until PEP 637 is ratified and implemented. Is there another way to dynamically construct a `Union` object? I’m not aware of any way to do so in a way that a type checker would be able to reason about it.
`Union.__getitem__(*components)`? I expect that would work at runtime at least. It's a bit icky, but can be explained with a comment perhaps. Although it's more of a problem that a typechecker wouldn't necessarily understand it. I suppose I can just use another cast()...
I don’t understand where `deunionize` and `unionize` operations (as shown in the examples) would be used. They provide no value for static type checking. Perhaps you have in mind use cases for dynamic (runtime) type checking? If the intent is to demonstrate a good static type checking use case for `TypeForm`, you might want to look for better examples.
I wanted to provide at least one concise example of a full function body that used TypeForm as a parameter or return type. A more compelling example like the implementation of trycast would be quite elaborate and longwinded to put inline. I could perhaps *link* to an implementation of a large real-world example though.
The section on “overloading” seems unnecessary to me. The fact that `TypeForm` works in an overload is implied by the previous sections of the spec. I guess it doesn’t hurt to say that it works with overloads, but it makes me wonder why overloads are specifically called out here.
I feel like when I was inspecting the mypy codebase that I thought I'd have to do something special to make @overload and TypeForm[T] work together. I'll take another look.
The section titled “Interactions with Type[C]” state, “it is not valid for a `TypeForm` and a `Type` to simultaneously constrain…”. I disagree with that statement. There are cases where such a constraint can be satisfied and cases where it cannot. For example, if someone were to call `bad_func(int, float)`, there is a valid solution (T => `float`) that satisfies all the constraints. In cases where the constraint cannot be met (e.g. `bad_func(Callable[…, Any], int)`), the constraint solver will detect that there is no valid solution and emit an error. So I don’t see a need to call this out as illegal. The normal rules of TypeVar constraint solving apply here, and no special case is needed. My recommendation is that you delete this entire section.
Another attempt to limit the scope and simplify, which apparently does not actually simplify. :) So sure, I'm good with removing the restriction. Thanks for all the suggestions Eric! Seems that I'll be able to make the next draft a lot more consistent with existing typechecker behavior and simplify the implementation further. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On 2/4/21 11:33 PM, Stefan van der Walt wrote:
Hi David,
Thank you for writing up the PEP; that was an interesting read.
On Thu, Feb 4, 2021, at 22:27, David Foster wrote:
PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`.
The question raised on the GitHub issue [0], "Why can't we make `Type[T]` also work for other special forms?", will likely come up for others reading this too. Would it be worth adding a sentence to explain?
Agreed. I'll plan to put a mention in the Rejected Ideas section, since extending Type[T] was also my first reaction when encountering the non-expressability problem that this PEP is aiming to solve.
[...] Phrased differently: why can the `trycast` definition not be:
``` def trycast(form: Type[T], value: object) -> Optional[T]: ... ```
That seems to work OK (see attached), although understanding why it isn't adequate would help motivate the document.
Good point. The above definition works if you pass in regular class types (like `int` or `str`) but rejects more complex arguments like `Union[int, str]`. And that is probably not obvious.
I don't know who the target audience of the PEP is, but presumably it would not hurt to make it a bit more widely accessible.
I'd say the target audience would be users of Python's static typing annotations, particularly relatively advanced users who were looking at implementing "new kinds of metaprogramming functions that operate on typing special forms". (Quote is from the first sentence in the PEP's "Motivation" section.) So far the main likely users of TypeForm seem to be libraries that manipulate abstract structures whose shape is defined by a type annotation. For example the pydantic and trycast libraries.
Crystallizing the advantage over what is currently available with usage examples of, say, `Union[int, str]`, would help.
I did make an example with `deunionize` which wouldn't be expressable at all with Type[C], although perhaps that's not immediately obvious. I will grant my only *early* example I had used `Shape` (a TypedDict), which I believe would still have worked with a Type[C].
I've never been active on a SIG before, so please accept my apologies if comments from those still learning the inner workings of types are considered unhelpful at this stage or a breach of social protocol.
I'm personally open to well-considered feedback from all, even with occasional mistakes and misunderstandings. 👍 It's worth considering that any particular expert is typically only an expert in a few areas in any case. For example a lot of the recent detailed technical discussion on the sister Variadic Generics thread is now a bit over my head. ^_^; -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
On 2/4/21 11:33 PM, Stefan van der Walt wrote:
The question raised on the GitHub issue [0], "Why can't we make `Type[T]` also work for other special forms?", will likely come up for others reading this too. Would it be worth adding a sentence to explain?
[0] https://github.com/python/mypy/issues/9773#issuecomment-736810721
Guido I tried to write up an explanation for this question, but couldn't come up with a very strong answer. Could you, Ivan, or Jukka (on CC) elaborate why it would be a bad idea to widen `Type` such that typing special forms like `Optional[str]` or `Any` would be assignable to it? My lukewarm attempt to explain the current rationale is at the bottom of this email. Personally I think it really *would* be a lot easier to just widen Type[T] to also include typing special forms. It's unclear to me why it's specifically important to have it only match things that can be used in isinstance(). It doesn't even do that job perfectly at the moment: `isinstance(MyTypedDict(value=1), MyTypedDict)` fails at runtime with TypeError but mypy treats `MyTypedDict` as assignable to `Type` anyway. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>
Rejected Ideas ============== Widen Type[T] to support all typing special forms ------------------------------------------------- `Type` was [designed] to only be used for things that can be the second argument of `isinstance()`. And those things must be actual class objects -- they cannot be typing special forms like `Any`, `Optional[int]` or `List[str]`. Since `Type` is restricted in this way, it is necessary to introduce a new spelling, `TypeForm`, to include the additional special forms as well. [designed]: https://github.com/python/mypy/issues/8992#issuecomment-643369024 <<<<<<<<<<
I think isinstance() is a distraction. Sorry for that. The intent when Type[] was introduced was that it would describe a class object. TypedDict is an odd case because it uses runtime erasure. (There is also an issue around the constructor signature, so there is a silent assumption that constructors follow Liskov.) But Union etc. are not class objects. Could we nevertheless forge ahead and widen the meaning of Type[]? I suppose so. But I think it would reduce clarity about the intent of the user. With TypeForm, I would expect the argument to be introspected. An example might be a function that renders a type prettier than the default str(). Or a JSON validator generator. With Type, I would expect the class to be instantiated. —Guido On Sat, Feb 13, 2021 at 22:22 David Foster <davidfstr@gmail.com> wrote:
On 2/4/21 11:33 PM, Stefan van der Walt wrote:
The question raised on the GitHub issue [0], "Why can't we make `Type[T]` also work for other special forms?", will likely come up for others reading this too. Would it be worth adding a sentence to explain?
[0] https://github.com/python/mypy/issues/9773#issuecomment-736810721
Guido I tried to write up an explanation for this question, but couldn't come up with a very strong answer. Could you, Ivan, or Jukka (on CC) elaborate why it would be a bad idea to widen `Type` such that typing special forms like `Optional[str]` or `Any` would be assignable to it?
My lukewarm attempt to explain the current rationale is at the bottom of this email.
Personally I think it really *would* be a lot easier to just widen Type[T] to also include typing special forms. It's unclear to me why it's specifically important to have it only match things that can be used in isinstance().
It doesn't even do that job perfectly at the moment: `isinstance(MyTypedDict(value=1), MyTypedDict)` fails at runtime with TypeError but mypy treats `MyTypedDict` as assignable to `Type` anyway.
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>
Rejected Ideas ==============
Widen Type[T] to support all typing special forms -------------------------------------------------
`Type` was [designed] to only be used for things that can be the second argument of `isinstance()`. And those things must be actual class objects -- they cannot be typing special forms like `Any`, `Optional[int]` or `List[str]`.
Since `Type` is restricted in this way, it is necessary to introduce a new spelling, `TypeForm`, to include the additional special forms as well.
[designed]: https://github.com/python/mypy/issues/8992#issuecomment-643369024
<<<<<<<<<<
-- --Guido (mobile)
The fourth draft for the TypeForm PEP is now ready! Major changes: * Improvements to motivation - Why not use Type[C] instead of TypeForm[T]? - Explain more in: §"Rejected Ideas" > §"Widen Type[T] to support all typing special forms" * Add/revise examples of functions that use TypeForm. * Explain more about the T in TypeForm[T]. No longer restrict it to being a TypeVar. * Clarify that a bare `TypeForm` is the same as `TypeForm[Any]`. * Eliminate redundant section on overloads. * Eliminate special rules around TypeVar constraint solving that involves a TypeForm. I have highlighted changed sections by marking them with ⭐. The PEP really seems to be coming together now. Next big steps I believe are: * Find a sponsor * RST-ize the PEP and submit a PR to the PEPs repo -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
PEP: 9999 Title: TypeForm: Type Hints for Typing Special Forms and Regular Types Author: David Foster <david at dafoster.net> Sponsor: TODO Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 647 (TypeGuard) Created: 21-Dec-2020 Python-Version: 3.10 Post-History: 24-Jan-2021, 28-Jan-2021, 04-Feb-2021, 14-Feb-2021 Abstract ======== PEP 484 [^type-c] defines the type `Type[C]` where `C` is a class, to refer to a class object that is a subtype of `C`. It explicitly does not allow `Type[C]` to refer to typing special forms such as the runtime object `Optional[str]` even if `C` is an unbounded `TypeVar`. This PEP proposes a new type `TypeForm` that allows referring to both any class object and *also* to any runtime object that is a typing special form, such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. [^type-c]: https://www.python.org/dev/peps/pep-0484/#the-type-of-class-objects Motivation ========== The introduction of `TypeForm` allows new kinds of metaprogramming functions that operate on typing special forms to be type-annotated. For example, here is a function that checks whether a value is assignable to a variable of a particular type, and if so returns the original value: ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... ``` It's very powerful to be able to define a function whose return type can be influenced by a `form` value passed at runtime. Here is another function that checks whether a value is assignable to a variable of a particular type, and if so returns True (as a special TypeGuard bool [^TypeGuardPep]): [^TypeGuardPep]: https://www.python.org/dev/peps/pep-0647/ ``` def isassignable(value: object, form: TypeForm[T]) -> TypeGuard[T]: ... ``` When combining TypeForm and TypeGuard together in this way, typecheckers can understand the relationship between the type provided to parameter `form` and the function's return type, and narrow the return type appropriately depending on what form is passed in: ``` request_json = ... # type: object assert isassignable(request_json, Shape) # request_json is narrowed to type Shape here! ``` NB: The preceding example functions implement the kinds of enhanced `isinstance` checks that were omitted in PEP 589[^typeddict-no-isinstance] which are very useful for, among other things, [checking whether a value decoded from JSON conforms to a particular structure] of nested `TypedDict`s, `List`s, `Optional`s, `Literal`s, and other types. [^typeddict-no-isinstance]: https://www.python.org/dev/peps/pep-0589/#using-typeddict-types [checking whether a value decoded from JSON conforms to a particular structure]: https://mail.python.org/archives/list/typing-sig@python.org/thread/I5ZOQICTJ... ⭐ One might think you could define the `trycast` or `isassignable` examples above to take a `Type[T]` - which is syntax that already exists - rather than a `TypeForm[T]`. However if you were to do that then certain typing special forms like `Optional[str]` - which are not class objects and therefore not `Type`s at runtime - would be rejected: ``` # uses a Type[T] parameter rather than a TypeForm[T] def trycast_type(form: Type[T], value: object) -> Optional[T]: ... trycast_type(str, 'hi') # ok; str is a Type trycast_type(Optional[str], 'hi') # ERROR; Optional[str] is not a Type trycast_type(Union[str, int], 'hi') # ERROR; Union[str, int] is not a Type trycast_type(MyTypedDict, dict(value='hi')) # questionable usage; but ok in Python 3.9 ``` For a longer explanation of why we don't just widen `Type[T]` to accept all special forms, see §"Widen Type[T] to support all typing special forms" in §"Rejected Ideas". Specification ============= A type-form type represents a `type` object or a special typing form such as `Optional[str]`, `Union[int, str]`, or `MyTypedDict`. A type-form type can be written as either `TypeForm[T]` where `T` is an type variable or as `TypeForm` with no argument. ⭐ The type parameter `T` in `TypeForm[T]` describes the type of an *instance* of `TypeForm[T]`. For example, `TypeForm[Union[int, str]]` describes a type that, when instantiated, is either an instance of `int` or an instance of `str`. The parameter is invariant, so an instance of `TypeForm[object]` *must* be the value `object` and cannot be, for example, `str` or any other subtype of `object`. ⭐ The syntax `TypeForm` alone, without a type argument, is equivalent to `TypeForm[Any]`. ``` T = TypeVar('T') def trycast(form: TypeForm[T], value: object) -> Optional[T]: ... def isassignable(value: object, form: T) -> TypeGuard[T]: ... def is_type(form: TypeForm) -> bool: ... ``` Here are some complete functions that use `TypeForm`: ⭐ ``` from typing import TypeForm, TypeVar TargetType = TypeVar('TargetType', int, float, str) def convert(value: object, form: TypeForm[TargetType]) -> TargetType: if form is str: return str(value) if form is int: return int(value) if form is float: return float(value) raise NotImplementedError('unsupported TargetType') print(convert('1', int) + convert('1.2', float)) ``` ⭐ ``` from typing import cast, get_args, get_origin, List, NoReturn, Sequence, TypeForm def deunionize(form: TypeForm) -> Tuple[TypeForm, ...]: """Decomposes a union or standalone type to its individual type components.""" if form is NoReturn: return () if get_origin(form) is Union: return get_args(form) else: return (form,) def unionize(forms: Sequence[TypeForm]) -> TypeForm: """Composes multiple type components to a single union type.""" components = [] # type: List[TypeForm] for form in forms: components.extend(deunionize(form)) if len(components) == 0: return NoReturn else: return Union.__getitem__(*components) # Union[*components] ``` ⭐ For an even longer example of a function that could use TypeForm, please see the initial implementation of the [trycast] function. [trycast]: https://github.com/davidfstr/trycast/blob/65859341ff9bab13fd5c39c0f5b1721d5d... Using TypeForm Types -------------------- TypeForm types may be used as function parameter types, return types, and variable types: ``` def is_type(form: TypeForm) -> bool: ... # parameter type ``` ``` S = TypeVar('S') T = TypeVar('T') def meet_types(s: TypeForm[S], t: TypeForm[T]) \ -> Union[TypeForm[S], TypeForm[T], TypeForm]: ... # return types ``` ``` NULL_TYPE: TypeForm # variable type NULL_TYPE = type(None) ``` ``` NULL_TYPE: TypeForm = type(None) # variable type ``` Note however that a typechecker won't automatically infer a TypeForm type for an unannotated variable assignment that contains a special typing form on the right-hand-side because PEP 484 [^type-alias-syntax] reserves that syntax for defining type aliases: [^type-alias-syntax]: https://www.python.org/dev/peps/pep-0484/#type-aliases ``` NULL_TYPE = type(None) # OOPS; treated as a type alias! ``` If you want a typechecker to use a TypeForm variable type for a bare assignment you'll need to explicitly declare the assignment-target as having `TypeForm` type: ``` NULL_TYPE: TypeForm = type(None) ``` ``` NULL_TYPE = type(None) # type: TypeForm # the type comment is significant ``` ``` NULL_TYPE: TypeForm NULL_TYPE = type(None) ``` Values of type TypeForm ----------------------- The type `TypeForm` has values corresponding to exactly those runtime objects that are valid on the right-hand-side of a variable declaration, ``` value: *form* ``` the right-hand-side of a parameter declaration, ``` def some_func(value: *form*): ``` or as the return type of a function: ``` def some_func() -> *form*: ``` Any runtime object that is valid in one of the above locations is a value of `TypeForm`. Incomplete forms like a bare `Optional` or `Union` are not values of `TypeForm`. Example of values include: * type objects like `int`, `str`, `object`, and `FooClass` * generic collections like `List`, `List[int]`, `Dict`, or `Dict[K, V]` * callables like `Callable`, `Callable[[Arg1Type, Arg2Type], ReturnType]`, `Callable[..., ReturnType]` * union forms like `Optional[str]`, `Union[int, str]`, or `NoReturn` * literal forms like `Literal['r', 'rb', 'w', 'wb']` * type variables like `T` or `AnyStr` * annotated types like `Annotated[int, ValueRange(-10, 5)]` * type aliases like `Vector` (where `Vector = list[float]`) * the `Any` form * the `Type` and `Type[C]` forms * the `TypeForm` and `TypeForm[T]` forms Forward References '''''''''''''''''' Runtime objects for forms which contain forward references such as `Union['Triangle', 'Shape']` are normalized at runtime to `Union[ForwardRef('Triangle'), ForwardRef('Shape')]`. These `ForwardRef(...)` objects are also values of TypeForm. Code that manipulates TypeForm values at runtime should be prepared to deal with ForwardRef(...) values. Type Consistency ---------------- Informally speaking, *type consistency* is a generalization of the is-subtype-of relation to support the `Any` type. It is defined more formally in PEP 483 [^type-consistency]. This section introduces the new rules needed to support type consistency for TypeForm types: [^type-consistency]: https://www.python.org/dev/peps/pep-0483/#summary-of-gradual-typing * `type`, `Type`, and `Type[C]` are consistent with `TypeForm`. `TypeForm` is consistent with `object`. ⭐ * `TypeForm[S]` is consistent with `TypeForm[T]` iff `S` and `T` are the same type. * `Any` is consistent with `TypeForm`. `TypeForm` is consistent with `Any`. Notably `TypeForm` is *not* consistent with `type`, `Type`, or `Type[C]`. ⭐ Interactions with Type[C] ------------------------- A group of type variables where some variables are constrained by TypeForms, some are constrained by `Type`s, and others are unconstrained may appear together as arguments to the same type constructor (i.e. `Union[...]`, `Dict[K, V]`, etc): ``` F = TypeVar('F') C = TypeVar('C') V = TypeVar('V') # Union[F, C, V] contains type variables with many kinds of constraints def pick_random(form: TypeForm[F], tp: Type[C], value: V) -> Union[F, C, V]: return random.choice((form, tp, value)) ``` No additional type-checking rules should be need to support this case; the normal rules of TypeVar constraint solving apply here. Backwards Compatibility ======================= No other backward incompatible changes are made by this PEP. Reference Implementation ======================== The following will be true when [mypy#9773](https://github.com/python/mypy/issues/9773) is implemented: The mypy type checker supports `TypeForm` types. A reference implementation of the runtime component is provided in the `typing_extensions` module. In addition an implementation of the `trycast` and `isassignable` functions mentioned in various examples above is being implemented in the [trycast PyPI module](https://github.com/davidfstr/trycast). Rejected Ideas ============== ⭐ Widen Type[T] to support all typing special forms ------------------------------------------------- `Type` was [designed] to only be used to describe class objects. A class object can always be instantiated by calling it and can always be used as the second argument of `isinstance()`. `TypeForm` on the other hand is typically introspected by the user in some way, is not necessarily directly instantiable, and is not necessary directly usable in a regular `isinstance()` check. It would be possible to widen `Type` to include the additional values allowed by `TypeForm` but it would reduce clarity about the user's intentions when working with a `Type`. Different concepts and usage patterns; different spellings. [designed]: https://mail.python.org/archives/list/typing-sig@python.org/message/D5FHORQV... Restricting TypeForm to "universal" forms only ---------------------------------------------- Originally it was considered important to restrict TypeForm such that its values only included forms that could be used as a local variable type, a parameter type, *and* as a parameter type. For example that would rule out `NoReturn` as a value because it can only be used as a return type and not as a local variable type. Functions taking TypeForms as parameters that only wish to accept "universal" forms (that can be used in all 3 of the aforementioned positions) can write code that raises TypeError or a similar exception at runtime when provided a non-universal form. Standardized type inference rules --------------------------------- This PEP intentionally does not specify any rules for implicitly inferring a `TypeForm` type for an expression. No other type-related PEPs prescribe inference rules, and existing type checkers vary in their behavior. So it would be inconsistent to dictate an inference rule for TypeForm. Individual type checkers may define their own inference rules if they wish. Open Issues =========== [Any points that are still being decided/discussed.] References ========== [A collection of URLs used as references through the PEP.] Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
On 2/14/21 10:06 AM, David Foster wrote:
The [TypeForm] PEP really seems to be coming together now. Next big steps I believe are: * Find a sponsor * RST-ize the PEP and submit a PR to the PEPs repo
I'm ready to RST-ize the TypeForm PEP now. Any problem doing that before I find a sponsor? In particular, will the PEPs repo accept a PR with a Sponsor of "TODO"? -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
You can create a PR, and get some review of the text that way, but I won’t merge it. We’re short on potential sponsors here... On Tue, Feb 23, 2021 at 23:13 David Foster <davidfstr@gmail.com> wrote:
On 2/14/21 10:06 AM, David Foster wrote:
The [TypeForm] PEP really seems to be coming together now. Next big steps I believe are: * Find a sponsor * RST-ize the PEP and submit a PR to the PEPs repo
I'm ready to RST-ize the TypeForm PEP now.
Any problem doing that before I find a sponsor? In particular, will the PEPs repo accept a PR with a Sponsor of "TODO"?
-- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
-- --Guido (mobile)
I'm deferring further work on TypeForm for 1-2 months so that I can focus on getting PEP 655 (Required[] and NotRequired[] for TypedDict) over the line and into Python 3.10a7 (due 2021-04-05). -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
participants (5)
-
David Foster -
Eric Traut -
Guido van Rossum -
Stefan van der Walt -
Tin Tvrtković