Spelling the type of a runtime type annotation object
I am excited about the idea of being able to match JSON data received by Python web applications at runtime using type annotations to describe the expected shape of the data. JSON data is typically in the form of structured dictionaries (TypedDicts) containing Lists and primitives (str, float, bool, None). Consider an example web application that wants to provide a service to draw shapes, perhaps on a connected physical billboard. The service has a '/draw_shape' endpoint which takes a JSON object (a Shape TypedDict) describing a shape to draw: from bottle import HTTPResponse, request, route from typing import Literal, TypedDict, Union class Point2D(TypedDict): x: float y: float class Circle(TypedDict): type: Literal['circle'] center: Point2D # has a nested TypedDict! radius: float class Rect(TypedDict): type: Literal['rect'] x: float y: float width: float height: float Shape = Union[Circle, Rect] # a Tagged Union / Discriminated Union @route('/draw_shape') def draw_shape_endpoint() -> None: request_json = request.json # type: object if isinstance(request_json, Shape): draw_shape(request_json) # type is narrowed to Shape else: return HTTPResponse(status=400) # Bad Request Now, it's not actually possible in draw_shape_endpoint() to use isinstance() to perform a runtime type check against a complex type annotation object such as a Shape (which is a Union[Circle, Rect]), a Circle/Rect (which is a TypedDict subtype), or a Literal[...]. Guido mentioned in thread [1] that it would be problematic to extend/alter isinstance() to match complex type annotations for a number of reasons. Notably: * the recursive checks would make isinstance() a speed trap in certain cases, and * the subtype rules used by isinstance() aren't convenient for matching JSON when working with {int, float, bool} types. - In fact the subtype rules used by isinstance() aren't even consistent with the subtype rules used by typecheckers conforming to PEP 484 ("Type Hints") for such types, and the rules currently used by typecheckers are generally more friendly to matching JSON. So I wrote my own function trycast() [2] that can do the kind of matching that I want. With it, I can write: @route('/draw_shape') def draw_shape_endpoint() -> None: if (shape := trycast(Shape, request.json)) is not None: draw_shape(shape) # type is narrowed to Shape else: return HTTPResponse(status=400) # Bad Request The idea behind trycast(...) is that it is similar to the existing cast(...) function, but actually performs a runtime check to see whether its value argument actually conforms to its type annotation argument, using typechecker subtype rules from PEP 484 ("Type Hints"). trycast() works beautifully at runtime. There's just one problem: I can't actually spell the types in the function signature for trycast(...)! You might think it would be: T = TypeVar('T') def trycast(type: Type[T], value: object) -> Optional[T]: ... However typecheckers won't currently allow me to pass a runtime value like `Union[int, str]` as an argument for parameter `type` (which requires a `Type[T]`) because `isinstance(Union[int, str], type) == False`! (Jukka explains this in thread [3].) So it seems I need to be able to spell "an object that can be used where a type annotation is expected", but there's currently no syntax for that. **So I'd like to propose that Python typecheckers make it possible to spell "an object that can be used where a type annotation is expected".** Having the existence of such a spelling would make it possible to type-annotate several new kinds of valuable metaprogramming functions. Consider the following uses: * A function that can check whether an object conforms to the shape described by a type annotation. This is similar to the idea of an isinstance check, but includes cases that isinstance cannot handle, like isinstance(x, Union[int, str]) or isinstance(x, SomeTypedDict). def conforms_to(typelike: TypeAnnotation, value: object) -> bool: ... ^^^^^^^^^^^^^^ tentative spelling * A function that can parse a value and return it if it happens to conform to the shape described by a type annotation, or None if it doesn't: def trycast(typelike: TypeAnnotation[T], value: object) -> Optional[T]: ... Notice that Optional here is accepting a T member that is bound by TypeAnnotation[T]. * A function that can parse a value and return it if it happens to conform to the shape described by a type annotation, or return a custom sentinel if it does not: def trycast(typelike: TypeAnnotation[T], value: object, failure: F) -> Union[T, F] Notice that the Union here is accepting a T member that is bound by TypeAnnotation[T] in addition to the bound created by the F TypeVar. There are two main approaches I can see for adding the capability to spell "an object that can be used where a type annotation is expected": 1. Widen the interpretation of Type[T] to cover not just cases where `isinstance(T, type)` but also allow Type[T] to match any type annotation object (like `Union[str, int]`). 2. Introduce an alternate distinct spelling, such as `TypeAnnotation[T]` (which I use in the examples above), that is in addition to the existing spelling for `Type[T]`. - Logically `TypeAnnotation[T]` would be a supertype of `Type[T]` since all instances of `type` are (I believe) valid type annotations in their own right. **Reactions to each of these approaches? Comments on tradeoffs?** If folks think the concept of this new spelling is overall a good idea and I can get some clarity around the above questions, I'd be willing to make a stab at implementing an initial version in mypy. (If using approach 2 where a new distinct spelling is added, I'd probably opt to put it in the mypy_extensions package for the time being.) [1]: https://mail.python.org/archives/list/python-ideas@python.org/message/AZ2JBL... [2]: https://github.com/davidfstr/trycast [3]: https://github.com/python/mypy/issues/9003#issuecomment-646690595 -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
Hello, trycast() works beautifully at runtime. There's just one problem:
I can't actually spell the types in the function signature for trycast(...)! You might think it would be:
T = TypeVar('T')
def trycast(type: Type[T], value: object) -> Optional[T]: ...
I've recently had to relax a type annotation in cattrs ( https://github.com/Tinche/cattrs/issues/105) due to a bug report for this exact problem, so I'm interested in a solution as well.
Slightly offtopic since we're working on similar problems: what's the benefit of using typed dictionaries instead of classes? To me structured data in Python automatically means a class.
I think this is a perfect use case for user-defined type guards, which has been discussed in a different typing-sig thread. The draft PEP can be found [here](https://github.com/erictraut/peps/blob/master/pep-9999.rst). This is implemented in preview form within [Pyright](https://github.com/microsoft/pyright). Here's how you'd use it: ```python def is_typed_dict(raw_dict: Dict[str, Any], typed_dict_class: Type[_T]) -> TypeGuard[_T]: # Perform check here and return bool value indicating whether raw_dict # conforms to the type _T return trycast(typed_dict_class, raw_dict) is not None ```
On 11/27/20 6:17 PM, Tin Tvrtković wrote:
[...] what's the benefit of using typed dictionaries instead of classes? To me structured data in Python automatically means a class.
Typed dictionaries are the natural form that JSON data comes in over the wire. They can be trivially serialized and deserialized (but not parsed) without any additional logic. For applications that use a lot of JSON data - such as web applications - using typed dictionaries is very convenient. -- David Foster | Seattle, WA, USA
**So I'd like to propose that Python typecheckers make it possible to spell "an object that can be used where a type annotation is expected".**
If folks think the concept of this new spelling is overall a good idea and I can get some clarity around the above questions, I'd be willing to make a stab at implementing an initial version in mypy.
I've started a thread on the mypy issue tracker, for those interested in following my efforts to get this kind of syntax supported in mypy: -> https://github.com/python/mypy/issues/9773 -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy
participants (3)
-
David Foster
-
Eric Traut
-
Tin Tvrtković