PEP 692: Using TypedDict for more precise **kwargs typing
Adding support for more precise **kwargs (consisting of different types) typing using TypedDict has been a very desired feature in python community [1]. The feature proposal has been discussed both on the mailing list and at a typing meetup and received positive feedback. I've submitted PEP 692 that aims to add support for more precise **kwargs typing. The PR is available at https://github.com/python/peps/pull/2620 And the preview at https://pep-previews--2620.org.readthedocs.build/pep-0692/ - **kwargs can consist of different types - A new syntax `def foo(**kwargs: **Movie): ...` that enables the feature is introduced Please let me know what you think about it. Franek [1] https://github.com/python/mypy/issues/4441
Thanks for pushing this effort forward, I think it will be quite valuable! I don't have much special to contribute regarding the PEP text beyond what others are discussing, but I did have one clarification question. The draft mentions that TypedDicts are currently not allowed to be assigned values with extra keys, although there is a separate discussion happening about possibly extending that. In terms of how that applies to used TD for **kwargs, could you clarify whether in this example class MovieBase(TypedDict): name: str class Movie(MovieBase): year: int movie: Movie = {name:"Holy Grail", year:1975} def takes_moviebase(**kwargs: **MovieBase): print(kwargs["name"]) takes_moviebase(**movie) is the function call allowed, or would checkers consider this an error?
I think this should be allowed - Movie also requires all of the MovieBase's required keys so this should be fine.
Allowing this is inconsistent with point 3 in the "Assignment" section of the PEP. It states that this should be allowed: def takes_name(name: str): ... def takes_moviebase(**kwargs: **MovieBase): ... takes_moviebase = takes_name But then takes_moviebase(**movie) would fail at runtime.
Typecheckers would also have to reject def takes_moviebase(**kwargs: **MovieBase): takes_name(**kwargs) since that would also make takes_moviebase(**movie) fail
This is tricky, it may be worth thinking hard about whether some of the other semantics need adjusting. It is already a requirement of TypedDict semantics that a value of type `Movie` must be considered compatible with type `MovieBase`. As a result, we have to allow splatting a `Movie` because this can't be fixed in PEP 692 ... even if we didn't allow *directly* splatting, the fact that upstream code would be allowed to upcast `Movie` to `MovieBase` would mean the extra arguments can be there. Only a final TypedDict can be assumed to have no extra keys. As a result, henbruas is correct that if we're going to allow using a TypedDict to deal with the type of `**kwargs` in a way where we validate bottom-of-the-stack calls against functions that fail with extra keyword argumetns (like `takes_name`), then we should probably insist that those typed dicts must be final. Just to make sure I'm communicating why we have to assume there are extra keys, it is always legal to write this code: ``` has_year_field: Movie = {"name":"Holy Grail", "year":1975} still_has_year_field: MovieBase = d ``` and as a result, any type checking rules we write need to make sense when `MovieDict` includes an extra field like `year`. This can be confusing because at the moment this code won't typecheck: ``` has_year_field_directly_assigned: MovieBase = {"name":"Holy Grail", "year":1975} ``` This creates a situation in which it's easy to mistakenly believe values of MovieBase can't have extra fields, but they can.
Right, this makes sense, thank you henbruas and Steven for pointing that out, it slipped my mind that in that case it is possible to have additional keys would fail at runtime. It makes sense that in those cases type checking could possibly pass only if the TypedDict is final, unfortunately mypy does not currently support that [1,2]. When it comes to assignment, I'd be fine with simply disallowing assignments as described in #3. It seems like this kind of thing would very rarely be needed in real life? As for the second use case, type safety could be maintained by dereferencing the value before calling the function: def takes_moviebase(**kwargs: **MovieBase): name = kwargs['name'] takes_name(name) But then this: def takes_moviebase(**kwargs: **MovieBase): takes_name(**kwargs) would have to be rejected for reasons that you've very well explained. On the other hand it feels like this: def takes_moviebase(**kwargs: **MovieBase): print(kwargs["name"]) takes_moviebase(**movie) should be allowed (there's no runtime error unless kwargs are passed to something that will fail if it encounters additional keys). So seems like for the second use case an error should be emitted if inside the function kwargs are passed as unpacked arguments to another function that specifies only keyword arguments and doesn't have any **kwargs. Now that I think about it, passing kwargs as a "normal" (not unpacked) argument could also fail at runtime: def bar(d: MovieBase): takes_name(**d) def foo(**kwargs: MovieBase): bar(kwargs) It might make sense to allow to pass **kwargs only to function that can accept **kwargs, but will need to give it more thought. [1] https://github.com/python/mypy/blob/master/test-data/unit/check-typeddict.te... [2] https://github.com/python/mypy/issues/7981
Has there been any discussion of symmetrical syntax for `*args` unpacking, eg. with a tuple annotation? Admittedly there are probably fewer use cases, but it feels odd to have one and not the other.
You already can using *tuple[Arg, Types, Here] On Mon, 18 Jul 2022 at 16:27, Artemis <artemisdev21+python-ideas@gmail.com> wrote:
Has there been any discussion of symmetrical syntax for `*args` unpacking, eg. with a tuple annotation? Admittedly there are probably fewer use cases, but it feels odd to have one and not the other. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gobot1234yt@gmail.com
Wouldn't it make more sense to use a Protocol class for this? It's pretty much what TS does, Protocol classes being the closest to its interfaces. It wouldn't limit **kwargs to whatever's declared in the type. Rafael
Could you explain a bit more what you mean by "It wouldn't limit **kwargs to whatever's declared in the type"? That is an interesting idea, but I don't think it is realistic. What should be the type of the kwargs parameter inside the function body in that case? Kwargs are a built in dict object. That means that inside the function body kwargs are used like a dict, so they need to support indexing (like kwargs["name"]) and all the methods that dict supports. TypedDict already takes care of that.
Hi all, I like the idea of the pep and the pep itself. But, from what I've observed, proposing syntax changes can sometimes create barriers to pep acceptance, which would be a shame, since I've wanted better typing of **kwargs forever. So, I wanted to throw out an alternative idea to requiring a syntax change (and effectively new operator/dunder method). I don't think I saw this mentioned; apologies if it has been already. Anyways, the idea: a future import (or equiv) that changes the meaning of what a **kwargs type annotation means. e.g. ``` from __future__ import exact_kwargs_annotations def foo(**kwargs: TypedDict[...]): ... def bar(**kwargs: dict[str, int]): ... # or alternatively, a decorator: from typing import exact_kwargs @exact_kwargs def foo(**kwargs: TypedDict): ... def bar(**kwargs: dict[str, int]): ... ``` (the decorator could also be something trailed in e.g. typing_extensions for early feedback and backwards compatibility) On Sat, Aug 6, 2022 at 11:06 AM Franek Magiera <framagie@gmail.com> wrote:
Could you explain a bit more what you mean by "It wouldn't limit **kwargs to whatever's declared in the type"?
That is an interesting idea, but I don't think it is realistic. What should be the type of the kwargs parameter inside the function body in that case? Kwargs are a built in dict object. That means that inside the function body kwargs are used like a dict, so they need to support indexing (like kwargs["name"]) and all the methods that dict supports. TypedDict already takes care of that. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: richardlev@gmail.com
Thanks for your comment! In the case you're mentioning I think we could use Unpack as proposed in the PEP.
Well, normal classes are just a nice syntax for a dictionary. If a Protocol can describe class without limiting its members or requiring explicit base declaration, why it shouldn't be able to describe a dictionary? Perhaps there could be TypedDictProtocol ABC that would behave like amalgam of TypedDict and Protocol? PS. Sorry for the delay. It's my first post on a python mailing list and it turned out the mail delivery was disabled.
What would be the added value of using Protocol instead of TypedDict? I can see that proposal working if we introduce TypedDictProtocol ABC, but that's one more thing that would have to be added and maintained, also I don't see an obvious way of how such an ABC should be implemented (but it feels like a lot of what TypedDict provides would have to be duplicated). In addition, I think Protocols are much more powerful than TypedDicts, but those features that make Protocols more powerful also make it a bad fit for this purpose and would require much more effort to implement and maintain (basically we would have to ban certain functionalities that Protocols provide, for example there shouldn't be any methods defined). On the other hand, TypedDicts naturally correspond to kwargs which are actual dictionaries. TypedDicts provide support for Required/NotRequired fields, which AFAIK Protocols don't. TypedDicts have a clear specification and if there are ever any changes to the TypedDict's specification it should be easy to incorporate them in the context of PEP 692 (like for example not limiting kwargs to what is declared in the type, if TypedDicts ever support additional keys). I think the proposal could work with Protocols instead of TypedDicts, but I am not convinced a Protocols based implementation would be more valuable than a TypedDict based one. At the same time I feel a Protocols based proposal would be more complex and more difficult to implement.
I agree with Franek. A TypedDict is effectively a protocol (a structural type) that describes a typed dictionary. It is theoretically possible to manually define a protocol class that is the equivalent of a TypedDict, but it would involve so much verbose boilerplate that no one would ever do it. Consider the following: ```python class Foo(TypedDict): x: Required[int] y: NotRequired[str] ``` This is effectively equivalent to the following protocol: ```python class FooProto(Protocol): def __init__(self, x: int, y: str): ... def copy(self: Self) -> Self: ... def update(self: _T, __m: _T) -> None: ... def items(self) -> ItemsView[str, object]: ... def keys(self) -> KeysView[str]: ... def values(self) -> ValuesView[object]: ... @overload def get(self, k: Literal["x"]) -> int: ... @overload def get(self, k: Literal["x"], default: _T) -> int | _T: ... @overload def get(self, k: Literal["y"]) -> str: ... @overload def get(self, k: Literal["y"], default: _T) -> str | _T: ... @overload def setdefault(self, k: Literal["x"], default: int) -> int: ... @overload def setdefault(self, k: Literal["y"], default: str) -> str: ... @overload def pop(self, k: Literal["y"]) -> int: ... @overload def pop(self, k: Literal["y"], default: _T) -> int | _T: ... ``` Also, as Franek said, the flexibility of protocols make them unsuitable for `**kwargs` because the runtime type for `kwargs` is a dict, so its type must conform to the interface of a dict. That's what a TypedDict is designed to do. -- Eric Traut Contributor to pyright & pylance Microsoft
Thank you for the clarification. I was sure that TypedDict doesn't allow extra keys, but since that's not the case, it makes perfect sense to use it
participants (10)
-
Artemis
-
asafspades@gmail.com
-
Eric Traut
-
Franek Magiera
-
Gobot1234
-
henbruas@gmail.com
-
r.krupinski@gmail.com
-
Rafael Krupinski
-
Richard Levasseur
-
Steven Troxler