TypedDict with dict comprehension
I see that both mypy and pyright both reject TypedDict being constructed with a dict comprehension, which from a static typing perspective is understandable. I do question if it's the correct behavior though. I have multiple dynamic typing cases where I construct a TypedDict using a dict comprehension a la: TypedDict("TD", {n: t for n, t in ...}). I propose static type checkers interpret this as equivalent to dict[Any, Any], and not flag it as an error (and not show red underlines in various IDEs), as there is nothing I can see in the documentation to prohibit it and works at runtime as expected.
Could you provide a more complete (self-contained) code sample? When you say "construct a TypedDict", do you mean "define a new TypedDict class"? Or are you talking about calling the constructor for an existing TypedDict class? It looks like you mean the former, but it's not clear from your terminology. The purpose of TypedDict is to support static type checking. If you don't care about static type checking, you can simply use `dict`. If you do care about static type checking, then a TypedDict that is the equivalent of dict[Any, Any] seem to be of no value, so I'm not sure why one would want to define a TypedDict class in that manner. Perhaps you could explain your use case in more detail? -Eric -- Eric Traut Contributor to pyright & pylance Microsoft
I hope the purpose of TypedDict is not to exclusively support *static* type checking. It's proven to be very useful in dynamic typing cases as well. Overly-simplistic sample code for demonstration of comprehension:
fields = ("a", "b", "c") TD = TypedDict("TD", {field: str for field in fields} TD.__annotations__ {'a': <class 'str'>, 'b': <class 'str'>, 'c': <class 'str'>} TD.__required_keys__ frozenset({'a', 'c', 'b'})
Real world use cases of dynamically-generated TypedDict: - deriving a typed dictionary from a SQL table schema to support dynamic data validation and interchange - deriving a typed dictionary from machine-readable API (e.g. OpenAPI) document to support dynamic data validation and interchange As demonstrated above, this works very well at runtime, and I can't think of a reason to prohibit the construction of a TypedDict from a comprehension. Static typing tools do complain (as well as IDEs that utilize them) and this seems spurious in light of evidence that it is both useful and fully functional. On Wed, 2022-10-19 at 16:11 +0000, Eric Traut wrote:
Could you provide a more complete (self-contained) code sample? When you say "construct a TypedDict", do you mean "define a new TypedDict class"? Or are you talking about calling the constructor for an existing TypedDict class? It looks like you mean the former, but it's not clear from your terminology.
The purpose of TypedDict is to support static type checking. If you don't care about static type checking, you can simply use `dict`. If you do care about static type checking, then a TypedDict that is the equivalent of dict[Any, Any] seem to be of no value, so I'm not sure why one would want to define a TypedDict class in that manner. Perhaps you could explain your use case in more detail?
-Eric
-- Eric Traut Contributor to pyright & pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
I see. Yeah, you're using TypedDict in a way that the authors of PEP 589 apparently didn't contemplate. First, you're using the "Alternate Syntax" which was provided (according to the PEP) for backward compatibility only. All versions of Python that are supported today do not have a need for this backward compatibility syntax, so I would expect its use to be waning at this point. Second, you're using a dynamic expression for the second argument the the TypedDict alternate syntax, which the PEP calls out as something that static type checkers need not support.
A type checker is only expected to accept a dictionary display expression as the second argument to TypedDict. In particular, a variable that refers to a dictionary object does not need to be supported, to simplify implementation.
For developers who are using TypedDict in accordance with its original intended purpose, I see utility in the errors that static type checkers currently generate in this case. If we were to suppress these errors and default the resulting type to dict[str, Any] in the presence of a dynamic expression, I think some developers would find that behavior to be surprising and non-intuitive because it would effectively render static type checking ineffective for that type. It would also not help with completion suggestions and other coding productivity features provided by language servers. I think I'd want to see some additional signal that your use case is something that is more broadly desirable before adding this support to pyright. -Eric
El mié, 19 oct 2022 a las 10:02, Eric Traut (<eric@traut.com>) escribió:
I see. Yeah, you're using TypedDict in a way that the authors of PEP 589 apparently didn't contemplate.
First, you're using the "Alternate Syntax" which was provided (according to the PEP) for backward compatibility only. All versions of Python that are supported today do not have a need for this backward compatibility syntax, so I would expect its use to be waning at this point.
There are remaining use cases for the alternate syntax: if your keys are not valid identifiers or are Python keywords, you cannot use the class-based syntax. We recently changed the CPython docs to call out these use cases.
Second, you're using a dynamic expression for the second argument the the TypedDict alternate syntax, which the PEP calls out as something that static type checkers need not support.
A type checker is only expected to accept a dictionary display expression as the second argument to TypedDict. In particular, a variable that refers to a dictionary object does not need to be supported, to simplify implementation.
For developers who are using TypedDict in accordance with its original intended purpose, I see utility in the errors that static type checkers currently generate in this case. If we were to suppress these errors and default the resulting type to dict[str, Any] in the presence of a dynamic expression, I think some developers would find that behavior to be surprising and non-intuitive because it would effectively render static type checking ineffective for that type. It would also not help with completion suggestions and other coding productivity features provided by language servers.
I agree that static type checkers should continue to error for dynamically generated TypedDicts. For Paul's use case, I would recommend a workaround like: if TYPE_CHECKING: TD = dict[str, Any] else: TD = TypedDict("TD", {k: v for ...}))
I think I'd want to see some additional signal that your use case is something that is more broadly desirable before adding this support to pyright.
-Eric _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Hi Paul, Is `TypedDict` strictly necessary in your use case? It seems to me that you can achieve the same goal by defining a plain dict where keys are field name strings and values are corresponding classes, e.g. ``` fields = ('a', 'b', 'c') TD = {field: str for field in fields} ``` There's no need to indirectly obtain the same info via `TypedDict.__annotations__`. Other goodies, like `__required_keys__`, can be obtained by creating your own wrapper class around dict with those additional fields. My understanding is that the entire motivation of `TypedDict` is to inform static type checkers what keys are in a dict and what are the corresponding value types. The bottom line here is that if you don't really know these info statically anyway, then going out of the way to construct a `TypedDict` feels a bit redundant. If your proposal is to re-purpose the pre-existing `TypedDict` class in stdlib, to re-interpret it to be the "standard" class that represents the runtime structure for your own use case, then I do see some value of it. Although if we are proposing new changes to the Python stdlib, my expectation is that the bar would be a bit higher than "it's useful for me". Do you have more data points that suggest the feature is actually needed (or could potentially be widely useful) in practice? - Jia
On Wed, 2022-10-19 at 17:16 +0000, Jia Chen via Typing-sig wrote:
Is `TypedDict` strictly necessary in your use case? It seems to me that you can achieve the same goal by defining a plain dict where keys are field name strings and values are corresponding classes, e.g.
Yes, because unlike my overly simplistic example, real use cases derive TypedDict from live SQL schemas and server-generated OpenAPI documents to facilitate runtime type validation and value encoding/decoding during data interchange.
My understanding is that the entire motivation of `TypedDict` is to inform static type checkers what keys are in a dict and what are the corresponding value types. The bottom line here is that if you don't really know these info statically anyway, then going out of the way to construct a `TypedDict` feels a bit redundant.
While it may have been initially motivated by static type checking use cases, it's proven invaluable in dynamic typing cases. I sincerely hope it won't be exclusively for static type checking use cases.
If your proposal is to re-purpose the pre-existing `TypedDict` class in stdlib, to re-interpret it to be the "standard" class that represents the runtime structure for your own use case, then I do see some value of it. Although if we are proposing new changes to the Python stdlib, my expectation is that the bar would be a bit higher than "it's useful for me". Do you have more data points that suggest the feature is actually needed (or could potentially be widely useful) in practice?
TypedDict already works as advertised and as expected. The issue I raised pertained to how static type checkers were interpreting a particular case of constructing TypedDict, which demonstrably works at runtime. Jelle's suggested workaround will serve to suppress static type checking errors.
participants (4)
-
Eric Traut -
Jelle Zijlstra -
Jia Chen -
Paul Bryan