PEP 589: no way to tell if a key is optional at runtime?
Hi All I'm a core developer of Hypothesis, and largely responsible for our `from_type()` strategy. As the name suggests, it takes a type and returns an object that can generate instances of it, including for things in the `typing` module that aren't really types at runtime. So far, so good! For obvious reasons, I'm looking to support TypedDict too, but I've run into a snag: I can't see any way to distinguish between optional and required keys at runtime! It's not actually *incorrect* to treat all keys as required, but we'll miss test cases that could have exposed real bugs. Consider the following cases: ```python class A(typing.TypedDict): a: int # a is required class B(A, total=False): b: bool # a is required, b is optional class C(B): c: str # a is required, b is optional, c is required again ``` The PEP is clear about the semantics, and this is obvious enough when reading the code. At runtime the __annotations__ attribute of each class gives us the set of allowed keys and the type of each corresponding value, but we have a problem: - C has __total__==True, but b is not actually required. - B has __total__==False, but a *is* required. - I can't see any way to get the parent classes of a TypedDict class! The _TypedDictMeta metaclass updates the attributes, but leaves no record of the parent type - at runtime A, B, and C all appear to inherit directly from dict. Is there a way to directly determine which keys are required and which are optional at runtime? If not, is there a way to determine the parent class(es) of a TypedDict? If not, can we please add one - preferably the former - for 3.8.1? If there's an agreed design I'd be happy to implement it - as a concrete suggestion, I'd add two frozensets, __required_keys__ and __optional_keys__. Obvious, useful, orthogonal to current attributes, and easy to fall back to treating all keys as required. Best, Zac Hatfield-Dodds
Hi Zac, We have received similar requests in the past. And I think this is something worth implementing. The main use-case for this I see is some kind of runtime validation of data at the "boundary" of statically typed code (for example a JSON coming from the network). I like the idea of two frozen sets, __required_keys__ and __optional_keys__, but I don't think it is possible to add this to Python 3.8.1, since this is technically a feature. We can merge this in CPython master (so it will be in 3.9), and update the backports in typing on PyPI as well as typing_extensions, so people can start using this soon. I would propose that you open an issue on https://bugs.python.org and make a PR against CPython master, and then we can discuss if we can make an exception an add this to 3.8.1 as a "bug-fix". -- Ivan On Thu, 14 Nov 2019 at 22:25, Zac Hatfield-Dodds < zac.hatfield.dodds@gmail.com> wrote:
Hi All
I'm a core developer of Hypothesis, and largely responsible for our `from_type()` strategy. As the name suggests, it takes a type and returns an object that can generate instances of it, including for things in the `typing` module that aren't really types at runtime. So far, so good!
For obvious reasons, I'm looking to support TypedDict too, but I've run into a snag: I can't see any way to distinguish between optional and required keys at runtime! It's not actually *incorrect* to treat all keys as required, but we'll miss test cases that could have exposed real bugs.
Consider the following cases:
```python class A(typing.TypedDict): a: int # a is required
class B(A, total=False): b: bool # a is required, b is optional
class C(B): c: str # a is required, b is optional, c is required again ```
The PEP is clear about the semantics, and this is obvious enough when reading the code. At runtime the __annotations__ attribute of each class gives us the set of allowed keys and the type of each corresponding value, but we have a problem:
- C has __total__==True, but b is not actually required. - B has __total__==False, but a *is* required. - I can't see any way to get the parent classes of a TypedDict class!
The _TypedDictMeta metaclass updates the attributes, but leaves no record of the parent type - at runtime A, B, and C all appear to inherit directly from dict.
Is there a way to directly determine which keys are required and which are optional at runtime? If not, is there a way to determine the parent class(es) of a TypedDict? If not, can we please add one - preferably the former - for 3.8.1?
If there's an agreed design I'd be happy to implement it - as a concrete suggestion, I'd add two frozensets, __required_keys__ and __optional_keys__. Obvious, useful, orthogonal to current attributes, and easy to fall back to treating all keys as required.
Best, Zac Hatfield-Dodds _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
Sounds good - I've opened https://bugs.python.org/issue38834 and will have a PR up later this week :-) Zac On Mon, 18 Nov 2019 at 06:56, Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Hi Zac,
We have received similar requests in the past. And I think this is something worth implementing. The main use-case for this I see is some kind of runtime validation of data at the "boundary" of statically typed code (for example a JSON coming from the network).
I like the idea of two frozen sets, __required_keys__ and __optional_keys__, but I don't think it is possible to add this to Python 3.8.1, since this is technically a feature. We can merge this in CPython master (so it will be in 3.9), and update the backports in typing on PyPI as well as typing_extensions, so people can start using this soon.
I would propose that you open an issue on https://bugs.python.org and make a PR against CPython master, and then we can discuss if we can make an exception an add this to 3.8.1 as a "bug-fix".
-- Ivan
On Thu, 14 Nov 2019 at 22:25, Zac Hatfield-Dodds < zac.hatfield.dodds@gmail.com> wrote:
Hi All
I'm a core developer of Hypothesis, and largely responsible for our `from_type()` strategy. As the name suggests, it takes a type and returns an object that can generate instances of it, including for things in the `typing` module that aren't really types at runtime. So far, so good!
For obvious reasons, I'm looking to support TypedDict too, but I've run into a snag: I can't see any way to distinguish between optional and required keys at runtime! It's not actually *incorrect* to treat all keys as required, but we'll miss test cases that could have exposed real bugs.
Consider the following cases:
```python class A(typing.TypedDict): a: int # a is required
class B(A, total=False): b: bool # a is required, b is optional
class C(B): c: str # a is required, b is optional, c is required again ```
The PEP is clear about the semantics, and this is obvious enough when reading the code. At runtime the __annotations__ attribute of each class gives us the set of allowed keys and the type of each corresponding value, but we have a problem:
- C has __total__==True, but b is not actually required. - B has __total__==False, but a *is* required. - I can't see any way to get the parent classes of a TypedDict class!
The _TypedDictMeta metaclass updates the attributes, but leaves no record of the parent type - at runtime A, B, and C all appear to inherit directly from dict.
Is there a way to directly determine which keys are required and which are optional at runtime? If not, is there a way to determine the parent class(es) of a TypedDict? If not, can we please add one - preferably the former - for 3.8.1?
If there's an agreed design I'd be happy to implement it - as a concrete suggestion, I'd add two frozensets, __required_keys__ and __optional_keys__. Obvious, useful, orthogonal to current attributes, and easy to fall back to treating all keys as required.
Best, Zac Hatfield-Dodds _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/
participants (3)
-
Ivan Levkivskyi -
Zac Hatfield Dodds -
Zac Hatfield-Dodds