Re: Type-hinting dictionaries for an arbitrary number of arbitrary key/value pairs? Counterpart to PEP 589?
Indeed — we essentially lie to mypy about the method resolution order for list, dict, etc (mypy thinks that list directly inherits from collections.abc.MutableSequence — see the typeshed stub here: https://github.com/python/typeshed/blob/32bc2161a107db20c2ebc85aad31c29730db...), but numeric ABCs are not special-cased by typeshed in the same way as collections ABCs. Fundamentally, mypy has no knowledge about virtual subclassing. Again, I covered this very comprehensively on StackOverflow ;) https://stackoverflow.com/questions/69334475/how-to-hint-at-number-types-i-e... Best, Alex
On Sat, 16 Oct 2021 at 14:07, Alex Waygood <alex.waygood@gmail.com> wrote:
Indeed — we essentially lie to mypy about the method resolution order for list, dict, etc (mypy thinks that list directly inherits from collections.abc.MutableSequence — see the typeshed stub here: https://github.com/python/typeshed/blob/32bc2161a107db20c2ebc85aad31c29730db...), but numeric ABCs are not special-cased by typeshed in the same way as collections ABCs. Fundamentally, mypy has no knowledge about virtual subclassing.
IMO, that's a bug in mypy. Maybe the fix is "too hard", maybe it isn't. But if we added runtime isinstance checks, they would pass, so arguing that the type is wrong or that the variable "needs" to be typed differently is incorrect. I guess my argument here is that flagging an error when you're not 100% sure the code is wrong is a problem - "in the case of ambiguity, refuse the temptation to guess" seems relevant here. Personally, I'd be unwilling to mess around with the typing in a situation like this - I'd be more likely to just remove the types (the type checking equivalent of `#noqa` when you don't agree with what your style checker says). I'd much prefer mypy to miss the odd problem rather than flagging usages that aren't actually errors, precisely because you don't (as far as I know) have a mechanism for telling it you know what you're doing... But this is probably off-topic. I'm not 100% sure what the OP's proposal was, but as far as I can tell it seems to me that "dict[str, Number] is the correct usage for static typing" is the answer, regardless of mypy's ability to process it. Paul
On Sat, Oct 16, 2021 at 02:49:42PM +0100, Paul Moore wrote:
Type annotations are still useful to the human reader, even if the type checker is absent or wrong. I presume that mypy does support some "skip this" directive? If not, it should.
That's what it looks like to me too. -- Steve
On Sat, 16 Oct 2021 at 18:01, Steven D'Aprano <steve@pearwood.info> wrote:
My impression (from the projects I've worked on using typing) was that it didn't. But I checked and I was wrong: `# type: ignore` does exactly that. Paul
Hi all, thanks for all the feedback.
Let me explain the rational. Albeit type annotations in Python are usually viewed as a tool for static code analysis, I really started to like them for run-time checks and testing. I am hammering my code with pytest, hypothesis and typeguard. It is this very combination that allows me to find the kind of odd cases that I am looking for. The cool thing is that, once the code is tested, I can turn the run-time checks (i.e. typeguard) off simply by setting `PYTHONOPTIMIZE` to `1` or by running CPython with the `-o` flag for production use. It kills the `assert` statements *and* the run-time type checks without touching the code. Now, what PEP 589 *theoretically* allows is something as follows: ```python from typing import TypedDict from typeguard import typechecked @typechecked class Movie(TypedDict): name: str year: int a = Movie(name = 'a', year = 1984) # ok a['height'] = 2.0 # fail b = Movie(something = 1.0) # fail ``` I am aware that typeguard does not support this at the moment, but based on its design, it could be implemented (I think). A simple decorator on top of a class with type annotations - done. Every instance of `Movie` could be automatically type-checked. For this approach to work, it is key that `Movie` is a class (which can be decorated). In contrast, something like `dict[str, Number]` is perfectly fine for static type checking and/or if there is type interference. Consider the following example: ```python Data = dict[str, Number] c: Data = {} c[3.0] = b'foo' # how to catch this at runtime? ``` In this case, `c` is just a "regular" instance of `dict`. I am not sure how a run-time checker could catch any subsequent errors as shown above (unless the dict is passed into or returned from a function/method). The type checker would need to dig into the `dict` class itself and/or deep into the interpreter. Correct me if I am wrong here - I'd be happy to learn something new. Now my potentially stupid idea is to have a counterpart to PEP 589's `TypedDict` classes which allows to subclass and decorate the result, e.g. something elegant like this: ```python @typechecked class Data(YetAnotherTypedDictThing): keys: str values: Number ``` At the end of the day, a variation of Steven's and Alex's solutions should do for me for the time being: ```python from numbers import Number from collections import UserDict @typechecked class Data(UserDict[str, Number]): def __setitem__(self, key: str, value: Number): super().__setitem__(key, value) ``` It is ok, it works, but it still looks and feels a little awkward: (1) I need to override a dunder method. (2) I need to subclass from something that already wraps `dict`. (3) I need to specify the types in two separate places (redundancy) if I am going for a generic solution - unless I am introducing type variables. Last but not least, typeguard is looking for a new maintainer at the moment, yes. I can not stress enough how useful this tool is for testing. Maybe it is of interest and relevance to someone around here. Thanks, Sebastian
On Sat, 16 Oct 2021 at 14:07, Alex Waygood <alex.waygood@gmail.com> wrote:
Indeed — we essentially lie to mypy about the method resolution order for list, dict, etc (mypy thinks that list directly inherits from collections.abc.MutableSequence — see the typeshed stub here: https://github.com/python/typeshed/blob/32bc2161a107db20c2ebc85aad31c29730db...), but numeric ABCs are not special-cased by typeshed in the same way as collections ABCs. Fundamentally, mypy has no knowledge about virtual subclassing.
IMO, that's a bug in mypy. Maybe the fix is "too hard", maybe it isn't. But if we added runtime isinstance checks, they would pass, so arguing that the type is wrong or that the variable "needs" to be typed differently is incorrect. I guess my argument here is that flagging an error when you're not 100% sure the code is wrong is a problem - "in the case of ambiguity, refuse the temptation to guess" seems relevant here. Personally, I'd be unwilling to mess around with the typing in a situation like this - I'd be more likely to just remove the types (the type checking equivalent of `#noqa` when you don't agree with what your style checker says). I'd much prefer mypy to miss the odd problem rather than flagging usages that aren't actually errors, precisely because you don't (as far as I know) have a mechanism for telling it you know what you're doing... But this is probably off-topic. I'm not 100% sure what the OP's proposal was, but as far as I can tell it seems to me that "dict[str, Number] is the correct usage for static typing" is the answer, regardless of mypy's ability to process it. Paul
On Sat, Oct 16, 2021 at 02:49:42PM +0100, Paul Moore wrote:
Type annotations are still useful to the human reader, even if the type checker is absent or wrong. I presume that mypy does support some "skip this" directive? If not, it should.
That's what it looks like to me too. -- Steve
On Sat, 16 Oct 2021 at 18:01, Steven D'Aprano <steve@pearwood.info> wrote:
My impression (from the projects I've worked on using typing) was that it didn't. But I checked and I was wrong: `# type: ignore` does exactly that. Paul
Hi all, thanks for all the feedback.
Let me explain the rational. Albeit type annotations in Python are usually viewed as a tool for static code analysis, I really started to like them for run-time checks and testing. I am hammering my code with pytest, hypothesis and typeguard. It is this very combination that allows me to find the kind of odd cases that I am looking for. The cool thing is that, once the code is tested, I can turn the run-time checks (i.e. typeguard) off simply by setting `PYTHONOPTIMIZE` to `1` or by running CPython with the `-o` flag for production use. It kills the `assert` statements *and* the run-time type checks without touching the code. Now, what PEP 589 *theoretically* allows is something as follows: ```python from typing import TypedDict from typeguard import typechecked @typechecked class Movie(TypedDict): name: str year: int a = Movie(name = 'a', year = 1984) # ok a['height'] = 2.0 # fail b = Movie(something = 1.0) # fail ``` I am aware that typeguard does not support this at the moment, but based on its design, it could be implemented (I think). A simple decorator on top of a class with type annotations - done. Every instance of `Movie` could be automatically type-checked. For this approach to work, it is key that `Movie` is a class (which can be decorated). In contrast, something like `dict[str, Number]` is perfectly fine for static type checking and/or if there is type interference. Consider the following example: ```python Data = dict[str, Number] c: Data = {} c[3.0] = b'foo' # how to catch this at runtime? ``` In this case, `c` is just a "regular" instance of `dict`. I am not sure how a run-time checker could catch any subsequent errors as shown above (unless the dict is passed into or returned from a function/method). The type checker would need to dig into the `dict` class itself and/or deep into the interpreter. Correct me if I am wrong here - I'd be happy to learn something new. Now my potentially stupid idea is to have a counterpart to PEP 589's `TypedDict` classes which allows to subclass and decorate the result, e.g. something elegant like this: ```python @typechecked class Data(YetAnotherTypedDictThing): keys: str values: Number ``` At the end of the day, a variation of Steven's and Alex's solutions should do for me for the time being: ```python from numbers import Number from collections import UserDict @typechecked class Data(UserDict[str, Number]): def __setitem__(self, key: str, value: Number): super().__setitem__(key, value) ``` It is ok, it works, but it still looks and feels a little awkward: (1) I need to override a dunder method. (2) I need to subclass from something that already wraps `dict`. (3) I need to specify the types in two separate places (redundancy) if I am going for a generic solution - unless I am introducing type variables. Last but not least, typeguard is looking for a new maintainer at the moment, yes. I can not stress enough how useful this tool is for testing. Maybe it is of interest and relevance to someone around here. Thanks, Sebastian
participants (4)
-
Alex Waygood -
Paul Moore -
Sebastian M. Ernst -
Steven D'Aprano