Type-hinting dictionaries for an arbitrary number of arbitrary key/value pairs? Counterpart to PEP 589?
Hi all, disclaimer: I have no idea on potential syntax or if it is applicable to a wide audience or if there is already a good solution to this. It is more like a "gap" in the type hint spec that I ran across in a project. In function/method signatures, I can hint at dictionaries for example as follows: ```python from numbers import Number from typing import Dict from typeguard import typechecked Data = Dict[str, Number] @typechecked def foo(bar: Data): print(bar) ``` Yes, this is using run-time checks (typeguard), which works just fine. Only strings as keys and Number objects as values are going through. (I was told that MyPy does not get this at the moment.) The issue is that `bar` itself still allows "everything" to go in (and out): ```python @typechecked def foo2(bar: Data): bar[1.0] = b'should not be allowed' ``` PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs. For reference, related question on SO: https://stackoverflow.com/q/69555006/1672565 Best regards, Sebastian
Mypy correctly rejects this: ❯ type .\t.py from numbers import Number from typing import Dict Data = Dict[str, Number] def foo(bar: Data): print(bar) bar[1.0] = b'hello' PS 17:48 00:00.008 C:\Work\Scratch\foo ❯ mypy .\t.py t.py:9: error: Invalid index type "float" for "Dict[str, Number]"; expected type "str" t.py:9: error: Incompatible types in assignment (expression has type "bytes", target has type "Number") Found 2 errors in 1 file (checked 1 source file) If typeguard doesn't, maybe you need to raise that as a bug against that project? Paul On Fri, 15 Oct 2021 at 17:20, Sebastian M. Ernst <ernst@pleiszenburg.de> wrote:
Hi all,
disclaimer: I have no idea on potential syntax or if it is applicable to a wide audience or if there is already a good solution to this. It is more like a "gap" in the type hint spec that I ran across in a project.
In function/method signatures, I can hint at dictionaries for example as follows:
```python from numbers import Number from typing import Dict
from typeguard import typechecked
Data = Dict[str, Number]
@typechecked def foo(bar: Data): print(bar) ```
Yes, this is using run-time checks (typeguard), which works just fine. Only strings as keys and Number objects as values are going through. (I was told that MyPy does not get this at the moment.)
The issue is that `bar` itself still allows "everything" to go in (and out):
```python @typechecked def foo2(bar: Data): bar[1.0] = b'should not be allowed' ```
PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs.
For reference, related question on SO: https://stackoverflow.com/q/69555006/1672565
Best regards, Sebastian _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DY5DPG... Code of Conduct: http://python.org/psf/codeofconduct/
Hi Paul, all, Am 15.10.21 um 18:49 schrieb Paul Moore:
Mypy correctly rejects this: [...]
interesting. Thanks for trying.
If typeguard doesn't, maybe you need to raise that as a bug against that project?
This is kind of contradicting the design of typeguard. It works on a call level checking arguments and return values for the most part. Typeguard does not provide any form of type inference like mypy does. From this perspective, typeguard would need to guard `dict.__init__`, `dict.__setitem__`, `dict.update` and friends directly, essentially requiring a subclass of `dict` of some sort. However, typeguard does have a special function to perform type checks on objects, e.g. `check_type('variablename', [1234], List[int])` Ignoring typeguard, my suggestion still stands, although slightly changed: Annotating a dictionary as described earlier in such a way that type inference is not required OR in such a way that run-time checkers have a chance to work more easily - if this makes any sense at all? Best regards, Sebastian
Just a pointer related to this, typeguard is abandoned. kttps://github.com/agronholm/typeguard/issues/198
On Fri, 15 Oct 2021 at 18:07, Sebastian M. Ernst <ernst@pleiszenburg.de> wrote:
Ignoring typeguard, my suggestion still stands, although slightly changed: Annotating a dictionary as described earlier in such a way that type inference is not required OR in such a way that run-time checkers have a chance to work more easily - if this makes any sense at all?
I'm not sure what your suggestion actually is, though. "I am looking for an arbitrary number of typed keys/value pairs." - isn't that Mapping[str, float]" (or something similar)? I see no value in making the programmer do more work so that type checkers can do less type inference. Far from it, I'd like to have as few annotations as possible, and have the type checkers do *more* work for me. Ideally, I feel that I should only have to annotate the bare minimum, and let the checker do the rest. Paul
On Fri, Oct 15, 2021 at 9:20 AM Sebastian M. Ernst <ernst@pleiszenburg.de> wrote:
PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs.
But that's exactly what Dict[int, str] already is -- the keys are ints, the values are strings, and there can be any number of items. What are you looking for that's different from that?? As far as I can tell, there is a bug / missing feature in typeguard. What could you possibly do to better specify this particular type? Maybe that's more clear in the SO thread you referenced, but SO seems to be down at the moment ... -CHB
For reference, related question on SO: https://stackoverflow.com/q/69555006/1672565
Best regards, Sebastian _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DY5DPG... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Fri, Oct 15, 2021 at 06:17:12PM +0200, Sebastian M. Ernst wrote:
Data = Dict[str, Number]
@typechecked def foo(bar: Data): print(bar) ```
Yes, this is using run-time checks (typeguard), which works just fine. Only strings as keys and Number objects as values are going through. (I was told that MyPy does not get this at the moment.)
Be careful about believing what you are told. [steve ~]$ cat test.py from numbers import Number from typing import Dict Data = Dict[str, Number] a: Data = {None: []} [steve ~]$ mypy test.py test.py:4: error: Dict entry 0 has incompatible type "None": "List[<nothing>]"; expected "str": "Number" Found 1 error in 1 file (checked 1 source file)
The issue is that `bar` itself still allows "everything" to go in (and out):
```python @typechecked def foo2(bar: Data): bar[1.0] = b'should not be allowed' ```
The Python interpreter intentionally doesn't know or care about typing annotations. The interpreter should absolutely allow that. However mypy correctly flags that assignment as a type error. How about typeguard? That's for the typeguard developers to answer, but my guess is that they will say that the typechecked decorator can enforce type checking of the input parameters and perhaps even the return result, but there is nothing they can do about what happens inside the body of the function.
PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs.
Something like this? from numbers import Number from collections import UserDict class RestrictedDict(UserDict): def __setitem__(self, key, value): if not isinstance(key, str): raise TypeError('key must be a string') if not isinstance(value, Number): raise TypeError('value must be a number') super().__setitem__(key, value) (I think that is the only method that needs to be overloaded.) -- Steve
On 16 Oct 2021, at 06:13, Steven D'Aprano <steve@pearwood.info> wrote: Be careful about believing what you are told.
Indeed, MyPy will correctly raise errors if you assign {None: []} to a variable annotated with dict[str, Number]. However, you'll find that MyPy also raises an error if you assign {'foo': 4} to a variable annotated with dict[str, Number]: https://mypy-play.net/?mypy=latest&python=3.10&gist=de7faea3b4d6a0ffefc64acf3bfa7b77 The issue isn't a matter of false negatives from Mypy; it's a matter of false positives. For more, see the (rather long — sorry!) answer I gave Sebastian to a previous question of his on Stack Overflow: https://stackoverflow.com/questions/69334475/how-to-hint-at-number-types-i-e... Having said that, I absolutely agree that type-hinting doesn't seem to have much to do with this specific problem, and that a UserDict subclass is probably the way to go here. Best, Alex
On 16 Oct 2021, at 06:13, Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Oct 15, 2021 at 06:17:12PM +0200, Sebastian M. Ernst wrote:
Data = Dict[str, Number]
@typechecked def foo(bar: Data): print(bar) ```
Yes, this is using run-time checks (typeguard), which works just fine. Only strings as keys and Number objects as values are going through. (I was told that MyPy does not get this at the moment.)
Be careful about believing what you are told.
[steve ~]$ cat test.py from numbers import Number from typing import Dict Data = Dict[str, Number] a: Data = {None: []}
[steve ~]$ mypy test.py test.py:4: error: Dict entry 0 has incompatible type "None": "List[<nothing>]"; expected "str": "Number" Found 1 error in 1 file (checked 1 source file)
The issue is that `bar` itself still allows "everything" to go in (and out):
```python @typechecked def foo2(bar: Data): bar[1.0] = b'should not be allowed' ```
The Python interpreter intentionally doesn't know or care about typing annotations. The interpreter should absolutely allow that.
However mypy correctly flags that assignment as a type error.
How about typeguard? That's for the typeguard developers to answer, but my guess is that they will say that the typechecked decorator can enforce type checking of the input parameters and perhaps even the return result, but there is nothing they can do about what happens inside the body of the function.
PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs.
Something like this?
from numbers import Number from collections import UserDict
class RestrictedDict(UserDict): def __setitem__(self, key, value): if not isinstance(key, str): raise TypeError('key must be a string') if not isinstance(value, Number): raise TypeError('value must be a number') super().__setitem__(key, value)
(I think that is the only method that needs to be overloaded.)
-- Steve _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HLGDIW... Code of Conduct: http://python.org/psf/codeofconduct/
On Sat, Oct 16, 2021 at 09:54:13AM +0100, Alex Waygood wrote:
On 16 Oct 2021, at 06:13, Steven D'Aprano <steve@pearwood.info> wrote: Be careful about believing what you are told.
Indeed, MyPy will correctly raise errors if you assign {None: []} to a variable annotated with dict[str, Number]. However, you'll find that MyPy also raises an error if you assign {'foo': 4} to a variable annotated with dict[str, Number]:
Hah, serves me right for not testing it for both positive and negative cases. In my very limited testing now, I see that the problem appears to be with the Number type. mypy correctly accepts this: Data = Dict[str, int] a: Data = {'spam': 42} (no errors), but if you change the annotation to use Number instead of int, it wrongly flags that as a type error. Possibly mypy doesn't know that ints and floats are instances of Number?
participants (6)
-
Alex Waygood
-
Christopher Barker
-
Paul Moore
-
Sebastian M. Ernst
-
Simão Afonso
-
Steven D'Aprano