Mailman 3 Type-hinting dictionaries for an arbitrary number of arbitrary key/value pairs? Counterpart to PEP 589? - Python-ideas

Type-hinting dictionaries for an arbitrary number of arbitrary key/value pairs? Counterpart to PEP 589?

Sebastian M. Ernst

15 Oct 2021 15 Oct '21

12:17 p.m.

Hi all, disclaimer: I have no idea on potential syntax or if it is applicable to a wide audience or if there is already a good solution to this. It is more like a "gap" in the type hint spec that I ran across in a project. In function/method signatures, I can hint at dictionaries for example as follows: ```python from numbers import Number from typing import Dict from typeguard import typechecked Data = Dict[str, Number] @typechecked def foo(bar: Data): print(bar) ``` Yes, this is using run-time checks (typeguard), which works just fine. Only strings as keys and Number objects as values are going through. (I was told that MyPy does not get this at the moment.) The issue is that `bar` itself still allows "everything" to go in (and out): ```python @typechecked def foo2(bar: Data): bar[1.0] = b'should not be allowed' ``` PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs. For reference, related question on SO: https://stackoverflow.com/q/69555006/1672565 Best regards, Sebastian

Show replies by thread

Paul Moore

15 Oct 15 Oct

12:49 p.m.

Mypy correctly rejects this: ❯ type .\t.py from numbers import Number from typing import Dict Data = Dict[str, Number] def foo(bar: Data): print(bar) bar[1.0] = b'hello' PS 17:48 00:00.008 C:\Work\Scratch\foo ❯ mypy .\t.py t.py:9: error: Invalid index type "float" for "Dict[str, Number]"; expected type "str" t.py:9: error: Incompatible types in assignment (expression has type "bytes", target has type "Number") Found 2 errors in 1 file (checked 1 source file) If typeguard doesn't, maybe you need to raise that as a bug against that project? Paul On Fri, 15 Oct 2021 at 17:20, Sebastian M. Ernst wrote:

...

Hi all,

disclaimer: I have no idea on potential syntax or if it is applicable to a wide audience or if there is already a good solution to this. It is more like a "gap" in the type hint spec that I ran across in a project.

In function/method signatures, I can hint at dictionaries for example as follows:

```python from numbers import Number from typing import Dict

from typeguard import typechecked

Data = Dict[str, Number]

@typechecked def foo(bar: Data): print(bar) ```

Yes, this is using run-time checks (typeguard), which works just fine. Only strings as keys and Number objects as values are going through. (I was told that MyPy does not get this at the moment.)

The issue is that `bar` itself still allows "everything" to go in (and out):

```python @typechecked def foo2(bar: Data): bar[1.0] = b'should not be allowed' ```

PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs.

For reference, related question on SO: https://stackoverflow.com/q/69555006/1672565

Best regards, Sebastian _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DY5DPG... Code of Conduct: http://python.org/psf/codeofconduct/

Sebastian M. Ernst

1:07 p.m.

Hi Paul, all, Am 15.10.21 um 18:49 schrieb Paul Moore:

...

Mypy correctly rejects this: [...]

interesting. Thanks for trying.

...

If typeguard doesn't, maybe you need to raise that as a bug against that project?

This is kind of contradicting the design of typeguard. It works on a call level checking arguments and return values for the most part. Typeguard does not provide any form of type inference like mypy does. From this perspective, typeguard would need to guard `dict.__init__`, `dict.__setitem__`, `dict.update` and friends directly, essentially requiring a subclass of `dict` of some sort. However, typeguard does have a special function to perform type checks on objects, e.g. `check_type('variablename', [1234], List[int])` Ignoring typeguard, my suggestion still stands, although slightly changed: Annotating a dictionary as described earlier in such a way that type inference is not required OR in such a way that run-time checkers have a chance to work more easily - if this makes any sense at all? Best regards, Sebastian

Simão Afonso

1:36 p.m.

Just a pointer related to this, typeguard is abandoned. kttps://github.com/agronholm/typeguard/issues/198

Paul Moore

2:15 p.m.

On Fri, 15 Oct 2021 at 18:07, Sebastian M. Ernst wrote:

...

Ignoring typeguard, my suggestion still stands, although slightly changed: Annotating a dictionary as described earlier in such a way that type inference is not required OR in such a way that run-time checkers have a chance to work more easily - if this makes any sense at all?

I'm not sure what your suggestion actually is, though. "I am looking for an arbitrary number of typed keys/value pairs." - isn't that Mapping[str, float]" (or something similar)? I see no value in making the programmer do more work so that type checkers can do less type inference. Far from it, I'd like to have as few annotations as possible, and have the type checkers do *more* work for me. Ideally, I feel that I should only have to annotate the bare minimum, and let the checker do the rest. Paul

Christopher Barker

10:55 p.m.

On Fri, Oct 15, 2021 at 9:20 AM Sebastian M. Ernst wrote:

...

PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs.

But that's exactly what Dict[int, str] already is -- the keys are ints, the values are strings, and there can be any number of items. What are you looking for that's different from that?? As far as I can tell, there is a bug / missing feature in typeguard. What could you possibly do to better specify this particular type? Maybe that's more clear in the SO thread you referenced, but SO seems to be down at the moment ... -CHB

...

For reference, related question on SO: https://stackoverflow.com/q/69555006/1672565

Best regards, Sebastian _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DY5DPG... Code of Conduct: http://python.org/psf/codeofconduct/

-- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

Steven D'Aprano

16 Oct 16 Oct

1:09 a.m.

On Fri, Oct 15, 2021 at 06:17:12PM +0200, Sebastian M. Ernst wrote:

...

Data = Dict[str, Number]

@typechecked def foo(bar: Data): print(bar) ```

Yes, this is using run-time checks (typeguard), which works just fine. Only strings as keys and Number objects as values are going through. (I was told that MyPy does not get this at the moment.)

Be careful about believing what you are told. [steve ~]$ cat test.py from numbers import Number from typing import Dict Data = Dict[str, Number] a: Data = {None: []} [steve ~]$ mypy test.py test.py:4: error: Dict entry 0 has incompatible type "None": "List[<nothing>]"; expected "str": "Number" Found 1 error in 1 file (checked 1 source file)

...

The issue is that `bar` itself still allows "everything" to go in (and out):

```python @typechecked def foo2(bar: Data): bar[1.0] = b'should not be allowed' ```

The Python interpreter intentionally doesn't know or care about typing annotations. The interpreter should absolutely allow that. However mypy correctly flags that assignment as a type error. How about typeguard? That's for the typeguard developers to answer, but my guess is that they will say that the typechecked decorator can enforce type checking of the input parameters and perhaps even the return result, but there is nothing they can do about what happens inside the body of the function.

...

PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs.

Something like this? from numbers import Number from collections import UserDict class RestrictedDict(UserDict): def __setitem__(self, key, value): if not isinstance(key, str): raise TypeError('key must be a string') if not isinstance(value, Number): raise TypeError('value must be a number') super().__setitem__(key, value) (I think that is the only method that needs to be overloaded.) -- Steve

Alex Waygood

4:54 a.m.

...

On 16 Oct 2021, at 06:13, Steven D'Aprano wrote: Be careful about believing what you are told.

Indeed, MyPy will correctly raise errors if you assign {None: []} to a variable annotated with dict[str, Number]. However, you'll find that MyPy also raises an error if you assign {'foo': 4} to a variable annotated with dict[str, Number]: https://mypy-play.net/?mypy=latest&python=3.10&gist=de7faea3b4d6a0ffefc64acf3bfa7b77 The issue isn't a matter of false negatives from Mypy; it's a matter of false positives. For more, see the (rather long — sorry!) answer I gave Sebastian to a previous question of his on Stack Overflow: https://stackoverflow.com/questions/69334475/how-to-hint-at-number-types-i-e... Having said that, I absolutely agree that type-hinting doesn't seem to have much to do with this specific problem, and that a UserDict subclass is probably the way to go here. Best, Alex

...

On 16 Oct 2021, at 06:13, Steven D'Aprano wrote:

On Fri, Oct 15, 2021 at 06:17:12PM +0200, Sebastian M. Ernst wrote:

...
Data = Dict[str, Number]

@typechecked def foo(bar: Data): print(bar) ```

Yes, this is using run-time checks (typeguard), which works just fine. Only strings as keys and Number objects as values are going through. (I was told that MyPy does not get this at the moment.)

Be careful about believing what you are told.

[steve ~]$ cat test.py from numbers import Number from typing import Dict Data = Dict[str, Number] a: Data = {None: []}

[steve ~]$ mypy test.py test.py:4: error: Dict entry 0 has incompatible type "None": "List[<nothing>]"; expected "str": "Number" Found 1 error in 1 file (checked 1 source file)

...
The issue is that `bar` itself still allows "everything" to go in (and out):

```python @typechecked def foo2(bar: Data): bar[1.0] = b'should not be allowed' ```

The Python interpreter intentionally doesn't know or care about typing annotations. The interpreter should absolutely allow that.

However mypy correctly flags that assignment as a type error.

How about typeguard? That's for the typeguard developers to answer, but my guess is that they will say that the typechecked decorator can enforce type checking of the input parameters and perhaps even the return result, but there is nothing they can do about what happens inside the body of the function.

...
PEP 589 introduces typed dictionaries, but for a fixed set of predefined keys (similar to struct-like constructs in other languages). In contrast, I am looking for an arbitrary number of typed keys/value pairs.

Something like this?

from numbers import Number from collections import UserDict

class RestrictedDict(UserDict): def __setitem__(self, key, value): if not isinstance(key, str): raise TypeError('key must be a string') if not isinstance(value, Number): raise TypeError('value must be a number') super().__setitem__(key, value)

(I think that is the only method that needs to be overloaded.)

-- Steve _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HLGDIW... Code of Conduct: http://python.org/psf/codeofconduct/

Steven D'Aprano

8:50 a.m.

On Sat, Oct 16, 2021 at 09:54:13AM +0100, Alex Waygood wrote:

...

...
On 16 Oct 2021, at 06:13, Steven D'Aprano wrote: Be careful about believing what you are told.

Indeed, MyPy will correctly raise errors if you assign {None: []} to a variable annotated with dict[str, Number]. However, you'll find that MyPy also raises an error if you assign {'foo': 4} to a variable annotated with dict[str, Number]:

Hah, serves me right for not testing it for both positive and negative cases. In my very limited testing now, I see that the problem appears to be with the Number type. mypy correctly accepts this: Data = Dict[str, int] a: Data = {'spam': 42} (no errors), but if you change the annotation to use Number instead of int, it wrongly flags that as a type error. Possibly mypy doesn't know that ints and floats are instances of Number?

920

Age (days ago)

921

Last active (days ago)

List overview

Download

8 comments

6 participants

participants (6)

Alex Waygood
Christopher Barker
Paul Moore
Sebastian M. Ernst
Simão Afonso
Steven D'Aprano

Type-hinting dictionaries for an arbitrary number of arbitrary key/value pairs? Counterpart to PEP 589?

tags

participants (6)