Some other considerations:

+ Using `kwargs` within the function body:

Within the function body, `kwargs` is treated as having type `Kwargs`.

```
class MovieKwargs(TypedDict):
    name: str
    year: int

def foo(**kwargs: **MovieKwargs) -> None:
    name: str = kwargs["name"]
    year: int = kwargs["year"]

    # => MovieKwargs
    reveal_type(kwargs)

    # invalid because `name` is `str`.
    kwargs["name"] = None
    # invalid because `year` is required.
    del kwargs["year"]
    # invalid.
    kwargs["extra"] = 1
```

+ What about compatibility checks for functions?

*Rule*: When checking compatibility for functions that use `**Kwargs`, we will have to unpack the TypedDict fields and treat them as we treat explicit keyword-only parameters.

```
class MovieBase(TypedDict):
    name: str

class Movie(MovieBase, total=False):
    year: int

def foo(**kwargs: Movie) -> None: ...

# foo must be treated as:

def foo(*, name: str, year: int = ...) -> None: ...
```

If we didn't do the above, we would have unintuitive behavior because of the compatibility rules for TypedDict.

> Value types behave invariantly, since TypedDict objects are mutable. This is similar to mutable container types such as List and Dict.
>
> -- https://www.python.org/dev/peps/pep-0589/#type-consistency

This is not a consideration when checking compatibility of functions with `**kwargs`, as far as I can see.

```
class ExpectsOptionalInt(Protocol):
    def foo(self, *, x: Optional[int]) -> None: ...

class ExpectsInt(Protocol):
    def foo(self, *, x: int) -> None: ...

def foo() -> None:
    x: ExpectsOptionalInt
    # valid
    y: ExpectsInt = x
```

However, if naively used the compatibility check that the TypedDict `Kwarg1` is compatible with `Kwarg2`, then we would get the following:

```

class OptionalIntKwargs(TypedDict):
    x: Optional[int]

class IntKwargs(TypedDict):
    x: int

class ExpectsOptionalInt(Protocol):
    def foo(self, **OptionalIntKwargs) -> None: ...

class ExpectsInt(Protocol):
    def foo(self, **IntKwargs) -> None: ...

def foo() -> None:
    x: ExpectsOptionalInt
    # We would consider this invalid!
    y: ExpectsInt = x

    # That is because we would check if `IntKwargs` was compatible with `OptionalIntKwargs`.
    # It is not compatible because of the TypedDict compatibility rules.
    # i.e.,
    x: IntKwargs
    y: OptionalIntKwargs = x  # invalid
```

Another example: a function with a non-required keyword-only parameter is compatible with a function that has a required keyword-only parameter.

```
class NonRequired(Protocol):
    def foo(self, *, x: int = ...) -> None: ...
class Required(Protocol):
    def foo(self, *, x: int) -> None: ...

def foo() -> None:
    x: NonRequired
    # valid
    y: Required = x
```

That wouldn't be the case if we tried to check if a TypedDict with a required field was compatible with a TypedDict having a non-required field.

The proposed rule above answers for the other issues raised in the TypedDict PEP.

+ What if the type supplied to `**Kwargs` is a Union?

Eric had raised this question. For the time being, we can consider only concrete TypedDicts.

(I imagine unpacking a Union of TypedDicts would give us multiple overloads, but I haven't thought about complications like the ordering.)



On Sun, Feb 7, 2021 at 5:43 PM S Pradeep Kumar <gohanpra@gmail.com> wrote:
I agree with Eric that `**kwargs: Foo` means that `kwargs` is a dictionary of Foo values, as per PEP 484. Changing that would be backward-incompatible, and special-casing it for TypedDict alone would be clumsy.

However, it *is* useful to be able to specify the individual types of the keyword-only parameters we want to accept.

*Proposal*: We allow typing `**kwargs: **KwargsTypedDict`.

Required keyword-only parameters will be required fields of the TypedDict. Keyword-only parameters with default values will be non-required fields of the TypedDict.

I wanted a real-world example to motivate this and thought of `json.loads` and co:

```python
# Simplified and modified slightly from `json.loads` and `json.load`, which
# share the same keyword-only parameters.

# Before.
def loads(
    s: Union[str, bytes],
    *,
    # No default value for this one.
    json_decoder: Type[JSONDecoder],
    # Has a default value.
    parse_int: Optional[Callable[[str], Any]] = ...,
    # And a bunch of other keyword-only parameters.
) -> JSON:
    ...


def load(
    fp: SupportsRead[Union[str, bytes]],
    *,
    json_decoder: Type[JSONDecoder],
    parse_int: Optional[Callable[[str], Any]] = ...,
) -> JSON:
    ...

class JSONDecoder:
    def __init__(
       self,
       *,
       parse_int: Optional[Callable[[str], Any]] = ...,
    ) -> None: ...
```

Note that `loads` and `load` internally just construct a `JSONDecoder` by passing on their kwargs to the given `json_decoder`. Clearly, these functions share the same keyword-only parameters.

```python
# These keyword-only parameters had default values. So, they are non-required
# fields in the TypedDict.
class JSONDecoderKwargs(TypedDict, total=False):
    parse_int: Callable[[str], Any]
    # And a bunch of others.


# `json_decoder` was a required keyword-only parameter (in my example above).
# So, it goes in a total TypedDict.
# We also inherit the other fields (preserving their non-requiredness).
class LoadKwargs(JSONDecoderKwargs, total=True):
    json_decoder: Type[JSONDecoder]


# After.
def loads(
    s: Union[str, bytes],
    **kwargs: **LoadKwargs,
) -> JSON:
    ...

def load(
    fp: SupportsRead[Union[str, bytes]],
    **kwargs: **LoadKwargs,
) -> JSON:
    ...

class JSONDecoder:
    def __init__(
       self,
        **kwargs: **JSONDecoderKwargs,
    ) -> None: ...

# valid
loads(s, json_decoder=MyJsonDecoder)
loads(s, json_decoder=MyJsonDecoder, parse_int=my_parse_int)

# invalid: missing keyword-only parameter `class_info`.
loads(s)
# invalid: type mismatch
loads(s, json_decoder=1)
# invalid: unexpected argument `foo`.
loads(s, json_decoder=MyJsonDecoder, foo=2)
```

Other such cases off the top of my head include

+ `subprocess.run`, `Popen.__init__`, and friends.
+ `sort` and `sorted`, which accept the same keyword-only parameters

Things to consider:

1. This is backward-compatible because it preserves the `**kwargs: int` behavior of PEP 484.

2. `**kwargs: **KwargsTypedDict` requires changes to the parser. If that's not worth it, we could settle for something like `**kwargs: UnpackTypedDict[KwargsTypedDict]`. This is analogous to PEP 646 making `Unpack[Ts]` a synonym for `*Ts` until syntax support lands.

3. We may also want to allow arbitrary keyword parameters beyond the ones specifically named.

That is, how would we type the following using the above TypedDict proposal?

```
def foo(*, required: int, non_required: str=..., **kwargs: int) -> None: ...

# valid.
foo(required=1)
# valid.
foo(required=1, extra=7)
# invalid: expected int, got str.
foo(required=1, extra="wrong type")
```

One option is to simply require users to type out the named keyword-only parameters by hand (as done above for `foo`). This wouldn't allow multiple such functions to share the same keyword-only parameters, but would not require any other changes.

A more long-term option is to allow open-ended TypedDicts - ones that allows arbitrary fields other than the named fields. I believe there was some discussion about this earlier, but there was no resulting PEP there: https://mail.python.org/archives/list/typing-sig@python.org/thread/66RITIHDQHVTUMJHH2ORSNWZ6DOPM367/#2S4YVCLI2FNMSP7QJTSDRV5VUVF27LEM. This might be impractical to wait for.

Yet another option is to always allow arbitrary kwargs beyond the fields in the TypedDict. I'm against this because it won't let us specify that we want a finite set of keyword-only parameters, like in the `json.loads` example above.

In any case, this is a backward-compatible feature that we can defer for this discussion.


On Sun, Feb 7, 2021 at 3:08 PM Eric Traut <eric@traut.com> wrote:
PEP 484 says that a type argument on a **kwargs parameter annotates the _value_ type of the dictionary. In other words, `**kwargs: X` indicates that kwargs is of type `Dict[str, X]`. If X is a TypedDict, then this would indicate that kwargs is a dictionary whose keys are strings and values are all typed dictionaries. I don't see any ambiguity here, but it's clearly not the behavior that you want.

I think you're proposing to make an exception to the PEP 484 rule specifically in the case that X evaluates to a TypedDict. I think that's an ugly inconsistency, and it sets a dangerous precedent. One could make the argument that `Dict` should be exempt from the normal PEP 484 rules here as well. It would mean that there's no way to specify the case where you intend for kwargs to be a dictionary whose values are all type dictionaries. It's also a change that would break backward compatibility, since the rules for PEP 484 are well established. It also opens up questions like what if X is a union that includes a TypedDict or multiple TypedDicts? For all of these reasons, I think this proposal is a no-go.

If I understand your motivation correctly, you are designing an interface where you have (presumably a large number of) keyword parameters that are shared across many methods. Have you considered changing your interface such that you don't expose individual keyword parameters and instead expose a single parameter that accepts a TypedDict? I realize this would change the way callers invoke these methods (e.g. `foo(a=3, b=5) would need to be changed to `foo({"a": 1, "b": 5})`), but it would be type safe and would work with existing Python type checkers.

--
Eric Traut
Contributor to Pyright and Pylance
Microsoft Corp.
_______________________________________________
Typing-sig mailing list -- typing-sig@python.org
To unsubscribe send an email to typing-sig-leave@python.org
https://mail.python.org/mailman3/lists/typing-sig.python.org/
Member address: gohanpra@gmail.com


--
S Pradeep Kumar


--
S Pradeep Kumar