Type narrowing for TypeGuard in the negative case
Since the introduction of PEP 647 (User-defined Type Guards), we've received a steady stream of input from users saying that they don't like the limitation that type narrowing is applied only in the positive case and is not applied in the negative case. In general, type narrowing is not safe to perform if a user-defined type guard returns False, so I've stood by the original decision not to provide type narrowing in the negative case, but there are cases where such narrowing is safe and desirable. Here's a recent thread where this was discussed in some detail: https://github.com/python/typing/issues/996#issuecomment-1002716830 In this discussion, Ilya Kamen proposed a solution. The proposal is to extend the existing `TypeGuard` to support an optional second type argument. If present, the second argument indicates the narrowed type in the negative type narrowing situation. Here's a simple (admittedly contrived) example: ```python def is_str(val: str | int) -> TypeGuard[str, int]: return isinstance(val, str) def func(val: str | int): if is_str(val): reveal_type(val) # str else: reveal_type(val) # int ``` I've implemented this proposal in the latest published version of pyright (1.1.202) so folks can experiment with it and see if they like it. If there's general consensus that it's the right approach, I can file an amendment for the existing PEP 647 to include this functionality. It was trivial to add to pyright, so I'm optimistic that it would likewise be easy to add to the other type checkers. Another common request for PEP 647 is the desire for an "assert" form of TypeGuard — a way to indicate that a function performs runtime validation of a type, raising an exception if the type is incorrect. It occurred to me that this two-argument form of `TypeGuard` could also be used to handle this use case. The second argument would be specified as a `NoReturn`. I've provisionally implemented this in pyright as well (although it's not in the currently-published version — you'll need to wait until 1.1.203 is published). Here's what this would look like: ```python def validate_sequence(val: Sequence[_T | None]) -> TypeGuard[Sequence[_T], NoReturn]: """Verifies that the sequence contains no None values""" if len([x for x in val if x is not None]) > 0: raise Exception() return True def func(val: Sequence[int | None]): validate_sequence(val) reveal_type(val) # Sequence[int] ``` I'm interested in input on these proposals. -Eric -- Eric Traut Contributor to Pyright & Pylance Microsoft
El jue, 30 dic 2021 a las 9:39, Eric Traut (<eric@traut.com>) escribió:
Since the introduction of PEP 647 (User-defined Type Guards), we've received a steady stream of input from users saying that they don't like the limitation that type narrowing is applied only in the positive case and is not applied in the negative case.
In general, type narrowing is not safe to perform if a user-defined type guard returns False, so I've stood by the original decision not to provide type narrowing in the negative case, but there are cases where such narrowing is safe and desirable.
Here's a recent thread where this was discussed in some detail: https://github.com/python/typing/issues/996#issuecomment-1002716830 In this discussion, Ilya Kamen proposed a solution. The proposal is to extend the existing `TypeGuard` to support an optional second type argument. If present, the second argument indicates the narrowed type in the negative type narrowing situation.
Here's a simple (admittedly contrived) example: ```python def is_str(val: str | int) -> TypeGuard[str, int]: return isinstance(val, str)
def func(val: str | int): if is_str(val): reveal_type(val) # str else: reveal_type(val) # int ```
I'm not sure this is useful in any non-contrived cases. The motivating use case in the issue was `numpy.isscalar`, which is essentially: def isscalar(element: object) -> TypeGuard[SupportsFloat]: ... In the use case, we have a variable of type `SupportsFloat | numpy.ndarray`, and we want it to narrow to `ndarray` if `isscalar()` returns False. But `isscalar()` returns False on everything that's not a scalar, so we can't just put ndarray as the second argument to TypeGuard. If it existed, we could express it with a type like TypeScript's Exclude ( https://www.typescriptlang.org/docs/handbook/utility-types.html#excludeunion... ): def isscalar(element: T) -> TypeGuard[SupportsFloat, Exclude[T, SupportsFloat]]: ... I'm having trouble coming up with a realistic use case for two-parameter TypeGuard other than the NoReturn case that wouldn't need this Exclude type.
I've implemented this proposal in the latest published version of pyright (1.1.202) so folks can experiment with it and see if they like it. If there's general consensus that it's the right approach, I can file an amendment for the existing PEP 647 to include this functionality. It was trivial to add to pyright, so I'm optimistic that it would likewise be easy to add to the other type checkers.
Another common request for PEP 647 is the desire for an "assert" form of TypeGuard — a way to indicate that a function performs runtime validation of a type, raising an exception if the type is incorrect. It occurred to me that this two-argument form of `TypeGuard` could also be used to handle this use case. The second argument would be specified as a `NoReturn`. I've provisionally implemented this in pyright as well (although it's not in the currently-published version — you'll need to wait until 1.1.203 is published).
Here's what this would look like:
```python def validate_sequence(val: Sequence[_T | None]) -> TypeGuard[Sequence[_T], NoReturn]: """Verifies that the sequence contains no None values""" if len([x for x in val if x is not None]) > 0: raise Exception() return True
def func(val: Sequence[int | None]): validate_sequence(val) reveal_type(val) # Sequence[int] ```
I'm interested in input on these proposals.
-Eric
-- Eric Traut Contributor to Pyright & Pylance Microsoft _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
Jelle Zijlstra wrote:
[...] The motivating use case in the issue was `numpy.isscalar`, which is essentially: def isscalar(element: object) -> TypeGuard[SupportsFloat]: ... In the use case, we have a variable of type `SupportsFloat | numpy.ndarray`, and we want it to narrow to `ndarray` if `isscalar()` returns False. But `isscalar()` returns False on everything that's not a scalar, so we can't just put ndarray as the second argument to TypeGuard. If it existed, we could express it with a type like TypeScript's Exclude ( https://www.typescriptlang.org/docs/handbook/utility-types.html#excludeunion... ): def isscalar(element: T) -> TypeGuard[SupportsFloat, Exclude[T, SupportsFloat]]: ...
You could try something like this: ```py def guard(var: SupportsFloat | T) -> TypeGuard[SupportsFloat, T]: ... def f(x: str | float) -> None: if guard(x): reveal_type(x) # SupportsFloat else: reveal_type(x) # str ``` This does work with the current pyright implementation. Although it is limited to a union of two types for `x`. `str | bytes | float` doesn't work unfortunately.
I'm not sure this is useful in any non-contrived cases. [...] I'm having trouble coming up with a realistic use case for two-parameter TypeGuard other than the NoReturn case that wouldn't need this Exclude type.
Personally, I would love being able to use it for something like `asyncio.iscoroutine` and `asyncio.iscoroutinefunction`. ```py T = TypeVar("T") _T_co = TypeVar("_T_co", covariant=True) _T_contra = TypeVar("_T_contra", contravariant=True) _V_co = TypeVar("_V_co", covariant=True) def iscoroutine(obj: Coroutine[_T_co, _T_contra, _V_co] | T) -> TypeGuard[Coroutine[_T_co, _T_contra, _V_co], T]: ... ```
I recently encountered usefulness in preserving type variable relationship as mentioned in this issue, https://github.com/microsoft/pyright/issues/3119. An example with current type guard both mypy and pyright produce an error on this code, from typing_extensions import TypeGuard def is_int(x: object) -> TypeGuard[int] return isinstance(x, int) def f(x: T) -> T: if is_int(x): return x # type here is int not int* so an error is produced return x While error is correct it would be nice for typeguards to preserve typevar relationship. But honestly right now I only have 2/3 times this issue appears in my current codebase of 100k lines (mostly typed). So it's fairly small need. Negative part of logic I remember thinking interesting, but then I reviewed my codebase for usages of typeguards and false positives due to lack of negative narrowing and found only 1/2 cases of it. On assertion type guards I don't understand what practical difference between an existing typeguard with an assertion and a typeguard with NoReturn as negative argument. I sometimes have code like, def is_type_list(...) -> TypeGuard[...]: ... def foo(config: object): assert is_type_list(config, int) and that looks to work. It does remove assert statement in foo, but if that's difference it feels quite minor. My overall sentiment is weakly positive, but I think biggest reason these issues are infrequently felt is a lot of libraries type hints/typeshed does not use TypeGuard enough to notice these interactions. Maybe as time passes if typeguard becomes more common in typeshed these issues will pop back up.
This proposal (and variants you raised in the typing issue, like <https://github.com/python/typing/issues/996#issuecomment-1001979242>) all rely on the typeguard function knowing the precise set of types that will be passed to it in the first place, and that type union being relatively small so that the `else` type is also useful to work on. I don't think this is very useful in practice: (1) it doesn't match how existing type-checking functions often work, and (2) doesn't match the behavior of type-checking inline in your `if` condition. Instead, I think for this to be useful, false TypeGuards (or some variant of TypeGuard, to avoid changing current behavior) need to operate on *value* unions, rather than *argument* unions. 1) The existing type-narrowing checks (isinstance(), issubclass(), callable()) all (effectively?) take Any as their input type; if they return False, the type of the output is just Any minus whatever they checked for. My own typeguards in my code do the same, precisely because I might be taking any of a number of values. Since we don't have subtraction types, these functions *can't* explicitly write out their false type. In general, I don't think it's reasonable to expect type-guarding functions to have a non-Any input type at all (or at least a sufficiently narrow type union that the subtraction gives a useful type); their whole point is discriminating the one type you care about from anything else -- sometimes I'm filtering Element nodes from Comment nodes, other times from strs, other times from Nones. 2) The existing type-narrowing checks operate on *value* types, not argument types; if I have a value of type `str | Element` and I write `if isinstance(x, Element)`, the `else` branch will have narrowed x's type to `str` for me. If x's type was `str | Element | None`, the else branch will narrow to just `str | None`. The type-narrowing examples all explicitly call this behavior out. But if I move my `isinstance()` check to a type-guarding function, I lose this behavior and have to manually narrow my types in the `else`. This is the *precise* behavior that I want to maintain, however! For the sake of being productive, I suggest a variant called `TotalTypeGuard[type]`, which implies that your type-guarding function returns True for *all* values of the specified type. (Not just potentially for some, as Guido points out can happen with the current TypeGuard.) The behavior when it's false is then identical to existing built-in type-narrowers: if the value's type is a union, the specified type is removed from that union.
I'm not sure I fully understand the terminology you're using. Can you explain what you mean by "value unions" versus "argument unions"? Those terms aren't part of the Python typing lexicon currently (at least to my knowledge), but it sounds like you see this as an important distinction. There have been deep discussions about supporting a variant of TypeGuard that supports narrowing in the negative case but is more limited than the existing PEP 647 `TypeGuard`. See https://github.com/python/typing/discussions/1013#discussioncomment-1966238 for a concrete proposal for a variant called `StrictTypeGuard`. Rebecca Chen (member of the pytype team) presented this idea at the 2020 PyCon Typing Summit (https://us.pycon.org/2022/events/typing-summit/), so you may want to watch the recording of that session. I think this idea be consistent with your `TotalTypeGuard` concept. I've implemented the proposed `StrictTypeGuard` in pyright, so you can try it today. `StrictTypeGuard` won't work at runtime because there is no runtime support currently, but it will work at type checking time. So far, support for `StrictTypeGuard` has been somewhere between positive and lukewarm within the typing community. I don't recall any significant negative feedback about the proposal, but no one has stepped up yet to shepherd the proposal through the standardization process. If someone is strongly motivated to push it further, I think the next step would be to write a formal PEP. Eric Traut Contributor to pyright & pylance Microsoft
Apologies for the non-standard terminology; I'm still relatively new to the community. I meant precisely the distinction between your original proposal in <https://github.com/python/typing/discussions/1013> ("argument unions") and your updated one in the same thread ("value unions") - that is, having the type guard narrow the type of the actual value passed to the typeguard function, rather than narrowing the type of the typeguard function's argument. I hadn't seen discussion 1013 earlier, but reading the comment you link, that is indeed *exactly* what I was wanting, yes! Thanks for that! (And while I don't do a lot of exception-raising, the *Assert types in that proposal also sound quite useful to me.) My day job is already web standards work (in the W3C and JavaScript communities), so I suspect shepherding a new PEP through would be doable, but it would be a new experience for me. ^_^
Eric started a new thread about this topic here in case anyone is interested: https://github.com/python/typing/discussions/1013 I'm working on a PEP for a suggestion that Eric had here: https://github.com/python/typing/discussions/1013#discussioncomment-6003799
participants (7)
-
Eric Traut -
jackalmage＠gmail.com -
Jelle Zijlstra -
Marc Mueller -
Mehdi2277 -
Neil Girdhar -
Rich Chiodo