Thoughts on a NotType?
Hi, I'd like to write a PEP for a new typing construct: NotType which, would be equivalent to any type but x. There has been an issue open on the typing repo for quite some time about this (https://github.com/python/typing/issues/801). After the recent-ish proposal for an IntersectionType, this would be a nice-to-have feature as it would complete the set theory operations for typed Python. I'd also like to propose implementing type.__invert__ to make this work, so it has built-in syntax like intersection and union as this is the closest thing to a not operator. Usecases: Something that takes any non-empty string ```py def fn(x: str & ~Literal[""]): ... ``` This also has nice integration with PEP 675 where the type of a Literal str is important. If you’d want to call this from any str you’d need a type guard. ```py def guard(x: str) -> TypeGuard[str & ~Literal[“”]]: return isinstance(x, str) and x != “” # or maybe bool(x) although this is probably much harder to implement ``` Something that takes any value but None ```py def fn(x: ~Literal[None]): ... ``` Something that requires not an integer ```py def not_an_int(x: Any) -> TypeGuard[~int]: ... ``` This also allows for narrowing in the negative case which is another requested feature (https://github.com/python/typing/issues/926) ```py @overload def is_none(value: None) -> Literal[True]: ... @overload def is_none(value: ~Literal[None]) -> Literal[False]: ... def is_none(value: Any) -> bool: return value is None def func(value: Optional[str]): if is_none(value): reveal_type(value) # None else: reveal_type(value) # str ``` Currently this isn’t the case ```py def is_none(value: Any) -> TypeGuard[None]: return value is None def func(value: Optional[str]): if is_none(value): reveal_type(value) # None else: reveal_type(value) # str | None ``` Overloading operators more correctly, more easily ```py @overload def __eq__(self, other: ~Self) -> Literal[False]: ... @overload def __eq__(self, other: Self) -> bool: ... def __eq__(self, other: object) -> bool: return isinstance(other, self.__class__) and ... ``` Something fun ```py reveal_type(~~str) # str ``` A few questions: Can you see any issues here? Do you have ideas for use cases I haven't thought of? Does anybody want to be a co-author on this?
I really like the `NotType` idea. I've seen several real-life use-cases when I needed it. I would love to help on the PEP's draft. Drop me a line if you wish! чт, 23 дек. 2021 г. в 16:48, James H-B <gobot1234yt@gmail.com>:
Hi,
I'd like to write a PEP for a new typing construct: NotType which, would be equivalent to any type but x. There has been an issue open on the typing repo for quite some time about this ( https://github.com/python/typing/issues/801). After the recent-ish proposal for an IntersectionType, this would be a nice-to-have feature as it would complete the set theory operations for typed Python. I'd also like to propose implementing type.__invert__ to make this work, so it has built-in syntax like intersection and union as this is the closest thing to a not operator.
Usecases:
Something that takes any non-empty string ```py def fn(x: str & ~Literal[""]): ... ``` This also has nice integration with PEP 675 where the type of a Literal str is important. If you’d want to call this from any str you’d need a type guard. ```py def guard(x: str) -> TypeGuard[str & ~Literal[“”]]: return isinstance(x, str) and x != “” # or maybe bool(x) although this is probably much harder to implement ```
Something that takes any value but None ```py def fn(x: ~Literal[None]): ... ```
Something that requires not an integer ```py def not_an_int(x: Any) -> TypeGuard[~int]: ... ``` This also allows for narrowing in the negative case which is another requested feature (https://github.com/python/typing/issues/926) ```py @overload def is_none(value: None) -> Literal[True]: ... @overload def is_none(value: ~Literal[None]) -> Literal[False]: ... def is_none(value: Any) -> bool: return value is None
def func(value: Optional[str]): if is_none(value): reveal_type(value) # None else: reveal_type(value) # str ``` Currently this isn’t the case ```py def is_none(value: Any) -> TypeGuard[None]: return value is None
def func(value: Optional[str]): if is_none(value): reveal_type(value) # None else: reveal_type(value) # str | None ```
Overloading operators more correctly, more easily ```py @overload def __eq__(self, other: ~Self) -> Literal[False]: ... @overload def __eq__(self, other: Self) -> bool: ... def __eq__(self, other: object) -> bool: return isinstance(other, self.__class__) and ... ```
Something fun ```py reveal_type(~~str) # str ```
A few questions: Can you see any issues here? Do you have ideas for use cases I haven't thought of? Does anybody want to be a co-author on this? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: n.a.sobolev@gmail.com
As a maintainer of pyright, I can say that implementing and maintaining support for a "Not Type" would be an immense amount of work within a static type checker. Assuming that all of the type rules could be worked out (and I'm skeptical of that — see below), it would require updating and expanding many parts of the type checking logic. The same is probably true for all of the runtime type checking libraries. I personally don't see enough value to justify the complexity this feature would add to the type system or the effort it would take to spec and resolve all of the issues. I can't think of any case where I've ever wanted this, and I don't find the provided examples all that compelling. It looks like the examples are also relying on some form of intersection type (an `&` operator), which presumably adds yet more complexity. Have you seen a construct like this in any other language? I'm not aware of any, but if you can find an example, it would be informative to look at how that language works through the many challenges this construct would add. If I understand the proposal correctly, a NotType would be equivalent to an "Any" except for the excluded type(s). That would be bad for strict type checking because most static type checking rules would be disabled for that symbol (as they are with "Any"). That means it would "blind" type checkers to many code errors. In general, we discourage people from using `Any` today for this reason. A NotType would also be bad for language server functionality like completion suggestions (Intellisense) because no completion suggestions can be offered for an "Any" type. If you pursue this idea, you'll need to work out all of the ways this feature intersects with existing type features. Here are a few off the top of my head: * What are the subtyping rules for types and NotTypes? * In what ways are Any and NotType equivalent, and in what ways are they not? * How does a NotType interact with generic classes? For example, what does `~list[T]` mean if `T` is used elsewhere in the same function signature? * How does a NotType interact with TypeVars? For example, what does ~T mean? * How does type narrowing work for NotType? * How does a union of a Type and a NotType work? * Can a TypeVar be bound to a NotType? What does that mean? * Can a NotType be used in an isinstance call? What does that mean? * Can a NotType be used as a type argument in a generic class? If so, how does it work with existing variance rules? * How should the TypeVar constraint solver handle the case where a TypeVar is being matched to both a type and a NotType? * How does NotType work with runtime type checking? The answers to many of these questions are not obvious to me. I predict it will take lots of effort (and likely debate) to work them out. I'm sure there are many other issues I'm forgetting. Discovering and resolving the full list of issues will likely require you to implement a full reference implementation in one of the existing type checkers as part of the spec'ing process. It's possible that the feature could be constrained in its use so it doesn't need to compose with all of these other type features, but a feature that is designed from the outset not to compose is probably not desirable. -Eric -- Eric Traut Contributor to Pylance and Pyright Microsoft
Thank you for the feedback, Eric. This is something I think is important to get right. These are my initial thoughts on the matter: * What are the subtyping rules for types and NotTypes? Any subtype of the type T to a NotType should be considered an excluded type. e.g. ```py def fn(x: ~Sequence): … list = [1, 2, 3, 4] tuple = (1, 2, 3, 4) collection = {1, 2, 3, 4} fn(list) # should not pass as list is a subtype of Sequence fn(tuple) # should not pass as tuple is a subtype of Sequence fn(collection) # should pass as set is not a subtype of Sequence its a subtype of Collection which is a supertype of Sequence ``` * In what ways are Any and NotType equivalent, and in what ways are they not? I’m in agreement here, it is very similar to Any in the fact that every type, just not the type T. * How does a NotType interact with generic classes? For example, what does `~list[T]` mean if `T` is used elsewhere in the same function signature? Not an instance of that class so in this case anything that isn’t a list[T] so tuple[T, …] is fine as is int. * How does a NotType interact with TypeVars? For example, what does ~T mean? Anything that isn’t T side note: currently ~T is the repr for an invariant TypeVar which is something that would probably need to be changed Related issue: https://github.com/python/typing/issues/599 * How does type narrowing work for NotType? I’m thinking you need to use __eq__/__ne__, however is none trivial for non-literals as understanding how __eq__ behaves is very difficult. Otherwise, I’d just hope you can fall back to using an isinstance check * How does a union of a Type and a NotType work? A picture demonstrates this well where A is the Type and B is the type that’s being not’ed https://imgur.com/a/5kdE8xl * Can a TypeVar be bound to a NotType? What does that mean? Yes, the TypeVar can be used in place of anything that’s not the type T. * Can a NotType be used as a type argument in a generic class? If so, how does it work with existing variance rules? Yes, see my first answer I’d suggest yes and it should follow the same variance rules as Unions. * How should the TypeVar constraint solver handle the case where a TypeVar is being matched to both a type and a NotType? * Can a NotType be used in an isinstance call? What does that mean? I’m inclined to say yes, it should just expand to not isinstance(thing, T), I’d not be entirely sure as I think it would be dependent the team working on intersection decides to do. * How does NotType work with runtime type checking? Introspection wise this would be very similar to UnionType so it would have __args__, __parameters__, support substitution and probably support instance checks. Other points: I can see the concern here with this type being similar to Any and it being bad for static type checking ideally, the user should cover both the not type and the type cases themselves and I'd perhaps consider adding a warning about what would be in essence un-typed code. With regards to examples using intersection I think it's a feature that’s undeniably important to the type system and I agree a lot of this hinges on that being accepted as a PEP. Another concern I would have is also about ParamSpec. Would it be any function signature that doesn’t match the currently bound one? I’m not sure. Would that ever even be useful? Again I’m not sure.
Could you point us to any other language (with a somewhat similar type system to PEP 483) that has a NOT type? I'm aware of languages with Intersection types but I haven't heard of languages with NOT. Also, the only realistic use case that I have ever heard people ask for is "NOT str", as in "any sequence except str". I'd like to clarify that most of your "use cases" aren't use cases -- they are just examples. A use case is more than a code snippet, it should be a little story based in real code experience that provides an actual problem people run into regularly and for which the proposal would be a better solution than the available workarounds. On Thu, Dec 23, 2021 at 1:09 PM James H-B <gobot1234yt@gmail.com> wrote:
Thank you for the feedback, Eric. This is something I think is important to get right.
These are my initial thoughts on the matter:
* What are the subtyping rules for types and NotTypes? Any subtype of the type T to a NotType should be considered an excluded type.
e.g. ```py def fn(x: ~Sequence): … list = [1, 2, 3, 4] tuple = (1, 2, 3, 4) collection = {1, 2, 3, 4}
fn(list) # should not pass as list is a subtype of Sequence fn(tuple) # should not pass as tuple is a subtype of Sequence fn(collection) # should pass as set is not a subtype of Sequence its a subtype of Collection which is a supertype of Sequence ```
* In what ways are Any and NotType equivalent, and in what ways are they not? I’m in agreement here, it is very similar to Any in the fact that every type, just not the type T.
* How does a NotType interact with generic classes? For example, what does `~list[T]` mean if `T` is used elsewhere in the same function signature? Not an instance of that class so in this case anything that isn’t a list[T] so tuple[T, …] is fine as is int.
* How does a NotType interact with TypeVars? For example, what does ~T mean?
Anything that isn’t T
side note: currently ~T is the repr for an invariant TypeVar which is something that would probably need to be changed Related issue: https://github.com/python/typing/issues/599
* How does type narrowing work for NotType? I’m thinking you need to use __eq__/__ne__, however is none trivial for non-literals as understanding how __eq__ behaves is very difficult. Otherwise, I’d just hope you can fall back to using an isinstance check
* How does a union of a Type and a NotType work? A picture demonstrates this well where A is the Type and B is the type that’s being not’ed https://imgur.com/a/5kdE8xl
* Can a TypeVar be bound to a NotType? What does that mean? Yes, the TypeVar can be used in place of anything that’s not the type T.
* Can a NotType be used as a type argument in a generic class? If so, how does it work with existing variance rules? Yes, see my first answer
I’d suggest yes and it should follow the same variance rules as Unions. * How should the TypeVar constraint solver handle the case where a TypeVar is being matched to both a type and a NotType?
* Can a NotType be used in an isinstance call? What does that mean? I’m inclined to say yes, it should just expand to not isinstance(thing, T), I’d not be entirely sure as I think it would be dependent the team working on intersection decides to do.
* How does NotType work with runtime type checking? Introspection wise this would be very similar to UnionType so it would have __args__, __parameters__, support substitution and probably support instance checks.
Other points:
I can see the concern here with this type being similar to Any and it being bad for static type checking ideally, the user should cover both the not type and the type cases themselves and I'd perhaps consider adding a warning about what would be in essence un-typed code.
With regards to examples using intersection I think it's a feature that’s undeniably important to the type system and I agree a lot of this hinges on that being accepted as a PEP.
Another concern I would have is also about ParamSpec. Would it be any function signature that doesn’t match the currently bound one? I’m not sure. Would that ever even be useful? Again I’m not sure. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Technically this is something that TypeScript supports with conditional types, Exclude, and intersections (this is not my own example). https://www.typescriptlang.org/play?#code/C4TwDgpgBAcg9sAggHkQGigFQHxQLxQCiA...
Cool. Do you know if it’s very useful in TS? Was it hard to implement there? On Thu, Dec 23, 2021 at 13:32 James H-B <gobot1234yt@gmail.com> wrote:
Technically this is something that TypeScript supports with conditional types, Exclude, and intersections (this is not my own example).
https://www.typescriptlang.org/play?#code/C4TwDgpgBAcg9sAggHkQGigFQHxQLxQCiA... _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido (mobile)
I think Eric would probably have the best insight into this as I have never used typescript for anything actually useful.
I have done some more digging, I can't find much on implementing the conditional types and Exclude and what not but I have actually found a PR for a type negation operator https://github.com/microsoft/TypeScript/pull/29317 and judging by the commit size yes this does look like a rather large undertaking as Eric mentioned.
Guido van Rossum wrote:
Also, the only realistic use case that I have ever heard people ask for is "NOT str", as in "any sequence except str".
I've just come across another good example: an int, but not bool. Here's an example over on Discuss: https://discuss.python.org/t/contravariant-typing-type/12741/1 How does one statically enforce the rule "an int, but not bool"? Or *only* an int, no subclasses? Doing it at runtime is easy. -- Steve
Also, the only realistic use case that I have ever heard people ask for is "NOT str", as in "any sequence except str".
Found this thread while trying to address some of the "type: ignore" comments in pandas. A couple of examples where this feature would be useful: A bunch of functions are just optimized isinstance checks where we'd like to overload: ``` @overload def is_float(x: float | np.floating) -> True: ... @overload def is_float(x: AnythingBut[float | np.floating]) -> False: ... ``` Similar but a bit more complicated, we have a `def find_common_type(types: list[np.dtype | ExtensionDtype]) -> np.dtype | ExtensionDtype:` where we'd like an overload to declare `def find_common_type(types: list[np.dtype]) -> np.dtype`. AFAICT we can't do this without an AnythingBut[X] annotation.
On Sun, Jan 2, 2022 at 8:32 PM <jbrockmendel@gmail.com> wrote:
Also, the only realistic use case that I have ever heard people ask for is "NOT str", as in "any sequence except str".
Found this thread while trying to address some of the "type: ignore" comments in pandas. A couple of examples where this feature would be useful:
A bunch of functions are just optimized isinstance checks where we'd like to overload:
``` @overload def is_float(x: float | np.floating) -> True: ...
@overload def is_float(x: AnythingBut[float | np.floating]) -> False: ... ```
Maybe this could use TypeGuard: def is_float(x: object) -> TypeGuard[float|np.floating]: ... (Though perhaps the new, *precise* typeguard that's being discussed in another thread <https://mail.python.org/archives/list/typing-sig@python.org/thread/EMUD2D424...> on this list would be better.)
Similar but a bit more complicated, we have a `def find_common_type(types: list[np.dtype | ExtensionDtype]) -> np.dtype | ExtensionDtype:` where we'd like an overload to declare `def find_common_type(types: list[np.dtype]) -> np.dtype`.
AFAICT we can't do this without an AnythingBut[X] annotation.
For this, I think overload actually works just fine. E.g. from typing import * @overload def foo(a: list[int]) -> int: ... @overload def foo(a: list[str]) -> str: ... @overload def foo(a: list[int|str]) -> int|str: ... def foo(a): ... reveal_type(foo([1,2,3])) # int reveal_type(foo(['a', 'b'])) # str x: list[int|str] = [1, 2, 'a', 'b'] reveal_type(foo(x)) # int|str -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
Guido van Rossum wrote:
Similar but a bit more complicated, we have a `def find_common_type(types: list[np.dtype | ExtensionDtype]) -> np.dtype | ExtensionDtype:` where we'd like an overload to declare `def find_common_type(types: list[np.dtype]) -> np.dtype`. AFAICT we can't do this without an AnythingBut[X] annotation. For this, I think overload actually works just fine. E.g. from typing import * @overload def foo(a: list[int]) -> int: ... @overload def foo(a: list[str]) -> str: ... @overload def foo(a: list[int|str]) -> int|str: ... def foo(a): ... reveal_type(foo([1,2,3])) # int reveal_type(foo(['a', 'b'])) # str x: list[int|str] = [1, 2, 'a', 'b'] reveal_type(foo(x)) # int|str
Another solution that might work would be to use a type variable. ``` from typing import TypeVar T = TypeVar("T") def foo(a: list[T]) -> T: ... ```
participants (7)
-
Eric Traut -
Guido van Rossum -
James H-B -
jbrockmendel＠gmail.com -
Marc Mueller -
steve＠pearwood.info -
Никита Соболев