I occasionally receive questions from pyright users about extending type narrowing to handle cases that are not possible with today’s Python type system.
Typescript provides a useful facility called [user-defined type guards]( https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defin…) that allows users to extend the notion of type narrowing. It would be straightforward to add this idea to Python. Do others think this would be useful, and is there any interest in standardizing this?
Here’s a concrete proposal. We could add a new generic type called `TypeGuard` in the `typing` module. It would be a subclass of bool, defined simply as:
```python
class TypeGuard(bool, Generic[_T]):
pass
```
If any function or method accepts one parameter and returns a `TypeGuard` type, a type checker would assume that the argument passed to this function can be narrowed to the type specified in the `TypeGuard` type argument.
Here are a few examples:
```python
from typing import Any, Generic, List, Tuple, TypeGuard, TypeVar
_T = TypeVar("_T")
def is_two_element_tuple(val: Tuple[_T, ...]) -> TypeGuard[Tuple[_T, _T]]:
return TypeGuard(len(val) == 2)
def foo(names: Tuple[str, ...]):
if is_two_element_tuple(names):
reveal_type(names) # Tuple[str, str]
def is_string_list(val: List[object]) -> TypeGuard[List[str]]:
return TypeGuard(all(isinstance(x, str) for x in val))
def bar(stuff: List[Any]):
if is_string_list(stuff):
reveal_type(stuff) # List[str]
```
Thoughts? Suggestions?
-Eric
---
Eric Traut
Contributor to Pyright and Pylance
Microsoft Corp.
PEP 561 indicates that “package maintainers who wish to support type checking of their code MUST add a marker file named py.typed…”. It doesn’t define what “support type checking” means or what expectations are implied. This has led to a situation where packages claim to support type checking but omit many type annotations. There’s currently no tooling that validates the level of “type completeness” for a package, so even well-intentioned package maintainers are unable to confirm that their packages are properly and completely annotated. This leads to situations where type checkers and language servers need to fall back on type inference, which is costly and gives inconsistent results across tools. Ideally, all py.typed packages would have their entire public interface completely annotated.
I’m working on a new feature in Pyright that allows package maintainers to determine whether any of the public symbols in their package are missing type annotations. To do this, I need to clearly define what constitutes a “public symbol”. In most cases, the rules are pretty straightforward and follow the naming guidelines set forth in PEP 8 and PEP 484. For example, symbols that begin with an underscore are excluded from the list of public symbols.
One area of ambiguity is related to import statements. PEP 484 indicates that within stub files, a symbol is not considered exported unless it is used within an import statement of the form `import x as y` or `from x import y as z` or `from x import *`. The problem is that this rule applies only to “.pyi” files and not to “.py” files. For packages that use inlined types, it’s ambiguous whether an import statement of the form `import x` or `from y import x` should treat `x` as a public symbol that is exported from that module.
I can think of a few solutions here:
1. For py.typed packages, type checkers should always apply PEP 484 import rules for “.py” files. If a symbol `x` is imported with an `import x` or `from y import x`, it is treated as “not public”, and any attempt to import it from another package will result in an error.
2. For py.typed packages, PEP 484 rules are _not_ applied for import statements. This maintains backward compatibility. Package maintainers can opt in to PEP 484 rules using some well-defined mechanism. For example, we could define a special flag “stub_import_rules” that can be added to a “py.typed” file. Type checkers could then conditionally use PEP 484 rules for imports.
Option 1 will likely break some assumptions for existing packages. Option 2 avoids that break, but it involves more complexity.
Any suggestions? Thoughts?
-Eric
---
Eric Traut
Contributor to Pyright and Pylance
Microsoft
Consider these two similar functions:
```
from typing import Union
def f() -> int:
x: Union[int, str] = 42
return x # Error here
def g() -> int:
x: Union[int, str]
x = 42
return x # But not here
```
Quite a while ago (2016-2017) there was a long discussion about this in the
mypy tracker: https://github.com/python/mypy/issues/2008
Not only is it surprising that these two functions are type-checked
differently, there are differences across type checkers. I tried mypy,
pytype, pyre and pyright. All of them consider the second function correct.
But all except pyright produce a type error on the first function,
something along the lines of
```
t.py:5: error: Incompatible return value type (got "Union[int, str]",
expected "int")
```
(It's possible that there are configuration settings to change the
behavior; I didn't look into this.)
I think it would be in everybody's interest if all type checkers agreed on
this. I also think it's surprising to users that the two functions are
treated differently. (But as the old mypy issue shows, not everybody agrees
they should be treated the same either. :-)
There's another twist (note the return type is str here):
```
from typing import Any
def h() -> str:
x: Any = 42
return x
```
This is an error in pyre but not in mypy, pytype or pyright. And for good
measure the other variant:
```
def i() -> str:
x: Any
x = 42
return x
```
This is accepted by mypy and pyright, but an error to pyre and pytype.
To summarize, it seems the type of x will be as follows:
```
x: Union[int, str] = 42
# mypy: Union; pytype: Union; pyre: Union; pyright: int
# majority doesn't narrow
x: Union[int, str]
x = 42
# mypy: int; pytype: int; pyre: int; pyright: int
# everybody narrows
x: Any = 42
# mypy: Any; pytype: Any; pyre: int; pyright: Any
# majority doesn't narrow
x: Any
x = 42
# mypy: Any; pytype: int; pyre: int; pyright: Any
# even split
```
Looking at this, I'd say if we follow the majority, we would have to make
the following changes:
- pyright should stop narrowing for `x: Union[..] = value`
- pyre should stop narrowing for `x: Any = value`
- mypy and pyright should start narrowing for `x: Any; x = value`
(The latter recommendation is debatable, since it's an even split, but I
like it better if Union and Any are treated similarly in this case.)
Let the debate begin.
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>
Let's say I have the following function:
```python
def spam(a: str, b: str, c: str, d: int = 42): ...
```
Now if I call that with:
```python
spam(*"a/b".split("/", 2), "hello")
```
That will execute fine and actually result in appropriate types for those
arguments.
Unfortunately both Pylance and mypy say that call is incompatible because
of `str.split(...) -> List`. And since 'typing' lacks an concept of a sized
list like it does for tuples, I don't know how to make this pass.
Am I overlooking something? Or is this a gap due to there not being an
equivalent of 'SizedList' in 'typing'?
Hello everyone. I created an issue on the mypy GitHub here, as I didn't know about this mailing list: https://github.com/python/mypy/issues/9472
I have been a Python programmer for 12 years now, and in my day-to-day job, I use TypeScript constantly now. The type checking in TypeScript is now very mature and quite easy to use, and I find myself every day wanting more for my back end Python code of what I already have for my front end TypeScript code. I think one lesson I've learned from my 3+ years of TypeScript use is that the TypeScript utility types are very useful, in particular the `Parameters` and `ReturnType` utility types, as documented here: https://www.typescriptlang.org/docs/handbook/utility-types.html#parameterst…
Shantanu on GitHub was kind enough to point out that these utility types have been brought up before in a PEP, and rejected. See here: https://www.python.org/dev/peps/pep-0612/#defining-parametersof
I believe it is worth reconsidering the efficacy of these utility types. As an end-user of TypeScript and Pyright, I can say that they make a lot of sense to me, and they are easy to explain to the people I train and manage. The specifics of how they are implemented I can't comment on, as I don't understand the full complexity of the Python type systems, but I can certainly say that they are obvious to the people I work with, and very useful, as we use them often in TypeScript code. The prior art on the part of TypeScript seems to indicate the utility types are useful as a general concept.
I believe that if these utility types are implemented, they will form a part of solving the problem of how to correctly specify types for function decorators. Because it will be very easy to extract the parameters and return type of a Callable, you will be able to write a new type for a function that aliases one part and changes another part, such as using the same arguments for a function F as before, with a new return type, or the same return type as before, with a ParamSpec or list of arguments, etc.
What does everyone else think about this topic?
This has previously been discussed here:
https://github.com/python/typing/issues/566.
In typeshed we have found that there a quite a few cases where functions
can return two or more different, incompatible types, that can't be
distinguished by the input types alone. A few examples:
* ip_address() can return either an IPv4Address or an IPv6Address,
depending on the input string.
* float ** float returns either a float or a complex. Currently,
typeshed marks this as returning just float.
* json.loads() et al. can return a selection of types, depending on
the input.
* urlopen() can return either an HTTPResponse or an addinfourl.
There are more examples in the linked issue.
In these cases, returning a Union would technically be correct, but
would also be inconvenient for callers that know what type to expect.
We usually give up and just return Any.
My suggestion would be to add a type tentatively called "AnyOf",
that acts as an unsafe union:
ip_address(address: str) -> AnyOf[IPv4Address, IPv6Address]
I believe this could be useful for type checkers. While the use of
AnyOf is unsafe, it is safer than just treating these return values
as Any. It's also theoretically possible to use some clever
narrowing. That said, there has been some pushback from the mypy
core team in the linked issue, especially since this is most likely
complex to implement. An easy way for type checkers to implement
this, without sacrificing any already existing type safety is to
just treating AnyOf exactly like Any. At least as a first step.
Should type checkers implement full support for AnyOf later, we
could already have better types in typeshed ready to use.
But in the last years, typeshed has also gained users outside of
type checkers. For example, PyCharm and jedi use it for
autocompletion. I believe that AnyOf could be useful for those
projects, as well as others.
Is there interest in a feature like this? I would be willing to
write a PEP and contribute the AnyOf = Any solution to mypy.
Unfortunately I don't have the bandwidth to learn how to implement
this "properly" in mypy.
- Sebastian
A few thoughts here (and apologies for the long post). First, I think we should try to avoid conflating _when_ to apply type narrowing with _how_ to apply type narrowing (i.e. the resulting type when narrowing is applied). We may want to standardize aspects of both the _when_ and the _how_, but it will help the discussion if we keep these two issues separate as much as possible.
I will first focus on _when_ type narrowing should be applied.
It appears that we have one primary point of contention — whether narrowing should be applied for all assignments. Some have argued that narrowing should be skipped if the assignment statement includes a variable type annotation.
In short, should these two cases be treated in a consistent manner?
```python
# (1)
a: T = x
# (2)
a: T
a = x
```
We (the maintainers of pyright) have a strong conviction that type narrowing should not be tied in any way with the location of type declarations. The two code samples above should be treated in a consistent manner, and narrowing should be applied in both cases.
Here’s the thought process behind this position. I’m hopeful that most of these points (at least through point 5) will be uncontentious and will form a solid basis for our point of view.
**Point 1**: A type annotation provided for a variable declares that the variable should be assigned only values whose types are compatible with the declared type. Any attempt to assign a value whose type is not compatible with a variable’s declared type should be flagged as an error by a type checker.
**Point 2**: A symbol can have only one declared type. A symbol can’t be declared as a function and in the same scope redeclared as a variable of type “str”. An instance variable cannot be declared as an `int` in one method but a `str` in another. A local variable cannot be declared as an `int` within an “if” clause and declared as a `str` within the “else” clause. Any attempt to redeclare a symbol in an incompatible manner should be flagged as an error.
**Point 3**: PEP 526 does not (nor should it) dictate that a variable’s type declaration must be provided upon “first use” of that variable. In fact, there’s no clear definition of “first use” in Python. An instance variable (e.g. “self.foo”) might be assigned a value in two different methods, and the order in which these methods are called at runtime cannot be known statically. A variable’s type declaration can therefore appear anywhere, in any order, and it must apply to all uses of that variable.
```python
# These are semantically equivalent.
class C1:
def f1(self):
self.var: Optional[int] = None
def f2(self):
self.var = 4
class C1:
def f1(self):
self.var = None
def f2(self):
self.var: Optional[int] = 4
```
**Point 4**: Absent any additional information about a variable whose type has been declared, a type checker should assume that it contains a value with the declared type.
```python
def foo(a: Optional[int]):
reveal_type(a) # “Optional[int]” at this point
class C1:
def __init__(self):
self.var: Optional[str] = None
def f1(self):
reveal_type(self.var) # “Optional[str]” at this point
```
**Point 5**: Type narrowing is a process by which a type checker can use additional contextual knowledge within a control flow graph to understand more detailed information about a variable’s contents _at a particular point in the code flow_. Knowledge of narrower types is beneficial because it allows the type checker to perform more accurate check, minimize false positives, and provide more accurate completion suggestions for developers. Additional knowledge about a variable’s type that can be discerned from the control flow graph should not be discarded or ignored!
**Point 6**: Type narrowing should be independent of (and not influenced by) the location of a variable’s type declaration. As we noted above, the location of a variable’s declaration is incidental, and the type declaration is independent of code flow considerations.
Any other interpretation results in inconsistent behaviors, an inconsistent mental model for developers — especially those who are new to type annotations, and hard-to-explain edge cases. Alternate interpretations also result in false-positive errors because they ignore valuable information derived from narrowing.
Some have argued that applying type narrowing in an inconsistent manner can be a useful technique for handling cases where the assigned variable is subsequently used to infer an invariant type argument for a generic type.
```python
a: Union[int, str] = 1 # Type is a is narrowed to int
b = [a] # Inferred type of b is List[int]
c: List[Union[int, str]] = b # Error
```
In this example, `b` does not have a declared type, so its type needs to be inferred. There are cases (like this) where a type cannot be inferred in an unambiguous manner, and the normal heuristics for type inference can produce an answer that is inconsistent with the programmer’s intent. The appropriate fix for removing this ambiguity is to introduce a type declaration and eliminate the need for type inference. In this case, the ambiguity is with the inference of `b`, so a type declaration should be added as follows:
```python
a: Union[int, str] = 1
b: List[Union[int, str]] = [a]
c = b
```
Solving the inference ambiguity for `b` by introducing an inconsistent type narrowing rule for `a` is not the right solution!
Finally, it’s worth noting that there are many different conditions where type narrowing can be applied, involving a variety of techniques. For example, pyright currently applies type narrowing in the following cases:
1. Assignments
2. Positive (“if”) and negative (“else”) conditional code flow constructs where the conditional expression contains:
* A combination of “and”, “or” or “not” boolean operators
* “is None” or “is not None” tests
* “type(X) is Y” or “type(X) is not Y” tests
* “X.Y == <literal>” or “X.Y != <literal>” tests where Y is used to discriminate among subtypes
* “X is Y” or “X is not Y” enum tests
* “X in Y” tests
* isinstance() tests
* issubclass() tests
* callable() tests
* Truthy or Falsy tests
Standardizing this list is going to be challenging because different type checkers implement different techniques, and this list will undoubtedly continue to grow over time as new innovations are introduced. It may still be worth trying to standardize some aspects of assignment-based narrowing, but this will invariably leave behavioral differences between type checkers. We should collectively decide on whether the benefits of standardizing these behaviors outweigh the downsides.
Now, I’ll focus on _how_ type narrowing is applied, specifically for assignments.
We seem to have reached some level of agreement about the following:
1. Narrowing should eliminate subtypes within a union where possible
2. Narrowing should replace base classes and protocols with derived classes and derived protocols where possible
3. Narrowing should not be applied if the declared type is “Any”
Guido identified a bug in pyright where rule #2 was not being applied when the RHS of the assignment was a call to a constructor. This is a regression that I introduced recently. I’m working on a fix as well as a more comprehensive test suite that should prevent similar regressions in the future.
As I started to fill out this test suite, I ran across additional cases that are not covered by the list above. The most notable case involves a type declaration that includes a generic type (or a generic protocol) where one or more type arguments is specified as “Any”. For example:
```python
def func(p: List[int]):
a: List[Any] = p
reveal_type(a) # ?
```
If we agree that “Any” is meant as a way to disable further type checking, then it probably follows that the `reveal_type` in the above example should indicate that `a` is type `List[Any]` rather than `List[int]`. Do others agree?
If so, how far do we take this? Should it apply to derived classes and protocols? I think the answer is probably yes, but I’m not fully convicted.
Should this nest arbitrarily or apply only to the first level of type arguments? I’m not sure.
```python
def func(p: List[int]):
a: Iterable[Any] = p
reveal_type(a) # List[Any]?
```
```python
def func(p: List[List[int]]):
a: Iterable[List[Any]] = p
reveal_type(a) # List[List[Any]]? List[List[int]]?
```
There are also interesting cases involving unions that include Any. I think these are straightforward if we apply the rules consistently.
```python
def func(p: List[int]):
a: Union[Any, Iterable[int]] = p
reveal_type(a) # Union[Any, List[int]]?
```
```python
def func(p: Union[Any, List[int]]):
a: Iterable[int] = p
reveal_type(a) # Union[Any, List[int]]?
```
Based on this exploration, I propose that we add the following fourth rule:
4. If a declared type contains an “Any” type argument, the corresponding type argument(s) in the narrowed type must also be “Any”
Here’s a partial test suite that I’ve started to compile based on these four rules. It builds on Guido’s examples using an `Animal` base class. Do these test cases look correct?
```python
from typing import Any, Dict, Generic, Iterable, List, Literal, Protocol, TypeVar, Union
_T1 = TypeVar("_T1")
_T2 = TypeVar("_T2")
_T3 = TypeVar("_T3")
class Animal(Generic[_T1, _T2]):
pass
class Bear(Animal[_T3, int]):
pass
class Donkey(Animal[int, int], Generic[_T3]):
pass
class Flyer(Protocol[_T1]):
def get_wingspan(self, p1: _T1) -> float:
raise NotImplemented
class CaveDweller(Generic[_T1]):
pass
class Bat(Animal[int, int], CaveDweller[int]):
def get_wingspan(self, p1: int) -> float:
raise NotImplemented
def s1():
b: Bear[str] = Bear()
a: Animal[str, int] = b
t: Literal["Bear[str]"] = reveal_type(a)
def s2():
a: Animal[str, int] = Bear()
t: Literal["Bear[str]"] = reveal_type(a)
def s3():
a: Animal[str, int] = Bear()
t: Literal["Bear[str]"] = reveal_type(a)
def s4():
a: Bear[Any] = Bear[int]()
t: Literal["Bear[Any]"] = reveal_type(a)
def s5():
a: Animal[Any, Any] = Bear[int]()
t: Literal["Bear[Any]"] = reveal_type(a)
def s6():
a: Union[Bat, Bear[str]] = Bear()
t: Literal["Bear[str]"] = reveal_type(a)
def s7(p: Union[Bat, Bear[int]]):
a: Animal[int, int] = p
t: Literal["Bat | Bear[int]"] = reveal_type(a)
def s8():
a: Animal[int, int] = Bear[int]()
t: Literal["Bear[int]"] = reveal_type(a)
def s9(p: Dict[str, str]):
a: Dict[str, Any] = p
t: Literal["Dict[str, Any]"] = reveal_type(a)
def s10(p: List[str]):
a: Iterable[Any] = p
t1: Literal["List[Any]"] = reveal_type(a)
b: Iterable[str] = []
t2: Literal["List[str]"] = reveal_type(b)
c: Iterable[str] = list()
t3: Literal["list[str]"] = reveal_type(c)
def s11():
a: Animal[Any, Any] = Donkey[int]()
t: Literal["Donkey[int]"] = reveal_type(a)
def s12(p: Bear[_T1]):
a: Animal[Any, int] = p
t: Literal["Bear[Any]"] = reveal_type(a)
def s13(p: Bat):
a: Flyer[int] = p
t: Literal["Bat"] = reveal_type(a)
def s14(p: Bat):
a: CaveDweller[int] = p
t: Literal["Bat"] = reveal_type(a)
```
---
Eric Traut
Microsoft
Having to always import 'typing' to use TYPE_CHECKING bugs me. It's
probably because I hide everything type-related under `if
typing.TYPE_CHECKING: ...` to clearly separate it from runtime stuff, but
this blocks me from putting the 'typing' import in that block.
Has it ever been discussed to have type checkers recognize `if
TYPE_CHECKING := False: ...` to allow skipping the 'typing' import? I guess
another option is some `__type_checking__` built-in like `__debug__`, but
defining a new built-in is a harder sell.
Hey everyone,
It's somehow September already, and time for our next tensor typing meeting!
WHEN? Monday 14th September, 9:30am San Francisco time, 5:30pm London time.
WHERE? Cyberspace, Google Meet. I'll post the link closer to the time.
WHAT?
1. Status updates
2. Alfonso: Arithmetic on tensor dimensions (Alfonso, do you wanna say
anything more on this?)
3. Matthew (short): Benchmarking stubs with thousands of overloads
I *might* also have something to present on the use of Ellipsis in the
annotations setup that Jörg and I and are working, depending on how this
week goes/how much time we have in the meeting.
Looking forward!
Matthew
Hi there,
Recently I have been working on an implementation for PEP 604 (https://www.python.org/dev/peps/pep-0604/). While working on this, I started reviewing and exploring what new Optional syntax would look like in the Python language.
I've written a rough draft of a PEP and wanted to present it to this email list before submitting a PR. I know this is a topic that has been discussed in many settings (Github issues, in person typing meetups, etc. )previously, and wanted to solicit feedback here first. Let me know if submitting a PR would be a better way to gauge sentiment and feedback on this proposal.
Draft Proposal (branch compare): https://github.com/python/peps/compare/master...MaggieMoss:optional-syntax-…
Gist: https://gist.github.com/MaggieMoss/c848cb3a581979f445d075c15629c950
Thanks again,
Maggie
