Mailman 3 September 2020 - Typing-sig

Any way to express that a dict is expected to have a key(s) but don't care about other keys?
by Brett Cannon 23 Dec '22

23 Dec '22

Looking at TypedDict it seems that it does not like the idea of superfluous keys existing in a dict. In my case I have a REST call returning JSON for an error and all I'm guaranteed is to get an "error" key. But there may be extra keys that provide more details about the specific error. Unfortunately I don't know the exhaustive list of keys so I can't use a TypedDict with `totality == False` to try to be complete. I can't think of any way to somehow say "require/expect this one key, but other keys are OK". Am I missing anything that would let me do this?

16 34

Explicit Type Aliases
by Shannon Zhu 17 Aug '22

17 Aug '22

Hi everyone, I’d like to continue the discussion from our last typing summit on explicit type aliases. A quick summary of the current state and the proposal -- Current state: ``` from typing import List x = List[int] # considered a type alias y : Type[List[int]] = List[int] # considered an expression z : Any # considered a type alias z : x = [] # fine z : y = [] # invalid type error ``` Proposal (with explicit aliases): ``` from typing import List, TypeAlias x = TypeAlias[List[int]] # considered a type alias y = List[int] # considered an expression z : Any # considered an expression z = TypeAlias[Any] # considered a type alias reveal_type(x) # Type[List[int]] reveal_type(y) # Any (return type of __getitem__) z : x = [] # fine z : y = [] # invalid type error ``` Some of the benefits we’re hoping to gain from explicit type aliases: * Clearly distinguish between an unannotated global assignment and a type alias, especially when parsing forward-referencing string annotations. * Avoid the confusing case of type aliases in which some part of the type is invalid or undefined. This would facilitate warnings on invalid types at the alias definition rather than later on when the alias is used. * Make valid and invalid types more intuitive to Python programmers by shifting from delineation of non-aliases with a meta annotation toward delineation of aliases with `TypeAlias`. * Remove special treatment of values annotated as `Any`, which currently break all type aliasing rules. This will also allow us to clean up some of the distinctions we maintain between unannotated values and explicit Anys. Note: We also considered denoting the alias as an annotation (e.g. `x: TypeAlias = int`) as an alternative to the syntax above. I’m interested to hear your thoughts and suggestions on making type aliasing explicit going forward – we’d like to start implementing a version of this in Pyre soon! Shannon

7 10

User-defined type guards
by Eric Traut 17 Jan '21

17 Jan '21

I occasionally receive questions from pyright users about extending type narrowing to handle cases that are not possible with today’s Python type system. Typescript provides a useful facility called [user-defined type guards]( https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defin…) that allows users to extend the notion of type narrowing. It would be straightforward to add this idea to Python. Do others think this would be useful, and is there any interest in standardizing this? Here’s a concrete proposal. We could add a new generic type called `TypeGuard` in the `typing` module. It would be a subclass of bool, defined simply as: ```python class TypeGuard(bool, Generic[_T]): pass ``` If any function or method accepts one parameter and returns a `TypeGuard` type, a type checker would assume that the argument passed to this function can be narrowed to the type specified in the `TypeGuard` type argument. Here are a few examples: ```python from typing import Any, Generic, List, Tuple, TypeGuard, TypeVar _T = TypeVar("_T") def is_two_element_tuple(val: Tuple[_T, ...]) -> TypeGuard[Tuple[_T, _T]]: return TypeGuard(len(val) == 2) def foo(names: Tuple[str, ...]): if is_two_element_tuple(names): reveal_type(names) # Tuple[str, str] def is_string_list(val: List[object]) -> TypeGuard[List[str]]: return TypeGuard(all(isinstance(x, str) for x in val)) def bar(stuff: List[Any]): if is_string_list(stuff): reveal_type(stuff) # List[str] ``` Thoughts? Suggestions? -Eric --- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.

12 59

Defaults for type vars
by Sebastian Rittau 25 Nov '20

25 Nov '20

This was previously discussed in https://github.com/python/typing/issues/307. It would often be useful to be able to specify a default type for type vars. A few examples: * Counter is most likely used with ints in the vast majority of cases. Being able to write "Counter" instead of "Counter[int]" makes sense to me. * Currently, when marking a previously non-generic type as generic, it will break existing type annotations. Having a default value works around this. To use Counter as an example again: Currently it is non-generic, but making it generic would break existing annotations. Adding a default of "int" would work around this problem. * Another example from the issue above: Using a default of None can prevent overloads, as in the following example: "def nullcontext(enter_result: _T = ...) -> ContextManager[_T]: ...". Without the default we need an explicit overload for the case where no argument is provided, because otherwise the type checker can't know the type of _T. While this is not a big problem in stubs, I find overloads quite obnoxious in implementations. While of course not completely comparable, I find myself using generic defaults quite often when working with TypeScript. A few examples as anectodal evidence: * We have functions that fetch JSON from an API. They are generic over the return type and default to a general JSON type, but can be overwritten if the response shape is known. (Because we control the API.) * We have a few cases, where an object is generic over a list of strings. Take a class wrapping HTML <select> elements. Sometimes all the possible options are known to the application, and have semantic meaning. (For example for choosing a view.) So it makes sense to declare a class like "class Select[Literal["foo", "bar"]]" (using Python syntax). It's then possible to have "Select.value" have the type "Literal["foo", "bar"]". In other cases, the possible options are unknown (taken from a database, for example). In that case, using "class Select[str]" would make sense. The latter is also a sensible default. * Generally I tend to use more type variables per type in TypeScript than in Python, partially due to the fact that some of them can be set to sensible defaults. While it would be obnoxious to always define four generics everytime you use a type, in typeshed I only have to define one of them with the others having sensible defaults. What do other people think about this and how would we proceed if this is deemed useful? - Sebastian

3 2

Update: Work on Optional Syntax PEP
by maggiemoss＠fb.com 07 Oct '20

07 Oct '20

Hi there, Recently I have been working on an implementation for PEP 604 (https://www.python.org/dev/peps/pep-0604/). While working on this, I started reviewing and exploring what new Optional syntax would look like in the Python language. I've written a rough draft of a PEP and wanted to present it to this email list before submitting a PR. I know this is a topic that has been discussed in many settings (Github issues, in person typing meetups, etc. )previously, and wanted to solicit feedback here first. Let me know if submitting a PR would be a better way to gauge sentiment and feedback on this proposal. Draft Proposal (branch compare): https://github.com/python/peps/compare/master...MaggieMoss:optional-syntax-… Gist: https://gist.github.com/MaggieMoss/c848cb3a581979f445d075c15629c950 Thanks again, Maggie

6 9

Validating type completeness for py.typed packages
by Eric Traut 02 Oct '20

02 Oct '20

PEP 561 indicates that “package maintainers who wish to support type checking of their code MUST add a marker file named py.typed…”. It doesn’t define what “support type checking” means or what expectations are implied. This has led to a situation where packages claim to support type checking but omit many type annotations. There’s currently no tooling that validates the level of “type completeness” for a package, so even well-intentioned package maintainers are unable to confirm that their packages are properly and completely annotated. This leads to situations where type checkers and language servers need to fall back on type inference, which is costly and gives inconsistent results across tools. Ideally, all py.typed packages would have their entire public interface completely annotated. I’m working on a new feature in Pyright that allows package maintainers to determine whether any of the public symbols in their package are missing type annotations. To do this, I need to clearly define what constitutes a “public symbol”. In most cases, the rules are pretty straightforward and follow the naming guidelines set forth in PEP 8 and PEP 484. For example, symbols that begin with an underscore are excluded from the list of public symbols. One area of ambiguity is related to import statements. PEP 484 indicates that within stub files, a symbol is not considered exported unless it is used within an import statement of the form `import x as y` or `from x import y as z` or `from x import *`. The problem is that this rule applies only to “.pyi” files and not to “.py” files. For packages that use inlined types, it’s ambiguous whether an import statement of the form `import x` or `from y import x` should treat `x` as a public symbol that is exported from that module. I can think of a few solutions here: 1. For py.typed packages, type checkers should always apply PEP 484 import rules for “.py” files. If a symbol `x` is imported with an `import x` or `from y import x`, it is treated as “not public”, and any attempt to import it from another package will result in an error. 2. For py.typed packages, PEP 484 rules are _not_ applied for import statements. This maintains backward compatibility. Package maintainers can opt in to PEP 484 rules using some well-defined mechanism. For example, we could define a special flag “stub_import_rules” that can be added to a “py.typed” file. Type checkers could then conditionally use PEP 484 rules for imports. Option 1 will likely break some assumptions for existing packages. Option 2 avoids that break, but it involves more complexity. Any suggestions? Thoughts? -Eric --- Eric Traut Contributor to Pyright and Pylance Microsoft

4 14

Unions and initialization across type checkers
by Guido van Rossum 25 Sep '20

25 Sep '20

Consider these two similar functions: ``` from typing import Union def f() -> int: x: Union[int, str] = 42 return x # Error here def g() -> int: x: Union[int, str] x = 42 return x # But not here ``` Quite a while ago (2016-2017) there was a long discussion about this in the mypy tracker: https://github.com/python/mypy/issues/2008 Not only is it surprising that these two functions are type-checked differently, there are differences across type checkers. I tried mypy, pytype, pyre and pyright. All of them consider the second function correct. But all except pyright produce a type error on the first function, something along the lines of ``` t.py:5: error: Incompatible return value type (got "Union[int, str]", expected "int") ``` (It's possible that there are configuration settings to change the behavior; I didn't look into this.) I think it would be in everybody's interest if all type checkers agreed on this. I also think it's surprising to users that the two functions are treated differently. (But as the old mypy issue shows, not everybody agrees they should be treated the same either. :-) There's another twist (note the return type is str here): ``` from typing import Any def h() -> str: x: Any = 42 return x ``` This is an error in pyre but not in mypy, pytype or pyright. And for good measure the other variant: ``` def i() -> str: x: Any x = 42 return x ``` This is accepted by mypy and pyright, but an error to pyre and pytype. To summarize, it seems the type of x will be as follows: ``` x: Union[int, str] = 42 # mypy: Union; pytype: Union; pyre: Union; pyright: int # majority doesn't narrow x: Union[int, str] x = 42 # mypy: int; pytype: int; pyre: int; pyright: int # everybody narrows x: Any = 42 # mypy: Any; pytype: Any; pyre: int; pyright: Any # majority doesn't narrow x: Any x = 42 # mypy: Any; pytype: int; pyre: int; pyright: Any # even split ``` Looking at this, I'd say if we follow the majority, we would have to make the following changes: - pyright should stop narrowing for `x: Union[..] = value` - pyre should stop narrowing for `x: Any = value` - mypy and pyright should start narrowing for `x: Any; x = value` (The latter recommendation is debatable, since it's an even split, but I like it better if Union and Any are treated similarly in this case.) Let the debate begin. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

12 19

Supporting list lengths
by Brett Cannon 24 Sep '20

24 Sep '20

Let's say I have the following function: ```python def spam(a: str, b: str, c: str, d: int = 42): ... ``` Now if I call that with: ```python spam(*"a/b".split("/", 2), "hello") ``` That will execute fine and actually result in appropriate types for those arguments. Unfortunately both Pylance and mypy say that call is incompatible because of `str.split(...) -> List`. And since 'typing' lacks an concept of a sized list like it does for tuples, I don't know how to make this pass. Am I overlooking something? Or is this a gap due to there not being an equivalent of 'SizedList' in 'typing'?

7 11

Reconsidering ParametersOf and ReturnType
by dev＠w0rp.com 24 Sep '20

24 Sep '20

Hello everyone. I created an issue on the mypy GitHub here, as I didn't know about this mailing list: https://github.com/python/mypy/issues/9472 I have been a Python programmer for 12 years now, and in my day-to-day job, I use TypeScript constantly now. The type checking in TypeScript is now very mature and quite easy to use, and I find myself every day wanting more for my back end Python code of what I already have for my front end TypeScript code. I think one lesson I've learned from my 3+ years of TypeScript use is that the TypeScript utility types are very useful, in particular the `Parameters` and `ReturnType` utility types, as documented here: https://www.typescriptlang.org/docs/handbook/utility-types.html#parameterst… Shantanu on GitHub was kind enough to point out that these utility types have been brought up before in a PEP, and rejected. See here: https://www.python.org/dev/peps/pep-0612/#defining-parametersof I believe it is worth reconsidering the efficacy of these utility types. As an end-user of TypeScript and Pyright, I can say that they make a lot of sense to me, and they are easy to explain to the people I train and manage. The specifics of how they are implemented I can't comment on, as I don't understand the full complexity of the Python type systems, but I can certainly say that they are obvious to the people I work with, and very useful, as we use them often in TypeScript code. The prior art on the part of TypeScript seems to indicate the utility types are useful as a general concept. I believe that if these utility types are implemented, they will form a part of solving the problem of how to correctly specify types for function decorators. Because it will be very easy to extract the parameters and return type of a Callable, you will be able to write a new type for a function that aliases one part and changes another part, such as using the same arguments for a function F as before, with a new return type, or the same return type as before, with a ParamSpec or list of arguments, etc. What does everyone else think about this topic?

3 2

AnyOf - Union for return types
by Sebastian Rittau 23 Sep '20

23 Sep '20

This has previously been discussed here: https://github.com/python/typing/issues/566. In typeshed we have found that there a quite a few cases where functions can return two or more different, incompatible types, that can't be distinguished by the input types alone. A few examples: * ip_address() can return either an IPv4Address or an IPv6Address, depending on the input string. * float ** float returns either a float or a complex. Currently, typeshed marks this as returning just float. * json.loads() et al. can return a selection of types, depending on the input. * urlopen() can return either an HTTPResponse or an addinfourl. There are more examples in the linked issue. In these cases, returning a Union would technically be correct, but would also be inconvenient for callers that know what type to expect. We usually give up and just return Any. My suggestion would be to add a type tentatively called "AnyOf", that acts as an unsafe union: ip_address(address: str) -> AnyOf[IPv4Address, IPv6Address] I believe this could be useful for type checkers. While the use of AnyOf is unsafe, it is safer than just treating these return values as Any. It's also theoretically possible to use some clever narrowing. That said, there has been some pushback from the mypy core team in the linked issue, especially since this is most likely complex to implement. An easy way for type checkers to implement this, without sacrificing any already existing type safety is to just treating AnyOf exactly like Any. At least as a first step. Should type checkers implement full support for AnyOf later, we could already have better types in typeshed ready to use. But in the last years, typeshed has also gained users outside of type checkers. For example, PyCharm and jedi use it for autocompletion. I believe that AnyOf could be useful for those projects, as well as others. Is there interest in a feature like this? I would be willing to write a PEP and contribute the AnyOf = Any solution to mypy. Unfortunately I don't have the bandwidth to learn how to implement this "properly" in mypy. - Sebastian

10 20

Typing-sig September 2020