I occasionally receive questions from pyright users about extending type narrowing to handle cases that are not possible with today’s Python type system.
Typescript provides a useful facility called [user-defined type guards]( https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defin…) that allows users to extend the notion of type narrowing. It would be straightforward to add this idea to Python. Do others think this would be useful, and is there any interest in standardizing this?
Here’s a concrete proposal. We could add a new generic type called `TypeGuard` in the `typing` module. It would be a subclass of bool, defined simply as:
```python
class TypeGuard(bool, Generic[_T]):
pass
```
If any function or method accepts one parameter and returns a `TypeGuard` type, a type checker would assume that the argument passed to this function can be narrowed to the type specified in the `TypeGuard` type argument.
Here are a few examples:
```python
from typing import Any, Generic, List, Tuple, TypeGuard, TypeVar
_T = TypeVar("_T")
def is_two_element_tuple(val: Tuple[_T, ...]) -> TypeGuard[Tuple[_T, _T]]:
return TypeGuard(len(val) == 2)
def foo(names: Tuple[str, ...]):
if is_two_element_tuple(names):
reveal_type(names) # Tuple[str, str]
def is_string_list(val: List[object]) -> TypeGuard[List[str]]:
return TypeGuard(all(isinstance(x, str) for x in val))
def bar(stuff: List[Any]):
if is_string_list(stuff):
reveal_type(stuff) # List[str]
```
Thoughts? Suggestions?
-Eric
---
Eric Traut
Contributor to Pyright and Pylance
Microsoft Corp.
Hi all,
Something that's on many of our wishlists is support for variadic generics.
Here's a first draft of a PEP detailing how they might work.
https://docs.google.com/document/d/1oXWyAtnv0-pbyJud8H5wkpIk8aajbkX-leJ8JXs…
(Attached is also an HTML render of the RST for easier reading.)
This is an early draft so there are likely many things we're still missing.
Please do take a look and give us your feedback (in the doc directly or by
email, whichever is easiest for you).
Thanks!
Matthew
Hi all,
I was hoping to get some opinions on whether something like the following
is valid, and if so, how it should be interpreted:
from typing import MutableSequence, TypeVar
T = TypeVar('T')
def f(seq: MutableSequence[T]) -> int:
return len(seq)
(This is, of course, *not* inside a generic class parameterized on `T`.)
Currently, pytype emits an [invalid-annotation] error for this piece of
code, with the explanation that `T` appears only once in `f`. Neither mypy
nor pyre emits an error. As far as I know, how a single TypeVar should be
treated is not covered in any of the typing PEPs.
pytype's reasons for making this an error are:
* The error messaging helps users avoid mistakes caused by misunderstanding
how to use a TypeVar.
* pytype's behavior when encountering a single TypeVar (especially when a
bound or constraints come into play) is ill-defined, so we don't want
anyone depending on it.
The reasons I've seen for not making it an error are:
* The TypeVar provides a way to annotate `seq` as a sequence of anything,
while still allowing usage errors to be caught in the body of `f`. `Any`
swallows all errors, and `object` does not work for invariant containers.
* There are potential future uses for a single TypeVar. One that came up in
the tensor typing call this morning is using a ListVariadic as a
placeholder to represent an unused part of an annotation.
* PEP 484 does not explicitly say that this should be an error, so it makes
sense to err on the side of allowing it.
* In spirit, `T` is an unused variable; flagging such things is not the job
of a type checker.
Thoughts?
Best,
Rebecca
This was previously discussed in
https://github.com/python/typing/issues/307.
It would often be useful to be able to specify a default type for type
vars. A few examples:
* Counter is most likely used with ints in the vast majority of cases.
Being able to write "Counter" instead of "Counter[int]" makes sense
to me.
* Currently, when marking a previously non-generic type as generic, it
will break existing type annotations. Having a default value works
around this. To use Counter as an example again: Currently it is
non-generic, but making it generic would break existing annotations.
Adding a default of "int" would work around this problem.
* Another example from the issue above: Using a default of None can
prevent overloads, as in the following example: "def
nullcontext(enter_result: _T = ...) -> ContextManager[_T]: ...".
Without the default we need an explicit overload for the case where
no argument is provided, because otherwise the type checker can't
know the type of _T. While this is not a big problem in stubs, I
find overloads quite obnoxious in implementations.
While of course not completely comparable, I find myself using generic
defaults quite often when working with TypeScript. A few examples as
anectodal evidence:
* We have functions that fetch JSON from an API. They are generic over
the return type and default to a general JSON type, but can be
overwritten if the response shape is known. (Because we control the
API.)
* We have a few cases, where an object is generic over a list of
strings. Take a class wrapping HTML <select> elements. Sometimes all
the possible options are known to the application, and have semantic
meaning. (For example for choosing a view.) So it makes sense to
declare a class like "class Select[Literal["foo", "bar"]]" (using
Python syntax). It's then possible to have "Select.value" have the
type "Literal["foo", "bar"]". In other cases, the possible options
are unknown (taken from a database, for example). In that case,
using "class Select[str]" would make sense. The latter is also a
sensible default.
* Generally I tend to use more type variables per type in TypeScript
than in Python, partially due to the fact that some of them can be
set to sensible defaults. While it would be obnoxious to always
define four generics everytime you use a type, in typeshed I only
have to define one of them with the others having sensible defaults.
What do other people think about this and how would we proceed if this
is deemed useful?
- Sebastian
Hey all,
Our next tensor typing meeting will be next week.
WHEN? Monday 12th October, 9:30am San Francisco Time, 5:30pm London time.
WHERE? http://meet.google.com/fft-dzjq-ksu
WHAT? We have a relatively sparse agenda this time. I'll lead a discussion,
but if anyone else would like to present/discuss something, let me know!
1. Status updates
2. Discussion: Use of ... (Ellipsis) for unspecified parts of shapes
Cheers!
Matthew
Hi everyone,
My friend and I have been interested in multiple dispatch for a while, and we noticed that it might be worth considering using it to resolve some fundamental difficulties type-checkers have understanding binary operators. We wrote up a collab summarizing our ideas: https://colab.research.google.com/drive/1-HQa5GOmV-uhTbv6EgnMfJYVgwuYMIrQ#s…
I initially posted a short version of this idea to python-ideas, but Guido suggested I flesh out the idea and post it here. Looking forward to everyone's insight and comments,
Best,
Neil
Hi everyone,
I am happy to announce that the first version of Higher Kinded Types
emulation is released with full `mypy` support as a part of
`dry-python/returns(a)0.15`.
I have made a post that describes all its features and (what is even more
exciting) all the hacks we have to apply to get this working:
https://sobolevn.me/2020/10/higher-kinded-types-in-python
Long story short: the biggest problem is that `mypy` treats `Kind` as a
regular type and not as a SpecialForm with some extra features. This forces
us to use direct inheritance from Kind and do some other hacks inside our
mypy plugin. The second problem is that we don't have variadic generics.
But, I know that it is in the work right now.
If mypy developers are interested, I can slowly work on adding Kind type
and all the required machinery into mypy itself (the good news is that even
`mypyc` works correctly with Kind). If so, please, guide me about the first
steps :)
Source code and docs: https://github.com/dry-python/returns
Simple example from the article to experiment:
https://gist.github.com/sobolevn/7f8ffd885aec70e55dd47928a1fb3e61
Best regards,
Nikita Sobolev
https://github.com/sobolevn
Hi there,
Recently I have been working on an implementation for PEP 604 (https://www.python.org/dev/peps/pep-0604/). While working on this, I started reviewing and exploring what new Optional syntax would look like in the Python language.
I've written a rough draft of a PEP and wanted to present it to this email list before submitting a PR. I know this is a topic that has been discussed in many settings (Github issues, in person typing meetups, etc. )previously, and wanted to solicit feedback here first. Let me know if submitting a PR would be a better way to gauge sentiment and feedback on this proposal.
Draft Proposal (branch compare): https://github.com/python/peps/compare/master...MaggieMoss:optional-syntax-…
Gist: https://gist.github.com/MaggieMoss/c848cb3a581979f445d075c15629c950
Thanks again,
Maggie
PEP 561 indicates that “package maintainers who wish to support type checking of their code MUST add a marker file named py.typed…”. It doesn’t define what “support type checking” means or what expectations are implied. This has led to a situation where packages claim to support type checking but omit many type annotations. There’s currently no tooling that validates the level of “type completeness” for a package, so even well-intentioned package maintainers are unable to confirm that their packages are properly and completely annotated. This leads to situations where type checkers and language servers need to fall back on type inference, which is costly and gives inconsistent results across tools. Ideally, all py.typed packages would have their entire public interface completely annotated.
I’m working on a new feature in Pyright that allows package maintainers to determine whether any of the public symbols in their package are missing type annotations. To do this, I need to clearly define what constitutes a “public symbol”. In most cases, the rules are pretty straightforward and follow the naming guidelines set forth in PEP 8 and PEP 484. For example, symbols that begin with an underscore are excluded from the list of public symbols.
One area of ambiguity is related to import statements. PEP 484 indicates that within stub files, a symbol is not considered exported unless it is used within an import statement of the form `import x as y` or `from x import y as z` or `from x import *`. The problem is that this rule applies only to “.pyi” files and not to “.py” files. For packages that use inlined types, it’s ambiguous whether an import statement of the form `import x` or `from y import x` should treat `x` as a public symbol that is exported from that module.
I can think of a few solutions here:
1. For py.typed packages, type checkers should always apply PEP 484 import rules for “.py” files. If a symbol `x` is imported with an `import x` or `from y import x`, it is treated as “not public”, and any attempt to import it from another package will result in an error.
2. For py.typed packages, PEP 484 rules are _not_ applied for import statements. This maintains backward compatibility. Package maintainers can opt in to PEP 484 rules using some well-defined mechanism. For example, we could define a special flag “stub_import_rules” that can be added to a “py.typed” file. Type checkers could then conditionally use PEP 484 rules for imports.
Option 1 will likely break some assumptions for existing packages. Option 2 avoids that break, but it involves more complexity.
Any suggestions? Thoughts?
-Eric
---
Eric Traut
Contributor to Pyright and Pylance
Microsoft
Hello, greetings from the QEMU project!
We've been investigating adopting mypy for some of our python utilities
that we use as a kind of turbocharged preprocessor for defining QEMU API
types in our project ('QAPI').
So far I've had good success (And only nice things to say about the
improvements made in each python version so far), but I do have a
question about a few paradigms I see here and again.
Is there a short-hand way to type certain dunder methods? For example,
we've got an __exit__ method on one of our VM management classes we use
for testing. The types for this are fairly arcane for the average C
programmer with only surface-level knowledge of Python.
e.g., we are left with this:
def __exit__(self,
# pylint: disable=duplicate-code
# see https://github.com/PyCQA/pylint/issues/3619
exc_type: Optional[Type[BaseException]],
exc_val: Optional[BaseException],
exc_tb: Optional[TracebackType]) -> None:
self.close()
That's a bit of noise! In this case, I assume that for well-known
dunders, mypy could be configured to *assume* the canonical types for
well-known names. Maybe I am mistaken and this is a horrible assumption
to make?
On a similar train of thought, I was wondering if there was any
discussion about the typing of things like **kwargs. I see this paradigm
in python code quite often:
def my_cool_wrapper(**kwargs):
logging.debug(kwargs['arbitrarily-important-key'])
return some_stdlib_function(**kwargs)
Is there any formalization or shorthand for declaring this function as a
pass-through type? i.e., being able to declare that the entirety of this
functions arguments simply match precisely some other function's arguments.
Usually, I have been looking into typeshed to copy-paste type signatures
back into my code when I need to type something to match the standard
library, but this is prone to breakage -- and not always possible,
because typeshed uses lots of internal types I can't copy out. Is there
a more elegant way to write strictly typed wrappers, or some existing
thought/discussion on this area of design?
Similar cases pop up for overriding a parent class's method: there are
times where I might be attempting to extend an interface with new
optional parameters, but sometimes I am merely implementing an
interface, and the replication of all those types becomes a source of
lots of repeated lines of code.
I would be happy to be pointed to archived mailing list discussions,
formal proposals, etc.
Thanks,
--js
