Mailman 3 - Typing-sig

by Ethan Furman

Greetings! I'd like to add Generics support for Enum... at least I think I do. :-/ The magic seems to happen in `SomeMetaClass.__getitem__`, and it should be doable as `EnumType.__getitem__` currently only accepts strings, so types could easily take a different branch in the code, but I am entirely unsure of what that separate branch should do, what it should return, etc. Are there any pure-python classes in the stdlib (besides Generics) that I could use for inspiration? Or some documentation somewhere I could use? A basic tutorial so I can get a grip on all the terminology? Any and all help appreciated! -- ~Ethan~

26 minutes

2
1
0 0

Proposal: typing.assert_never()

by Jelle Zijlstra

Another idea that came up in a few recent threads is a `typing.assert_never()` function, which indicates that some code is supposed to be unreachable. I will submit an implementation to CPython, and we can use this thread to work out any details (or to let me know that we shouldn't do this after all). # Specification We will add a new function `typing.assert_never()` with the following signature: def assert_never(arg: NoReturn, /) -> NoReturn: ... At runtime, this will unconditionally raise `RuntimeError(f"Unexpectedly reached {arg!r}")` (wording suggestions welcome). Type checkers should emit an error if the function is called with an argument that has a non-bottom type. Examples: match bool(): case True: pass case False: pass case _ as x: assert_never(x) # ok match int(): case 0: pass case 1: pass case _ as x: assert_never(x) # error: x is an int, not NoReturn # Motivation PEP 484 only allows NoReturn as a return type, but in practice all type checkers allow it in argument positions as a bottom type. Thus, a manually defined `assert_never()` function already has the behavior specified above. Adding the function to the standard library will make this behavior more discoverable for users. Relatedly, I opened https://bugs.python.org/issue46475 to document NoReturn being a general bottom type. # Open questions Suggestions welcome for better wording in the runtime error. There is a related proposal to rename NoReturn to Never. I'm lukewarm about that because renaming things always causes churn, but in any case let's discuss that separately.

1 hour, 59 minutes

4
3
0 0

Re: Proposal: typing.assert_type() and with typing.assert_error()

by Anton Agestam

I like both of these suggestions. I use both assertions on revealed types and on yielded errors on the types in phantom-types. The names are clear to me, and the fact they'd be namespaced under "typing" makes it blatantly clear that they are related to concepts of typing. I'd suggest defining assert_error() to accept arbitrary kwargs, so that context can be passed in specific for type checkers, e.g.: assert_error(mypy=["arg-type"], pyright=[...])

2 hours, 53 minutes

2
1
0 0

Proposal: typing.assert_type() and with typing.assert_error()

by Jelle Zijlstra

Don't worry, this is the last one for today, but probably the trickiest. These two functions are intended to help users who want to check how their API behaves with a type checker: they want to assert that some type is inferred correctly, or that some code that they want the type checker to catch is rejected. # Specification The signature of typing.assert_type() is: def assert_type(obj: T, typ: TypeForm[T], /) -> T: ... (Using TypeForm from David Foster's pending proposal.) At runtime, this simply returns the first argument unchanged. As with my proposed behavior for reveal_type(), this allows using the function within an expression. A static type checker will emit an error if it encounters a call to assert_type() and the inferred type for the obj argument is not exactly equal to the provided type. The type passed in as the `typ` argument is interpreted the same way the type checker would interpret a type annotation, just as with the first argument to the existing `typing.cast()` function. For example: def f(x: int): assert_type(x, int) # ok assert_type(x, str) # error: type of x is int, not str assert_type(x, Any) # error: type of x is int, not Any The signature of typing.assert_error() is: def assert_error() -> ContextManager[None]: ... At runtime, this returns a context manager that does nothing. At type checking time, the context asserts that an error is found within the block. A static type checker will handle a `with assert_error():` block as follows: - Type check the block as normal, but don't emit any errors to the user - If there were no errors, emit an error - If there were errors, continue as normal For example: with assert_error(): # ok 1 + "x" with assert_error(): # error 1 + 1 # Motivation There has been a longstanding need for a way to check that typed code is understood correctly by the type checker: - https://github.com/python/typeshed/issues/1339 - typeshed issue looking for a way to test typeshed stubs - https://github.com/python/typing/discussions/1030 - recent discussion looking for the same thing in user code - https://pypi.org/project/pytest-mypy-plugins/ provides a mypy-specific way to test inferred types - https://google.github.io/pytype/faq.html#can-i-find-out-what-pytype-thinks-… - pytype added an assert_type() function as proposed here based on user feedback Adding these two functions to the standard library with defined semantics will create a portable, well-documented way to test type checker behavior. This will be useful for typeshed and for any library author who needs to support multiple type checkers. # Use case As a motivating example, I'd like to go over a recent change I made to typeshed: https://github.com/python/typeshed/pull/6941. I wanted to make it so that you cannot call `dict.setdefault()` with one argument if the dict's value type does not include `None`, so I wrote some tricky overloads. With `assert_type()` and `assert_error()`, I could have written tests like these to make sure my stubs work with all type checkers: int_int: dict[int, int] = {} int_any: dict[int, Any] = {} int_optional: dict[int, int | None] = {} with assert_error(): int_int.setdefault(1) # should error because this may set the value to None assert_type(int_int.setdefault(1, 1), int) assert_type(int_any.setdefault(1), Any) assert_type(int_optional.setdefault(1), int | None) What I did instead was run some samples manually with mypy, but that of course doesn't guarantee that the sample continues to work. This example comes from typeshed, but the utility isn't limited to typeshed; it can help any user who wants to make sure the types they wrote behave as they expect. # Open issues The assert_error() context manager is unlike other existing special forms in the typing module, since no other context manager currently has special meaning to the type checker. I'd be happy to hear any alternative ideas for covering this use case. Here are two alternative approaches I considered: - A magical comment, like TypeScript's "// $ExpectError". But a magical comment is not as explicit, is harder to understand for users, and may cause problems with autoformatters. - A function call: perhaps `typing.assert_type(expr, typing.Error)` would assert that evaluating `expr` produces an error. But this wouldn't allow asserting that an error occurs in a statement as opposed to an expression. In https://github.com/python/typing/discussions/1030, we explored several variants for the behavior of assert_type(), such as allowing an explicit way to provide the type as a string. I think the behavior specified here is the most intuitive and consistent with the behavior of other typing constructs, but I'm happy to discuss this further.

10 hours, 44 minutes

5
8
0 0

Adding `typing.reveal_type`

by Jelle Zijlstra

I just opened https://bugs.python.org/issue46414 to propose adding `typing.reveal_type` to Python 3.11. Copying my message there: --- The `reveal_type()` primitive is injected by type checkers into the builtins. When the type checker sees a call, it prints the inferred type of the argument. This has been implemented across all type checkers, but adding an implementation to `typing` would help document the behavior and make it more discoverable for users. Also, it means code with `reveal_type()` calls can run without runtime errors, useful if you want to run your tests at the same time as you're debugging a typing issue. The runtime implementation can be very simple: def reveal_type(obj: _T, /) -> _T: print("Runtime type is", type(obj)) return obj reveal_type() is supported by all type checkers that I'm aware of (docs include https://google.github.io/pytype/faq.html#can-i-find-out-what-pytype-thinks-… for pytype and https://mypy.readthedocs.io/en/stable/common_issues.html#reveal-type for mypy). One area of divergence is the return value. Pyright returns the inferred type of the expression as a string (and uses that in its test suite for testing type inference). Mypy returns the argument, which has the advantage that you can insert `reveal_type()` in the middle of an expression without having to put it on its own line. Also, the Pyright behavior cannot sensibly be implemented at runtime. Therefore, I suggest using Mypy's behavior for `typing.reveal_type`. ---- I'm hoping this will be simple and noncontroversial enough that we can add it quickly, but if there are concerns from any type checker authors, I'd love to hear them.

19 hours, 30 minutes

9
21
2 0

Proposal: typing.reveal_locals()

by Jelle Zijlstra

In the previous thread about reveal_type(), several people proposed also adding `typing.reveal_locals()`. In this thread I'd like to iron out any details and make sure we're all in agreement. # Specification We will add a new function `typing.reveal_locals()` with the following signature: def reveal_locals() -> None: ... At runtime, this function will do nothing. When a type checker encounters a call to reveal_locals(), it will emit the types of all variables in the local scope. For example: def f() -> None: a = 1 b = "x" reveal_locals() When typechecked, this may produce: Revealed local types are: a: builtins.int b: builtins.str # Motivation `reveal_locals()` is already implemented in mypy ( https://mypy.readthedocs.io/en/stable/common_issues.html#displaying-the-typ…) and pyright (undocumented). It is not as commonly used as reveal_type(), but can be useful for debugging type checker behavior. # Open questions What should the runtime behavior be? We could do something like sys._getframe(1).f_locals to get the calling function's locals and print them, but that isn't code I'd like to write.

20 hours, 46 minutes

3
3
0 0

RFC on Arbitrary Literal Strings PEP

by S Pradeep Kumar

Hello all, We've drafted a PEP to support arbitrary literal string types [1]. Feedback appreciated! Scala has a very similar concept. See this talk on using Scala's literal string types for "Preventing SQL injection at compile time" [2] Bikeshedding: We're open to any name that seems easy to understand :) -- Best, S Pradeep Kumar Graham Bleaney [1]: https://docs.google.com/document/d/1fbAbA2MCoAcSO1c8gecmUJYZ-Q0utT3CmaXlQ3I… [2]: https://slideslive.com/38907881/literal-types-what-they-are-good-for (slides 52 to 68) [3]: Previous typing-sig thread: https://mail.python.org/archives/list/typing-sig@python.org/thread/JJWAA5AH…

1 day, 3 hours

12
16
1 0

Exhaustiveness checking in match statements

by Henry F. Schreiner

I’d like to gather some opinions or suggestions on exhaustiveness checking on match statements, stemming from https://github.com/python/mypy/pull/10191 (though most future discussion should happen on https://github.com/python/mypy/issues/12010). I’m especially interested in any experience from working with this already, such as in pyright. # case _: assert False would trigger a type check error if reachable. This would allow users to mark match statements as exhaustive. This would require assert False to be treated specially by type checkers in a “match all” case statement, but it’s inline with a common usage of assert false at the end of an if chain today, and wouldn’t introduce new constructs. # --exhaustive-match flag for checkers like mypy This would be a new “strict' flag added to mypy “strict” that would force match statements to be statically exhaustive - this is not (and maybe could not) be enforced at runtime by Python, but could for a static type checker, and would force case _: pass to be added if fallthrough was expected - to make it explicit, and avoid forgetting to add case _: assert False. At some some codebases would likely benefit from this flag. I assume this could be done by other checkers too. It would be a normal mypy strictness flag, so could be enabled/disabled per module, etc. # Current workaround The current workaround for this is to have a function assert_never that takes a NoReturn - NoReturn happens to be (the only) way to spell an empty Union (at least in mypy?). This also works with type narrowing in if statements, so is usable before requiring 3.10+. Until one or both of the above are implemented, it’s the only way to check pattern matching for exhaustiveness, too. Personally, I’d like a slightly nicer way to write this as well, with assert_unreabable or something, or at least a Nothing/EmptyUnion spelling for NoReturn. I think the current leaning is to implement the first two, and do nothing on the third. Thoughts? Especially on the exhaustive-match flag. - Henry

1 day, 4 hours

8
9
2 0

PEP 655: Marking individual TypedDict items as required or potentially-missing

by David Foster

The new syntax Required[...] and NotRequired[...] for marking individual keys of a TypedDict has now been formally proposed as a draft PEP. Please see the latest text at the bottom of this email, or online at: https://www.python.org/dev/peps/pep-0655/ Comments welcome. :) Meanwhile I'll proceed to implement this syntax in mypy. -- David Foster | Seattle, WA, USA Contributor to TypedDict support for mypy >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PEP: 655 Title: Marking individual TypedDict items as required or potentially-missing Author: David Foster <david at dafoster.net> Sponsor: Guido van Rossum <guido at python.org> Discussions-To: typing-sig at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: 604 Created: 30-Jan-2021 Python-Version: 3.10 Post-History: 31-Jan-2021, 11-Feb-2021, 20-Feb-2021, 26-Feb-2021 Abstract ======== `PEP 589 <https://www.python.org/dev/peps/pep-0589/>`__ defines syntax for declaring a TypedDict with all required keys and syntax for defining a TypedDict with `all potentially-missing keys <https://www.python.org/dev/peps/pep-0589/#totality>`__ however it does not provide any syntax to declare some keys as required and others as potentially-missing. This PEP introduces two new syntaxes: ``Required[...]`` which can be used on individual items of a TypedDict to mark them as required, and ``NotRequired[...]`` which can be used on individual items to mark them as potentially-missing. Motivation ========== It is not uncommon to want to define a TypedDict with some keys that are required and others that are potentially-missing. Currently the only way to define such a TypedDict is to declare one TypedDict with one value for ``total`` and then inherit it from another TypedDict with a different value for ``total``: :: class _MovieBase(TypedDict): # implicitly total=True title: str class Movie(_MovieBase, total=False): year: int Having to declare two different TypedDict types for this purpose is cumbersome. Rationale ========= One might think it unusual to propose syntax that prioritizes marking *required* keys rather than syntax for *potentially-missing* keys, as is customary in other languages like TypeScript: :: interface Movie { title: string; year?: number; // ? marks potentially-missing keys } The difficulty is that the best word for marking a potentially-missing key, ``Optional[...]``, is already used in Python for a completely different purpose: marking values that could be either of a particular type or ``None``. In particular the following does not work: :: class Movie(TypedDict): ... year: Optional[int] # means int|None, not potentially-missing! Attempting to use any synonym of “optional” to mark potentially-missing keys (like ``Missing[...]``) would be too similar to ``Optional[...]`` and be easy to confuse with it. Thus it was decided to focus on positive-form phrasing for required keys instead, which is straightforward to spell as ``Required[...]``. Nevertheless it is common for folks wanting to extend a regular (``total=True``) TypedDict to only want to add a small number of potentially-missing keys, which necessitates a way to mark keys that are *not* required and potentially-missing, and so we also allow the ``NotRequired[...]`` form for that case. Specification ============= The ``typing.Required`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a required key: :: class Movie(TypedDict, total=False): title: Required[str] year: int Additionally the ``typing.NotRequired`` type qualifier is used to indicate that a variable declared in a TypedDict definition is a potentially-missing key: :: class Movie(TypedDict): # implicitly total=True title: str year: NotRequired[int] It is an error to use ``Required[...]`` or ``NotRequired[...]`` in any location that is not an item of a TypedDict. It is valid to use ``Required[...]`` and ``NotRequired[...]`` even for items where it is redundant, to enable additional explicitness if desired: :: class Movie(TypedDict): title: Required[str] # redundant year: NotRequired[int] Backwards Compatibility ======================= No backward incompatible changes are made by this PEP. How to Teach This ================= To define a TypedDict where most keys are required and some are potentially-missing, define a single TypedDict as normal and mark those few keys that are potentially-missing with ``NotRequired[...]``. To define a TypedDict where most keys are potentially-missing and a few are required, define a ``total=False`` TypedDict and mark those few keys that are required with ``Required[...]``. If some items accept ``None`` in addition to a regular value, it is recommended that the ``TYPE|None`` syntax be preferred over ``Optional[TYPE]`` for marking such item values, to avoid using ``Required[...]`` or ``NotRequired[...]`` alongside ``Optional[...]`` within the same TypedDict definition: Yes: :: from __future__ import annotations # for Python 3.7-3.9 class Dog(TypedDict): name: str owner: NotRequired[str|None] Avoid (unless Python 3.5-3.6): :: class Dog(TypedDict): name: str # ick; avoid using both Optional and NotRequired owner: NotRequired[Optional[str]] Reference Implementation ======================== The goal is to be able to make the following statement: The `mypy <http://www.mypy-lang.org/>`__ type checker supports ``Required`` and ``NotRequired``. A reference implementation of the runtime component is provided in the `typing_extensions <https://github.com/python/typing/tree/master/typing_extensions>`__ module. The mypy implementation is currently still being worked on. Rejected Ideas ============== Special syntax around the *key* of a TypedDict item --------------------------------------------------- :: class MyThing(TypedDict): opt1?: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value or: :: class MyThing(TypedDict): Optional[opt1]: str # may not exist, but if exists, value is string opt2: Optional[str] # always exists, but may have null value These syntaxes would require Python grammar changes and it is not believed that marking TypedDict items as required or potentially-missing would meet the high bar required to make such grammar changes. Also, “let’s just not put funny syntax before the colon.” [1]_ Marking required or potentially-missing keys with an operator ------------------------------------------------------------- We could use unary ``+`` as shorthand to mark a required key, unary ``-`` to mark a potentially-missing key, or unary ``~`` to mark a key with opposite-of-normal totality: :: class MyThing(TypedDict, total=False): req1: +int # + means a required key, or Required[...] opt1: str req2: +float class MyThing(TypedDict): req1: int opt1: -str # - means a potentially-missing key, or NotRequired[...] req2: float class MyThing(TypedDict): req1: int opt1: ~str # ~ means a opposite-of-normal-totality key req2: float Such operators could be implemented on ``type`` via the ``__pos__``, ``__neg__`` and ``__invert__`` special methods without modifying the grammar. It was decided that it would be prudent to introduce longform syntax (i.e. ``Required[...]`` and ``NotRequired[...]``) before introducing any shortform syntax. Future PEPs may reconsider introducing this or other shortform syntax options. Marking absence of a value with a special constant -------------------------------------------------- We could introduce a new type-level constant which signals the absence of a value when used as a union member, similar to JavaScript’s ``undefined`` type, perhaps called ``Missing``: :: class MyThing(TypedDict): req1: int opt1: str|Missing req2: float Such a ``Missing`` constant could also be used for other scenarios such as the type of a variable which is only conditionally defined: :: class MyClass: attr: int|Missing def __init__(self, set_attr: bool) -> None: if set_attr: self.attr = 10 :: def foo(set_attr: bool) -> None: if set_attr: attr = 10 reveal_type(attr) # int|Missing Misalignment with how unions apply to values '''''''''''''''''''''''''''''''''''''''''''' However this use of ``...|Missing``, equivalent to ``Union[..., Missing]``, doesn’t align well with what a union normally means: ``Union[...]`` always describes the type of a *value* that is present. By contrast missingness or non-totality is a property of a *variable* instead. Current precedent for marking properties of a variable include ``Final[...]`` and ``ClassVar[...]``, which the proposal for ``Required[...]`` is aligned with. Misalignment with how unions are subdivided ''''''''''''''''''''''''''''''''''''''''''' Furthermore the use of ``Union[..., Missing]`` doesn’t align with the usual ways that union values are broken down: Normally you can eliminate components of a union type using ``isinstance`` checks: :: class Packet: data: Union[str, bytes] def send_data(packet: Packet) -> None: if isinstance(packet.data, str): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # bytes packet_bytes = packet.data socket.send(packet_bytes) However if we were to allow ``Union[..., Missing]`` you’d either have to eliminate the ``Missing`` case with ``hasattr`` for object attributes: :: class Packet: data: Union[str, Missing] def send_data(packet: Packet) -> None: if hasattr(packet, 'data'): reveal_type(packet.data) # str packet_bytes = packet.data.encode('utf-8') else: reveal_type(packet.data) # Missing? error? packet_bytes = b'' socket.send(packet_bytes) or a check against ``locals()`` for local variables: :: def send_data(packet_data: Optional[str]) -> None: packet_bytes: Union[str, Missing] if packet_data is not None: packet_bytes = packet.data.encode('utf-8') if 'packet_bytes' in locals(): reveal_type(packet_bytes) # bytes socket.send(packet_bytes) else: reveal_type(packet_bytes) # Missing? error? or a check via other means, such as against ``globals()`` for global variables: :: warning: Union[str, Missing] import sys if sys.version_info < (3, 6): warning = 'Your version of Python is unsupported!' if 'warning' in globals(): reveal_type(warning) # str print(warning) else: reveal_type(warning) # Missing? error? Weird and inconsistent. ``Missing`` is not really a value at all; it’s an absence of definition and such an absence should be treated specially. Difficult to implement '''''''''''''''''''''' Eric Traut from the Pyright type checker team has stated that implementing a ``Union[..., Missing]``-style syntax would be difficult. [2]_ Introduces a second null-like value into Python ''''''''''''''''''''''''''''''''''''''''''''''' Defining a new ``Missing`` type-level constant would be very close to introducing a new ``Missing`` value-level constant at runtime, creating a second null-like runtime value in addition to ``None``. Having two different null-like constants in Python (``None`` and ``Missing``) would be confusing. Many newcomers to JavaScript already have difficulty distinguishing between its analogous constants ``null`` and ``undefined``. Replace Optional with Nullable. Repurpose Optional to mean “optional item”. --------------------------------------------------------------------------- ``Optional[...]`` is too ubiquitous to deprecate. Although use of it *may* fade over time in favor of the ``T|None`` syntax specified by `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`__. Change Optional to mean “optional item” in certain contexts instead of “nullable” --------------------------------------------------------------------------------- Consider the use of a special flag on a TypedDict definition to alter the interpretation of ``Optional`` inside the TypedDict to mean “optional item” rather than its usual meaning of “nullable”: :: class MyThing(TypedDict, optional_as_missing=True): req1: int opt1: Optional[str] or: :: class MyThing(TypedDict, optional_as_nullable=False): req1: int opt1: Optional[str] This would add more confusion for users because it would mean that in *some* contexts the meaning of ``Optional[...]`` is different than in other contexts, and it would be easy to overlook the flag. Various synonyms for “potentially-missing item” ----------------------------------------------- - Omittable – too easy to confuse with optional - OptionalItem, OptionalKey – two words; too easy to confuse with optional - MayExist, MissingOk – two words - Droppable – too similar to Rust’s ``Drop``, which means something different - Potential – too vague - Open – sounds like applies to an entire structure rather then to an item - Excludable - Checked References ========== .. [1] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIW… .. [2] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG… Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

1 day, 4 hours

9
24
0 0

dataclass_transform support for converter field descriptor parameter

by Erik De Bonte

Thanks to everyone that has provided feedback on the dataclass_transform PEP so far! There's one open issue<https://github.com/debonte/peps/blob/dataclass_transforms/pep-9999.rst#conv…> right now, and I wondered if anyone had feedback specifically on it: converter field descriptor parameter The attrs library supports a converter field descriptor parameter, which is passed a callable that is called by the generated __init__ method to convert the supplied value to some other desired value. This is tricky to support since the parameter type in the synthesized __init__ method needs to accept uncovered values, but the resulting field is typed according to the output of the converter. There may be no good way to support this because there's not enough information to derive the type of the input parameter. We currently have two ideas: 1. Add support for a converter field descriptor parameter but then use the Any type for the corresponding parameter in the __init__ method. 2. Say that converters are unsupported and recommend that attrs users avoid them. Some aspects of this issue are discussed on the pyright GitHub site. [https://github.com/microsoft/pyright/discussions/1782?sort=old#discussionco…] Thanks, Erik

4 days, 23 hours

4
3
0 0

2022

2021

2020

2019

2018

Typing-sig