I still think that we should reconsider deferring what happens to class and instance methods. The arguments given in https://www.python.org/dev/peps/pep-0647/#id13 seem insufficient, specially considering than the workaround provided is quite awkward. The author suggests to write def check(self, self2) -> Typeguard[T] and call it as self.check(self).

That pattern is used almost as much as multi-argument guards. I checked the typings from DefinitelyTyped (JS equivalent of typeshed) and found the following statistics: 7501 packages 100 (1.3%) packages defining type guards 13 (0.17%) packages defining multi-argument type guards 10 (0.13%) packages defining type guards for this (self) On Fri, Feb 12, 2021 at 12:00 AM Guido van Rossum <guido@python.org> wrote:

I think the use case (for x.is_foo()) is rare. And instead of writing x.is_foo(x), if you make the guard a function you can write is_foo(x).

On Thu, Feb 11, 2021 at 6:51 PM Sebastian Kreft <skreft@gmail.com> wrote:

...

I still think that we should reconsider deferring what happens to class and instance methods.

The arguments given in https://www.python.org/dev/peps/pep-0647/#id13 seem insufficient, specially considering than the workaround provided is quite awkward.

The author suggests to write def check(self, self2) -> Typeguard[T] and call it as self.check(self). _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CA2MFS4X... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

-- Sebastian Kreft

On Fri, Feb 12, 2021 at 12:14 PM Jim J. Jewett <jimjjewett@gmail.com> wrote:

Current PEP 647 draft says:

"Some built-in type guards provide narrowing for both positive and negative tests (in both the if and else clauses)"

Should there be a separate (sub-?) type for those TypeGuards that *do* allow narrowing even on a False? Leaving it as an implementation detail available only to (some) built-in types seems to confuse the issue in both directions.

I think the PEP is using the term in a wider sense here, referring to things like `isinstance(x, C)` and `x is not None`. Note that mypy (at least) also recognizes `if x:` as narrowing `Optional[T]` to `T`, but there in the else clause we *cannot* assume that x is not a T, because it could be a falsey instance. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

My primary reaction seems similar to Mark Shannon's. When I see this code: def is_str_list(val: List[object]) -> TypeGuard[List[str]]: ... I cannot tell what it returns. There is no readable indication in that this returns a boolean so the reader cannot immediately see how to use the function. In fact my first reaction is to assume it returns some form of List[str]. Which it most definitely does not do. Additionally, it seems like restricting this concept of a type guard to only a function that returns a bool is wrong. It is also a natural idiom to encounter functions that raise an exception if their type conditions aren't met. Those should also narrow types within a non-exception caught codepath after their return. Be simple and assume that catching any exception from their call ends their narrowing? A narrowing is meta-information about a typing side effect, it isn't a type itself. It isn't a value. So replacing a return type with it seems like too much. But if we do wind up going that route, at least make the name indicate that the return value is a bool. i.e.: def parse_num(thing: Any) -> NarrowingBool[float] The term TypeGuard is too theoretical for anyone reading code. I don't care if TypeScript uses it in their language... looking that up at a quick glance - https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-define... - it doesn't look like they ever use the term TypeGuard in the language syntax itself? good. Perhaps this would be better expressed as an annotation on the argument(s) that the function narrows? def is_str_list(val: Constrains[List[object]:List[str]) -> bool: ... I'm using Constrains as an alternative to Narrows here... still open to suggestions. But I do not think we should get hung up on TypeScript and assume that TypeGuard is somehow a good name. This particular proposed example uses : slice notation rather than a comma as a proposal that may be more readable. I'd also consider using the 'as' keyword instead of , or : but that wouldn't even parse today. *Really* what might be nice for this is our -> arrow. def assert_str_list(var: Narrows[List[object] -> List[str]]) -> None: ... as the arrow carries the implication of something effective "after" the call. I suspect adding new parsing syntax isn't what anyone had in mind for now though. : seems like a viable stopgap, whether syntax happens or not. (i'm intentionally avoiding the comma as an exploration in readability) The key point I'm trying to represent is that declaring it per argument allows for more expressive functions and doesn't interfere with their return value (if they even return anything). Clearly there's a use case that _does_ want to make the narrowing conditional on the return value boolness. I still suggest expressing that on the arguments themselves. More specific flavors of the guarding parameter such as ConstrainsWhenTrue or NarrowsWhenFalse perhaps. def is_set_of(val: Constrains[Set[Any]:Set[_T]], type: Type[_T]) -> bool: ... which might also be written more concisely as: def is_set_of(val: Set[Constrains[Any:_T]], type: Type[_T]) -> bool: If we allow the full concept. hopefully food for thought, i'm -1 right now. -gps On Fri, Feb 12, 2021 at 2:34 AM Mark Shannon <mark@hotpy.org> wrote:

Hi,

First of all, sorry for not commenting on this earlier. I only became aware of this PEP yesterday.

I like the general idea of adding a marker to show that a boolean function narrows the type of one (or more?) of its arguments. However, the suggested approach seems clunky and impairs readability.

It impairs readability, because it muddles the return type. The function in the example returns a bool. The annotation is also misleading as the annotation is on the return type, not on the parameter that is narrowed.

At a glance, most programmers should be able to work out what

def is_str_list(val: List[object]) -> bool:

returns.

But,

def is_str_list(val: List[object]) -> TypeGuard[List[str]]:

is likely to confuse and require careful reading. Type hints are for humans as well as type checkers.

Technical review. -----------------

For an annotation of this kind to be useful to a checker, that checker must perform both flow-sensitive and call-graph analysis. Therefore it is theoretically possible to remove the annotation altogether, using the following approach:

1. Scan the code looking for functions that return boolean and potentially narrow the type of their arguments. 2. Inline those functions in the analysis 3. Rely on pre-existing flow-senstive analysis to determine the correct types.

However, explicit is better and implicit. So some sort of annotation seems sensible.

I would contend that the minimal:

@narrows def is_str_list(val: List[object]) -> bool:

is sufficient for a checker, as the checker can inline anything marked @narrows. Plus, it does not mislead the reader by mangling the return type.

An alternative, and more explicit, approach would be to use variable annotations. So:

def is_str_list(val: List[object]) -> bool: """Determines whether all objects in the list are strings""" return all(isinstance(x, str) for x in val)

might become:

def is_str_list(val: List[object]) -> bool: """Determines whether all objects in the list are strings""" val: NarrowsTo[List[str]] return all(isinstance(x, str) for x in val)

Although the above lacks flow control and is thus ambiguous without the convention that `NarrowsTo` only applies if the result is True.

An alternative formulation would require the annotation to dominate the function exit:

def is_str_list(val: List[object]) -> bool: """Determines whether all objects in the list are strings""" if all(isinstance(x, str) for x in val): val: NarrowsTo[List[str]] return True return False

This is unambiguous.

Finally, I would ask for a change of name.

The "Type" part is redundant, since it is a type annotation, and the "Guard" part is incorrect. It is only a guard when used, the function itself is a predicate that narrows the type of an argument. "Narrows" or "NarrowsTo" would be better.

Cheers, Mark.

On 09/02/2021 4:21 pm, Guido van Rossum wrote:

...

I think we have reached consensus on PEP 647 in typing-sig. We have implementations for mypy and pyright, not sure about the rest. This PEP does not affect CPython directly except for the addition of one special item (TypeGuard) to typing.py -- it would be nice to get that in the 3.10 stdlib.

I'm CC'ing python-dev here to see if there are any further comments; if not, we can initiate the approval process by creating an issue at https://github.com/python/steering-council.

-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ < http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NOLCFYLY... Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4LKI6IGC... Code of Conduct: http://python.org/psf/codeofconduct/

Hello, On Sat, 13 Feb 2021 18:08:30 +1100 Steven D'Aprano <steve@pearwood.info> wrote:

On Fri, Feb 12, 2021 at 10:27:01AM +0000, Mark Shannon wrote:

...

It impairs readability, because it muddles the return type. The function in the example returns a bool. The annotation is also misleading as the annotation is on the return type, not on the parameter that is narrowed.

At a glance, most programmers should be able to work out what

def is_str_list(val: List[object]) -> bool:

returns.

But,

def is_str_list(val: List[object]) -> TypeGuard[List[str]]:

is likely to confuse and require careful reading. Type hints are for humans as well as type checkers.

This!

Without reading the PEP, how is anyone supposed to know that this returns a bool? This is obfuscated code. It looks like it returns an object of type `TypeGuard`. Even having read the PEP, it's still misleading: instead of a straightforward return type, I have to memorise that `TypeGuard[...]` is a magic thing that means

* the return type is actually a bool;

* and the type checker can narrow the type of the first parameter.

Yuck.

My quick reading of PEP647 shows that it's yet another patchy/throw-away PEP whose purpose is to work around deficiencies in the current generation of Python typecheckers. In that regard, I'd take it easy, and be +1 with it - it's better to support gradual evolution of Python typecheckers with hacks like that, rather than block it with proverbial "we can solve all the problems of universe right in the version 1". [] -- Best regards, Paul mailto:pmiscml@gmail.com

On Sat, 13 Feb 2021 at 07:11, Steven D'Aprano <steve@pearwood.info> wrote:

...

Without reading the PEP, how is anyone supposed to know that this returns a bool?

By looking at the name, or at the return statements in the body. These are expected to be really short. Tooling can certainly easily be taught what TypeGuard[...] means -- the whole *point* is to enable tooling (it provides an essential, albeit minor, piece of new functionality for type checkers). On Sat, Feb 13, 2021 at 2:38 AM Paul Moore <p.f.moore@gmail.com> wrote:

I have to agree here. I'm not a frequent user of type hints yet, but I am starting to have to maintain code that has type hints, and from a maintenance perspective, I have to say that this would be really hard to deal with. If I saw this for the first time "in the wild", I'd have no idea what to do with it.

Think of it as new syntax. Of course you have to learn it (or Google it). Is it easy to learn? I think so. Is it easy to recognize once you've seen and understood it once? Yes. Is it easy to use it correctly given an example? Yes again. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

I think it's a reasonable criticism that it's not obvious that a function annotated with a return type of `TypeGuard[x]` should return a bool. That said, the idea of a user-defined type guard comes from TypeScript, where the syntax is described [here](https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-define...). As you can see, the return type annotation here is also not a boolean. To my knowledge, that has not been a barrier for TypeScript developers. As Guido said, it's something that a developer can easily look up if they are confused about what it means. I'm open to alternative formulations that meet the following requirements: 1. It must be possible to express the type guard within the function signature. In other words, the implementation should not need to be present. This is important for compatibility with type stubs and to guarantee consistent behaviors between type checkers. 2. It must be possible to annotate the input parameter types _and_ the resulting (narrowed) type. It's not sufficient to annotate just one or the other. 3. It must be possible for a type checker to determine when narrowing can be applied and when it cannot. This implies the need for a bool response. 4. It should not require changes to the grammar because that would prevent this from being adopted in most code bases for many years. Mark, none of your suggestions meet these requirements. Gregory, one of your suggestions meets these requirements: ```python def is_str_list(val: Constrains[List[object]:List[str]) -> bool: ... ``` So, the question is whether this is more understandable and readable than this: ```python def is_str_list(val: List[object]) -> TypeGuard[List[str]]: ... ``` Both of these approaches would require a developer to do a search to understand the meaning. The former also introduces a novel use of the slice notation, which I find very unintuitive. Between these two, I find the latter to be clearly preferable, but that's admittedly a subjective opinion. As for choosing the name of the annotation... Most annotations in Python are nouns, for good reason. (There are a few exceptions like `Optional` that are adjectives.) For that reason, I'm not a fan of `Narrows`. I definitely wouldn't use `Constrains` because there's already a meaning in the Python type system for a "constrained type variable" (a TypeVar can constrained to two or more different types). `TypeGuard` is the term that is used in other languages to describe this notion, so it seems reasonable to me to adopt this term rather than making up a new term. Yes, it's introducing a new term that most Python users are not yet familiar with, but I can tell you from experience that very few Python developers know what "type narrowing" means. Some education will be required regardless of the formulation we choose. Steven, you said you'd like to explore a decorator-based formulation. Let's explore that. Here's what that it look like if we were to meet all of the above requirements. ```python @type_guard(List[str]) def is_str_list(val: List[object]) -> bool: ... ``` The problem here, as I mention in the "rejected ideas" section of the PEP, is that even with postponed type evaluations (as described in PEP 563), the interpreter cannot postpone the evaluation of an expression if it's used as the argument to a decorator. That's because it's not being used as a type annotation in this context. So while Mark is correct to point out that there has been a mechanism available for forward references since PEP 484, we've been trying to eliminate the use of quoted type expressions in favor of postponed evaluation. This would add a new case that can't be handled through postponed evaluation. Perhaps you still don't see that as a strong enough justification for rejecting the decorator-based formulation. I'm not entirely opposed to using a decorator here, but I think on balance that the `TypeGuard[x]` formulation is better. Once again, that's a subjective opinion. Paul said:

...to work around deficiencies in the current generation of Python typecheckers

It sounds like you're implying that this functionality will be no longer needed at some point in the future when type checkers improve in some (unspecified) manner. If that's what you meant to convey, then I disagree. I think there will be an ongoing need for this functionality. There's good evidence for this in TypeScript, where user-defined type guards have been adopted widely by developers. -- Eric Traut Contributor to Pyright & Pylance Microsoft Corp.

On 14/02/21 8:48 am, Eric Traut wrote:

def is_str_list(val: Constrains[List[object]:List[str]) -> bool: ...

Maybe something like this? def is_str_list(val: List[str] if True else List[object]) -> bool: ... -- Greg

Here's another suggestion: PEP 593 introduced the `Annotated` type annotation. This could be used to annotate a TypeGuard like this: `def is_str_list(val: List[object]) -> Annotated[bool, TypeGuard(List[str])]` Note that I used ( ) instead of [ ] for the TypeGuard, as it is no longer a type. This should fulfill all four requirements, but is a lot more verbose and therefore also longer. It would also be extensible for other annotations. For the most extensible approach both `-> TypeGuard(...)` and `-> Annotated[bool, TypeGuard(...)]` could be allowed, which would open the path for future non-type-annotations, which could be used regardless of whether the code is type-annotated. -- Adrian On February 14, 2021 2:20:14 PM GMT+01:00, Steven D'Aprano <steve@pearwood.info> wrote:

On Sat, Feb 13, 2021 at 07:48:10PM -0000, Eric Traut wrote:

...

I think it's a reasonable criticism that it's not obvious that a function annotated with a return type of `TypeGuard[x]` should return a bool. [...] As Guido said, it's something that a developer can easily look up if they are confused about what it means.

Yes, developers can use Bing and Google :-)

But it's not the fact that people have to look it up. It's the fact that they need to know that this return annotation is not what it seems, but a special magic value that needs to be looked up.

That's my objection: we're overloading the return annotation to be something other than the return annotation, but only for this one special value. (So far.) If you don't already know that it is special, you won't know that you need to look it up to learn that its special.

...

I'm open to alternative formulations that meet the following requirements:

1. It must be possible to express the type guard within the function signature. In other words, the implementation should not need to be present. This is important for compatibility with type stubs and to guarantee consistent behaviors between type checkers.

When you say "implementation", do you mean the body of the function?

Why is this a hard requirement? Stub files can contain function bodies, usually `...` by convention, but alternatives are often useful, such as docstrings, `raise NotImplementedError()` etc.

https://mypy.readthedocs.io/en/stable/stubs.html

I don't think that the need to support stub files implies that the type guard must be in the function signature. Have I missed something?

...

2. It must be possible to annotate the input parameter types _and_ the resulting (narrowed) type. It's not sufficient to annotate just one or the other.

Naturally :-)

That's the whole point of a type guard, I agree that this is a truly hard requirement.

...

3. It must be possible for a type checker to determine when narrowing can be applied and when it cannot. This implies the need for a bool response.

Do you mean a bool return type? Sorry Eric, sometimes the terminology you use is not familiar to me and I have to guess what you mean.

...

4. It should not require changes to the grammar because that would prevent this from being adopted in most code bases for many years.

Fair enough.

...

Mark, none of your suggestions meet these requirements.

Mark's suggestion to use a variable annotation in the body meets requirements 2, 3, and 4. As I state above, I don't think that requirement 1 needs to be a genuinely hard requirement: stub files can include function bodies.

To be technically precise, stub functions **must** include function bodies. It's just that by convention we use typically use `...` as the body.

...

Gregory, one of your suggestions meets these requirements:

```python def is_str_list(val: Constrains[List[object]:List[str]) -> bool: ... ```

That still misleadingly tells the reader (or naive code analysis software) that parameter val is of type

Contrains[List[object]:List[str]]

whatever that object is, rather than what it *actually* is, namely `List[object]`. I dislike code that misleads the reader.

...

As for choosing the name of the annotation [...] `TypeGuard` is the term that is used in other languages to describe this notion, so it seems reasonable to me to adopt this term

Okay, this reasoning makes sense to me. Whether spelled as a decorator or an annotation, using TypeGuard is reasonable.

...

Steven, you said you'd like to explore a decorator-based formulation. Let's explore that. Here's what that it look like if we were to meet all of the above requirements.

```python @type_guard(List[str]) def is_str_list(val: List[object]) -> bool: ... ```

Okay.

I note that this could be easily extended to support narrowing in the negative case as well:

@type_guard(List[str], List[float]) def is_str_list(val: List[Union[str, float]]) -> bool: ...

...

The problem here, as I mention in the "rejected ideas" section of the PEP, is that even with postponed type evaluations (as described in PEP 563), the interpreter cannot postpone the evaluation of an expression if it's used as the argument to a decorator. That's because it's not being used as a type annotation in this context.

Agreed.

...

So while Mark is correct to point out that there has been a mechanism available for forward references since PEP 484,

That was actually me who pointed out the quoting mechanism for forward references. (Unless Mark also did so.)

...

we've been trying to eliminate the use of quoted type expressions in favor of postponed evaluation. This would add a new case that can't be handled through postponed evaluation. Perhaps you still don't see that as a strong enough justification for rejecting the decorator-based formulation. I'm not entirely opposed to using a decorator here, but I think on balance that the `TypeGuard[x]` formulation is better. Once again, that's a subjective opinion.

I understand the desire to minimize the use of quoted forward references. But the cost to save two quote characters seems high: changing an obvious and straight-forward return annotation to an actively misleadingly special case. (Also, see below for the `Callable` case.)

I'm not convinced that forward references will be common. Did I miss something, or are there no examples in the PEP that require a forward-reference?

# Spam is not defined yet, so a forward reference is needed.

def is_list_of_spam(values:List[object]) -> TypeGuard[List[Spam]]: return all(isinstance(x, Spam) for x in values)

# versus decorator

@type_guard('List[Spam]') def is_list_of_spam(values:List[object]) -> bool: return all(isinstance(x, Spam) for x in values)

Of course, neither of these examples will actually work, because Spam doesn't exist so you can't refer to it in the body. I don't get the sense that this will require forward-references very often. At least not often enough to justify obfuscating the return type.

This obfuscation appears to have a critical consequence too. Please correct me if I am wrong, but quoting your PEP:

""" In all other respects, TypeGuard is a distinct type from bool. It is not a subtype of bool. Therefore, Callable[..., TypeGuard[int]] is not assignable to Callable[..., bool]. """

If I am reading this correctly that implies that if I define these functions:

``` def is_str_list(val: List[object]) -> TypeGuard[List[str]]: return all(isinstance(x, str) for x in val)

def my_filter(func:Callable[object, bool], values:List[object]) -> List[object]: return [x for x in values if func(x)] ```

the type checker would be expected to flag this as invalid:

my_filter(is_str_list, ['a', 'b'])

If I have not misunderstood, surely this is a critical flaw with the PEP?

Eric, for what it's worth, I think that this will be an excellent feature for type checkers, thank you for writing the PEP. It's just the syntax gives me the willies:

- special case handling of TypeGuard in a way that obfuscates the actual return type;

- actively misleads the reader, and any naive code analysis tools that don't know about type guards;

- prevents user type guards from being seen as `Callable[..., bool]` even though they actually are.

And the justification for eliminating decorators seems to be weak to me.

However, I will give you one possible point in its favour: runtime introspection of the annotations by typeguard-aware tools. If an introspection tool is aware of the special meaning of `TypeGuard` in the return annotation, then it is easy to introspect that value at runtime:

# untested... T = func.__annotations__['return'] if T.startswith("TypeGuard"): print("return type is bool") print("narrows to type ...") # TODO: extract from T else: print(f"return type is {T}")

With a decorator, we would need some other solution for runtime introspection of the type guard. That's easy enough to solve, the obvious solution is to just add the information to the function as an attribute. But it is one additional complication.

Still, I think this is a minor issue, especially compared to the critical issue of `Callable[..., bool]`.

I think that the runtime introspection issue will probably rule out Mark's "variable annotation in the body" idea. So considering all the factors as I see them, including familiarity to TypeScript devs:

* Decorator: +1

* Mark's variable annotation, if runtime introspection is neglected: +0.1

* Gregory's idea to annotate the parameter itself: -0.1

* The PEP's return value annotation: -0.2

* Mark's variable annotation, if runtime introspection is required: -1

-- Steve _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OE6N443A... Code of Conduct: http://python.org/psf/codeofconduct/

(there's a small TL;DR towards the end of my reply if you want to read in reverse to follow my thought process from possible conclusions to how i got there - please don't reply without reading the whole thing first) *TL;DR of my TL;DR* - Not conveying bool-ness directly in the return annotation is my only complaint. A BoolTypeGuard spelling would alleviate that. I'm +0.3 now. Otherwise I elaborate on other guarding options and note a few additional Rejected/Postponed/Deferred Ideas sections that the PEP should mention as currently out of scope: Unconditional guards, multiple guarded parameters, and type mutators. Along the way I work my way towards suggestions for those, but I think they don't belong in _this_ PEP and could serve as input for future ones if/when desired. On Sun, Feb 14, 2021 at 8:53 AM Paul Bryan <pbryan@anode.ca> wrote:

I'm a +1 on using Annotated in this manner. Guido mentioned that it was intended for only third-parties though. I'd like to know more about why this isn't a good pattern for use by Python libraries.

On Sun, 2021-02-14 at 16:29 +0100, Adrian Freund wrote:

Here's another suggestion:

PEP 593 introduced the `Annotated` type annotation. This could be used to annotate a TypeGuard like this:

`def is_str_list(val: List[object]) -> Annotated[bool, TypeGuard(List[str])]`

I like Annotated better than not having it for the sake of not losing the return type. BUT I still feel like this limits things too much and disconnects the information about what parameter(s) are transformed. It also doesn't solve the problem of why the guard _must_ be tied to the return value. Clearly sometimes it is desirable to do that. But on many other scenarios the act of not raising an exception is the narrowing action: ie - it should be declared as always happening. Nothing in the above annotation reads explicitly to me as saying that the return value determines the type outcome.

Note that I used ( ) instead of [ ] for the TypeGuard, as it is no longer a type.

This should fulfill all four requirements, but is a lot more verbose and therefore also longer. It would also be extensible for other annotations.

For the most extensible approach both `-> TypeGuard(...)` and `-> Annotated[bool, TypeGuard(...)]` could be allowed, which would open the path for future non-type-annotations, which could be used regardless of whether the code is type-annotated.

-- Adrian

On February 14, 2021 2:20:14 PM GMT+01:00, Steven D'Aprano < steve@pearwood.info> wrote:

On Sat, Feb 13, 2021 at 07:48:10PM -0000, Eric Traut wrote:

I think it's a reasonable criticism that it's not obvious that a function annotated with a return type of `TypeGuard[x]` should return a bool.

[...]

As Guido said, it's something that a developer can easily look up if they are confused about what it means.

Yes, developers can use Bing and Google :-)

But it's not the fact that people have to look it up. It's the fact that they need to know that this return annotation is not what it seems, but a special magic value that needs to be looked up.

That's my objection: we're overloading the return annotation to be something other than the return annotation, but only for this one special value. (So far.) If you don't already know that it is special, you won't know that you need to look it up to learn that its special.

I'm open to alternative formulations that meet the following requirements:

1. It must be possible to express the type guard within the function signature. In other words, the implementation should not need to be present. This is important for compatibility with type stubs and to guarantee consistent behaviors between type checkers.

When you say "implementation", do you mean the body of the function?

Why is this a hard requirement? Stub files can contain function bodies, usually `...` by convention, but alternatives are often useful, such as docstrings, `raise NotImplementedError()` etc.

https://mypy.readthedocs.io/en/stable/stubs.html

I don't think that the need to support stub files implies that the type guard must be in the function signature. Have I missed something?

2. It must be possible to annotate the input parameter types _and_ the resulting (narrowed) type. It's not sufficient to annotate just one or the other.

Naturally :-)

That's the whole point of a type guard, I agree that this is a truly hard requirement.

3. It must be possible for a type checker to determine when narrowing can be applied and when it cannot. This implies the need for a bool response.

Do you mean a bool return type? Sorry Eric, sometimes the terminology you use is not familiar to me and I have to guess what you mean.

4. It should not require changes to the grammar because that would prevent this from being adopted in most code bases for many years.

Fair enough.

Mark, none of your suggestions meet these requirements.

Mark's suggestion to use a variable annotation in the body meets requirements 2, 3, and 4. As I state above, I don't think that requirement 1 needs to be a genuinely hard requirement: stub files can include function bodies.

To be technically precise, stub functions **must** include function bodies. It's just that by convention we use typically use `...` as the body.

Gregory, one of your suggestions meets these requirements:

```python def is_str_list(val: Constrains[List[object]:List[str]) -> bool: ... ```

That still misleadingly tells the reader (or naive code analysis software) that parameter val is of type

Contrains[List[object]:List[str]]

whatever that object is, rather than what it *actually* is, namely `List[object]`. I dislike code that misleads the reader.

Something like this wouldn't mislead the reader: def is_strs(items: List[Any] narrows to List[str]) -> bool: ... But that's unlikely desirable from an added syntax perspective. To use an existing token as I suggested in the text of my earlier reply: def is_strs(items: List[Any] -> List[str]) -> bool: ... again, requires parsing updates but we're not introducing new soft keywords. def is_strs(items: Narrows[List[Any] -> List[str]]) -> bool: ... Even if those are written using existing syntax parsable today such as Narrows[List[Any]*, *List[str]] Constrains[List[Any]*:*List[str]] Narrows*(*List[Any], List[str]*)* Constrains*TypeTo(*List[Any], List[str]) NarrowsTypeTo(List[Any], List[str]) Words like *Constrains* and *Narrows* are intentionally *verbs*. TypeGuard is a noun. A verb conveys an action so it is reasonable to read those as not being a type themselves but indicating that something is happening to the types. Adrian's suggestion of using () instead of [] on TypeGuard to help convey the action. This could also be used here (as in a couple of those examples above) to make it even more obvious to the reader that Narrows, Constrains, NarrowsTo, ConstrainsTo, ConstrainsTypeTo, (whatever verb name we choose)... is not a type itself but a descriptor of the action being taken on this type. def is_strs(items: Constrains(List[str], List[int])) -> bool: ... The return type is not consumed, hidden, or altered by these proposals. The specific parameter(s) being acted upon are annotated with the action verb. This allows for more than just trivial signature single parameter bool return functions to be used as type guards. def validate_and_count( identifiers: ConstrainsTypeTo(List[Any], List[str]), results: ConstrainsTypeTo(List[Any], List[Optional[float]]) ) -> int: """Validates all identifiers and results. Returns the number of non-None values.""" Clearly a made up example for illustration, but my point is more a question of why restrict this feature to mere `f(a)->bool` functions? We could take this further and instead of merely offering a type guard that constrains a type to a narrower definition, offer the ability to annotate functions that mutate the type of their mutable parameters. Today that isn't supported at all, we require a return value for such an action. But with this form of annotation one can easily imagine: def sanitize_dates(dates: ConvertsTypeTo(List[str], List[datetime]) -> Set[str]: """Converts the date strings to datetime objects in place, extracts and returns any unparsable ones.""" Whether you think that function is good API designs or not: Functions that do this sort of thing exist. Indicating things on the parameter gives the ability for them to contribute to static analysis by conveying their action. I'm still not sure how to convey when a type guarding, narrowing, constraining, or converting action is tied to the return value of the function or not. When indicating things solely on the arguments themselves, that seems to be more likely to read as an un-tied non return value based constraint. Perhaps that's what we want. We'd end up with this for the always unless raises case, and an annotation on the return value for the limited type guard "def f(a)->bool" situation. *TL;DR* Does that take us full circle to: def is_str_list(value: List[Any]) -> BoolTypeGuard[List[str]]: ... for the PEP 647 being proposed that *only* covers f(x)->bool cases? *(note my simple addition of Bool to the name - the unaware reader can see that and assume "eh, it's a boolean")* and a potential follow-on PEP for more complicated situations of constraining or mutating the types of potentially multiple parameters regardless of return value as I've described above? I *do like* how conservative the PEP 647 is in what it supports despite anything I complain about above. The existing "User-defined type guards apply narrowing only in the positive case (the if clause). The type is not narrowed in the negative case." text... is +1 for a first implementation. It's more a matter of spelling and readability for me, while opening my mind to the idea of other type transformations that we don't have a way to communicate that I've tried to brainstorm on here. Capturing the scope and potential directions to explore for the future in the PEP would be good. You've got a little bit of that in the Conditionally Applying TypeGuard Type and Narrowing Arbitrary Parameters sections. Perhaps add Rejected/Postponed ideas sections: One for "Unconditional TypeGuard" to cover functions that raise if the guarding logic fails? And another for "Multiple Guarded Parameters" and maybe one more for "Mutating Type Conversions"? I'm not the one driving the need for this to be implemented. I'm just aiming to ensure we end up with a readable result rather than something for readers eyes to glaze over on. Or feel like the language is gatekeeping by making readers feel like impostors if they don't understand type theory. +0.3 I'm coming around on the narrow (guarded?) existing PEP, it's mostly a matter of spelling to convey the boolean return value. -gps

On Sun, Feb 14, 2021 at 12:51 PM David Mertz <mertz@gnosis.cx> wrote:

On Sun, Feb 14, 2021, 2:53 PM Gregory P. Smith <greg@krypto.org> wrote:

...

*TL;DR of my TL;DR* - Not conveying bool-ness directly in the return annotation is my only complaint. A BoolTypeGuard spelling would alleviate that.

This is exactly my feeling as well. In fact, I do not understand why it cannot simply be a parameterized Bool. That would avoid all confusion. Yes, it's not the technical jargon type system designers use... But the existing proposal moves all the mental effort to non-experts who may never use type checking tools.

But note that 'bool' in Python is not subclassable.

I.e. if I read this:

def is_str_list(v: List[Any]) -> Bool[List[str]]: ...

I just think, "OK, it returns a bool, and those mypy people are doing something else funny that I don't need to worry about." If I see TypeGuard[...whatever..] I automatically think "that's a weird thing I don't understand, but it looks like it returns a boolean when I read the body."

Then again, someone who tries to understand your version beyond "fuzzy matching of words while ignoring stuff they don't comprehend" (that's a real thing and we all do it all the time :-) might be confused by the conundrum posed by a parameterized boolean.

FWIW, this hypothetical non-expert is me. I've actually gotten in the habit of using simply type annotations in most of the code I write (a lot of which is for teaching). But I never actually check using type checkers. It's quite likely some of my annotations are not actually sound, but to me they are documentation. I.e. I'll use an annotation of str/list/float, but not precisely parameterized types, union, generics, etc.

So maybe you should just stick to "bool". TypeGuard is really only useful for people who care about what type checkers do (a surprisingly large contingent, actually, but I would not recommend it for the education department). And I'm sure that when five years from now you're seeing TypeGuard in the wild, you'll remember this thread and you'll know that it is a boolean. :-) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On Sun, Feb 14, 2021, 5:34 PM Guido van Rossum <guido@python.org> wrote:

But note that 'bool' in Python is not subclassable.

Sure. But that's why I suggested 'Bool' rather than 'bool'. It's a different spelling, but one with a really obvious connection.

I.e. if I read this:

...

def is_str_list(v: List[Any]) -> Bool[List[str]]: ...

I just think, "OK, it returns a bool, and those mypy people are doing something else funny that I don't need to worry about."

...

So maybe you should just stick to "bool". TypeGuard is really only useful for people who care about what type checkers do (a surprisingly large contingent, actually, but I would not recommend it for the education department).

I'm fairly sure I *will* just stick with 'bool' in code I write, no matter the outcome of this PEP. But I'm also sure I'll read code written by other people in the future... And EXPLAIN code written by other people to people less knowledgeable than I am. When explaining code to students or junior colleagues, it feels MUCH easier to explain "Bool[...] is a funny way of spelling 'bool' (in annotations)" than explaining the same of TypeGuard [...].

On Sun, Feb 14, 2021 at 11:49 AM Gregory P. Smith <greg@krypto.org> wrote:

*TL;DR of my TL;DR* - Not conveying bool-ness directly in the return annotation is my only complaint. A BoolTypeGuard spelling would alleviate that. I'm +0.3 now. Otherwise I elaborate on other guarding options and note a few additional Rejected/Postponed/Deferred Ideas sections that the PEP should mention as currently out of scope: Unconditional guards, multiple guarded parameters, and type mutators. Along the way I work my way towards suggestions for those, but I think they don't belong in _this_ PEP and could serve as input for future ones if/when desired.

Thanks for having an open mind. We should try to find a name that conveys the boolean-ness to casual readers. But I note that "naming is hard" and there are other aspects that already convey the boolean-ness (e.g. the naming convention starting with 'is', and the presence of 'return True' in the body :-), and I don't think that the proposed 'BoolTypeGuard' adds readability. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Could you summarize your proposal in a few lines? I've tried to read that email several times now and I still don't follow the proposal. You can leave the reasoning *why* you believe your proposal is better out -- I just want to see what it will look like (how to define a type guard, and how to use it). On Tue, Feb 16, 2021 at 11:39 AM Joseph Perez <joperez@hotmail.fr> wrote:

I've proposed PEP 593 `Annotated` too, but in the typing-sig mailing list: https://mail.python.org/archives/list/typing-sig@python.org/message/CVLLRWU7... and Guido had the following answer:

...

I see PEP 593 as a verbose solution to the problem "how do we use annotations for static typing and for runtime metadata simultaneously". Type guards solve a problem that's entirely in the realm of static typing, so IMO it would be an abuse of Annotated.

(I've also written in the mail about checked cast as an alternative solution to type guard.)

...

For the most extensible approach both -> TypeGuard(...) and -> Annotated[bool, TypeGuard(...)] could be allowed, which would open the path for future non-type-annotations, which could be used regardless of whether the code is type-annotated. I've proposed a possible implementation in my mail linked above.

---

Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/B3QZTPW6... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Eric Traut wrote: """It must be possible to express the type guard within the function signature. In other words, the implementation should not need to be present. This is important for compatibility with type stubs and to guarantee consistent behaviors between type checkers.""" Can type stubs include a docstring? If so, then adding subsequent lines that are only parameter annotations doesn't seem like a problem. And frankly, allowing subsequent lines (possibly restricted to first-except-docstrings) doesn't seem any more invasive than adding another magic type that has to be interpreted differently. (I understand it might be more coding in practice, based on one solution having already been written.) def is_str_list(val: List[object]) -> bool: """Determines whether all objects in the list are strings""" val: NarrowsTo[List[str]] I agree that Greg's suggestion of def is_str_list(val: Constrains[List[object]:List[str]) -> bool: also meets the criteria you list, but ... as a human reader, that signature is getting too heavy. I'll also note that I don't think TypeScript is a good model for "this won't be a problem." It is probably a good model for "this will work for the people who *want* to go whole hog on explicit typing", but for people who are at best agnostic about typing ... they would stick with JavaScript, instead of using TypeScript. Alas, this change isn't being proposed just for TypedPython; it is being proposed for baseline python. -jJ