I'm a +1 on using Annotated in this manner. Guido mentioned that it was intended for only third-parties though. I'd like to know more about why this isn't a good pattern for use by Python libraries. 

On Sun, 2021-02-14 at 16:29 +0100, Adrian Freund wrote:
Here's another suggestion:

PEP 593 introduced the `Annotated` type annotation. This could be used to annotate a TypeGuard like this:

`def is_str_list(val: List[object]) -> Annotated[bool, TypeGuard(List[str])]`

Note that I used ( ) instead of [ ] for the TypeGuard, as it is no longer a type.

This should fulfill all four requirements, but is a lot more verbose and therefore also longer.
It would also be extensible for other annotations.

For the most extensible approach both `-> TypeGuard(...)` and `-> Annotated[bool, TypeGuard(...)]` could be allowed, which would open the path for future non-type-annotations, which could be used regardless of whether the code is type-annotated.


On February 14, 2021 2:20:14 PM GMT+01:00, Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Feb 13, 2021 at 07:48:10PM -0000, Eric Traut wrote:

I think it's a reasonable criticism that it's not obvious that a 
function annotated with a return type of `TypeGuard[x]` should return
a bool.

As Guido said, it's something that a developer can easily 
look up if they are confused about what it means.

Yes, developers can use Bing and Google :-)

But it's not the fact that people have to look it up. It's the fact that
they need to know that this return annotation is not what it seems, but
a special magic value that needs to be looked up.

That's my objection: we're overloading the return annotation to be
something other than the return annotation, but only for this one
special value. (So far.) If you don't already know that it is special,
you won't know that you need to look it up to learn that its special.

 I'm open to alternative formulations that meet the following requirements:

1. It must be possible to express the type guard within the function
signature. In other words, the implementation should not need to be
present. This is important for compatibility with type stubs and to
guarantee consistent behaviors between type checkers.

When you say "implementation", do you mean the body of the function?

Why is this a hard requirement? Stub files can contain function
bodies, usually `...` by convention, but alternatives are often useful,
such as docstrings, `raise NotImplementedError()` etc.


I don't think that the need to support stub files implies that the type
guard must be in the function signature. Have I missed something?

2. It must be possible to annotate the input parameter types _and_ the 
resulting (narrowed) type. It's not sufficient to annotate just one or
the other.

Naturally :-)

That's the whole point of a type guard, I agree that this is a truly
hard requirement.

3. It must be possible for a type checker to determine when narrowing 
can be applied and when it cannot. This implies the need for a bool

Do you mean a bool return type? Sorry Eric, sometimes the terminology
you use is not familiar to me and I have to guess what you mean.

4. It should not require changes to the grammar because that would 
prevent this from being adopted in most code bases for many years.

Fair enough.

Mark, none of your suggestions meet these requirements.

Mark's suggestion to use a variable annotation in the body meets
requirements 2, 3, and 4. As I state above, I don't think that
requirement 1 needs to be a genuinely hard requirement: stub files can
include function bodies.

To be technically precise, stub functions **must** include function
bodies. It's just that by convention we use typically use `...` as the

Gregory, one of your suggestions meets these requirements:

def is_str_list(val: Constrains[List[object]:List[str]) -> bool:

That still misleadingly tells the reader (or naive code analysis
software) that parameter val is of type


whatever that object is, rather than what it *actually* is, namely
`List[object]`. I dislike code that misleads the reader.

As for choosing the name of the annotation

`TypeGuard` is the term that is used in other languages to describe 
this notion, so it seems reasonable to me to adopt this term

Okay, this reasoning makes sense to me. Whether spelled as a decorator
or an annotation, using TypeGuard is reasonable.

Steven, you said you'd like to explore a decorator-based formulation. 
Let's explore that. Here's what that it look like if we were to meet
all of the above requirements.

def is_str_list(val: List[object]) -> bool: ...


I note that this could be easily extended to support narrowing in the
negative case as well:

@type_guard(List[str], List[float])
def is_str_list(val: List[Union[str, float]]) -> bool: ...

The problem here, as I mention in the "rejected ideas" section of the 
PEP, is that even with postponed type evaluations (as described in PEP
563), the interpreter cannot postpone the evaluation of an expression
if it's used as the argument to a decorator. That's because it's not
being used as a type annotation in this context.


So while Mark is 
correct to point out that there has been a mechanism available for
forward references since PEP 484,

That was actually me who pointed out the quoting mechanism for forward
references. (Unless Mark also did so.)

we've been trying to eliminate the 
use of quoted type expressions in favor of postponed evaluation. This
would add a new case that can't be handled through postponed
evaluation. Perhaps you still don't see that as a strong enough
justification for rejecting the decorator-based formulation. I'm not
entirely opposed to using a decorator here, but I think on balance
that the `TypeGuard[x]` formulation is better. Once again, that's a
subjective opinion.

I understand the desire to minimize the use of quoted forward
references. But the cost to save two quote characters seems high:
changing an obvious and straight-forward return annotation to an
actively misleadingly special case. (Also, see below for the `Callable`

I'm not convinced that forward references will be common. Did I miss
something, or are there no examples in the PEP that require a

# Spam is not defined yet, so a forward reference is needed.

def is_list_of_spam(values:List[object]) -> TypeGuard[List[Spam]]:
return all(isinstance(x, Spam) for x in values)

# versus decorator

def is_list_of_spam(values:List[object]) -> bool:
return all(isinstance(x, Spam) for x in values)

Of course, neither of these examples will actually work, because Spam
doesn't exist so you can't refer to it in the body. I don't get the
sense that this will require forward-references very often. At least not
often enough to justify obfuscating the return type.

This obfuscation appears to have a critical consequence too. Please
correct me if I am wrong, but quoting your PEP:

In all other respects, TypeGuard is a distinct type from bool. It is
not a subtype of bool. Therefore, Callable[..., TypeGuard[int]] is not
assignable to Callable[..., bool].

If I am reading this correctly that implies that if I define these

def is_str_list(val: List[object]) -> TypeGuard[List[str]]:
return all(isinstance(x, str) for x in val)

def my_filter(func:Callable[object, bool], values:List[object]) ->
return [x for x in values if func(x)]

the type checker would be expected to flag this as invalid:

my_filter(is_str_list, ['a', 'b'])

If I have not misunderstood, surely this is a critical flaw with the

Eric, for what it's worth, I think that this will be an excellent
feature for type checkers, thank you for writing the PEP. It's just the
syntax gives me the willies:

- special case handling of TypeGuard in a way that obfuscates the actual
return type;

- actively misleads the reader, and any naive code analysis tools that
don't know about type guards;

- prevents user type guards from being seen as `Callable[..., bool]`
even though they actually are.

And the justification for eliminating decorators seems to be weak to me.

However, I will give you one possible point in its favour: runtime
introspection of the annotations by typeguard-aware tools. If an
introspection tool is aware of the special meaning of `TypeGuard` in the
return annotation, then it is easy to introspect that value at runtime:

# untested...
T = func.__annotations__['return']
if T.startswith("TypeGuard"):
print("return type is bool")
print("narrows to type ...") # TODO: extract from T
print(f"return type is {T}")

With a decorator, we would need some other solution for runtime
introspection of the type guard. That's easy enough to solve, the
obvious solution is to just add the information to the function as an
attribute. But it is one additional complication.

Still, I think this is a minor issue, especially compared to the
critical issue of `Callable[..., bool]`.

I think that the runtime introspection issue will probably rule out
Mark's "variable annotation in the body" idea. So considering all the
factors as I see them, including familiarity to TypeScript devs:

* Decorator: +1

* Mark's variable annotation, if runtime introspection is neglected: +0.1

* Gregory's idea to annotate the parameter itself: -0.1

* The PEP's return value annotation: -0.2

* Mark's variable annotation, if runtime introspection is required: -1