Moving typeshed representation of generic callable types to lowest common denominator
I recently discovered that MyPy supports a generic callable type shorthand that Pyre does not, namely things of the form ``` from typing import Callable, TypeVar T = TypeVar("T") def produce_no_op_decorator() -> Callable[[T], T]:... ``` MyPy interprets this as being of type `Callable[[], Callable<T>[[T], T]]` (mod https://github.com/python/mypy/issues/3924), while Pyre interprets it as being of type `Callable<T>[[], Callable[[T], T]]`, which we reject as being invalid. Pyre interprets all free type variables in a def or class as being in the scope of that define/class. MyPy also does this in most circumstances with Callable[[T], T], as both checkers have identical interpretations of ``` T = TypeVar("T") def f(x: T) -> Callable[[T], T]: ... # def <T> (T) -> Callable[[T], T] # and def f(f: Callable[[T], T]) -> T: ... # def <T> (Callable[[T], T]) -> T #and def i() -> Tuple[Callable[[T], T], T]: ... # def <T> () -> Tuple[Callable[[T], T], T]: ... # and class Child(Base[Callable[[T], T]]): ... # class <T> Child(Base[Callable[[T], T]]) ``` Pyre and MyPy only differ here in that MyPy introduces a unique special case that alters the behavior for free variables: * inside a Callable type * in the return type of a def * not present in any parameter of the def * not present anywhere else in the return type that isn't also inside of a Callable type. I can definitely see how this special case is helpful in making definitions for decorator factories more compact, but for us the inconsistency of this behavior from a user's perspective outweighs the concision. With all that being said, given that the semantics of this are unspecified in any PEPs, I think it's fine for there to be a divergence in interpretation here. However, there are instances of this pattern in typeshed, which have been leading to strange behavior in Pyre. Luckily, both MyPy and Pyre have a spelling of `produce_no_op_decorator` that they both agree on, and is mostly codified by a PEP: ``` from typing import Callable, TypeVar, Protocol T = TypeVar("T") class IdentityFunction(Protocol): def __call__(self, __x: T) -> T: ... def produce_no_op_decorator() -> IdentityFunction... ``` What are people's thoughts on modifying typeshed (like in https://github.com/python/typeshed/pull/4045) to change these instances to the commonly accepted spelling?
I like the `Callable[[T], T]` shorthand. pytype started supporting it because people were annotating decorator factories this way and getting confused when it didn't work, which matches my intuition that this is the "obvious" way of annotating such things, and it's a good user experience when the obvious things just work. I find the Protocol much less readable. Best, Rebecca On Mon, Jun 1, 2020 at 7:39 PM Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
I recently discovered that MyPy supports a generic callable type shorthand that Pyre does not, namely things of the form
``` from typing import Callable, TypeVar T = TypeVar("T") def produce_no_op_decorator() -> Callable[[T], T]:... ```
MyPy interprets this as being of type `Callable[[], Callable<T>[[T], T]]` (mod https://github.com/python/mypy/issues/3924), while Pyre interprets it as being of type `Callable<T>[[], Callable[[T], T]]`, which we reject as being invalid.
Pyre interprets all free type variables in a def or class as being in the scope of that define/class. MyPy also does this in most circumstances with Callable[[T], T], as both checkers have identical interpretations of
``` T = TypeVar("T") def f(x: T) -> Callable[[T], T]: ... # def <T> (T) -> Callable[[T], T] # and def f(f: Callable[[T], T]) -> T: ... # def <T> (Callable[[T], T]) -> T #and def i() -> Tuple[Callable[[T], T], T]: ... # def <T> () -> Tuple[Callable[[T], T], T]: ... # and class Child(Base[Callable[[T], T]]): ... # class <T> Child(Base[Callable[[T], T]]) ```
Pyre and MyPy only differ here in that MyPy introduces a unique special case that alters the behavior for free variables: * inside a Callable type * in the return type of a def * not present in any parameter of the def * not present anywhere else in the return type that isn't also inside of a Callable type.
I can definitely see how this special case is helpful in making definitions for decorator factories more compact, but for us the inconsistency of this behavior from a user's perspective outweighs the concision.
With all that being said, given that the semantics of this are unspecified in any PEPs, I think it's fine for there to be a divergence in interpretation here. However, there are instances of this pattern in typeshed, which have been leading to strange behavior in Pyre.
Luckily, both MyPy and Pyre have a spelling of `produce_no_op_decorator` that they both agree on, and is mostly codified by a PEP:
``` from typing import Callable, TypeVar, Protocol T = TypeVar("T") class IdentityFunction(Protocol): def __call__(self, __x: T) -> T: ... def produce_no_op_decorator() -> IdentityFunction... ```
What are people's thoughts on modifying typeshed (like in https://github.com/python/typeshed/pull/4045) to change these instances to the commonly accepted spelling? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: rechen@google.com
In isolation, I do agree with you that `def produces_identity() -> Callable[[T], T]:...` is the most intuitive way to spell a function that produces an identity function. However, I don't agree that it's intuitive to treat that Callable type there differently from `def takes_identity(f: Callable[[T], T]) -> None:..`, where I believe that pytype, mypy and pyre all agree that those `T`s are in the scope of the `def`. With all that being said, we're not arguing that anyone else should modify the behavior of their type checkers. Instead, we're just asking that in non-standardized areas such as these we have typeshed correspond to the lowest common denominator between the type checkers and spell things in a way that we all agree on. Best, Mark
On 2 Jun 2020, at 04:38, Mark Mendoza <mendoza.mark.a@gmail.com> wrote:
I recently discovered that MyPy supports a generic callable type shorthand that Pyre does not, namely things of the form
``` from typing import Callable, TypeVar T = TypeVar("T") def produce_no_op_decorator() -> Callable[[T], T]:... ```
MyPy interprets this as being of type `Callable[[], Callable<T>[[T], T]]` (mod https://github.com/python/mypy/issues/3924), while Pyre interprets it as being of type `Callable<T>[[], Callable[[T], T]]`, which we reject as being invalid.
Pyre interprets all free type variables in a def or class as being in the scope of that define/class. MyPy also does this in most circumstances with Callable[[T], T], as both checkers have identical interpretations of
``` T = TypeVar("T") def f(x: T) -> Callable[[T], T]: ... # def <T> (T) -> Callable[[T], T] # and def f(f: Callable[[T], T]) -> T: ... # def <T> (Callable[[T], T]) -> T #and def i() -> Tuple[Callable[[T], T], T]: ... # def <T> () -> Tuple[Callable[[T], T], T]: ... # and class Child(Base[Callable[[T], T]]): ... # class <T> Child(Base[Callable[[T], T]]) ```
Pyre and MyPy only differ here in that MyPy introduces a unique special case that alters the behavior for free variables: * inside a Callable type * in the return type of a def * not present in any parameter of the def * not present anywhere else in the return type that isn't also inside of a Callable type.
I can definitely see how this special case is helpful in making definitions for decorator factories more compact, but for us the inconsistency of this behavior from a user's perspective outweighs the concision.
With all that being said, given that the semantics of this are unspecified in any PEPs, I think it's fine for there to be a divergence in interpretation here. However, there are instances of this pattern in typeshed, which have been leading to strange behavior in Pyre.
Luckily, both MyPy and Pyre have a spelling of `produce_no_op_decorator` that they both agree on, and is mostly codified by a PEP:
``` from typing import Callable, TypeVar, Protocol T = TypeVar("T") class IdentityFunction(Protocol): def __call__(self, __x: T) -> T: ... def produce_no_op_decorator() -> IdentityFunction... ```
What are people's thoughts on modifying typeshed (like in https://github.com/python/typeshed/pull/4045) to change these instances to the commonly accepted spelling? _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jakub@stasiak.at
For what it’s worth I, as a user, would really expect def f() -> Callable[[T], T] # and similar to be non-generic function returning a generic one (consistent with Mypy) and I’ve been using this pattern in my code for a long time now. But then I’d also expect def g() -> T to be rejected (how can the caller of the function influence T when there’s no parameter(s) of type T and it’s a free function as opposed to a method belonging to a class generic in T?). Mypy currently allows it but I don’t really see how it’s useful (I may not be seeing bigger picture): % cat test.py from typing import cast, TypeVar T = TypeVar('T') def foo() -> T: return cast(T, 'surprise') reveal_type(foo) reveal_type(foo()) % mypy test.py test.py:9: note: Revealed type is 'def [T] () -> T`-1' test.py:10: note: Revealed type is '<nothing>' What does Pyre do in this case? Best, Jakub
Pyre errors on that function definition (and the cast) with `Invalid type variable [34]: The type variable `Variable[T]` isn't present in the function's parameters.`.
1. I would like to re-focus the discussion on the pull request linked to in the initial post rather than the particular semantics of unbound type variables in callables: https://github.com/python/typeshed/pull/4045 Specifically,
@mrkmndz Sorry for forgetting to answer. Yes, considering theconsequences this has for the typing infrastructure, I think this should be discussed on typing-sig.
@srittau: could you clarify what consequences you meant? What's the specific concern that should be discussed? 2. I would also like to hear people's thoughts on how to deal with disagreements on semantics where not specified in PEPs in shared infrastructure. This is one particular example but I'm sure this will happen more in the future.
Am 09.06.20 um 03:35 schrieb Dominik Gabi:
1. I would like to re-focus the discussion on the pull request linked to in the initial post rather than the particular semantics of unbound type variables in callables: https://github.com/python/typeshed/pull/4045
Specifically,
@mrkmndz Sorry for forgetting to answer. Yes, considering theconsequences this has for the typing infrastructure, I think this should be discussed on typing-sig. @srittau: could you clarify what consequences you meant? What's the specific concern that should be discussed?
Please see the referenced PR. In this case, for example, replacing all cases of Callable[[_T], _T] and similar with a protocol in typeshed. As I wrote earlier, now that we have a _typeshed.pyi file, I could live with the definition of a generic IdentityFunction type in that file. But this covers only this particular case. - Sebastian
Not sure I understand. Does that mean you'd be comfortable with the PR if the protocol moved into that common file? I would also like to hear your thoughts on (2). It's important for us to understand what the expectations are – specifically whether the PEPs or mypy's implementation will be considered the source of truth for semantics in typeshed.
On 9 Jun 2020, at 03:35, Dominik Gabi <dkgispam@gmail.com> wrote:
1. I would like to re-focus the discussion on the pull request linked to in the initial post rather than the particular semantics of unbound type variables in callables: https://github.com/python/typeshed/pull/4045
Specifically,
@mrkmndz Sorry for forgetting to answer. Yes, considering theconsequences this has for the typing infrastructure, I think this should be discussed on typing-sig.
@srittau: could you clarify what consequences you meant? What's the specific concern that should be discussed?
2. I would also like to hear people's thoughts on how to deal with disagreements on semantics where not specified in PEPs in shared infrastructure. This is one particular example but I'm sure this will happen more in the future. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jakub@stasiak.at
Sure – I merely meant to bring some context to the thread. With my type hint writer and occasional consumer hat on I’d like to both be able to write and consume def fun() -> Callable[[T], T] because the other interpretation of the relevant specification doesn’t provide me value (I get an error). In general case, all things considered, if there’s one interpretation of some pattern that causes an error and another that does something desirable, precise, unambiguous and not really causing any errors by itself I’d like to be able to use this pattern and for the tools to converge on it. Replacing Callable[[T], T] with IdentityFunction adds an indirection which makes the type hints slightly worse to consume in my opinion. And I expect there are many uses of this pattern outside typeshed that’ll be difficult to track down and convert in similar fashion. Best, Jakub
I believe typeshed should generally follow the semantics defined in the typing PEPs, but we should also accommodate implementation choices made by the various type checkers (within reason). That's especially true where the PEPs are ambiguous or don't define concrete semantics. In this specific case, that means I'm in support of using the pattern Mark is suggesting in typeshed. El mar., 9 jun. 2020 a las 19:32, Jakub Stasiak (<jakub@stasiak.at>) escribió:
On 9 Jun 2020, at 03:35, Dominik Gabi <dkgispam@gmail.com> wrote:
1. I would like to re-focus the discussion on the pull request linked to in the initial post rather than the particular semantics of unbound type variables in callables: https://github.com/python/typeshed/pull/4045
Specifically,
@mrkmndz Sorry for forgetting to answer. Yes, considering theconsequences this has for the typing infrastructure, I think this should be discussed on typing-sig.
@srittau: could you clarify what consequences you meant? What's the specific concern that should be discussed?
2. I would also like to hear people's thoughts on how to deal with disagreements on semantics where not specified in PEPs in shared infrastructure. This is one particular example but I'm sure this will happen more in the future. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jakub@stasiak.at
Sure – I merely meant to bring some context to the thread. With my type hint writer and occasional consumer hat on I’d like to both be able to write and consume
def fun() -> Callable[[T], T]
because the other interpretation of the relevant specification doesn’t provide me value (I get an error).
In general case, all things considered, if there’s one interpretation of some pattern that causes an error and another that does something desirable, precise, unambiguous and not really causing any errors by itself I’d like to be able to use this pattern and for the tools to converge on it. Replacing Callable[[T], T] with IdentityFunction adds an indirection which makes the type hints slightly worse to consume in my opinion. And I expect there are many uses of this pattern outside typeshed that’ll be difficult to track down and convert in similar fashion.
Best, Jakub _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
participants (6)
-
Dominik Gabi -
Jakub Stasiak -
Jelle Zijlstra -
Mark Mendoza -
Rebecca Chen -
Sebastian Rittau