RFC on Callable Type Syntax
Hello all, Typing-sig has been discussing user-friendly syntax for the type used to represent callables. [1] Since this affects the Python language syntax, we wanted to get some high-level feedback from you before putting up a detailed PEP. TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) -> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are there concerns we need to keep in mind about such a syntax change? 2. Should we propose syntax for additional features in the same PEP or in a future PEP? # Motivation Guido has pointed out that `Callable` is one of the most frequently used type forms but has the least readable syntax. For example, say we have a callback `formatter` that accepts a record and a list of permissions: ```python def print_purchases( user, formatter, # <-- callback ): <...> output = formatter(record, permissions) print(output) ``` To give it a type, we currently have to write: ```python from typing import Callable def print_purchases( user: User, formatter: Callable[[PurchaseRecord, List[AuthPermission]], FormattedItem] # Hard to read. ) -> None: <...> output = formatter(record, permissions) print(output) ``` `Callable` can be hard to understand for new users since it doesn't resemble existing function signatures and there can be multiple square brackets. It also requires an import from `typing`, which is inconvenient. Around 40% of the time [2], users just give up on precisely typing the parameters and return type of their callbacks and just leave it as a blank Callable. In other cases, they don't add a type for their callback at all. Both mean that they lose the safety guarantees from typing and leave room for bugs. We believe that adding friendly syntax for Callable types will improve readability and encourage users to type their callbacks more precisely. Other modern, gradually-typed languages like TypeScript (JS), Hack (PHP), etc. have special syntax for Callable types. (As a precedent, PEP 604 recently added clean, user-friendly syntax for the widely-used `Union` type. Instead of importing `Union` and writing `expr: Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr | SubtractExpr`.) # Proposal and Questions We have a two-part proposal and questions for you: 1. Syntax to replace Callable After a lot of discussion, there is strong consensus in typing-sig about adding syntax to replace Callable. So, the above example would be written as: ```python def print_purchases( user: User, formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem, ) -> None: <...> output = formatter(record, permissions) print(output) ``` This feels more natural and succinct. Async callbacks currently need to be written as ``` from typing import Callable async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]] ``` With the new syntax, they would be written as ``` async_callback: async (HttpRequest) -> HttpResponse ``` which again seems more natural. There is similar syntax for the type of decorators that pass on *args and **kwargs to the decorated function. Note that we considered and rejected using a full def-signature syntax like ```` (record: PurchaseRecord, permissions: List[AuthPermission], /) -> FormattedItem ```` because it would be more verbose for common cases and could lead to subtle bugs; more details in [3]. The Callable type is also usable as an expression, like in type aliases `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls. **Question 1**: Are there concerns we should keep in mind about such a syntax proposal? 2. Syntax for callback types beyond Callable `Callable` can't express the type of all possible callbacks. For example, it doesn't support callbacks where some parameters have default values: `formatter(record)` (the user didn't pass in `permissions`). It *is* possible to express these advanced cases using Callback Protocols (PEP 544) [4] but it gets verbose. There are two schools of thought on typing-sig on adding more syntax on top of (1): (a) Some, including Guido, feel that it would be a shame to not have syntax for core Python features like default values, keyword arguments, etc. One way to represent default values or optionally name parameters would be: ``` # permissions is optional formatter: (PurchaseRecord, List[AuthPermission]=...) -> FormattedItem # permissions can be called using a keyword argument. formatter: (PurchaseRecord, permissions: List[AuthPermission]) -> FormattedItem ``` There are also alternative syntax proposals. (b) Others want to wait till we have more real-world experience with the syntax in (1). The above cases occur < 5% of the time in typed or untyped code [5]. And the syntax in (1) is forward-compatible with the additional proposals. So we could add them later if needed or leave them out, since we can always use callback protocols. **Question 2**: Do you have preferences either way? Do we propose (1) alone or (1) + (2)? Once we get some high-level feedback here, we will draft a PEP going into the details for various use cases. Best, Pradeep Kumar Srinivasan Steven Troxler Eric Traut PS: We've linked to more details below. Happy to provide more details as needed. [1]: typing-sig thread about the proposal: https://mail.python.org/archives/list/typing-sig@python.org/message/JZLYRAXJ... [2]: Stats about loosely-typed Callables: https://github.com/pradeep90/annotation_collector#typed-projects---callable-... [3]: Comparison and rejection of proposals: https://www.dropbox.com/s/sshgtr4p30cs0vc/Python%20Callable%20Syntax%20Propo... [4]: Callback protocols: https://www.python.org/dev/peps/pep-0544/#callback-protocols [5]: Stats on callbacks not expressible with Callable: https://drive.google.com/file/d/1k_TqrNKcbWihRZdhMGf6K_GcLmn9ny3m/view?usp=s...
On Fri, Oct 8, 2021 at 1:45 PM S Pradeep Kumar <gohanpra@gmail.com> wrote:
The Callable type is also usable as an expression, like in type aliases `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls.
**Question 1**: Are there concerns we should keep in mind about such a syntax proposal?
Either I'm reading this wrongly, or this is oddly inconsistent with tuples. With a tuple, it's the comma that defines it; with function parameters here, it looks like the parentheses are mandatory, and they are what define it? Which of these will be valid? x = int -> int y = int, int -> int # tuple containing (int, Callable) or function taking two args? z = (int,) -> int Given the examples you've shown, my intuition is that it should match def statement syntax - namely, the args are always surrounded by parentheses, a trailing comma is legal but meaningless, and it has nothing to do with tuple display. It would be very convenient for one-arg functions to be able to be defined the way x is, and I suspect people will try it, but it introduces annoying ambiguities. There are parallel proposals to support a lambda syntax like "x => x.spam" as equivalent to "lambda x: x.spam", so I'm curious what the rules would be for the two types of arrows, and whether they'd be consistent with each other. ChrisA
pt., 8 paź 2021 o 04:48 S Pradeep Kumar <gohanpra@gmail.com> napisał(a):
... Note that we considered and rejected using a full def-signature syntax like ```` (record: PurchaseRecord, permissions: List[AuthPermission], /) -> FormattedItem ```` because it would be more verbose for common cases and could lead to subtle bugs; more details in [3].
Did you considered full def-signature with optional argument names, so the common cases would look like ```` (:PurchaseRecord, :List[AuthPermission]) -> FormattedItem ```` Bare name signatures like '(record) -> FormattedItem' could be disallowed to prevent bugs.
How would this work for a function that returns a function? Would you just put it on the def line like this? def foo() -> (int) -> str: return some_function_from_int_to_str Or would it have to be: def foo() -> ((int) -> str): ... The former is a little confusing to read - at first glance, it looks like it might return an int; the latter is ugly. I'd prefer a (soft?) keyword, like `(func (int) -> str)`. That keyword could be `def` or `lambda`, but those look like they might preclude custom callables. And just using `callable` seems like little improvement... On 07/10/2021 17:41, S Pradeep Kumar wrote:
Hello all,
Typing-sig has been discussing user-friendly syntax for the type used to represent callables. [1] Since this affects the Python language syntax, we wanted to get some high-level feedback from you before putting up a detailed PEP.
TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) -> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are there concerns we need to keep in mind about such a syntax change? 2. Should we propose syntax for additional features in the same PEP or in a future PEP?
# Motivation
Guido has pointed out that `Callable` is one of the most frequently used type forms but has the least readable syntax. For example, say we have a callback `formatter` that accepts a record and a list of permissions:
```python def print_purchases( user, formatter, # <-- callback ): <...> output = formatter(record, permissions) print(output) ```
To give it a type, we currently have to write:
```python from typing import Callable
def print_purchases( user: User, formatter: Callable[[PurchaseRecord, List[AuthPermission]], FormattedItem] # Hard to read. ) -> None:
<...> output = formatter(record, permissions) print(output) ```
`Callable` can be hard to understand for new users since it doesn't resemble existing function signatures and there can be multiple square brackets. It also requires an import from `typing`, which is inconvenient. Around 40% of the time [2], users just give up on precisely typing the parameters and return type of their callbacks and just leave it as a blank Callable. In other cases, they don't add a type for their callback at all. Both mean that they lose the safety guarantees from typing and leave room for bugs.
We believe that adding friendly syntax for Callable types will improve readability and encourage users to type their callbacks more precisely. Other modern, gradually-typed languages like TypeScript (JS), Hack (PHP), etc. have special syntax for Callable types.
(As a precedent, PEP 604 recently added clean, user-friendly syntax for the widely-used `Union` type. Instead of importing `Union` and writing `expr: Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr | SubtractExpr`.)
# Proposal and Questions
We have a two-part proposal and questions for you:
1. Syntax to replace Callable
After a lot of discussion, there is strong consensus in typing-sig about adding syntax to replace Callable. So, the above example would be written as: ```python def print_purchases( user: User, formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem, ) -> None:
<...> output = formatter(record, permissions) print(output) ``` This feels more natural and succinct.
Async callbacks currently need to be written as ``` from typing import Callable
async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]] ```
With the new syntax, they would be written as ``` async_callback: async (HttpRequest) -> HttpResponse ``` which again seems more natural. There is similar syntax for the type of decorators that pass on *args and **kwargs to the decorated function.
Note that we considered and rejected using a full def-signature syntax like ```` (record: PurchaseRecord, permissions: List[AuthPermission], /) -> FormattedItem ```` because it would be more verbose for common cases and could lead to subtle bugs; more details in [3].
The Callable type is also usable as an expression, like in type aliases `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls.
**Question 1**: Are there concerns we should keep in mind about such a syntax proposal?
2. Syntax for callback types beyond Callable
`Callable` can't express the type of all possible callbacks. For example, it doesn't support callbacks where some parameters have default values: `formatter(record)` (the user didn't pass in `permissions`). It *is* possible to express these advanced cases using Callback Protocols (PEP 544) [4] but it gets verbose.
There are two schools of thought on typing-sig on adding more syntax on top of (1):
(a) Some, including Guido, feel that it would be a shame to not have syntax for core Python features like default values, keyword arguments, etc.
One way to represent default values or optionally name parameters would be:
``` # permissions is optional formatter: (PurchaseRecord, List[AuthPermission]=...) -> FormattedItem # permissions can be called using a keyword argument. formatter: (PurchaseRecord, permissions: List[AuthPermission]) -> FormattedItem ```
There are also alternative syntax proposals.
(b) Others want to wait till we have more real-world experience with the syntax in (1).
The above cases occur < 5% of the time in typed or untyped code [5]. And the syntax in (1) is forward-compatible with the additional proposals. So we could add them later if needed or leave them out, since we can always use callback protocols.
**Question 2**: Do you have preferences either way? Do we propose (1) alone or (1) + (2)?
Once we get some high-level feedback here, we will draft a PEP going into the details for various use cases.
Best, Pradeep Kumar Srinivasan Steven Troxler Eric Traut
PS: We've linked to more details below. Happy to provide more details as needed.
[1]: typing-sig thread about the proposal: https://mail.python.org/archives/list/typing-sig@python.org/message/JZLYRAXJ... [2]: Stats about loosely-typed Callables: https://github.com/pradeep90/annotation_collector#typed-projects---callable-... [3]: Comparison and rejection of proposals: https://www.dropbox.com/s/sshgtr4p30cs0vc/Python%20Callable%20Syntax%20Propo... [4]: Callback protocols: https://www.python.org/dev/peps/pep-0544/#callback-protocols [5]: Stats on callbacks not expressible with Callable: https://drive.google.com/file/d/1k_TqrNKcbWihRZdhMGf6K_GcLmn9ny3m/view?usp=s...
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VBHJOS3L... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, 8 Oct 2021 at 03:45, S Pradeep Kumar <gohanpra@gmail.com> wrote:
Typing-sig has been discussing user-friendly syntax for the type used to represent callables. [1] Since this affects the Python language syntax, we wanted to get some high-level feedback from you before putting up a detailed PEP.
Disclaimer: I personally find that use of complex type annotations like Callable make code unreadable for the *human* reader, as compared to out of line information like a clearly-written comment or documentation. Therefore I'd probably be extremely reluctant to use this new syntax in real-life projects, regardless of the form that it takes. However, I do have some experience working on projects that use type annotations like this for callbacks, so I will answer in spite of my personal reservations.
**Question 1**: Are there concerns we should keep in mind about such a syntax proposal?
In my experience, a common pattern for functions that take non-trivial callbacks is to call them with named arguments rather than positional, in order to preserve future compatibility. So I would like callback annotations to be able to express the type "a function returning X, taking arguments `arg1` of type Y and `arg2` of type Z". The lack of keyword argument support in this proposal makes it unusable for APIs like this. Conversely, for callbacks with single arguments like f(cb: int->int) -> int, in my experience a lot of uses are lambda expressions, so something like f(lambda x: x+1). I would be a strong -1 on having to add types to lambda expressions (they are ugly enough already) so how does this interact with the annotation? Are type checkers able to correctly infer the type of lambda x: x+1, or would the whole expression end up being untyped? Similarly with functools.partial - would that correctly match the type? If common uses of callbacks end up being untyped in practice, that drastically reduces the value of the new syntax. (Note that this point is equally applicable to Callable - so I guess it's not so much about the new syntax as it is about whether annotating callbacks is sufficiently useful to be *worth* improving). Also, note that I automatically used a type of int->int up there. As someone else asked, is that form allowed, or is it required to be (int)->int? In my view, if we require the latter, I expect there will be a *lot* of people making that mistake and having to correct it.
**Question 2**: Do you have preferences either way? Do we propose (1) alone or (1) + (2)?
See above. My experience is that using named arguments for callbacks is a useful approach, and should probably be encouraged rather than made difficult, so I'd say that named argument support is important. Having said that, I think that making simple types like int->int or (int, int)->int more awkward to write would be a disadvantage - they are used a *lot* in things like the `key` argument of `sort`. So I'm -1 on things like (for example) requiring a colon as in (:int, :int) -> int. But to reiterate, I'm probably not likely to ever be a significant consumer of this feature, so consider the above with that in mind ;-) Paul
El vie, 8 oct 2021 a las 0:54, Paul Moore (<p.f.moore@gmail.com>) escribió:
On Fri, 8 Oct 2021 at 03:45, S Pradeep Kumar <gohanpra@gmail.com> wrote:
Typing-sig has been discussing user-friendly syntax for the type used to represent callables. [1] Since this affects the Python language syntax, we wanted to get some high-level feedback from you before putting up a detailed PEP.
Disclaimer: I personally find that use of complex type annotations like Callable make code unreadable for the *human* reader, as compared to out of line information like a clearly-written comment or documentation. Therefore I'd probably be extremely reluctant to use this new syntax in real-life projects, regardless of the form that it takes. However, I do have some experience working on projects that use type annotations like this for callbacks, so I will answer in spite of my personal reservations.
The point of this change is to make Callable annotations more readable: (int) -> int is a lot less dense than Callable[[int], int]. We hope that the new syntax will make it easier to use more complex annotations. Of course, if you think the annotations are still too complex, that's your choice.
**Question 1**: Are there concerns we should keep in mind about such a syntax proposal?
In my experience, a common pattern for functions that take non-trivial callbacks is to call them with named arguments rather than positional, in order to preserve future compatibility. So I would like callback annotations to be able to express the type "a function returning X, taking arguments `arg1` of type Y and `arg2` of type Z". The lack of keyword argument support in this proposal makes it unusable for APIs like this.
That's one of the differences between the proposals discussed here. The basic proposal Pradeep pointed out would not support named arguments, the more complete syntax favored by Guido (and also by me; 2(a) in Pradeep's email) would support it. Pradeep has done some empirical analysis that shows that in practice it is not very common to use such types, though.
Conversely, for callbacks with single arguments like f(cb: int->int) -> int, in my experience a lot of uses are lambda expressions, so something like f(lambda x: x+1). I would be a strong -1 on having to add types to lambda expressions (they are ugly enough already) so how does this interact with the annotation? Are type checkers able to correctly infer the type of lambda x: x+1, or would the whole expression end up being untyped? Similarly with functools.partial - would that correctly match the type? If common uses of callbacks end up being untyped in practice, that drastically reduces the value of the new syntax. (Note that this point is equally applicable to Callable - so I guess it's not so much about the new syntax as it is about whether annotating callbacks is sufficiently useful to be *worth* improving).
Type checkers should be able to infer this type. You would not need to write any explicit type when you call f(). However, the definition of `f` would look something like `def f(callback: (int) -> int) -> None: ...`. Currently, you'd write that as `def f(callback: Callable[[int], int]) -> None: ...`.
Also, note that I automatically used a type of int->int up there. As someone else asked, is that form allowed, or is it required to be (int)->int? In my view, if we require the latter, I expect there will be a *lot* of people making that mistake and having to correct it.
That's not something we discussed before. I'd be OK with allowing this unless it makes the grammar too ambiguous.
**Question 2**: Do you have preferences either way? Do we propose (1) alone or (1) + (2)?
See above. My experience is that using named arguments for callbacks is a useful approach, and should probably be encouraged rather than made difficult, so I'd say that named argument support is important. Having said that, I think that making simple types like int->int or (int, int)->int more awkward to write would be a disadvantage - they are used a *lot* in things like the `key` argument of `sort`. So I'm -1 on things like (for example) requiring a colon as in (:int, :int) -> int.
But to reiterate, I'm probably not likely to ever be a significant consumer of this feature, so consider the above with that in mind ;-)
Paul _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ANGLWMP4... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Oct 8, 2021 at 8:31 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
El vie, 8 oct 2021 a las 0:54, Paul Moore (<p.f.moore@gmail.com>) escribió:
Also, note that I automatically used a type of int->int up there. As someone else asked, is that form allowed, or is it required to be (int)->int? In my view, if we require the latter, I expect there will be a *lot* of people making that mistake and having to correct it.
That's not something we discussed before. I'd be OK with allowing this unless it makes the grammar too ambiguous.
Even if it didn't make the grammar ambiguous (and I think it doesn't, as long as the argument type isn't a union or another callable type), I'd be against this. An argument list is not a tuple, and we don't allow omitting the parentheses in other call-related situations: you can't write "def f x: return x+1" and you can't write "f 42". Allowing the omission of the parentheses here would be inconsistent (even if some other languages allow it). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Thanks for the responses! 1. Chris Angelico:
it looks like the parentheses are mandatory, and they are what define it?
Yes, we always expect parens. Like you said, this keeps things consistent, resembles the function signature, and avoids ambiguities. It also won't look like Haskell function types `int -> str -> bool` (which can be confusing for non-functional programmers). This was pretty explicitly rejected when we polled people at the Typing Summit.
Which of these will be valid?
(int) -> int (int, int) -> int (int,) -> int should also be fine, but I don't have strong opinions here. e.g., what about (,) -> int? We'll make sure to address this edge case explicitly, thanks.
There are parallel proposals to support a lambda syntax like "x => x.spam" as equivalent to "lambda x: x.spam", so I'm curious what the rules would be for the two types of arrows, and whether they'd be consistent with each other.
Did you considered full def-signature with optional argument names, so
Yeah, this had come up in our discussions. Like in other languages (such as Hack), we believe that lambdas should have a different arrow type to disambiguate the two. So, lambdas could be (x, y) => x + y. 2. Piotr Duda the common cases would look like
```` (:PurchaseRecord, :List[AuthPermission]) -> FormattedItem ```` Bare name signatures like '(record) -> FormattedItem' could be disallowed
to prevent bugs. The additional-features proposal in (2) actually does have optional parameter names. There are two variants: (a) the "hybrid" proposal, which will let you write (int, str) -> bool, but also hybrids like (int, y: str) -> bool; (x: int, y: str) -> bool; (int, str=...) -> bool. (b) the "xor" proposal, which will let you use the shorthand syntax (int, str) -> bool for the most frequently-used cases and the full def-signature for all other advanced cases using familiar syntax: (x: int, y: str=...) -> bool. We haven't reached a strong consensus on the above two yet (flexibility vs familiarity). We decided to get some thoughts here about whether we need to propose additional-features at all (2) in our initial PEP. 3. Patrick Reader
How would this work for a function that returns a function? Would you just put it on the def line like this?
We will allow using the callable syntax in the return type without parens: def foo() -> (int) -> str: return int_to_str The reason for not making that a syntax error is that it would be pretty bad UX. Users would see the callable syntax being used without parens everywhere: f: (int) -> str, but when they try to use that as the return type, they would get a confusing syntax error. So, given how frequently functions return functions, we decided to not require parens. People can specify parens in their coding guidelines (and auto-formatters) if they find parens more readable. 4. Paul Moore +1 to what Jelle said.
That's one of the differences between the proposals discussed here. The basic proposal Pradeep pointed out would not support named arguments, the more complete syntax favored by Guido (and also by me; 2(a) in Pradeep's email) would support it.
Pradeep has done some empirical analysis that shows that in practice it is not very common to use such types, though.
Yeah, that's basically Question 2 - whether to just replace Callable in the initial PEP (option 1) or to specify a more complete syntax from the beginning (option 2). 5. Serhiy Storchaka
How could you replace Callable[..., int] and Callable[Concatenate[str, P], int] ?
To represent a Callable that accepts arbitrary arguments: (...) -> int To represent a Callable that accepts ParamSpec (for decorators, etc.), we're leaning towards the below: (**P) -> int To represent your Concatenate example: (str, **P) -> int We would no longer need the Concatenate operator. We can naturally just add types up front. -- S Pradeep Kumar On Fri, Oct 8, 2021 at 8:48 AM Guido van Rossum <guido@python.org> wrote:
On Fri, Oct 8, 2021 at 8:31 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
El vie, 8 oct 2021 a las 0:54, Paul Moore (<p.f.moore@gmail.com>) escribió:
Also, note that I automatically used a type of int->int up there. As someone else asked, is that form allowed, or is it required to be (int)->int? In my view, if we require the latter, I expect there will be a *lot* of people making that mistake and having to correct it.
That's not something we discussed before. I'd be OK with allowing this unless it makes the grammar too ambiguous.
Even if it didn't make the grammar ambiguous (and I think it doesn't, as long as the argument type isn't a union or another callable type), I'd be against this.
An argument list is not a tuple, and we don't allow omitting the parentheses in other call-related situations: you can't write "def f x: return x+1" and you can't write "f 42". Allowing the omission of the parentheses here would be inconsistent (even if some other languages allow it).
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
-- S Pradeep Kumar
Pradeep, Thanks for all the clarifications. I think it's time that you put together a formal grammar. That should also make clear that "(,) -> int" is disallowed while "(int,) -> int" is allowed. --Guido On Fri, Oct 8, 2021 at 8:50 AM S Pradeep Kumar <gohanpra@gmail.com> wrote:
Thanks for the responses!
1. Chris Angelico:
it looks like the parentheses are mandatory, and they are what define it?
Yes, we always expect parens. Like you said, this keeps things consistent, resembles the function signature, and avoids ambiguities. It also won't look like Haskell function types `int -> str -> bool` (which can be confusing for non-functional programmers).
This was pretty explicitly rejected when we polled people at the Typing Summit.
Which of these will be valid?
(int) -> int (int, int) -> int
(int,) -> int should also be fine, but I don't have strong opinions here. e.g., what about (,) -> int? We'll make sure to address this edge case explicitly, thanks.
There are parallel proposals to support a lambda syntax like "x => x.spam" as equivalent to "lambda x: x.spam", so I'm curious what the rules would be for the two types of arrows, and whether they'd be consistent with each other.
Yeah, this had come up in our discussions. Like in other languages (such as Hack), we believe that lambdas should have a different arrow type to disambiguate the two. So, lambdas could be (x, y) => x + y.
2. Piotr Duda
Did you considered full def-signature with optional argument names, so the common cases would look like
```` (:PurchaseRecord, :List[AuthPermission]) -> FormattedItem ```` Bare name signatures like '(record) -> FormattedItem' could be disallowed to prevent bugs.
The additional-features proposal in (2) actually does have optional parameter names. There are two variants:
(a) the "hybrid" proposal, which will let you write (int, str) -> bool, but also hybrids like (int, y: str) -> bool; (x: int, y: str) -> bool; (int, str=...) -> bool.
(b) the "xor" proposal, which will let you use the shorthand syntax (int, str) -> bool for the most frequently-used cases and the full def-signature for all other advanced cases using familiar syntax: (x: int, y: str=...) -> bool.
We haven't reached a strong consensus on the above two yet (flexibility vs familiarity). We decided to get some thoughts here about whether we need to propose additional-features at all (2) in our initial PEP.
3. Patrick Reader
How would this work for a function that returns a function? Would you just put it on the def line like this?
We will allow using the callable syntax in the return type without parens:
def foo() -> (int) -> str: return int_to_str
The reason for not making that a syntax error is that it would be pretty bad UX. Users would see the callable syntax being used without parens everywhere: f: (int) -> str, but when they try to use that as the return type, they would get a confusing syntax error. So, given how frequently functions return functions, we decided to not require parens.
People can specify parens in their coding guidelines (and auto-formatters) if they find parens more readable.
4. Paul Moore
+1 to what Jelle said.
That's one of the differences between the proposals discussed here. The basic proposal Pradeep pointed out would not support named arguments, the more complete syntax favored by Guido (and also by me; 2(a) in Pradeep's email) would support it.
Pradeep has done some empirical analysis that shows that in practice it is not very common to use such types, though.
Yeah, that's basically Question 2 - whether to just replace Callable in the initial PEP (option 1) or to specify a more complete syntax from the beginning (option 2).
5. Serhiy Storchaka
How could you replace Callable[..., int] and Callable[Concatenate[str, P], int] ?
To represent a Callable that accepts arbitrary arguments:
(...) -> int
To represent a Callable that accepts ParamSpec (for decorators, etc.), we're leaning towards the below:
(**P) -> int
To represent your Concatenate example:
(str, **P) -> int
We would no longer need the Concatenate operator. We can naturally just add types up front. -- S Pradeep Kumar
On Fri, Oct 8, 2021 at 8:48 AM Guido van Rossum <guido@python.org> wrote:
On Fri, Oct 8, 2021 at 8:31 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
El vie, 8 oct 2021 a las 0:54, Paul Moore (<p.f.moore@gmail.com>) escribió:
Also, note that I automatically used a type of int->int up there. As someone else asked, is that form allowed, or is it required to be (int)->int? In my view, if we require the latter, I expect there will be a *lot* of people making that mistake and having to correct it.
That's not something we discussed before. I'd be OK with allowing this unless it makes the grammar too ambiguous.
Even if it didn't make the grammar ambiguous (and I think it doesn't, as long as the argument type isn't a union or another callable type), I'd be against this.
An argument list is not a tuple, and we don't allow omitting the parentheses in other call-related situations: you can't write "def f x: return x+1" and you can't write "f 42". Allowing the omission of the parentheses here would be inconsistent (even if some other languages allow it).
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
-- S Pradeep Kumar
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Fri, 8 Oct 2021 at 16:29, Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
That's one of the differences between the proposals discussed here. The basic proposal Pradeep pointed out would not support named arguments, the more complete syntax favored by Guido (and also by me; 2(a) in Pradeep's email) would support it.
Pradeep has done some empirical analysis that shows that in practice it is not very common to use such types, though.
I believe resolvelib (used by pip) makes a point of always calling callbacks with named arguments, to ensure compatibility. I thought PEP 517 hooks also required callers to use keywords, but when I checked the PEP, apparently not. I think with hindsight that we should have, though. But yes, I can imagine that simpler callbacks (like key= in sort) would be more common, and would be fine with positional arguments. Paul
07.10.21 19:41, S Pradeep Kumar пише:
1. Syntax to replace Callable
After a lot of discussion, there is strong consensus in typing-sig about adding syntax to replace Callable. So, the above example would be written as: ```python def print_purchases( user: User, formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem, ) -> None:
<...> output = formatter(record, permissions) print(output) ```
How could you replace Callable[..., int] and Callable[Concatenate[str, P], int] ?
Thanks for thinking more deeply about how we can make type annotations for callables more user friendly. The intersection between the syntax for general Python code and type annotations has been a topic of discussion within the Steering Council (on and off) for quite some time. We intend to provide more clarity about our thinking, probably in the form of an informational PEP, around this topic, and my apologies because I haven’t had time to draft it yet. But since this came up, I want to at least say that we are unanimous in our belief that the type annotation language must also be standard Python syntax. In other words, we are not in favor of letting type annotation syntax deviate from Python syntax. As a effect of this, the PSC does not intend to issue a blanket delegation of type-focused PEPs. Of course, individual typing PEPs can be delegated just like any other PEP, but especially where syntax change proposals are made, I expect that the PSC will want keep final decision making. This means that any PEP which proposes syntactic changes to support typing features must also address the implications of those syntax changes for the general Python language. PEP 646 (Variadic Generics) is a good example of this. Early on we recognized that the proposed syntax change had implications for Python, and we asked the PEP authors to address those implications for Python, which they added: https://www.python.org/dev/peps/pep-0646/#grammar-changes This is the kind of analysis the PSC needs in order to weigh the cost and benefits of making such changes to Python. TL;RA - One Syntax to Rule Them All Cheers, -Barry
On Oct 7, 2021, at 09:41, S Pradeep Kumar <gohanpra@gmail.com> wrote:
Hello all,
Typing-sig has been discussing user-friendly syntax for the type used to represent callables. [1] Since this affects the Python language syntax, we wanted to get some high-level feedback from you before putting up a detailed PEP.
TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) -> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are there concerns we need to keep in mind about such a syntax change? 2. Should we propose syntax for additional features in the same PEP or in a future PEP?
# Motivation
Guido has pointed out that `Callable` is one of the most frequently used type forms but has the least readable syntax. For example, say we have a callback `formatter` that accepts a record and a list of permissions:
```python def print_purchases( user, formatter, # <-- callback ): <...> output = formatter(record, permissions) print(output) ```
To give it a type, we currently have to write:
```python from typing import Callable
def print_purchases( user: User, formatter: Callable[[PurchaseRecord, List[AuthPermission]], FormattedItem] # Hard to read. ) -> None:
<...> output = formatter(record, permissions) print(output) ```
`Callable` can be hard to understand for new users since it doesn't resemble existing function signatures and there can be multiple square brackets. It also requires an import from `typing`, which is inconvenient. Around 40% of the time [2], users just give up on precisely typing the parameters and return type of their callbacks and just leave it as a blank Callable. In other cases, they don't add a type for their callback at all. Both mean that they lose the safety guarantees from typing and leave room for bugs.
We believe that adding friendly syntax for Callable types will improve readability and encourage users to type their callbacks more precisely. Other modern, gradually-typed languages like TypeScript (JS), Hack (PHP), etc. have special syntax for Callable types.
(As a precedent, PEP 604 recently added clean, user-friendly syntax for the widely-used `Union` type. Instead of importing `Union` and writing `expr: Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr | SubtractExpr`.)
# Proposal and Questions
We have a two-part proposal and questions for you:
1. Syntax to replace Callable
After a lot of discussion, there is strong consensus in typing-sig about adding syntax to replace Callable. So, the above example would be written as: ```python def print_purchases( user: User, formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem, ) -> None:
<...> output = formatter(record, permissions) print(output) ``` This feels more natural and succinct.
Async callbacks currently need to be written as ``` from typing import Callable
async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]] ```
With the new syntax, they would be written as ``` async_callback: async (HttpRequest) -> HttpResponse ``` which again seems more natural. There is similar syntax for the type of decorators that pass on *args and **kwargs to the decorated function.
Note that we considered and rejected using a full def-signature syntax like ```` (record: PurchaseRecord, permissions: List[AuthPermission], /) -> FormattedItem ```` because it would be more verbose for common cases and could lead to subtle bugs; more details in [3].
The Callable type is also usable as an expression, like in type aliases `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls.
**Question 1**: Are there concerns we should keep in mind about such a syntax proposal?
2. Syntax for callback types beyond Callable
`Callable` can't express the type of all possible callbacks. For example, it doesn't support callbacks where some parameters have default values: `formatter(record)` (the user didn't pass in `permissions`). It *is* possible to express these advanced cases using Callback Protocols (PEP 544) [4] but it gets verbose.
There are two schools of thought on typing-sig on adding more syntax on top of (1):
(a) Some, including Guido, feel that it would be a shame to not have syntax for core Python features like default values, keyword arguments, etc.
One way to represent default values or optionally name parameters would be:
``` # permissions is optional formatter: (PurchaseRecord, List[AuthPermission]=...) -> FormattedItem
# permissions can be called using a keyword argument. formatter: (PurchaseRecord, permissions: List[AuthPermission]) -> FormattedItem ```
There are also alternative syntax proposals.
(b) Others want to wait till we have more real-world experience with the syntax in (1).
The above cases occur < 5% of the time in typed or untyped code [5]. And the syntax in (1) is forward-compatible with the additional proposals. So we could add them later if needed or leave them out, since we can always use callback protocols.
**Question 2**: Do you have preferences either way? Do we propose (1) alone or (1) + (2)?
Once we get some high-level feedback here, we will draft a PEP going into the details for various use cases.
Best, Pradeep Kumar Srinivasan Steven Troxler Eric Traut
PS: We've linked to more details below. Happy to provide more details as needed.
[1]: typing-sig thread about the proposal: https://mail.python.org/archives/list/typing-sig@python.org/message/JZLYRAXJ... [2]: Stats about loosely-typed Callables: https://github.com/pradeep90/annotation_collector#typed-projects---callable-... [3]: Comparison and rejection of proposals: https://www.dropbox.com/s/sshgtr4p30cs0vc/Python%20Callable%20Syntax%20Propo... [4]: Callback protocols: https://www.python.org/dev/peps/pep-0544/#callback-protocols [5]: Stats on callbacks not expressible with Callable: https://drive.google.com/file/d/1k_TqrNKcbWihRZdhMGf6K_GcLmn9ny3m/view?usp=s... _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VBHJOS3L... Code of Conduct: http://python.org/psf/codeofconduct/
Thanks for summarizing the PSC position, Barry! I agree that type annotation syntax should be consistent with the rest of the language, but I'm curious why it has to be "standard Python syntax"? Could you elaborate a bit more on the reasoning behind that constraint? Cheers, Sergei On Fri, Oct 8, 2021 at 8:38 PM Barry Warsaw <barry@python.org> wrote: > Thanks for thinking more deeply about how we can make type annotations for > callables more user friendly. > > The intersection between the syntax for general Python code and type > annotations has been a topic of discussion within the Steering Council (on > and off) for quite some time. We intend to provide more clarity about our > thinking, probably in the form of an informational PEP, around this topic, > and my apologies because I haven’t had time to draft it yet. > > But since this came up, I want to at least say that we are unanimous in > our belief that the type annotation language must also be standard Python > syntax. In other words, we are not in favor of letting type annotation > syntax deviate from Python syntax. As a effect of this, the PSC does not > intend to issue a blanket delegation of type-focused PEPs. Of course, > individual typing PEPs can be delegated just like any other PEP, but > especially where syntax change proposals are made, I expect that the PSC > will want keep final decision making. > > This means that any PEP which proposes syntactic changes to support typing > features must also address the implications of those syntax changes for the > general Python language. PEP 646 (Variadic Generics) is a good example of > this. Early on we recognized that the proposed syntax change had > implications for Python, and we asked the PEP authors to address those > implications for Python, which they added: > > https://www.python.org/dev/peps/pep-0646/#grammar-changes > > This is the kind of analysis the PSC needs in order to weigh the cost and > benefits of making such changes to Python. > > TL;RA - One Syntax to Rule Them All > > Cheers, > -Barry > > > On Oct 7, 2021, at 09:41, S Pradeep Kumar <gohanpra@gmail.com> wrote: > > > > Hello all, > > > > Typing-sig has been discussing user-friendly syntax for the type used to > represent callables. [1] Since this affects the Python language syntax, we > wanted to get some high-level feedback from you before putting up a > detailed PEP. > > > > TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) > -> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are > there concerns we need to keep in mind about such a syntax change? 2. > Should we propose syntax for additional features in the same PEP or in a > future PEP? > > > > > > # Motivation > > > > Guido has pointed out that `Callable` is one of the most frequently used > type forms but has the least readable syntax. For example, say we have a > callback `formatter` that accepts a record and a list of permissions: > > > > ```python > > def print_purchases( > > user, > > formatter, # <-- callback > > ): > > <...> > > output = formatter(record, permissions) > > print(output) > > ``` > > > > To give it a type, we currently have to write: > > > > ```python > > from typing import Callable > > > > def print_purchases( > > user: User, > > formatter: Callable[[PurchaseRecord, List[AuthPermission]], > FormattedItem] # Hard to read. > > ) -> None: > > > > <...> > > output = formatter(record, permissions) > > print(output) > > ``` > > > > `Callable` can be hard to understand for new users since it doesn't > resemble existing function signatures and there can be multiple square > brackets. It also requires an import from `typing`, which is inconvenient. > Around 40% of the time [2], users just give up on precisely typing the > parameters and return type of their callbacks and just leave it as a blank > Callable. In other cases, they don't add a type for their callback at all. > Both mean that they lose the safety guarantees from typing and leave room > for bugs. > > > > We believe that adding friendly syntax for Callable types will improve > readability and encourage users to type their callbacks more precisely. > Other modern, gradually-typed languages like TypeScript (JS), Hack (PHP), > etc. have special syntax for Callable types. > > > > (As a precedent, PEP 604 recently added clean, user-friendly syntax for > the widely-used `Union` type. Instead of importing `Union` and writing > `expr: Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr | > SubtractExpr`.) > > > > > > # Proposal and Questions > > > > We have a two-part proposal and questions for you: > > > > 1. Syntax to replace Callable > > > > After a lot of discussion, there is strong consensus in typing-sig about > adding syntax to replace Callable. So, the above example would be written > as: > > ```python > > def print_purchases( > > user: User, > > formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem, > > ) -> None: > > > > <...> > > output = formatter(record, permissions) > > print(output) > > ``` > > This feels more natural and succinct. > > > > Async callbacks currently need to be written as > > ``` > > from typing import Callable > > > > async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]] > > ``` > > > > With the new syntax, they would be written as > > ``` > > async_callback: async (HttpRequest) -> HttpResponse > > ``` > > which again seems more natural. There is similar syntax for the type of > decorators that pass on *args and **kwargs to the decorated function. > > > > Note that we considered and rejected using a full def-signature syntax > like > > ```` > > (record: PurchaseRecord, permissions: List[AuthPermission], /) -> > FormattedItem > > ```` > > because it would be more verbose for common cases and could lead to > subtle bugs; more details in [3]. > > > > The Callable type is also usable as an expression, like in type aliases > `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls. > > > > **Question 1**: Are there concerns we should keep in mind about such a > syntax proposal? > > > > > > > > 2. Syntax for callback types beyond Callable > > > > `Callable` can't express the type of all possible callbacks. For > example, it doesn't support callbacks where some parameters have default > values: `formatter(record)` (the user didn't pass in `permissions`). It > *is* possible to express these advanced cases using Callback Protocols (PEP > 544) [4] but it gets verbose. > > > > There are two schools of thought on typing-sig on adding more syntax on > top of (1): > > > > (a) Some, including Guido, feel that it would be a shame to not have > syntax for core Python features like default values, keyword arguments, etc. > > > > One way to represent default values or optionally name parameters > would be: > > > > ``` > > # permissions is optional > > formatter: (PurchaseRecord, List[AuthPermission]=...) -> > FormattedItem > > > > # permissions can be called using a keyword argument. > > formatter: (PurchaseRecord, permissions: List[AuthPermission]) -> > FormattedItem > > ``` > > > > There are also alternative syntax proposals. > > > > (b) Others want to wait till we have more real-world experience with the > syntax in (1). > > > > The above cases occur < 5% of the time in typed or untyped code [5]. > And the syntax in (1) is forward-compatible with the additional proposals. > So we could add them later if needed or leave them out, since we can always > use callback protocols. > > > > **Question 2**: Do you have preferences either way? Do we propose (1) > alone or (1) + (2)? > > > > > > Once we get some high-level feedback here, we will draft a PEP going > into the details for various use cases. > > > > Best, > > Pradeep Kumar Srinivasan > > Steven Troxler > > Eric Traut > > > > PS: We've linked to more details below. Happy to provide more details as > needed. > > > > [1]: typing-sig thread about the proposal: > https://mail.python.org/archives/list/typing-sig@python.org/message/JZLYRAXJV34WAV5TKEOMA32V7ZLPOBFC/ > > [2]: Stats about loosely-typed Callables: > https://github.com/pradeep90/annotation_collector#typed-projects---callable-type > > [3]: Comparison and rejection of proposals: > https://www.dropbox.com/s/sshgtr4p30cs0vc/Python%20Callable%20Syntax%20Proposals.pdf?dl=0 > > [4]: Callback protocols: > https://www.python.org/dev/peps/pep-0544/#callback-protocols > > [5]: Stats on callbacks not expressible with Callable: > https://drive.google.com/file/d/1k_TqrNKcbWihRZdhMGf6K_GcLmn9ny3m/view?usp=sharing > > _______________________________________________ > > Python-Dev mailing list -- python-dev@python.org > > To unsubscribe send an email to python-dev-leave@python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/VBHJOS3LOXGVU6I4FABM6DKHH65GGCUB/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-leave@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/4TY3MVJQOLKSTMCJRTGAZEOSIIMAWQ45/ > Code of Conduct: http://python.org/psf/codeofconduct/ >
On Oct 8, 2021, at 13:02, Sergei Lebedev <sergei.a.lebedev@gmail.com> wrote:
I agree that type annotation syntax should be consistent with the rest of the language, but I'm curious why it has to be "standard Python syntax"? Could you elaborate a bit more on the reasoning behind that constraint?
Hi Sergei. I don’t mean anything more than just that there should be a single syntax for all of Python. I was just trying to describe “the bits of Python that aren’t type annotations”. Cheers, -Barry
On Fri, Oct 8, 2021 at 3:36 PM Barry Warsaw <barry@python.org> wrote:
On Oct 8, 2021, at 13:02, Sergei Lebedev <sergei.a.lebedev@gmail.com> wrote:
I agree that type annotation syntax should be consistent with the rest
of the language, but I'm curious why it has to be "standard Python syntax"? Could you elaborate a bit more on the reasoning behind that constraint?
Hi Sergei. I don’t mean anything more than just that there should be a single syntax for all of Python. I was just trying to describe “the bits of Python that aren’t type annotations”.
I interpret this as follows: - Whatever we choose (say, "(int, int) -> int") should be allowed where expressions are allowed, so that e.g. foo = (int, int) -> int assigns *something* at runtime -- it shouldn't be a syntax error and it shouldn't raise an exception. My preference would be for it to return something that's introspectable, i.e. an object lets one recover the original info. That's what we did when we introduced "A | B" as union syntax for types. (It's slightly different because that was already valid syntax, but the principle that it must return something introspectable is the same.) Union objects have an `__args__` attribute that gives the underlying types, and a `__parameters__` attribute giving any type variables. Note that the implementation of this introspectable type should ultimately be in C. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 7 Oct 2021, at 18:41, S Pradeep Kumar <gohanpra@gmail.com> wrote:
Note that we considered and rejected using a full def-signature syntax like ```` (record: PurchaseRecord, permissions: List[AuthPermission], /) -> FormattedItem ```` because it would be more verbose for common cases and could lead to subtle bugs; more details in [3].
Is this also why re-using an actual callable at a type was rejected? I always found the following more obvious: def data_to_table(d: Iterable[Mapping[str, float]], *, sort: bool = False, reversed: bool = False) -> Table: ... @dataclass class Stream: converter: data_to_table | None def add_converter(self, converter: data_to_table) -> None: self.converter = converter This solves the following problems with the `(P, Q) -> R` proposal: - how should this look like for "runtime" Python - how should we teach this - how can we express callables with complex signatures One disadvantage of this is that now arguments HAVE TO be named which raises questions: - should they be considered at type checking time? - how to express "I don't care"? To this I say: - yes, they should be considered at runtime (because kwargs have to be anyway) - ...unless they begin with an underscore This still leaves a minor problem that you can't have more than one argument literally named `_` so you'd have to do `_1`, `_2`, and so on. I don't think this is a big problem. In fact, forcing users to name callable arguments can be added as a fourth advantage to this design: making the annotations maximally informative to the human reader. The only remaining disadvantage that can't be addressed is that you can't create an *inline* callable type this way. I don't think this is a deal breaker as neither TypedDicts, Protocols, nor for this matter any PlainOldClasses can be defined inline inside a type annotation. - Ł
We drew up a draft syntax to clarify the three styles that have been discussed in typing-sig: - shorthand syntax, which supports exactly what Callable currently supports - XOR syntax, which would allow a def-like style in addition to shorthand - hybrid syntax, which extends shorthand to support named, default-value, and variadic args https://docs.google.com/document/d/1oCRBAAPs3efTYoIRSUwWLtOr1AUVqc8OCEYn4vKY... Within the discussion in typing-sig, - everyone would like to see support for at least shorthand - a little over half would like to have the more complex hybrid option - I don't think anyone has XOR as their first choice, but several of us think it might be better than hybrid
On 11/10/2021 16:45, Steven Troxler wrote:
https://docs.google.com/document/d/1oCRBAAPs3efTYoIRSUwWLtOr1AUVqc8OCEYn4vKY...
I think ParamSpecs should not have any special syntax. `**P` looks way too much like kwargs, and any other syntax would be totally new precedent. I'd suggest simply `(P) -> bool`, because P is already clearly a ParamSpec because it was declared as such. If `Concatenate[T, P]` is clear enough without special syntax decorating the `P`, so is `(T, P) ->`. (I posted this as a comment on the document, but I'm reposting it here as well for visibility)
It isn't clear to me that reusing the inspect.Signature object was even considered. If it was rejected as too complicated for most signatures, please put that in the PEP. My own opinion, admittedly not as a typing fan, is that if you need enough details to want more than Callable, then you probably do want the full power of inspect.Signature, though you may also want some convenience methods to indicate that certain portions of the signature are undefined, rather than following the example function. -jJ
Is this also why re-using an actual callable at a type was rejected?
The idea to use a function as a type (which Łukasz proposed at the PyCon typing summit) is still under consideration. Our thought was that it would probably be a separate PEP from a syntax change. Personally, I think we should both adopt function-as-a-type and shorthand syntax, and reject both the XOR and hybrid syntax because function-as-a-type is convenient for expressing the same things (named, default-value, and variadic args). We don't think function-as-a-type solves all cases for a few reasons: (1) in existing code, Pradeep's data analysis shows that the use of named arguments in calls is quite rare, less than 5% of cases; positional arguments are at least 12 times more common. As a result, we think it's important to have an option where positional arguments are convenient. We don't think the `/` or underscore conventions would be very convenient; `/` in particular would be easy to forget, which would lead to accidentally pinning argument names. It's easy to imagine library authors not realizing they did this, particularly their unit tests use the same argument names. (2) Callable is one of the most heavily used types. So we do think the ability to inline it may be important. (3) Around 20% of Callables in existing code Pradeep analyzed require a ParamSpec. That's 4 times the number of callables that made use of default arguments or named arguments. It is not easy to make use of ParamSpec in function-as-a-type, but it is easy to support with a new shorthand syntax.
On 11/10/2021 17:03, Steven Troxler wrote:
Personally, I think we should both adopt function-as-a-type and shorthand syntax, and reject both the XOR and hybrid syntax because function-as-a-type is convenient for expressing the same things (named, default-value, and variadic args).
+1 This combination of a shorthand syntax for simple callbacks and a longhand interface-style method for more complex signatures is the best option IMO. It provides good flexibility, in the pragmatic sense of "there should be only one way to do it": there is a dead-simple way which is pretty good, and a more flexible way for more complex cases which is pretty easy to switch to if necessary.
1. Barry Warsaw
This means that any PEP which proposes syntactic changes to support typing features must also address the implications of those syntax changes for the general Python language. PEP 646 (Variadic Generics) is a good example of this. Early on we recognized that the proposed syntax change had implications for Python, and we asked the PEP authors to address those implications for Python, which they added:
Yes, we definitely plan to include syntax implications in the PEP. (I'm co-author of PEP 646 :) ) This was one of the reasons we wanted to get high-level feedback here, since our proposal includes a syntax change for Python. 2. Łukasz As Steven mentioned, your proposal of using a function name as its type is definitely something we're keeping in mind. It comes under the "alternative proposals" I talked about in part 2. As a replacement for Callable itself (part 1), writing a separate function signature is much more verbose and inconvenient. Given how frequently we need to express simple positional-only callables (like `Callable[[Request], Response]`) and ParamSpec callables, the consensus was strongly in favor of having simple inline syntax. However, when it comes to complex features like named parameters, default values, and `**kwargs` (part 2), your proposal might well be more readable than "hybrid" syntax. These use cases rarely come up, based on our stats, so a simple, syntax-free change like yours might be better than trying to duplicate everything that can be expressed in function signatures. So, shorthand syntax + function-name-as-type could be a good overall solution for part 1 and part 2. This would give us simple and friendly syntax for the most common cases and readable syntax for the advanced cases. What do you think?
On Mon, Oct 11, 2021 at 1:52 AM Łukasz Langa <lukasz@langa.pl> wrote:
On 7 Oct 2021, at 18:41, S Pradeep Kumar <gohanpra@gmail.com> wrote:
Note that we considered and rejected using a full def-signature syntax like ```` (record: PurchaseRecord, permissions: List[AuthPermission], /) -> FormattedItem ```` because it would be more verbose for common cases and could lead to subtle bugs; more details in [3].
Is this also why re-using an actual callable at a type was rejected?
I always found the following more obvious:
def data_to_table(d: Iterable[Mapping[str, float]], *, sort: bool = False, reversed: bool = False) -> Table: ...
@dataclass class Stream: converter: data_to_table | None
def add_converter(self, converter: data_to_table) -> None: self.converter = converter
This solves the following problems with the `(P, Q) -> R` proposal: - how should this look like for "runtime" Python - how should we teach this - how can we express callables with complex signatures
One disadvantage of this is that now arguments HAVE TO be named which raises questions: - should they be considered at type checking time? - how to express "I don't care"?
To this I say: - yes, they should be considered at runtime (because kwargs have to be anyway) - ...unless they begin with an underscore
This still leaves a minor problem that you can't have more than one argument literally named `_` so you'd have to do `_1`, `_2`, and so on. I don't think this is a big problem.
In fact, forcing users to name callable arguments can be added as a fourth advantage to this design: making the annotations maximally informative to the human reader.
The only remaining disadvantage that can't be addressed is that you can't create an *inline* callable type this way. I don't think this is a deal breaker as neither TypedDicts, Protocols, nor for this matter any PlainOldClasses can be defined inline inside a type annotation.
Not everyone on typing-sig has the same preferences. For me personally, your proposal is at best a better syntax for callback protocols, not for inline callback types, precisely because it uses statement-level syntax and requires you to think of a name for the type. It also would cause some confusion because people aren't used for functions to be used as type aliases. If I see A = dict[str, list[int]] it's easy enough to guess that A is a type alias. But if I see def Comparison(a: T, b: T) -> Literal[-1, 0, 1]: ... my first thought is that it's a comparison function that someone hasn't finished writing yet, not a function type -- since if it did have at least one line of code in the body, it *would* be that. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
12.10.21 00:03, Guido van Rossum пише:
But if I see
def Comparison(a: T, b: T) -> Literal[-1, 0, 1]: ...
my first thought is that it's a comparison function that someone hasn't finished writing yet, not a function type -- since if it did have at least one line of code in the body, it *would* be that.
I agree. There are also problems with variables substitution (indexing), union (operator "|") and introspection. We would need to add __getitem__, __or__, __args__, __parameters__ etc to all functions. There may be also conflicts with NewType which returns a callable. But can it be expressed via Protocol? class Comparison(Protocol, Generic[T]): def __call__(self, /, a: T, b: T) -> Literal[-1, 0, 1]: ... Then we can just add a decorator which turns a function into class: @functype def Comparison(a: T, b: T) -> Literal[-1, 0, 1]: ...
If I can make a wild suggestion: why not create a little language for type specifications? If you look at other programming languages you’ll see that the “type definition sub-language” is often completely different from the “execution sub-language”, with only some symbols in common and used in vaguely related ways. `bool (*myfuncptr)(int, float*)` uses a completely different set of syntactic rules than `rv = (*myfunptr)(*myintptr, &myfloat)`. So with some grains of salt you could say that C is comprised of a declarative typing sublanguage and an imperative execution sublanguage. Python typing uses basically a subset of the execution expression syntax as its declarative typing language. What if we created a little language that is clearly flagged, for example as t”….” or t’….’? Then we could simply define the typestring language to be readable, so you could indeed say t”(int, str) -> bool”. And we could even allow escapes (similar to f-strings) so that the previous expression could also be specified, if you really wanted to, as t”{typing.Callable[[int, str], bool}”. Jack
On 7 Oct 2021, at 18:41, S Pradeep Kumar <gohanpra@gmail.com> wrote:
Hello all,
Typing-sig has been discussing user-friendly syntax for the type used to represent callables. [1] Since this affects the Python language syntax, we wanted to get some high-level feedback from you before putting up a detailed PEP.
TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) -> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are there concerns we need to keep in mind about such a syntax change? 2. Should we propose syntax for additional features in the same PEP or in a future PEP?
# Motivation
Guido has pointed out that `Callable` is one of the most frequently used type forms but has the least readable syntax. For example, say we have a callback `formatter` that accepts a record and a list of permissions:
```python def print_purchases( user, formatter, # <-- callback ): <...> output = formatter(record, permissions) print(output) ```
To give it a type, we currently have to write:
```python from typing import Callable
def print_purchases( user: User, formatter: Callable[[PurchaseRecord, List[AuthPermission]], FormattedItem] # Hard to read. ) -> None:
<...> output = formatter(record, permissions) print(output) ```
`Callable` can be hard to understand for new users since it doesn't resemble existing function signatures and there can be multiple square brackets. It also requires an import from `typing`, which is inconvenient. Around 40% of the time [2], users just give up on precisely typing the parameters and return type of their callbacks and just leave it as a blank Callable. In other cases, they don't add a type for their callback at all. Both mean that they lose the safety guarantees from typing and leave room for bugs.
We believe that adding friendly syntax for Callable types will improve readability and encourage users to type their callbacks more precisely. Other modern, gradually-typed languages like TypeScript (JS), Hack (PHP), etc. have special syntax for Callable types.
(As a precedent, PEP 604 recently added clean, user-friendly syntax for the widely-used `Union` type. Instead of importing `Union` and writing `expr: Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr | SubtractExpr`.)
# Proposal and Questions
We have a two-part proposal and questions for you:
1. Syntax to replace Callable
After a lot of discussion, there is strong consensus in typing-sig about adding syntax to replace Callable. So, the above example would be written as: ```python def print_purchases( user: User, formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem, ) -> None:
<...> output = formatter(record, permissions) print(output) ``` This feels more natural and succinct.
Async callbacks currently need to be written as ``` from typing import Callable
async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]] ```
With the new syntax, they would be written as ``` async_callback: async (HttpRequest) -> HttpResponse ``` which again seems more natural. There is similar syntax for the type of decorators that pass on *args and **kwargs to the decorated function.
Note that we considered and rejected using a full def-signature syntax like ```` (record: PurchaseRecord, permissions: List[AuthPermission], /) -> FormattedItem ```` because it would be more verbose for common cases and could lead to subtle bugs; more details in [3].
The Callable type is also usable as an expression, like in type aliases `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls.
**Question 1**: Are there concerns we should keep in mind about such a syntax proposal?
2. Syntax for callback types beyond Callable
`Callable` can't express the type of all possible callbacks. For example, it doesn't support callbacks where some parameters have default values: `formatter(record)` (the user didn't pass in `permissions`). It *is* possible to express these advanced cases using Callback Protocols (PEP 544) [4] but it gets verbose.
There are two schools of thought on typing-sig on adding more syntax on top of (1):
(a) Some, including Guido, feel that it would be a shame to not have syntax for core Python features like default values, keyword arguments, etc.
One way to represent default values or optionally name parameters would be:
``` # permissions is optional formatter: (PurchaseRecord, List[AuthPermission]=...) -> FormattedItem
# permissions can be called using a keyword argument. formatter: (PurchaseRecord, permissions: List[AuthPermission]) -> FormattedItem ```
There are also alternative syntax proposals.
(b) Others want to wait till we have more real-world experience with the syntax in (1).
The above cases occur < 5% of the time in typed or untyped code [5]. And the syntax in (1) is forward-compatible with the additional proposals. So we could add them later if needed or leave them out, since we can always use callback protocols.
**Question 2**: Do you have preferences either way? Do we propose (1) alone or (1) + (2)?
Once we get some high-level feedback here, we will draft a PEP going into the details for various use cases.
Best, Pradeep Kumar Srinivasan Steven Troxler Eric Traut
PS: We've linked to more details below. Happy to provide more details as needed.
[1]: typing-sig thread about the proposal: https://mail.python.org/archives/list/typing-sig@python.org/message/JZLYRAXJ... <https://mail.python.org/archives/list/typing-sig@python.org/message/JZLYRAXJV34WAV5TKEOMA32V7ZLPOBFC/> [2]: Stats about loosely-typed Callables: https://github.com/pradeep90/annotation_collector#typed-projects---callable-... <https://github.com/pradeep90/annotation_collector#typed-projects---callable-type> [3]: Comparison and rejection of proposals: https://www.dropbox.com/s/sshgtr4p30cs0vc/Python%20Callable%20Syntax%20Propo... <https://www.dropbox.com/s/sshgtr4p30cs0vc/Python%20Callable%20Syntax%20Proposals.pdf?dl=0> [4]: Callback protocols: https://www.python.org/dev/peps/pep-0544/#callback-protocols <https://www.python.org/dev/peps/pep-0544/#callback-protocols> [5]: Stats on callbacks not expressible with Callable: https://drive.google.com/file/d/1k_TqrNKcbWihRZdhMGf6K_GcLmn9ny3m/view?usp=s... <https://drive.google.com/file/d/1k_TqrNKcbWihRZdhMGf6K_GcLmn9ny3m/view?usp=sharing>_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VBHJOS3L... Code of Conduct: http://python.org/psf/codeofconduct/
Thanks for the responses, everyone. Overall, it seems like there were no strong objections to the proposal. I didn't hear much about Question 2, though: Should we propose features beyond present-day `Callable` in the same PEP or defer it to a future PEP? In case that question got lost in the other details, feel free to respond here. If not, I'll take it there aren't strong opinions either way. Some of my other takeaways: + Address the implications of the syntax changes for Python. + Address edge cases like trailing commas, `Concatenate` for `ParamSpec`, and runtime value of the expression. + Explicitly discuss the function-name-as-a-type proposal. We will be drafting the PEP over the coming month.
On Fri, Oct 15, 2021 at 12:22 PM Pradeep Kumar Srinivasan < gohanpra@gmail.com> wrote:
Thanks for the responses, everyone. Overall, it seems like there were no strong objections to the proposal.
I didn't hear much about Question 2, though: Should we propose features beyond present-day `Callable` in the same PEP or defer it to a future PEP?
In case that question got lost in the other details, feel free to respond here. If not, I'll take it there aren't strong opinions either way.
Sorry I didn't make it to the meeting. You know my opinion. :-)
Some of my other takeaways:
+ Address the implications of the syntax changes for Python.
In particular, when the syntax is used in a non-annotation position, it must evaluate to some object that represents the information present in the syntax (like Callable does ATM). + Address edge cases like trailing commas, `Concatenate` for `ParamSpec`,
and runtime value of the expression. + Explicitly discuss the function-name-as-a-type proposal.
We will be drafting the PEP over the coming month.
Awesome. You have my blessing. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
participants (16)
-
Barry Warsaw -
Chris Angelico -
gohanpra＠gmail.com -
Guido van Rossum -
jack.jansen＠cwi.nl -
Jelle Zijlstra -
Jim J. Jewett -
Patrick Reader -
Paul Moore -
Piotr Duda -
Pradeep Kumar Srinivasan
-
S Pradeep Kumar
-
Sergei Lebedev -
Serhiy Storchaka -
Steven Troxler -
Łukasz Langa