Re: RFC on Callable Syntax PEP
First of all, thank you for authoring a really well-written PEP. tl;dr: I find it very troubling that we are going on a path where need to increase the language complexity (syntax) only in the cause 'easier' typing. So I am opposed to this change. Most of the native typing use cases until now (PEP 585 / PEP 604) was using the existing syntax, with slight adjustments on runtime. And personally, I really liked them (being able to write `list[int | str]` is nice), but not as much as I like the simplicity of the language. I think the new syntax proposal for only a typing use case is really destructive when we already have a very formal way of creating callable types. I personally don't think it is verbose (as claimed by the PEP), since the only thing that we have an extra is the `Callable` name. Aside from these concerns, a few comments on the actual proposal: - The syntax itself would probably confuse people, considering in other languages (like javascript) the arrow functions are a thing that loosely translates to our lambdas. People might try to write code like `(x, y) -> x + y`, with the expectation that it would work (since it is a valid syntax now), but it would immediately throw an error (if x/y is not defined) and create confusion. - Callables that return callables look very awkward with a double arrow. `) -> (int, str) -> R:` (compared to `) -> Callable[[int, str], R]:`. This was one of the examples that really made me look twice to understand. - `async` keyword used to be coupled with `def`/`for`/`with`, but without any of these, it doesn't feel good to have it as some sort of prefix to these expressions. I think it would be nice if the PEP can address how much of the code out there uses `Callable[..., Awaitable[...]]` syntax, since from what I understand by the current proposal `async (int, str) -> bool` is the same with `(int, str) -> Awaitable[bool]`, which is, in theory, a new syntax that we might be adding for a very restricted use case. - The parameter part (`(x, y) -> `) seem to be even diverging from an actual expression (by allowing standalone [double-]starred operations, and disallowing `(int, ...) ->`), which I think would complicate the actual parsing process (even though we have an advanced parser in CPython, that does not mean community maintained projects such as jedi, parso, black will have one). - Out of 4 concerns that are listed on the PEP about the current callable syntax; only 1 is about the actual runtime behaviour, which even if I did not agree initially, we have other solutions. Like an open issue on the tracker about making `callable()` generic https://bugs.python.org/issue42102. - The other 3 is pretty much the same, claiming the syntax is dense. Which I partly agree on. We probably could find other solutions (using tuples to represent the parameter group, etc.), but I do not see those points as very big deals (at least not as big as adding a very complicated piece of new syntax). I would assume some of these were previously discussed during the typing-sig/python-dev threads, but as someone who recently learned about this proposal, I feel like this is really overworked for a problem that is (subjectively, again) small.
Batuhan expresses my concerns better than I could, so I just add my agreement. On 12/18/2021 3:13 PM, Batuhan Taskaya wrote:
tl;dr: I find it very troubling that we are going on a path where need to increase the language complexity (syntax) only in the cause 'easier' typing. So I am opposed to this change.
-- Terry Jan Reedy
I agree. The same concerns and reservations apply for me.
On Sat, 18 Dec 2021 at 21:13, Terry Reedy
Batuhan expresses my concerns better than I could, so I just add my agreement.
On 12/18/2021 3:13 PM, Batuhan Taskaya wrote:
tl;dr: I find it very troubling that we are going on a path where need to increase the language complexity (syntax) only in the cause 'easier' typing. So I am opposed to this change.
-- Terry Jan Reedy
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/G6PD5KYV... Code of Conduct: http://python.org/psf/codeofconduct/
I've been thinking about readability hard because I share many of your concerns about readability. I'm starting to think parenthesizing the outside might work much better in practice: ``` (int, str -> bool) ``` instead of ``` (int, str) -> bool ``` This looks a bit less familiar, but it - eliminates any situation - helps the eye group together functions (and lets us use an editor's go-to-delimiter if we get lost!) An example ========= The function ---------------- To get at what I mean, here's a nice simple function: ``` def zip(f, g): def zipped(x: int, y: str): return f(x), g(y) return zipped ``` typing.Callable -------------------- Here's it's type using typing.Callable: ``` typing.Callable[ [typing.Callable[[int], bool], typing.Callable[[str], float]], typing.Callable[[int, str], tuple[bool, float] ] ``` which seems ugly. It's actually not bad compared to a lot of production code, but this is the kind of thing that led to PEP 677. current PEP 677 proposal ---------------------------------- With the current PEP 677 proposal, writing the type on one line we get ``` ((int) -> float, (str) -> bool) -> (int, str) -> tuple[float, bool] ``` I prefer this to the Callable version. But it’s dense, and I do agree it illustrates some readability issues with PEP 677. Using (int, str -> bool) syntax -------------------------------- Here’s the type if we change the syntax to put the right parenthesis after the return type: ``` ((int -> float), (str -> bool) -> (int, str -> tuple[float, bool]) ``` To my eyes, most of the pain points are now eliminated. - there’s never a double-arrow due to callable in return position - even for argument types, to my eyes it’s now easier to read An added bonus is we no longer have to think about double-arrows when a callable type is in the return position of a function, e.g.: ``` def f() -> (int, str -> bool): ... ``` And it solves another major usability problem we found - writing optional callables - because it’s now no problem at all to write ``` (int, str -> bool) | None ``` as the type of an optional callable argument. New edge cases / readability issues? ------------------------------------- I’m pretty sure *all* existing edge cases and readability issues are basically solved by this syntax because the grouping becomes obvious. The only new edge case I can see is that we might not like `(-> bool)` for parameterless callables. I’d be okay with that, but I’d also be down with a hack like `(() -> bool)`. I don’t actually think it matters much - by making this change we would trade multiple edge cases in *complicated* examples for a single edge case in the *simplest possible* example where readability isn’t a big problem.
On Tue, 21 Dec 2021 at 18:35, Steven Troxler
I've been thinking about readability hard because I share many of your concerns about readability.
Before I comment on syntax, I'd like to question the example:
An example =========
The function ----------------
To get at what I mean, here's a nice simple function: ``` def zip(f, g): def zipped(x: int, y: str): return f(x), g(y) return zipped ```
So my first question is why a generic function like this would limit the parameters x and y to int and str. Surely this is an entirely workable function for *any* arguments with the appropriate "shape"? So how is this a good example of somewhere you'd use types like Callable[[int], bool] (by the way, where did that "bool" appear from?) in a signature? My immediate thought when seeing that zip function is that its type is "obvious"¹, (takes two functions, returns a function which takes 2 arguments matching the args of the 2 functions and returns a tuple matching the results of the 2 functions). That type is almost impossible to express clearly, because it needs pretty complex generics. But the more important point is that I'd never, **ever**, want to write that type out longhand. I'd expect a type system to either infer the type, and it would be anonymous¹, or to give up and treat it as untyped. One thing I absolutely would not want to do is over-constrain any of the arguments just to make it possible to write the type down. The real issue with this function, in my view, is not expressing callables, but rather generic type variables ("return a function whose first argument has the same type as the single argument of the function which is the first argument of this function..." !!!) Can you suggest a more "real world" function as an example, which focuses on the callable syntax and doesn't use things like arguments called f and g? Maybe a GUI callback with a function argument like on_click? ¹ By which I mean intuitive, not easy to express in words!!! :-) ² Some languages may allow syntax like `typeof(zip)` to refer to that anonymous type, but that's a separate point.
Here's it's type using typing.Callable: ``` typing.Callable[ [typing.Callable[[int], bool], typing.Callable[[str], float]], typing.Callable[[int, str], tuple[bool, float] ] ``` which seems ugly. It's actually not bad compared to a lot of production code, but this is the kind of thing that led to PEP 677.
It's massively over-constrained. I assume that's because you're trying to make a point about Callable[] rather than about generics, but can you give a realistic example that *doesn't* involve over-constraining higher order functions? On a side note, why not name at least some of those function types? And why not use "from typing import Callable"? It feels like you're not making enough effort to make your example readable, which undermines your point as a result.
((int) -> float, (str) -> bool) -> (int, str) -> tuple[float, bool]
To your credit, you've made this pretty unreadable, which gives some balance here :-) Seriously, making it a one-liner with all those -> arrows is a disaster. Changing the location of the parentheses doesn't alter that at all. Rewriting as a multi-line expression: ( (int) -> float, (str) -> bool ) -> (int, str) -> tuple[float, bool] helps quite a bit, but returning a function looks bad here. We're not writing Haskell, you know ;-) I'd prefer a "mixed" notation here: ( (int) -> float, (str) -> bool ) -> Callable[(int, str), tuple[float, bool]] I don't honestly think there's a readable "function returning a function" form here - the chained -> tokens is just awkward. Although that's clearly a matter of preference, there's never going to be an objective answer here.
Here’s the type if we change the syntax to put the right parenthesis after the return type: ``` ((int -> float), (str -> bool) -> (int, str -> tuple[float, bool]) ```
To my eyes, most of the pain points are now eliminated. - there’s never a double-arrow due to callable in return position - even for argument types, to my eyes it’s now easier to read
To my mind, the eye is still drawn to the arrows, and the readability is no better. And the parentheses give me a lisp vibe, for reasons I can't really pin down but which makes this version *less* readable.
An added bonus is we no longer have to think about double-arrows when a callable type is in the return position of a function, e.g.: ``` def f() -> (int, str -> bool): ... ```
Still looks like double arrows to me, I'm afraid. The parentheses don't group strongly enough to override the "chain of arrows" impression.
And it solves another major usability problem we found - writing optional callables - because it’s now no problem at all to write ``` (int, str -> bool) | None ``` as the type of an optional callable argument.
I guess, but it feels like punctuation soup to me, I'm afraid. Optional[Callable[[int, str], bool]] is more obvious to me (the ugliness in that version comes from the square brackets and the capitalisation, which are present for different reasons, not the use of words rather than symbols). Paul
In the example I was aiming for something easy to understand that produced a type illustrating potential problems of PEP 677, which is at its worst when there are callables in both argument and return position. I don't have a great real-world example of this worst-case, most of what I've seen involves simpler and the current PEP 677 proposal isn't as bad. As for formatting I agree that I wouldn't hand-write the type as ``` ((int) -> float, (str) -> bool) -> (int, str) -> tuple[float, bool] ``` but a lot of code formatters might fight me on this, so I think it's worthy of consideration, and I was trying to illustrate a readability problem and possible fix. Formatting the code nicely so that my proposal looks good would have meant not really engaging with the concern. If you're looking for examples from real code where Callable is unweildy, Pradeep collected a few at [1] from typeshed although most of them look just fine with the current PEP 677 proposal. A couple examples: Callable[[AnyStr, Callable[[AnyStr, AnyStr, AnyStr], AnyStr]], AnyStr] Callable[[Optional[str], tuple[_Marshallable, ...]], Union[Fault, tuple[_Marshallable, ...]]] Callable[[str, Callable[[Iterable[str], str], str]], None] versus the same types written using the current PEP 677 syntax: (AnyStr, (AnyStr, AnyStr, AnyStr) -> AnyStr) -> AnyStr (Optional[str], tuple[_Marshallable, ...]) -> Union[Fault, tuple[_Marshallable, ...]] (str, (Iterable[str], str) -> str) -> None versus with outer parentheses: (AnyStr, (AnyStr, AnyStr, AnyStr -> AnyStr) -> AnyStr) (Optional[str], tuple[_Marshallable, ...] -> Union[Fault, tuple[_Marshallable, ...]]) (str, (Iterable[str], str -> str) -> None) and another idea, requiring both outer parentheses and argument parentheses: ((AnyStr, ((AnyStr, AnyStr, AnyStr )-> AnyStr)) -> AnyStr) ((Optional[str], tuple[_Marshallable, ...]) -> Union[Fault, tuple[_Marshallable, ...]]) (str, (Iterable[str], str) -> str) -> None) To me, these are convincing examples of where Callable is hard to read and an arrow syntax is easier, but that doesn't necessarily mean it's worth the price of new syntax.
To me, these are mostly convincing examples that people need to name
parts of a complex type like this :-)
I don't actually find any of the syntaxes better than any other. They
are all a bit bad, but I view that as the fault of the complex nested
types, not the syntax (hence my preference for naming things). It's
hard to give examples of how I'd name things, because you need to know
the "business logic" to name things well, and I don't for these.
Frankly, if people were writing normal Python expressions like this,
everyone would be telling them that Python isn't about writing
one-liners and they should factor out named subexpressions. Why should
type annotations be any different?
Paul
PS If I wanted any improvement for callables, it would be to use
parentheses for the arguments rather than square brackets (which look
too "heavy" in my view). So Callable[(int, str), bool] rather than
Callable[[int, str], bool]. I understand why the outer brackets have
to be square (abuse of indexing notation) but as far as I can see
there's no reason the inner ones need to be (OK, for single arguments,
Callable[(int,), bool] is a bit clumsy...)
On Tue, 21 Dec 2021 at 22:10, Steven Troxler
In the example I was aiming for something easy to understand that produced a type illustrating potential problems of PEP 677, which is at its worst when there are callables in both argument and return position. I don't have a great real-world example of this worst-case, most of what I've seen involves simpler and the current PEP 677 proposal isn't as bad.
As for formatting I agree that I wouldn't hand-write the type as ``` ((int) -> float, (str) -> bool) -> (int, str) -> tuple[float, bool] ``` but a lot of code formatters might fight me on this, so I think it's worthy of consideration, and I was trying to illustrate a readability problem and possible fix. Formatting the code nicely so that my proposal looks good would have meant not really engaging with the concern.
If you're looking for examples from real code where Callable is unweildy, Pradeep collected a few at [1] from typeshed although most of them look just fine with the current PEP 677 proposal. A couple examples:
Callable[[AnyStr, Callable[[AnyStr, AnyStr, AnyStr], AnyStr]], AnyStr] Callable[[Optional[str], tuple[_Marshallable, ...]], Union[Fault, tuple[_Marshallable, ...]]] Callable[[str, Callable[[Iterable[str], str], str]], None]
versus the same types written using the current PEP 677 syntax:
(AnyStr, (AnyStr, AnyStr, AnyStr) -> AnyStr) -> AnyStr (Optional[str], tuple[_Marshallable, ...]) -> Union[Fault, tuple[_Marshallable, ...]] (str, (Iterable[str], str) -> str) -> None
versus with outer parentheses:
(AnyStr, (AnyStr, AnyStr, AnyStr -> AnyStr) -> AnyStr) (Optional[str], tuple[_Marshallable, ...] -> Union[Fault, tuple[_Marshallable, ...]]) (str, (Iterable[str], str -> str) -> None)
and another idea, requiring both outer parentheses and argument parentheses:
((AnyStr, ((AnyStr, AnyStr, AnyStr )-> AnyStr)) -> AnyStr) ((Optional[str], tuple[_Marshallable, ...]) -> Union[Fault, tuple[_Marshallable, ...]]) (str, (Iterable[str], str) -> str) -> None)
To me, these are convincing examples of where Callable is hard to read and an arrow syntax is easier, but that doesn't necessarily mean it's worth the price of new syntax. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KWQVEQDZ... Code of Conduct: http://python.org/psf/codeofconduct/
On 12/18/2021 3:13 PM, Batuhan Taskaya wrote:
tl;dr: I find it very troubling that we are going on a path where need to increase the language complexity (syntax) only in the cause 'easier' typing.
Which brings up the question is whether it's worth adding syntax for typing, but only in the context of typing. As of right now, typing.get_type_hints() will evaluate a string annotation, e.g. In [62]: def f(x:"int"): ...: pass ...: In [63]: typing.get_type_hints(f) Out[63]: {'x': int} so get_type_hints could extend its acceptable syntax with this new use of -> -- and it could get used by wrapping it in quotes. And depending on how PEP 563 gets resolved, the quotes may not be necessary in the future. And this could open up some other nifty things, like extending what's allowable inside [] -- there was a discussion a while back on python-ideas about extending the __getitem__ protocol, partly motivated by type hints. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sun, Dec 19, 2021 at 8:26 PM Christopher Barker
On 12/18/2021 3:13 PM, Batuhan Taskaya wrote:
tl;dr: I find it very troubling that we are going on a path where need to increase the language complexity (syntax) only in the cause 'easier' typing.
Which brings up the question is whether it's worth adding syntax for typing, but only in the context of typing.
The SC has already said we don't like that idea as that bifurcates the knowledge one needs in order to even have a chance at comprehending a type hint. Plus PEP 649 wouldn't be possible in that instance unless we ship a second parser just for type hints in order to translate the type-specific syntax to type-related objects. -Brett
As of right now, typing.get_type_hints() will evaluate a string annotation, e.g.
In [62]: def f(x:"int"): ...: pass ...:
In [63]: typing.get_type_hints(f) Out[63]: {'x': int}
so get_type_hints could extend its acceptable syntax with this new use of -> -- and it could get used by wrapping it in quotes. And depending on how PEP 563 gets resolved, the quotes may not be necessary in the future.
And this could open up some other nifty things, like extending what's allowable inside [] -- there was a discussion a while back on python-ideas about extending the __getitem__ protocol, partly motivated by type hints.
-CHB
-- Christopher Barker, PhD (Chris)
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JWPMHLSP... Code of Conduct: http://python.org/psf/codeofconduct/
18.12.21 23:07, Terry Reedy пише:
Batuhan expresses my concerns better than I could, so I just add my agreement.
On 12/18/2021 3:13 PM, Batuhan Taskaya wrote:
tl;dr: I find it very troubling that we are going on a path where need to increase the language complexity (syntax) only in the cause 'easier' typing. So I am opposed to this change.
I concur.
Thanks for the feedback. I have a few thoughts. (1) Concerns about complexity of the syntax make sense to me, it's definitely possible to write confusing types with this syntax. Readability would be a good reason to reject this idea, but it does cut both ways because `Callable` can be hard to read today. Typing-sig as a whole is confident that an arrow syntax would be an improvement but I think the submission process is a good time for wider opinions, I can see that it might not be a good idea overall even if it's a better type syntax. (2) I do like the idea of allowing parameters in a tuple in the existing Callable type, that seems like a very clear easy win if we *don't* accept callable syntax. I agree that tuple-for-parameters combined with making `builtins.callable` subscriptable would address a chunk of the concerns. I can look into getting more stats on async callables, I don't think they are terribly common in most projects, but in certain contexts like async webservers they can come up quite a lot.
I agree with what Batuhan said. Adding on, I'm very concerned about the potential maintenance burden for Python implementations. Just for typing in CPython, some combination of the following knowledge is required to contribute code: 1. Metaclasses 2. Descriptors and CPython dunders 3. C internals due to PEP 585 and 604 4. If we throw in annotations internals, maybe we need some parser knowledge too, but this is usually not required. With the new callable syntax, that likely needs strong parser and compiler knowledge. I'm not claiming everyone needs to know all of the above to contribute (to the contrary, not knowing most of these is fine), but IMO the list is getting very long. I'd be sad if one day new typing contributors feel too overwhelmed by typing's implementation complexity (I admit sometimes feeling overwhelmed too). That said, thank you PEP authors for working to make Python's typing syntax more elegant. I hope my concerns didn't come across as overly harsh. - KJ
participants (8)
-
Batuhan Taskaya -
Brett Cannon -
Christopher Barker -
Ken Jin -
Paul Moore -
Serhiy Storchaka -
Steven Troxler -
Terry Reedy