Compact syntax for Callable: `(Arg1, Arg2) -> Ret`
Hi typing-sig, I’d like to start a discussion about moving forward with improved syntax for typing.Callable based on discussion in the Typing Summit: * Allow `(ArgType0, ArgType1) -> ReturnType` * In place of `Callable[[ArgType0, ArgType1], ReturnType]` A lot of folks also want a way to express things like named, optional, and variadic args, but this is more complicated and has harder tradeoffs. I'll start a separate thread next week to discuss that, let's aim to keep this thread focused on the syntactic sugar described above. If we think this is a good idea, I'm happy to work on drafting a PEP. ## Background The existing Callable type is useful for simple functions with required arguments passed by position, but there’s consensus that it can be hard to keep track of brackets, especially as the types get complex. For example, even in very simple functional-style code with higher-order functions the brackets can be cumbersome: ``` from typing import Callable def accepts_callback(callback: Callable[[str, int], str]) -> None: print(callback("answer", 42) def f(s: str, x: int) -> str: return s + ": " + str(x) accepts_callback(f) accepts_callback(lambda s, _: s) ``` In agreement with a poll from the Typing Summit, I’d like to propose allowing * `(Argtype1, Argtype2) -> ReturnType` to be used in place of * `Callable[[Argtype1, Argtype2], ReturnType]` This allows us to rewrite the example above as: ``` def accepts_callback(callback: (str, int) -> str) -> None: print(callback("answer", 42) def f(s: str, x: int) -> str: return s + ": " + str(x) accepts_callback(f) accepts_callback(lambda s, _: s) ``` Advantages of arrow syntax include: * eliminate the need to import Callable * easier-to-skim types * they are shorter, and one of the problems with Callables is just verbosity * the parentheses and arrow have more visual structure than existing bracket-based Callable syntax * a syntax that should feel familiar: * it mimics the syntax for ordinary functions and stubs * it is similar to several other languages e.g. * [typescript](https://basarat.gitbook.io/typescript/type-system/callable#arrow-syntax) * [flow](https://flow.org/en/docs/types/functions/#toc-function-types) * [kotlin](https://kotlinlang.org/docs/lambdas.html#function-types) ## Open questions about syntax rules Would we require parentheses around a callable return type as above, or allow it to be bare (which causes two ->s to be visually at the same level of the code, even though one is nested in the AST)? Consider this function (annotated using a conservative syntax where we wrap the return type in paranetheses): ``` def identity( f: (str, int, int) -> int, ) -> ((str, int, int) -> int): return f ``` In this case, would we want to * require parentheses around a callable return type as above, or * allow it to be bare (which causes two ->s to look a little bit as if they are at the same code level) Here’s what the example above would look like if we allow bare arrow syntax in the return type: ``` def identity( f: (str, int, int) -> int, ) -> (str, int, int) -> int: return f ``` Another option, if we feel like the outer parentheses are necessary would be to make the parameter grouping optional in this case, for example: ``` def identity( f: (str, int, int) -> int, ) -> (str, int, int -> int): return f ``` ## Alternatives considered We ran a poll at the typing summit comparing three options for nicer Callable syntax: * This proposal `(ArgType1, ArgType2, ArgType3) -> ReturnType` * 72% of participants upvoted this * A typescript-like syntax `(a: ArgType1, b: ArgType2) => ReturnType` * 62% of participants upvoted this * Disadvantages are * a new symbol `=>` * argument names are nuisance parameters for positional args (which is all the existing Callable supports) * `[ArgType1, ArgType2, ArgType3] -> ReturnType` * 8% of participants upvoted this * The main disadvantage is that brackets don’t seem as intuitive - Steven
Am 10.06.21 um 02:25 schrieb Steven Troxler:
## Alternatives considered
We ran a poll at the typing summit comparing three options for nicer Callable syntax:
* This proposal `(ArgType1, ArgType2, ArgType3) -> ReturnType` * 72% of participants upvoted this * A typescript-like syntax `(a: ArgType1, b: ArgType2) => ReturnType` * 62% of participants upvoted this * Disadvantages are * a new symbol `=>` * argument names are nuisance parameters for positional args (which is all the existing Callable supports) * `[ArgType1, ArgType2, ArgType3] -> ReturnType` * 8% of participants upvoted this * The main disadvantage is that brackets don’t seem as intuitive
I think option 1 is the best. This could later be extended to also allow "(a: ArgType) -> None" or "(ArgType = ...)", but for now having syntactic sugar for Callable would be nice. One thing I miss is an alternative for "Callable[..., None]", though. This is be part of this PEP, to be able to fully replace Callable. Options I can see are "(...) -> None" or "... -> None". I think the main obstacle with this PEP is that this is the first time that we allow special syntax inside type annotations. This will be a hard sell. Finally, a bit off-topic for this proposal, but related: Łukasz suggested to be able to use ordinary "def"s for more complicated definitions: def Foo(x: int, *, y: str = ...) -> Any: ... def bar(cb: Foo) -> None: cb(123, y="abc") This could be an easy replacement for callable protocols. - Sebastian
Something to consider is whether this new form should be limited to the functionality provided by `Callable` or whether it should be expanded to support the full range of supported callable signatures. This would include: 1. The use of *args and **kwargs parameters 2. Keyword (named) parameters 3. Position-only and keyword-only separators (`/` and `*`, respectively) The new proposed syntax could support all of these if desired. I suspect this is where Łukasz was going with his line of thinking. But his proposal (if I understand it correctly) blurs the lines between a function declaration and a callable type declaration. I'd encourage us to keep those distinct. We can (and should) adopt similar syntax for both cases though. Examples: ```python C1 = (x: float, /) -> float # Equivalent to Callable[[float], float] C2 = (x: float) -> float # Not expressible with Callable C3 = (*, x: float) -> float # Not expressible with Callable C4 = (v: int, /, x: float) -> float # Not expressible with Callable C5 = (x: int, *args: str, **kwargs: str) -> int # Not expressible with Callable C6 = (*args: Any, **kwargs: Any) -> None # Equivalent to Callable[..., None] ``` Note that `C6` above handles the `Callable[..., None]` case that Sebastian raised. Another potential consideration is support for PEP 612. Callable handles `ParamSpec` and `Concatenate` in a way that would be difficult to support using this alternate syntax. These are edge cases, so perhaps it's fine to say that you need to fall back on `Callable` if you want to use `ParamSpec` and `Concatenate`. -Eric --- Eric Traut Contributor to Pylance & Pyright Microsoft Corp.
Am 11.06.21 um 01:03 schrieb Eric Traut:
Examples: ```python C1 = (x: float, /) -> float # Equivalent to Callable[[float], float] C2 = (x: float) -> float # Not expressible with Callable C3 = (*, x: float) -> float # Not expressible with Callable C4 = (v: int, /, x: float) -> float # Not expressible with Callable C5 = (x: int, *args: str, **kwargs: str) -> int # Not expressible with Callable C6 = (*args: Any, **kwargs: Any) -> None # Equivalent to Callable[..., None] ```
I really like that these look exactly like function declaration minus "def" and the name. The question is whether to even allow something like "(float) -> float" as a "shortcut" for C1 above as suggested originally by Steven. The advantage is that it's a bit more concise for the common callback case, where names don't matter, the disadvantage is that it's inconsistent with existing Python syntax. If we go for the shortcut syntax we should always require parentheses around the arguments for readability and disambiguation. - Sebastian
The shortcut (float) -> float for C1 makes sense to me, because the other interpretation would be equivalent to (float: Any) -> float, which isn’t very useful. From a readability point of view I think making the parentheses mandatory is desirable. I do think we need to seriously consider how a similar notation could also be used as an alternative for lambda (JavaScript has this, as does Scala IIRC). On Fri, Jun 11, 2021 at 03:45 Sebastian Rittau <srittau@rittau.biz> wrote:
Am 11.06.21 um 01:03 schrieb Eric Traut:
Examples: ```python C1 = (x: float, /) -> float # Equivalent to Callable[[float], float] C2 = (x: float) -> float # Not expressible with Callable C3 = (*, x: float) -> float # Not expressible with Callable C4 = (v: int, /, x: float) -> float # Not expressible with Callable C5 = (x: int, *args: str, **kwargs: str) -> int # Not expressible with Callable C6 = (*args: Any, **kwargs: Any) -> None # Equivalent to Callable[..., None] ```
I really like that these look exactly like function declaration minus "def" and the name. The question is whether to even allow something like "(float) -> float" as a "shortcut" for C1 above as suggested originally by Steven. The advantage is that it's a bit more concise for the common callback case, where names don't matter, the disadvantage is that it's inconsistent with existing Python syntax.
If we go for the shortcut syntax we should always require parentheses around the arguments for readability and disambiguation.
- Sebastian
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido (mobile)
Am 11.06.21 um 16:20 schrieb Guido van Rossum:
The shortcut (float) -> float for C1 makes sense to me, because the other interpretation would be equivalent to (float: Any) -> float, which isn’t very useful.
From a readability point of view I think making the parentheses mandatory is desirable.
I do think we need to seriously consider how a similar notation could also be used as an alternative for lambda (JavaScript has this, as does Scala IIRC).
I'd like a shorter lambda syntax. One thing to consider though: "(float) -> None" would mean different things in lambdas ((float: Any) -> None) and type annotations ((x: float, /) -> None). - Sebastian
Any thoughts about handy syntax for coroutine functions? On Fri, 2021-06-11 at 07:20 -0700, Guido van Rossum wrote:
The shortcut (float) -> float for C1 makes sense to me, because the other interpretation would be equivalent to (float: Any) -> float, which isn’t very useful.
From a readability point of view I think making the parentheses mandatory is desirable.
I do think we need to seriously consider how a similar notation could also be used as an alternative for lambda (JavaScript has this, as does Scala IIRC).
On Fri, Jun 11, 2021 at 03:45 Sebastian Rittau <srittau@rittau.biz> wrote:
Am 11.06.21 um 01:03 schrieb Eric Traut:
Examples: ```python C1 = (x: float, /) -> float # Equivalent to Callable[[float], float] C2 = (x: float) -> float # Not expressible with Callable C3 = (*, x: float) -> float # Not expressible with Callable C4 = (v: int, /, x: float) -> float # Not expressible with Callable C5 = (x: int, *args: str, **kwargs: str) -> int # Not expressible with Callable C6 = (*args: Any, **kwargs: Any) -> None # Equivalent to Callable[..., None] ```
I really like that these look exactly like function declaration minus "def" and the name. The question is whether to even allow something like "(float) -> float" as a "shortcut" for C1 above as suggested originally by Steven. The advantage is that it's a bit more concise for the common callback case, where names don't matter, the disadvantage is that it's inconsistent with existing Python syntax.
If we go for the shortcut syntax we should always require parentheses around the arguments for readability and disambiguation.
- Sebastian
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: pbryan@anode.ca
I was planning to start a second thread about a more powerful syntax for Callable, but I think given concerns about forward-compatibility it makes sense to discuss here: We’d like an easy way to express a callable using language features that the existing Callable type cannot describe: * optional arguments (with default values) * named and keyword-only arguments * variadic *args and *kwargs The only general way to do this right now is via a protocol, as suggested in the mypy docs (https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols): ``` from typing import Protocol, Iterable, List class Combiner(Protocol): def __call__( self, *vals: bytes, maxlen: int | None = None ) -> List[bytes]: ... def batch_proc(data: Iterable[bytes], cb_results: Combiner) -> bytes: for item in data: ... ``` This has a bunch of disadvantages: * It requires lots of boilerplate to import Protocol and build the dummy wrapper class * There’s an extraneous self in the __call__ signature that isn’t part of implementations * Using a Protocol (probably) isn’t obvious to folks who haven’t seen this suggestion already * The type isn’t first-class, we must define a Combiner alias because there’s no way to directly specify it Pyre has a syntax (only used to display type errors) for general callable types. The type of a Combiner would be displayed as `Callable[[Args(vals, bytes), KeywordOnly(maxlen, int | None)`. This probably shouldn't be used for a PEP because * it’s even more cumbersome than the Callable syntax we’re proposing improving above * it doesn’t clearly distinguish optional vs non-optional arguments ## A quick solution: function-as-a-type A proposal from Łukasz is to allow functions to be used as types, which would allow us to rewrite the example above as ``` from typing import Iterable, List def combiner(*vals: bytes, maxlen: int | None = None) -> List[bytes]: ... def batch_proc(data: Iterable[bytes], cb_results: combiner) -> bytes: for item in data: ... ``` This has some awesome advantages: * it’s a pure-typing change, the python interpreter would not need any updates * the syntax is familiar for anyone used to annotations already * in some situations a default function would already be available to use as a type The main disadvantage is that the type still isn’t first-class: we can’t directly specify the type, instead we have to define a combiner function and refer to it in our annotations. Moreover, any positional-only arguments require dummy names that are irrelevant to the actual signature. One thing we should be clear on is that accepting Łukasz's proposal does not at all prevent us from also working toward a first-class callable syntax that can express these things, so it might be well-worth making a PEP for just the function-as-a-type semantics. ## first-class callable syntax: Eric's proposal Another option as proposed by Eric would be to use a stub-compatible syntax ``` ( _positional0: float, # required positional-only _positional1: float = ..., # optional positional-only /, a: bool, # required b: bool = ..., # optional *args: object, **kwargs: object, ) -> list[int] ``` This gives us a first-class callable type. My main concern is that the argument names for positional-only arguments are nuisance parameters. This creates two headaches: * when generating error messages, we'd probably have to auto-generate unique parameter names * This could be mildly annoying to implement and also mildly confusing for users * What I think is more important is that the nuisance parameters are likely to confuse users * I would guess that a minority of python devs are even aware of the `/` feature for positional args * Suddenly they'd have to use fake argnames and `/` for everything expressible using the existing Callable ## first-class callable syntax, no argnames for positional-only args Another option would be to extend the syntax proposed above, for example something like ``` ( float, # required positional-only float = ..., # optional positional-only a: bool, # required b: bool = ..., # optional *args: object, **kwargs: object, ) -> list[int] where general arguments are named and positional-only arguments have no name. There would be no need to accept a / for positional-only arguments since they are indicated by lack of a name, but we would probably want to accept a bare * for keyword-only arguments: ``` ( int, # positional-only a: int, # normal * b: int, # required keyword-only c: int = ..., # optional keyword-only ) -> list[int] ``` This has an advantage in that the representation of a type is first-class and canonical because the nuisance argument names for positional-only arguments are eliminated. It also would be pretty compact for simple signatures, e.g. ``` (a: int, b: list[int]) -> list[int] ``` while scaling reasonably well over multiple lines for more complicated ones. The disadvantage is that the syntax is slightly different from stub syntax. In my mind this is okay, because * Łukasz's proposal gives folks a stub-like alternative * stubs and types are not the same * a stub is an annotation of a concrete function with an implementation, so the arg names have a meaning * but in a type that abstracts implementations, the positional-only args don't have a name at all The biggest reasons I'd prefer a no-name syntax are: * easy compatibility with the existing `Callable` type, which seems to me like a top priority * I suspect that in the long run teaching a new syntax will be less work than teaching that the argnames aren't part of the type
Re: nuisance parameter names: - In TypeScript there is a similar issue and they just live with it - In C the type comes first and the arg name is optional; but the name is often useful as documentation So perhaps the nuisance parameter names aren't the worst compromise. When you don't want to make up a name, you can write _ if there's only one (surely common), or _1, _2, ... if there are multiple. On Fri, Jun 11, 2021 at 9:17 AM Steven Troxler <steven.troxler@gmail.com> wrote:
I was planning to start a second thread about a more powerful syntax for Callable, but I think given concerns about forward-compatibility it makes sense to discuss here:
We’d like an easy way to express a callable using language features that the existing Callable type cannot describe: * optional arguments (with default values) * named and keyword-only arguments * variadic *args and *kwargs
The only general way to do this right now is via a protocol, as suggested in the mypy docs ( https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols): ``` from typing import Protocol, Iterable, List
class Combiner(Protocol): def __call__( self, *vals: bytes, maxlen: int | None = None ) -> List[bytes]: ...
def batch_proc(data: Iterable[bytes], cb_results: Combiner) -> bytes: for item in data: ... ``` This has a bunch of disadvantages:
* It requires lots of boilerplate to import Protocol and build the dummy wrapper class * There’s an extraneous self in the __call__ signature that isn’t part of implementations * Using a Protocol (probably) isn’t obvious to folks who haven’t seen this suggestion already * The type isn’t first-class, we must define a Combiner alias because there’s no way to directly specify it
Pyre has a syntax (only used to display type errors) for general callable types. The type of a Combiner would be displayed as `Callable[[Args(vals, bytes), KeywordOnly(maxlen, int | None)`. This probably shouldn't be used for a PEP because
* it’s even more cumbersome than the Callable syntax we’re proposing improving above * it doesn’t clearly distinguish optional vs non-optional arguments
## A quick solution: function-as-a-type
A proposal from Łukasz is to allow functions to be used as types, which would allow us to rewrite the example above as
``` from typing import Iterable, List
def combiner(*vals: bytes, maxlen: int | None = None) -> List[bytes]: ...
def batch_proc(data: Iterable[bytes], cb_results: combiner) -> bytes: for item in data: ... ``` This has some awesome advantages:
* it’s a pure-typing change, the python interpreter would not need any updates * the syntax is familiar for anyone used to annotations already * in some situations a default function would already be available to use as a type
The main disadvantage is that the type still isn’t first-class: we can’t directly specify the type, instead we have to define a combiner function and refer to it in our annotations. Moreover, any positional-only arguments require dummy names that are irrelevant to the actual signature.
One thing we should be clear on is that accepting Łukasz's proposal does not at all prevent us from also working toward a first-class callable syntax that can express these things, so it might be well-worth making a PEP for just the function-as-a-type semantics.
## first-class callable syntax: Eric's proposal
Another option as proposed by Eric would be to use a stub-compatible syntax ``` ( _positional0: float, # required positional-only _positional1: float = ..., # optional positional-only /, a: bool, # required b: bool = ..., # optional *args: object, **kwargs: object, ) -> list[int] ``` This gives us a first-class callable type. My main concern is that the argument names for positional-only arguments are nuisance parameters. This creates two headaches: * when generating error messages, we'd probably have to auto-generate unique parameter names * This could be mildly annoying to implement and also mildly confusing for users * What I think is more important is that the nuisance parameters are likely to confuse users * I would guess that a minority of python devs are even aware of the `/` feature for positional args * Suddenly they'd have to use fake argnames and `/` for everything expressible using the existing Callable
## first-class callable syntax, no argnames for positional-only args
Another option would be to extend the syntax proposed above, for example something like ``` ( float, # required positional-only float = ..., # optional positional-only a: bool, # required b: bool = ..., # optional *args: object, **kwargs: object, ) -> list[int]
where general arguments are named and positional-only arguments have no name.
There would be no need to accept a / for positional-only arguments since they are indicated by lack of a name, but we would probably want to accept a bare * for keyword-only arguments: ``` ( int, # positional-only a: int, # normal * b: int, # required keyword-only c: int = ..., # optional keyword-only ) -> list[int] ```
This has an advantage in that the representation of a type is first-class and canonical because the nuisance argument names for positional-only arguments are eliminated. It also would be pretty compact for simple signatures, e.g. ``` (a: int, b: list[int]) -> list[int] ``` while scaling reasonably well over multiple lines for more complicated ones.
The disadvantage is that the syntax is slightly different from stub syntax. In my mind this is okay, because * Łukasz's proposal gives folks a stub-like alternative * stubs and types are not the same * a stub is an annotation of a concrete function with an implementation, so the arg names have a meaning * but in a type that abstracts implementations, the positional-only args don't have a name at all
The biggest reasons I'd prefer a no-name syntax are: * easy compatibility with the existing `Callable` type, which seems to me like a top priority * I suspect that in the long run teaching a new syntax will be less work than teaching that the argnames aren't part of the type _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I wonder if it would make sense to allow _ for all positional parameter names. That would be reasonably consistent with the interpreter (although not with stub/function headers), for example this is legal:
(_, _) = (5, 6)
After a closer look `*args` and `**kwargs` are also nuisance argument names (thanks to Pradeep for pointing this out)
That would be a pretty simple change -- the grammar already accepts that, we would just have to disable the separate check that raises "SyntaxError: duplicate argument '_' in function definition" (and only if the name is "_"). On Fri, Jun 11, 2021 at 10:10 AM Steven Troxler <steven.troxler@gmail.com> wrote:
I wonder if it would make sense to allow _ for all positional parameter names.
That would be reasonably consistent with the interpreter (although not with stub/function headers), for example this is legal:
(_, _) = (5, 6)
After a closer look `*args` and `**kwargs` are also nuisance argument names (thanks to Pradeep for pointing this out) _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I agree that nuisance parameter names offer a good compromise. It's better IMO than creating an inconsistent syntax with subtly different rules in different contexts. Steven mentioned that pyre uses a special syntax for Callable types that appear in diagnostic messages. Pyright also uses a special syntax in its diagnostic messages. It happens to be consistent with my proposal above. :) You can see this if you run pyright on the following code. ```python from typing import Callable, ParamSpec, Protocol, TypeVar class Callback1(Protocol): def __call__(self, x: int, /, y: str, *, z: int = ...) -> None: ... _P = ParamSpec("_P") _T = TypeVar("_T") def get_sig(x: Callable[_P, _T]) -> Callable[_P, _T]: """ Converts a callable protocol instance into its callable signature """ ... def func(x: Callable[[int, str], None], y: Callback1): reveal_type(x) # "(_p0: int, _p1: str) -> None" reveal_type(get_sig(y)) # "(x: int, /, y: str, *, z: int = ...) -> None" ``` Note that parameter names (e.g. `_p0` and `_p1) are synthesized when they are not known. I agree that most Python users are not aware of `/`, so that could be an issue with adoption. The distinction between positional-only and position-or-keyword parameters is significant, so one could argue that it's important to educate Python programmers about this distinction. However, if we decide that `/` is too ugly or onerous to include, we could support the pre-PEP-570 convention of naming position-only parameters with double underscores (`__p0`, etc.). ``` C1 = (__p0: int, __p1: int) -> int # Equivalent to Callable[[int, int], int] ``` Alternatively, we could support the name `_` for all parameters that are position-only and suspend the normal semantic rule that parameters names need to be unique within a signature. I kind of like this option because it's readable, concise, and still follows the standard syntax rules for `def`. ``` C1 = (_: int, _: int) -> int # Equivalent to Callable[[int, int], int] ``` -Eric
It might be worth thinking about whether leading underscores make sense. Especially the convention of allowing multiple `_` I do think it might be more intuitive than requiring `/` (my unsubstantiated hunch is that the fraction of users who could guess from context what `_` does without having to use a search engine would be much higher) I think if we coalesce on a stub-consistent syntax like this for generic Callable, there's still a question of whether to allow a shorthand without argument names for all-positional-argument callables e.g. ``` (int, str) -> float ``` as shorthand for ``` (_: int, _: str, / ) -> float # if we require the slash (_: int, _: str) -> float # if not ``` The downside is more complex syntax with special rules, but it would be nice for our "improved" syntax to be concise for existing `Callable` annotations and the most common, simple cases
Wanted to chime in on the leading underscores workaround – I think this makes sense as a shorthand if we’re forced to provide a name, but the ‘_’ symbol to me is still associated with “unused” rather than “positional” argument, and that this isn’t intuitive unless someone has read the rules in the PEP and knows this is something they need to add to every callable type. I’m also worried about the extra verbosity when many callables may not need to deal with or think about named parameters at all. I’d be supportive of the shorthand where `(int, str) -> float` is still acceptable if we do not have named params. Additionally, what are the concerns with allowing a hybrid of named and unnamed params, ie.: f: (int, str, *, foo: int = ..., **kwargs: str) -> bool From: Steven Troxler <steven.troxler@gmail.com> Date: Friday, June 11, 2021 at 1:49 PM To: typing-sig@python.org <typing-sig@python.org> Subject: [Typing-sig] Re: Compact syntax for Callable: `(Arg1, Arg2) -> Ret` It might be worth thinking about whether leading underscores make sense. Especially the convention of allowing multiple `_` I do think it might be more intuitive than requiring `/` (my unsubstantiated hunch is that the fraction of users who could guess from context what `_` does without having to use a search engine would be much higher) I think if we coalesce on a stub-consistent syntax like this for generic Callable, there's still a question of whether to allow a shorthand without argument names for all-positional-argument callables e.g. ``` (int, str) -> float ``` as shorthand for ``` (_: int, _: str, / ) -> float # if we require the slash (_: int, _: str) -> float # if not ``` The downside is more complex syntax with special rules, but it would be nice for our "improved" syntax to be concise for existing `Callable` annotations and the most common, simple cases _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/<https://mail.python.org/mailman3/lists/typing-sig.python.org/> Member address: szhu@fb.com
On Wed, Jun 16, 2021 at 10:07 AM Shannon Zhu via Typing-sig < typing-sig@python.org> wrote:
Wanted to chime in on the leading underscores workaround –
I think this makes sense as a shorthand if we’re forced to provide a name, but the ‘_’ symbol to me is still associated with “unused” rather than “positional” argument, and that this isn’t intuitive unless someone has read the rules in the PEP and knows this is something they need to add to every callable type.
This is a good point: sometimes you really *do* have a parameter position that is unused (needed for historical reasons / compatibility).
I’m also worried about the extra verbosity when many callables may not need to deal with or think about named parameters at all.
Yeah, this is the main reason why we need to keep looking.
I’d be supportive of the shorthand where `(int, str) -> float` is still acceptable if we do not have named params.
Yeah, I think this is a reasonable compromise, even if it means that there's a discrepancy: it would mean that def foo(bar, baz) -> int: ... does not correspond to the signature foo: (bar, baz) -> int instead it corresponds roughly to foo: (Any, Any) -> int But really the latter would be a precise match for def foo(bar, baz, /) -> int: ...
Additionally, what are the concerns with allowing a hybrid of named and unnamed params, ie.:
f: (int, str, *, foo: int = ..., **kwargs: str) -> bool
If we were to allow this, it'd be harder to give a simple rule for when the argument names can be omitted or not -- for example would we allow this? f: (int, name: str) -> bool or this? f: (count: int, str) -> bool or even this? f: (a: int, b, c: str) -> bool -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I would favor a shorthand where a bare name means a typed positional-only argument. That means we both get a nice shorthand (e.g., `(int, str) -> int` corresponds to `Callable[[int, str], int]`) and full power to represent all signatures (you can write `(a: int, /, b: str, *, c: int = ...) -> int` if you really need it). El mié, 16 jun 2021 a las 11:06, Guido van Rossum (<guido@python.org>) escribió:
On Wed, Jun 16, 2021 at 10:07 AM Shannon Zhu via Typing-sig < typing-sig@python.org> wrote:
Wanted to chime in on the leading underscores workaround –
I think this makes sense as a shorthand if we’re forced to provide a name, but the ‘_’ symbol to me is still associated with “unused” rather than “positional” argument, and that this isn’t intuitive unless someone has read the rules in the PEP and knows this is something they need to add to every callable type.
This is a good point: sometimes you really *do* have a parameter position that is unused (needed for historical reasons / compatibility).
I’m also worried about the extra verbosity when many callables may not need to deal with or think about named parameters at all.
Yeah, this is the main reason why we need to keep looking.
I’d be supportive of the shorthand where `(int, str) -> float` is still acceptable if we do not have named params.
Yeah, I think this is a reasonable compromise, even if it means that there's a discrepancy: it would mean that
def foo(bar, baz) -> int: ...
does not correspond to the signature
foo: (bar, baz) -> int
instead it corresponds roughly to
foo: (Any, Any) -> int
But really the latter would be a precise match for
def foo(bar, baz, /) -> int: ...
Additionally, what are the concerns with allowing a hybrid of named and unnamed params, ie.:
f: (int, str, *, foo: int = ..., **kwargs: str) -> bool
If we were to allow this, it'd be harder to give a simple rule for when the argument names can be omitted or not -- for example would we allow this?
f: (int, name: str) -> bool
This would be shorthand for a function that takes a positional-only argument of type int and a positional-or-keyword argument named "name" of type str.
or this?
f: (count: int, str) -> bool
or even this?
f: (a: int, b, c: str) -> bool
These two would be illegal because a positional-only argument cannot follow a positional-or-keyword argument.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
On Wed, Jun 16, 2021 at 11:29 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
I would favor a shorthand where a bare name means a typed positional-only argument. That means we both get a nice shorthand (e.g., `(int, str) -> int` corresponds to `Callable[[int, str], int]`) and full power to represent all signatures (you can write `(a: int, /, b: str, *, c: int = ...) -> int` if you really need it).
El mié, 16 jun 2021 a las 11:06, Guido van Rossum (<guido@python.org>) escribió:
[...]
Additionally, what are the concerns with allowing a hybrid of named and unnamed params, ie.:
f: (int, str, *, foo: int = ..., **kwargs: str) -> bool
If we were to allow this, it'd be harder to give a simple rule for when the argument names can be omitted or not -- for example would we allow this?
f: (int, name: str) -> bool
This would be shorthand for a function that takes a positional-only argument of type int and a positional-or-keyword argument named "name" of type str.
But if you spell that out, you'd have to add a /, i.e. def f(unused: int, /, name: str) -> bool: ... I would prefer to use a simpler rule: - Either there are *no* colons, stars, slashes or equal signs, and then it's a bunch of positional-only args given by type; example: (int, str) -> bool. - Or the argument list follows the rules of 'def'; example: (a: int, b: str) -> bool. In this form a "bare" name represents an argument name and has the implicit type Any, like it has in function definitions. (Or Undefined, in checkers that make a distinction, like pyright.) or this?
f: (count: int, str) -> bool
or even this?
f: (a: int, b, c: str) -> bool
These two would be illegal because a positional-only argument cannot follow a positional-or-keyword argument.
Hm, so with my "simpler" rule, those would be untyped arguments (in the first example, the argument name would be confusingly 'str' and the type 'Any'; in the second, we'd have 'b: Any'). But I honestly don't know what users would expect that to mean -- probably if the name is a known type, they'd expect it to be a type, but if it wasn't, they'd expect it to be an argument name -- but that's not a rule we can implement. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
El mié, 16 jun 2021 a las 11:42, Guido van Rossum (<guido@python.org>) escribió:
On Wed, Jun 16, 2021 at 11:29 AM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
I would favor a shorthand where a bare name means a typed positional-only argument. That means we both get a nice shorthand (e.g., `(int, str) -> int` corresponds to `Callable[[int, str], int]`) and full power to represent all signatures (you can write `(a: int, /, b: str, *, c: int = ...) -> int` if you really need it).
El mié, 16 jun 2021 a las 11:06, Guido van Rossum (<guido@python.org>) escribió:
[...]
Additionally, what are the concerns with allowing a hybrid of named and unnamed params, ie.:
f: (int, str, *, foo: int = ..., **kwargs: str) -> bool
If we were to allow this, it'd be harder to give a simple rule for when the argument names can be omitted or not -- for example would we allow this?
f: (int, name: str) -> bool
This would be shorthand for a function that takes a positional-only argument of type int and a positional-or-keyword argument named "name" of type str.
But if you spell that out, you'd have to add a /, i.e.
def f(unused: int, /, name: str) -> bool: ...
I would prefer to use a simpler rule:
- Either there are *no* colons, stars, slashes or equal signs, and then it's a bunch of positional-only args given by type; example: (int, str) -> bool.
- Or the argument list follows the rules of 'def'; example: (a: int, b: str) -> bool. In this form a "bare" name represents an argument name and has the implicit type Any, like it has in function definitions. (Or Undefined, in checkers that make a distinction, like pyright.)
I would be OK with that too; I don't think mixing the two forms is an important use case.
or this?
f: (count: int, str) -> bool
or even this?
f: (a: int, b, c: str) -> bool
These two would be illegal because a positional-only argument cannot follow a positional-or-keyword argument.
Hm, so with my "simpler" rule, those would be untyped arguments (in the first example, the argument name would be confusingly 'str' and the type 'Any'; in the second, we'd have 'b: Any').
But I honestly don't know what users would expect that to mean -- probably if the name is a known type, they'd expect it to be a type, but if it wasn't, they'd expect it to be an argument name -- but that's not a rule we can implement.
That's a good point, and that seems like an additional argument to make this an error. This confusion can also appear under your proposal though: maybe a user will write (a, b) -> bool expecting a function with two arguments of any type. As a side note, CPython is already able to parse a subset of this syntax, though the feature is undocumented:
ast.dump(compile("(a, b) -> int", "", mode="func_type", flags=ast.PyCF_ONLY_AST)) "FunctionType(argtypes=[Name(id='a', ctx=Load()), Name(id='b', ctx=Load())], returns=Name(id='int', ctx=Load()))"
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
In practice, I’m not as concerned with the potential pitfall of confusion between a type annotation and an argument name. I think it may look similar when we’re using lowercase variable placeholders in hypothetical examples, but in practice if someone wrote `(a, b) -> bool` where `a` and `b` are parameter names, they’ll get an UndefinedType error complaining that `a` and `b` aren’t valid types. Even before that, I think it would be somewhat unintuitive for someone decide to specify a callable type, then want to put parameters in without types, and go fetch names for those parameters instead of just using “…”. Or if there’s a mix of typed/Any params, that would mean they’re actively alternating between types and param names. If we want to support either all-or-nothing named params that seems reasonable to me, ie., what Guido mentioned about either fully shorthand or fully spelled out. * In this form a "bare" name represents an argument name and has the implicit type Any, like it has in function definitions What are your thoughts on erroring on this and mandating an explicit Any if you’re using the full form syntax – so all parameters must both have a name and a type if you aren’t using the shorthand version? From: Jelle Zijlstra <jelle.zijlstra@gmail.com> Date: Wednesday, June 16, 2021 at 2:58 PM To: Guido van Rossum <guido@python.org> Cc: Shannon Zhu <szhu@fb.com>, Steven Troxler <steven.troxler@gmail.com>, typing-sig@python.org <typing-sig@python.org> Subject: Re: [Typing-sig] Re: Compact syntax for Callable: `(Arg1, Arg2) -> Ret` El mié, 16 jun 2021 a las 11:42, Guido van Rossum (<guido@python.org<mailto:guido@python.org>>) escribió: On Wed, Jun 16, 2021 at 11:29 AM Jelle Zijlstra <jelle.zijlstra@gmail.com<mailto:jelle.zijlstra@gmail.com>> wrote: I would favor a shorthand where a bare name means a typed positional-only argument. That means we both get a nice shorthand (e.g., `(int, str) -> int` corresponds to `Callable[[int, str], int]`) and full power to represent all signatures (you can write `(a: int, /, b: str, *, c: int = ...) -> int` if you really need it). El mié, 16 jun 2021 a las 11:06, Guido van Rossum (<guido@python.org<mailto:guido@python.org>>) escribió: [...] Additionally, what are the concerns with allowing a hybrid of named and unnamed params, ie.: f: (int, str, *, foo: int = ..., **kwargs: str) -> bool If we were to allow this, it'd be harder to give a simple rule for when the argument names can be omitted or not -- for example would we allow this? f: (int, name: str) -> bool This would be shorthand for a function that takes a positional-only argument of type int and a positional-or-keyword argument named "name" of type str. But if you spell that out, you'd have to add a /, i.e. def f(unused: int, /, name: str) -> bool: ... I would prefer to use a simpler rule: - Either there are *no* colons, stars, slashes or equal signs, and then it's a bunch of positional-only args given by type; example: (int, str) -> bool. - Or the argument list follows the rules of 'def'; example: (a: int, b: str) -> bool. In this form a "bare" name represents an argument name and has the implicit type Any, like it has in function definitions. (Or Undefined, in checkers that make a distinction, like pyright.) I would be OK with that too; I don't think mixing the two forms is an important use case. or this? f: (count: int, str) -> bool or even this? f: (a: int, b, c: str) -> bool These two would be illegal because a positional-only argument cannot follow a positional-or-keyword argument. Hm, so with my "simpler" rule, those would be untyped arguments (in the first example, the argument name would be confusingly 'str' and the type 'Any'; in the second, we'd have 'b: Any'). But I honestly don't know what users would expect that to mean -- probably if the name is a known type, they'd expect it to be a type, but if it wasn't, they'd expect it to be an argument name -- but that's not a rule we can implement. That's a good point, and that seems like an additional argument to make this an error. This confusion can also appear under your proposal though: maybe a user will write (a, b) -> bool expecting a function with two arguments of any type. As a side note, CPython is already able to parse a subset of this syntax, though the feature is undocumented:
ast.dump(compile("(a, b) -> int", "", mode="func_type", flags=ast.PyCF_ONLY_AST)) "FunctionType(argtypes=[Name(id='a', ctx=Load()), Name(id='b', ctx=Load())], returns=Name(id='int', ctx=Load()))"
-- --Guido van Rossum (python.org/~guido<http://python.org/~guido>) Pronouns: he/him (why is my pronoun here?)<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On Wed, Jun 16, 2021 at 12:13 PM Shannon Zhu <szhu@fb.com> wrote:
In practice, I’m not as concerned with the potential pitfall of confusion between a type annotation and an argument name.
I think it may look similar when we’re using lowercase variable placeholders in hypothetical examples, but in practice if someone wrote `(a, b) -> bool` where `a` and `b` are parameter names, they’ll get an UndefinedType error complaining that `a` and `b` aren’t valid types. Even before that, I think it would be somewhat unintuitive for someone decide to specify a callable type, then want to put parameters in without types, and go fetch names for those parameters instead of just using “…”. Or if there’s a mix of typed/Any params, that would mean they’re actively alternating between types and param names.
If we want to support either all-or-nothing named params that seems reasonable to me, ie., what Guido mentioned about either fully shorthand or fully spelled out.
- In this form a "bare" name represents an argument name and has the implicit type Any, like it has in function definitions
What are your thoughts on erroring on this and mandating an explicit Any if you’re using the full form syntax – so all parameters must both have a name and a type if you aren’t using the shorthand version?
That sounds reasonable. The one case where it would be a bit verbose is if you have a bunch of mandatory positional parameters and one optional keyword. Without this restriction you could write this as f: (int, int, str, str, mode: str = "r") -> bool whereas in your version it would have to become f: (a1: int, a2: int, a3: str, a4: str, mode: str = "r") -> bool -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
El mié, 16 jun 2021 a las 12:26, Guido van Rossum (<guido@python.org>) escribió:
On Wed, Jun 16, 2021 at 12:13 PM Shannon Zhu <szhu@fb.com> wrote:
In practice, I’m not as concerned with the potential pitfall of confusion between a type annotation and an argument name.
I think it may look similar when we’re using lowercase variable placeholders in hypothetical examples, but in practice if someone wrote `(a, b) -> bool` where `a` and `b` are parameter names, they’ll get an UndefinedType error complaining that `a` and `b` aren’t valid types. Even before that, I think it would be somewhat unintuitive for someone decide to specify a callable type, then want to put parameters in without types, and go fetch names for those parameters instead of just using “…”. Or if there’s a mix of typed/Any params, that would mean they’re actively alternating between types and param names.
If we want to support either all-or-nothing named params that seems reasonable to me, ie., what Guido mentioned about either fully shorthand or fully spelled out.
- In this form a "bare" name represents an argument name and has the implicit type Any, like it has in function definitions
What are your thoughts on erroring on this and mandating an explicit Any if you’re using the full form syntax – so all parameters must both have a name and a type if you aren’t using the shorthand version?
That sounds reasonable. The one case where it would be a bit verbose is if you have a bunch of mandatory positional parameters and one optional keyword. Without this restriction you could write this as
f: (int, int, str, str, mode: str = "r") -> bool
whereas in your version it would have to become
f: (a1: int, a2: int, a3: str, a4: str, mode: str = "r") -> bool
That's actually subtly different, because it would require positional-or-keyword arguments, so a callable with differently named arguments would not be compatible. The unambiguous equivalent would need `/,`. Though of course we could decide that the rules should be more lenient here.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I still favor a combination of requiring `Any` and using the lenient syntax where
f: (int, int, str, str, mode: str = ...) -> bool
can be used in place of
f: (_0: int, _1: int, _2: str, _3: str, /, mode: str = ...) -> bool
It's a pain that there is a gap between this syntax and stub syntax, but I think new syntax is usually easier to explain than familiar syntax with confusing semantics. Especially because we use a search engine when they see unfamiliar syntax, whereas we make wrong assumptions and get very confused when familiar syntax doesn't do what we expect. In particular I like that allowing no argument name - avoids having a special-case compact syntax - expresses most clearly that argument names don't actually exist as part of the type The ugliest thing in my mind is that positional-only parameters with default values would wind up having `<type> = ...` which looks weird, but it still expresses what's going on pretty well.
On Wed, Jun 16, 2021 at 12:29 PM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
El mié, 16 jun 2021 a las 12:26, Guido van Rossum (<guido@python.org>) escribió: That sounds reasonable. The one case where it would be a bit verbose is if you have a bunch of mandatory positional parameters and one optional keyword. Without this restriction you could write this as
f: (int, int, str, str, mode: str = "r") -> bool
whereas in your version it would have to become
f: (a1: int, a2: int, a3: str, a4: str, mode: str = "r") -> bool That's actually subtly different, because it would require positional-or-keyword arguments, so a callable with differently named arguments would not be compatible. The unambiguous equivalent would need `/,`. Though of course we could decide that the rules should be more lenient here.
You're right, I forgot the / there. I agree that the shorthand should always imply the /. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I realize there are different philosophies to language design, but I generally value consistency and explicitness over brevity in most cases. Every time a new inconsistency is introduced into a language, it begets more inconsistencies and exceptions. It also complicates the mental model for users of the language because they need to remember different rules and exceptions for different contexts. Python is already full of such inconsistencies, and I see Python developers trip on them every day. Today, type annotations associated with parameters are optional in a `def` statement and, if provided, always appear after a `:` token. If a `:` is not present, then it means that there is no type annotation for that parameter. Those are the syntactic and semantic rules Python developers are used to. Deviating from this to save a few extra characters doesn't strike me as a good tradeoff. For that reason, I would advise against using "naked" types without a parameter identifier and without a `:` token. I think TypeScript got it right here. -Eric ---- Eric Traut Contributor to Pyright & Pylance Microsoft Corp.
I'd like to recap what I understand from discussion so far ## Stub-like syntax A full stub-like syntax. On a complex example it would look something like this: ``` (a, b: int, c: float = ..., /, d: str, e: str | None = ..., *args: int, f: bool = ..., **kwargs: str) -> int ``` where - we might allow the duplicate use of `_` as argument names where they aren't part of the type - we might require an explicit Any for untyped arguments like `a` If we did both of those, the example above might instead look like: ``` (_: Any, _: int, _: float = ..., /, d: str, e: str | None = ..., *args: int, f: bool = ..., **kwargs: str) -> int ``` ## Compact syntax Compact syntax, which there's some debate about, like ``` (int, str) -> float ``` which we could either restrict to all-positional or allow in general, so that equivalent to the above would be: ``` (Any, Int, float = ..., /, d: str, e: str | None = ..., *args: int, f: bool = ..., **kwargs: str) -> int ``` I'll try to put together some examples of existing Callable types as well as some functions from github so we can get a little more taste for these alternatives
I collected up some examples of existing Callables and what they look like under each of the conventions we've discussed. I also have a simple example of a type that Callable can't express in each of the conventions, as well as some examples of stubs that might be used with each. The examples are a little hard to follow in email so I put them in a gist: https://gist.github.com/stroxler/119d2586a04ec35940d449fb78801b2e One thing I want to draw attention to is that stub syntax is very natural for specifying the most specific possible type for a given function signature (because a stub is 1:1 with an implementation). When we're annotating callable types, which is something we would only do when many implementations may be useful, we'll tend to use much more general types. Whether this is a reason to adjust the syntax isn't clear, but I do think it's important to realize that some things very rarely used in implementations (like positional-only args) will be extremely common to use in type annotations.
Am 16.06.21 um 20:05 schrieb Guido van Rossum:
If we were to allow this, it'd be harder to give a simple rule for when the argument names can be omitted or not -- for example would we allow this?
f: (int, name: str) -> bool
or this?
f: (count: int, str) -> bool
or even this?
f: (a: int, b, c: str) -> bool
If we allow the shorthand, I believe we should allow either the shorthand or the full syntax, but not to mix both. My current stance is +1 for the full syntax, +0 for also allowing the shorthand, and -1 for special casing underscores. I'm not totally sold on the shorthand, due to the inconsistency and possible confusion with normal Python syntax and the two ways to do things. On the other hand writing `def foo(cb: (int) -> Any) -> None:` is more practical than writing `def foo(cb: (x: int, /) -> Any) -> None: ...`. But I'm not sure the conciseness is worth it. - Sebastian
Am 17.06.21 um 10:51 schrieb Sebastian Rittau:
Am 16.06.21 um 20:05 schrieb Guido van Rossum:
If we were to allow this, it'd be harder to give a simple rule for when the argument names can be omitted or not -- for example would we allow this?
f: (int, name: str) -> bool
or this?
f: (count: int, str) -> bool
or even this?
f: (a: int, b, c: str) -> bool
If we allow the shorthand, I believe we should allow either the shorthand or the full syntax, but not to mix both. My current stance is +1 for the full syntax, +0 for also allowing the shorthand, and -1 for special casing underscores. I'm not totally sold on the shorthand, due to the inconsistency and possible confusion with normal Python syntax and the two ways to do things. On the other hand writing `def foo(cb: (int) -> Any) -> None:` is more practical than writing `def foo(cb: (x: int, /) -> Any) -> None: ...`. But I'm not sure the conciseness is worth it.
P.S.: Maybe we should start with only the full syntax and see how that goes. If it turns out that it is too cumbersome in practice, we can always allow the shorthand later. - Sebastian
On Thu, Jun 17, 2021 at 1:52 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Am 16.06.21 um 20:05 schrieb Guido van Rossum:
If we were to allow this, it'd be harder to give a simple rule for when the argument names can be omitted or not -- for example would we allow this?
f: (int, name: str) -> bool
or this?
f: (count: int, str) -> bool
or even this?
f: (a: int, b, c: str) -> bool
If we allow the shorthand, I believe we should allow either the shorthand or the full syntax, but not to mix both. My current stance is +1 for the full syntax, +0 for also allowing the shorthand, and -1 for special casing underscores. I'm not totally sold on the shorthand, due to the inconsistency and possible confusion with normal Python syntax and the two ways to do things. On the other hand writing `def foo(cb: (int) -> Any) -> None:` is more practical than writing `def foo(cb: (x: int, /) -> Any) -> None: ...`. But I'm not sure the conciseness is worth it.
Yes, it is worth it. Of all the typing constructs currently expressed using the form SomeOperator[arg, arg, ...], Callable is second after Union (which already has a new notation, a|b). So I think the new notation is important, and excess verbosity is undesirable. We can debate whether the shorthand and full notation can be mixed, and about underscores (which we won't need if we have the shorthand). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
This is my order of preference among options so far: (1) mixed shorthand syntax where explicit parameter types are required and no name implies positional - I'd prefer to make no-name-given the *only* way to do positional-only args, i.e. no `/` in the syntax - this is not consistent with stub syntax, but it avoids some edge cases we hit otherwise - I don't think requiring explicit types is a huge burden, the existing `Callable` type requires it (2) full stub syntax, with a shorthand syntax for positional-only callables - we would have to require explicit Any annotations: - otherwise, we get collisions between shorthand syntax and implicit Any - for example, is `(a, b) -> int` of type `Callable[[Any, Any], int]` or `Callable[[a, b], int`? (3) full stub syntax only, with no shorthand - it seems like a majority thinks this is burdensome because of existing Callables - If we want to be able to revisit shorthand syntax later if we have to require explicit Any - otherwise, the ambiguous case described above plus backward compatibility will trap us If we move forward with options (2) or (3) we'll still have to decide whether to allow _ as the argument name for positional-only args. My preference would be to allow it so that nuisance argnames aren't required, but to require an explicit `/`
Steven, I’ve lost track of the details of each version, and the examples you give don’t seem to disambiguate the alternatives completely. Could you write up the alternatives a bit more precisely without requiring context of the thread? On Thu, Jun 17, 2021 at 16:04 Steven Troxler <steven.troxler@gmail.com> wrote:
This is my order of preference among options so far:
(1) mixed shorthand syntax where explicit parameter types are required and no name implies positional - I'd prefer to make no-name-given the *only* way to do positional-only args, i.e. no `/` in the syntax - this is not consistent with stub syntax, but it avoids some edge cases we hit otherwise - I don't think requiring explicit types is a huge burden, the existing `Callable` type requires it
(2) full stub syntax, with a shorthand syntax for positional-only callables - we would have to require explicit Any annotations: - otherwise, we get collisions between shorthand syntax and implicit Any - for example, is `(a, b) -> int` of type `Callable[[Any, Any], int]` or `Callable[[a, b], int`?
(3) full stub syntax only, with no shorthand - it seems like a majority thinks this is burdensome because of existing Callables - If we want to be able to revisit shorthand syntax later if we have to require explicit Any - otherwise, the ambiguous case described above plus backward compatibility will trap us
If we move forward with options (2) or (3) we'll still have to decide whether to allow _ as the argument name for positional-only args. My preference would be to allow it so that nuisance argnames aren't required, but to require an explicit `/` _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido (mobile)
Steven, when you say "stub syntax", do you mean a syntactic form that would be legal only in stub files and not in ".py" files? I hope that's not the implication because there's no precedent for that. Today the grammar is the same between ".pyi" and ".py" files, and I don't think we should deviate from that. Perhaps by "stub syntax" you mean "the form that will more commonly be used within stub files"? Just wanted to make sure I understood. -Eric -- Eric Traut Contributor to Pyright & Pylance Microsoft Corp.
Guido - I'll make a publicly-sharable google doc to outline my understanding of current proposals. Inline comments will hopefully help with the details. Eric - when I say "stub syntax" I just mean a callable syntax that's mostly compatible with how we write stubs, i.e. - argument names are required, and a \ is required for positional-only arguments - (possibly - I would argue against) types are optional and default to Any - (possibly - I would argue in favor) we allow using _ as the name of all positional-only arguments
Pradeep and I put together a doc outlining the various proposals we are aware of. There's a table of examples comparing each proposal side-by-side, followed by a section with more details and discussion about each proposed syntax: https://docs.google.com/document/d/11VkNk9N8whxS1SrBv2BNN1Csl5yeTlgG7WyLrCB5... Using comments for discussion help
Thanks for writing the document. It's a very helpful step forward. One thing that occurs to me is that all of the syntax proposals create ambiguities when the `->` operator is combined with `|` (union). The order of precedence between `->` and `|` would need to be established and documented, and parentheses would need to be supported to override the normal precedence. Take this example: ```python # Does this mean `Union[Callable[[int], int], str]` or `Callable[[int], Union[int, str]]`? (int) -> int | str # With parentheses, it is unambiguous. ((int) -> int) | str (int) -> (int | str) ``` For what it's worth, TypeScript chose to bind `|` more tightly. If we were to adopt this precedent, `(int) -> int | str` would imply `(int) -> (int | str)`. -Eric
On Sat, Jun 19, 2021 at 8:26 AM Eric Traut <eric@traut.com> wrote:
Thanks for writing the document. It's a very helpful step forward.
Indeed. One thing that occurs to me is that all of the syntax proposals create
ambiguities when the `->` operator is combined with `|` (union). The order of precedence between `->` and `|` would need to be established and documented, and parentheses would need to be supported to override the normal precedence.
Take this example:
```python # Does this mean `Union[Callable[[int], int], str]` or `Callable[[int], Union[int, str]]`? (int) -> int | str
# With parentheses, it is unambiguous. ((int) -> int) | str (int) -> (int | str) ```
For what it's worth, TypeScript chose to bind `|` more tightly. If we were to adopt this precedent, `(int) -> int | str` would imply `(int) -> (int | str)`.
I think we should do the same, because in the one other context where '->' is used ('def f(...) -> ...'), '}' also binds more tightly than '->'. Since this usage is so closely coupled to function types I think we have little choice in the matter, else we'd just confuse users. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
I think that `|` binding more tightly is the better choice here, but I'm having trouble with the behavior in some cases; would bool | (T) -> bool | int parse as bool | Callable[[T], bool | int] ? That makes it seem like `|` has a different precedence in different contexts. Would we require parenthesis here? On Sat, Jun 19, 2021, at 12:29 PM, Guido van Rossum wrote:
On Sat, Jun 19, 2021 at 8:26 AM Eric Traut <eric@traut.com> wrote:
Thanks for writing the document. It's a very helpful step forward.
Indeed.
One thing that occurs to me is that all of the syntax proposals create ambiguities when the `->` operator is combined with `|` (union). The order of precedence between `->` and `|` would need to be established and documented, and parentheses would need to be supported to override the normal precedence.
Take this example:
```python # Does this mean `Union[Callable[[int], int], str]` or `Callable[[int], Union[int, str]]`? (int) -> int | str
# With parentheses, it is unambiguous. ((int) -> int) | str (int) -> (int | str) ```
For what it's worth, TypeScript chose to bind `|` more tightly. If we were to adopt this precedent, `(int) -> int | str` would imply `(int) -> (int | str)`.
I think we should do the same, because in the one other context where '->' is used ('def f(...) -> ...'), '}' also binds more tightly than '->'. Since this usage is so closely coupled to function types I think we have little choice in the matter, else we'd just confuse users.
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Typing-sig mailing list -- typing-sig@python.org <mailto:typing-sig%40python.org> To unsubscribe send an email to typing-sig-leave@python.org <mailto:typing-sig-leave%40python.org> https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: rbt@sent.as <mailto:rbt%40sent.as>
On Sat, Jun 19, 2021 at 11:04 AM Rebecca Turner <rbt@sent.as> wrote:
I think that `|` binding more tightly is the better choice here, but I'm having trouble with the behavior in some cases; would bool | (T) -> bool | int parse as bool | Callable[[T], bool | int] ? That makes it seem like `|` has a different precedence in different contexts. Would we require parenthesis here?
Right, that would be a syntax error, you'd have to write bool | ((T) -> bool | int) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Yes, it sounds like binding | more tightly would be the right choice. I added a "Further Notes" section to the bottom of the doc to keep track of concerns like this that aren't tied to choosing a specific syntax.
I was curious about what kind of callable types are needed in existing code. So, I spent some time analyzing the Callable annotations in typeshed. Hopefully, we can use the data to help drive our discussions. There are three kinds of Callables we might hope to annotate using the new syntax: 1. Callables that are already expressed using `Callable`. These have positional-only parameters. 2. Callables that are used as `Callable[..., None]` or even just `f: Callable`. 3. Callables that have no type right now, either because the author could have added `Callable` types but didn't or because they needed more than positional-only parameters. Stats: [1] 1. 72% of Callables in typeshed have explicit parameter types like `Callable[[int], str]`. (941 callables) 2. 28% have no parameter types, such as `f: Callable` or `f: Callable[..., None]`. (364 callables) # Callables without parameter types How many of the poorly-typed Callables are that way because they need features beyond positional-only parameters (named parameters, default values, `*args: int`)? To answer that, I sampled several of the Callables stubs with `...` parameter type and checked their source code to see how they were actually used. + Most of the callables I looked at used `Callable[..., Any]` because they couldn't express the relation that `*args` and `**kwargs` would be passed into the callback. This is expressible with ParamSpec or TypeVarTuple. For example, the docstring of the below function says: callable [is] invoked with specified positional and keyword arguments. It needs to use ParamSpec: ```python # unittest/case.pyi # typeshed stub (slightly edited) def assertWarns( self, callable: Callable[..., Any], *args: Any, **kwargs: Any, ) -> None: ... # desired stub using ParamSpec def assertWarns( self, callable: Callable[P, Any], *args: P.args, **kwargs: P.kwargs, ) -> None: ... ``` As another example, the docstring of the below function says: Any positional arguments after the callback will be passed to the callback when it is called. It needs to use TypeVarTuple (PEP 646): ``` # asyncio/base_events.pyi has 12 such Callable[..., Any] types # typeshed stub (slightly edited) def call_soon(self, callback: Callable[..., Any], *args: Any) -> Handle: ... # desired stub using PEP 646: def call_soon(self, callback: Callable[[*Ts], Any], *args: *Ts) -> Handle: ... ``` + The remaining callables I saw were too dynamic to capture using types. For example, `multiprocessing/managers.py` had a registry dictionary that mapped each string key to a different Callable. There's no way for us to capture arbitrary dictionaries like that, so it makes sense to use `Dict[str, Callable[..., Any]]`. # Questions In my limited exploration above, I did not find callables that were given `Callable[..., Any]` for the lack of things like named parameters, default values, or `*args: int`. This suggests that the shorthand syntax `(int, bool) -> str` might cover almost all of our use cases. [2] + Do we have real-world examples where a Callable type was given `Callable[..., Any]` because it could not be expressed with positional-only parameters or ParamSpec or TypeVarTuple? + Do we have real-world examples of untyped Python code where a callable is called with named parameters? This is the remaining category (3) where features beyond shorthand syntax might potentially be useful. Given that overloads are unlikely to be represented in our callable syntax, as David pointed out, we can't hope to capture all function types. We will have to draw the line somewhere. Is it worth the extra complexity to include things like named parameters? # Footnotes Curious: What are some large, typed, open-source Python projects (apart from typeshed) that we can analyze for stats? [1]: Script to get type annotations using LibCST: https://github.com/pradeep90/annotation_collector This includes annotations from parameter types, return types, attribute types, etc. Also extracts Callables from within `List[Callable[[int], str]]`, etc. List of callables in typeshed: https://github.com/pradeep90/annotation_collector/blob/master/typeshed-calla... Breakdown of the callables that have explicit parameter types: + 0 parameters: 220 (16%) -- i.e., Callable[[], str] + 1 parameter: 521 (39%) -- i.e., Callable[[int], str] + 2 parameters: 137 (10%) + 3 parameters: 42 (3%) + 4 parameters: 13 (~0%) + 5 parameters: 8 (~0%) [2]: We could alternatively use `(_: int, _: bool) -> str` instead of `(int, bool) -> str` if we want it to be consistent with current syntax, while still restricting it to positional-only parameters. [3]: Fwiw, Pyre does show overloads when revealing Callable types, but it gets *really* clunky and is definitely not something we want: `Callable[[Named(x, bool)], str][[[Named(x, Literal[True])], str][[Named(x, Literal[False])], str]]`
El sáb, 19 jun 2021 a las 20:20, S Pradeep Kumar (<gohanpra@gmail.com>) escribió:
<snip> # Questions
In my limited exploration above, I did not find callables that were given `Callable[..., Any]` for the lack of things like named parameters, default values, or `*args: int`. This suggests that the shorthand syntax `(int, bool) -> str` might cover almost all of our use cases. [2]
+ Do we have real-world examples where a Callable type was given `Callable[..., Any]` because it could not be expressed with positional-only parameters or ParamSpec or TypeVarTuple?
+ Do we have real-world examples of untyped Python code where a callable is called with named parameters? This is the remaining category (3) where features beyond shorthand syntax might potentially be useful.
Given that overloads are unlikely to be represented in our callable syntax, as David pointed out, we can't hope to capture all function types. We will have to draw the line somewhere. Is it worth the extra complexity to include things like named parameters?
Thanks, this is great data to have for discussion! More complicated callable types are currently often expressed using callback protocols. For example, I found this one in `stdlib/tkinter/__init__.pyi`: class _ExceptionReportingCallback(Protocol): def __call__(self, __exc: Type[BaseException], __val: BaseException, __tb: TracebackType) -> Any: ... This could be expressed with Callable but the names provide better documentation. Similarly, this one is in sharedctypes.pyi: class _AcquireFunc(Protocol): def __call__(self, block: bool = ..., timeout: Optional[float] = ...) -> bool: ...
# Footnotes
Curious: What are some large, typed, open-source Python projects (apart from typeshed) that we can analyze for stats?
The mypy-primer corpus ( https://github.com/hauntsaninja/mypy_primer/blob/master/mypy_primer.py#L914) would be a good place to look. I suspect complicated callable types are more common in application codebases than in reusable libraries (the code you see in typeshed). I haven't tested that assumption though.
More complicated callable types are currently often expressed using callback protocols
Good point! Have updated the script and output data for callback protocols. I found 37 callback protocols in typeshed (2.7% of all callables). [1]
The mypy-primer corpus ( https://github.com/hauntsaninja/mypy_primer/blob/master/mypy_primer.py#L914) would be a good place to look.
Thanks, Jelle. I'll take a look at some of those projects. [1]: https://github.com/pradeep90/annotation_collector/blob/master/typeshed-calla... On Sat, Jun 19, 2021 at 8:57 PM Jelle Zijlstra <jelle.zijlstra@gmail.com> wrote:
El sáb, 19 jun 2021 a las 20:20, S Pradeep Kumar (<gohanpra@gmail.com>) escribió:
<snip> # Questions
In my limited exploration above, I did not find callables that were given `Callable[..., Any]` for the lack of things like named parameters, default values, or `*args: int`. This suggests that the shorthand syntax `(int, bool) -> str` might cover almost all of our use cases. [2]
+ Do we have real-world examples where a Callable type was given `Callable[..., Any]` because it could not be expressed with positional-only parameters or ParamSpec or TypeVarTuple?
+ Do we have real-world examples of untyped Python code where a callable is called with named parameters? This is the remaining category (3) where features beyond shorthand syntax might potentially be useful.
Given that overloads are unlikely to be represented in our callable syntax, as David pointed out, we can't hope to capture all function types. We will have to draw the line somewhere. Is it worth the extra complexity to include things like named parameters?
Thanks, this is great data to have for discussion!
More complicated callable types are currently often expressed using callback protocols. For example, I found this one in `stdlib/tkinter/__init__.pyi`:
class _ExceptionReportingCallback(Protocol): def __call__(self, __exc: Type[BaseException], __val: BaseException, __tb: TracebackType) -> Any: ...
This could be expressed with Callable but the names provide better documentation.
Similarly, this one is in sharedctypes.pyi:
class _AcquireFunc(Protocol): def __call__(self, block: bool = ..., timeout: Optional[float] = ...) -> bool: ...
# Footnotes
Curious: What are some large, typed, open-source Python projects (apart from typeshed) that we can analyze for stats?
The mypy-primer corpus ( https://github.com/hauntsaninja/mypy_primer/blob/master/mypy_primer.py#L914) would be a good place to look.
I suspect complicated callable types are more common in application codebases than in reusable libraries (the code you see in typeshed). I haven't tested that assumption though.
-- S Pradeep Kumar
+ Do we have real-world examples where a Callable type was given `Callable[..., Any]` because it could not be expressed with positional-only parameters or ParamSpec or TypeVarTuple?
+ Do we have real-world examples of untyped Python code where a callable is called with named parameters? This is the remaining category (3) where features beyond shorthand syntax might potentially be useful.
To both: **kwargs/dict-splatting is a major use case that can't be expressed with the current or proposed tools. I'm working on some features to encapsulate this use-case with TypedDict but I don't have more than a draft right now. On Sat, Jun 19, 2021, at 11:20 PM, S Pradeep Kumar wrote:
I was curious about what kind of callable types are needed in existing code. So, I spent some time analyzing the Callable annotations in typeshed. Hopefully, we can use the data to help drive our discussions.
There are three kinds of Callables we might hope to annotate using the new syntax:
1. Callables that are already expressed using `Callable`. These have positional-only parameters. 2. Callables that are used as `Callable[..., None]` or even just `f: Callable`. 3. Callables that have no type right now, either because the author could have added `Callable` types but didn't or because they needed more than positional-only parameters.
Stats: [1]
1. 72% of Callables in typeshed have explicit parameter types like `Callable[[int], str]`. (941 callables) 2. 28% have no parameter types, such as `f: Callable` or `f: Callable[..., None]`. (364 callables)
# Callables without parameter types
How many of the poorly-typed Callables are that way because they need features beyond positional-only parameters (named parameters, default values, `*args: int`)?
To answer that, I sampled several of the Callables stubs with `...` parameter type and checked their source code to see how they were actually used.
+ Most of the callables I looked at used `Callable[..., Any]` because they couldn't express the relation that `*args` and `**kwargs` would be passed into the callback.
This is expressible with ParamSpec or TypeVarTuple.
For example, the docstring of the below function says: callable [is] invoked with specified positional and keyword arguments.
It needs to use ParamSpec:
```python # unittest/case.pyi # typeshed stub (slightly edited) def assertWarns( self, callable: Callable[..., Any], *args: Any, **kwargs: Any, ) -> None: ...
# desired stub using ParamSpec def assertWarns( self, callable: Callable[P, Any], *args: P.args, **kwargs: P.kwargs, ) -> None: ... ```
As another example, the docstring of the below function says: Any positional arguments after the callback will be passed to the callback when it is called.
It needs to use TypeVarTuple (PEP 646):
``` # asyncio/base_events.pyi has 12 such Callable[..., Any] types # typeshed stub (slightly edited) def call_soon(self, callback: Callable[..., Any], *args: Any) -> Handle: ...
# desired stub using PEP 646: def call_soon(self, callback: Callable[[*Ts], Any], *args: *Ts) -> Handle: ... ```
+ The remaining callables I saw were too dynamic to capture using types. For example, `multiprocessing/managers.py` had a registry dictionary that mapped each string key to a different Callable. There's no way for us to capture arbitrary dictionaries like that, so it makes sense to use `Dict[str, Callable[..., Any]]`.
# Questions
In my limited exploration above, I did not find callables that were given `Callable[..., Any]` for the lack of things like named parameters, default values, or `*args: int`. This suggests that the shorthand syntax `(int, bool) -> str` might cover almost all of our use cases. [2]
+ Do we have real-world examples where a Callable type was given `Callable[..., Any]` because it could not be expressed with positional-only parameters or ParamSpec or TypeVarTuple?
+ Do we have real-world examples of untyped Python code where a callable is called with named parameters? This is the remaining category (3) where features beyond shorthand syntax might potentially be useful.
Given that overloads are unlikely to be represented in our callable syntax, as David pointed out, we can't hope to capture all function types. We will have to draw the line somewhere. Is it worth the extra complexity to include things like named parameters?
# Footnotes
Curious: What are some large, typed, open-source Python projects (apart from typeshed) that we can analyze for stats?
[1]: Script to get type annotations using LibCST: https://github.com/pradeep90/annotation_collector
This includes annotations from parameter types, return types, attribute types, etc. Also extracts Callables from within `List[Callable[[int], str]]`, etc.
List of callables in typeshed: https://github.com/pradeep90/annotation_collector/blob/master/typeshed-calla...
Breakdown of the callables that have explicit parameter types:
+ 0 parameters: 220 (16%) -- i.e., Callable[[], str] + 1 parameter: 521 (39%) -- i.e., Callable[[int], str] + 2 parameters: 137 (10%) + 3 parameters: 42 (3%) + 4 parameters: 13 (~0%) + 5 parameters: 8 (~0%)
[2]: We could alternatively use `(_: int, _: bool) -> str` instead of `(int, bool) -> str` if we want it to be consistent with current syntax, while still restricting it to positional-only parameters.
[3]: Fwiw, Pyre does show overloads when revealing Callable types, but it gets *really* clunky and is definitely not something we want:
`Callable[[Named(x, bool)], str][[[Named(x, Literal[True])], str][[Named(x, Literal[False])], str]]`
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org <mailto:typing-sig%40python.org> To unsubscribe send an email to typing-sig-leave@python.org <mailto:typing-sig-leave%40python.org> https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: rbt@sent.as <mailto:rbt%40sent.as>
I'm following up on the post with the link to the doc because I just realized that your "hybrid" form is different from what I had imagined. In your hybrid form you write f: (int, int, int=42) -> int which is not what I had in mind when I proposed it (but maybe you didn't get it from my proposal!). The hybrid form above can only specify optional *positional* arguments, whereas I think it would be more useful if you could specify optional *keyword* arguments. So what I had thought of as "hybrid" is not even in your list! I meant to allow f: (int, int, kw: int = 42) -> int which in stub-style syntax would be f: (_0: int, _1: int, /, kw: int = 42) -> int I therefore have a new proposal, let's call it "Guido". :-) In my proposal you can use stub-style syntax but you can optionally prefix that with any number of bare types. The semantics are that if there is no '/' in the stub-style portion, one is inserted between the bare args and the stub-style args, and the bare args are translated into named args by giving them dummy names (or naming them all "_" if that part of the proposal is accepted). Both rules can be seen in action in my little example above. Separately, I am +0 on allowing multiple positional arguments to be named "_", but I don't want to make it a deal-breaker. (I am not worried that this will confuse users more than once.) Finally, I'd like to make sure that we all agree that where we write 'int', 'str' etc. in the examples, an *arbitrary* type should be allowed, e.g. 'list[T]' or 'int | str' or even '(int, int) -> str'. On Fri, Jun 18, 2021 at 9:58 PM Steven Troxler <steven.troxler@gmail.com> wrote:
Pradeep and I put together a doc outlining the various proposals we are aware of.
There's a table of examples comparing each proposal side-by-side, followed by a section with more details and discussion about each proposed syntax:
https://docs.google.com/document/d/11VkNk9N8whxS1SrBv2BNN1Csl5yeTlgG7WyLrCB5...
Using comments for discussion help _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Sorry, it's a little easy to get lost in all the details because there are so many variations listed, but we had indeed intended to support named args with default in the hybrid syntax, using the style you suggest. I added a comment with @Guido to the section of the table where we specified named-argument-with-default syntax. I could add another syntax proposal that allows mixing stub and hybrid format as suggested if you'd like, or we can just keep in mind that this is also on the table. And yes, agreed that arbitrary types should be allowed, I think we have some examples of callable parameters inside callable types as part of the table. - Steven
On Thu, Jun 24, 2021 at 1:49 PM Steven Troxler <steven.troxler@gmail.com> wrote:
Sorry, it's a little easy to get lost in all the details because there are so many variations listed, but we had indeed intended to support named args with default in the hybrid syntax, using the style you suggest.
But did you also intend hybrid syntax to allow (int, int, int=42) -> int? That would seems super-confusing.
I added a comment with @Guido to the section of the table where we specified named-argument-with-default syntax.
Got it.
I could add another syntax proposal that allows mixing stub and hybrid format as suggested if you'd like, or we can just keep in mind that this is also on the table.
I wouldn't characterize my proposal as such. It mixes stub and shorthand. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
We did intend to allow ``` (int, str = ...) -> float ``` in hybrid syntax, because positional-only arguments *must* be nameless in hybrid as proposed. ## Mixed Syntax I added another option "mixed" to the doc I tried to capture my understanding of your suggestions: - allows nameless positional-only arguments as long as they don't have default values - optionally lets users specify names as in stub-style, but in that case a `/` is required - forces users to specify a name (possibly just _) for a positional-only argument with default So in mixed (assuming we allow `_`), the signature above could be written as either ``` (int, _: str = ..., /) -> float ``` or ``` (_: int, _: str = ..., /) -> float ``` I think I like the mixed idea. It has most of the benefits of hybrid syntax: - simple callables are compact - it has a canonical representation without nuisance parameters we can use in type errors - we can mostly avoid `/` (my guess is that positional-with-default args are pretty rare) There are multiple options for how users can represent a type, which is the big drawback. But allowing folks to name their positional arguments means they can make callback types much more self-documenting, since a name plus a type can express what a callback is supposed to do much better than a type signature alone. Obviously library authors tend to put this information in docstrings, but having it as part of the callback type would mean the argument names can be available in type error messages and IDE hovers.
On Fri, Jun 25, 2021 at 10:16 AM Steven Troxler <steven.troxler@gmail.com> wrote:
We did intend to allow ``` (int, str = ...) -> float ``` in hybrid syntax, because positional-only arguments *must* be nameless in hybrid as proposed.
No wonder you're not in favor of hybrid syntax. :-)
## Mixed Syntax
I added another option "mixed" to the doc
I tried to capture my understanding of your suggestions: - allows nameless positional-only arguments as long as they don't have default values - optionally lets users specify names as in stub-style, but in that case a `/` is required - forces users to specify a name (possibly just _) for a positional-only argument with default
So in mixed (assuming we allow `_`), the signature above could be written as either ``` (int, _: str = ..., /) -> float ``` or ``` (_: int, _: str = ..., /) -> float ```
I think I like the mixed idea. It has most of the benefits of hybrid syntax: - simple callables are compact - it has a canonical representation without nuisance parameters we can use in type errors - we can mostly avoid `/` (my guess is that positional-with-default args are pretty rare)
It's not that rare, except people typically don't use the / unless they are feeling pedantic. I suspect that many functions exist like this: ``` def func(x, y, z=0): ... ``` where the user's intention is really ``` def func(x, y, z=0, /): ... ``` but their code needs to be compatible with Python 3.7, or perhaps it was written when 3.7 (or earlier) was current and the code wasn't updated to add the /.
There are multiple options for how users can represent a type, which is the big drawback.
Is it though? It doesn't strike me as a big problem. Yes, the Zen of Python says TOOWTDI, but you have to take that with a grain of salt. I also like that there's a nice simple upgrade path for code that currently uses Callable[]: once it can drop support for 3.10 and earlier, Callable[[T, S], R] becomes (T, S) -> R, which is much cleaner than (_: T, _: S, /) -> R. I suspect many, many use cases will be satisfied with this.
But allowing folks to name their positional arguments means they can make callback types much more self-documenting, since a name plus a type can express what a callback is supposed to do much better than a type signature alone.
Agreed, for new code this is a nice option. Then again I suspect there are plenty of Callable types with a single argument whose meaning is obvious from the name of the callback. And it's not possible to insert names when mechanically replacing Callable[] with the new notation.
Obviously library authors tend to put this information in docstrings, but having it as part of the callback type would mean the argument names can be available in type error messages and IDE hovers.
Yes, that would be nice. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Do we think it might be worth starting with the original plan of a PEP that just introduces shorthand syntax (as proposed at the start of this thread)? That would allow us to defer a lot of discussion until later: - How to add support for named / default / variadic arguments, if at all - Whether it's more valuable to support param specs (e.g. for decorators) than ^ - Pradeep's initial analysis of existing packages suggests this may be the case I think the main thing we have to decide is whether we're aligned on any future changes being backward-compatible with shorthand, which rules out strict stub-style syntax with implicit Any but leaves all other options on the table.
El lun, 28 jun 2021 a las 13:08, Steven Troxler (<steven.troxler@gmail.com>) escribió:
Do we think it might be worth starting with the original plan of a PEP that just introduces shorthand syntax (as proposed at the start of this thread)?
I'm not sure. I feel like the biggest hurdle is going to be convincing the SC to add a new piece of syntax that is only useful for typing. In my view, either approach has risks. If we only add the shorthand syntax, perhaps the feature isn't compelling enough because it doesn't unlock new capabilities. If we do add the full syntax, there is more opportunity for bikeshedding to derail the discussion.
That would allow us to defer a lot of discussion until later: - How to add support for named / default / variadic arguments, if at all - Whether it's more valuable to support param specs (e.g. for decorators) than ^ - Pradeep's initial analysis of existing packages suggests this may be the case
I think the main thing we have to decide is whether we're aligned on any future changes being backward-compatible with shorthand, which rules out strict stub-style syntax with implicit Any but leaves all other options on the table
_______________________________________________
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: jelle.zijlstra@gmail.com
At this point I feel that the choices are sufficiently clear-cut that we should just forge ahead and settle on a full proposal. And I agree with Jelle that the key point is introducing new syntax for callable types at all. The main point of contention may well be whether the "(...) -> ..." syntax shouldn't be a shorthand for lambda instead. On Mon, Jun 28, 2021 at 1:08 PM Steven Troxler <steven.troxler@gmail.com> wrote:
Do we think it might be worth starting with the original plan of a PEP that just introduces shorthand syntax (as proposed at the start of this thread)?
That would allow us to defer a lot of discussion until later: - How to add support for named / default / variadic arguments, if at all - Whether it's more valuable to support param specs (e.g. for decorators) than ^ - Pradeep's initial analysis of existing packages suggests this may be the case
I think the main thing we have to decide is whether we're aligned on any future changes being backward-compatible with shorthand, which rules out strict stub-style syntax with implicit Any but leaves all other options on the table. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
# How frequently does *untyped* Python code use complex callbacks? I'd shared stats earlier about Callable usage in typed Python code [1]. By looking at the source code for stubbed files, we'd seen that practically all of the Callable uses in typed Python projects were either (a) expressible using positional-only parameters, (b) expressible using ParamSpec or TypeVarTuple, or (c) simply too dynamic. The remaining possible source of callables is *untyped* Python code. Perhaps such code has a lot of callbacks that requires named parameters, default values, etc. Untyped Python is not constrained by our Callable type syntax, so if we are going to find complex uses anywhere, it will be in untyped Python. I tested the above claim by looking at a few large, untyped, open-source Python projects (Django with 80k+ LOC of Python, Sentry with 150k+ LOC of Python). Findings [2]: + About half of the callbacks were called with just positional arguments: `func(a, b)`. + More than a third of callbacks were called with `*args, **kwargs`: `func(*args, **kwargs)` or `func(prefix, *args, **kwargs)`. These would need ParamSpec. + The remaining ~13% were called with only `*args` or were too dynamic. This is in line with the pattern I saw for typed Python projects [3]: 90+% of callbacks were either positional-only or ParamSpec. # How often are named arguments used for callbacks? Very rarely. I saw a couple of examples and they were basically all cases where the "callback" was a class, not a function: ``` def _generate_altered_foo_together(self, operation): x = operation( name=model_name, **{option_name: new_value} ) self._generate_altered_foo_together(operations.AlterUniqueTogether) self._generate_altered_foo_together(operations.AlterIndexTogether) ``` In other words, the type would not be `Callable`. It would be `Type[ModelOperation]`: ``` def _generate_altered_foo_together(self, operation: Type[ModelOperation]): ``` # Conclusion Based on the above stats, I feel confident that the vast majority of cases can be handled by the shorthand syntax proposal: + positional-only parameters: `(int, str) -> bool` instead of Callable[[int, str], bool] + ParamSpec: `(**P) -> R` instead of Callable[P, R] Given how frequently ParamSpec is needed (20-30% of all callbacks), the simpler syntax above would lower the barrier to better types and increase type safety. More details are in the syntax proposal doc under the name "shorthand syntax". [4] Finally, in order to support any remaining edge cases, we could adopt Łukasz's proposal about using the name of a function as a type (like we do with classes). This would be easier to use than a callback protocol. At this point, if we want to add extra syntax for things like named parameters, default values, etc., I think we should justify the extra complexity using real-world cases. If not, I don't feel it's worth adding them and going through a ton of bikeshedding for something that is not going to add much value to the above. None of the syntax proposals handle overloads, so no syntax will capture all the cases that a callback protocol can. We have to draw the line somewhere; I suggest we draw it at the shorthand proposal. # Footnotes [1]: Earlier mail with stats for typeshed: https://mail.python.org/archives/list/typing-sig@python.org/message/TTPVR7QU... [2]: Summary of stats for untyped projects: https://github.com/pradeep90/annotation_collector#projects-with-no-types List of callbacks in Django: https://github.com/pradeep90/annotation_collector/blob/master/data/django-ca... I updated the libcst script to collect functions that called one of their parameters. This helped automatically analyze untyped code. I cross-checked that by sampling a significant number of functions (~400) from each project and manually checking how callbacks were used. I also looked for functions and parameters that had suggestive names like `func` or `callback` and checked how they were used. [3]: Stats for many large, typed Python projects: https://github.com/pradeep90/annotation_collector#stats [4]: Callable syntax proposals document: https://docs.google.com/document/d/11VkNk9N8whxS1SrBv2BNN1Csl5yeTlgG7WyLrCB5... On Mon, Jun 28, 2021 at 1:30 PM Guido van Rossum <guido@python.org> wrote:
At this point I feel that the choices are sufficiently clear-cut that we should just forge ahead and settle on a full proposal. And I agree with Jelle that the key point is introducing new syntax for callable types at all. The main point of contention may well be whether the "(...) -> ..." syntax shouldn't be a shorthand for lambda instead.
On Mon, Jun 28, 2021 at 1:08 PM Steven Troxler <steven.troxler@gmail.com> wrote:
Do we think it might be worth starting with the original plan of a PEP that just introduces shorthand syntax (as proposed at the start of this thread)?
That would allow us to defer a lot of discussion until later: - How to add support for named / default / variadic arguments, if at all - Whether it's more valuable to support param specs (e.g. for decorators) than ^ - Pradeep's initial analysis of existing packages suggests this may be the case
I think the main thing we have to decide is whether we're aligned on any future changes being backward-compatible with shorthand, which rules out strict stub-style syntax with implicit Any but leaves all other options on the table. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gohanpra@gmail.com
-- S Pradeep Kumar
Another quick datapoint: At the moment, typeshed has 16 callable protocols in the stdlib alone that don't use positional only arguments. Many of these protocols have arguments with default values. These protocols can't be replaced using the shorthand syntax, but would need a more comprehensive syntax. - Sebastian Am 22.07.21 um 05:51 schrieb S Pradeep Kumar:
# How frequently does *untyped* Python code use complex callbacks?
I'd shared stats earlier about Callable usage in typed Python code [1]. By looking at the source code for stubbed files, we'd seen that practically all of the Callable uses in typed Python projects were either (a) expressible using positional-only parameters, (b) expressible using ParamSpec or TypeVarTuple, or (c) simply too dynamic.
The remaining possible source of callables is *untyped* Python code. Perhaps such code has a lot of callbacks that requires named parameters, default values, etc. Untyped Python is not constrained by our Callable type syntax, so if we are going to find complex uses anywhere, it will be in untyped Python.
I tested the above claim by looking at a few large, untyped, open-source Python projects (Django with 80k+ LOC of Python, Sentry with 150k+ LOC of Python).
Findings [2]:
+ About half of the callbacks were called with just positional arguments: `func(a, b)`. + More than a third of callbacks were called with `*args, **kwargs`: `func(*args, **kwargs)` or `func(prefix, *args, **kwargs)`. These would need ParamSpec. + The remaining ~13% were called with only `*args` or were too dynamic.
This is in line with the pattern I saw for typed Python projects [3]: 90+% of callbacks were either positional-only or ParamSpec.
# How often are named arguments used for callbacks?
Very rarely. I saw a couple of examples and they were basically all cases where the "callback" was a class, not a function:
``` def _generate_altered_foo_together(self, operation): x = operation( name=model_name, **{option_name: new_value} )
self._generate_altered_foo_together(operations.AlterUniqueTogether) self._generate_altered_foo_together(operations.AlterIndexTogether) ```
In other words, the type would not be `Callable`. It would be `Type[ModelOperation]`:
``` def _generate_altered_foo_together(self, operation: Type[ModelOperation]): ```
# Conclusion
Based on the above stats, I feel confident that the vast majority of cases can be handled by the shorthand syntax proposal:
+ positional-only parameters: `(int, str) -> bool` instead of Callable[[int, str], bool] + ParamSpec: `(**P) -> R` instead of Callable[P, R]
Given how frequently ParamSpec is needed (20-30% of all callbacks), the simpler syntax above would lower the barrier to better types and increase type safety. More details are in the syntax proposal doc under the name "shorthand syntax". [4]
Finally, in order to support any remaining edge cases, we could adopt Łukasz's proposal about using the name of a function as a type (like we do with classes). This would be easier to use than a callback protocol.
At this point, if we want to add extra syntax for things like named parameters, default values, etc., I think we should justify the extra complexity using real-world cases. If not, I don't feel it's worth adding them and going through a ton of bikeshedding for something that is not going to add much value to the above. None of the syntax proposals handle overloads, so no syntax will capture all the cases that a callback protocol can. We have to draw the line somewhere; I suggest we draw it at the shorthand proposal.
# Footnotes
[1]: Earlier mail with stats for typeshed: https://mail.python.org/archives/list/typing-sig@python.org/message/TTPVR7QU... <https://mail.python.org/archives/list/typing-sig@python.org/message/TTPVR7QUK6MGSYRYMHHR4O23OZG6B6VP/>
[2]: Summary of stats for untyped projects: https://github.com/pradeep90/annotation_collector#projects-with-no-types <https://github.com/pradeep90/annotation_collector#projects-with-no-types>
List of callbacks in Django: https://github.com/pradeep90/annotation_collector/blob/master/data/django-ca... <https://github.com/pradeep90/annotation_collector/blob/master/data/django-callback-parameters.txt>
I updated the libcst script to collect functions that called one of their parameters. This helped automatically analyze untyped code. I cross-checked that by sampling a significant number of functions (~400) from each project and manually checking how callbacks were used. I also looked for functions and parameters that had suggestive names like `func` or `callback` and checked how they were used.
[3]: Stats for many large, typed Python projects: https://github.com/pradeep90/annotation_collector#stats <https://github.com/pradeep90/annotation_collector#stats>
[4]: Callable syntax proposals document: https://docs.google.com/document/d/11VkNk9N8whxS1SrBv2BNN1Csl5yeTlgG7WyLrCB5... <https://docs.google.com/document/d/11VkNk9N8whxS1SrBv2BNN1Csl5yeTlgG7WyLrCB5wgs/edit#heading=h.s6u1i3i5ct8r>
On Mon, Jun 28, 2021 at 1:30 PM Guido van Rossum <guido@python.org <mailto:guido@python.org>> wrote:
At this point I feel that the choices are sufficiently clear-cut that we should just forge ahead and settle on a full proposal. And I agree with Jelle that the key point is introducing new syntax for callable types at all. The main point of contention may well be whether the "(...) -> ..." syntax shouldn't be a shorthand for lambda instead.
On Mon, Jun 28, 2021 at 1:08 PM Steven Troxler <steven.troxler@gmail.com <mailto:steven.troxler@gmail.com>> wrote:
Do we think it might be worth starting with the original plan of a PEP that just introduces shorthand syntax (as proposed at the start of this thread)?
That would allow us to defer a lot of discussion until later: - How to add support for named / default / variadic arguments, if at all - Whether it's more valuable to support param specs (e.g. for decorators) than ^ - Pradeep's initial analysis of existing packages suggests this may be the case
I think the main thing we have to decide is whether we're aligned on any future changes being backward-compatible with shorthand, which rules out strict stub-style syntax with implicit Any but leaves all other options on the table. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org <mailto:typing-sig@python.org> To unsubscribe send an email to typing-sig-leave@python.org <mailto:typing-sig-leave@python.org> https://mail.python.org/mailman3/lists/typing-sig.python.org/ <https://mail.python.org/mailman3/lists/typing-sig.python.org/> Member address: guido@python.org <mailto:guido@python.org>
-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) /Pronouns: he/him //(why is my pronoun here?)/ <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Typing-sig mailing list -- typing-sig@python.org <mailto:typing-sig@python.org> To unsubscribe send an email to typing-sig-leave@python.org <mailto:typing-sig-leave@python.org> https://mail.python.org/mailman3/lists/typing-sig.python.org/ <https://mail.python.org/mailman3/lists/typing-sig.python.org/> Member address: gohanpra@gmail.com <mailto:gohanpra@gmail.com>
-- S Pradeep Kumar
_______________________________________________ Typing-sig mailing list --typing-sig@python.org To unsubscribe send an email totyping-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address:srittau@rittau.biz
Another quick datapoint: At the moment, typeshed has 16 callable protocols in the stdlib alone that don't use positional only arguments. Many of these protocols have arguments with default values.
Sebastian: Yes, I'd shared callback protocol usage in an earlier mail: + projects with detailed types (like mypy, typeshed, spark) have callback protocols making up 2.3% of all callables. [1] + other projects with loosely-typed callables (sphinx, jax, etc.) don't use any callback protocols. [2] The remaining question was whether there were a lot more uses of non-positional callbacks in code that is not yet typed, perhaps due to the lack of easy syntax. That is why I also looked at untyped Python projects.
These protocols can't be replaced using the shorthand syntax, but would need a more comprehensive syntax.
I agree that they aren't handled by the shorthand syntax. The question is whether we want the burden of the more comprehensive syntax to handle such a small minority of cases. Given that these are very rare, in both typed and untyped projects, and that Łukasz's proposal will give us easy syntax for those cases, I feel we shouldn't add the extra complexity. But in any case, I hope the stats are useful for moving the discussion forward :) [1]: https://github.com/pradeep90/annotation_collector#projects-with-well-typed-c... You can see the 37 callback protocols in typeshed at https://github.com/pradeep90/annotation_collector/blob/master/typeshed-calla.... These would all be easily handled by Łukasz's proposal of using function names as types. [2]: https://github.com/pradeep90/annotation_collector#projects-with-loosely-type... On Thu, Jul 22, 2021 at 3:47 AM Sebastian Rittau <srittau@rittau.biz> wrote:
Another quick datapoint: At the moment, typeshed has 16 callable protocols in the stdlib alone that don't use positional only arguments. Many of these protocols have arguments with default values. These protocols can't be replaced using the shorthand syntax, but would need a more comprehensive syntax.
- Sebastian
Am 22.07.21 um 05:51 schrieb S Pradeep Kumar:
# How frequently does *untyped* Python code use complex callbacks?
I'd shared stats earlier about Callable usage in typed Python code [1]. By looking at the source code for stubbed files, we'd seen that practically all of the Callable uses in typed Python projects were either (a) expressible using positional-only parameters, (b) expressible using ParamSpec or TypeVarTuple, or (c) simply too dynamic.
The remaining possible source of callables is *untyped* Python code. Perhaps such code has a lot of callbacks that requires named parameters, default values, etc. Untyped Python is not constrained by our Callable type syntax, so if we are going to find complex uses anywhere, it will be in untyped Python.
I tested the above claim by looking at a few large, untyped, open-source Python projects (Django with 80k+ LOC of Python, Sentry with 150k+ LOC of Python).
Findings [2]:
+ About half of the callbacks were called with just positional arguments: `func(a, b)`. + More than a third of callbacks were called with `*args, **kwargs`: `func(*args, **kwargs)` or `func(prefix, *args, **kwargs)`. These would need ParamSpec. + The remaining ~13% were called with only `*args` or were too dynamic.
This is in line with the pattern I saw for typed Python projects [3]: 90+% of callbacks were either positional-only or ParamSpec.
# How often are named arguments used for callbacks?
Very rarely. I saw a couple of examples and they were basically all cases where the "callback" was a class, not a function:
``` def _generate_altered_foo_together(self, operation): x = operation( name=model_name, **{option_name: new_value} )
self._generate_altered_foo_together(operations.AlterUniqueTogether) self._generate_altered_foo_together(operations.AlterIndexTogether) ```
In other words, the type would not be `Callable`. It would be `Type[ModelOperation]`:
``` def _generate_altered_foo_together(self, operation: Type[ModelOperation]): ```
# Conclusion
Based on the above stats, I feel confident that the vast majority of cases can be handled by the shorthand syntax proposal:
+ positional-only parameters: `(int, str) -> bool` instead of Callable[[int, str], bool] + ParamSpec: `(**P) -> R` instead of Callable[P, R]
Given how frequently ParamSpec is needed (20-30% of all callbacks), the simpler syntax above would lower the barrier to better types and increase type safety. More details are in the syntax proposal doc under the name "shorthand syntax". [4]
Finally, in order to support any remaining edge cases, we could adopt Łukasz's proposal about using the name of a function as a type (like we do with classes). This would be easier to use than a callback protocol.
At this point, if we want to add extra syntax for things like named parameters, default values, etc., I think we should justify the extra complexity using real-world cases. If not, I don't feel it's worth adding them and going through a ton of bikeshedding for something that is not going to add much value to the above. None of the syntax proposals handle overloads, so no syntax will capture all the cases that a callback protocol can. We have to draw the line somewhere; I suggest we draw it at the shorthand proposal.
# Footnotes
[1]: Earlier mail with stats for typeshed: https://mail.python.org/archives/list/typing-sig@python.org/message/TTPVR7QU...
[2]: Summary of stats for untyped projects: https://github.com/pradeep90/annotation_collector#projects-with-no-types
List of callbacks in Django: https://github.com/pradeep90/annotation_collector/blob/master/data/django-ca...
I updated the libcst script to collect functions that called one of their parameters. This helped automatically analyze untyped code. I cross-checked that by sampling a significant number of functions (~400) from each project and manually checking how callbacks were used. I also looked for functions and parameters that had suggestive names like `func` or `callback` and checked how they were used.
[3]: Stats for many large, typed Python projects: https://github.com/pradeep90/annotation_collector#stats
[4]: Callable syntax proposals document: https://docs.google.com/document/d/11VkNk9N8whxS1SrBv2BNN1Csl5yeTlgG7WyLrCB5...
On Mon, Jun 28, 2021 at 1:30 PM Guido van Rossum <guido@python.org> wrote:
At this point I feel that the choices are sufficiently clear-cut that we should just forge ahead and settle on a full proposal. And I agree with Jelle that the key point is introducing new syntax for callable types at all. The main point of contention may well be whether the "(...) -> ..." syntax shouldn't be a shorthand for lambda instead.
On Mon, Jun 28, 2021 at 1:08 PM Steven Troxler <steven.troxler@gmail.com> wrote:
Do we think it might be worth starting with the original plan of a PEP that just introduces shorthand syntax (as proposed at the start of this thread)?
That would allow us to defer a lot of discussion until later: - How to add support for named / default / variadic arguments, if at all - Whether it's more valuable to support param specs (e.g. for decorators) than ^ - Pradeep's initial analysis of existing packages suggests this may be the case
I think the main thing we have to decide is whether we're aligned on any future changes being backward-compatible with shorthand, which rules out strict stub-style syntax with implicit Any but leaves all other options on the table. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gohanpra@gmail.com
-- S Pradeep Kumar
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.orghttps://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: srittau@rittau.biz
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gohanpra@gmail.com
-- S Pradeep Kumar
Adding onto this, i think something similar to `async () -> ...` would be nice with this, currently type hinting something as a coroutine function can be annoying as it requires making the functions return a `Coroutine[Any, Any, T]` which can make the signature longer than needed.
That's not a bad idea, especially given that async function *definitions* (as opposed to types) have their own syntax as well ('async def'). On Sat, Aug 14, 2021 at 7:34 PM Zomatree <angelokontaxis@hotmail.com> wrote:
Adding onto this, i think something similar to `async () -> ...` would be nice with this, currently type hinting something as a coroutine function can be annoying as it requires making the functions return a `Coroutine[Any, Any, T]` which can make the signature longer than needed. _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
participants (10)
-
Eric Traut
-
Guido van Rossum
-
Jelle Zijlstra
-
Paul Bryan
-
Rebecca Turner
-
S Pradeep Kumar
-
Sebastian Rittau
-
Shannon Zhu
-
Steven Troxler
-
Zomatree