
+ Do we have real-world examples where a Callable type was given `Callable[..., Any]` because it could not be expressed with positional-only parameters or ParamSpec or TypeVarTuple?
+ Do we have real-world examples of untyped Python code where a callable is called with named parameters? This is the remaining category (3) where features beyond shorthand syntax might potentially be useful.
To both: **kwargs/dict-splatting is a major use case that can't be expressed with the current or proposed tools. I'm working on some features to encapsulate this use-case with TypedDict but I don't have more than a draft right now. On Sat, Jun 19, 2021, at 11:20 PM, S Pradeep Kumar wrote:
I was curious about what kind of callable types are needed in existing code. So, I spent some time analyzing the Callable annotations in typeshed. Hopefully, we can use the data to help drive our discussions.
There are three kinds of Callables we might hope to annotate using the new syntax:
1. Callables that are already expressed using `Callable`. These have positional-only parameters. 2. Callables that are used as `Callable[..., None]` or even just `f: Callable`. 3. Callables that have no type right now, either because the author could have added `Callable` types but didn't or because they needed more than positional-only parameters.
Stats: [1]
1. 72% of Callables in typeshed have explicit parameter types like `Callable[[int], str]`. (941 callables) 2. 28% have no parameter types, such as `f: Callable` or `f: Callable[..., None]`. (364 callables)
# Callables without parameter types
How many of the poorly-typed Callables are that way because they need features beyond positional-only parameters (named parameters, default values, `*args: int`)?
To answer that, I sampled several of the Callables stubs with `...` parameter type and checked their source code to see how they were actually used.
+ Most of the callables I looked at used `Callable[..., Any]` because they couldn't express the relation that `*args` and `**kwargs` would be passed into the callback.
This is expressible with ParamSpec or TypeVarTuple.
For example, the docstring of the below function says: callable [is] invoked with specified positional and keyword arguments.
It needs to use ParamSpec:
```python # unittest/case.pyi # typeshed stub (slightly edited) def assertWarns( self, callable: Callable[..., Any], *args: Any, **kwargs: Any, ) -> None: ...
# desired stub using ParamSpec def assertWarns( self, callable: Callable[P, Any], *args: P.args, **kwargs: P.kwargs, ) -> None: ... ```
As another example, the docstring of the below function says: Any positional arguments after the callback will be passed to the callback when it is called.
It needs to use TypeVarTuple (PEP 646):
``` # asyncio/base_events.pyi has 12 such Callable[..., Any] types # typeshed stub (slightly edited) def call_soon(self, callback: Callable[..., Any], *args: Any) -> Handle: ...
# desired stub using PEP 646: def call_soon(self, callback: Callable[[*Ts], Any], *args: *Ts) -> Handle: ... ```
+ The remaining callables I saw were too dynamic to capture using types. For example, `multiprocessing/managers.py` had a registry dictionary that mapped each string key to a different Callable. There's no way for us to capture arbitrary dictionaries like that, so it makes sense to use `Dict[str, Callable[..., Any]]`.
# Questions
In my limited exploration above, I did not find callables that were given `Callable[..., Any]` for the lack of things like named parameters, default values, or `*args: int`. This suggests that the shorthand syntax `(int, bool) -> str` might cover almost all of our use cases. [2]
+ Do we have real-world examples where a Callable type was given `Callable[..., Any]` because it could not be expressed with positional-only parameters or ParamSpec or TypeVarTuple?
+ Do we have real-world examples of untyped Python code where a callable is called with named parameters? This is the remaining category (3) where features beyond shorthand syntax might potentially be useful.
Given that overloads are unlikely to be represented in our callable syntax, as David pointed out, we can't hope to capture all function types. We will have to draw the line somewhere. Is it worth the extra complexity to include things like named parameters?
# Footnotes
Curious: What are some large, typed, open-source Python projects (apart from typeshed) that we can analyze for stats?
[1]: Script to get type annotations using LibCST: https://github.com/pradeep90/annotation_collector
This includes annotations from parameter types, return types, attribute types, etc. Also extracts Callables from within `List[Callable[[int], str]]`, etc.
List of callables in typeshed: https://github.com/pradeep90/annotation_collector/blob/master/typeshed-calla...
Breakdown of the callables that have explicit parameter types:
+ 0 parameters: 220 (16%) -- i.e., Callable[[], str] + 1 parameter: 521 (39%) -- i.e., Callable[[int], str] + 2 parameters: 137 (10%) + 3 parameters: 42 (3%) + 4 parameters: 13 (~0%) + 5 parameters: 8 (~0%)
[2]: We could alternatively use `(_: int, _: bool) -> str` instead of `(int, bool) -> str` if we want it to be consistent with current syntax, while still restricting it to positional-only parameters.
[3]: Fwiw, Pyre does show overloads when revealing Callable types, but it gets *really* clunky and is definitely not something we want:
`Callable[[Named(x, bool)], str][[[Named(x, Literal[True])], str][[Named(x, Literal[False])], str]]`
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org <mailto:typing-sig%40python.org> To unsubscribe send an email to typing-sig-leave@python.org <mailto:typing-sig-leave%40python.org> https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: rbt@sent.as <mailto:rbt%40sent.as>