@overloads and arguments without defaults following arguments with defaults
When working with overloads, it would often make sense if an argument without default could follow an argument with defaults. Take this simplified example: @overload def foo(arg1: str = ..., arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., arg2: Literal[False] = ...) -> bytes: ... This is invalid syntax. Currently, the workaround is fairly cumbersome, redundant, and hard to read (especially in non-toy examples): @overload def foo(arg1: str, arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., *, arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., arg2: Literal[False] = ...) -> bytes: ... Would it make sense to relax Python's syntax to allow this case? And what would that mean for non-overloaded functions using this syntax? - Sebastian
+1 for this being a potentially useful future. We've run into this when trying to write type stubs for certain TensorFlow operations in https://github.com/deepmind/tensor_annotations - as a concrete example: ```python # reduce_sum has a signature (input_tensor, axis=..., keepdims=..., name=...) # But when keepdims=True, because the result is always the same rank # as `input_tensor`, we don't care about the `axis` argument: @overload def reduce_sum(input_tensor: Tensor2[A1, A2], axis=..., keepdims: Literal[True], name=...) -> Tensor2[A1, A2]: ... @overload def reduce_sum(input_tensor: Tensor3[A1, A2, A3], axis=..., keepdims: Literal[True], name=...) -> Tensor3[A1, A2, A3]: ... ``` On Sat, 23 Jan 2021 at 14:15, Sebastian Rittau <srittau@rittau.biz> wrote:
When working with overloads, it would often make sense if an argument without default could follow an argument with defaults. Take this simplified example:
@overload def foo(arg1: str = ..., arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., arg2: Literal[False] = ...) -> bytes: ...
This is invalid syntax. Currently, the workaround is fairly cumbersome, redundant, and hard to read (especially in non-toy examples):
@overload def foo(arg1: str, arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., *, arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., arg2: Literal[False] = ...) -> bytes: ...
Would it make sense to relax Python's syntax to allow this case? And what would that mean for non-overloaded functions using this syntax?
- Sebastian _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
I think this is reasonable, even outside annotations. Someone should start a discussion on python-dev and draft a PEP. —Guido On Mon, Jan 25, 2021 at 02:50 Matthew Rahtz via Typing-sig < typing-sig@python.org> wrote:
+1 for this being a potentially useful future. We've run into this when trying to write type stubs for certain TensorFlow operations in https://github.com/deepmind/tensor_annotations - as a concrete example:
```python # reduce_sum has a signature (input_tensor, axis=..., keepdims=..., name=...) # But when keepdims=True, because the result is always the same rank # as `input_tensor`, we don't care about the `axis` argument:
@overload def reduce_sum(input_tensor: Tensor2[A1, A2], axis=..., keepdims: Literal[True], name=...) -> Tensor2[A1, A2]: ...
@overload def reduce_sum(input_tensor: Tensor3[A1, A2, A3], axis=..., keepdims: Literal[True], name=...) -> Tensor3[A1, A2, A3]: ... ```
On Sat, 23 Jan 2021 at 14:15, Sebastian Rittau <srittau@rittau.biz> wrote:
When working with overloads, it would often make sense if an argument without default could follow an argument with defaults. Take this simplified example:
@overload def foo(arg1: str = ..., arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., arg2: Literal[False] = ...) -> bytes: ...
This is invalid syntax. Currently, the workaround is fairly cumbersome, redundant, and hard to read (especially in non-toy examples):
@overload def foo(arg1: str, arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., *, arg2: Literal[True]) -> str: ... @overload def foo(arg1: str = ..., arg2: Literal[False] = ...) -> bytes: ...
Would it make sense to relax Python's syntax to allow this case? And what would that mean for non-overloaded functions using this syntax?
- Sebastian _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: guido@python.org
-- --Guido (mobile)
I'd strongly advise against this. It would seem to lead to ambiguities when matching arguments with parameters. For example: ```python @overload def foo(arg1: bool=..., arg2: Literal[True]) -> str: ... foo(True) # Does this match the overload? ``` Would the rule be "if there's any possible way to construe an argument list as matching an overload, it should be matched"? The code for matching arguments with parameters is already extremely complicated in a type checker because it needs to deal with *args parameters, **kwargs parameters, position-only separators, keyword-only separators, unpack operators on args, TypeVar assignments, ParamSpec parameter assignments (P.args, P.kwargs), and *args with variadic type variables. More complexity in this code is not needed! -- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.
Am 25.01.21 um 17:48 schrieb Eric Traut:
I'd strongly advise against this.
It would seem to lead to ambiguities when matching arguments with parameters. For example:
```python @overload def foo(arg1: bool=..., arg2: Literal[True]) -> str: ...
foo(True) # Does this match the overload? ```
Would the rule be "if there's any possible way to construe an argument list as matching an overload, it should be matched"?
This would follow normal Python rules. In the example above, it doesn't match as calling this would be a runtime error: ```python def foo(arg1: bool=..., arg2: Literal[True]) -> str: ... foo(True) ```
The code for matching arguments with parameters is already extremely complicated in a type checker because it needs to deal with *args parameters, **kwargs parameters, position-only separators, keyword-only separators, unpack operators on args, TypeVar assignments, ParamSpec parameter assignments (P.args, P.kwargs), and *args with variadic type variables. More complexity in this code is not needed!
This seems to be a trade-off between type checker complexity and user code/stubs complexity. I am obviously biased here, but simpler user code and stubs seems more desirable to me than simpler type checkers as it scales far worse. - Sebastian
Thanks for the clarification. If argument/parameter matching follows normal Python rules, then my concern is lessened. Do you happen to know where in the Python parser and/or runtime this limitation is enforced today? In pyright, the check for parameters without defaults after parameters with defaults is done in the parser. This check would need to be moved out of the parser to a later stage because it would need to be conditionalized based on the presence of the @overload decorator. That's doable, but I'm wondering if that same work will need to be done in all type checkers and the Python parser and runtime as well. -- Eric Traut Contributor to Pyright and Pylance Microsoft Corp.
On Mon, Jan 25, 2021 at 11:15 AM Eric Traut <eric@traut.com> wrote:
Thanks for the clarification. If argument/parameter matching follows normal Python rules, then my concern is lessened.
Do you happen to know where in the Python parser and/or runtime this limitation is enforced today? In pyright, the check for parameters without defaults after parameters with defaults is done in the parser. This check would need to be moved out of the parser to a later stage because it would need to be conditionalized based on the presence of the @overload decorator. That's doable, but I'm wondering if that same work will need to be done in all type checkers and the Python parser and runtime as well.
You're right, this is generally done in the parser. IMO the proposal could only succeed if it made the point that *generally* we should allow this in function definitions. Reasonable and unsurprising semantics are available, the question is how often is this needed? I think I have come across this need occasionally, but I can't quantify it. I think only changing this for `@overload` is not going to fly (since it really does require changes to the parser). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
The problem could also be solved by putting a sentinel in the default value. For example, where FAKE_DEFAULT_VALUE is something to be bikeshedded: ``` @overload def foo(arg1: str = ..., arg2: Literal[True] = FAKE_DEFAULT_VALUE) -> str: ... @overload def foo(arg1: str = ..., arg2: Literal[False] = ...) -> bytes: ... ``` The semantics would be something like: type checkers never match overloads to calls which look like they would result in an argument being assigned FAKE_DEFAULT_VALUE. On Mon, 25 Jan 2021 at 13:47, Guido van Rossum <guido@python.org> wrote:
On Mon, Jan 25, 2021 at 11:15 AM Eric Traut <eric@traut.com> wrote:
Thanks for the clarification. If argument/parameter matching follows normal Python rules, then my concern is lessened.
Do you happen to know where in the Python parser and/or runtime this limitation is enforced today? In pyright, the check for parameters without defaults after parameters with defaults is done in the parser. This check would need to be moved out of the parser to a later stage because it would need to be conditionalized based on the presence of the @overload decorator. That's doable, but I'm wondering if that same work will need to be done in all type checkers and the Python parser and runtime as well.
You're right, this is generally done in the parser. IMO the proposal could only succeed if it made the point that *generally* we should allow this in function definitions. Reasonable and unsurprising semantics are available, the question is how often is this needed? I think I have come across this need occasionally, but I can't quantify it. I think only changing this for `@overload` is not going to fly (since it really does require changes to the parser).
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...> _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: hauntsaninja@gmail.com
participants (5)
-
Eric Traut -
Guido van Rossum -
Matthew Rahtz -
Sebastian Rittau -
Shantanu Jain