PEP 484 change proposal: Allowing @overload outside stub files
Ben Darnell (Tornado lead) brought up a good use case for allowing @overload in regular Python files. There's some discussion (some old, some new) here: https://github.com/ambv/typehinting/issues/72 I now propose to allow @overload in non-stub (i.e. .py) files, but with the following rule: a series of @overload-decorated functions must be followed by an implementation function that's not @overload-decorated. Calling an @overload-decorated function is still an error (I propose NotImplemented). Due to the way repeated function definitions with the same name replace each other, leaving only the last one active, this should work. E.g. for Tornado's utf8() the full definition would look like this: @overloaddef utf8(value: None) -> None: ...@overloaddef utf8(value: bytes) -> bytes: ...@overloaddef utf8(value: str) -> bytes: ... # or (unicode)->bytes, in PY2def utf8(value): # Real implementation goes here. NOTE: If you are trying to understand why we can't use a stub file here or why we can't solve this with type variables or unions, please read the issue and comment there if things are not clear. Here on python-ideas I'd like to focus on seeing whether this amendment is non-controversial (apart from tea party members who just want to repeal PEP 484 entirely :-). I know that previously we wanted to come up with a complete solution for multi-dispatch based on type annotations first, and there are philosophical problems with using @overload (though it can be made to work using sys._getframe()). The proposal here is *not* that solution. If you call the @overload-decorated function, it will raise NotImplemented. (But if you follow the rule, the @overload-decorated function objects are inaccessible so this would only happen if you forgot or misspelled the final, undecorated implementation function). -- --Guido van Rossum (python.org/~guido)
On 1/22/2016 3:00 PM, Guido van Rossum wrote:
From a naive point of view, it is the prohibition that is exceptional and in need of justification. So its removal would seem non-problematical.
Again, from a naive point of view, treating 'overload' the same as 'x', this seems wasteful, so the usage must be a consenting-adult tradeoff between the time taken to create function objects that get thrown away and the side-effect of 'overload'. I do understand that non-beginners with expectation based on other languages, who don't know Pythons specific usage of 'overload', may get confused. Your proposed implementation is missing a return statement. def overload(func): def overload_dummy(*args, **kwds): raise NotImplemented( "You should not call an overloaded function. " "A series of @overload-decorated functions " "outside a stub module should always be followed " "by an implementation that is not @overloaded.") To avoid throwing away two functions with each def, I suggested moving the constant replacement outside of overload. def _overload_dummy(*args, **kwds): raise NotImplemented( "You should not call an overloaded function. " "A series of @overload-decorated functions " "outside a stub module should always be followed " "by an implementation that is not @overloaded.") def overload(func): return _overload_dummy
Sorry, I don't see any connection between tea party philosophy and type hinting, except maybe in the opposite direction. Maybe we should continue leaving external politics, US or otherwise, out of pydev discussions.
It would be possible for _overload_dummy to add a reference to each func arg to a list or set thereof. Perhaps you meant more or less the same with 'using sys._getframe()'. The challenge would be writing a new 'overload_complete' decorator, for the last function, that would combine the pieces. -- Terry Jan Reedy
On Jan 22, 2016, at 12:00, Guido van Rossum <guido@python.org> wrote:
It feels like this is too similar to single_dispatch to be so different in details. I get the distinction (the former is for a function that has a single implementation but acts like a bunch of overloads that switch on type, the latter is for a function that's actually implemented as a bunch of overloads that switch on type), and I also get that it'll be much easier to extend overload to compile-time multiple dispatch than to extend single_dispatch to run-time multiple dispatch (and you don't want the former to have to wait on the latter), and so on. But it still feels like someone has stapled together two languages here. (Of course I feel the same way about typing.protocols and implicit ABCs, and I know you disagreed there, so I wouldn't be too surprised if you disagree here as well. But this is even _more_ of a distinct and parallel system than that was--at least typing.Sized is in some way related to collections.abc.Sized, while overload is not related to single_dispatch at all, so someone who finds the wrong one in a search seems much more liable to assume that Python just doesn't have the one he wants than to find it.) Other than that detail, I like everything else: some feature like this should definitely be part of the type checker (the only alternative is horribly complex type annotations); if it's allowed in stub files, it should be allowed in source files; and the rule of "follow the overloads with the real implementation" seems like by far the simplest rule that could make this work.
On 23 January 2016 at 06:00, Guido van Rossum <guido@python.org> wrote:
I share Andrew's concerns about the lack of integration between this and functools.singledispatch, so would it be feasible to apply the "magic comments" approach here, similar to the workarounds for variable annotations and Py2 compatible function annotations? That is, would it be possible to use a notation like the following?: def utf8(value): # type: (None) -> None # type: (bytes) -> bytes # type: (unicode) -> bytes ... You're already going to have to allow this for single lines to handle Py2 compatible annotations, so it seems reasonable to also extend it to handle overloading while you're still figuring out a native syntax for that. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Jan 22, 2016 at 8:04 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
That's clever. I'm sure we could make it work if we wanted to. But it doesn't map to anything else -- in stub files do already have @overload, and there's no way to translate this directly to Python 3 in-line annotations. Regarding the confusion with @functools.singledispatch, hopefully all documentation for @overload (including StackOverflow :-) would quickly point out how you are supposed to use it. There's also a deep distinction between @overload in PEP 484 and singledispatch, multidispatch or even the (ultimately deferred) approach from PEP 3124, also called @overload. PEP 484's @overload (whether in stubs or in this proposed form in .py files) talks to the *type checker* and it can be used with generic types. For example, suppose you have @overload def foo(a: Sequence[int]) -> int: ... @overload def foo(a: Sequence[str]) -> float: ... def foo(a): return sum(float(x) for x in a) (NOTE: Don't be fooled to think that the implementation is the last word on the supported types and hence the list of overloads is "obviously" incomplete. The type checker needs to take the overloads at their word and reject calls to e.g. foo([3.14]). A future implementation that matches the overloaded signatures given here might not work for float arguments.) Here the implementation will have to somehow figure out whether its argument is a list of integers or strings, e.g. by checking the type of the first item -- that should be okay since passing the type check implies a promise that the argument is homogeneous. But a purely runtime dispatcher would not be able to make that distinction so easily, since PEP 484 assumes type erasure -- at runtime the argument is just a Sequence. Of course, functools.singledispatch sidesteps this by not supporting generic types at all. But the example illustrates that the two are more different than you'd think from the utf8() example (which just distinguishes between unicode, bytes and None -- no generic types there).
Clearly both @overload and functools.singledispatch are stepping stones on the way to an elusive better solution. Hopefully until that solution is found they can live together? -- --Guido van Rossum (python.org/~guido)
On 23 January 2016 at 15:43, Guido van Rossum <guido@python.org> wrote:
Right, my assumption is that it would eventually be translated to a full multi-dispatch solution for Python 3, whatever that spelling turns out to be - I'm just assuming that spelling *won't* involve annotating empty functions the way that stub files currently do, but rather annotating separate implementations for a multidispatch algorithm, or perhaps gaining a way to more neatly compose multiple sets of annotations.
While the disconnect with functools.singledispatch is one concern, another is the sheer visual weight of this approach. The real function has to go last to avoid getting clobbered, but the annotations for multiple dispatch end up using a lot of space beforehand. It gets worse if you need to combine it with Python 2 compatible type hinting comments since you can't squeeze the function definitions onto one line anymore: @overload def foo(a): # type: (Sequence[int]) -> int ... @overload def foo(a): # type: (Sequence[str]) -> float ... def foo(a): return sum(float(x) for x in a) If you were to instead go with a Python 2 compatible comment based inline solution for now, you'd then get to design the future official spelling for multi-dispatch annotations based on your experience with both that and with the decorator+annotations approach used in stub files. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Jan 23, 2016 at 6:17 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
We don't have a proposal for multidispatch even though people have been hoping for it to happen for a long time. It's a much harder problem than providing multiple signatures for a function, and it's also arguably a different problem, even if there is some overlap. @overload might still be preferable to a multidispatch solution in a lot of cases even if both existed since @overload is conceptually pretty simple. However, this is all conjecture since we don't know what multidispatch would look like. I'd love to see a multidispatch proposal.
There is little evidence that @overload answers a *common* need, even if the need is important -- originally we left out @overload in .py files because we hadn't found a convincing use case. I don't consider the visual weight to be a major problem, as this would only be used rarely, at least based on our current understanding. But clearly the proposed syntax won't win any prettiness awards. Singledispatch solves a different problem and makes different tradeoffs -- for example, it adds more runtime overhead, and it doesn't lend itself to specifying multiple return types for a single function body that depend on argument types. It also lives in a different module. I don't worry too much about the overlap.
Your proposed comment based solution looks nicer in Python 2 code than @overload. I'd prefer optimizing any syntax we choose for Python 3 as that's where the future is. I'd rather not be forced to use comment-based signatures in Python 3 only code. Jukka
On 24 January 2016 at 04:13, Jukka Lehtosalo <jlehtosalo@gmail.com> wrote:
For the benefit of folks reading this thread, but not the linked issue: Guido pointed out some cases with variable signatures (e.g. annotating a range-style API) & keyword args where the stacked comments idea doesn't work, so I switched to being +0 on the "@overload in .py files" interim solution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan <ncoghlan@...> writes:
I find that https://pypi.python.org/pypi/multipledispatch looks quite nice: line 155, in __call__ func = self._cache[types] KeyError: (<class 'float'>, <class 'int'>) Stefan Krah
On Jan 23, 2016, at 13:12, Stefan Krah <skrah.temporarily@gmail.com> wrote:
Of course you still have to work out how that would fit with type annotations. Presumably you could just move the dispatched types from the decorator to the annotations, and add a return type on each overload. And you could make the dispatch algorithm ignore element types in generic types (so add(a: Sequence[T], b: Sequence[T]) gets called on any pair of Sequences). But even then, it's hard to imagine how a type checker could understand your code unless it had special-case code for this special decorator. Not to mention that you're not supposed to runtime-dispatch on typing.Sequence (isinstance and issubclass only work by accident), but you can't genericize collections.abc.Sequence. Plus, as either Guido or Jukka pointed out earlier, you may want to specify that Sequence[T] normally returns T but Sequence[Number] always returns float or something; at runtime, those are the same type, so they have to share a single implementation, which takes you right back to needing a way to specify overloads for the type checker. Still, I would love to see someone take that library and mypy and experiment with making them work together and solving all of these problems. (As a side note, every time I look at this stuff, I start thinking I want type computation so I can specify that add(a: Sequence[T], b: Sequence[U]) -> Sequence[decltype(T + U)], until I spend a few minutes trying to find a way to write that that isn't as horrible as C++ without introducing all of Haskell into Python, and then appreciating again why maybe building the simple thing first was a good idea...)
Andrew Barnert via Python-ideas <python-ideas@...> writes: [multipledispatch]
Still, I would love to see someone take that library and mypy and experiment with making them work together
Exactly: I posted that link mainly in the hope of not having a simple @overload now and perhaps a fully typed-checked @dispatch version later. But apparently people really want the simple version right now. Stefan Krah
Overall, that's an interesting idea although I share the concern about the visual heaviness of the proposal. Not sure if that can be resolved properly. I somehow like the comment idea but that doesn't fit into the remaining concept well. On 22.01.2016 21:00, Guido van Rossum wrote:
Calling an @overload-decorated function is still an error (I propose NotImplemented).
Not sure if that applies here, but would that be rather NotImplementedError? Best, Sven
On 1/22/2016 3:00 PM, Guido van Rossum wrote:
From a naive point of view, it is the prohibition that is exceptional and in need of justification. So its removal would seem non-problematical.
Again, from a naive point of view, treating 'overload' the same as 'x', this seems wasteful, so the usage must be a consenting-adult tradeoff between the time taken to create function objects that get thrown away and the side-effect of 'overload'. I do understand that non-beginners with expectation based on other languages, who don't know Pythons specific usage of 'overload', may get confused. Your proposed implementation is missing a return statement. def overload(func): def overload_dummy(*args, **kwds): raise NotImplemented( "You should not call an overloaded function. " "A series of @overload-decorated functions " "outside a stub module should always be followed " "by an implementation that is not @overloaded.") To avoid throwing away two functions with each def, I suggested moving the constant replacement outside of overload. def _overload_dummy(*args, **kwds): raise NotImplemented( "You should not call an overloaded function. " "A series of @overload-decorated functions " "outside a stub module should always be followed " "by an implementation that is not @overloaded.") def overload(func): return _overload_dummy
Sorry, I don't see any connection between tea party philosophy and type hinting, except maybe in the opposite direction. Maybe we should continue leaving external politics, US or otherwise, out of pydev discussions.
It would be possible for _overload_dummy to add a reference to each func arg to a list or set thereof. Perhaps you meant more or less the same with 'using sys._getframe()'. The challenge would be writing a new 'overload_complete' decorator, for the last function, that would combine the pieces. -- Terry Jan Reedy
On Jan 22, 2016, at 12:00, Guido van Rossum <guido@python.org> wrote:
It feels like this is too similar to single_dispatch to be so different in details. I get the distinction (the former is for a function that has a single implementation but acts like a bunch of overloads that switch on type, the latter is for a function that's actually implemented as a bunch of overloads that switch on type), and I also get that it'll be much easier to extend overload to compile-time multiple dispatch than to extend single_dispatch to run-time multiple dispatch (and you don't want the former to have to wait on the latter), and so on. But it still feels like someone has stapled together two languages here. (Of course I feel the same way about typing.protocols and implicit ABCs, and I know you disagreed there, so I wouldn't be too surprised if you disagree here as well. But this is even _more_ of a distinct and parallel system than that was--at least typing.Sized is in some way related to collections.abc.Sized, while overload is not related to single_dispatch at all, so someone who finds the wrong one in a search seems much more liable to assume that Python just doesn't have the one he wants than to find it.) Other than that detail, I like everything else: some feature like this should definitely be part of the type checker (the only alternative is horribly complex type annotations); if it's allowed in stub files, it should be allowed in source files; and the rule of "follow the overloads with the real implementation" seems like by far the simplest rule that could make this work.
On 23 January 2016 at 06:00, Guido van Rossum <guido@python.org> wrote:
I share Andrew's concerns about the lack of integration between this and functools.singledispatch, so would it be feasible to apply the "magic comments" approach here, similar to the workarounds for variable annotations and Py2 compatible function annotations? That is, would it be possible to use a notation like the following?: def utf8(value): # type: (None) -> None # type: (bytes) -> bytes # type: (unicode) -> bytes ... You're already going to have to allow this for single lines to handle Py2 compatible annotations, so it seems reasonable to also extend it to handle overloading while you're still figuring out a native syntax for that. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Jan 22, 2016 at 8:04 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
That's clever. I'm sure we could make it work if we wanted to. But it doesn't map to anything else -- in stub files do already have @overload, and there's no way to translate this directly to Python 3 in-line annotations. Regarding the confusion with @functools.singledispatch, hopefully all documentation for @overload (including StackOverflow :-) would quickly point out how you are supposed to use it. There's also a deep distinction between @overload in PEP 484 and singledispatch, multidispatch or even the (ultimately deferred) approach from PEP 3124, also called @overload. PEP 484's @overload (whether in stubs or in this proposed form in .py files) talks to the *type checker* and it can be used with generic types. For example, suppose you have @overload def foo(a: Sequence[int]) -> int: ... @overload def foo(a: Sequence[str]) -> float: ... def foo(a): return sum(float(x) for x in a) (NOTE: Don't be fooled to think that the implementation is the last word on the supported types and hence the list of overloads is "obviously" incomplete. The type checker needs to take the overloads at their word and reject calls to e.g. foo([3.14]). A future implementation that matches the overloaded signatures given here might not work for float arguments.) Here the implementation will have to somehow figure out whether its argument is a list of integers or strings, e.g. by checking the type of the first item -- that should be okay since passing the type check implies a promise that the argument is homogeneous. But a purely runtime dispatcher would not be able to make that distinction so easily, since PEP 484 assumes type erasure -- at runtime the argument is just a Sequence. Of course, functools.singledispatch sidesteps this by not supporting generic types at all. But the example illustrates that the two are more different than you'd think from the utf8() example (which just distinguishes between unicode, bytes and None -- no generic types there).
Clearly both @overload and functools.singledispatch are stepping stones on the way to an elusive better solution. Hopefully until that solution is found they can live together? -- --Guido van Rossum (python.org/~guido)
On 23 January 2016 at 15:43, Guido van Rossum <guido@python.org> wrote:
Right, my assumption is that it would eventually be translated to a full multi-dispatch solution for Python 3, whatever that spelling turns out to be - I'm just assuming that spelling *won't* involve annotating empty functions the way that stub files currently do, but rather annotating separate implementations for a multidispatch algorithm, or perhaps gaining a way to more neatly compose multiple sets of annotations.
While the disconnect with functools.singledispatch is one concern, another is the sheer visual weight of this approach. The real function has to go last to avoid getting clobbered, but the annotations for multiple dispatch end up using a lot of space beforehand. It gets worse if you need to combine it with Python 2 compatible type hinting comments since you can't squeeze the function definitions onto one line anymore: @overload def foo(a): # type: (Sequence[int]) -> int ... @overload def foo(a): # type: (Sequence[str]) -> float ... def foo(a): return sum(float(x) for x in a) If you were to instead go with a Python 2 compatible comment based inline solution for now, you'd then get to design the future official spelling for multi-dispatch annotations based on your experience with both that and with the decorator+annotations approach used in stub files. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Jan 23, 2016 at 6:17 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
We don't have a proposal for multidispatch even though people have been hoping for it to happen for a long time. It's a much harder problem than providing multiple signatures for a function, and it's also arguably a different problem, even if there is some overlap. @overload might still be preferable to a multidispatch solution in a lot of cases even if both existed since @overload is conceptually pretty simple. However, this is all conjecture since we don't know what multidispatch would look like. I'd love to see a multidispatch proposal.
There is little evidence that @overload answers a *common* need, even if the need is important -- originally we left out @overload in .py files because we hadn't found a convincing use case. I don't consider the visual weight to be a major problem, as this would only be used rarely, at least based on our current understanding. But clearly the proposed syntax won't win any prettiness awards. Singledispatch solves a different problem and makes different tradeoffs -- for example, it adds more runtime overhead, and it doesn't lend itself to specifying multiple return types for a single function body that depend on argument types. It also lives in a different module. I don't worry too much about the overlap.
Your proposed comment based solution looks nicer in Python 2 code than @overload. I'd prefer optimizing any syntax we choose for Python 3 as that's where the future is. I'd rather not be forced to use comment-based signatures in Python 3 only code. Jukka
On 24 January 2016 at 04:13, Jukka Lehtosalo <jlehtosalo@gmail.com> wrote:
For the benefit of folks reading this thread, but not the linked issue: Guido pointed out some cases with variable signatures (e.g. annotating a range-style API) & keyword args where the stacked comments idea doesn't work, so I switched to being +0 on the "@overload in .py files" interim solution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan <ncoghlan@...> writes:
I find that https://pypi.python.org/pypi/multipledispatch looks quite nice: line 155, in __call__ func = self._cache[types] KeyError: (<class 'float'>, <class 'int'>) Stefan Krah
On Jan 23, 2016, at 13:12, Stefan Krah <skrah.temporarily@gmail.com> wrote:
Of course you still have to work out how that would fit with type annotations. Presumably you could just move the dispatched types from the decorator to the annotations, and add a return type on each overload. And you could make the dispatch algorithm ignore element types in generic types (so add(a: Sequence[T], b: Sequence[T]) gets called on any pair of Sequences). But even then, it's hard to imagine how a type checker could understand your code unless it had special-case code for this special decorator. Not to mention that you're not supposed to runtime-dispatch on typing.Sequence (isinstance and issubclass only work by accident), but you can't genericize collections.abc.Sequence. Plus, as either Guido or Jukka pointed out earlier, you may want to specify that Sequence[T] normally returns T but Sequence[Number] always returns float or something; at runtime, those are the same type, so they have to share a single implementation, which takes you right back to needing a way to specify overloads for the type checker. Still, I would love to see someone take that library and mypy and experiment with making them work together and solving all of these problems. (As a side note, every time I look at this stuff, I start thinking I want type computation so I can specify that add(a: Sequence[T], b: Sequence[U]) -> Sequence[decltype(T + U)], until I spend a few minutes trying to find a way to write that that isn't as horrible as C++ without introducing all of Haskell into Python, and then appreciating again why maybe building the simple thing first was a good idea...)
Andrew Barnert via Python-ideas <python-ideas@...> writes: [multipledispatch]
Still, I would love to see someone take that library and mypy and experiment with making them work together
Exactly: I posted that link mainly in the hope of not having a simple @overload now and perhaps a fully typed-checked @dispatch version later. But apparently people really want the simple version right now. Stefan Krah
Overall, that's an interesting idea although I share the concern about the visual heaviness of the proposal. Not sure if that can be resolved properly. I somehow like the comment idea but that doesn't fit into the remaining concept well. On 22.01.2016 21:00, Guido van Rossum wrote:
Calling an @overload-decorated function is still an error (I propose NotImplemented).
Not sure if that applies here, but would that be rather NotImplementedError? Best, Sven
participants (7)
-
Andrew Barnert -
Guido van Rossum -
Jukka Lehtosalo -
Nick Coghlan -
Stefan Krah -
Sven R. Kunze -
Terry Reedy