Proposal: Allow non-default after default arguments
Currently, Python doesn't allow non-default arguments after default arguments:
def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument
I believe that at the time this was introduced, no use cases for this were known and this is is supposed to prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important: The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great: def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ... To me, the "natural" solution looks like this: def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2) This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment. The more important use case involves @overloads. A condensed example of a function where the return type depends on an "encoding" parameter, followed by further parameters that could be called like this: foo(123, "utf-8") # would return bytes foo(encoding="utf-8") foo(123, None) # would return str foo(encoding=None) foo(x=123) # would return str foo() This could ideally be written as: @overload def foo(x: int = ..., encoding: None = ...) -> str: ... @overload def foo(x: int = ..., encoding: str) -> bytes: ... # plus the actual implementation But due to the syntax constraint, this needs to be hacked around with a third overload: @overload def foo(x: int = ... encoding: None = ...) -> str: ... @overload def foo(x: int, encoding: str) -> bytes: ... # for foo(123, "utf-8") @overload def foo(*, encoding: str) -> bytes: ... # for foo(encoding="utf-8") Not only is this hard to read, real examples in typeshed are usually more complex, with many arguments before or after the affected argument or even multiple affected arguments. This often becomes too complex to write or maintain. Here is one example from the wild: https://github.com/python/typeshed/blob/b95b729b9e07ab21d252701af0f5b7404672... Allowing non-default arguments after default arguments would solve both use cases above and eliminates a special case. I'm also not sure what exactly the current SyntaxError really protects us from. Adding a non-default after a default argument can't really lead bugs. - Sebastian
On Tue, Nov 9, 2021 at 8:38 PM Sebastian Rittau <srittau@rittau.biz> wrote:
Currently, Python doesn't allow non-default arguments after default arguments:
def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument
I believe that at the time this was introduced, no use cases for this were known and this is is supposed to prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important:
The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great:
def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ...
To me, the "natural" solution looks like this:
def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2)
This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment.
What would this mean, though: foo(2) Is that legal? If it is, it has to be the same as foo(y=2), by your definition. But that would mean that it's hard to get your head around the mapping of arguments and parameters. foo(1, 2) # x=1, y=2 foo(1) # x=None, y=1 There are a very very few functions in Python that have this sort of odd behaviour (range() being probably the only one most programmers will ever come across), and it's not something to encourage. I would instead recommend making the parameters keyword-only, which would allow any of them to have defaults or not have defaults. In terms of useful API design, this is usually more helpful than having an early parameter omitted. ChrisA
Am 09.11.21 um 10:50 schrieb Chris Angelico:
On Tue, Nov 9, 2021 at 8:38 PM Sebastian Rittau <srittau@rittau.biz> wrote:
Currently, Python doesn't allow non-default arguments after default arguments:
def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument
I believe that at the time this was introduced, no use cases for this were known and this is is supposed to prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important:
The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great:
def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ...
To me, the "natural" solution looks like this:
def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2)
This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment.
What would this mean, though:
foo(2)
Is that legal?
No. This would be equal to foo(x=2) (same as now), meaning the required argument "y" is missing.
I would instead recommend making the parameters keyword-only, which would allow any of them to have defaults or not have defaults. In terms of useful API design, this is usually more helpful than having an early parameter omitted.
This might be better API design (although I don't think Python should be opinionated about this outside the standard library), but this still leaves the API change example and the very real problem of @overloads unsolved. - Sebastian
On Tue, 9 Nov 2021 at 10:39, Sebastian Rittau <srittau@rittau.biz> wrote:
This might be better API design (although I don't think Python should be opinionated about this outside the standard library), but this still leaves the API change example and the very real problem of @overloads unsolved.
You can handle this using *args and/or **kwargs, so Python does support this API design if you want to use it. I disagree, however, with the statement that Python shouldn't be "opinionated" about this. It's not a matter of being opinionated, IMO, it's about whether Python has to have explicit language support for every possible use case that anyone comes up with. In this case, the situation seems rare enough, and the API design is sufficiently unusual (which is a polite way of me saying that I think it's a bad design...), that I think it's entirely reasonable for Python to not support it outside of the generic *args/**kwargs machinery. Paul
By the way, this discussion is probably better suited to the Python-Ideas mailing list. But since we're here... On Tue, Nov 09, 2021 at 11:37:40AM +0100, Sebastian Rittau wrote:
To me, the "natural" solution looks like this:
def foo(x=None, y): ... [...]
Chris Angelico asked:
What would this mean, though:
foo(2)
Is that legal?
No. This would be equal to foo(x=2) (same as now), meaning the required argument "y" is missing.
That's an odd interpretation. What you described earlier is very similar to the calling convention of range, which conceptually looks like this: range([start=0,] end, [step=1]) With your example of "foo(x=None, y)" I would expect foo(2) to mean that x gets the default and y gets the passed in argument 2, similar to the way that range(2) works. -- Steve
On Tue, Nov 9, 2021 at 6:31 AM Steven D'Aprano <steve@pearwood.info> wrote:
By the way, this discussion is probably better suited to the Python-Ideas mailing list. But since we're here...
On Tue, Nov 09, 2021 at 11:37:40AM +0100, Sebastian Rittau wrote:
To me, the "natural" solution looks like this:
def foo(x=None, y): ... [...]
Chris Angelico asked:
What would this mean, though:
foo(2)
Is that legal?
No. This would be equal to foo(x=2) (same as now), meaning the required argument "y" is missing.
That's an odd interpretation. What you described earlier is very similar to the calling convention of range, which conceptually looks like this:
range([start=0,] end, [step=1])
With your example of "foo(x=None, y)" I would expect foo(2) to mean that x gets the default and y gets the passed in argument 2, similar to the way that range(2) works.
But we all understand range() because we have been using it for years and there's a mathematical understanding of what it represents. If you were taught "range(2, 5) returns the numbers 2 through 5, exclusive" and then were asked, "what does range(2) do?", I'm not sure what a beginner would assume, but thinking it goes from 5 to infinity wouldn't be an unreasonable leap to make. An API like range() is also not pervasive, so it can be supported without any special parameter support and not feel icky; `def range(x, y=None, z=1, /)` covers its unique situation. But for me, the key issue is simply remembering what one argument means compares to two. foo(1) versus foo(1, 2) is not self-documenting whether that `1` in the first call would get bound to the same thing as the `1` in the second call. I would need to have quite the understanding of the API to know that without having to look something up in the docs. But the current semantics don't have this issue, and `1` means the same thing in both scenarios (unless you're doing something really weird like range(), which is something I wouldn't want to encourage).
On 11/9/2021 9:23 AM, Steven D'Aprano wrote:
By the way, this discussion is probably better suited to the Python-Ideas mailing list. But since we're here...
On Tue, Nov 09, 2021 at 11:37:40AM +0100, Sebastian Rittau wrote:
To me, the "natural" solution looks like this:
def foo(x=None, y): ... [...]
Chris Angelico asked:
What would this mean, though:
foo(2)
Is that legal?
No. This would be equal to foo(x=2) (same as now), meaning the required argument "y" is missing.
That's an odd interpretation. What you described earlier is very similar to the calling convention of range, which conceptually looks like this:
range([start=0,] end, [step=1])
With your example of "foo(x=None, y)" I would expect foo(2) to mean that x gets the default and y gets the passed in argument 2, similar to the way that range(2) works.
Implementing with *args is possible, but awkward as one must explicitly handle each of the 0, 1, 2, 3, and >3 args cases. The exceptions for the 0 and >3 cases should match those given by the normal processing, which can change from version to version. The actual signature is range_(start_stop, stop=_NotGiven, step=1, /), as implemented in _NotGiven = object() def range_(start_stop, stop=_NotGiven, step=1, /): if stop is _NotGiven: start = 0 stop = start_stop else: start = start_stop # Stand-in for actual code. ... return range(start, stop, step)
list(range_(4)) [0, 1, 2, 3] list(range_(3,5)) [3, 4] list(range_(3,9,2)) [3, 5, 7]
The signature of Sebastian's function with honest parameter names is foo(x_or_y, required_y=_NotGiven, /). It is the 2nd argument, not the first, that is optional, as with range. If required_y is not given, than x_or_y must be y, and x is given a default that is not part of the signature because it is explicitly bound when called. If required_y *is* given, then x_or_y can be x. The reason Python should not accept defaulted args before required args is that it would have to create sentinels, rewrite the signature to what it actually is, and write the code to bind the input arguments to the intended parameters. Very messy. Even if possible, adding code would, for instance, mess up tracing and disassembly. -- Terry Jan Reedy
Am 09.11.21 um 19:26 schrieb Terry Reedy:
The signature of Sebastian's function with honest parameter names is foo(x_or_y, required_y=_NotGiven, /). It is the 2nd argument, not the first, that is optional, as with range. If required_y is not given, than x_or_y must be y, and x is given a default that is not part of the signature because it is explicitly bound when called. If required_y *is* given, then x_or_y can be x.
Just to clarify: This proposal works differently than how range() works. foo(3) would be illegal as the required second parameter ("y") is missing. This can be solved by either supplying both "x" and "y", e.g. foo(None, 3), or by using a named parameter for "y": foo(y=3). Therefore the honest names are foo(x=None, y) in my proposal. - Sebastian
On 11/9/2021 1:52 PM, Sebastian Rittau wrote:
Am 09.11.21 um 19:26 schrieb Terry Reedy:
The signature of Sebastian's function with honest parameter names is foo(x_or_y, required_y=_NotGiven, /). It is the 2nd argument, not the first, that is optional, as with range. If required_y is not given, than x_or_y must be y, and x is given a default that is not part of the signature because it is explicitly bound when called. If required_y *is* given, then x_or_y can be x.
Just to clarify: This proposal works differently than how range() works. foo(3) would be illegal as the required second parameter ("y") is missing.
No it is not. If there is one required positional parameter and one supplies one positional argument, then that argument must be bound to that parameter name.
This can be solved by either supplying both "x" and "y", e.g. foo(None, 3), or by using a named parameter for "y": foo(y=3).
You are asking that 'x' be required if 'y' is passed by position but not if 'y' is passed by name. This is contradictory. Note that range does not allow passing by keyword and I specified the same for foo. If you make 'y' keyword only, then there is no problem making 'x' optional.
the honest names are foo(x=None, y) in my proposal.
No, because you want 'x' is required or not depending on how 'y' is passed. -- Terry Jan Reedy
On Tue, Nov 9, 2021 at 12:27 PM Terry Reedy <tjreedy@udel.edu> wrote:
On 11/9/2021 1:52 PM, Sebastian Rittau wrote:
Am 09.11.21 um 19:26 schrieb Terry Reedy:
The signature of Sebastian's function with honest parameter names is foo(x_or_y, required_y=_NotGiven, /). It is the 2nd argument, not the first, that is optional, as with range. If required_y is not given, than x_or_y must be y, and x is given a default that is not part of the signature because it is explicitly bound when called. If required_y *is* given, then x_or_y can be x.
Just to clarify: This proposal works differently than how range() works. foo(3) would be illegal as the required second parameter ("y") is missing.
No it is not. If there is one required positional parameter and one supplies one positional argument, then that argument must be bound to that parameter name.
Terry, maybe that is *your* proposal. But Sebastian's proposal works like he describes. You can argue that there is a problem with those semantics, but you cannot argue that that is not what Sebastian proposes. And please remain civil. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 11/9/21 3:24 PM, Terry Reedy wrote:
On 11/9/2021 1:52 PM, Sebastian Rittau wrote:
Am 09.11.21 um 19:26 schrieb Terry Reedy:
The signature of Sebastian's function with honest parameter names is foo(x_or_y, required_y=_NotGiven, /). It is the 2nd argument, not the first, that is optional, as with range. If required_y is not given, than x_or_y must be y, and x is given a default that is not part of the signature because it is explicitly bound when called. If required_y *is* given, then x_or_y can be x.
Just to clarify: This proposal works differently than how range() works. foo(3) would be illegal as the required second parameter ("y") is missing.
No it is not. If there is one required positional parameter and one supplies one positional argument, then that argument must be bound to that parameter name.
This can be solved by either supplying both "x" and "y", e.g. foo(None, 3), or by using a named parameter for "y": foo(y=3).
You are asking that 'x' be required if 'y' is passed by position but not if 'y' is passed by name. This is contradictory.
Note that range does not allow passing by keyword and I specified the same for foo. If you make 'y' keyword only, then there is no problem making 'x' optional.
the honest names are foo(x=None, y) in my proposal.
No, because you want 'x' is required or not depending on how 'y' is passed.
One big issue with that method is there are surprise traps. Given that definition, and a example function defined as: def fun(x=None, y): then if in future we want to add a default to y, we can't as it would be a silent breaking change for a call like fun(2). Now, if that call needed to be fun(y=2) then we avoided that break. By first feeling is that the problems caused by skipping optional positional parameters is enough to not make it worth it. There is also that fact that if we initially require the y only call to need to be a keyword parameter, but still be required and could be positional if both are provided, we could later allow it to be positional if that seems REALLY important to do, but it is much harder to take back a syntax that has been given. -- Richard Damon
On 09. 11. 21 10:50, Chris Angelico wrote:
On Tue, Nov 9, 2021 at 8:38 PM Sebastian Rittau <srittau@rittau.biz> wrote:
Currently, Python doesn't allow non-default arguments after default arguments:
def foo(x=None, y): pass File "<stdin>", line 1 def foo(x=None, y): pass ^ SyntaxError: non-default argument follows default argument
I believe that at the time this was introduced, no use cases for this were known and this is is supposed to prevent a source of bugs. I have two use cases for this, one fringe, but valid, the other more important:
The fringe use case: Suppose you have a function that takes a 2D coordinate value as separate "x" and "y" arguments. The "x" argument is optional, the "y" argument isn't. Currently there are two ways to do this, none of them particularly great:
def foo(y, x): # reverse x and y arguments, confusing ... def foo(x, y=None): # Treat the x argument as y if only one argument is provided if y is None: x, y = y, x ...
To me, the "natural" solution looks like this:
def foo(x=None, y): ... # Called like this: foo(1, 2) foo(y=2)
This could also be useful when evolving APIs. For example there is a function "bar" that takes two required arguments. In a later version, the first argument gains a useful default, the second doesn't. There is no sensible way to evolve the API at the moment.
What would this mean, though:
foo(2)
Is that legal? If it is, it has to be the same as foo(y=2), by your definition. But that would mean that it's hard to get your head around the mapping of arguments and parameters.
foo(1, 2) # x=1, y=2 foo(1) # x=None, y=1
There are a very very few functions in Python that have this sort of odd behaviour (range() being probably the only one most programmers will ever come across), and it's not something to encourage.
A more extreme case is functions with an optional *group*. In curses, the first two arguments to addch are optional. In `help()` it's documented as `window.addch([y, x,] ch[, attr=...])` and you can call it as one of: window.addch(ch) window.addch(ch, attr) window.addch(y, x, ch) window.addch(y, x, ch, attr) see: https://docs.python.org/3/library/curses.html#curses.window.addch Supporting this was a headache for Argument Clinic (PEP 436), and AFAIK it still isn't possible to express this as an inspect.Signature (PEP 362). Allowing non-default arguments after default arguments would mean introspection tools (and code that uses them) would need to be changed to prepare for the new possibilities. It's not free. And for the "encoding" case: IMO, varying the return type based on an optional "encoding" argument" is a holdover from the pre-typing era, when return types were only specified in the documentation -- just like "addch" is a holdover from the days when function signatures were only described in the docs. Nowadays, I'd consider it bad API design. The @overloads are ugly but they work -- just like the API itself. IMO we shouldn't add special cases to encourage more of it.
I would instead recommend making the parameters keyword-only, which would allow any of them to have defaults or not have defaults. In terms of useful API design, this is usually more helpful than having an early parameter omitted.
+1. I'm not sure if it's possible to mark args as keyword-only in the type stubs while keeping actual implementation backwards-compatible, but if it is, it might be a good option.
IMO it was a bad idea to merge 2 ncurses C functions into a single Python function. In the C API, there are two different functions: * mvwadd_wch(win, y, x, char): 4 arguments * wadd_wch(win, char): 2 arguments The Python curses module could/can have a separated function when (y, x) arguments are needed. Maybe it's not too late to deprecate the current complex API when passing y and x, and add a new function which accept (y, x). As Petr explained, it's a headache to have such very complicated function prototype :-( Think about completion in code editor. What is the expected behavior when pressing TAB key to complete on the code "window.addch(arg, " ? Victor On Tue, Nov 9, 2021 at 1:47 PM Petr Viktorin <encukou@gmail.com> wrote:
A more extreme case is functions with an optional *group*. In curses, the first two arguments to addch are optional. In `help()` it's documented as `window.addch([y, x,] ch[, attr=...])` and you can call it as one of:
window.addch(ch) window.addch(ch, attr) window.addch(y, x, ch) window.addch(y, x, ch, attr)
see: https://docs.python.org/3/library/curses.html#curses.window.addch
Supporting this was a headache for Argument Clinic (PEP 436), and AFAIK it still isn't possible to express this as an inspect.Signature (PEP 362).
Allowing non-default arguments after default arguments would mean introspection tools (and code that uses them) would need to be changed to prepare for the new possibilities. It's not free.
-- Night gathers, and now my watch begins. It shall not end until my death.
Am 09.11.21 um 13:44 schrieb Petr Viktorin:
And for the "encoding" case: IMO, varying the return type based on an optional "encoding" argument" is a holdover from the pre-typing era, when return types were only specified in the documentation -- just like "addch" is a holdover from the days when function signatures were only described in the docs. Nowadays, I'd consider it bad API design. The @overloads are ugly but they work -- just like the API itself. IMO we shouldn't add special cases to encourage more of it.
"encoding" arguments might be the most common case, but it's certainly not limited to that. And the fact remains that however much we desire it to be different, there are loads of APIs out there, both in the standard library and in (well received and not so well received) third-party libraries that use these constructs. This is not going to change. Also, API design is always in the eye of the beholder. Personally I prefer my example foo(x=None, y) to the design of range() of using some *args/**kwargs hacks as was suggested. I don't believe Python's users are best served by artificially limiting possible API design, especially if it causes genuine problems (as it in the case of @overload). - Sebastian
+1 on this. Typeshed is full of examples of extra overloads due to this. Mainly because of a case where a parameter with a default argument isn't actually optional (ie, using the default argument would necessary lead to an error). openpyxl stubs in particular is chockful of them.
participants (11)
-
Brett Cannon -
Chris Angelico -
Guido van Rossum -
Paul Moore -
Petr Viktorin -
Richard Damon -
Samuel T. -
Sebastian Rittau -
Steven D'Aprano -
Terry Reedy -
Victor Stinner