PEP 671 (late-bound arg defaults), next round of discussion!
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere. *PEP 671: Syntax for late-bound function argument defaults* Questions, for you all: 1) If this feature existed in Python 3.11 exactly as described, would you use it? 2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? (It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.) 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? 4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.) 5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best! The reference implementation currently has some test failures, which I'm looking into. I'm probably going to make this my personal default Python interpreter for a while, to see how things go. ChrisA
On 1 Dec 2021, at 10:16 AM, Chris Angelico <rosuav@gmail.com> wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it? Yes I will use it.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? No, because it would look like a lambda (if the new lambda syntax were approved), indicating this will be evaluated each time the function is run.
(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
NA
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I haven’t done it from source. I might try to learn how to do it in the next weekend and give it a try.
I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. I will show this to some of my coworkers who are python experts and I will report back.
Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!
The reference implementation currently has some test failures, which I'm looking into. I'm probably going to make this my personal default Python interpreter for a while, to see how things go.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UVOQEK... Code of Conduct: http://python.org/psf/codeofconduct/
Answering questions:
1) If this feature existed in Python 3.11 exactly as described, would you use it? I would definitely use it. 2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? No, it isn't much of a cognitive burden. 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? Probably all options other than (d). 5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) Yes I do know how to, and I'm gonna try it out soon enough. I'll reply to this comment once I've done so.
On Wed, Dec 01, 2021 at 07:47:49AM -0000, Jeremiah Vivian wrote:
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? No, it isn't much of a cognitive burden.
You say that now, but if you read function definitions that looked like this: def process(func:List->int=>xs=>expression)->int: ... would you still agree? -- Steve
On Wed, Dec 1, 2021 at 6:58 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Wed, Dec 01, 2021 at 07:47:49AM -0000, Jeremiah Vivian wrote:
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? No, it isn't much of a cognitive burden.
You say that now, but if you read function definitions that looked like this:
def process(func:List->int=>xs=>expression)->int: ...
would you still agree?
I'm not sure what that's supposed to mean. Is List->int an annotation, and if so, who's deciding on that syntax? You seem to have used two different arrows in two different ways: List->int # presumably an annotation meaning "function that takes List, returns int" func:ann=>dflt # late-bound default, completely unnecessary here xs=>expression # presumably a lambda function def process(args)->int # return value annotation Why should List->int and xs=>expression use different arrows? Wouldn't it be much more reasonable for them to use the same one, whichever that be? And if that does turn out to be "=>", then yes, I would be looking at changing PEP 671 to recommend := or =: or something, for clarity (but still an equals sign with one other symbol next to it). It's always possible to come up with pathological code. But this is only really as bad as you describe if it has zero spaces in it. Otherwise, it's pretty easy to clarify which parts go where: def process(func: List->int => xs=>expression) -> int: ... Tada, easy grouping. ChrisA
On Wed, Dec 1, 2021 at 7:07 PM Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Dec 1, 2021 at 6:58 PM Steven D'Aprano <steve@pearwood.info> wrote:
You say that now, but if you read function definitions that looked like this:
def process(func:List->int=>xs=>expression)->int: ...
would you still agree?
I'm not sure what that's supposed to mean.
(To clarify: by "that", I mean "the hypothetical line of code", not the question you're asking. I'm not sure what the code example is supposed to mean.) ChrisA
On Wed, Dec 01, 2021 at 07:07:20PM +1100, Chris Angelico wrote:
def process(func:List->int=>xs=>expression)->int: ...
I'm not sure what that's supposed to mean.
You did a pretty good job of working it out :-)
Is List->int an annotation, and if so, who's deciding on that syntax? You seem to have used two different arrows in two different ways:
Exactly my point. Beyond your arrow syntax for default parameters, and the existing return annotation use, there are two other hypothetical proposals for arrow syntax on the table: - using `->` as an alias for typing.Callable; - using `=>` as a more compact lambda; See here: https://lwn.net/Articles/847960/ I rate the first one (the typing.Callable alias) as extremely likely. It is hard to annotate parameters which are functions, and the typing community is actively working on making annotations easier. So my gut feeling is that it is only a matter of time before we have annotations like `func:List->int`. And probably closer to 3.11 than 3.20 :-) The second, the compact lambda, I rate as only moderately likely. Guido seems to like it, but whether somebody writes a PEP and the Steering Council accepts it is uncertain. But it is something that does keep coming up, and I think we should consider it as a possibility. Even if `=>` doesn't get used for compact lambda, it is an obvious syntax to use for *something* and so we should consider how late-binding will look, not just with existing syntax, but also with *likely* future syntax. (When you build a road, you don't plan for just the population you have now, but also the population you are likely to have in the future.) I don't expect anyone to take into account *arbitrary* and unpredictable future syntax, but it is wise to take into account future syntax that we're already planning or that seems likely.
List->int # presumably an annotation meaning "function that takes List, returns int"
Correct.
func:ann=>dflt # late-bound default, completely unnecessary here
Come on Chris, how can you say that it is "completely unnecessary"? Unless that's an admission that late-bound defaults are all unnecessary... *wink* There is a difference between these two: def func(arg=lambda a: expression): ... and this: def func(arg=None): if arg is None: arg = lambda a: expression ... therefore there will be a difference between: def func(arg=lambda a: expression): def func(arg=>lambda a: expression): If nothing else, in the first case (early binding) you get the same function object every time. In the second, you get a freshly made function object each time. Since function objects are mutable (they have a writable `__dict__` that's a visible difference even if the bodies are identical. And they may not be. But even if it is unnecessary, it will still be permitted, just as we will be able to write: # Now this *actually is* totally unnecessary use of late-binding def func(arg=>None):
xs=>expression # presumably a lambda function def process(args)->int # return value annotation
Why should List->int and xs=>expression use different arrows?
Because they do different things. To avoid confusion. Chris, you wrote the PEP for the walrus operator. Why should the assignment operator use a different symbol from the assignment statement? Same reason that typing.Callable and lambda will likely use different arrows. Anyway, if you disagree, take it up with Guido, it was his suggestion to use different arrows :-P
Wouldn't it be much more reasonable for them to use the same one, whichever that be? And if that does turn out to be "=>", then yes, I would be looking at changing PEP 671 to recommend := or =: or something, for clarity (but still an equals sign with one other symbol next to it).
Oh great, now you're going to conflict with walrus... def process(obj:Union[T:=something, List[T]]:=func(x:=expression)+x)->T: We ought to at least try to avoid clear and obvious conflicts between new and existing syntax. Using `:=` is even worse than `=>`, and `=:` is just *begging* to confuse newcomers "why do we have THREE different assignment symbols?"
It's always possible to come up with pathological code. But this is only really as bad as you describe if it has zero spaces in it. Otherwise, it's pretty easy to clarify which parts go where:
def process(func: List->int => xs=>expression) -> int: ...
"My linter complains about spaces around operators! Take them out!" Or maybe we should put more in? Spaces are optional. def process(func : List -> int => xs => expression) -> int: ... I'm not saying that an experienced Pythonista who is a careful reader can't work out what the meaning is. It's not *ambiguous* syntax to a careful reader. But it's *confusing* syntax to somebody who may not be as careful and experienced as you, or is coding late at night, or in a hurry, or coding while tired and emotional or distracted. We should prefer to avoid confusable syntax when we can. The interpreter can always disambiguate assignment as an expression from assignment as a statement, but we still chose to make it easy on the human reader by using distinct symbols. def process(arg>=expression) would be totally unambiguous to the intepreter, but I trust you would reject that because it reads like "greater than" to the human reader. -- Steve
On Wed, Dec 1, 2021 at 9:09 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Wed, Dec 01, 2021 at 07:07:20PM +1100, Chris Angelico wrote:
def process(func:List->int=>xs=>expression)->int: ...
I'm not sure what that's supposed to mean.
You did a pretty good job of working it out :-)
Okay, cool, thanks.
Exactly my point. Beyond your arrow syntax for default parameters, and the existing return annotation use, there are two other hypothetical proposals for arrow syntax on the table:
- using `->` as an alias for typing.Callable;
- using `=>` as a more compact lambda;
That confuses me. Why would they use different arrows? If we have a compact lambda syntax, why can't actual functions be used as annotations to represent functions? I believe there's plans to have [str] mean List[str], which makes a lot of sense. Why not have "lambda str: int" or "str=>int" be an annotation too? But okay. Supposing that annotations have to use one arrow and lambda functions use another, then yes, this is a good reason to use "=:" for late-bound defaults. It's not that big a change.
func:ann=>dflt # late-bound default, completely unnecessary here
Come on Chris, how can you say that it is "completely unnecessary"? Unless that's an admission that late-bound defaults are all unnecessary... *wink*
There is a difference between these two:
def func(arg=lambda a: expression): ...
and this:
def func(arg=None): if arg is None: arg = lambda a: expression ...
therefore there will be a difference between:
def func(arg=lambda a: expression): def func(arg=>lambda a: expression):
If nothing else, in the first case (early binding) you get the same function object every time. In the second, you get a freshly made function object each time. Since function objects are mutable (they have a writable `__dict__` that's a visible difference even if the bodies are identical. And they may not be.
Yyyyyes. Okay. There is a technical way in which you might want this alternate behaviour. Is that REALLY something you're planning on doing - having a function which takes another function as an argument, and which behaves differently based on whether it's given the same function every time or multiple different functions with the same behaviour?
But even if it is unnecessary, it will still be permitted, just as we will be able to write:
# Now this *actually is* totally unnecessary use of late-binding def func(arg=>None):
Yes. That one is absolutely unnecessary, since there is no way you'll ever get back a different result. Except that you could then mutate func's dunders and change the behaviour. So in a purely technical sense, nothing can be called "unnecessary". Now, in real terms: late-binding a lambda function with no default arguments seems like a pretty pointless thing to do. I stand by my original statement :) And I also stand by my original statement that proper use of the space bar can not only improve readability, it can also help you find a date with space girls and space guys... or maybe that last part only works on Mars.
xs=>expression # presumably a lambda function def process(args)->int # return value annotation
Why should List->int and xs=>expression use different arrows?
Because they do different things. To avoid confusion.
Chris, you wrote the PEP for the walrus operator. Why should the assignment operator use a different symbol from the assignment statement? Same reason that typing.Callable and lambda will likely use different arrows.
Anyway, if you disagree, take it up with Guido, it was his suggestion to use different arrows :-P
Fair enough, but typing also started out with List[str] and is now looking at using list[str] and, I think, [str], because it's more consistent :)
Wouldn't it be much more reasonable for them to use the same one, whichever that be? And if that does turn out to be "=>", then yes, I would be looking at changing PEP 671 to recommend := or =: or something, for clarity (but still an equals sign with one other symbol next to it).
Oh great, now you're going to conflict with walrus...
def process(obj:Union[T:=something, List[T]]:=func(x:=expression)+x)->T:
We ought to at least try to avoid clear and obvious conflicts between new and existing syntax.
Using `:=` is even worse than `=>`, and `=:` is just *begging* to confuse newcomers "why do we have THREE different assignment symbols?"
If you look at every way the equals sign is used in Python grammar, there are a lot of subtle differences. The difference between "f(x=1)" and "x=1" doesn't bother people, and those are exactly the same symbol. There are only so many symbols that we can type on everyone's keyboards. But you're right, and that's why I would very much like to reuse an existing symbol rather than create new ones.
It's always possible to come up with pathological code. But this is only really as bad as you describe if it has zero spaces in it. Otherwise, it's pretty easy to clarify which parts go where:
def process(func: List->int => xs=>expression) -> int: ...
"My linter complains about spaces around operators! Take them out!"
Yes, take your linter out the back and execute it :)
Or maybe we should put more in? Spaces are optional.
def process(func : List -> int => xs => expression) -> int: ...
I'm not saying that an experienced Pythonista who is a careful reader can't work out what the meaning is. It's not *ambiguous* syntax to a careful reader. But it's *confusing* syntax to somebody who may not be as careful and experienced as you, or is coding late at night, or in a hurry, or coding while tired and emotional or distracted.
We should prefer to avoid confusable syntax when we can. The interpreter can always disambiguate assignment as an expression from assignment as a statement, but we still chose to make it easy on the human reader by using distinct symbols.
def process(arg>=expression)
would be totally unambiguous to the intepreter, but I trust you would reject that because it reads like "greater than" to the human reader.
Actually that, to me, is no different from the other options - it's still an arrow (but now more like a funnel than an arrow, which is a bit weird). It's no more a greater-than than "=>" is. ChrisA
On 01/12/2021 10:05, Steven D'Aprano wrote:
Oh great, now you're going to conflict with walrus...
def process(obj:Union[T:=something, List[T]]:=func(x:=expression)+x)->T:
We ought to at least try to avoid clear and obvious conflicts between new and existing syntax.
Using `:=` is even worse than `=>`, and `=:` is just *begging* to confuse newcomers "why do we have THREE different assignment symbols?"
You can't have it both ways. Either you re-use `:=`, or you add a third symbol. (There would be no actual conflict with the walrus operator.) Best wishes Rob Cliffe
On Wed, Dec 1, 2021 at 7:12 PM Jeremiah Vivian <nohackingofkrowten@gmail.com> wrote:
I'll reply to this comment once I've done so. I just realized that this message replied on a comment instead of the thread. Anyways, it's a really impressive implementation (hope this statement doesn't spark some sort of debate).
I'm.... going to take that at face value and interpret it as a compliment. Thank you. :) If you intended that to be taken differently, then yeah, you're probably not wrong there too - it is, in a sense, quite... impressive. ChrisA
To be honest, I like the `@param=value` syntax. It is sort of easier to read than `param=>value`, though I do not have problems with distinguishing the arrow `=>` from other things.
Exactly like you. I can read => to mean late-bound or supposedly lambda function and I can distinguish it from the callable typing -> but "@param=value” is definitely better in my eyes. => won’t prevent me from using it for default mutable objects.
On 2 Dec 2021, at 3:45 AM, Jeremiah Vivian <nohackingofkrowten@gmail.com> wrote:
To be honest, I like the `@param=value` syntax. It is sort of easier to read than `param=>value`, though I do not have problems with distinguishing the arrow `=>` from other things.
On Wed, Dec 1, 2021 at 6:43 PM Abdulla Al Kathiri <alkathiri.abdulla@gmail.com> wrote:
On 1 Dec 2021, at 10:16 AM, Chris Angelico <rosuav@gmail.com> wrote: 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).
Very interesting. That doesn't normally seem like a function default - is the callback ever going to be passed two arguments (self and actual text) such that the default would be ignored? But, hey, if it makes sense in your code to make it a parameter, sure!
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I haven’t done it from source. I might try to learn how to do it in the next weekend and give it a try.
I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. I will show this to some of my coworkers who are python experts and I will report back.
Thank you! All feedback greatly appreciated. Building CPython from source can be done by following these instructions: https://devguide.python.org/setup/ Instead of creating your own clone from the pristine master copy, instead clone my repository at https://github.com/rosuav/cpython and checkout the pep-671 branch. That'll give you my existing reference implementation (it's pretty crummy but it mostly works). ChrisA
On 1 Dec 2021, at 12:01 PM, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Dec 1, 2021 at 6:43 PM Abdulla Al Kathiri <alkathiri.abdulla@gmail.com> wrote:
On 1 Dec 2021, at 10:16 AM, Chris Angelico <rosuav@gmail.com> wrote: 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).
Very interesting. That doesn't normally seem like a function default - is the callback ever going to be passed two arguments (self and actual text) such that the default would be ignored? Yeah. Let’s say the callback prints the text on the main window console. I could use the same function to print something on the console not related to the default widget changing text. Maybe another push button that prints literal “WOW”. If I made the second argument a default widget (def callback(self, line_edit=self.line_edit):...) and then call line_edit.text() inside the function, I would be able to pass any other widget that has the method text() but I wouldn’t be able to pass a literal text.
But, hey, if it makes sense in your code to make it a parameter, sure!
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I haven’t done it from source. I might try to learn how to do it in the next weekend and give it a try.
I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. I will show this to some of my coworkers who are python experts and I will report back.
Thank you! All feedback greatly appreciated.
Building CPython from source can be done by following these instructions:
https://devguide.python.org/setup/
Instead of creating your own clone from the pristine master copy, instead clone my repository at https://github.com/rosuav/cpython and checkout the pep-671 branch. That'll give you my existing reference implementation (it's pretty crummy but it mostly works).
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7VOA7B... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Dec 1, 2021 at 8:18 PM Abdulla Al Kathiri <alkathiri.abdulla@gmail.com> wrote:
On 1 Dec 2021, at 12:01 PM, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Dec 1, 2021 at 6:43 PM Abdulla Al Kathiri <alkathiri.abdulla@gmail.com> wrote:
On 1 Dec 2021, at 10:16 AM, Chris Angelico <rosuav@gmail.com> wrote: 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).
Very interesting. That doesn't normally seem like a function default - is the callback ever going to be passed two arguments (self and actual text) such that the default would be ignored? Yeah. Let’s say the callback prints the text on the main window console. I could use the same function to print something on the console not related to the default widget changing text. Maybe another push button that prints literal “WOW”. If I made the second argument a default widget (def callback(self, line_edit=self.line_edit):...) and then call line_edit.text() inside the function, I would be able to pass any other widget that has the method text() but I wouldn’t be able to pass a literal text.
Ahh gotcha. Then yes, you have a function that takes up to two parameters (or one if you don't count self), and yes, omitting the second argument will have the same effect as fetching the text from that control. So, sounds like a great use for it! ChrisA
On Wednesday, December 1, 2021 at 1:18:33 AM UTC-5 Chris Angelico wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
No, I would avoid for the sake of readers. Anyway, it can't be used in any public projects until thy drop support for 3.10, which is many years off. Also, I think this question is quite the biased sample on python-ideas. Please consider asking this to less advanced python users, e.g., reddit.com/r/python or learnpython.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)
Yes.
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
No.
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative.
Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!
This PEP has a lot of interesting ideas. I still think that none-aware operators (PEP 505) are an easier, more readable general solution to binding calculated default values to arguments succinctly. I think the problems with this idea include: * The caller cannot explicitly ask for default behaviour except by omitting the parameter. This can be very annoying to set up when the parameter values are provided from other places, e.g., if need_x: # do lots of stuff x = whatever else: # do more x = None f(x=x, ...) # easy, but with this PEP, you would have to find a way to remove x from the parameter list. Typically, removing a parameter from a dynamically-created parameter list is hard. * The function code becomes unreadable if the parameter-setting code is long. Having a long piece of code inside the function after "if parameter is None" is just fine. Having none-aware operators would make such code more succinct. * People nearly always avoid writing code in the parameter defaults themselves, and this new practice adds a lot of cognitive load. E.g., people rarely write: def f(x: int = 1+g()) -> None: ... Parameter lists are already busy enough with parameter names, annotations, and defaults. We don't need to encourage this practice. In short, I think this is a creative idea, a great exploration. While optional parameters are common, and some of them have defaults that are calculated inside the function, my feeling is that people will continue to set their values inside the function. Best, Neil
On Wed, Dec 1, 2021 at 8:57 PM Neil Girdhar <mistersheik@gmail.com> wrote:
On Wednesday, December 1, 2021 at 1:18:33 AM UTC-5 Chris Angelico wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
No, I would avoid for the sake of readers. Anyway, it can't be used in any public projects until thy drop support for 3.10, which is many years off.
Also, I think this question is quite the biased sample on python-ideas. Please consider asking this to less advanced python users, e.g., reddit.com/r/python or learnpython.
I know it is :) I never expected to get an unbiased sample (from anywhere, least of all here).
This PEP has a lot of interesting ideas. I still think that none-aware operators (PEP 505) are an easier, more readable general solution to binding calculated default values to arguments succinctly. I think the problems with this idea include: * The caller cannot explicitly ask for default behaviour except by omitting the parameter. This can be very annoying to set up when the parameter values are provided from other places, e.g., if need_x: # do lots of stuff x = whatever else: # do more x = None f(x=x, ...) # easy, but with this PEP, you would have to find a way to remove x from the parameter list. Typically, removing a parameter from a dynamically-created parameter list is hard.
You can always do that with *args or **kwargs if that's what you need, and that actually DOES represent the concept of "don't pass this argument". Passing None is a hack that doesn't actually mean "don't pass this argument", and if you're depending on that behaviour, it's been locked in as the function's API, which causes problems if None ever becomes a meaningful non-default value. None-aware operators are solving a different problem. They allow you to accept None, but then simplify the replacement of it with some other value. This proposal allows you to leave None out of the equation altogether. And it only works if None isn't a valid value.
* The function code becomes unreadable if the parameter-setting code is long. Having a long piece of code inside the function after "if parameter is None" is just fine. Having none-aware operators would make such code more succinct.
Again, that only works if None is not a valid parameter value, and PEP 505 doesn't generalize to other sentinels at all. If the code is too long, don't put it into the parameter default. But "if x is None: x = []" can easily be replaced with "x=>[]" and the parameters will actually become shorter. ANY feature can be used badly. You can daisy-chain everything into a gigantic expression with horrendous abuses of lambda functions, but that doesn't mean you should. It also doesn't mean that lambda functions are bad :)
* People nearly always avoid writing code in the parameter defaults themselves, and this new practice adds a lot of cognitive load. E.g., people rarely write: def f(x: int = 1+g()) -> None: ... Parameter lists are already busy enough with parameter names, annotations, and defaults. We don't need to encourage this practice.
They don't become any more noisy by being able to use defaults that aren't constant. The main goal is to put the function header in the function header, and the function body in the function body, instead of having the function body doing part of the work of the header :)
In short, I think this is a creative idea, a great exploration. While optional parameters are common, and some of them have defaults that are calculated inside the function, my feeling is that people will continue to set their values inside the function.
In some cases, they certainly will. And that's not a problem. But for the cases where it's better to put it in the header, it's better to have that option than to restrict it for an arbitrary technical limitation. Some languages don't have ANY function defaults. They simply have "optional arguments", where an omitted optional argument will always and only have a specific null value. Python allows you to show what the default REALLY is, and that's an improvement. I want to be able to show what the default is, even if that is defined in a non-constant way. ChrisA
"The caller cannot explicitly ask for default behaviour except by omitting the parameter. This can be very annoying to set up when the parameter values are provided from other places, e.g., if need_x: # do lots of stuff x = whatever else: # do more x = None f(x=x, ...) # easy, but with this PEP, you would have to find a way to remove x from the parameter list. Typically, removing a parameter from a dynamically-created parameter list is hard." If removing a parameter from a dynamically-created parameter list is hard, maybe that's a tools problem. Maybe we should consider crafting better tools for such tasks. It also sounds like a lot of people believe the inspectability of callables that use late-bound defaults is a point of contention. Maybe there should be more inspection hooks than described in the implementation details section of the PEP <https://www.python.org/dev/peps/pep-0671/#id8>? On Wednesday, December 1, 2021 at 3:56:35 AM UTC-6 miste...@gmail.com wrote:
On Wednesday, December 1, 2021 at 1:18:33 AM UTC-5 Chris Angelico wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
No, I would avoid for the sake of readers. Anyway, it can't be used in any public projects until thy drop support for 3.10, which is many years off.
Also, I think this question is quite the biased sample on python-ideas. Please consider asking this to less advanced python users, e.g., reddit.com/r/python or learnpython.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)
Yes.
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
No.
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative.
Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!
This PEP has a lot of interesting ideas. I still think that none-aware operators (PEP 505) are an easier, more readable general solution to binding calculated default values to arguments succinctly. I think the problems with this idea include: * The caller cannot explicitly ask for default behaviour except by omitting the parameter. This can be very annoying to set up when the parameter values are provided from other places, e.g., if need_x: # do lots of stuff x = whatever else: # do more x = None f(x=x, ...) # easy, but with this PEP, you would have to find a way to remove x from the parameter list. Typically, removing a parameter from a dynamically-created parameter list is hard.
* The function code becomes unreadable if the parameter-setting code is long. Having a long piece of code inside the function after "if parameter is None" is just fine. Having none-aware operators would make such code more succinct.
* People nearly always avoid writing code in the parameter defaults themselves, and this new practice adds a lot of cognitive load. E.g., people rarely write: def f(x: int = 1+g()) -> None: ... Parameter lists are already busy enough with parameter names, annotations, and defaults. We don't need to encourage this practice.
In short, I think this is a creative idea, a great exploration. While optional parameters are common, and some of them have defaults that are calculated inside the function, my feeling is that people will continue to set their values inside the function.
Best,
Neil
On Wed, Dec 1, 2021 at 2:17 AM Chris Angelico <rosuav@gmail.com> wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
Currently, I'm thinking "absolutely not".
However, I thought the same about the walrus operator and I now miss not being able to use it in a program that includes support for Python 3.6 and where I have literally dozens of places where I would use it if I could.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
Yes. I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
*Perhaps* if a keyword would be used instead of symbols, I might reconsider. I find the emphasis of trying to cram too much information in single lines of code to be really a burden. Many years ago, I argued very unsuccessfully for using a 'where:' code block for annotations. (To this day, I still believe it would make the code much more readable, at the cost of a slight duplication.) Using what is at first glance a cryptic operator like => for late binding is not helping readability, especially when type annotations are thrown in the mix. Aside: at the same time, I can see how using => instead of lambda as a potential win in readability, including for beginners.
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
Sorry, I'm not interested enough at this point but, given the amount of work you put into this, I decided that the least I could do is provide feedback rather than be a passive reader. André Roberge
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UVOQEK... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Dec 1, 2021 at 10:30 PM André Roberge <andre.roberge@gmail.com> wrote:
On Wed, Dec 1, 2021 at 2:17 AM Chris Angelico <rosuav@gmail.com> wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
Currently, I'm thinking "absolutely not".
However, I thought the same about the walrus operator and I now miss not being able to use it in a program that includes support for Python 3.6 and where I have literally dozens of places where I would use it if I could.
Very fair :)
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
Yes. I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.
Also fair. I'm not a fan of keywords for this sort of thing, since it implies that you could do this: def f(x=defer []): ... dflt = defer [] def f(x=dflt): ... which is a completely different proposal (eg it would be evaluated only when you "touch" that, rather than being guaranteed to be evaluated before the first line of the function body). That's why I want to adorn the equals sign and nothing else.
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
*Perhaps* if a keyword would be used instead of symbols, I might reconsider.
I find the emphasis of trying to cram too much information in single lines of code to be really a burden. Many years ago, I argued very unsuccessfully for using a 'where:' code block for annotations. (To this day, I still believe it would make the code much more readable, at the cost of a slight duplication.) Using what is at first glance a cryptic operator like => for late binding is not helping readability, especially when type annotations are thrown in the mix.
Aside: at the same time, I can see how using => instead of lambda as a potential win in readability, including for beginners.
It's interesting how different people's views go on that sort of thing. It depends a lot on how much people expect to use something. Features you use a lot want to have short notations, features you seldom use are allowed to have longer words.
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
Sorry, I'm not interested enough at this point but, given the amount of work you put into this, I decided that the least I could do is provide feedback rather than be a passive reader.
That's absolutely fine. I don't by any means expect everyone to be able or willing to compile CPython. Feedback is absolutely appreciated, and I asked the question with full expectation of getting a few "No" responses :) Thank you for taking the time to respond. Thoughtful feedback is incredibly helpful. ChrisA
1) If this feature existed in Python 3.11 exactly as described, would you use it?
Absolutely!
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
Nope, I don't think this is an undue cognitive burden. If anything I think the symmetry between the proposed '=>' syntax and the arrow syntax for lambdas in other languages (potentially even in python in the future) reduces the cognitive burden significantly, given the there is an equivalent symmetry with their semantics (in both cases the code is being evaluated later when something is called). Steven gave the following example of a function signature that would be difficult to visually parse if this proposal and arrow lambdas were accepted: def process(func:List->int=>xs=>expression)->int: And while I agree that it does sort of stop you in your tracks when you see this, I think there are a couple of reasons why this is not as big of a problem as it appears. Firstly, I think if you're writing this sort of code, you can improve the legibility a lot by using appropriate spacing and adding parentheses around the lambda arguments (even if they're unnecessary for a lambda with only a single argument, like in Steven's example): def process(func: (List) -> int => (xs) => expression) -> int: Personally I find this much easier to visually parse since I'm used to seeing lambdas in the form: (*args) => expr So my brain just kind of groups `(List) -> int` and `(xs) => expression` into atomic chunks fairly effortlessly. And this is before even mentioning the *massive* cognitive assistance provided by syntax highlighting :) But at this point I would also like to point out that code like this is probably going to be vanishingly rare in the wild. The majority of use-cases for late-bound defaults as far as I can tell are going to be for mutable defaults (and maybe some references to early-bound arguments). I don't like the idea of compromising on what is a very clean and intuitive syntax in 99% of cases to cater to the hypothetical 1% of cases where it admittedly takes a bit more cognitive effort to parse. I agree with the statement Chris made above where he mentioned that the same symbols sometimes have different meanings based on context, and that's okay. For example: def some_func(some_arg: int) -> int: ... def another_func(another_arg: int = some_func(some_arg=3)) -> int: ... In the second function definition above you've got the '=' symbol pulling double-duty both as declaring an argument for a function def and as a keyword arg for a function call in the same statement. I think this is perfectly fine, and I think the same thing is true of the => symbol being used for both lambdas and late-bound arguments, especially given they are semantically related. 3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
(a) is the primary use-case I can see myself using this feature for. I'm not sure what you mean by (b). I can also definitely see some situations where (c) would be dead useful, though I don't think this would come up in the sort of code I write as much as (a). Still, when the use-case for (c) did present itself, I would be *extremely* grateful to have late-bound defaults to reach for.
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
Not relevant, since my strong preference for late-bound default syntax is the '=>' symbol. On Wed, Dec 1, 2021 at 11:50 AM Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Dec 1, 2021 at 10:30 PM André Roberge <andre.roberge@gmail.com> wrote:
On Wed, Dec 1, 2021 at 2:17 AM Chris Angelico <rosuav@gmail.com> wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
Currently, I'm thinking "absolutely not".
However, I thought the same about the walrus operator and I now miss not being able to use it in a program that includes support for Python 3.6 and where I have literally dozens of places where I would use it if I could.
Very fair :)
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
Yes. I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.
Also fair. I'm not a fan of keywords for this sort of thing, since it implies that you could do this:
def f(x=defer []): ...
dflt = defer [] def f(x=dflt): ...
which is a completely different proposal (eg it would be evaluated only when you "touch" that, rather than being guaranteed to be evaluated before the first line of the function body). That's why I want to adorn the equals sign and nothing else.
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
*Perhaps* if a keyword would be used instead of symbols, I might reconsider.
I find the emphasis of trying to cram too much information in single lines of code to be really a burden. Many years ago, I argued very unsuccessfully for using a 'where:' code block for annotations. (To this day, I still believe it would make the code much more readable, at the cost of a slight duplication.) Using what is at first glance a cryptic operator like => for late binding is not helping readability, especially when type annotations are thrown in the mix.
Aside: at the same time, I can see how using => instead of lambda as a potential win in readability, including for beginners.
It's interesting how different people's views go on that sort of thing. It depends a lot on how much people expect to use something. Features you use a lot want to have short notations, features you seldom use are allowed to have longer words.
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
Sorry, I'm not interested enough at this point but, given the amount of work you put into this, I decided that the least I could do is provide feedback rather than be a passive reader.
That's absolutely fine. I don't by any means expect everyone to be able or willing to compile CPython. Feedback is absolutely appreciated, and I asked the question with full expectation of getting a few "No" responses :)
Thank you for taking the time to respond. Thoughtful feedback is incredibly helpful.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WTU355... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Dec 01, 2021 at 12:26:33PM +0000, Matt del Valle wrote:
If anything I think the symmetry between the proposed '=>' syntax and the arrow syntax for lambdas in other languages (potentially even in python in the future) reduces the cognitive burden significantly, given the there is an equivalent symmetry with their semantics (in both cases the code is being evaluated later when something is called).
There is not as much symmetry as you might think between a hypothetical lambda arrow and the proposed late-bound default arrow. arg => arg + 1 # lambda arg=>expr # late-bound default The first case is (or could be some day...) an actual expression that returns a function object, which we will explicitly call at some point. Or at least pass the function to another function, which will call it. But the late bound default is not an expression, it is a declaration. It declares the default value used for arg. We don't have to call anything to get access to the default value. It just shows up when we access the parameter without providing an argument for it. We certainly don't need to call arg explicitly to evaluate the default. It may be that behind the scenes the default expression is stored as a callable function which the interpreter calls. (I believe that list comprehensions do something similar.) But that's an implementation detail that can change: it might just as well store the source code as a string, and pass it to eval(). Or use some other mechanism that I'm not clever enough to think of, so I shall just call "deepest black magic". There is no *neccessity* for the late-bound default to be a hidden function, and it is certainly not part of the semantics of late-bound defaults. Just the implementation. If you disagree, and still think that the symmetry is powerful enough to use the same syntax for both lambdas and default arguments, well, how about if we *literally* do that? def function(spam=expression, # regular default lambda eggs: expression, # late-bound default ) Heh, perhaps the symmetry is not that strong after all :-) -- Steve
On Thu, Dec 2, 2021 at 1:30 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Wed, Dec 01, 2021 at 12:26:33PM +0000, Matt del Valle wrote:
If anything I think the symmetry between the proposed '=>' syntax and the arrow syntax for lambdas in other languages (potentially even in python in the future) reduces the cognitive burden significantly, given the there is an equivalent symmetry with their semantics (in both cases the code is being evaluated later when something is called).
It may be that behind the scenes the default expression is stored as a callable function which the interpreter calls. (I believe that list comprehensions do something similar.) But that's an implementation detail that can change: it might just as well store the source code as a string, and pass it to eval().
In my reference implementation, there is no object that stores it; it's simply part of the function. A good parallel is the if/else expression: x = float("inf") if z == 0 else y/z Is there an object that represents the if/else expression (or the deferred "y/z" part)? No, although it could be implemented that way if you chose: x = iff(z == 0, lambda: y/z, lambda: float("inf")) But just because you CAN use a function to simulate this behaviour, that doesn't mean that it's inherently a function. So I agree with Steve that the parallel is quite weak. That said, though, I still like the arrow notation, not because of any parallel with a lambda function, but because of the parallel with assignment. (And hey. You're allowed to think of things in any way you like. I'm not forcing you to interpret everything using the same justifications I do.)
Or use some other mechanism that I'm not clever enough to think of, so I shall just call "deepest black magic".
Remind me some time to use some slightly shallower black magic.
There is no *neccessity* for the late-bound default to be a hidden function, and it is certainly not part of the semantics of late-bound defaults. Just the implementation.
If you disagree, and still think that the symmetry is powerful enough to use the same syntax for both lambdas and default arguments, well, how about if we *literally* do that?
def function(spam=expression, # regular default lambda eggs: expression, # late-bound default )
Heh, perhaps the symmetry is not that strong after all :-)
That..... is hilariously confusing. I like it. Just not EVER in the language :) ChrisA
I like the => syntax and would use it probably 80% of the time for mutable defaults. I don't think it causes cognitive load. I especially disagree with the argument that it looks like arbitrary symbols and adds more 'line noise' than a keyword. My eye picks it up as a distinct arrow rather than a random assembly of punctuation. Maybe it's not as hard as people claim to visually parse strings of punctuation that make little pictures ¯\_(ツ)_/¯ I also disagree that it makes Python harder to teach or learn. You *have to* teach students about early binding anyway. That's the hard part. Explaining how parameter binding works so that they can avoid the pit traps is the hard part. Then you either show them a verbose and clumsy work around of checking for a sentinel or you show them a cute little arrow. I think people are missing the overall point of doing in the function header what belongs in the function header so that it doesn't spill into the body. My favorite alternative is ?= if people think => and -> are getting overly loaded. What I really don't like is @param=[] because it puts the emphasis on the parameter name rather than the act of binding. Not only does it make it look like @param is a special kind of variable, it also mimics the *args and **kwargs syntax which makes them seem related. On Wed, Dec 1, 2021, 8:36 PM Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Dec 2, 2021 at 1:30 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Wed, Dec 01, 2021 at 12:26:33PM +0000, Matt del Valle wrote:
If anything I think the symmetry between the proposed '=>' syntax and the arrow syntax for lambdas in other languages (potentially even in python in the future) reduces the cognitive burden significantly, given the there is an equivalent symmetry with their semantics (in both cases the code is
being
evaluated later when something is called).
It may be that behind the scenes the default expression is stored as a callable function which the interpreter calls. (I believe that list comprehensions do something similar.) But that's an implementation detail that can change: it might just as well store the source code as a string, and pass it to eval().
In my reference implementation, there is no object that stores it; it's simply part of the function. A good parallel is the if/else expression:
x = float("inf") if z == 0 else y/z
Is there an object that represents the if/else expression (or the deferred "y/z" part)? No, although it could be implemented that way if you chose:
x = iff(z == 0, lambda: y/z, lambda: float("inf"))
But just because you CAN use a function to simulate this behaviour, that doesn't mean that it's inherently a function.
So I agree with Steve that the parallel is quite weak. That said, though, I still like the arrow notation, not because of any parallel with a lambda function, but because of the parallel with assignment.
(And hey. You're allowed to think of things in any way you like. I'm not forcing you to interpret everything using the same justifications I do.)
Or use some other mechanism that I'm not clever enough to think of, so I shall just call "deepest black magic".
Remind me some time to use some slightly shallower black magic.
There is no *neccessity* for the late-bound default to be a hidden function, and it is certainly not part of the semantics of late-bound defaults. Just the implementation.
If you disagree, and still think that the symmetry is powerful enough to use the same syntax for both lambdas and default arguments, well, how about if we *literally* do that?
def function(spam=expression, # regular default lambda eggs: expression, # late-bound default )
Heh, perhaps the symmetry is not that strong after all :-)
That..... is hilariously confusing. I like it. Just not EVER in the language :)
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/AQWFM7... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Dec 01, 2021 at 09:58:11PM -0600, Abe Dillon wrote:
My favorite alternative is ?= if people think => and -> are getting overly loaded. What I really don't like is @param=[] because it puts the emphasis on the parameter name rather than the act of binding. Not only does it make it look like @param is a special kind of variable, it also mimics the *args and **kwargs syntax which makes them seem related.
But it *is* a special kind of parameter: it is a parameter that uses late binding for the default value, instead of early binding. Putting the emphasis on the parameter name is entirely appropriate. Late bound parameters don't have a different sort of binding to other parameters. There aren't two kinds of binding: 1. Parameters with no default, and parameters with early bound default, use the same old type of name binding that other local variables use (local slots in CPython, maybe a dict for other implementations); 2. and parameters with late bound defaults use a different sort of name binding, and we need a different syntax to reflect that. That is wrong: there is only one sort of binding in Python (although it can have a few different storage mechanisms: slots, cells and dict namespaces, maybe even others). The storage mechanism is irrelevant. It's all just name binding, and the implementation details are handled by the interpreter. By the time the function body is entered, all the parameters have been bound to a value. (If there are parameters that don't have a value, the interpreter raises an exception and you never enter the function body.) It doesn't matter where those values came from, whether they were passed in by the caller, or early bound defaults loaded from the cache, or late bound defaults freshly evaluated. The value is bound to the parameter in exactly the same way, and you cannot determine where that value came from. -- Steve
On Fri, Dec 3, 2021 at 2:17 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Wed, Dec 01, 2021 at 09:58:11PM -0600, Abe Dillon wrote:
My favorite alternative is ?= if people think => and -> are getting overly loaded. What I really don't like is @param=[] because it puts the emphasis on the parameter name rather than the act of binding. Not only does it make it look like @param is a special kind of variable, it also mimics the *args and **kwargs syntax which makes them seem related.
But it *is* a special kind of parameter: it is a parameter that uses late binding for the default value, instead of early binding.
Putting the emphasis on the parameter name is entirely appropriate.
Late bound parameters don't have a different sort of binding to other parameters. There aren't two kinds of binding:
1. Parameters with no default, and parameters with early bound default, use the same old type of name binding that other local variables use (local slots in CPython, maybe a dict for other implementations);
2. and parameters with late bound defaults use a different sort of name binding, and we need a different syntax to reflect that.
That is wrong: there is only one sort of binding in Python (although it can have a few different storage mechanisms: slots, cells and dict namespaces, maybe even others). The storage mechanism is irrelevant. It's all just name binding, and the implementation details are handled by the interpreter.
In these examples, is the name binding any different? x = 1 x += 1 (x := 1) for x in [1]: pass They use different equals signs, but once the assignment happens, it's the exact same name binding. But exactly what IS different about them? Is the name 'x' different? The value 1? The name binding itself?
By the time the function body is entered, all the parameters have been bound to a value. (If there are parameters that don't have a value, the interpreter raises an exception and you never enter the function body.) It doesn't matter where those values came from, whether they were passed in by the caller, or early bound defaults loaded from the cache, or late bound defaults freshly evaluated. The value is bound to the parameter in exactly the same way, and you cannot determine where that value came from.
Yes, that is correct. And the parameter is the exact same kind of thing either way, too, so adorning the parameter name is wrong too. And the expression being evaluated is still just an expression, so adorning THAT is wrong too. There's nothing about late-bound defaults that is fundamentally different from the rest of Python, so there's no obvious "this is the bit to adorn" thing. In contrast, *args and **kwargs actually change the parameter that gets received: one gives a sequence, the other a mapping, whereas leaving off both markers gives you a single value. ChrisA
Steven D'Aprano "> My favorite alternative is ?= if people think => and -> are getting
overly loaded. What I really don't like is @param=[] because it puts the emphasis on the parameter name rather than the act of binding. Not only does it make it look like @param is a special kind of variable, it also mimics the *args and **kwargs syntax which makes them seem related.
But it *is* a special kind of parameter: it is a parameter that uses late binding for the default value, instead of early binding. Putting the emphasis on the parameter name is entirely appropriate. Late bound parameters don't have a different sort of binding to other parameters. There aren't two kinds of binding:" Again: the important part is the *binding*, not the name of the parameter. Right now, the order of reading something like `def load_tweets(... hash_tags: List[str] = None, ...)` goes like this: 1) there's a parameter named 'hash_tags' 2) it's a list of strings 3) it has a default value (bound at function declaration) 4) the default value is None With `def load_tweets(... @hash_tags: List[str] = [], ...)` it goes like: 1) There's some parameter bound late 2) It's a parameter named 'hash_tags' 3) It's a list of strings 4) It has a default value (of course because of information we learned 3 steps ago) 5) The default value is an empty list With `def load_tweets(... hash_tags: List[str] => [], ...)` it goes like: 1) there's a parameter named 'hash_tags' 2) it's a list of strings 3) it has a default value (bound at call-time) 4) the default value is None Notice how it makes way more sense to put related parts closer together rather than saying "here's a little information. Now put a pin in that: we'll get back to it later on". To drive the point home: The @ and = signs are connected, but not visually, which is confusing. You couldn't write @param without an '=' somewhere down the line: `def load_tweets(... @hash_tags, ...) # Error!` I probably should have been more clear about what I meant when I said "it make[s] it look like @param is a special kind of variable, it also mimics the *args and **kwargs syntax which makes them seem related." Of course the way the parameter is bound *is* special making it, in the ways you pointed out, a "special kind of variable", however, again: the specialness is all about the binding of the variable, and: unlike *args and **kwargs, @param doesn't imply any specific type (in the 'type(kwargs)' sense) for the variable. We know that args and kwargs are an iterable and mapping, respectively. The other question is: how important is the information that a parameter is late-binding. I've always been against the coding standard of making enums and constants ALL_CAPS because it makes code shouty and obnoxious just as it makes any other communication shouty and obnoxious. It doesn't seem like the fact that something is an enum or constant is important enough to warrant anything more than some syntax coloring from my IDE (if that). People always point out that the fact that something is an enum or constant is, in fact, information, and how else would they possibly communicate that information? The answer is: you communicate it by declaring an enum or constant and stop shouting at me! There's other information I you could put in your variable names that most sane people don't. You don't always append the name of the class of an object to the variable's name. You say 'hash_tags' not 'hash_tags_list_of_strings'. In the same vein, I also don't see the need here to make a big fuss about the binding behavior of a parameter such that it *must* be up-front ahead of the name, annotation, and other binding info. It may seem like a big deal now because it's new and scary so we must prefix it with "HERE_BE_DRAGONS!!!", but years down the line when late-binding params are common place, it'll be a wart. On Thursday, December 2, 2021 at 9:16:33 AM UTC-6 Steven D'Aprano wrote:
On Wed, Dec 01, 2021 at 09:58:11PM -0600, Abe Dillon wrote:
My favorite alternative is ?= if people think => and -> are getting overly loaded. What I really don't like is @param=[] because it puts the emphasis on the parameter name rather than the act of binding. Not only does it make it look like @param is a special kind of variable, it also mimics the *args and **kwargs syntax which makes them seem related.
But it *is* a special kind of parameter: it is a parameter that uses late binding for the default value, instead of early binding.
Putting the emphasis on the parameter name is entirely appropriate.
Late bound parameters don't have a different sort of binding to other parameters. There aren't two kinds of binding:
1. Parameters with no default, and parameters with early bound default, use the same old type of name binding that other local variables use (local slots in CPython, maybe a dict for other implementations);
2. and parameters with late bound defaults use a different sort of name binding, and we need a different syntax to reflect that.
That is wrong: there is only one sort of binding in Python (although it can have a few different storage mechanisms: slots, cells and dict namespaces, maybe even others). The storage mechanism is irrelevant. It's all just name binding, and the implementation details are handled by the interpreter.
By the time the function body is entered, all the parameters have been bound to a value. (If there are parameters that don't have a value, the interpreter raises an exception and you never enter the function body.) It doesn't matter where those values came from, whether they were passed in by the caller, or early bound defaults loaded from the cache, or late bound defaults freshly evaluated. The value is bound to the parameter in exactly the same way, and you cannot determine where that value came from.
-- Steve _______________________________________________ Python-ideas mailing list -- python...@python.org To unsubscribe send an email to python-id...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python...@python.org/message/LYRLA3TH4... <https://mail.python.org/archives/list/python-ideas@python.org/message/LYRLA3TH47H4EKOR3NCIUZKXJOICJ5R7/> Code of Conduct: http://python.org/psf/codeofconduct/
On 2021-12-01 18:35, Chris Angelico wrote:
In my reference implementation, there is no object that stores it; it's simply part of the function. A good parallel is the if/else expression:
x = float("inf") if z == 0 else y/z
Is there an object that represents the if/else expression (or the deferred "y/z" part)? No, although it could be implemented that way if you chose:
This is not a good parallel. There is nothing deferred there. The entire line is evaluated when it is encountered and you get a result and no part of the if/else expression can ever impact anything else again unless, via some external control flow, execution returns and re-executes the entire line. That is not comparable to a function default, which is STORED and evaluated later independently of the context in which it was originally written (i.e., the function default is re-executed but the function definition itself is not re-executed). The ternary expression vanishes without a trace by the next line, leaving only its evaluated result. There would be no use in being able to access some part of it, since the whole (i.e., the ternary expression) is completely finished by the time you would be able to access it. This is not the case with a function definition. The function definition leaves behind a function object, and that function object needs to "know" about the late-bound default as an independent entity so that it can be evaluated later. It is bad for the function to store that late-bound default only in some private format for its exclusive future use without providing any means for other code to access it as a first-class value. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Thu, Dec 2, 2021 at 6:27 PM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2021-12-01 18:35, Chris Angelico wrote:
In my reference implementation, there is no object that stores it; it's simply part of the function. A good parallel is the if/else expression:
x = float("inf") if z == 0 else y/z
Is there an object that represents the if/else expression (or the deferred "y/z" part)? No, although it could be implemented that way if you chose:
This is not a good parallel. There is nothing deferred there. The entire line is evaluated when it is encountered and you get a result and no part of the if/else expression can ever impact anything else again unless, via some external control flow, execution returns and re-executes the entire line. That is not comparable to a function default, which is STORED and evaluated later independently of the context in which it was originally written (i.e., the function default is re-executed but the function definition itself is not re-executed).
The ternary expression vanishes without a trace by the next line, leaving only its evaluated result. There would be no use in being able to access some part of it, since the whole (i.e., the ternary expression) is completely finished by the time you would be able to access it. This is not the case with a function definition. The function definition leaves behind a function object, and that function object needs to "know" about the late-bound default as an independent entity so that it can be evaluated later. It is bad for the function to store that late-bound default only in some private format for its exclusive future use without providing any means for other code to access it as a first-class value.
That's exactly why it's such a close parallel. The late-evaluated default is just code, nothing else. It's not "stored" in any way - it is evaluated as part of the function beginning execution. There is no "first class object" for a late-evaluated default any more than there is one for the "y/z" part of the ternary. ChrisA
On 2021-12-01 23:36, Chris Angelico wrote:
That's exactly why it's such a close parallel. The late-evaluated default is just code, nothing else. It's not "stored" in any way - it is evaluated as part of the function beginning execution.
But it IS stored! There is no way for it to be evaluated without it being stored! I know we're talking past each other here but it is quite obvious that something has to be stored if it is going to be evaluated later. You can say that it is "just code" but that doesn't change the fact that that code has to be stored. You can say that it is just prepended to the function body but that's still storing it. That is still not parallel to a ternary operator in which no part of the expression is EVER re-executed unless control flow causes execution to return to that same source code line and re-execute it as a whole. Actually this raises a question that maybe was answered in the earlier thread but if so I forgot: if a function has a late-bound default, will the code to evaluate it be stored as part of the function's code object? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Thu, Dec 2, 2021 at 6:59 PM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2021-12-01 23:36, Chris Angelico wrote:
That's exactly why it's such a close parallel. The late-evaluated default is just code, nothing else. It's not "stored" in any way - it is evaluated as part of the function beginning execution.
But it IS stored! There is no way for it to be evaluated without it being stored!
I know we're talking past each other here but it is quite obvious that something has to be stored if it is going to be evaluated later. You can say that it is "just code" but that doesn't change the fact that that code has to be stored. You can say that it is just prepended to the function body but that's still storing it. That is still not parallel to a ternary operator in which no part of the expression is EVER re-executed unless control flow causes execution to return to that same source code line and re-execute it as a whole.
I'm not sure I understand you here. How is the late-bound default "stored" when one side of a ternary is "not stored"?
Actually this raises a question that maybe was answered in the earlier thread but if so I forgot: if a function has a late-bound default, will the code to evaluate it be stored as part of the function's code object?
Yes. To be precise, it is part of the code object's co_code attribute - the bytecode (or wordcode if you prefer) of the function. Here's how a ternary if looks:
def f(n): ... return 0 if n == 0 else 42/n ... dis.dis(f) 2 0 LOAD_FAST 0 (n) 2 LOAD_CONST 1 (0) 4 COMPARE_OP 2 (==) 6 POP_JUMP_IF_FALSE 6 (to 12) 8 LOAD_CONST 1 (0) 10 RETURN_VALUE >> 12 LOAD_CONST 2 (42) 14 LOAD_FAST 0 (n) 16 BINARY_TRUE_DIVIDE 18 RETURN_VALUE
The "42/n" part is stored in f.__code__.co_code as the part that says "LOAD_CONST 42, LOAD_FAST n, BINARY_TRUE_DIVIDE". It's not an object. It's just code - three instructions. Here's how (in the reference implementation - everything is subject to change) a late-bound default looks:
def f(x=>[]): print(x) ... dis.dis(f) 1 0 QUERY_FAST 0 (x) 2 POP_JUMP_IF_TRUE 4 (to 8) 4 BUILD_LIST 0 6 STORE_FAST 0 (x) >> 8 LOAD_GLOBAL 0 (print) 10 LOAD_FAST 0 (x) 12 CALL_FUNCTION 1 14 POP_TOP 16 LOAD_CONST 0 (None) 18 RETURN_VALUE
The "=>[]" part is stored in f.__code__.co_code as the part that says "QUERY_FAST x, and if false, BUILD_LIST, STORE_FAST x". It's not an object. It's four instructions in the bytecode. In both cases, no part of the expression is ever re-executed. I'm not understanding the distinction here. Can you explain further please? ChrisA
On Thu, Dec 2, 2021 at 3:33 AM Chris Angelico <rosuav@gmail.com> wrote:
But it IS stored! There is no way for it to be evaluated
without it
being stored!
I'm not sure I understand you here. How is the late-bound default "stored" when one side of a ternary is "not stored"?
def foo(a): ... b = a + 1 ... print(b) ... foo.__code__ <code object foo at 0x7f167e539710, file "<ipython-input-7-3c44060a0872>",
This seems like dishonest argument. I'm not even sure what point you think it is making. Every time I write a function, everything the function does needs to be STORED. The body is *stored* in the .__code__ attribute. Other things are stored in .__annotations__ and elsewhere. A function is an OBJECT, and everything about it has to be attributes of that object. line 1> A late binding isn't that one thing about a function that never gets stored, but floats in the ether magically ready to operate on a function call by divine intervention. It HAS TO describe *something* attached to the function object, doing *something* by some means. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Fri, Dec 3, 2021 at 12:43 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
On Thu, Dec 2, 2021 at 3:33 AM Chris Angelico <rosuav@gmail.com> wrote:
But it IS stored! There is no way for it to be evaluated without it being stored!
I'm not sure I understand you here. How is the late-bound default "stored" when one side of a ternary is "not stored"?
This seems like dishonest argument. I'm not even sure what point you think it is making.
My point was a response to a claim that a late-bound default fundamentally has to be stored somewhere, yet half of a ternary conditional isn't. My point was that there is no difference, and they are both code. Please examine the surrounding context and respond in more detail, rather than calling my argument "dishonest". ChrisA
On 2021-12-02 00:31, Chris Angelico wrote:
Here's how a ternary if looks:
>def f(n): ... return 0 if n == 0 else 42/n ... >dis.dis(f) 2 0 LOAD_FAST 0 (n) 2 LOAD_CONST 1 (0) 4 COMPARE_OP 2 (==) 6 POP_JUMP_IF_FALSE 6 (to 12) 8 LOAD_CONST 1 (0) 10 RETURN_VALUE >> 12 LOAD_CONST 2 (42) 14 LOAD_FAST 0 (n) 16 BINARY_TRUE_DIVIDE 18 RETURN_VALUE
The "42/n" part is stored in f.__code__.co_code as the part that says "LOAD_CONST 42, LOAD_FAST n, BINARY_TRUE_DIVIDE". It's not an object. It's just code - three instructions.
Here's how (in the reference implementation - everything is subject to change) a late-bound default looks:
>def f(x=>[]): print(x) ... >dis.dis(f) 1 0 QUERY_FAST 0 (x) 2 POP_JUMP_IF_TRUE 4 (to 8) 4 BUILD_LIST 0 6 STORE_FAST 0 (x) >> 8 LOAD_GLOBAL 0 (print) 10 LOAD_FAST 0 (x) 12 CALL_FUNCTION 1 14 POP_TOP 16 LOAD_CONST 0 (None) 18 RETURN_VALUE
The "=>[]" part is stored in f.__code__.co_code as the part that says "QUERY_FAST x, and if false, BUILD_LIST, STORE_FAST x". It's not an object. It's four instructions in the bytecode.
In both cases, no part of the expression is ever re-executed. I'm not understanding the distinction here. Can you explain further please?
Your explanation exactly shows how it IS re-executed. I'm not totally clear on this disassembly since this is new behavior, but if I understand right, BUILD_LIST is re-executing the expression `[]` and STORE_FAST is re-assigning it to x. The expression `[]` is syntactically present in the function definition but its execution has been shoved into the function body where it may be re-executed many times (any time the function is called without passing a value). What do you mean when you say it is not re-executed? Is it not the case that `[]` is syntactically present in the `def` line (which is executed only once, and whose bytecode is not shown) yet its implementation (BUILD_LIST) is in the function body, which may be executed many times? How is the BUILD_LIST opcode there not being re-executed on later calls of the function? Perhaps what you are saying is that what is stored is not the literal string "[]" but bytecode that implements it? That distinction is meaningless to me. The point is that you wrote `[]` once, in a line that is executed once (the function definition itself), but the `[]` is executed many times, separately from the function definition. Another way to put it is, again, the examples are not parallel. In your first example the ternary expression is (syntactically) part of the function BODY, so of course it appears in the disassembly. In your second example, the late-bound default is not in the body, it is in the signature. The disassembly is only showing the bytecode of the function body, but the late-bound default is syntactically enmeshed with the function DEFINITION. So I don't want the late-bound default code to be re-executed unless the function is re-defined. Or, if you're going to store that `x = []`, I don't want it to be "stored" by just shoving it in the function body, I want it to be some kind of separate object. Here's an example that may make my point clearer. some_function(x+1, lambda x: x+2) This is our best current approximation to some kind of "late evaluation". The first argument, `x+1`, is evaluated before the function is called, and the function gets only the result. The second argument is also of course evaluated before the function is called, but what is evaluated is a lambda; the `x+2` is not evaluated. But the idea of "evaluate x+2 later" is encapsulated in a function object. Without knowing anything about `some_function`, I still know that there is no way it can ever evaluate `x+2` without going through the interface of that function object. There is no way to pass, as the second argument to `some_function` some kind of amorphous "directive" that says "evaluate x+2 later" without wrapping that directive up into some kind of Python object. Perhaps the fundamental point that I feel you're missing about my position is that a ternary expression does not have a "definition" and a "body" that are executed at separate times. There is just the whole ternary expression and it is evaluated all at once. Thus there can be no parallel with functions, which do have a separation between definition time and call time. Indeed, it's because these are separate that late vs. early binding of defaults is even a meaningful concept for functions (but not for ternary expressions). So what I am saying is if I see a line like this: def f(a=x+1, b@=x+2): The x+2 is syntactically embedded within the `def` line, that is, the function definition (not the body). Thus there are only two kinds of semantics that are going to make me happy: 1) That `x+2`(or any bytecode derived from it, etc.) will never be re-executed unless the program execution again reaches the line with the `def f` (which is what we have with early-bound defaults) 2) A Python object is created that encapsulates the expression `x+2` somehow and defines what it means to "evaluate it in a context" (e.g., by referencing local variables in the scope where it is evaluated) Maybe another way to say this is just "I want any kind of late-bound default to really be an early-bound default whose value is some object that provides a way to evaluate it later". (I'm trying to think of different ways to say this because it seems what I'm saying is not clear to you. :-) -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Fri, Dec 3, 2021 at 9:26 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2021-12-02 00:31, Chris Angelico wrote:
Here's how a ternary if looks:
>>def f(n): ... return 0 if n == 0 else 42/n ... >>dis.dis(f) 2 0 LOAD_FAST 0 (n) 2 LOAD_CONST 1 (0) 4 COMPARE_OP 2 (==) 6 POP_JUMP_IF_FALSE 6 (to 12) 8 LOAD_CONST 1 (0) 10 RETURN_VALUE >> 12 LOAD_CONST 2 (42) 14 LOAD_FAST 0 (n) 16 BINARY_TRUE_DIVIDE 18 RETURN_VALUE
The "42/n" part is stored in f.__code__.co_code as the part that says "LOAD_CONST 42, LOAD_FAST n, BINARY_TRUE_DIVIDE". It's not an object. It's just code - three instructions.
Here's how (in the reference implementation - everything is subject to change) a late-bound default looks:
>>def f(x=>[]): print(x) ... >>dis.dis(f) 1 0 QUERY_FAST 0 (x) 2 POP_JUMP_IF_TRUE 4 (to 8) 4 BUILD_LIST 0 6 STORE_FAST 0 (x) >> 8 LOAD_GLOBAL 0 (print) 10 LOAD_FAST 0 (x) 12 CALL_FUNCTION 1 14 POP_TOP 16 LOAD_CONST 0 (None) 18 RETURN_VALUE
The "=>[]" part is stored in f.__code__.co_code as the part that says "QUERY_FAST x, and if false, BUILD_LIST, STORE_FAST x". It's not an object. It's four instructions in the bytecode.
In both cases, no part of the expression is ever re-executed. I'm not understanding the distinction here. Can you explain further please?
Your explanation exactly shows how it IS re-executed. I'm not totally clear on this disassembly since this is new behavior, but if I understand right, BUILD_LIST is re-executing the expression `[]` and STORE_FAST is re-assigning it to x. The expression `[]` is syntactically present in the function definition but its execution has been shoved into the function body where it may be re-executed many times (any time the function is called without passing a value).
Ah, I think I get you. The problem is that code is in the def line but is only executed when the function is called, is that correct? Because the code would be "re-executed" just as much if it were written in the function body. It's executed (at most) once for each call to the function, just like the ternary's side is. I suppose that's a consideration, but it's not nearly as strong in practice as you might think. A lot of people aren't even aware of the difference between compilation time and definition time (even people on this list have made that mistake). Function default args are executed when the function is defined, not when it's called, and that's something that changes with this proposal; but there are many other subtleties to execution order and timing that don't really matter in practice. Perhaps the key point here is to consider function decorators. We could avoid them altogether: def f(): @deco def g(x): ... def h(x): ... h = deco(h) But as well as having the name replication problem, this buries important information down in the body of the surrounding code, rather than putting it at the signature of g/h where it belongs. Even though, semantically, this is actually part of the body of f, we want to be able to read it as part of the signature of g. Logically and conceptually, it is part of the signature. Now compare these two: def f2(): _SENTINEL = object() def g(x=_SENTINEL): if x is _SENTINEL: x = [] ... def h(x=>[]): ... Which one has its signature where its signature belongs? Yes, semantically, the construction of the empty list happens at function call time, not at definition time. But what you're saying is: if there are no args passed, behave as if a new empty list was passed. That's part of the signature. In neither case will you find an object representing the expression [] in the function's signature, because that's not an object, it's an instruction to build an empty list. In the case of g, you can find a meaningless and useless object stashed away in __defaults__, but that doesn't tell you anything about the true behaviour of the function. At least in the case of h, you can find the descriptive string "[]" stashed there, which can tell a human what's happening. ChrisA
Ah, I think I get you. The problem is that code is in the def line but is only executed when the function is called, is that correct? Because the code would be "re-executed" just as much if it were written in the function body. It's executed (at most) once for each call to the function, just like the ternary's side is.
Hmmm, is that really the root of our disagreement? That's surprising. :-) Anyway, yes. The function body and the function signature are two different things. Function definition time and function call time are two different things.
I suppose that's a consideration, but it's not nearly as strong in practice as you might think. A lot of people aren't even aware of the difference between compilation time and definition time (even people on this list have made that mistake). Function default args are executed when the function is defined, not when it's called, and that's something that changes with this proposal; but there are many other subtleties to execution order and timing that don't really matter in practice.
That is a surprising statement to me. I consider the distinction between definition time and call time to be quite fundamental to Python. At a minimum one has to understand that the function body isn't executed when the function is defined (i.e., you have to call it later to "activate" the defined behavior). This is something I've seen beginner programmers struggle with. Understanding the difference between def time and call time is also important for understanding how decorators work. It's why most decorators have a structure that is confusing to many newcomers, where the decorator itself is a function that defines another function and then returns it. It has to be this way because of the distinction between the time the decorated function is defined (which is when the decorator is called) and the time it is called (which is when the "inner function" that the decorator defines is called). Or, to put it bluntly, I think people just need to understand the difference between def time and call time to understand Python's semantics. Of course even experienced programmers may sometimes get mixed up in a particular case, but I fully reject the idea that this distinction is some kind of obscure technicality that we can ignore or blur by including call-time behavior in the def-time signature. The distinction is fundamental to how functions work in Python.
Perhaps the key point here is to consider function decorators. We could avoid them altogether:
def f(): @deco def g(x): ...
def h(x): ... h = deco(h)
But as well as having the name replication problem, this buries important information down in the body of the surrounding code, rather than putting it at the signature of g/h where it belongs. Even though, semantically, this is actually part of the body of f, we want to be able to read it as part of the signature of g. Logically and conceptually, it is part of the signature. Now compare these two:
Decorators aren't part of the signature of the function they decorate. deco is not part of the signature of g. A decorator is sort of like a "transform" applied to the function after it is defined. It's true we have decorators and like them because they allow us to put that transform up front rather than at the bottom, but that's not because of anything about signatures. It's just because some transforms are nice to know about up front.
def f2(): _SENTINEL = object() def g(x=_SENTINEL): if x is _SENTINEL: x = [] ...
def h(x=>[]): ...
Which one has its signature where its signature belongs? Yes, semantically, the construction of the empty list happens at function call time, not at definition time. But what you're saying is: if there are no args passed, behave as if a new empty list was passed. That's part of the signature.
Again, I disagree. Any behavior that happens at call time is not part of the signature. The signature specifies the function's parameter names and optionally maps any or all of them to objects to be used in case they are not passed at call time.
In neither case will you find an object representing the expression [] in the function's signature, because that's not an object, it's an instruction to build an empty list. In the case of g, you can find a meaningless and useless object stashed away in __defaults__, but that doesn't tell you anything about the true behaviour of the function. At least in the case of h, you can find the descriptive string "[]" stashed there, which can tell a human what's happening.
You say "In the case of g, you can find a meaningless and useless object stashed away in __defaults__, but that doesn't tell you anything about the true behaviour of the function." Yes. And that's fine. As I've mentioned in other messages in these threads, I am totally fine with the idea that there is stuff in the function signature that does not completely characterize the function's behavior. The place for the function author to explain what the functions defaults mean is in documentation, and the place for the user to learn what the defaults mean is also the documentation. I think again that another area of our disagreement is the relationship between the function signature and "the true behavior of the function". To me the signature is to the "true behavior" as the cover of a thick tome to its contents. The "true behavior" of a function can be arbitrarily complex in myriad ways which we can never dream of communicating via the signature. The signature cannot hope to even begin to convey more than an echo of a whisper of a shadowy reflection of "the true behavior of the function". We should not worry ourselves in the slightest about the fact that "the true behavior of the function" may not be transparently represented in the signature. Sure, we should choose good names for the parameters, but that's about all I expect from a function signature. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On 02/12/2021 07:05, Brendan Barnwell wrote:
On 2021-12-01 18:35, Chris Angelico wrote:
In my reference implementation, there is no object that stores it; it's simply part of the function. A good parallel is the if/else expression:
x = float("inf") if z == 0 else y/z
Is there an object that represents the if/else expression (or the deferred "y/z" part)? No, although it could be implemented that way if you chose:
This is not a good parallel. There is nothing deferred there. The entire line is evaluated when it is encountered and you get a result and no part of the if/else expression can ever impact anything else again unless, via some external control flow, execution returns and re-executes the entire line. That is not comparable to a function default, which is STORED and evaluated later independently of the context in which it was originally written (i.e., the function default is re-executed but the function definition itself is not re-executed).
The ternary expression vanishes without a trace by the next line, leaving only its evaluated result. There would be no use in being able to access some part of it, It could be useful for debugging. Best wishes Rob Cliffe since the whole (i.e., the ternary expression) is completely finished by the time you would be able to access it. This is not the case with a function definition. The function definition leaves behind a function object, and that function object needs to "know" about the late-bound default as an independent entity so that it can be evaluated later. It is bad for the function to store that late-bound default only in some private format for its exclusive future use without providing any means for other code to access it as a first-class value.
On Wed, Dec 01, 2021 at 12:26:33PM +0000, Matt del Valle wrote:
Steven gave the following example of a function signature that would be difficult to visually parse if this proposal and arrow lambdas were accepted:
def process(func:List->int=>xs=>expression)->int:
And while I agree that it does sort of stop you in your tracks when you see this, I think there are a couple of reasons why this is not as big of a problem as it appears.
I totally agree with you that, for even moderately experienced Pythonistas, it is possible to parse that. It's not literally ambiguous syntax that cannot be resolved. I am confident that the parser will be able to work it out just fine :-) I care more about the poor reader who may not be a moderately experienced Pythonista, and will be trying to work out why there are two different arrow symbols with four different meanings: * function return annotation * typing.Callable type annotation * lambda alternative syntax * late-bound defaults I'm sure that people will learn the many uses of the arrow symbols. After all, people learn Perl and APL :-) Closer to home, we also learn all the many different uses of the star symbol `*`. Yes, spaces will help the reader *parse* the pieces of the function signature. But spaces doesn't help the reader decipher and remember the different meanings of the arrows. Also the use of extra spaces goes against the usual style guides that we *don't* use spaces between the colon and the annotation, or the equals sign and the default. For the sake of discussion, I've been using Chris' arrow symbol, but that doesn't mean I've warmed to it. Aside from the other issues, it is the wrong way around: parameter => expression implies moving the parameter into the expression, which is the wrong way around. The expression moves into the parameter. E.g. in R, you can write assignment with an equals sign, or an arrow, but the arrow points from the value to the variable: > x <- 1 > 2 -> y > c(x, y) [1] 1 2 I've never seen a language or pseudo-code that does assignment with the arrow pointing from the variable to the value. Does anyone know of any? -- Steve
On 05/12/2021 15:16, Steven D'Aprano wrote:
On Wed, Dec 01, 2021 at 12:26:33PM +0000, Matt del Valle wrote:
I'm sure that people will learn the many uses of the arrow symbols. After all, people learn Perl and APL :-) Closer to home, we also learn all the many different uses of the star symbol `*`.
Not to mention the half-dozen or so uses of colon (introduce a suite, slicing, dictionary displays, annotations, lambdas). Anybody still have a problem with them? Raise your hand and I'll give you your money back.😁 Best wishes Rob Cliffe
On Wed, Dec 1, 2021 at 7:51 AM Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Dec 1, 2021 at 10:30 PM André Roberge <andre.roberge@gmail.com> wrote:
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
Yes. I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.
Also fair. I'm not a fan of keywords for this sort of thing, since it implies that you could do this:
def f(x=defer []): ...
dflt = defer [] def f(x=dflt): ...
which is a completely different proposal (eg it would be evaluated only when you "touch" that, rather than being guaranteed to be evaluated before the first line of the function body). That's why I want to adorn the equals sign and nothing else.
Shouldn't the PEP contain a rejected idea section where this could be mentioned?
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
*Perhaps* if a keyword would be used instead of symbols, I might reconsider.
I find the emphasis of trying to cram too much information in single lines of code to be really a burden. Many years ago, I argued very unsuccessfully for using a 'where:' code block for annotations. (To this day, I still believe it would make the code much more readable, at the cost of a slight duplication.) Using what is at first glance a cryptic operator like => for late binding is not helping readability, especially when type annotations are thrown in the mix.
Aside: at the same time, I can see how using => instead of lambda as a potential win in readability, including for beginners.
It's interesting how different people's views go on that sort of thing. It depends a lot on how much people expect to use something. Features you use a lot want to have short notations, features you seldom use are allowed to have longer words.
I rarely use lambda in my own code, and have never written a line of code
anywhere that uses a '=>' operator. If Python had a 'function' keyword instead of 'lambda', I would prefer to keep the function keyword instead of adding => as a symbol. For me, it is not a question of terseness for commonly used features, but one of easing the learning curve. Starting from zero, I do believe that => would be easier to grasp than learning about lambda as a keyword and the syntactic rules to use with it. With function as a keyword, I believe that it is the other way around. No doubt many others will disagree! André Roberge
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WTU355... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Dec 2, 2021 at 1:33 AM André Roberge <andre.roberge@gmail.com> wrote:
On Wed, Dec 1, 2021 at 7:51 AM Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Dec 1, 2021 at 10:30 PM André Roberge <andre.roberge@gmail.com> wrote:
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
Yes. I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.
Also fair. I'm not a fan of keywords for this sort of thing, since it implies that you could do this:
def f(x=defer []): ...
dflt = defer [] def f(x=dflt): ...
which is a completely different proposal (eg it would be evaluated only when you "touch" that, rather than being guaranteed to be evaluated before the first line of the function body). That's why I want to adorn the equals sign and nothing else.
Shouldn't the PEP contain a rejected idea section where this could be mentioned?
Hmm, maybe. It's such a completely different proposal, but it does get asked a few times. If someone could actually put it forward as a full proposal, I'd gladly mention it as an interaction with another PEP. Otherwise, I'll write up a very brief thing on deferred expressions and how they're not what this is about. ChrisA
Chris Angelico wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ Questions, for you all: 1) If this feature existed in Python 3.11 exactly as described, would you use it?
When needed, but I'm uncomfortable with the syntax.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
Yes. A lot of arrows with different meanings require the reader of the code to know and remember what each does. Something more distinctive feels better. I'm way too new to Python to have much say, but anyway, arrows make more sense for other uses. Late binding feels to me like it is best served by something more distinctive.
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
I'd certainly use it only when I felt it was absolutely necessary. The readability is not aided by utilising syntax to alter behaviour when the altered behaviour is not required. We only quote, escape, etc., when we need to. (Silly example to illustrate, I know.)
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
When I first saw this (very few months ago indeed!!), I wondered about something rather clunkier but more explicit, such as, for example -- ```def foo(x=__late__['bar']): def bar(): #something etc... ``` Ugly though. And I subsequently realised some use cases are not well-served by this. (Particularly when [speculatively] considering some implementation issues.) The point in any case, is that for the occasions when the mechanism is needed, then a clearly explicit way to indicate something is evaluated late, and a tidy place to put the definition of how it's handled (late), seemed to me to be preferable. Of the **cited** examples, I find the `@hi=len(a)` spelling most readable.
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
Sadly not yet, but time permitting I will one day figure that out, for sure.
bikeshedding is what we do best!
Should we introduce choice of colour, as we run out of new symbols to use? /jk JL
On Wed, Dec 1, 2021 at 1:18 AM Chris Angelico <rosuav@gmail.com> wrote:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
No. ... except in the sense that as I trainer I have to teach the warts in Python, and would need to warn students they might see that.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
YES! A few weeks later than the prior long discussion that I read in full, it took a triple take not to read it as >= (which would mean something syntactical in many cases, just not what is intended).
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
I would always recommend against its use if I had any influence on code review.
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
Yes, the delay/later/defer keyword approach is not confusing, and does not preempt a later feature that would actually be worth having. 5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
I do know how, but it is unlikely I'll have time. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Thu, Dec 2, 2021 at 12:42 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
Yes, the delay/later/defer keyword approach is not confusing, and does not preempt a later feature that would actually be worth having.
Do you mean changing the spelling of the existing proposal, or a completely different proposal for deferred objects that are actually objects? Because that is NOT what I mean by a "small change". :) ChrisA
On Wed, Dec 1, 2021 at 10:12 AM Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Dec 2, 2021 at 12:42 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
Yes, the delay/later/defer keyword approach is not confusing, and does not preempt a later feature that would actually be worth having.
Do you mean changing the spelling of the existing proposal, or a completely different proposal for deferred objects that are actually objects? Because that is NOT what I mean by a "small change". :)
The spelling of the existing proposal. I.e. if the proposal were: def fun(things: list[int] = defer []) -> int: # ... some implementation I'd be -0 on the idea rather than -100. Yes, my change in attitude is largely because I want *some future PEP* to address the more general situation like: result = defer really_expensive_calculation() if predicate: doubled = result * 2 But I do not think your PEP does (nor even should) include that potential future behavior/syntax. Such a hypothetical future PEP would have a continuity with the syntax of your feature, albeit DEFINITELY need to address many independent concerns/issues that yours does not create. However, even if I assume the mythical future PEP never happens, in terms of readability, a WORD is vastly less confusing than a combination of punctuation that has no obvious or natural interpretation like '=>'. Or rather, I think that spelling is kinda-sorta obvious for the lambda meaning, and the use you want is kinda-sorta similar to a lambda. So I *do* understand how you get there... but it still seems like much too much line noise for a very minimal need. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Thu, Dec 2, 2021 at 2:40 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
On Wed, Dec 1, 2021 at 10:12 AM Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Dec 2, 2021 at 12:42 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
Yes, the delay/later/defer keyword approach is not confusing, and does not preempt a later feature that would actually be worth having.
Do you mean changing the spelling of the existing proposal, or a completely different proposal for deferred objects that are actually objects? Because that is NOT what I mean by a "small change". :)
The spelling of the existing proposal. I.e. if the proposal were:
def fun(things: list[int] = defer []) -> int: # ... some implementation
I'd be -0 on the idea rather than -100.
Okay. If it's JUST the spelling, then yes, I'll take that into consideration (though I am still against keywords myself).
Yes, my change in attitude is largely because I want *some future PEP* to address the more general situation like:
result = defer really_expensive_calculation() if predicate: doubled = result * 2
But I do not think your PEP does (nor even should) include that potential future behavior/syntax. Such a hypothetical future PEP would have a continuity with the syntax of your feature, albeit DEFINITELY need to address many independent concerns/issues that yours does not create.
However, even if I assume the mythical future PEP never happens, in terms of readability, a WORD is vastly less confusing than a combination of punctuation that has no obvious or natural interpretation like '=>'. Or rather, I think that spelling is kinda-sorta obvious for the lambda meaning, and the use you want is kinda-sorta similar to a lambda. So I *do* understand how you get there... but it still seems like much too much line noise for a very minimal need.
The trouble is that this actually would be incompatible. If you can defer an expensive calculation and have some "placeholder" value in the variable 'result', then logically, you should be able to have that placeholder as a function default argument, which would be an early-bound default of the placeholder. That is quite different in behaviour from a late-bound default, so if I were to use the word "defer" for late-bound defaults, it would actually prevent the more general proposal. ChrisA
On 01/12/2021 15:39, David Mertz, Ph.D. wrote:
However, even if I assume the mythical future PEP never happens, in terms of readability, a WORD is vastly less confusing than a combination of punctuation that has no obvious or natural interpretation like '=>'. Or rather, I think that spelling is kinda-sorta obvious for the lambda meaning, and the use you want is kinda-sorta similar to a lambda. So I *do* understand how you get there... but it still seems like much too much line noise for a very minimal need.
Hm. A word is "vastly less confusing". OK. Should Python have been designed with x assigned y+1 rather than x = y +1 (note that '=' here does not have its "obvious or natural interpretation", viz. "is equal to"). Should we have (x becomes y+1) rather than (x := y+1) Conciseness *is* a virtue, even if it is often trumped by other considerations. Best wishes Rob Cliffe
On Wed, 1 Dec 2021 at 06:19, Chris Angelico <rosuav@gmail.com> wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
Probably not. Mainly because I don't have any real use for it rather than because I have any inherent problem with it (but see below, the more I thought about it the more uncomfortable with it I became).
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)
Not especially. The idea of having both late-bound and early-bound parameters is probably more of a cognitive burden than the syntax.
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
N/A, except to say that when you enumerate the use cases like this, none of them even tempt me to use this feature. I think that the only thing I might use it for is to make it easier to annotate defaults (as f(a: list[int] => []) rather than as f(a: list[int] | None = None). So I'll revise my answer to (1) and say that I *might* use this, but only in a way it wasn't intended to be used in, and mostly because I hate how verbose it is to express optional arguments in type annotations. (And the fact that the annotation exposes the sentinel value, even when you want it to be opaque). I hope I don't succumb and do that, though ;-)
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
Not really. It addresses a wart in the language, but on consideration, it feels like the cure is no better than the disease.
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
Sorry, I really don't have time to, in the foreseeable future. If I did have time, one thing I would experiment with is how this interacts with typing and tools like pyright and mypy (yes, I know type checkers would need updating for the new syntax, so that would mostly be a thought experiment) - as I say, I'd expect to annotate a function with an optional list argument defaulting to an empty list as f(a: list[int] => []), which means that __annotations__ needs to distinguish between this case and f(a: list[int]) with no default.
def f(a: list[int]): pass ... f.__annotations__ {'a': list[int]}
def f(a: list[int] => []): pass ... f.__annotations__ ???
I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative.
Sorry, I don't have any feedback like that. What I can say, though, is I'd find it quite hard to express the question, in the sense that I'd struggle to explain the difference between early and late bound parameters to a non-expert, much less explain why we need both. I'd probably just say "it's short for a default of None and a check" which doesn't really capture the point...
Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!
I hope this was useful feedback. Paul
On Thu, Dec 2, 2021 at 1:21 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Wed, 1 Dec 2021 at 06:19, Chris Angelico <rosuav@gmail.com> wrote:
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
N/A, except to say that when you enumerate the use cases like this, none of them even tempt me to use this feature.
I'm just listing some of the common use-cases, not all of them.
I think that the only thing I might use it for is to make it easier to annotate defaults (as f(a: list[int] => []) rather than as f(a: list[int] | None = None).
That's exactly the goal! It's hugely simpler to say "this is always a list, and defaults to a new empty list" than "this is a list or None, defaults to None, and hey, if it's None, I'll make a new empty list".
So I'll revise my answer to (1) and say that I *might* use this, but only in a way it wasn't intended to be used in, and mostly because I hate how verbose it is to express optional arguments in type annotations. (And the fact that the annotation exposes the sentinel value, even when you want it to be opaque). I hope I don't succumb and do that, though ;-)
That's definitely an intended use-case ("mutable defaults" above); you're focusing on the fact that it improves the annotations, I'm focusing on the fact that it improves documentation and introspection, but it's all the same improvement :)
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
Sorry, I really don't have time to, in the foreseeable future. If I did have time, one thing I would experiment with is how this interacts with typing and tools like pyright and mypy (yes, I know type checkers would need updating for the new syntax, so that would mostly be a thought experiment) - as I say, I'd expect to annotate a function with an optional list argument defaulting to an empty list as f(a: list[int] => []), which means that __annotations__ needs to distinguish between this case and f(a: list[int]) with no default.
No problem, yup.
Sorry, I don't have any feedback like that. What I can say, though, is I'd find it quite hard to express the question, in the sense that I'd struggle to explain the difference between early and late bound parameters to a non-expert, much less explain why we need both. I'd probably just say "it's short for a default of None and a check" which doesn't really capture the point...
The point is that it's a shortcut for "omitted" rather than "a default of None", but there are many ways to explain it, and I only have one brother who knows enough Python to be able to talk to about this (and I got feedback from him VERY early).
Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!
I hope this was useful feedback.
It was. Thank you. ChrisA
On Wed, Dec 1, 2021 at 9:24 AM Paul Moore <p.f.moore@gmail.com> wrote:
I think that the only thing I might use it for is to make it easier to annotate defaults (as f(a: list[int] => []) rather than as f(a: list[int] | None = None).
Why not `f(a: Optional[list[int]] = None)`? I'm not counting characters, but that form seems to express the intention better than either of the others IMHO. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Thu, Dec 2, 2021 at 2:24 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
On Wed, Dec 1, 2021 at 9:24 AM Paul Moore <p.f.moore@gmail.com> wrote:
I think that the only thing I might use it for is to make it easier to annotate defaults (as f(a: list[int] => []) rather than as f(a: list[int] | None = None).
Why not `f(a: Optional[list[int]] = None)`?
I'm not counting characters, but that form seems to express the intention better than either of the others IMHO.
"a: list[int] => []" fully expresses the intention IMO, but yes, Optional is usually better than "| None". That's just a minor spelling problem though. ChrisA
On Wed, 1 Dec 2021 at 15:24, David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
On Wed, Dec 1, 2021 at 9:24 AM Paul Moore <p.f.moore@gmail.com> wrote:
I think that the only thing I might use it for is to make it easier to annotate defaults (as f(a: list[int] => []) rather than as f(a: list[int] | None = None).
Why not `f(a: Optional[list[int]] = None)`?
I'm not counting characters, but that form seems to express the intention better than either of the others IMHO.
If None were a valid argument, and I was using an opaque sentinel, Optional doesn't work, and exposing the type of the sentinel is not what I intend (as it's invalid to explicitly supply the sentinel value). Also, Optional[list[int]] doesn't express the intent accurately - the intended use is that people must supply a list[int] or not supply the argument *at all*. Optional allows them to supply None as well. As I say, I don't consider this an intended use case for the feature, because what I'm actually discussing here is optional arguments and sentinels, which is a completely different feature. All I'm saying is that the only case when I can imagine using this feature is for when I want a genuinely opaque way of behaving differently if the caller omitted an argument (and using None or a sentinel has been good enough all these years, so it's not exactly a pressing need). Let's just go back to the basic point, which is that I can't think of a realistic case where I'd want to actually use the new feature. Paul
On 2/12/21 4:40 am, Paul Moore wrote:
the intended use is that people must supply a list[int] or not supply the argument *at all*.
I don't think this is a style of API that we should be encouraging people to create, because it results in things that are very awkward to wrap. -- Greg
On Wed, 1 Dec 2021 at 22:27, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 2/12/21 4:40 am, Paul Moore wrote:
the intended use is that people must supply a list[int] or not supply the argument *at all*.
I don't think this is a style of API that we should be encouraging people to create, because it results in things that are very awkward to wrap.
Hmm, interesting point, I agree with you. It's particularly telling that I got sucked into designing that sort of API, even though I know it's got this problem. I guess that counts as an argument against the late bound defaults proposal - or maybe even two: 1. It's hard (if not impossible) to wrap functions that use late-bound defaults. 2. The feature encourages people to write such unwrappable functions when an alternative formulation that is wrappable is just as good. (That may actually only be one point - obviously a feature encourages people to use it, and any feature can be over-used. But the point about wrappability stands). Paul
On Thu, 2 Dec 2021 at 08:29, Paul Moore <p.f.moore@gmail.com> wrote:
On Wed, 1 Dec 2021 at 22:27, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 2/12/21 4:40 am, Paul Moore wrote:
the intended use is that people must supply a list[int] or not supply the argument *at all*.
I don't think this is a style of API that we should be encouraging people to create, because it results in things that are very awkward to wrap.
Hmm, interesting point, I agree with you. It's particularly telling that I got sucked into designing that sort of API, even though I know it's got this problem. I guess that counts as an argument against the late bound defaults proposal - or maybe even two:
1. It's hard (if not impossible) to wrap functions that use late-bound defaults. 2. The feature encourages people to write such unwrappable functions when an alternative formulation that is wrappable is just as good.
(That may actually only be one point - obviously a feature encourages people to use it, and any feature can be over-used. But the point about wrappability stands).
Actually, Chris - does functools.wraps work properly in your implementation when wrapping functions with late-bound defaults?
def dec(f): ... @wraps(f) ... def inner(*args, **kw): ... print("Calling") ... return f(*args, **kw) ... return inner ... @dec ... def g(a => []): ... return len(a)
Paul
On Thu, Dec 2, 2021 at 7:36 PM Paul Moore <p.f.moore@gmail.com> wrote:
Actually, Chris - does functools.wraps work properly in your implementation when wrapping functions with late-bound defaults?
def dec(f): ... @wraps(f) ... def inner(*args, **kw): ... print("Calling") ... return f(*args, **kw) ... return inner ... @dec ... def g(a => []): ... return len(a)
Yes, it does. There are two things happening here which, from a technical standpoint, are actually orthogonal; the function *call* is simply using *a,**kw notation, so it's passing along all the positional and keyword parameters untouched; but the function *documentation* just says "hey look over there for the real signature".
g([1,2,3]) Calling 3 g() Calling 0
(In your example there's no proof that it's late-bound, but it is.)
help(g) Help on function g in module __main__:
g(a=>[])
g.__wrapped__ <function g at 0x7fb5581efa00> g <function g at 0x7fb5581efab0>
When you do "inner = wraps(f)(inner)", what happens is that the function's name and qualname get updated, and then __wrapped__ gets added as a pointer saying "hey, assume that I have the signature of that guy over there". There's unfortunately no way to merge signatures (you can't do something like "def f(*a, timeout=500, **kw):" and then use the timeout parameter in your wrapper and pass the rest on - help(f) will just show the base function's signature), and nothing is actually aware of the underlying details. But on the plus side, this DOES work correctly. Documentation for late-bound defaults is done by a (compile-time) string snapshot of the AST that defined the default, so it's about as accurate as for early-bound defaults. One thing I'm not 100% happy with in the reference implementation is that the string for help() is stored on the function object, but the behaviour of the late-bound default is inherent to the code object. I'm not sure of a way around that, but it also doesn't seem terribly problematic in practice. ChrisA
On Thu, Dec 2, 2021 at 7:31 PM Paul Moore <p.f.moore@gmail.com> wrote:
On Wed, 1 Dec 2021 at 22:27, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 2/12/21 4:40 am, Paul Moore wrote:
the intended use is that people must supply a list[int] or not supply the argument *at all*.
I don't think this is a style of API that we should be encouraging people to create, because it results in things that are very awkward to wrap.
Hmm, interesting point, I agree with you. It's particularly telling that I got sucked into designing that sort of API, even though I know it's got this problem. I guess that counts as an argument against the late bound defaults proposal - or maybe even two:
1. It's hard (if not impossible) to wrap functions that use late-bound defaults. 2. The feature encourages people to write such unwrappable functions when an alternative formulation that is wrappable is just as good.
(That may actually only be one point - obviously a feature encourages people to use it, and any feature can be over-used. But the point about wrappability stands).
Wrappability when using an arbitrary sentinel looks like this: _SENTINEL = object() def f(value=_SENTINEL): if value is _SENTINEL: value = ... # in another module def wrap_f(): othermodule.f(othermodule._SENTINEL) Or: def wrap_f(): othermodule.f(othermodule.f.__defaults__[0]) I'm not sure either of these is any better than the alternatives. You're reaching into a module to access its internal implementation details. Is that really better than messing with *args? def wrap_f(): args = [42] or [] # pass or don't pass? othermodule.f(*args) It only looks easy when None is used, and even then, it's largely an implementation detail. ChrisA
On Thu, Dec 02, 2021 at 08:29:58AM +0000, Paul Moore wrote:
1. It's hard (if not impossible) to wrap functions that use late-bound defaults.
I don't understand that argument. We can implement late-bound defaults right now, using the usual sentinel jiggery-pokery. def func(spam, eggs, cheese=None, aardvark=42): if cheese is None: ... # You know the drill. We can wrap functions like this. You don't generally even care what the sentinel is. A typical wrapper function looks something like this: @functools.wraps(func) def inner(*args, **kwargs): # wrapper implementation func(*args, **kwargs) with appropriate pre-processing or post-processing as needed. Why would argument unpacking to call the wrapped function stop working? -- Steve
On 11/30/21 10:16 PM, Chris Angelico wrote:
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
No.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
Yes.
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
a, b, c
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
Have the token/keyword be at the beginning instead of in the middle. -- ~Ethan~
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
Yes
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
No, but it will be cognitive burden with shorthand lambda proposed syntax, for example def x(a: (b, c)=>c): is annotation for a (b, c) or maybe (b, c)=>c
(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
mostly (a), sometimes (c)
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
While I answered yes to question 1, personally I would prefer not adding new syntax, but switching completly to late defaults (requiring future import for some next versions)
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
Don't have enough time.
On 1 Dec 2021, at 06:16, Chris Angelico <rosuav@gmail.com> wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
no because of name=>
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
yes.
(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
yes (a) What does (b) mean? example please. yes (c)
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
Use the @name to avoid the confusing with the set of = things.
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
no promises, if I get spare time I'll give it a go, should be easy to hack the Fedora python RPM to build your version.
I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative.
Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!
The reference implementation currently has some test failures, which I'm looking into. I'm probably going to make this my personal default Python interpreter for a while, to see how things go.
Barry
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UVOQEK... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Dec 2, 2021 at 4:40 AM Barry Scott <barry@barrys-emacs.org> wrote:
On 1 Dec 2021, at 06:16, Chris Angelico <rosuav@gmail.com> wrote: 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
yes (a) What does (b) mean? example please. yes (c)
global_default = 500 def do_thing(timeout=>global_default): ... If the global_default timeout changes between function definition and call, omitting timeout will use the updated global. Similarly, you could say "file=>sys.stdout" and if code elsewhere changes sys.stdout, you'll use that. ChrisA
On 1 Dec 2021, at 17:59, Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Dec 2, 2021 at 4:40 AM Barry Scott <barry@barrys-emacs.org> wrote:
On 1 Dec 2021, at 06:16, Chris Angelico <rosuav@gmail.com> wrote: 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
yes (a) What does (b) mean? example please. yes (c)
global_default = 500 def do_thing(timeout=>global_default): ...
If the global_default timeout changes between function definition and call, omitting timeout will use the updated global.
Similarly, you could say "file=>sys.stdout" and if code elsewhere changes sys.stdout, you'll use that.
On a case-by-case basis I might still put defaulting into the body of the function if that made the intent clearer. I could see me using @file=sys.stdout. Barry
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/KVLO3C... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Dec 2, 2021 at 8:50 AM Barry Scott <barry@barrys-emacs.org> wrote:
On 1 Dec 2021, at 17:59, Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Dec 2, 2021 at 4:40 AM Barry Scott <barry@barrys-emacs.org> wrote:
On 1 Dec 2021, at 06:16, Chris Angelico <rosuav@gmail.com> wrote: 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
yes (a) What does (b) mean? example please. yes (c)
global_default = 500 def do_thing(timeout=>global_default): ...
If the global_default timeout changes between function definition and call, omitting timeout will use the updated global.
Similarly, you could say "file=>sys.stdout" and if code elsewhere changes sys.stdout, you'll use that.
On a case-by-case basis I might still put defaulting into the body of the function if that made the intent clearer.
I could see me using @file=sys.stdout.
That's a simplified version, but you might have the same default shown as get_default_timeout() or Defaults.timeout or something like that. The point is that it might be a simple integer, but it could change at any time, and the default should always be "whatever this name refers to". In any case, it's just one of many use-cases. I was curious what people would use and what they wouldn't. ChrisA
On Wed, Dec 01, 2021 at 05:16:34PM +1100, Chris Angelico wrote:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
Yes I would, but probably not as often as counting cases of the "if param is None: ..." idiom might lead you to expect. The problem is, as Neil also pointed out, that it becomes tricky to explicitly ask for the default behaviour except by leaving the argument out altogether. Not impossible, as Chris mentions elsewhere, you can mess about with `*args` or `**kwargs`, but it is decidedly less convenient, more verbose, and likely to have a performance hit. So if I were messing about in the interactive interpreter, I would totally use this for the convenience: def func(L=>[]): ... but if I were writing a library, I reckon that probably at least half the time I'll stick to the old idiom so I can document the parameter: "If param is missing **or None**, the default if blah..." I reject Chris' characterisation of this as a hack. There are function parameters where None will *never* in any conceivable circumstances become a valid argument value, and it is safe to use it as a sentinel. For example, I have a function that takes a `collation=None` parameter. The collation is a sequence of characters, usually a string. Using None to indicate "use the default collation" will never conflict with some possible future use of "use None as the actual collation" because None is not a sequence of characters. In this specific case, my function's current signature is: def itoa(n, base, collation=None, plus='', minus='-', width=0, ): I could re-write it as: def itoa(n, base, collation=>_default_collation(base), plus='', minus='-', width=0, ): but that would lose the ability to explicitly say "use the default collation" by passing None. So I think that, on balance, I would likely stick to the existing idiom for this function. On the other hand, if None were a valid value, so the signature used a private and undocumented sentinel: collation=_MISSING (and I've used that many times in other functions) then I would be likely to swap to this feature and drop the private sentinel.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
I think you know my opinion on the `=>` syntax :-) I shall not belabour it further here. I reserve the right to belabour it later on :-)
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
Any of the above.
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
There's lots of Python features that I use even though I don't like the spelling. Not a day goes by that I don't wish we spelled the base class of the object hierarchy "Object", so it were easier to distinguish between Object the base class and object as a generic term for any object. If the Steering Council loves your `=>` syntax then I would disagree with their decision but still use it. -- Steve
On Wed, Dec 1, 2021, 10:38 PM Steven D'Aprano
"If param is missing **or None**, the default if blah..." I reject Chris' characterisation of this as a hack. There are function parameters where None will *never* in any conceivable circumstances become a valid argument value, and it is safe to use it as a sentinel.
In particular, the cases where None will never, in any conceivable circumstances, be a non-sentinel value are at least 98% of all the functions (that have named parameters) I've ever written in Python. All of this discussion of a syntax change is for, at most, 2% of functions that need a different sentinel.
2% of functions is a lot of functions. We're talking about a language that's been around 30 years. The feature set is pretty mature. If it were lacking features that would improve a much larger percent of code for so long, I don't think it would be as popular. It's not like the next PIP is going to be as big as the for-loop. On Wednesday, December 1, 2021 at 9:56:40 PM UTC-6 David Mertz, Ph.D. wrote:
On Wed, Dec 1, 2021, 10:38 PM Steven D'Aprano
"If param is missing **or None**, the default if blah..." I reject Chris' characterisation of this as a hack. There are function parameters where None will *never* in any conceivable circumstances become a valid argument value, and it is safe to use it as a sentinel.
In particular, the cases where None will never, in any conceivable circumstances, be a non-sentinel value are at least 98% of all the functions (that have named parameters) I've ever written in Python.
All of this discussion of a syntax change is for, at most, 2% of functions that need a different sentinel.
Steven D'Aprano ""If param is missing **or None**, the default if blah..." I reject Chris' characterisation of this as a hack. There are function parameters where None will *never* in any conceivable circumstances become a valid argument value, and it is safe to use it as a sentinel." Yes, we know *why* the hack works. We're all familiar with it. That doesn't mean it's not a hack. The bottom line is: you *don't actually* want the parameter to default to the value of a sentinel. you *have* to use that hack because you can't express what you want the default to actually be. You're doing something misleading to work around a shortcoming of the language. That's a hack. You have to write something that you don't actually intend. On Wednesday, December 1, 2021 at 9:39:12 PM UTC-6 Steven D'Aprano wrote:
On Wed, Dec 01, 2021 at 05:16:34PM +1100, Chris Angelico wrote:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
Yes I would, but probably not as often as counting cases of the "if param is None: ..." idiom might lead you to expect.
The problem is, as Neil also pointed out, that it becomes tricky to explicitly ask for the default behaviour except by leaving the argument out altogether. Not impossible, as Chris mentions elsewhere, you can mess about with `*args` or `**kwargs`, but it is decidedly less convenient, more verbose, and likely to have a performance hit.
So if I were messing about in the interactive interpreter, I would totally use this for the convenience:
def func(L=>[]): ...
but if I were writing a library, I reckon that probably at least half the time I'll stick to the old idiom so I can document the parameter:
"If param is missing **or None**, the default if blah..."
I reject Chris' characterisation of this as a hack. There are function parameters where None will *never* in any conceivable circumstances become a valid argument value, and it is safe to use it as a sentinel.
For example, I have a function that takes a `collation=None` parameter. The collation is a sequence of characters, usually a string. Using None to indicate "use the default collation" will never conflict with some possible future use of "use None as the actual collation" because None is not a sequence of characters.
In this specific case, my function's current signature is:
def itoa(n, base, collation=None, plus='', minus='-', width=0, ):
I could re-write it as:
def itoa(n, base, collation=>_default_collation(base), plus='', minus='-', width=0, ):
but that would lose the ability to explicitly say "use the default collation" by passing None. So I think that, on balance, I would likely stick to the existing idiom for this function.
On the other hand, if None were a valid value, so the signature used a private and undocumented sentinel:
collation=_MISSING
(and I've used that many times in other functions) then I would be likely to swap to this feature and drop the private sentinel.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
I think you know my opinion on the `=>` syntax :-)
I shall not belabour it further here. I reserve the right to belabour it later on :-)
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
Any of the above.
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
There's lots of Python features that I use even though I don't like the spelling. Not a day goes by that I don't wish we spelled the base class of the object hierarchy "Object", so it were easier to distinguish between Object the base class and object as a generic term for any object.
If the Steering Council loves your `=>` syntax then I would disagree with their decision but still use it.
-- Steve _______________________________________________ Python-ideas mailing list -- python...@python.org To unsubscribe send an email to python-id...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python...@python.org/message/G3V47ST6I... <https://mail.python.org/archives/list/python-ideas@python.org/message/G3V47ST6INLI7Y3C5RZSWVXFLNUTELQT/> Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, Dec 01, 2021 at 09:00:50PM -0800, abed...@gmail.com wrote:
Steven D'Aprano ""If param is missing **or None**, the default if blah..."
The bottom line is:
you *don't actually* want the parameter to default to the value of a sentinel.
Yes I do. I *do* want to be able to give a convenient sentinel value in order to explicitly tell the function "give me the default value".
you *have* to use that hack because you can't express what you want the default to actually be.
The point of having default values is so that the caller doesn't have to express what the default will actually be. If the caller has to express that value, it's not a default, it's a passed-in argument.
You're doing something misleading to work around a shortcoming of the language.
How is it misleading? The parameter is explicitly documented as taking None to have a certain effect. None is behaving here as a convenient, common special constant to trigger a certain behaviour, no different than passing (for example) buffering=-1 to open() to trigger a very complex set of behaviour. (With buffering=-1, the buffer depends on the platform details, and whether the file is binary, text, and whether or not it is a tty.) For historical reasons, probably related to C, the default value for buffering is -1. But it could have just as easily be None, or the string "default", or an enumeration, or some builtin constant. -- Steve
Steven D'Aprano "> ""If param is missing **or None**, the default if blah..."
The bottom line is:
you *don't actually* want the parameter to default to the value of a sentinel.
Yes I do. I *do* want to be able to give a convenient sentinel value in order to explicitly tell the function "give me the default value"." If you want the parameter to default to an empty list, it's categorically more explicit to have the parameter default to an empty list. It's a hack to say "I wan't this to default to an empty list" then have it actually default to a sentinel value that you'll replace with an empty list later on. That's the opposite of explicit. If you actually wanted to have the parameter default to a sentinel value, why would you then overwrite that value? This isn't a very complicated point. Steven D'Aprano "> you *have* to use that hack because you can't express what you want the
default to actually be.
The point of having default values is so that the caller doesn't have to express what the default will actually be. If the caller has to express that value, it's not a default, it's a passed-in argument." None of my comment had anything to do with the caller's perspective. I know how optional parameters work. I'm specifically talking about the programmer's ability to map their intent to code and the ability of the would-be reader of the code (NOT CALLER) to decipher that intent. If my intent is that I want a parameter to default to an empty list, I wouldn't assign it to an object called "monkey_pants" by default then add code to the body of the function to check if it's "monkey_pants". "But wait", I hear you say, "assigning my optional parameter to 'monkey_pants' isn't a hack because I know will *never* in any conceivable circumstances become a valid argument value!" "assigning a value to 'monkey_pants' is how I explicitly communicate tell the function "give me the default value"!" Those are both quotes from you with 'None' replaced with 'monkey_pants'. Steven D'Aprano "How is it misleading?" Your way of having a parameter default to an empty list (or whatever) is by having the parameter default to None. That's how it's misleading. I know how the hack works. I know why the hack works. I know how default parameters work. You can spare trying to explain all that again. What you can't explain is why it's more explicit to have a parameter default to a sentinel value when the actual intent is to have it default to a mutable value. If it were more explicit, then why not replace `def func(x=0)` with `def func(x=SENTINEL): if x is SENTINEL: x = 0`? I'm at a loss for ways to explain my position any clear than that. On Thursday, December 2, 2021 at 9:52:28 AM UTC-6 Steven D'Aprano wrote:
On Wed, Dec 01, 2021 at 09:00:50PM -0800, abed...@gmail.com wrote:
Steven D'Aprano ""If param is missing **or None**, the default if blah..."
The bottom line is:
you *don't actually* want the parameter to default to the value of a sentinel.
Yes I do. I *do* want to be able to give a convenient sentinel value in order to explicitly tell the function "give me the default value".
you *have* to use that hack because you can't express what you want the default to actually be.
The point of having default values is so that the caller doesn't have to express what the default will actually be. If the caller has to express that value, it's not a default, it's a passed-in argument.
You're doing something misleading to work around a shortcoming of the language.
How is it misleading? The parameter is explicitly documented as taking None to have a certain effect. None is behaving here as a convenient, common special constant to trigger a certain behaviour, no different than passing (for example) buffering=-1 to open() to trigger a very complex set of behaviour.
(With buffering=-1, the buffer depends on the platform details, and whether the file is binary, text, and whether or not it is a tty.)
For historical reasons, probably related to C, the default value for buffering is -1. But it could have just as easily be None, or the string "default", or an enumeration, or some builtin constant.
-- Steve _______________________________________________ Python-ideas mailing list -- python...@python.org To unsubscribe send an email to python-id...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python...@python.org/message/QGPPY4JLA... <https://mail.python.org/archives/list/python-ideas@python.org/message/QGPPY4JLABDSKICIVJQZ5IRWMXMYAOH4/> Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Dec 02, 2021 at 01:20:27PM -0800, abed...@gmail.com wrote:
If you want the parameter to default to an empty list, it's categorically more explicit to have the parameter default to an empty list.
You've gone from talking about *my* intentions, which you got wrong, to an argument about which is "more explicit". In the simple example of the default being a new empty list: # Using Chris' preferred syntax, not mine def func(arg=>[]): then I agree: once you know what the syntax means, then it is more explicit at telling the reader that the default value is an empty list. Great! This sort of thing is the primary use-case of the feature. But let's talk about more complex examples: def open(filename, mode='r', buffering=-1): ... How would you make that "more explicit"? def open(filename, mode='r', buffering=>( 1 if 'b' not in mode and os.fdopen(os.open(filename)).isatty() else _guess_system_blocksize() or io.DEFAULT_BUFFER_SIZE ) ): ... Too much information! We're now drowning in explicitness. Now how do I, the caller, *explicitly* indicate that I want to use that system-dependent value? I just want a nice, simple value I can explicitly pass as buffering to say "just use the default". with open('myfile.txt', buffering= what? ) as f: I can't. So in this case, your agonisingly explicit default expression forces the caller to be *less* explicit when they call the function. What the right hand giveth, the left hand taketh away. Can we agree that, like salt, sugar, and bike-shedding, sometimes you can have *too much* explicitness in code? Sometimes the right level of explicitness is to have an agreed upon symbol that acts as a short, easy to read and write, documented and explicit signal to the function "give me the default value". And that symbol can be -1, or an enumeration, or a string, or None. This is not a hack, and its not "implicit".
None of my comment had anything to do with the caller's perspective.
The caller's perspective is important. The caller has to read the function signature (either in the documentation, or if they use `help(function)` in the interpreter, or when their IDE offers up tooltips or argument completion). The caller has to actually use the function. We should consider their perspective.
If my intent is that I want a parameter to default to an empty list, I wouldn't assign it to an object called "monkey_pants" by default then add code to the body of the function to check if it's "monkey_pants".
Funny you should say that, but *totally by coincidence*, the Bulgarian word for "default" happens to be transliterated into English as *monkeypants*, so if I were Bulgarian that is exactly what I might do! How weird is that? Nah, I'm kidding of course. But the point I am making is that you are mocking the concept by using a totally silly example. "monkey_pants" indeed, ha ha how very droll. But if you used a more meaningful symbol, like "default", or some enumeration with a meaningful name, or a value which is a standard idiom such as -1, then the idea isn't so ridiculous. And None is one such standard idiom. *Especially* when the default value is not some trivially simple expression such as an empty list, but a more complicated expression such as the default buffering used by open files, or in my earlier example of the default collation (a sequence of letters which depends on the other input arguments).
Steven D'Aprano "How is it misleading?"
Your way of having a parameter default to an empty list (or whatever) is by having the parameter default to None. That's how it's misleading.
All you have done is repeat that it is misleading. I'm still no clearer why you think it is misleading to document "if the argument is missing or None, the default behaviour is ...". Is the documentation false? Does the function not do exactly what it says it will do if you pass that symbol? Let's be more concrete. Do you feel that using -1 for the buffer size in open() is "misleading"? Do you think that there is ever a possibility that someday Python's file I/O will use a buffer with *literally* a size of -1? -- Steve
"You've gone from talking about *my* intentions, which you got wrong, to an argument about which is "more explicit"." No. You were the one who brought up explicitness by claiming that using a sentinel value was a way to "explicitly tell the function "give me the default value" (whatever the heck that means). Now that I've deconstructed that argument, you're moving the goal post from "my way is more explicit" to "there is such a thing as *too* explicit, don't ya know?!" and blaming me for shifting the focus to "which is more explicit"... cool. "In the simple example of the default being a new empty list: # Using Chris' preferred syntax, not mine def func(arg=>[]): then I agree" Which is, what? 99% of the use cases? great! "But let's talk about more complex examples: def open(filename, mode='r', buffering=-1): ... How would you make that "more explicit"? def open(filename, mode='r', buffering=>( 1 if 'b' not in mode and os.fdopen(os.open(filename)).isatty() else _guess_system_blocksize() or io.DEFAULT_BUFFER_SIZE ) ): ... Too much information! We're now drowning in explicitness." Then just use the sentinel hack, or write a helper function that has a nice name that says what the default will be and use that. I'm positive there will always be corner cases where logic that's concerned with checking and setting the right values for parameters before the actual business logic starts. "Now how do I, the caller, *explicitly* indicate that I want to use that system-dependent value? I just want a nice, simple value I can explicitly pass as buffering to say "just use the default". with open('myfile.txt', buffering= what? ) as f: I can't. So in this case, your agonisingly explicit default expression forces the caller to be *less* explicit when they call the function." This seems like an exceedingly minor complaint. The obvious answer is: you invoke the default behaviour by not passing the argument. And yes, I know that's less explicit, but that's kinda the whole point of defaults to begin with. It's behavior when the user decides not to specify their own. Nobody's asking "what do I pass to logging.config.fileConfig to explicitly invoke the logging module's default behavior?!" The answer is simply: don't configure logging. It's not rocket science. This want to explicitly pass something to invoke default behavior brings this old Raymond Hetinger talk <https://www.youtube.com/watch?v=OSGv2VnC0go&t=1870s> to mind where he suggests that sometimes passing positional arguments by name can help make your code more readable. I wholeheartedly agree with the practice discussed in Hettinger's talk, but I think your example is completely different. When you see `twitter_search('@obama', False, 20, True)` that naturally invokes the question "what the heck are those arguments??" When you see `with open('myfile.txt') as f` it doesn't raise the question "what is the buffering argument?" I don't know if that's why you think the ability to explicitly invoke default behavior is so important, but I thought I'd address it anyway. "Can we agree that, like salt, sugar, and bike-shedding, sometimes you can have *too much* explicitness in code?" Surely *you* would never agree that there's such a thing as *too much* bike-shedding! Isn't that your raison d'être?! I never said one had to be a Nazi about stamping out every use of the sentinel... er... idiom (if you prefer it to hack). Sure, use it in those rare corner cases where expressing a late-bound default would be so very burdensome. The fact that there are such cases isn't a very good argument against this proposal. It's more like "HA! You can't fix *everything* can you?!". Great. You got me, Steve. *slow claps* "The caller's perspective is important. The caller has to read the function signature (either in the documentation, or if they use `help(function)` in the interpreter, or when their IDE offers up tooltips or argument completion). The caller has to actually use the function. We should consider their perspective." I never said we shouldn't. That's just not what I was talking about at that time and you were misinterpreting my argument. "Nah, I'm kidding of course. But the point I am making is that you are mocking the concept by using a totally silly example. "monkey_pants" indeed, ha ha how very droll. But if you used a more meaningful symbol, like "default", or some enumeration with a meaningful name, or a value which is a standard idiom such as -1, then the idea isn't so ridiculous. And None is one such standard idiom." You missed my point entirely. The point is that the reasons you were giving for sentinels both being explicit and not being a hack don't hold water. Namely: The fact that it's guaranteed to work doesn't make it not a hack, and your inexplicable reasoning that defaulting to a sentinel instead of the actual default value that parameter is supposed to take after all is said and done is somehow more explicit than defaulting to the default value in the first place. Just because it's an idiom, doesn't mean it's actually good. I think the `if __name__ == "__main__":` idiom is a horrible, round-about way of saying "if this file is being run as a script". You might look at that and say "What do you mean? That's exactly what it says!" because you've seen the idiom so many times, just like how you think "x=None" explicitly says "give x the default value", but if you look closely `if __name__ == "__main__":` *doesn't* actually say anything like "if this file is being run as a script", it says a bunch of esoteric and confusing garbage that you've learned to read as "if this file is being run as a script". "> Steven D'Aprano
"How is it misleading?"
Your way of having a parameter default to an empty list (or whatever) is by having the parameter default to None. That's how its misleading.
All you have done is repeat that it is misleading. I'm still no clearer why you think it is misleading" Seriously? If I said "I'm going to pat my head" then proceeded to rub my tummy, would you not think that's a bit misleading? Again, if you look closely, I *didn't* simply repeat that it's misleading. I pointed out the contradiction between intent and code. On Thu, Dec 2, 2021 at 8:57 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Thu, Dec 02, 2021 at 01:20:27PM -0800, abed...@gmail.com wrote:
If you want the parameter to default to an empty list, it's categorically more explicit to have the parameter default to an empty list.
You've gone from talking about *my* intentions, which you got wrong, to an argument about which is "more explicit".
In the simple example of the default being a new empty list:
# Using Chris' preferred syntax, not mine def func(arg=>[]):
then I agree: once you know what the syntax means, then it is more explicit at telling the reader that the default value is an empty list. Great! This sort of thing is the primary use-case of the feature.
But let's talk about more complex examples:
def open(filename, mode='r', buffering=-1): ...
How would you make that "more explicit"?
def open(filename, mode='r', buffering=>( 1 if 'b' not in mode and os.fdopen(os.open(filename)).isatty() else _guess_system_blocksize() or io.DEFAULT_BUFFER_SIZE ) ): ...
Too much information! We're now drowning in explicitness.
Now how do I, the caller, *explicitly* indicate that I want to use that system-dependent value? I just want a nice, simple value I can explicitly pass as buffering to say "just use the default".
with open('myfile.txt', buffering= what? ) as f:
I can't. So in this case, your agonisingly explicit default expression forces the caller to be *less* explicit when they call the function.
What the right hand giveth, the left hand taketh away.
Can we agree that, like salt, sugar, and bike-shedding, sometimes you can have *too much* explicitness in code? Sometimes the right level of explicitness is to have an agreed upon symbol that acts as a short, easy to read and write, documented and explicit signal to the function "give me the default value".
And that symbol can be -1, or an enumeration, or a string, or None.
This is not a hack, and its not "implicit".
None of my comment had anything to do with the caller's perspective.
The caller's perspective is important. The caller has to read the function signature (either in the documentation, or if they use `help(function)` in the interpreter, or when their IDE offers up tooltips or argument completion). The caller has to actually use the function. We should consider their perspective.
If my intent is that I want a parameter to default to an empty list, I wouldn't assign it to an object called "monkey_pants" by default then add code to the body of the function to check if it's "monkey_pants".
Funny you should say that, but *totally by coincidence*, the Bulgarian word for "default" happens to be transliterated into English as *monkeypants*, so if I were Bulgarian that is exactly what I might do! How weird is that?
Nah, I'm kidding of course. But the point I am making is that you are mocking the concept by using a totally silly example. "monkey_pants" indeed, ha ha how very droll. But if you used a more meaningful symbol, like "default", or some enumeration with a meaningful name, or a value which is a standard idiom such as -1, then the idea isn't so ridiculous.
And None is one such standard idiom.
*Especially* when the default value is not some trivially simple expression such as an empty list, but a more complicated expression such as the default buffering used by open files, or in my earlier example of the default collation (a sequence of letters which depends on the other input arguments).
Steven D'Aprano "How is it misleading?"
Your way of having a parameter default to an empty list (or whatever) is by having the parameter default to None. That's how it's misleading.
All you have done is repeat that it is misleading. I'm still no clearer why you think it is misleading to document "if the argument is missing or None, the default behaviour is ...".
Is the documentation false? Does the function not do exactly what it says it will do if you pass that symbol?
Let's be more concrete.
Do you feel that using -1 for the buffer size in open() is "misleading"? Do you think that there is ever a possibility that someday Python's file I/O will use a buffer with *literally* a size of -1?
-- Steve _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SZVP4K... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Dec 3, 2021 at 3:47 PM Abe Dillon <abedillon@gmail.com> wrote:
This seems like an exceedingly minor complaint. The obvious answer is: you invoke the default behaviour by not passing the argument. And yes, I know that's less explicit, but that's kinda the whole point of defaults to begin with. It's behavior when the user decides not to specify their own. Nobody's asking "what do I pass to logging.config.fileConfig to explicitly invoke the logging module's default behavior?!" The answer is simply: don't configure logging. It's not rocket science.
I'd say it isn't "less explicit". It's actually the ultimate definition of not doing something: you just... don't do it. The same line of explanation is important when working with callback functions: def callback(*a): print("Doing stuff") button.signal_connect("clicked", callback()) Oops, you called the callback instead of passing it to the signal_connect function. What's the solution? DON'T call the callback at this point. How do you NOT call something? Well, how do you call it? You put parentheses after it. So how do you not call it? You don't put the parentheses after it. Python is fairly good at giving you syntactic features that do things, such that you can not do them by omitting them. So I don't think anyone is truly concerned about the pure "don't pass the argument" case. The problem is the "maybe pass the argument" case. That ends up looking like this: c = 42 if cond: func(a, b) else: func(a, b, c) # or args = {} if not cond: args["c"] = 42 func(a, b, **args) # or if you know what the default is: c = some_default if not cond: c = 42 func(a, b, c) The first two will, of course, continue to work. The second one is a fairly clear way to indicate whether or not a parameter should get a value. The problem with the third is that it requires a documented and future-guaranteed default value, published and promised to work, and that simply isn't always possible or practical. So, yes, some forms of default might be a little unclear. And if you're converting an existing codebase, it may be necessary to maintain "if c is None: c = some_default" in the body forever, or until such time as you can remove the support for None as a parameter. But None isn't truly the default value.
"Can we agree that, like salt, sugar, and bike-shedding, sometimes you can have *too much* explicitness in code?"
Surely *you* would never agree that there's such a thing as *too much* bike-shedding! Isn't that your raison d'être?!
I never said one had to be a Nazi about stamping out every use of the sentinel... er... idiom (if you prefer it to hack).
Idiom. Sentinels are extremely important, and PEP 671 isn't proposing to eliminate them. The "hack" part comes when, for technical reasons, a sentinel is used when one isn't logically part of the API. I agree with Steve here: sometimes, the true default is extremely clunky, and it's better to use a sentinel. That's part of why they're never going away (and None will continue to be a popular sentinel). PEP 671 doesn't propose changing every function ever written, because the vast majority of them are perfectly fine as they are.
"The caller's perspective is important. The caller has to read the function signature (either in the documentation, or if they use `help(function)` in the interpreter, or when their IDE offers up tooltips or argument completion). The caller has to actually use the function. We should consider their perspective."
I never said we shouldn't. That's just not what I was talking about at that time and you were misinterpreting my argument.
Yes, absolutely. And that's why the function should be able to put the information in the signature when it's appropriate to, and in the body when it isn't.
Just because it's an idiom, doesn't mean it's actually good. I think the `if __name__ == "__main__":` idiom is a horrible, round-about way of saying "if this file is being run as a script". You might look at that and say "What do you mean? That's exactly what it says!" because you've seen the idiom so many times, just like how you think "x=None" explicitly says "give x the default value", but if you look closely `if __name__ == "__main__":` *doesn't* actually say anything like "if this file is being run as a script", it says a bunch of esoteric and confusing garbage that you've learned to read as "if this file is being run as a script".
TBH I think that the name-is-main idiom isn't too bad, but mainly because I've seen far worse idioms. If someone has a better proposal, I'd be happy to hear it, but the existing idiom isn't horrible. (But you're right. Well-known idioms can indeed be really REALLY bad. Just look at the sorts of things that pre-C99 C code would do.) ChrisA
On Fri, Dec 03, 2021 at 04:15:37PM +1100, Chris Angelico wrote:
So I don't think anyone is truly concerned about the pure "don't pass the argument" case. The problem is the "maybe pass the argument" case. That ends up looking like this:
c = 42 if cond: func(a, b) else: func(a, b, c) [...]
Yes, that's exactly it! I agree with the rest of your post here. -- Steve
"# or if you know what the default is: c = some_default if not cond: c = 42 func(a, b, c)" I would argue that, of the three examples; this one is the most problematic because it overly couples your code to an implementation detail of the function and makes it unnecessarily brittle to change. On top of that, in the case of a sentinel; it might even encourage people to circumvent the "guarantees" that your API relies upon. For example, I might make a function where the optional parameter could be anything even a None is valid, so I make my own secret sentinel that, as Steven D'Aprano put it "will *never* in any conceivable circumstances become a valid argument value": _default = object() def func(a, b, c=_default): ... Now the pattern above violates the "nobody will *ever* pass me this" condition because it imports _default. Then some intern sees your code and starts importing _default and using it elsewhere because it's such a neat pattern... "The "hack" part comes when, for technical reasons, a sentinel is used when one isn't logically part of the API." That's my entire point. "TBH I think that the name-is-main idiom isn't too bad" The bad part is a lot of Python newbies see it on day one or two because it's so common and try explaining it without talking about the interpreter and watching their eyes glaze over. You just showed them how to assign values to variables and that unbound variables cause errors and now you're pulling __name__ out of thin air? Where the heck did that come from? Anyway, this is a tangent I promised myself I wouldn't go on... On Thu, Dec 2, 2021 at 11:16 PM Chris Angelico <rosuav@gmail.com> wrote:
On Fri, Dec 3, 2021 at 3:47 PM Abe Dillon <abedillon@gmail.com> wrote:
This seems like an exceedingly minor complaint. The obvious answer is: you invoke the default behaviour by not passing the argument. And yes, I know that's less explicit, but that's kinda the whole point of defaults to begin with. It's behavior when the user decides not to specify their own. Nobody's asking "what do I pass to logging.config.fileConfig to explicitly invoke the logging module's default behavior?!" The answer is simply: don't configure logging. It's not rocket science.
I'd say it isn't "less explicit". It's actually the ultimate definition of not doing something: you just... don't do it. The same line of explanation is important when working with callback functions:
def callback(*a): print("Doing stuff")
button.signal_connect("clicked", callback())
Oops, you called the callback instead of passing it to the signal_connect function. What's the solution? DON'T call the callback at this point. How do you NOT call something? Well, how do you call it? You put parentheses after it. So how do you not call it? You don't put the parentheses after it.
Python is fairly good at giving you syntactic features that do things, such that you can not do them by omitting them.
So I don't think anyone is truly concerned about the pure "don't pass the argument" case. The problem is the "maybe pass the argument" case. That ends up looking like this:
c = 42 if cond: func(a, b) else: func(a, b, c)
# or args = {} if not cond: args["c"] = 42 func(a, b, **args)
# or if you know what the default is: c = some_default if not cond: c = 42 func(a, b, c)
The first two will, of course, continue to work. The second one is a fairly clear way to indicate whether or not a parameter should get a value. The problem with the third is that it requires a documented and future-guaranteed default value, published and promised to work, and that simply isn't always possible or practical.
So, yes, some forms of default might be a little unclear. And if you're converting an existing codebase, it may be necessary to maintain "if c is None: c = some_default" in the body forever, or until such time as you can remove the support for None as a parameter. But None isn't truly the default value.
"Can we agree that, like salt, sugar, and bike-shedding, sometimes you can have *too much* explicitness in code?"
Surely *you* would never agree that there's such a thing as *too much* bike-shedding! Isn't that your raison d'être?!
I never said one had to be a Nazi about stamping out every use of the sentinel... er... idiom (if you prefer it to hack).
Idiom. Sentinels are extremely important, and PEP 671 isn't proposing to eliminate them. The "hack" part comes when, for technical reasons, a sentinel is used when one isn't logically part of the API.
I agree with Steve here: sometimes, the true default is extremely clunky, and it's better to use a sentinel. That's part of why they're never going away (and None will continue to be a popular sentinel). PEP 671 doesn't propose changing every function ever written, because the vast majority of them are perfectly fine as they are.
"The caller's perspective is important. The caller has to read the function signature (either in the documentation, or if they use `help(function)` in the interpreter, or when their IDE offers up tooltips or argument completion). The caller has to actually use the function. We should consider their perspective."
I never said we shouldn't. That's just not what I was talking about at that time and you were misinterpreting my argument.
Yes, absolutely. And that's why the function should be able to put the information in the signature when it's appropriate to, and in the body when it isn't.
Just because it's an idiom, doesn't mean it's actually good. I think the `if __name__ == "__main__":` idiom is a horrible, round-about way of saying "if this file is being run as a script". You might look at that and say "What do you mean? That's exactly what it says!" because you've seen the idiom so many times, just like how you think "x=None" explicitly says "give x the default value", but if you look closely `if __name__ == "__main__":` *doesn't* actually say anything like "if this file is being run as a script", it says a bunch of esoteric and confusing garbage that you've learned to read as "if this file is being run as a script".
TBH I think that the name-is-main idiom isn't too bad, but mainly because I've seen far worse idioms. If someone has a better proposal, I'd be happy to hear it, but the existing idiom isn't horrible.
(But you're right. Well-known idioms can indeed be really REALLY bad. Just look at the sorts of things that pre-C99 C code would do.)
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QM2ALI... Code of Conduct: http://python.org/psf/codeofconduct/
On 02/12/2021 03:35, Steven D'Aprano wrote:
On Wed, Dec 01, 2021 at 05:16:34PM +1100, Chris Angelico wrote:
1) If this feature existed in Python 3.11 exactly as described, would you use it? Yes I would, but probably not as often as counting cases of the "if param is None: ..." idiom might lead you to expect.
The problem is, as Neil also pointed out, that it becomes tricky to explicitly ask for the default behaviour except by leaving the argument out altogether. Not impossible, as Chris mentions elsewhere, you can mess about with `*args` or `**kwargs`, but it is decidedly less convenient, more verbose, and likely to have a performance hit.
So if I were messing about in the interactive interpreter, I would totally use this for the convenience:
def func(L=>[]): ...
but if I were writing a library, I reckon that probably at least half the time I'll stick to the old idiom so I can document the parameter:
"If param is missing **or None**, the default if blah..."
I reject Chris' characterisation of this as a hack. There are function parameters where None will *never* in any conceivable circumstances become a valid argument value, and it is safe to use it as a sentinel.
There are also circumstances when you start off thinking that None will never be a valid argument value, but later you have to cater for it. Now it does start to look as if you used a hack. Best wishes Rob Cliffe
On 2021-11-30 22:16, Chris Angelico wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would you use it?
I hope not. :-) Realistically I might wind up using it at some point way down the line. I wouldn't start using it immediately. I still almost never use the walrus and only occasionally use f-strings. You didn't ask how people would feel about READING this rather than writing it, but what I would do is get really annoyed at seeing it in code as people start to use it and confuse me and others.
2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?
(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)
Yes, it is yet another reason not to do this.
3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?
N/A.
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
No. As I mentioned in the earlier thread, I don't support any proposal in which an argument can "have a default" but that default is not a first-class Python object of some sort.
5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)
No, I don't. I know I said this before, but I really hope this change is not adopted. It is to me a classic example of adding significant complexity to the language and reducing readability for only a very small benefit in expressiveness. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
Brendan Barnwell wrote:
No. As I mentioned in the earlier thread, I don't support any proposal in which an argument can "have a default" but that default is not a first-class Python object of some sort.
What if a default is a function? I was inspired by learning Django and saw in models that fields can have a default which is either a regular (early-bound) default such as a first-class Python object as one would expect, *or* a function -- which will be called 'later' when needed. That prompted me to contemplate a syntax for late-bound defaults, albeit a bit clunky, but I did think it suited a special-case requirement met by late-bound defaults. I still think that littering function arguments throughout all code with large numbers of arrows would make things less readable. Special requirements need special treatment, I'm thinking. The problem is passing arguments to such a function without it looking like it's being called at definition time.
On Thu, Dec 2, 2021 at 6:12 PM <role.pythonorg-readers@jlassocs.com> wrote:
Brendan Barnwell wrote:
No. As I mentioned in the earlier thread, I don't support any proposal in which an argument can "have a default" but that default is not a first-class Python object of some sort.
What if a default is a function?
I was inspired by learning Django and saw in models that fields can have a default which is either a regular (early-bound) default such as a first-class Python object as one would expect, *or* a function -- which will be called 'later' when needed.
That prompted me to contemplate a syntax for late-bound defaults, albeit a bit clunky, but I did think it suited a special-case requirement met by late-bound defaults. I still think that littering function arguments throughout all code with large numbers of arrows would make things less readable. Special requirements need special treatment, I'm thinking.
The problem is passing arguments to such a function without it looking like it's being called at definition time.
Also has the same problem of other deferreds, which is: when exactly is it evaluated? def func(stuff, n=>len(stuff)): stuff.append("spam") print(n) func(["spam", "ham"]) What will this print? If it's a function default, it MUST print 2, since len(stuff) is 2 as the function starts. But if it's a deferred object of some sort, then it should probably print 3, since len(stuff) is 3 at the time that n gets printed. Which is it to be? That's why PEP 671 is *not* about generic deferred evaluation. It is specifically about function default arguments, and guarantees that they will be evaluated prior to the first line of code in the function body. ChrisA
On Wed, Dec 01, 2021 at 10:50:38PM -0800, Brendan Barnwell wrote:
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
No. As I mentioned in the earlier thread, I don't support any proposal in which an argument can "have a default" but that default is not a first-class Python object of some sort.
I don't understand this criticism. Of course the default value will be a first-class Python object of some sort. *Every* value in Python is a first-class object. There are no machine values or unboxed values, and this proposal will not change that. All that this proposal changes is *when* and *how often* the default will be evaluated, not the nature of the value. Status quo: default values are evaluated once, when the def statement is executed. With optional late-binding: default values are evaluated as often as they are needed, when the function is called. But the value will still be an object. I suppose that there will be one other change, relating to introspection of the function. You will no longer be able to inspect the function and see the default values as constants in a cache: >>> (lambda x=1.25: None).__defaults__ (1.25,) Depending on the implementation, you *might* be able to inspect the function and see the default expression as some sort of callable function, or evaluatable code object. (That would be nice.) Or even as a plain old string. All of which are first-class objects. Or it might be that the default expression will be compiled into the body of the function, where is it effectively invisible. So I guess that's a third change: when, how often, and the difference to introspection. -- Steve
On Thu, Dec 2, 2021 at 8:40 PM Steven D'Aprano <steve@pearwood.info> wrote:
Depending on the implementation, you *might* be able to inspect the function and see the default expression as some sort of callable function, or evaluatable code object. (That would be nice.)
Unfortunately not, since the default expression could refer to other parameters, or closure variables, or anything else from the context of the called function. So you won't be able to externally evaluate it.
Or even as a plain old string. All of which are first-class objects.
For documentation purposes, it is indeed available as a plain old string. In colloquial terms, it is the source code for the expression, although technically it's reconstructed from the AST. ( This is approximately as accurate as saying that the repr of an object is its source code. Possibly more accurate, actually.)
Or it might be that the default expression will be compiled into the body of the function, where is it effectively invisible. So I guess that's a third change: when, how often, and the difference to introspection.
Among our changes are when, how often, the difference to introspection, and the ability to externally manipulate the defaults. And nice red uniforms. Although the ability to manipulate them could be considered part of introspection, not sure. I'm still unsure whether this is a cool feature or an utter abomination:
def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f.__defaults_extra__ = ("n/a",) f(42) You passed x as 42 f() You didn't pass x
(It's an implementation detail and not part of the specification, but if CPython adopts this behaviour, it will become de facto part of the expected behaviour, and someone somewhere will use this deliberately.) ChrisA
On Thu, Dec 02, 2021 at 11:00:33PM +1100, Chris Angelico wrote:
On Thu, Dec 2, 2021 at 8:40 PM Steven D'Aprano <steve@pearwood.info> wrote:
Depending on the implementation, you *might* be able to inspect the function and see the default expression as some sort of callable function, or evaluatable code object. (That would be nice.)
Unfortunately not, since the default expression could refer to other parameters, or closure variables, or anything else from the context of the called function. So you won't be able to externally evaluate it.
Why not? Functions can do all those things: refer to other variables, or closures, or anything else. You can call functions. Are you sure that this limitation of the default expression is not just a limitation of your implementation?
I'm still unsure whether this is a cool feature or an utter abomination:
def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f.__defaults_extra__ = ("n/a",) f(42) You passed x as 42 f() You didn't pass x
That is absolutely an abomination. If your implementation has the side-effect that setting a regular early-bound default to Ellipsis makes the parameter unable to retrieve the default, then the implementation is fatally broken. It absolutely is not a feature. -- Steve
On 02/12/2021 14:47, Steven D'Aprano wrote:
def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f.__defaults_extra__ = ("n/a",) f(42) You passed x as 42 f() You didn't pass x That is absolutely an abomination. If your implementation has the side-effect that setting a regular early-bound default to Ellipsis makes
On Thu, Dec 02, 2021 at 11:00:33PM +1100, Chris Angelico wrote: I'm still unsure whether this is a cool feature or an utter abomination: the parameter unable to retrieve the default, then the implementation is fatally broken.
It absolutely is not a feature.
It's backward incompatible: 15:03:04 R:\>python Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.
def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f() You passed x as Ellipsis
So I must agree with Steven that this should not be a feature. Best wishes Rob Cliffe
On Fri, Dec 3, 2021 at 2:11 AM Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:
On 02/12/2021 14:47, Steven D'Aprano wrote:
def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f.__defaults_extra__ = ("n/a",) f(42) You passed x as 42 f() You didn't pass x That is absolutely an abomination. If your implementation has the side-effect that setting a regular early-bound default to Ellipsis makes
On Thu, Dec 02, 2021 at 11:00:33PM +1100, Chris Angelico wrote: I'm still unsure whether this is a cool feature or an utter abomination: the parameter unable to retrieve the default, then the implementation is fatally broken.
It absolutely is not a feature.
It's backward incompatible:
15:03:04 R:\>python Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.
def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f() You passed x as Ellipsis
So I must agree with Steven that this should not be a feature.
Clearly I shouldn't post code examples without lots and lots of explanatory comments. https://www.python.org/dev/peps/pep-0671/#implementation-details # REDEFINE THE INTERPRETER'S UNDERSTANDING # OF THE LATE BOUND DEFAULTS FOR THE FUNCTION f.__defaults_extra__ = ("n/a",) There, now it's a bit clearer what's going on. ChrisA
On Fri, Dec 3, 2021 at 1:53 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Thu, Dec 02, 2021 at 11:00:33PM +1100, Chris Angelico wrote:
On Thu, Dec 2, 2021 at 8:40 PM Steven D'Aprano <steve@pearwood.info> wrote:
Depending on the implementation, you *might* be able to inspect the function and see the default expression as some sort of callable function, or evaluatable code object. (That would be nice.)
Unfortunately not, since the default expression could refer to other parameters, or closure variables, or anything else from the context of the called function. So you won't be able to externally evaluate it.
Why not? Functions can do all those things: refer to other variables, or closures, or anything else. You can call functions. Are you sure that this limitation of the default expression is not just a limitation of your implementation?
def f(): a = 1 def f(b, c=>a+b): return c a = 2 return f If there were a function to represent the late-bound default value for c, what parameters should it accept? How would you externally evaluate this? And also: what do you gain by it being a function, other than a lot of unnecessary overhead? And it is potentially a LOT of unnecessary overhead. Consider this edge case: def f(a, b=>c:=len(a)): ... In what context should the name c be bound? If there's a function for the evaluation of b, then that implies making c a closure cell, just for the sake of that. Every reference to c anywhere in the function (not just where it's set to len(a), but anywhere in f()) has to dereference that. It's a massive amount of completely unnecessary overhead AND a difficult question of which parts belong in the closure and which parts belong as parameters, which means that this is nearly impossible to define usefully.
I'm still unsure whether this is a cool feature or an utter abomination:
def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f.__defaults_extra__ = ("n/a",) f(42) You passed x as 42 f() You didn't pass x
That is absolutely an abomination. If your implementation has the side-effect that setting a regular early-bound default to Ellipsis makes the parameter unable to retrieve the default, then the implementation is fatally broken.
It absolutely is not a feature.
That's not what the example shows. It shows that changing dunder attributes can do this. I'm not sure why you think that the implementation is as restricted as you imply. The assignment to __defaults_extra__ is kinda significant here :) ChrisA
On Fri, Dec 03, 2021 at 02:10:12AM +1100, Chris Angelico wrote:
Unfortunately not, since the default expression could refer to other parameters, or closure variables, or anything else from the context of the called function. So you won't be able to externally evaluate it.
Why not? Functions can do all those things: refer to other variables, or closures, or anything else. You can call functions. Are you sure that this limitation of the default expression is not just a limitation of your implementation?
def f(): a = 1 def f(b, c=>a+b): return c a = 2 return f
If there were a function to represent the late-bound default value for c, what parameters should it accept?
I'm not saying that it *must* be a function. It could be a bare code object, that is `eval()`ed. Or something completely new. Dunno. But you're saying something is impossible, and that seems implausible to me, because things that seems *very similar* are totally possible.
How would you externally evaluate this?
inner = f() # Get the inner function. default = inner.__code__.__late_defaults__.wibble[1] # whatever try: value = default() except NameError: # Well, what did you expect to happen? value = eval(default.__code__, globals(), {'a': 101, 'b': 202}) Or something. The point is, rather than dismissing the possibility outright, this should be something we discuss, and carefully consider, before the PEP is complete.
And also: what do you gain by it being a function, other than a lot of unnecessary overhead?
Nicer introspection. Brendan goes from strongly opposed to the PEP to its biggest and most tireless supporter *wink* Cleaner separation of concerns: the defaults get handled independently of the function body. Plays nice with other tools that (say) use byte-code manipulation on the function body.
And it is potentially a LOT of unnecessary overhead. Consider this edge case:
def f(a, b=>c:=len(a)): ...
In what context should the name c be bound?
The obvious (which is not necessarily correct) answer is, the same scope that the expression is evaluated in, unless its declared global or nonlocal. (Which is only relevant if the function f() is nested in another function.) With regular defaults, the expression is evaluated at function definition time, and c gets bound to the surrounding scope. With late-bound defaults, the expression is evaluated at function call time, in the scope of f()'s locals. So c would be a local. Or the other obvious answer is that c will always be in the surrounding scope, for both early and late bound defaults. Consider the existence of walrus expressions in comprehensions:
def demo(a, b=((w:=i**2)*str(w) for i in range(5))): ... return b ... it = demo(None) next(it) '' next(it) '1' next(it) '4444' w 4
So there is precedent for having function-like entities (in this case, a comprehension) exposing their walrus variables in the surrounding scope. The third obvious answer is that if either the decision or the implementation is really too hard, then make it a syntax error for now, and revisit it in the future.
If there's a function for the evaluation of b, then that implies making c a closure cell, just for the sake of that. Every reference to c anywhere in the function (not just where it's set to len(a), but anywhere in f()) has to dereference that.
Okay. Is this a problem? If it really is a problem, then make it a syntax error to use walrus expressions inside late bound defaults.
It's a massive amount of completely unnecessary overhead AND a difficult question of which parts belong in the closure and which parts belong as parameters, which means that this is nearly impossible to define usefully.
I've given you two useful definitions. Its not clear what overhead you are worried about. Accessing variables in cells is almost as fast as accessing locals, but even if they were as slow as globals, premature optimization is the root of all evil. Globals are fast enough. Or are you worried about the memory overhead of the closures? The extra cost of fetching and calling the functions when evaluating the defaults? None of these things seem to be good reasons to dismiss the idea that default expressions should be independent of the function body. "Using a walrus expression in the default expression will make your function 3% slower and 1% larger, so therefore we must not make the default expression an introspectable code object..."
I'm still unsure whether this is a cool feature or an utter abomination:
def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f.__defaults_extra__ = ("n/a",) f(42) You passed x as 42 f() You didn't pass x
[...]
That's not what the example shows. It shows that changing dunder attributes can do this. I'm not sure why you think that the implementation is as restricted as you imply. The assignment to __defaults_extra__ is kinda significant here :)
Ah, well that is not so clear to people who aren't as immersed in the implementation as you :-) Messing about with function dunders can do weird shit:
def func(a=1, b=2): ... return a+b ... func.__defaults__ = (1, 2, 3, 4, 5) func() 9
I wouldn't worry about it. -- Steve
On Fri, Dec 3, 2021 at 12:48 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Dec 03, 2021 at 02:10:12AM +1100, Chris Angelico wrote:
Unfortunately not, since the default expression could refer to other parameters, or closure variables, or anything else from the context of the called function. So you won't be able to externally evaluate it.
Why not? Functions can do all those things: refer to other variables, or closures, or anything else. You can call functions. Are you sure that this limitation of the default expression is not just a limitation of your implementation?
def f(): a = 1 def f(b, c=>a+b): return c a = 2 return f
If there were a function to represent the late-bound default value for c, what parameters should it accept?
I'm not saying that it *must* be a function. It could be a bare code object, that is `eval()`ed. Or something completely new. Dunno.
But you're saying something is impossible, and that seems implausible to me, because things that seems *very similar* are totally possible.
How would you externally evaluate this?
inner = f() # Get the inner function. default = inner.__code__.__late_defaults__.wibble[1] # whatever try: value = default() except NameError: # Well, what did you expect to happen? value = eval(default.__code__, globals(), {'a': 101, 'b': 202})
Or something. The point is, rather than dismissing the possibility outright, this should be something we discuss, and carefully consider, before the PEP is complete.
How, with external calling, are you going to know which name references to look up, and where to get their values from? This is already a virtually impossible task in Python, which is why f-strings cannot be implemented as str.format(**something) for any known meaning of "something". You cannot get your *current* variable set, much less the available variables in some other context.
And it is potentially a LOT of unnecessary overhead. Consider this edge case:
def f(a, b=>c:=len(a)): ...
In what context should the name c be bound?
With late-bound defaults, the expression is evaluated at function call time, in the scope of f()'s locals. So c would be a local.
In other words, if you're trying to evaluate b's default externally, you have to set c in a context that doesn't even exist yet. Is that correct?
The third obvious answer is that if either the decision or the implementation is really too hard, then make it a syntax error for now, and revisit it in the future.
Everything's perfectly well defined, save that you can't externally evaluate the defaults. You can quite happily use the walrus in a late-bound default, and it'll bind to the function's locals. (Though I don't recommend it. I think that that's going to make for less-readable code than alternatives. Still, it's perfectly legal and well-defined.)
It's a massive amount of completely unnecessary overhead AND a difficult question of which parts belong in the closure and which parts belong as parameters, which means that this is nearly impossible to define usefully.
I've given you two useful definitions.
Its not clear what overhead you are worried about.
Accessing variables in cells is almost as fast as accessing locals, but even if they were as slow as globals, premature optimization is the root of all evil. Globals are fast enough.
Or are you worried about the memory overhead of the closures? The extra cost of fetching and calling the functions when evaluating the defaults? None of these things seem to be good reasons to dismiss the idea that default expressions should be independent of the function body.
The problem is that you're trying to reference cell variables in a closure that might not even exist yet, so *just in case* you might have nonlocals, you have to construct a closure... or find an existing one. Ill-defined and inefficient.
That's not what the example shows. It shows that changing dunder attributes can do this. I'm not sure why you think that the implementation is as restricted as you imply. The assignment to __defaults_extra__ is kinda significant here :)
Ah, well that is not so clear to people who aren't as immersed in the implementation as you :-)
Maybe, but I did think that the fact that I was assigning to a dunder should be obvious to people who are reading here :)
Messing about with function dunders can do weird shit:
def func(a=1, b=2): ... return a+b ... func.__defaults__ = (1, 2, 3, 4, 5) func() 9
I wouldn't worry about it.
Good, so, not an abomination. (In your opinion. Others are, of course, free to abominate as they please.) ChrisA
On Fri, Dec 03, 2021 at 01:08:50PM +1100, Chris Angelico wrote:
How, with external calling, are you going to know which name references to look up, and where to get their values from?
Isn't the source code available as a string? I will see the names in the expression. Or I will disassemble the code object and inspect its byte-code and/or AST. Or read the documentation. Or maybe the code object will expose those names in a dunder and I can just look at that.
This is already a virtually impossible task in Python, which is why f-strings cannot be implemented as str.format(**something) for any known meaning of "something".
f-strings cannot be implemented as str.format because str.format doesn't evaluate arbitrary expressions.
f'{1000+len(repr(None))}' '1004'
You cannot get your *current* variable set, much less the available variables in some other context.
Why not? The default expression object can record nonlocal variables in a closure, and you can provide your own globals and/or locals. You're saying "can't, impossible" but I could take your function object, extract the code object, disassemble the code to AST or byte-code, extract the portions that represent the default expression, and evaluate that. So not *impossible*, just bloody inconvenient and annoying. (Well, when I say *I could*, I mean *somebody with mad leet AST or byte- code hacking skillz. Not actually me.)
And it is potentially a LOT of unnecessary overhead. Consider this edge case:
def f(a, b=>c:=len(a)): ...
In what context should the name c be bound?
With late-bound defaults, the expression is evaluated at function call time, in the scope of f()'s locals. So c would be a local.
In other words, if you're trying to evaluate b's default externally, you have to set c in a context that doesn't even exist yet. Is that correct?
Yes? You seem to be saying that as if it were some fatal flaw in my proposal. mylocals = {} There. Now the context exists. result = eval(default_expression.__code__, globals(), mylocals())
The third obvious answer is that if either the decision or the implementation is really too hard, then make it a syntax error for now, and revisit it in the future.
Everything's perfectly well defined, save that you can't externally evaluate the defaults.
You keep saying that, but your arguments are not convincing.
You can quite happily use the walrus in a late-bound default, and it'll bind to the function's locals.
Cool :-)
The problem is that you're trying to reference cell variables in a closure that might not even exist yet, so *just in case* you might have nonlocals, you have to construct a closure... or find an existing one. Ill-defined and inefficient.
That doesn't make sense to me. When you compile the default expression, you know what variable names are assigned to with a walrus expression (if any), you know what names exist in the surrounding nonlocal scope (if any), you know what names are parameters. If the default expression doesn't use walrus, doesn't refer to any nonlocal names, and doesn't refer to any other parameters, why would you construct a closure "just in case"? What would you put in it? -- Steve
On Fri, Dec 3, 2021 at 2:24 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Dec 03, 2021 at 01:08:50PM +1100, Chris Angelico wrote:
How, with external calling, are you going to know which name references to look up, and where to get their values from?
Isn't the source code available as a string? I will see the names in the expression.
Or I will disassemble the code object and inspect its byte-code and/or AST. Or read the documentation. Or maybe the code object will expose those names in a dunder and I can just look at that.
This is already a virtually impossible task in Python, which is why f-strings cannot be implemented as str.format(**something) for any known meaning of "something".
f-strings cannot be implemented as str.format because str.format doesn't evaluate arbitrary expressions.
f'{1000+len(repr(None))}' '1004'
You cannot get your *current* variable set, much less the available variables in some other context.
Why not? The default expression object can record nonlocal variables in a closure, and you can provide your own globals and/or locals.
You're saying "can't, impossible" but I could take your function object, extract the code object, disassemble the code to AST or byte-code, extract the portions that represent the default expression, and evaluate that. So not *impossible*, just bloody inconvenient and annoying.
(Well, when I say *I could*, I mean *somebody with mad leet AST or byte- code hacking skillz. Not actually me.)
When f-strings were first proposed, one of the counter-arguments was basically "just use globals()/locals()". https://www.python.org/dev/peps/pep-0498/#no-use-of-globals-or-locals But you cannot, in a fully general way, solve the problem that way. It is *impossible*. Not just difficult. Impossible. You cannot, in Python as it currently is, look up arbitrary names in any reliable way. This is especially true now that we have assignment expressions, such that evaluating the function default could create a new local or nonlocal name. Not just inconvenient and annoying. So impossible that a str method could not be written to do it. ChrisA
On 2021-12-02 01:35, Steven D'Aprano wrote:
4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)
No. As I mentioned in the earlier thread, I don't support any proposal in which an argument can "have a default" but that default is not a first-class Python object of some sort. I don't understand this criticism.
Of course the default value will be a first-class Python object of some sort.*Every* value in Python is a first-class object. There are no machine values or unboxed values, and this proposal will not change that.
All that this proposal changes is*when* and*how often* the default will be evaluated, not the nature of the value.
As has happened often in these threads, it seems different people mean different things by "default value". What you are calling "the default value" is "a thing that is used at call time if no value is passed for the argument". What I am calling "the default value" is "a thing that is noted at definition time to be used later if no value is passed for the argument". What I'm saying is that I want that "thing" to exist. At the time the function is defined, I want there to be a Python object which represents the behavior to be activated at call time if the argument is not passed. In the current proposal there is no such "thing". The function just has behavior melded with its body that does stuff, but there is no addressable "thing" where you can say "if you call the function and the argument isn't passed were are going to take this default-object-whatchamacallit and 'use' it (in some defined way) to get the default value". This is what we already have for early-bound defaults in the function's `__defaults__` attribute. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Fri, Dec 3, 2021 at 6:24 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
What I'm saying is that I want that "thing" to exist. At the time the function is defined, I want there to be a Python object which represents the behavior to be activated at call time if the argument is not passed. In the current proposal there is no such "thing". The function just has behavior melded with its body that does stuff, but there is no addressable "thing" where you can say "if you call the function and the argument isn't passed were are going to take this default-object-whatchamacallit and 'use' it (in some defined way) to get the default value". This is what we already have for early-bound defaults in the function's `__defaults__` attribute.
What would you do with this object? Suppose you have this function: def f(lst=>[], n=>len(lst)): ... What can you usefully do with this hypothetical object representing the default for n? There is no "thing" and no way you could "use" that thing to predict the value of n without first knowing the full context of the function. With early-bound defaults, you can meaningfully inspect them and see what the value is going to be. With late-bound defaults, should the default for lst be an empty list? Should the default for n be 0? Neither is truly correct (but they're wrong in different ways). Do you actually need that object for anything, or is it simply for the sake of arbitrary consistency ("we have objects for early-bound defaults, why can't we have objects for late-bound defaults")? The value that will be assigned to the parameter does not exist until the function is actually called, and may depend on all kinds of things about the function and its context. Until then, there is no value, no object, that can represent it. I put to you the same question I put to David Mertz: how is a late-bound default different from half of a conditional expression? def f(*args): lst = args[0] if len(args) > 0 else [] n = args[1] if len(args) > 1 else len(lst) ... Do the expressions "[]" and "len(lst)" have to have object representations in this form? If not, why should they need object representations when written in the more natural way as "lst=>[]"? ChrisA
On Thu, Dec 2, 2021 at 2:40 PM Chris Angelico <rosuav@gmail.com> wrote:
How is a late-bound default different from half of a conditional expression?
def f(lst=>[], n=>len(lst)):
def f(*args): lst = args[0] if len(args) > 0 else [] n = args[1] if len(args) > 1 else len(lst)
Although such is obviously not your intention, I think you have provided a stronger argument against this feature/PEP than any other I've seen raised so far. Well, maybe just against the specific implementation you have in mind. You are correct, of course, that the second form does not provide inspectability for `lst` and `n`. Well, it does, but only in a fairly contorted way of disassembling f.__code__.co_code. Or maybe with an equally indirect look at the parse tree or something. The body of a function is very specifically A BODY. What your proposal/implementation does is put things into the function signature that are simultaneously excluded from direct inspectability as function attributes. Python, like almost all programming languages, makes a pretty clear distinction between function signatures and function bodies. You propose to remove that useful distinction, or at least weaken it. For the reasons Eric Smith and others have pointed out, I really WANT to keep inspectability of function signatures. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Fri, Dec 3, 2021 at 8:07 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
On Thu, Dec 2, 2021 at 2:40 PM Chris Angelico <rosuav@gmail.com> wrote:
How is a late-bound default different from half of a conditional expression?
def f(lst=>[], n=>len(lst)):
def f(*args): lst = args[0] if len(args) > 0 else [] n = args[1] if len(args) > 1 else len(lst)
Although such is obviously not your intention, I think you have provided a stronger argument against this feature/PEP than any other I've seen raised so far. Well, maybe just against the specific implementation you have in mind.
Fascinating.
You are correct, of course, that the second form does not provide inspectability for `lst` and `n`. Well, it does, but only in a fairly contorted way of disassembling f.__code__.co_code. Or maybe with an equally indirect look at the parse tree or something. The body of a function is very specifically A BODY.
What your proposal/implementation does is put things into the function signature that are simultaneously excluded from direct inspectability as function attributes. Python, like almost all programming languages, makes a pretty clear distinction between function signatures and function bodies. You propose to remove that useful distinction, or at least weaken it.
Actually, no. I want to put the default arguments into the signature, and the body in the body. The distinction currently has a technical restriction that means that, in certain circumstances, what belongs in the signature has to be hacked into the body. I'm trying to make it so that those can be put where they belong.
For the reasons Eric Smith and others have pointed out, I really WANT to keep inspectability of function signatures.
Here's what you get:
def f(lst=>[], n=>len(lst)): ... ... f.__defaults_extra__ ('[]', 'len(lst)')
String representation, but exactly what the default is. Any tool that inspects a signature (notably, help() etc) will be able to access this. Now write the function the other way, and show me how easy it is to determine the behaviour when arguments are omitted. What can you access? ChrisA
On 2021-12-02 15:40, Chris Angelico wrote:
Actually, no. I want to put the default arguments into the signature, and the body in the body. The distinction currently has a technical restriction that means that, in certain circumstances, what belongs in the signature has to be hacked into the body. I'm trying to make it so that those can be put where they belong.
Chris, I know this is probably not your intention, but I feel the discussion is continually muddle by you just saying "default arguments" as if everyone agrees on what those are and the issue is just where to put them. But clearly that is not the case. You seem to take it for granted that "len(x) evaluated when the function is called" "is" a "default argument" in the same sense that an early-bound default like the number 3 is. I do not agree, and it's pretty clear David Mertz does not agree, and I think there are others here who also do not agree. It's not as is there is some pre-existing notion of "default argument" in Python and you are just proposing to add an implementation of it that was left out due to some kind of oversight. Your proposal is CHANGING the idea of what a default argument is. I get that it's natural to refer to your new proposal as "default arguments" but I've now seen a number of messages here where you say "but no we should do X because this is a default argument", taking for granted that what you're talking about is already agreed to be a default argument. (No doubt I and many others are also guilty of similar missteps by saying "default argument" as a shorthand for something or other, and I'll try to be careful about it myself.) By my definition as of now, a default argument has to be an object. Thus what your proposal envisions are not default arguments. You can't just say "I want to do this with default arguments". You need to provide an argument for why we should even consider these to be default arguments at all. Perhaps a way of stating this without reference to arguments is this: you want to put code in the function signature but not have it executed until the function is called. I do not agree with that choice. The function signature and the function body are different syntactic environments with different semantics. Everything in the function signature should be executed when the function is defined (although of course that execution can result in an object which contains deferred behavior itself, like if we pass a function as an argument to another). Only the function body should be executed when the function is called. If we want to provide some way to augment the function body to assign values to missing arguments, I wouldn't rule that out completely, but in my view the function signature is not an available space for that. The function signature is ONLY for things that happen when the function is defined. It is too confusing to have the function signature mix definition-time and call-time behavior. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Fri, Dec 3, 2021 at 2:30 PM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2021-12-02 15:40, Chris Angelico wrote:
Actually, no. I want to put the default arguments into the signature, and the body in the body. The distinction currently has a technical restriction that means that, in certain circumstances, what belongs in the signature has to be hacked into the body. I'm trying to make it so that those can be put where they belong.
Chris, I know this is probably not your intention, but I feel the discussion is continually muddle by you just saying "default arguments" as if everyone agrees on what those are and the issue is just where to put them. But clearly that is not the case. You seem to take it for granted that "len(x) evaluated when the function is called" "is" a "default argument" in the same sense that an early-bound default like the number 3 is. I do not agree, and it's pretty clear David Mertz does not agree, and I think there are others here who also do not agree.
It's not as is there is some pre-existing notion of "default argument" in Python and you are just proposing to add an implementation of it that was left out due to some kind of oversight. Your proposal is CHANGING the idea of what a default argument is. I get that it's natural to refer to your new proposal as "default arguments" but I've now seen a number of messages here where you say "but no we should do X because this is a default argument", taking for granted that what you're talking about is already agreed to be a default argument. (No doubt I and many others are also guilty of similar missteps by saying "default argument" as a shorthand for something or other, and I'll try to be careful about it myself.)
Some functions most assuredly DO have a pre-existing notion of "default argument". Some do not. Allow me to give a few examples: def getattr(object, name, default): Omitting the third argument does not have a default. It will cause different behaviour (getattr will raise an exception). def dict.get(key, default=None): Omitting the last argument behaves exactly as if you passed None. This has a default; the function behaves identically whether you pass an argument or not, and you can determine what the behaviour would be if you do. (These two are a little confusing in that they use the name "default", but the fact is that default arguments are often used for defaults, surprise surprise. So I'm going to be reusing that word a lot.) def open(file, mode='r', encoding=????): Omitting mode is exactly the same as passing 'r'. Omitting encoding is exactly the same as passing locale.getpreferredencoding(False). Both of these have real defaults. def bisect.bisect(a, x, lo=0, hi=len(a)): Omitting lo is exactly the same as passing 0. Omitting hi is exactly the same as passing len(a). Both of these have real defaults. For parameters that do not have defaults, PEP 671 has nothing to say. Continue doing what you're already doing (whether that's a sentinel, or *args, or whatever), as there's nothing needing to be changed. For parameters whose defaults are simple constants, PEP 671 also has nothing to say. Continue defining them as early-bound defaults, and the behaviour will not change. The difference comes with those arguments whose true default is calculated in some way, or is dynamic. These really truly do have default values, and there is absolutely no behavioural difference between hi=len(a) and omitting the hi parameter to bisect(). Due to a technical limitation, though, the documentation for bisect() has to explain this in the docstring rather than the function signature. Would this code pass review? def spaminate(msg, times=None): if times is None: times = 50 ... Unless there's a very VERY good reason for using None as the default, shouldn't it use 50 in the signature? It would be a clearer declaration of intent: omitting this argument is the same as passing 50.
By my definition as of now, a default argument has to be an object. Thus what your proposal envisions are not default arguments. You can't just say "I want to do this with default arguments". You need to provide an argument for why we should even consider these to be default arguments at all.
Yes, that is a current, and technical, limitation. Consider this: https://en.wikipedia.org/wiki/Default_argument#Evaluation Python is *notable* for always evaluating default arguments just once. The concept of default args, as described there and as understood by anyone that I have asked (mainly students, which I admit isn't an unbiased sample, but nothing is), is that they should be evaluated at call time. My argument for why they should be considered default arguments is that, aside from Python being unable to represent them, they *are* precisely what default arguments are. Suppose you're on the ECMAScript committee, and someone proposes that the language support bignums. (That's not truly hypothetical, incidentally - I saw references to such a proposal.) Would you argue that 9007199254740993 is not really a number, on the basis that current ECMAScript cannot represent it? Would you force someone to prove that it is really a number? It's only special because of implementation restrictions.
Perhaps a way of stating this without reference to arguments is this: you want to put code in the function signature but not have it executed until the function is called. I do not agree with that choice. The function signature and the function body are different syntactic environments with different semantics. Everything in the function signature should be executed when the function is defined (although of course that execution can result in an object which contains deferred behavior itself, like if we pass a function as an argument to another).
That's reasonable, but I disagree from a perspective of practicality: logically and usefully, it is extremely helpful to be able to describe arguments that way. Passing positional parameters is approximately equivalent to: def func(*args): """func(a, b, c=1, d=a+b)""" a, args = args b, args = args if args: c, args = args else: c = 1 if args: d, args = args else: d = a + b If you try to explain it to someone who's learning about default argument values, do you first have to explain that they get evaluated and recorded somewhere, and these indescribable values are what's actually assigned? Or would you describe it mostly like this (or as "if c is omitted: c = 1", which comes to the same thing)? Argument default snapshotting is an incredibly helpful performance advantage, but it isn't inherent to the very concept of default arguments.
Only the function body should be executed when the function is called. If we want to provide some way to augment the function body to assign values to missing arguments, I wouldn't rule that out completely, but in my view the function signature is not an available space for that. The function signature is ONLY for things that happen when the function is defined. It is too confusing to have the function signature mix definition-time and call-time behavior.
The trouble is that the function signature MUST be the one and only place where you figure out whether the argument is optional or mandatory. Otherwise there is an endless sea of debugging nightmares - just ask anyone who works regularly in JavaScript, where all arguments are optional and will default to the special value 'undefined' if omitted. So you'd need to separate "this is optional" from "if omitted, do this". That might be a possibility, but it would need two separate compiler features: def func(a, b, c=1, d=?): if unset d: d = a+b where "unset N" is a unary operator which returns True if the name would raise, False if not. I'm not proposing that, but if someone wants to, I would be happy to share my reference implementation, since 90% of the code is actually there :) ChrisA
On Fri, Dec 03, 2021 at 10:40:42AM +1100, Chris Angelico wrote:
Here's what you get:
def f(lst=>[], n=>len(lst)): ... ... f.__defaults_extra__ ('[]', 'len(lst)')
String representation, but exactly what the default is.
Excellent. And you've just proven that we can evaluate the defaults. >>> a = '[]' # f.__defaults_extra__[0] >>> b = 'len(lst)' # f.__defaults_extra__[1] >>> lst = eval(a, globals()) >>> n = eval(b, globals(), dict(lst=lst)) >>> print(lst, n) [] 0 Worst case scenario, we can eval the string representation, and that should work for the great majority of cases that don't involve nonlocals. But if the defaults are proper code objects, complete with a closure to capture their nonlocal environment, then we should be able to do even better and capture nonlocals as well. Could there be odd corner cases that don't quite work? Say, like a comprehension inside a class body? https://github.com/satwikkansal/wtfpython#-name-resolution-ignoring-class-sc... Oh well. Let's not make the perfect the enemy of the good. -- Steve
On Sat, Dec 4, 2021 at 2:34 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Dec 03, 2021 at 10:40:42AM +1100, Chris Angelico wrote:
Here's what you get:
def f(lst=>[], n=>len(lst)): ... ... f.__defaults_extra__ ('[]', 'len(lst)')
String representation, but exactly what the default is.
Excellent. And you've just proven that we can evaluate the defaults.
>>> a = '[]' # f.__defaults_extra__[0] >>> b = 'len(lst)' # f.__defaults_extra__[1] >>> lst = eval(a, globals()) >>> n = eval(b, globals(), dict(lst=lst)) >>> print(lst, n) [] 0
Worst case scenario, we can eval the string representation, and that should work for the great majority of cases that don't involve nonlocals.
Yes, those awkward cases that involve nonlocals. A pity about those. But hey, if that's not a problem to you, then sure, go ahead, just eval it. The string is there for you to use.
But if the defaults are proper code objects, complete with a closure to capture their nonlocal environment, then we should be able to do even better and capture nonlocals as well.
Could there be odd corner cases that don't quite work? Say, like a comprehension inside a class body?
https://github.com/satwikkansal/wtfpython#-name-resolution-ignoring-class-sc...
Oh well. Let's not make the perfect the enemy of the good.
Lots and lots and lots of potential problems. Consider: def f(): a = 1 def f(b, x=>a+b): def g(): return x, a, b Both a and b are closure variables - one because it comes from an outer scope, one because it's used in an inner scope. So to evaluate a+b, you have to look up an existing closure cell, AND construct a new closure cell. The only way to do that is for the compiled code of a+b to exist entirely within the context of f's code object. Which means that there isn't really anything to usefully pull out and evaluate, since it can't be executed until you've set up a stack frame to call f, and once you've done that, you're basically just... calling f anyway. If you still dispute that it's impossible, you're absolutely welcome to write your own reference implementation. I've argued this point with as much detail as I can, and short of pointing stuff out in the code, I can't do anything more. If you want to keep on saying "but of COURSE it's possible", then go ahead, prove it to me. ChrisA
On Sat, Dec 04, 2021 at 03:14:46PM +1100, Chris Angelico wrote:
Lots and lots and lots of potential problems. Consider:
def f(): a = 1 def f(b, x=>a+b): def g(): return x, a, b
Both a and b are closure variables - one because it comes from an outer scope, one because it's used in an inner scope. So to evaluate a+b, you have to look up an existing closure cell, AND construct a new closure cell.
The only way to do that is for the compiled code of a+b to exist entirely within the context of f's code object.
I dispute that is the only way. Let's do a thought experiment. First, we add a new flag to the co_flags field on code objects. Call it the "LB" flag, for late-binding. Second, we make this: def f(b, x=>a+b): ... syntactic sugar for this: def f(b, x=lambda b: a+b): ... except that the lambda has the LB flag set. And third, when the interpreter fetches a default from func.__defaults__, if it is a LB function, it automatically calls that function with the parameters to the left of x (which in this case would be just b). Here's your function, with a couple of returns to make it actually do something: def f(): a = 1 def f(b, x=>a+b): def g(): return x, a, b return g return f We can test that right now (well, almost all of it) with this: def func(): # change of name to distinguish inner and outer f a = 1 def f(b, x=lambda b: a+b): def g(): return x, a, b return g return f and just pretend that x is automatically evaluated by the interpreter. But as a proof of concept, it's enough that we can demonstrate that *we* can manually evaluate it, by calling the lambda. We can call func() to get the inner function f, and call f to get g: >>> f = func() >>> print(f) <function func.<locals>.f at 0x7fc945c41f30> >>> g = f(100) >>> print(g) <function func.<locals>.f.<locals>.g at 0x7fc945e1f520> Calling g works: >>> print(g()) (<function func.<locals>.<lambda> at 0x7fc945c40f70>, 1, 100) with the understanding that the real implementation will have automatically called that lambda, so we would have got 101 instead of the lambda. That step requires interpreter support, so for now we just have to pretend that we get (101, 1, 100) instead of the lambda. But we can demonstrate that calling the lambda works, by manually calling it: >>> x = g()[0] >>> print(x) <function func.<locals>.<lambda> at 0x7fc945c40f70> >>> print(x(100)) # the interpreter knows that b=100 101 Now let's see if we can extract the default and play around with it: >>> default_expression = f.__defaults__[0] >>> print(default_expression) <function func.<locals>.<lambda> at 0x7fc945c40f70> The default expression is just a function (with the new LB flag set). So we can inspect its name, its arguments, its cell variables, etc: >>> default_expression.__closure__ (<cell at 0x7fc945de74f0: int object at 0x7fc94614c0f0>,) We can do anything that we could do with any other other function object. Can we evaluate it? Of course we can. And we can test it with any value we like, we're not limited to the value of b that we originally passed to func(). >>> default_expression(3000) 3001 Of course, if we are in a state of *maximal ignorance* we might have no clue what information is needed to evaluate that default expression: >>> default_expression() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: func.<locals>.<lambda>() missing 1 required positional argument: 'b' Oh look, we get a useful diagnostic message for free! What are we missing? The source code of the original expression, as text. That's pretty easy too: the compiler knows the source, it can cram it into the default expression object: >>> default_expression.__expression__ = 'a+b' Introspection tools like help() can learn to look for that. What else are we missing? A cool repr. >>> print(default_expression) # Simulated. <late bound default expression a+b> We can probably come up with a better repr, and a better name than "late bound default expression". We already have other co_flags that change the repr: 32 GENERATOR 128 COROUTINE 256 ITERABLE_COROUTINE so we need a name that is at least as cool as "generator" or "coroutine". Summary of changes: * add a new co_flag with a cool name better than "LB"; * add an `__expression__` dunder to hold the default expression; (possibly missing for regular functions -- we don't necessarily need *every* function to have this dunder) * change the repr of LB functions to display the expression; * teach the interpreter to compile late-bound defaults into one of these LB functions, including the source expression; * teach the interpreter that when retrieving default values from the function's `__defaults__`, if they are a LB function, it must call the function and use its return result as the actual default value; * update help() and other introspection tools to handle these LB functions; but if any tools don't get updated, you still get a useful result with an informative repr. -- Steve
On Sat, Dec 4, 2021 at 8:48 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Dec 04, 2021 at 03:14:46PM +1100, Chris Angelico wrote:
Lots and lots and lots of potential problems. Consider:
def f(): a = 1 def f(b, x=>a+b): def g(): return x, a, b
Both a and b are closure variables - one because it comes from an outer scope, one because it's used in an inner scope. So to evaluate a+b, you have to look up an existing closure cell, AND construct a new closure cell.
The only way to do that is for the compiled code of a+b to exist entirely within the context of f's code object.
I dispute that is the only way. Let's do a thought experiment.
First, we add a new flag to the co_flags field on code objects. Call it the "LB" flag, for late-binding.
Second, we make this:
def f(b, x=>a+b): ...
syntactic sugar for this:
def f(b, x=lambda b: a+b): ...
except that the lambda has the LB flag set.
Okay. So the references to 'a' and 'b' here are one more level of function inside the actual function we're defining, which means you're paying the price of nonlocals just to be able to late-evaluate defaults. Not a deal-breaker, but that is a notable cost (every reference to them inside the function will be slower).
And third, when the interpreter fetches a default from func.__defaults__, if it is a LB function, it automatically calls that function with the parameters to the left of x (which in this case would be just b).
Plausible. Okay. What this does mean, though, is that there are "magic objects" that cannot be used like other objects. Consider: def make_printer(dflt): def func(x=dflt): print("x is", x) return func Will make_printer behave the same way for all objects? Clearly the expectation is that it will display the repr of whichever object is passed to func, or if none is, whichever object is passed to make_printer. But if you pass it a function with the magic LB flag set, it will *execute* that function. I don't like the idea that some objects will be invisibly different like that.
Here's your function, with a couple of returns to make it actually do something:
def f(): a = 1 def f(b, x=>a+b): def g(): return x, a, b return g return f
We can test that right now (well, almost all of it) with this:
def func(): # change of name to distinguish inner and outer f a = 1 def f(b, x=lambda b: a+b): def g(): return x, a, b return g return f
and just pretend that x is automatically evaluated by the interpreter. But as a proof of concept, it's enough that we can demonstrate that *we* can manually evaluate it, by calling the lambda.
Okay, sure. It's a bit hard to demo it (since it has to ONLY do that magic if the arg was omitted), but sure, we can pretend.
We can call func() to get the inner function f, and call f to get g:
>>> f = func() >>> print(f) <function func.<locals>.f at 0x7fc945c41f30>
>>> g = f(100) >>> print(g) <function func.<locals>.f.<locals>.g at 0x7fc945e1f520>
Calling g works:
>>> print(g()) (<function func.<locals>.<lambda> at 0x7fc945c40f70>, 1, 100)
with the understanding that the real implementation will have automatically called that lambda, so we would have got 101 instead of the lambda. That step requires interpreter support, so for now we just have to pretend that we get
(101, 1, 100)
instead of the lambda. But we can demonstrate that calling the lambda works, by manually calling it:
>>> x = g()[0] >>> print(x) <function func.<locals>.<lambda> at 0x7fc945c40f70> >>> print(x(100)) # the interpreter knows that b=100 101
Now let's see if we can extract the default and play around with it:
>>> default_expression = f.__defaults__[0] >>> print(default_expression) <function func.<locals>.<lambda> at 0x7fc945c40f70>
The default expression is just a function (with the new LB flag set). So we can inspect its name, its arguments, its cell variables, etc:
>>> default_expression.__closure__ (<cell at 0x7fc945de74f0: int object at 0x7fc94614c0f0>,)
We can do anything that we could do with any other other function object.
Yup. As long as it doesn't include any assignment expressions, or anything else that would behave differently.
Can we evaluate it? Of course we can. And we can test it with any value we like, we're not limited to the value of b that we originally passed to func().
>>> default_expression(3000) 3001
Of course, if we are in a state of *maximal ignorance* we might have no clue what information is needed to evaluate that default expression:
>>> default_expression() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: func.<locals>.<lambda>() missing 1 required positional argument: 'b'
Oh look, we get a useful diagnostic message for free!
What are we missing? The source code of the original expression, as text. That's pretty easy too: the compiler knows the source, it can cram it into the default expression object:
>>> default_expression.__expression__ = 'a+b'
Introspection tools like help() can learn to look for that.
What else are we missing? A cool repr.
>>> print(default_expression) # Simulated. <late bound default expression a+b>
We can probably come up with a better repr, and a better name than "late bound default expression". We already have other co_flags that change the repr:
32 GENERATOR 128 COROUTINE 256 ITERABLE_COROUTINE
so we need a name that is at least as cool as "generator" or "coroutine".
Those parts are trivial, no problem.
Summary of changes:
* add a new co_flag with a cool name better than "LB";
* add an `__expression__` dunder to hold the default expression; (possibly missing for regular functions -- we don't necessarily need *every* function to have this dunder)
* change the repr of LB functions to display the expression;
* teach the interpreter to compile late-bound defaults into one of these LB functions, including the source expression;
* teach the interpreter that when retrieving default values from the function's `__defaults__`, if they are a LB function, it must call the function and use its return result as the actual default value;
* update help() and other introspection tools to handle these LB functions; but if any tools don't get updated, you still get a useful result with an informative repr.
Great. So now we have some magnificently magical behaviour in the language, which will have some nice sharp edge cases, but which nobody will ever notice. Totally. I'm sure. Plus, we pay a performance price in any function that makes use of argument references, not just for the late-bound default, but in the rest of the code. We also need to have these special functions that get stored as separate code objects. All to buy what, exactly? The ability to manually synthesize an equivalent parameter value, as long as there's no assignment expressions, no mutation, no other interactions, etc, etc, etc? That's an awful lot of magic for not a lot of benefit. I *really* don't like the idea that some types of object will be executed instead of being used, just because they have a flag set. That strikes me as the sort of thing that should be incredibly scary, but since I can't think of any specific reasons, I just have to call it "extremely off-putting". But hey. Go ahead and build a reference implementation. I'll compile it and give it a whirl. ChrisA
On 2021-12-04 03:50, Chris Angelico wrote:
On Sat, Dec 4, 2021 at 8:48 PM Steven D'Aprano <steve@pearwood.info> wrote:
And third, when the interpreter fetches a default from func.__defaults__, if it is a LB function, it automatically calls that function with the parameters to the left of x (which in this case would be just b).
Plausible. Okay.
What this does mean, though, is that there are "magic objects" that cannot be used like other objects. Consider:
Your proposal also has the same problem, since it involves "magic functions" that do not have usable values for their argument defaults, instead having some kind of Ellipsis two-step. It's all a matter of what you consider magic.
Great. So now we have some magnificently magical behaviour in the language, which will have some nice sharp edge cases, but which nobody will ever notice. Totally. I'm sure. Plus, we pay a performance price in any function that makes use of argument references, not just for the late-bound default, but in the rest of the code. We also need to have these special functions that get stored as separate code objects.
All to buy what, exactly? The ability to manually synthesize an equivalent parameter value, as long as there's no assignment expressions, no mutation, no other interactions, etc, etc, etc? That's an awful lot of magic for not a lot of benefit.
I would consider most of what you say here an accurate description of your own proposal. :-) Now we have magnificently magical behavior in the language, which will take expressions in the function signature and behind the scenes "inline" them into the function body. We also need to have these special function arguments that do NOT get stored as separate objects, unlike ordinary function arguments. All to buy what, exactly? The ability to write something in the function signature that we can already write in the body, and that quite naturally belongs in the body, because it is executed when the function is called, not when it is defined.
I *really* don't like the idea that some types of object will be executed instead of being used, just because they have a flag set. That strikes me as the sort of thing that should be incredibly scary, but since I can't think of any specific reasons, I just have to call it "extremely off-putting".
I *really* don't like the idea that some types of argument will be inlined into the function body instead of being stored as first-class values like other `__defaults__`, just because there happens to be this one extra character next to the equals sign in the function signature. That strikes me as the sort of thing that should be incredibly scary. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Sun, Dec 5, 2021 at 6:16 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2021-12-04 03:50, Chris Angelico wrote:
On Sat, Dec 4, 2021 at 8:48 PM Steven D'Aprano <steve@pearwood.info> wrote:
And third, when the interpreter fetches a default from func.__defaults__, if it is a LB function, it automatically calls that function with the parameters to the left of x (which in this case would be just b).
Plausible. Okay.
What this does mean, though, is that there are "magic objects" that cannot be used like other objects. Consider:
Your proposal also has the same problem, since it involves "magic functions" that do not have usable values for their argument defaults, instead having some kind of Ellipsis two-step. It's all a matter of what you consider magic.
My proposal allows any object to be used as a function default argument. There's a minor technical difference that means that there's a second lookup if you use Ellipsis, but you can still use Ellipsis just fine.
def f(x=...): ... print(type(x), x) ... f() <class 'ellipsis'> Ellipsis f(None) <class 'NoneType'> None f("spam") <class 'str'> spam
There are no objects that will behave differently if used in this way. EVERY object can be a function default argument. Steve's proposal has some objects (functions with the LB flag set) actually behave differently - they *will not behave correctly* if used in this way. This is a restriction placed on the rest of the language.
Great. So now we have some magnificently magical behaviour in the language, which will have some nice sharp edge cases, but which nobody will ever notice. Totally. I'm sure. Plus, we pay a performance price in any function that makes use of argument references, not just for the late-bound default, but in the rest of the code. We also need to have these special functions that get stored as separate code objects.
All to buy what, exactly? The ability to manually synthesize an equivalent parameter value, as long as there's no assignment expressions, no mutation, no other interactions, etc, etc, etc? That's an awful lot of magic for not a lot of benefit.
I would consider most of what you say here an accurate description of your own proposal. :-)
That's highly unfair. No, I won't let that pass. Please retract or justify that statement. You are quoting the conclusion of a lengthy post in which I show significant magic in Steve's proposal, contrasting it with mine which has much clearer behaviour, and you then say that my proposal has the same magic. Frankly, that is not a reasonable assertion, and I take offense.
Now we have magnificently magical behavior in the language, which will take expressions in the function signature and behind the scenes "inline" them into the function body. We also need to have these special function arguments that do NOT get stored as separate objects, unlike ordinary function arguments. All to buy what, exactly? The ability to write something in the function signature that we can already write in the body, and that quite naturally belongs in the body, because it is executed when the function is called, not when it is defined.
You assert that it "belongs in the body", but only because Python currently doesn't allow it to be anywhere else. Other languages have this exact information in the function signature. This is a much larger distinction than what Steve shows, which is the exact same feature but with these magic callables.
I *really* don't like the idea that some types of object will be executed instead of being used, just because they have a flag set. That strikes me as the sort of thing that should be incredibly scary, but since I can't think of any specific reasons, I just have to call it "extremely off-putting".
I *really* don't like the idea that some types of argument will be inlined into the function body instead of being stored as first-class values like other `__defaults__`, just because there happens to be this one extra character next to the equals sign in the function signature. That strikes me as the sort of thing that should be incredibly scary.
You're still being highly offensive here. There's a HUGE difference between these two assertions. Steve's proposal makes some objects *behave differently when used in existing features*. It would be like creating a new type of string which, when printed out, would eval itself. That proposal wouldn't fly, and it's why f-strings are most assuredly NOT first-class objects. Why is it such a big deal for these function default expressions to not be first-class objects? None of these are first-class either: print(f"An f-string's {x+y} subexpressions") print(x/y if y else "An if/else expression's sides") assign(x.y[42], "An assignment target") We don't have a problem with these being unable to be externally referenced, manipulated, etc, as first-class objects. Why is it a problem to be unable to refer to "new empty list" as some sort of object when used like this? def f(x=>[]): ... Can you explain why it is necessary? And then, after that, explain why you claim that Steve's proposal, which makes some objects *not even work in early-bound defaults*, is just as magical? ChrisA
On Sun, Dec 05, 2021 at 08:15:34AM +1100, Chris Angelico wrote:
There are no objects that will behave differently if used in this way. EVERY object can be a function default argument. Steve's proposal has some objects (functions with the LB flag set) actually behave differently - they *will not behave correctly* if used in this way. This is a restriction placed on the rest of the language.
Its a restriction in name only. I've already given you two work-arounds for the restriction, and one solution to that (non-)problem. We can change the trigger condition if its really an issue. But if we don't, there are still ways to get the result that you want. How many ways are there to remove the restriction that operator dunders cannot return NotImplemented as a first-class value? Zero. Another precedence is the `__new__` dunder. Classes can return anything they like from the constructor, but if you return an instance of the class, it will automatically trigger calling the `__init__` method, whether you want it to or not. You probably see that as a feature. In my "late-bound protocol", the fact that a function with the "late bound" flag triggers calling the function is likewise a feature, not a bug. Unlike NotImplemented and `__new__`, there are at least two work-arounds for the rare case where you don't want that. And as I said, if this is really a show-stopper, we can change the trigger condition. There are pros and cons and I'm not wedded to one way or the other.
You assert that it "belongs in the body", but only because Python currently doesn't allow it to be anywhere else. Other languages have this exact information in the function signature.
This argument about where the evaluation of the default expression is surreal. Brendan argues in favour of the status quo, because he thinks that the evaluation of the default expression "belongs in the body". You (Chris) argue in favour of your PEP, where the bytecode of the default expression is inlined into the function's body, because you insist that it belongs in the body. You justify that claim by making spurious arguments that it is "impossible" to do otherwise (your term, not mine). But it isn't impossible, and I shall justify that claim in another post. A bold claim for somebody who knows nothing about the C implementation :-) The bottom line is, you both are arguing at each other because you both want the default expression to be executed in the body of the function o_O
You're still being highly offensive here.
Everyone please chill a bit. Taking offence at vigourous but still respectful intellectual disagreement is not kind or welcoming and most of all it is not *helpful*. Brendan has not accused you of molesting children or betraying a position of trust. He hasn't even suggested you have bad breath. He has merely disagreed with your interpretation of a proposed programming language feature. Is it really worth taking offence over that? And if you think Brendan is not being *respectful* because his opinion is so obviously *wrong* (in your view), well, fine. Suppose you are right. Does it matter? Should you challenge him to pistols at dawn for besmirching your honour? Chris, I know this has been a long, hard PEP for you. Here is a feature that people in the Python community say they want, you write a PEP proposing it, and you get nothing but negativity. Ouch :-( Well, not nothing but negativity. There has been some support. And I didn't spend six hours researching and writing up evaluation strategies of nearly two dozen languages because I'm trying to sabotage your proposal. You and I may disagree with some details of this PEP, but I'm supporting it and want it to be the best late-bound feature it can be. -- Steve
On Mon, Dec 6, 2021 at 10:02 PM Steven D'Aprano <steve@pearwood.info> wrote:
You assert that it "belongs in the body", but only because Python currently doesn't allow it to be anywhere else. Other languages have this exact information in the function signature.
This argument about where the evaluation of the default expression is surreal.
Brendan argues in favour of the status quo, because he thinks that the evaluation of the default expression "belongs in the body".
You (Chris) argue in favour of your PEP, where the bytecode of the default expression is inlined into the function's body, because you insist that it belongs in the body. You justify that claim by making spurious arguments that it is "impossible" to do otherwise (your term, not mine). But it isn't impossible, and I shall justify that claim in another post. A bold claim for somebody who knows nothing about the C implementation :-)
I am arguing that it belongs in the *header*. I think I once mistyped that and said "body" by mistake, but I have never intentionally advocated for that (and you called me out on it promptly for a correction). Default expressions, whether evaluated at definition time or call time, belong in the header. With definition-time expressions, their code is inlined into the body of the surrounding context. With call-time expressions, their code is inlined into the body of the function. Neither of these is part of the signature, but that's because the signature doesn't itself have any code.
The bottom line is, you both are arguing at each other because you both want the default expression to be executed in the body of the function o_O
You're conflating implementation with syntax. I want it to be implemented in the body, because that's clean and efficient *for implementation*. I want it to be written in the signature, because that's where function defaults belong. You're asking for it to be written in the signature, same as I am, but then to have it executed... somewhere else. But still in basically the same place.
You're still being highly offensive here.
Everyone please chill a bit.
Taking offence at vigourous but still respectful intellectual disagreement is not kind or welcoming and most of all it is not *helpful*. Brendan has not accused you of molesting children or betraying a position of trust. He hasn't even suggested you have bad breath. He has merely disagreed with your interpretation of a proposed programming language feature. Is it really worth taking offence over that?
He apologized privately and we have settled that matter. Let's drop it.
Chris, I know this has been a long, hard PEP for you. Here is a feature that people in the Python community say they want, you write a PEP proposing it, and you get nothing but negativity. Ouch :-(
Well, not nothing but negativity. There has been some support. And I didn't spend six hours researching and writing up evaluation strategies of nearly two dozen languages because I'm trying to sabotage your proposal. You and I may disagree with some details of this PEP, but I'm supporting it and want it to be the best late-bound feature it can be.
Thank you, I do appreciate that. Still, it does get somewhat tiresome when there is a dogpile of negativity about the proposal, much of it unbacked by any sort of code, and just declaring that what I've said doesn't work or isn't ideal or should be left off because of a hypothetical future feature. Very tiresome. ChrisA
On Mon, Dec 06, 2021 at 10:16:06PM +1100, Chris Angelico wrote:
You (Chris) argue in favour of your PEP, where the bytecode of the default expression is inlined into the function's body, because you insist that it belongs in the body. You justify that claim by making spurious arguments that it is "impossible" to do otherwise (your term, not mine). But it isn't impossible, and I shall justify that claim in another post. A bold claim for somebody who knows nothing about the C implementation :-)
I am arguing that it belongs in the *header*.
The *source code* of the expression. But you are inlining its *bytecode* into the function body.
With definition-time expressions, their code is inlined into the body of the surrounding context. With call-time expressions, their code is inlined into the body of the function.
Right! That's what I said. -- Steve
Brendan Barnwell writes:
The ability to write something in the function signature that we can already write in the body, and that quite naturally belongs in the body, because it is executed when the function is called, not when it is defined.
I'm basically in sympathy with your conclusion, but I don't think it's useful to prejudice the argument by saying it *naturally belongs* in the body. Some languages quite naturally support thunks/blocks (Ruby) or even go so far as to equate code to data (Lisp), and execute that code in lieu of what appear to be variable references. But maybe it's *Pythonic* to necessarily place the source code in the body? I can't say that.
I *really* don't like the idea that some types of object will be executed instead of being used, just because they have a flag set.
From a syntactic point of view, that's how Ruby blocks work. Closer to home, that's how properties work. And in the end, *all* objects are accessed by executing code. This is a distinction without a difference, except in our heads. I wouldn't want to be asked to explain the dividing line between objects that were "just used" and objects that were "produced by code that was executed instead of being just used".
I *really* don't like the idea that some types of argument will be inlined into the function body instead of being stored as first-class values like other `__defaults__`, just because there happens to be this one extra character next to the equals sign in the function signature. That strikes me as the sort of thing that should be incredibly scary.
Properties have *no* visible syntax if they're imported from a module. Properties are extremely useful, and we all use them all the time without noticing or caring. I see no reason in principle why the same kind of feature wouldn't be useful and just as invisible and just as "natural" for local or global variables -- or callable parameters, as long as properly restricted. Chris's proposal is nothing if not restricted! :-) My issues with Chris's proposal are described elsewhere, but I don't really see a problem in principle.
On Sat, Dec 04, 2021 at 10:50:14PM +1100, Chris Angelico wrote:
syntactic sugar for this:
def f(b, x=lambda b: a+b): ...
except that the lambda has the LB flag set.
Okay. So the references to 'a' and 'b' here are one more level of function inside the actual function we're defining, which means you're paying the price of nonlocals just to be able to late-evaluate defaults. Not a deal-breaker, but that is a notable cost (every reference to them inside the function will be slower).
How much slower? By my tests: - access to globals is 25% more expensive than access to locals; - access to globals is 19% more expensive than nonlocals; - and nonlocals are 6% more expensive than locals. Or if you do the calculation the other way (the percentages don't match because the denominators are different): - locals are 20% faster than globals; - and 5% faster than nonlocals; - nonlocals are 16% faster than globals. Premature optimization is the root of all evil. We would be better off spending effort making nonlocals faster for everyone than throwing out desirable features and a cleaner design just to save 5% on a microbenchmark. [...]
What this does mean, though, is that there are "magic objects" that cannot be used like other objects.
NotImplemented says hello :-) You are correct that one cannot use a LB function as a standard, early bound default without triggering the "evaluate this at call time" behaviour. If we're happy with this behaviour, it would need to be documented for people to ignore *wink* There's precedence though. You cannot overload an operator method to return NotImplemented without triggering the special "your object doesn't support this operator" behaviour. And there are two obvious workarounds: 1. Just pass the LB function in as an explicit argument. The trigger only operates when looking up a default, not on every access to a function. 2. Or you can wrap the LB function you actually want to be the default in a late-bound expression that returns that function. And if you still think that we should care, we can come up with a more complex trigger condition: - the parameter was flagged as using a late-default; - AND the default is a LB function. Problem solved. Now you can use LB functions as early-bound defaults, and all it costs is to record and check a flag for each parameter. Is it worth it? Dunno. [...]
The default expression is just a function (with the new LB flag set). So we can inspect its name, its arguments, its cell variables, etc:
>>> default_expression.__closure__ (<cell at 0x7fc945de74f0: int object at 0x7fc94614c0f0>,)
We can do anything that we could do with any other other function object.
Yup. As long as it doesn't include any assignment expressions, or anything else that would behave differently.
I don't get what you mean here. Functions with the walrus operator are still just functions that we can introspect:
f = lambda a, b: (len(w:=str(a))+b)*w f('spam', 2) 'spamspamspamspamspamspam' f.__code__ <code object <lambda> at 0x7fc945e07c00, file "<stdin>", line 1>
What sort of "behave differently" do you think would prevent us from introspecting the function object? "Differently" from what?
Great. So now we have some magnificently magical behaviour in the language, which will have some nice sharp edge cases, but which nobody will ever notice. Totally. I'm sure.
NotImplemented. Document it and move on. There are two work-arounds for those who care. And if you still think it matters, you can record a flag for each parameter recording whether it actually used a late-bound default or not.
Plus, we pay a performance price in any function that makes use of argument references, not just for the late-bound default, but in the rest of the code.
Using a late-bound default doesn't turn every local variable in your function into a cell variable. For any function that does a meaningful amount of work, the cost of making one or two parameters into cell variables instead of local variables is negligible. At worst, if you do *no other work at all*, it's a cost of about 5% on two-fifths of bugger-all. But if your function does a lot of real work, the difference between using cell variables instead of locals is going to be insignificant compared to ~~the power of the Force~~ the rest of the work done in the function. And if you have some unbelievably critical function that you need to optimize up the wahzoo? def func(a, b=None): if b is None: # Look ma, no cell variables! b = expression Python trades off convenience for speed and safety all the time. This will just be another such example. You want the convenience of a late-bound default? Use this feature. You want it to be 3ns faster? Use the old "if arg is None" idiom. Or write your code in C, and make it 5000000000ns faster.
We also need to have these special functions that get stored as separate code objects.
That's not a cost, that's a feature. Seriously. We're doing that so that we can introspect them individually, not just as the source string, but as actual callable objects that can be: - introspected; - tested; - monkey-patched and modified in place (to the degree that any function can be modified, which is not a lot); - copied or replaced with a new function. Testing is probably the big one. Test frameworks will soon develop a way to let you write tests to confirm that your late bound defaults do what you expect them to do. That's trivial for `arg=[]` expressions, but for complex expressions in complex functions, being able to isolate them for testing is a big plus. -- Steve
On Sun, Dec 5, 2021 at 11:34 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Dec 04, 2021 at 10:50:14PM +1100, Chris Angelico wrote:
syntactic sugar for this:
def f(b, x=lambda b: a+b): ...
except that the lambda has the LB flag set.
Okay. So the references to 'a' and 'b' here are one more level of function inside the actual function we're defining, which means you're paying the price of nonlocals just to be able to late-evaluate defaults. Not a deal-breaker, but that is a notable cost (every reference to them inside the function will be slower).
How much slower? By my tests:
- access to globals is 25% more expensive than access to locals; - access to globals is 19% more expensive than nonlocals; - and nonlocals are 6% more expensive than locals.
Or if you do the calculation the other way (the percentages don't match because the denominators are different):
- locals are 20% faster than globals; - and 5% faster than nonlocals; - nonlocals are 16% faster than globals.
Premature optimization is the root of all evil.
We would be better off spending effort making nonlocals faster for everyone than throwing out desirable features and a cleaner design just to save 5% on a microbenchmark.
Fair, but the desirable feature can be achieved without this cost, and IMO your design isn't cleaner than the one I'm already using, and 5% is a lot for no benefit.
[...]
What this does mean, though, is that there are "magic objects" that cannot be used like other objects.
NotImplemented says hello :-)
Good point. Still, I don't think we want more magic like that.
And if you still think that we should care, we can come up with a more complex trigger condition:
- the parameter was flagged as using a late-default; - AND the default is a LB function.
Problem solved. Now you can use LB functions as early-bound defaults, and all it costs is to record and check a flag for each parameter. Is it worth it? Dunno.
Uhh.... so..... the parameter has to be flagged AND the value has to be flagged? My current proposal just flags the parameter. So I ask again: what are you gaining by this change? You've come right back to where you started, and added extra costs and requirements, all for.... what?
[...]
The default expression is just a function (with the new LB flag set). So we can inspect its name, its arguments, its cell variables, etc:
>>> default_expression.__closure__ (<cell at 0x7fc945de74f0: int object at 0x7fc94614c0f0>,)
We can do anything that we could do with any other other function object.
Yup. As long as it doesn't include any assignment expressions, or anything else that would behave differently.
I don't get what you mean here. Functions with the walrus operator are still just functions that we can introspect:
f = lambda a, b: (len(w:=str(a))+b)*w f('spam', 2) 'spamspamspamspamspamspam' f.__code__ <code object <lambda> at 0x7fc945e07c00, file "<stdin>", line 1>
What sort of "behave differently" do you think would prevent us from introspecting the function object? "Differently" from what?
Wrapping it in a function means the walrus would assign in that function's context, not the outer function. I think it'd be surprising if this works: def f(x=>(a:=1)+a): # default is 2 but this doesn't: def g(x=>(a:=1), y=>a): # default is UnboundLocalError It's not a showstopper, but it is most definitely surprising. The obvious solution is to say that, in this context, a is a nonlocal. But this raises a new problem: The function object, when created, MUST know its context. A code object says "this is a nonlocal", and a function object says "when I'm called, this is my context". Which means you can't have a function object that gets called externally, because it's the code, not the function, that is what you need here. And that means it's not directly executable, but it needs a context. So, once again, we come right back around to what I have already: code that you can't lift out and call externally. The difference is that, by your proposal, there's a lot more overhead, for the benefit of maybe under some very specific circumstances being able to synthesize the result.
We also need to have these special functions that get stored as separate code objects.
That's not a cost, that's a feature. Seriously. We're doing that so that we can introspect them individually, not just as the source string, but as actual callable objects that can be:
- introspected;
- tested;
- monkey-patched and modified in place (to the degree that any function can be modified, which is not a lot);
- copied or replaced with a new function.
Testing is probably the big one. Test frameworks will soon develop a way to let you write tests to confirm that your late bound defaults do what you expect them to do.
That's trivial for `arg=[]` expressions, but for complex expressions in complex functions, being able to isolate them for testing is a big plus.
I'm still not convinced that it's as useful as you say. Compare these append-and-return functions: def build1(value, lst=None): if lst is None: lst = [] lst.append(value) return lst _SENTINEL = object() def build2(value, lst=_SENTINEL): if lst is _SENTINEL: lst = [] lst.append(value) return lst def hidden_sentinel(): _SENTINEL = object() def build3(value, lst=_SENTINEL): if lst is _SENTINEL: lst = [] lst.append(value) return lst return build3 build3 = hidden_sentinel() def build4(value, *optional_lst): if len(optional_lst) > 1: raise TypeError("too many args") if not optional_lst: optional_lst = [[]] optional_lst[0].append(value) return optional_lst[0] def build5(value, lst=>[]): lst.append(value) return lst (Have I missed any other ways to do this?) In which of them can you introspect the []? Three of them have an externally-visible sentinel, but you can't usefully change it in any way. You can look at it, and you'll see None or "<object object at 0xYourCapitalCity>", but that doesn't tell you that you'll get a new empty list. In which of them do you have a thing you can call that will generate an equivalent empty list? In which can you monkey-patch or modify-in-place the []? Not a big deal though, you admit that there wouldn't be much monkey-patching in any form. But you get none whatsoever with other idioms. In which of these can you separately test the []? Do any current test frameworks have a way to let you write tests to make sure that these do what you expect? In which of these can you copy the [] into another context, or replace the [] with something else? Why are these such major problems for build5 if they're not for the other four? ChrisA
On Sun, Dec 05, 2021 at 01:19:08PM +1100, Chris Angelico wrote:
And if you still think that we should care, we can come up with a more complex trigger condition:
- the parameter was flagged as using a late-default; - AND the default is a LB function.
Problem solved. Now you can use LB functions as early-bound defaults, and all it costs is to record and check a flag for each parameter. Is it worth it? Dunno.
Uhh.... so..... the parameter has to be flagged AND the value has to be flagged? My current proposal just flags the parameter.
*shrug* Is that a problem? Okay, then we can just flag the parameter, and use a plain ol' function with no special co_flag, although we do surely still want to give it an attribute `__expression__` to hold the source expression.
So I ask again: what are you gaining by this change?
Separation of concerns. Testing. Cleaner design (the body of the function contains only the byte code from the body of the function). Testing. Introspection. Testing. You don't have to hack the byte-code of the function in order to monkey-patch defaults. Testing. In case it wasn't obvious, I think that testing is important :-) Other people might think of other uses. [...]
What sort of "behave differently" do you think would prevent us from introspecting the function object? "Differently" from what?
Wrapping it in a function means the walrus would assign in that function's context, not the outer function.
Unless the variable was a cell variable of the outer function. Which I think that they need to be. Didn't you already decide that walrus assignment has to be in the context of the function? That's not a rhetorical question, I genuinely don't remember.
The obvious solution is to say that, in this context, a is a nonlocal.
Right.
But this raises a new problem: The function object, when created, MUST know its context. A code object says "this is a nonlocal", and a function object says "when I'm called, this is my context". Which means you can't have a function object that gets called externally, because it's the code, not the function, that is what you need here. And that means it's not directly executable, but it needs a context.
Sorry Chris, I don't understand what you are trying to say here. If I take what you are saying literally, I would take you as trying to say that closures can never be executed. But they clearly can be, and I know that you know that. So obviously I am misunderstanding you. I don't understand why you think that we can't take one of those Late-Bound (LB) functions, which will be a closure, and call it like we can call any other closure. By the time we have access to the LB functions, the owner function will exist, so there shouldn't be any problem with the context not existing.
So, once again, we come right back around to what I have already: code that you can't lift out and call externally. The difference is that, by your proposal, there's a lot more overhead, for the benefit of maybe under some very specific circumstances being able to synthesize the result.
Surely it is the other way around? If you are correct, there are some extremely specific cicumstances involving the use of walrus operator in two different default expressions such that you cannot call one of the two functions, but the 99.99% of straight-forward non-walrus, non-tricky default expressions will work perfectly fine as independently callable functions.
I'm still not convinced that it's as useful as you say. Compare these append-and-return functions:
def build1(value, lst=None): if lst is None: lst = [] [...] In which of them can you introspect the []?
"Here are lots of ways to solve a problem that don't let you test or introspect part of your code. Therefore we shouldn't add a feature that will let us more easily test and introspect that part of the code." Yes Chris, I know that emulated late-bound defaults are currently effectively invisible at runtime (short of byte-code hacking). That weakness of the status quo is why you have written this PEP. That invisibility is not a feature we should duplicate in late-bound defaults, but an argument for introducing defaults that overcome that weakness. Right now, your PEP adds a new feature that gives us *zero* new functionality. There is nothing your PEP allows us to do that we cannot already do. It adds no new functionality. It's just syntactic sugar for "if arg is missing: return expression". You compile that code (or something very close to it) into the body of the function. The only wins are (1) when people stumble into the "mutable default" gotcha, it is easy to tell them how to fix it (but not much easier than the status quo) and (2) a small increase in clarity, since we will be able to write the default expression directly in the signature instead of hiding behind a sentinel. That's not nothing, but we've seen on this list a fair number of people who think that the increase in clarity is not enough to justify the feature. Do you think the Steering Council will consider those two small wins sufficient? But with a different design, we have not just the increase in clarity, but also open the door to a ton of new possibilities that come about thanks to the separation of those default expressions into their own objects. I don't know whether that will be enough to sway the Steering Council. They might hate my idea even more. But feedback on this list suggests that your design's lack of introspectability is putting people off the idea. It may be that this feature will never be used for anything more complex than `param=>[]`. In which case, okay, I concede, moving the default expression into its own code unit (a function? something else?) is overkill. But I don't think that people will only use this for such trivial expressions. Maybe 60% of the cases will be just a literal list display or dict display, and maybe another 30% will be a simple call to an existing function, like `time.ctime()`. But its the last 10% of non-trivial, interesting expressions where people will want to do more than just inspect the string. And having the expressions as first-class code units will open up those new opportunities that we currently lack. -- Steve
On Mon, Dec 6, 2021 at 2:56 AM Steven D'Aprano <steve@pearwood.info> wrote:
What sort of "behave differently" do you think would prevent us from introspecting the function object? "Differently" from what?
Wrapping it in a function means the walrus would assign in that function's context, not the outer function.
Unless the variable was a cell variable of the outer function. Which I think that they need to be. Didn't you already decide that walrus assignment has to be in the context of the function?
That's not a rhetorical question, I genuinely don't remember.
Yes, it does, and that won't be the case if you create an invisible nested function.
But this raises a new problem: The function object, when created, MUST know its context. A code object says "this is a nonlocal", and a function object says "when I'm called, this is my context". Which means you can't have a function object that gets called externally, because it's the code, not the function, that is what you need here. And that means it's not directly executable, but it needs a context.
Sorry Chris, I don't understand what you are trying to say here. If I take what you are saying literally, I would take you as trying to say that closures can never be executed. But they clearly can be, and I know that you know that. So obviously I am misunderstanding you.
Closures cannot be executed without a context. Consider: def f(x=lambda: (a:=[])): if isinstance(x, FunctionType): x = x() print(a) Here's the problem: The name 'a' should be in the context of f, but that context *does not exist* until f starts executing. As an early-bound default, like this, it's not going to work, because the function object has to be fully constructed at function definition time. If you try to do this sort of thing with a magical function that is synthesized as part of late-bound default handling, you'll still need to figure out when f's context gets formed. Normally, it won't get formed until the stack frame for f gets constructed - which is when arguments get assigned to parameters, etc, etc. Trying to call that inner function without first giving it a context won't work. And if all you have is a code object without a function, what's the use? How is it better than just having code in the function itself?
I don't understand why you think that we can't take one of those Late-Bound (LB) functions, which will be a closure, and call it like we can call any other closure.
By the time we have access to the LB functions, the owner function will exist, so there shouldn't be any problem with the context not existing.
The function will exist. The instance of it won't. Consider this basic demo of closures: def counter(): n = 1 def incr(): nonlocal n n += 1 return n return incr Prior to calling counter(), you can disassemble incr's bytecode and see what it does. But you can't call incr. Its context does not yet exist. Until you call counter, there is actually no function for incr, only the code object. And while it is possible to eval() a code object...
eval(counter.__code__.co_consts[2]) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: code object passed to eval() may not contain free variables
... you can't if it has nonlocals, because it needs a context. So if there's any possibility of nonlocals in latebound argument defaults, you won't be able to call this externally, which leaves me wondering: why all this hassle if it isn't even going to work?
So, once again, we come right back around to what I have already: code that you can't lift out and call externally. The difference is that, by your proposal, there's a lot more overhead, for the benefit of maybe under some very specific circumstances being able to synthesize the result.
Surely it is the other way around? If you are correct, there are some extremely specific cicumstances involving the use of walrus operator in two different default expressions such that you cannot call one of the two functions, but the 99.99% of straight-forward non-walrus, non-tricky default expressions will work perfectly fine as independently callable functions.
To be honest, 99.99% of late-bound defaults would probably work just fine if you simply eval their texts. But that doesn't justify having functions either.
I'm still not convinced that it's as useful as you say. Compare these append-and-return functions:
def build1(value, lst=None): if lst is None: lst = [] [...] In which of them can you introspect the []?
"Here are lots of ways to solve a problem that don't let you test or introspect part of your code. Therefore we shouldn't add a feature that will let us more easily test and introspect that part of the code."
Yes Chris, I know that emulated late-bound defaults are currently effectively invisible at runtime (short of byte-code hacking). That weakness of the status quo is why you have written this PEP.
That invisibility is not a feature we should duplicate in late-bound defaults, but an argument for introducing defaults that overcome that weakness.
Okay, but you're asking for a LOT of complexity for a very small part of the feature.
Right now, your PEP adds a new feature that gives us *zero* new functionality. There is nothing your PEP allows us to do that we cannot already do. It adds no new functionality. It's just syntactic sugar for "if arg is missing: return expression". You compile that code (or something very close to it) into the body of the function.
MOST new features give us zero new functionality. That's what it means for a language to be Turing-complete. But "if arg is missing:" doesn't currently exist, nor does "this argument is optional and has no default", so you can't quite write this currently, and that's why we need sentinels.
The only wins are (1) when people stumble into the "mutable default" gotcha, it is easy to tell them how to fix it (but not much easier than the status quo) and (2) a small increase in clarity, since we will be able to write the default expression directly in the signature instead of hiding behind a sentinel.
I consider those to be LARGE wins.
That's not nothing, but we've seen on this list a fair number of people who think that the increase in clarity is not enough to justify the feature. Do you think the Steering Council will consider those two small wins sufficient?
I'm hoping so. That's the main thrust of what I'm doing. Clarity, and an easier way to make mutable defaults working. And more intuitive behaviour in some situations. Clarity, mutable defaults, and intuitive behaviour. And nice red uniforms.
But with a different design, we have not just the increase in clarity, but also open the door to a ton of new possibilities that come about thanks to the separation of those default expressions into their own objects. I don't know whether that will be enough to sway the Steering Council. They might hate my idea even more. But feedback on this list suggests that your design's lack of introspectability is putting people off the idea.
Or it's suggesting that people on this list love to argue. I don't think we're surprised by that. :) (And I don't have a problem with it.)
It may be that this feature will never be used for anything more complex than `param=>[]`. In which case, okay, I concede, moving the default expression into its own code unit (a function? something else?) is overkill.
But I don't think that people will only use this for such trivial expressions. Maybe 60% of the cases will be just a literal list display or dict display, and maybe another 30% will be a simple call to an existing function, like `time.ctime()`.
But its the last 10% of non-trivial, interesting expressions where people will want to do more than just inspect the string. And having the expressions as first-class code units will open up those new opportunities that we currently lack.
If you want to write up a competing reference implementation, go ahead. I don't think it will be as easy as you claim. And I'm not going to mention the possibility in the PEP *without* a reference implementation, nor do I intend to write one myself (since I dispute its usefulness). Will the benefits outweigh the costs? I don't know. But I think not, and that the much simpler proposal will be better. ChrisA
On Mon, Dec 06, 2021 at 03:14:36AM +1100, Chris Angelico wrote:
Closures cannot be executed without a context. Consider:
def f(x=lambda: (a:=[])): if isinstance(x, FunctionType): x = x() print(a)
Here's the problem: The name 'a' should be in the context of f,
Not in the code as you show it. The a in the signature is local to the lambda, and the a in the print() call is a global. The code exactly as shown works fine, in exactly that way. So I'm going to assume that you meant: def f(x=>(a:=[])) which compiles to a separate `lambda:(a:=[])`, as per my suggestion.
but that context *does not exist* until f starts executing.
If this really cannot be solved, then we could prohibit walrus operators in late bound defaults. If something is too hard to get right, we can prohibit it until such time (if ever) that we work out how to do it. For many years, we had a restriction that you could have try...except and you could have try...finally but you couldn't have try...except... finally in a single block. (And then we solved that technical limitation.) Or we make them global, like they are for early bound defaults. Its not *mandatory* that walrus bindings in the default expression go into the function locals. They could go into the function's surrounding scope (usually globals). That's a perfectly acceptable solution too. Or... be bold. Think outside the box. "Do not go where the path may lead, go instead where there is no path and leave a trail." To boldly go where no science fiction show has gone before, and other cliches. If we can't execute the expression without the context existing, we make it exist. Details to follow in another post.
As an early-bound default, like this, it's not going to work, because the function object has to be fully constructed at function definition time.
a = "this is a global" f()
This paragraph confused me for the longest time, because as an early-bound default, it does work. If you put `return x` after the print, you get the expected result: this is a global []
If you try to do this sort of thing with a magical function that is synthesized as part of late-bound default handling, you'll still need to figure out when f's context gets formed. Normally, it won't get formed until the stack frame for f gets constructed - which is when arguments get assigned to parameters, etc, etc. Trying to call that inner function without first giving it a context won't work.
Right. So make the context. [...]
If you want to write up a competing reference implementation, go ahead. I don't think it will be as easy as you claim. And I'm not going to mention the possibility in the PEP *without* a reference implementation, nor do I intend to write one myself (since I dispute its usefulness).
It is the job of the PEP to describe rejected alternatives. If you choose to disregard my clearly superior suggestion *wink* but don't explain why you are rejecting it, then the PEP is not doing its job. -- Steve
On Tue, Dec 07, 2021 at 12:53:39AM +1100, Steven D'Aprano wrote:
If we can't execute the expression without the context existing, we make it exist. Details to follow in another post.
Here is some handwavy pseudo-code for setting up the context prior to calling a function "func": code = func.__code__ locals = [NIL pointer for _ in range(code.co_nlocals)] cells = [cell() for _ in range(len(code.co_cellvars))] assign args, kwargs, early defaults to locals # make a frame frame = Frame() frame.f_back = current frame frame.f_lasti = 0 frame.f_locals = locals + cells + list(func.__closure__) frame.f_code = code and then the interpreter does its funky thang with the frame and the function is executed. But here's the thing... there is nothing that says that the f_code has to have come from func.__code__. It just needs to be code that expects the same environment (locals, cell, etc) as the frame is set up for. So here's how we could run the late bound default expression alone: 1. Ensure that the default expression code object has the same environment (locals, cells etc) as the function that owns it; this could be a copy, or it could be an reference back to the owner. 2. Set up the frame, as above, for the function that owns the expression. 3. But instead of setting f_code to the owner's code object, we set it to the default expression's code object. They share the same environment (cells, etc) so that's safe. 4. And then just run the function. And to run the owner function: 1. Set up the frame. 2. Run each of the default expressions as above. 3. Then run the owner function. Obviously this is very handwavy. But I am confident that it demonstrates that the idea of factoring default expressions out into their own code objects is not "impossible". -- Steve
On Tue, Dec 7, 2021 at 1:28 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Tue, Dec 07, 2021 at 12:53:39AM +1100, Steven D'Aprano wrote:
If we can't execute the expression without the context existing, we make it exist. Details to follow in another post.
Here is some handwavy pseudo-code for setting up the context prior to calling a function "func":
code = func.__code__ locals = [NIL pointer for _ in range(code.co_nlocals)] cells = [cell() for _ in range(len(code.co_cellvars))] assign args, kwargs, early defaults to locals # make a frame frame = Frame() frame.f_back = current frame frame.f_lasti = 0 frame.f_locals = locals + cells + list(func.__closure__) frame.f_code = code
and then the interpreter does its funky thang with the frame and the function is executed.
But here's the thing... there is nothing that says that the f_code has to have come from func.__code__. It just needs to be code that expects the same environment (locals, cell, etc) as the frame is set up for.
Okay. So you want to break out a separate code object from the main body, but it has to expect the exact same environment as the main body. Which means you cannot replace it with anything else, you cannot call it externally without constructing the exact same environment, and frankly, I cannot for the life of me see what you gain above just having it part of the body. You keep arguing in favour of having it be somehow separate, but to what end? Please. Go make a reference implementation. At least then we'll know what's possible and what's not. ChrisA
On 2021-12-05 08:14, Chris Angelico wrote:
Closures cannot be executed without a context. Consider:
def f(x=lambda: (a:=[])): if isinstance(x, FunctionType): x = x() print(a)
Here's the problem: The name 'a' should be in the context of f, but that context*does not exist* until f starts executing.
Frankly, I would consider this another disadvantage of late-bound arguments as defined under your proposal. I do not want argument defaults to be able to have the side effect of creating additional local variables in the function. (There is also the question of whether they could assign in this manner to names already used by other arguments, so that one argument's default could potentially override the default of another.) -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On Tue, Dec 7, 2021 at 6:16 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2021-12-05 08:14, Chris Angelico wrote:
Closures cannot be executed without a context. Consider:
def f(x=lambda: (a:=[])): if isinstance(x, FunctionType): x = x() print(a)
Here's the problem: The name 'a' should be in the context of f, but that context*does not exist* until f starts executing.
Frankly, I would consider this another disadvantage of late-bound arguments as defined under your proposal. I do not want argument defaults to be able to have the side effect of creating additional local variables in the function. (There is also the question of whether they could assign in this manner to names already used by other arguments, so that one argument's default could potentially override the default of another.)
Unless it's otherwise denied, all valid forms of expression should be valid in an argument default. They are already valid in early-bound defaults:
def f(): ... def g(a=(b:=42)): ... ... return locals() ... f() {'b': 42, 'g': <function f.<locals>.g at 0x7f9a7083ef00>}
Since early-bound defaults are evaluated at definition time, the assignment happens in the outer function. If anything, it's more logical for it to be a part of the inner function, but for early-bound defaults, that's not possible. Though I wouldn't recommend that people actually *do* this sort of thing. Legal doesn't mean recommended. But if someone does, it needs to behave logically and correctly. ChrisA
On 2021-12-06 11:21, Chris Angelico wrote:
On Tue, Dec 7, 2021 at 6:16 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:
On 2021-12-05 08:14, Chris Angelico wrote:
Closures cannot be executed without a context. Consider:
def f(x=lambda: (a:=[])): if isinstance(x, FunctionType): x = x() print(a)
Here's the problem: The name 'a' should be in the context of f, but that context*does not exist* until f starts executing.
Frankly, I would consider this another disadvantage of late-bound arguments as defined under your proposal. I do not want argument defaults to be able to have the side effect of creating additional local variables in the function. (There is also the question of whether they could assign in this manner to names already used by other arguments, so that one argument's default could potentially override the default of another.)
Unless it's otherwise denied, all valid forms of expression should be valid in an argument default. They are already valid in early-bound defaults:
def f(): ... def g(a=(b:=42)): ... ... return locals() ... f() {'b': 42, 'g': <function f.<locals>.g at 0x7f9a7083ef00>}
That's true. I guess what I really mean is "I still think the walrus was a bad idea". :-) -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
On 4 Dec 2021, at 09:44, Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Dec 04, 2021 at 03:14:46PM +1100, Chris Angelico wrote:
Lots and lots and lots of potential problems. Consider:
def f(): a = 1 def f(b, x=>a+b): def g(): return x, a, b
Both a and b are closure variables - one because it comes from an outer scope, one because it's used in an inner scope. So to evaluate a+b, you have to look up an existing closure cell, AND construct a new closure cell.
The only way to do that is for the compiled code of a+b to exist entirely within the context of f's code object.
I dispute that is the only way. Let's do a thought experiment.
There are many possible implementation of the late bound idea that could create an object/default expression. But is it reasonable to bother with that added complexity/maintenance burden for a first implementation. And maybe no one will care enough to ever implement the ability to modify the code of a late bound variables expression as a separate object later. I think I understand the argument as being along the lines of for early bound defaults they can be inspected and modified. Therefore being able to do the same for late bound defaults must be implemented. I'm not convinced that that is reasonable to require is implemented. If python had always had late bound defaults, as it is with most languages in the survey posted earlier in this thread, would that have been implemented as an object/expression? Maybe, but I doubt it. Summary: I agree it's not impossible, I do not agree that it's needed. Barry
Barry Scott writes:
There are many possible implementation of the late bound idea that could create an object/default expression. But is it reasonable to bother with that added complexity/maintenance burden for a first implementation.
Yes. If you don't do it, you'll have backward compatibility issues or technical debt. I'm not saying that's a compelling argument here, except that one of the main alleged problems is that users don't understand mutable defaults. So adding more and more layers of support for default arguments is making matters worse, I suspect. (Remember, they're going to be reading "arg=None" and "@arg=[]" for a long long time.) This one is Worth Doing Right the first time, I think. And IMO David Mertz is right: doing it right means a more general deferred-evaluation object (not to be confused with Deferreds that need to be queried about their value).
And maybe no one will care enough to ever implement the ability to modify the code of a late bound variables expression as a separate object later.
Hear! Hear! That's exactly how I feel about *this* proposal! With all due respect to Chris and Steve who have done great work advocating, implementing, and clarifying the proposal, IAGNU (I am gonna not use). Too much muscle memory, and more important, existing code whose style I want to be consistent and don't wanna mess with because it works, around "arg=None".
On Sun, Dec 5, 2021 at 3:08 PM Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
Barry Scott writes:
There are many possible implementation of the late bound idea that could create an object/default expression. But is it reasonable to bother with that added complexity/maintenance burden for a first implementation.
Yes. If you don't do it, you'll have backward compatibility issues or technical debt.
I'm not saying that's a compelling argument here, except that one of the main alleged problems is that users don't understand mutable defaults. So adding more and more layers of support for default arguments is making matters worse, I suspect. (Remember, they're going to be reading "arg=None" and "@arg=[]" for a long long time.)
This one is Worth Doing Right the first time, I think. And IMO David Mertz is right: doing it right means a more general deferred-evaluation object (not to be confused with Deferreds that need to be queried about their value).
If you think that deferred evaluation objects are the right way to do it, then write up a proposal to compete with PEP 671. In my opinion, it is a completely independent idea, which has its own merit, and which is not a complete replacement for late-bound defaults; the two could coexist in Python simultaneously, or either one could be accepted without the other, or we could continue to have neither. Yes, there's some overlap in the problems they solve, just as there's overlap between PEP 671 and PEP 661 on named sentinels; but there's also overlap between plenty of other language features, and we don't deem try/finally or context managers to be useless because of the other. ChrisA
Chris Angelico writes:
On Sun, Dec 5, 2021 at 3:08 PM Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
This one is Worth Doing Right the first time, I think. And IMO David Mertz is right: doing it right means a more general deferred-evaluation object (not to be confused with Deferreds that need to be queried about their value).
If you think that deferred evaluation objects are the right way to do it, then write up a proposal to compete with PEP 671.
That's not your call, I'm afraid. "Keep the status quo" is always a viable option, regardless of other options. And other things equal, it's the preferred option.
In my opinion, it is a completely independent idea,
You're welcome to your opinion, of course. But if you want to claim that's a reason for implementing your proposal, you need to support it. You also need to explain why the additional potential complexity of a third kind of default argument (evaluated at definition, evaluated during call, evaluated when referenced) isn't a big problem.
which is not a complete replacement for late-bound defaults;
Why not? If we have such objects, we could simply specify that in the case where such an object is specified as a function parameter default, it is evaluated in the same environment as your late-bound defaults. So we can have your schsemantics if that's what we want. On the other hand, it might turn out that 90% of the time, it doesn't matter if it's evaluated as part of the calling process, 9% of the time the natural place to evaluate it is at the point of first use, and 1% of the time it should be evaluated before the function body proper is entered. In that case the late-bound default would be of some utility, but is it worth it? David's "infinitesimal utility" argument seems likely to apply. What else would such a deferred-evaluation object be unable to do that your late-bound default can do?
we could continue to have neither.
That's where I am. More than any of these issues, the lack of a well- defined, properly introspectable object bothers me. In 2021, we should provide that.
On Sat, Dec 04, 2021 at 06:11:08PM +0000, Barry Scott wrote:
There are many possible implementation of the late bound idea that could create an object/default expression. But is it reasonable to bother with that added complexity/maintenance burden for a first implementation.
I don't think we can conclude that factoring out the late-bound defaults into their own routines is "added complexity/maintenance burden". Usually, factoring stand-alone code into its own routines *reduces* the maintenance burden, it doesn't increase it. You can test chunks of code in isolation. You can refactor it. It is easier to analyse and understand. The ability to introspect code is not a "burden", it is a feature that reduces the difficulty of testing and maintaining code. That's why we have functions in the first place, instead of having one giant ball of mud. https://exceptionnotfound.net/big-ball-of-mud-the-daily-software-anti-patter... Chris wants to throw the late-bound defaults into the body of the function because that's what we are currently forced to do to emulate late-bound defaults. But we are designing the feature from scratch, and we shouldn't be bound by the limitations of the existing idiom. Encapsulation is good. Putting late-bound expressions in their own function so that they can be tested is a good thing, not a problem to be avoided. Most of us have, from time to time, already moved the late bound default into its own function just to make it easy to test in isolation: def func(arg=None): if arg is None: arg = _implementation() # and now handle arg With my strategy, we get that isolation for almost for free. There is no extra effort needed on the programmer's side, no new name or top-level function needed. Will people take advantage of it? Of course people won't bother if the default is `[]` but beyond a certain level of complexity, people will want to test the default expressions in isolation, to be sure that it does the right thing. -- Steve
On Mon, Dec 6, 2021 at 1:45 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Dec 04, 2021 at 06:11:08PM +0000, Barry Scott wrote:
There are many possible implementation of the late bound idea that could create an object/default expression. But is it reasonable to bother with that added complexity/maintenance burden for a first implementation.
Chris wants to throw the late-bound defaults into the body of the function because that's what we are currently forced to do to emulate late-bound defaults. But we are designing the feature from scratch, and we shouldn't be bound by the limitations of the existing idiom.
Not quite true. I want to have them syntactically as part of the body of the function, and semantically as part of the function call. As a function begins executing, a ton of stuff happens, including allocating arguments to parameters, and providing default values for optional parameters that weren't passed. All I want to change is the way that defaults can be provided. Tell me, what's the first bytecode instruction in the function g here? def f(x): print(x) return lambda: x In current CPython, it's not "LOAD_GLOBAL print". It's "MAKE_CELL x". Conceptually, that's part of the framework of setting up the function - setting up its local variables, providing a place for nonlocals. But it's a bytecode at the start of the function's code object. (I'm not sure when that became a bytecode instruction. It wasn't one in Python 2.7.) CPython bytecode is an implementation detail, and the fact that there's bytecode to do this or that is not part of the language definition.
Encapsulation is good. Putting late-bound expressions in their own function so that they can be tested is a good thing, not a problem to be avoided. Most of us have, from time to time, already moved the late bound default into its own function just to make it easy to test in isolation:
def func(arg=None): if arg is None: arg = _implementation() # and now handle arg
With my strategy, we get that isolation for almost for free. There is no extra effort needed on the programmer's side, no new name or top-level function needed.
And if you want an _implementation function for external testing, you're still welcome to do that. def func(arg=>_implementation()): ... No magic, just perfectly normal coding practices. If something's important enough to test separately, refactor it into a private function so you can call it. Why should this happen automatically for late-bound argument defaults? From what I can tell, the only syntactic constructs which form separate code objects are those which need them for namespacing purposes: class blocks, nested functions, genexps, comprehensions. Default argument values are not separate namespaces, so they shouldn't need dedicated stack frames, dedicated code objects, or anything like that.
Will people take advantage of it? Of course people won't bother if the default is `[]` but beyond a certain level of complexity, people will want to test the default expressions in isolation, to be sure that it does the right thing.
Beyond a certain level of complexity, the fact that argument defaults are in the signature will naturally pressure people to refactor them into their own functions. I don't think we'll see people putting multiline argument default expressions directly into the function header. ChrisA
On Mon, Dec 06, 2021 at 02:08:46AM +1100, Chris Angelico wrote:
I want to have them syntactically as part of the body of the function, and semantically as part of the function call.
Then you'll love the status quo, because that's exactly what we have now! *wink*
As a function begins executing, a ton of stuff happens, including allocating arguments to parameters, and providing default values for optional parameters that weren't passed.
And neither of those things are part of the function body. They are set up *before* the function's code object is called.
All I want to change is the way that defaults can be provided.
Right. And the community reaction on this mailing list is underwhelming. Apart from Abe, who is rather vigourly in favour of the PEP as it stands, I'm not really sure this PEP has much support as it stands now. And one of the most frequent issues raised is that **all** you want to do is change the way that the late defaults can be provided. You're the PEP author, but maybe you should consider listening to community feedback? Just sayin'.
Tell me, what's the first bytecode instruction in the function g here?
def f(x): print(x) return lambda: x
Why do you think that is an argument against my position? As you said:
CPython bytecode is an implementation detail, and the fact that there's bytecode to do this or that is not part of the language definition.
Bytecode can and does change. I'm talking about the language definition: late bound defaults should be compiled into their own publicly accessible callable functions. I don't care what bytecode is emitted to do that. Whatever it is, it will change in the future, without changing the Python semantics one whit.
And if you want an _implementation function for external testing, you're still welcome to do that.
Oh, thank goodness! I thought your PEP was going to ban the use of function calls as the default expression!!! *wink*
def func(arg=>_implementation()): ...
No magic, just perfectly normal coding practices.
You've just made a good argument against your own PEP. We don't need new syntax for late-bound defaults, just use the perfectly normal coding practices that have worked fine for 30 years.
If something's important enough to test separately, refactor it into a private function so you can call it. Why should this happen automatically for late-bound argument defaults?
Sure. And why should early-bound defaults be available in a publicly accessible dunder attribute, when they could be compiled directly into the function code object? Hell, the entire code object, and most of the function object, could be compiled into one great big opaque ball of mud. Aside from standard attributes inherited from object, all the function really needs is a name and a binary blob of everything else. But that's not the sort of language we have. -- Steve
On Mon, Dec 6, 2021 at 4:13 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Mon, Dec 06, 2021 at 02:08:46AM +1100, Chris Angelico wrote:
I want to have them syntactically as part of the body of the function, and semantically as part of the function call.
Then you'll love the status quo, because that's exactly what we have now! *wink*
Aaaaaand that's what I get for typing emails at 2AM. LOL. Syntactically as part of the signature of the function, is what I meant to say. Thanks for the correction.
As a function begins executing, a ton of stuff happens, including allocating arguments to parameters, and providing default values for optional parameters that weren't passed.
And neither of those things are part of the function body. They are set up *before* the function's code object is called.
Correct. Including handling default values.
All I want to change is the way that defaults can be provided.
Right. And the community reaction on this mailing list is underwhelming. Apart from Abe, who is rather vigourly in favour of the PEP as it stands, I'm not really sure this PEP has much support as it stands now.
And one of the most frequent issues raised is that **all** you want to do is change the way that the late defaults can be provided.
You're the PEP author, but maybe you should consider listening to community feedback? Just sayin'.
Believe you me, I am listening. And quite frankly, the tone of this list is sounding like "shut up, go away, don't do anything, because there are other proposals that nobody can be bothered writing up, but if they existed, they'd be way better than what you're doing". Not exactly encouraging, since nobody is willing to do the work of writing up proposals, but is plenty willing to criticize.
Tell me, what's the first bytecode instruction in the function g here?
def f(x): print(x) return lambda: x
Why do you think that is an argument against my position? As you said:
CPython bytecode is an implementation detail, and the fact that there's bytecode to do this or that is not part of the language definition.
Bytecode can and does change.
I'm talking about the language definition: late bound defaults should be compiled into their own publicly accessible callable functions. I don't care what bytecode is emitted to do that. Whatever it is, it will change in the future, without changing the Python semantics one whit.
Yes. So why is it a problem for the default's evaluation to be part of the function's bytecode? I don't understand why this is such a big hassle.
def func(arg=>_implementation()): ...
No magic, just perfectly normal coding practices.
You've just made a good argument against your own PEP. We don't need new syntax for late-bound defaults, just use the perfectly normal coding practices that have worked fine for 30 years.
Hmm.... you mean that "arg=>_implementation()" works currently? You're complaining that complicated argument defaults can't be separately tested, and that we need magic to do the refactoring for us and make them testable. I say that, if it needs to be separately tested, refactor it, and if it's simple enough to not be worth refactoring, it doesn't get separately tested. That's an argument against your "break it into a separate magical subfunction" proposal, but not against the PEP itself.
Sure. And why should early-bound defaults be available in a publicly accessible dunder attribute, when they could be compiled directly into the function code object?
Because they're part of the FUNCTION, not the CODE. There's a big difference. Consider: for i in range(10): def func(i=i): ... There is only one code object. There are multiple function objects, each with their own early-bound defaults. By definition, since the defaults are evaluated as part of the function definition, the code to evaluate that MUST be part of the surrounding context, not the function being defined. And it's not introspectable at the moment, only the resulting value is. You are asking for far more than the language currently offers in any related context.
Hell, the entire code object, and most of the function object, could be compiled into one great big opaque ball of mud. Aside from standard attributes inherited from object, all the function really needs is a name and a binary blob of everything else.
But that's not the sort of language we have.
Indeed. We have all kinds of useful introspection tools. And late-bound defaults already have quite a few introspection features, most notably a source-code-reconstruction of the expression. That's more than you get for early-bound defaults. (In discussing this, I have noted a hole in the specification: it would be nice to have a way to see, on the code object, which args have late-bound defaults. Consider that a specification bug to be fixed.) You're setting the bar for PEP 671 a long way higher than literally everything else in the language... and yet you refuse to code your own reference implementation for the better version that you propose. Any plans to put some code behind your words? ChrisA
On Sun, Dec 5, 2021, 12:33 PM Chris Angelico
And quite frankly, the tone of this list is sounding like "shut up, go away, don't do anything, because there are other proposals that nobody can be bothered writing up, but if they existed, they'd be way better than what you're doing". Not exactly encouraging, since nobody is willing to do the work of writing up proposals, but is plenty willing to criticize.
I'll write up my proposal: "Keep the status quo" All done. I admit I may be a overly vociferous in my opposition to this particular change. But I also think your tone has been rather consistently pugnacious, and a bit combative, in dismissing objections or disagreements. I know you genuinely wish to improve Python, and believe this PEP would do so. But I think you've become attached to your idea in a way that becomes non-constructive and unlikely to garner support. For example, your careful exclusion of alternative ideas from the PEP feels less than forthcoming. It almost feels like you are trying to hide the alternatives from the SC. Obviously, many of them read this list, and all of them will think of basically the same ideas others have suggested on their own anyway. I first discussed the idea of a "generalized deferred object/type" on this list at least two years ago, probably more than three (I haven't looked through archives lately to be sure the dates). The idea got some vague interest, but I was too lazy, or too busy, or whatever, to write an actual PEP or implementation. It's fine to criticize my inaction in advancing the more general idea. But the result of my failing isn't "therefore PEP 671 should be adopted" as you keep claiming. It's just that I haven't done the work to flesh out the encompassing idea that would cover late-binding as a minor aspect. As an analogy, PEP 275 was written in 2001 and rejected/neglected. PEP 3103 was rejected in 2006. The very simple case switch on literals was thought not to be broad enough to change Python syntax, despite being a construct in numerous other programming languages. Then in 2020, PEP 622 was written, widely discussed and refined, and adopted. PEP 622 does EVERYTHING that PEP 275 would have, 19 years earlier, and even with pretty much the same syntax. But it also does MUCH more as well, and hence was thought to be worth a syntax change. PEP 671 is very much the same. It does something worthwhile. But it does vastly less than needed to warrant new syntax and semantics. I hope it takes less than 19 years, but a generalized deferred construct is worth waiting for.
On Mon, Dec 6, 2021 at 5:38 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
On Sun, Dec 5, 2021, 12:33 PM Chris Angelico
And quite frankly, the tone of this list is sounding like "shut up, go away, don't do anything, because there are other proposals that nobody can be bothered writing up, but if they existed, they'd be way better than what you're doing". Not exactly encouraging, since nobody is willing to do the work of writing up proposals, but is plenty willing to criticize.
I'll write up my proposal:
"Keep the status quo"
All done.
PEP 671 is very much the same. It does something worthwhile. But it does vastly less than needed to warrant new syntax and semantics. I hope it takes less than 19 years, but a generalized deferred construct is worth waiting for.
You: "Keep the status quo, all done" Also you: "Let's wait for something better" That's what I take issue with. You are simultaneously telling me that this proposal is bad because there's another proposal that would be better, AND saying that you don't want to push for any other proposal. So you're welcome to keep waiting. Meanwhile, I'm going to try to actually accomplish something NOW, not wait for some hypothetical future. ChrisA
On Mon, Dec 6, 2021 at 5:51 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
On Sun, Dec 5, 2021, 1:48 PM Chris Angelico
You: "Keep the status quo, all done" Also you: "Let's wait for something better"
Now is better than never. Although never is often better than *right* now.
Neither of which says "the distant and uncertain future of an unwritten proposal is better than either now or never". If you JUST said "keep the status quo, all done", that is a reasonable and consistent position (which I disagree with, but I fully respect). But you then taint your claims with this statement that the reason for keeping the status quo is not that the status quo is better, but that there is a hypothetical idea that might be even better. That's not "now is better than never". That's not "never is better than right now" either. I think it's time for me to drop this entire topic for a while. This list is getting more and more noisy and I'm going to stop contributing to that. ChrisA
On Sun, Dec 5, 2021, 12:33 PM Chris Angelico
And quite frankly, the tone of this list is sounding like "shut up, go away, don't do anything, because there are other proposals that nobody can be bothered writing up, but if they existed, they'd be way better than what you're doing". Not exactly encouraging, since nobody is willing to do the work of writing up proposals, but is plenty willing to criticize.
I'll write up my proposal:
"Keep the status quo"
All done.
I admit I may be a overly vociferous in my opposition to this particular change. But I also think your tone has been rather consistently pugnacious, and a bit combative, in dismissing objections or disagreements.
I know you genuinely wish to *improve Python*, and believe this PEP *would* do so. But I think you've become attached to your idea in a way that becomes non-constructive and unlikely to garner support.
For example, your careful exclusion of alternative ideas from the PEP feels less than forthcoming. It almost feels like you are trying to hide the alternatives from the SC. Obviously, many of them read this list, and all of them will think of basically the same ideas others have suggested on their own anyway. I am sure Chris A can answer for himself, but IMO the above is, frankly, insulting. It almost feels like you have run out of arguments and are resorting to a personal attack.
I first discussed the idea of a "generalized deferred object/type" on this list at least two years ago, probably more than three (I haven't looked through archives lately to be sure the dates). The idea got some vague interest, but I was too lazy, or too busy, or whatever, to write an actual PEP or implementation.
It's fine to criticize my inaction in advancing the more general idea. But the result of my failing isn't "therefore PEP 671 should be adopted" as you keep claiming. It's just that I haven't done the work to flesh out the encompassing idea that would cover late-binding as a minor aspect. Nobody has attempted (or at least completed) a PEP, never mind an implementation, of a "generalized deferred object/type", in the last N years or decades. And no reason to suppose that anyone will in the next N years or decades. (I am sure it is very difficult.) And I think it is fair to say that opinion is mixed on the benefits of such a
On 05/12/2021 18:37, David Mertz, Ph.D. wrote: proposal. I think it is also fair to say that such a proposal need not be incompatible with PEP 671. Meanwhile we have a completed PEP and implementation (though obviously changes might still be made) that "*would*" "*improve Python*".
As an analogy, PEP 275 was written in 2001 and rejected/neglected. PEP 3103 was rejected in 2006. The very simple case switch on literals was thought not to be broad enough to change Python syntax, despite being a construct in numerous other programming languages.
Then in 2020, PEP 622 was written, widely discussed and refined, and adopted. PEP 622 does EVERYTHING that PEP 275 would have, 19 years earlier, and even with pretty much the same syntax. But it also does MUCH more as well, and hence was thought to be worth a syntax change.
That is a fair point which, I admit, I struggle somewhat to refute. Let me say: It would carry more weight if we had a time-line, even a vague one, for implementing a "generalized deferred object/type". Nobody seems to be able and prepared to do the work. Without that, Python could wait indefinitely for this (PEP 671's) improvement. In my very personal, subjective opinion, PEP 622 feels more like a luxury. Anything that can be done with it could be done with pre-existing syntax. Whereas late-bound defaults feels more like a ... hm, "necessity" is obviously the wrong word, I think I mean "basic feature". (And AFAIU can not really be implemented with existing syntax - well, Turing-complete and all that, but I hope you know what I mean.) As witness, 12 of the 16 languages that Steven d'Aprano surveyed provide it. YMMV. Best wishes Rob Cliffe
PEP 671 is very much the same. It does something worthwhile. But it does vastly less than needed to warrant new syntax and semantics. I hope it takes less than 19 years, but a generalized deferred construct is worth waiting for.
_______________________________________________ Python-ideas mailing list --python-ideas@python.org To unsubscribe send an email topython-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived athttps://mail.python.org/archives/list/python-ideas@python.org/message/JPCEPJ... Code of Conduct:http://python.org/psf/codeofconduct/
Rob Cliffe via Python-ideas writes:
Nobody has attempted (or at least completed) a PEP, never mind an implementation, of a "generalized deferred object/type", in the last N years or decades.
Haskell anything. Ruby blocks. Closer to home, properties and closures. So I don't think the object part is that hard, it's the syntax and semantics that's devilish. The evaluation syntax doesn't seem hard: you just reference the identifier the object is bound to. The creation syntax, though, probably needs to be concise, maybe even familiar, as Chris's proposal is. For the semantics, just in the case of defining an alternative to PEP 671, consider: if you just stuff a generalized deferred expression object into a formal argument's default, you still have to solve the problems that Chris's proposal does: where do you get the appropriate namespace to extract variable values from? and when do you do it? I do admire the elegance of Chris's solution to those questions, but if I were to fit a generalized deferred into that role, I might not adopt the "evaluate deferreds just before entering the function body proper" strategy. And then there's the question of whether referencing the deferred "collapses the waveform" into an object and binds it to the original identifier (as in Chris's proposal, but for more general contexts), or whether the deferred gets reevaluated each time in the current context (as Ruby blocks and Python properties usually do). Steve
On 06/12/2021 09:44, Stephen J. Turnbull wrote:
Rob Cliffe via Python-ideas writes:
Nobody has attempted (or at least completed) a PEP, never mind an implementation, of a "generalized deferred object/type", in the last N years or decades.
Haskell anything. Ruby blocks. Closer to home, properties and closures.
So I don't think the object part is that hard, it's the syntax and semantics that's devilish. The evaluation syntax doesn't seem hard: you just reference the identifier the object is bound to. The creation syntax, though, probably needs to be concise, maybe even familiar, as Chris's proposal is.
For the semantics, just in the case of defining an alternative to PEP 671, consider: if you just stuff a generalized deferred expression object into a formal argument's default, you still have to solve the problems that Chris's proposal does: where do you get the appropriate namespace to extract variable values from? and when do you do it? I do admire the elegance of Chris's solution to those questions, but if I were to fit a generalized deferred into that role, I might not adopt the "evaluate deferreds just before entering the function body proper" strategy.
And then there's the question of whether referencing the deferred "collapses the waveform" into an object and binds it to the original identifier (as in Chris's proposal, but for more general contexts), or whether the deferred gets reevaluated each time in the current context (as Ruby blocks and Python properties usually do).
Steve
I think you're making my point. You're saying that the object part isn't that hard, but other parts of it are. Which overall means that it is a hard problem. And there seems to be no likelihood of anyone tackling it soon. (If anyone is working on it now, or planning to soon, please correct me.) I can't see the point of rejecting something that provides a tangible benefit, now, because some fictitious vapourware *might*, one day, provide another way of doing it. Best wishes Rob Cliffe
Rob Cliffe via Python-ideas writes:
I think you're making my point.
*shrug* You wrote "object", I took you at your word.
You're saying that the object part isn't that hard, but other parts of it are.
For values of "hard" == "non-trivial but mostly bikeshedding". I don't think it will be that much harder to get through than Chris's. And in theory it could be easier: it could be implemented with a new builtin such as "quote_expression" that takes a string, and thus needing no new syntax. I actually don't think that will pass, too clumsy to be of practical use. Although it might fly on the same logic as function annotations did: we add the instance flag that says that "this object is a thunk to be evaluated in the current context" and the code in the interpreter that's needed to use it, and postpone syntax until somebody comes up with a really good proposal. In fact, we could combine this strategy with Steven d'Aprano's proposal for a thunk object, in which case Chris doesn't need syntax himself. I'm sure he won't like that, but it's an idea.
And there seems to be no likelihood of anyone tackling it soon.
There's no way to judge that. David Mertz might post a PEP tomorrow for all you know.
I can't see the point of rejecting something that provides a tangible benefit, now, because some fictitious vapourware *might*, one day, provide another way of doing it.
That's a strawman. The argument is not "Your proposal is good, but not perfect, so we reject it." The basic argument is "Your proposal isn't good enough to deserve syntax for reasons given elsewhere, but if you do this other stuff we would likely support it." Sure, there's some component of "it might interfere with the general facility, or be a small cognitive burden, a warty technical debt, if the general facility is implemented", but that's only a tiny part of why people opposing the proposal oppose it.
Rob Cliffe via Python-ideas writes:
I think you're making my point.
*shrug* You wrote "object", I took you at your word.
You're saying that the object part isn't that hard, but other parts of it are.
For values of "hard" == "non-trivial but mostly bikeshedding". I don't think it will be that much harder to get through than Chris's.
And in theory it could be easier: it could be implemented with a new builtin such as "quote_expression" that takes a string, and thus needing no new syntax. I actually don't think that will pass, too clumsy to be of practical use. Although it might fly on the same logic as function annotations did: we add the instance flag that says that "this object is a thunk to be evaluated in the current context" and the code in the interpreter that's needed to use it, and postpone syntax until somebody comes up with a really good proposal.
In fact, we could combine this strategy with Steven d'Aprano's proposal for a thunk object, in which case Chris doesn't need syntax himself. I'm sure he won't like that, but it's an idea. All this seems like a matter of opinion, on which we shall have to disagree. But ...
And there seems to be no likelihood of anyone tackling it soon.
There's no way to judge that. David Mertz might post a PEP tomorrow for all you know. Surely you're pulling my leg. If he had something in the pipeline that would greatly strengthen his argument, we would have heard about it by now.
I can't see the point of rejecting something that provides a tangible benefit, now, because some fictitious vapourware *might*, one day, provide another way of doing it.
That's a strawman. The argument is not "Your proposal is good, but not perfect, so we reject it." That IMO is exactly the argument. It's like saying "I won't buy a car today because in 10/20/50 years time I can probably buy a driverless one". The basic argument is "Your proposal isn't good enough to deserve syntax for reasons given elsewhere, Er, what reasons? but if_*you*_ do this other stuff we would likely support it." [My emphasis - RC.] You want *Chris* to implement deferred-evaluation objects? Hasn't he done enough? You want him to produce a second PEP and a second reference implementation in competition with his first one? (He has on several occasions mentioned problems that he sees with the idea.) Surely if other people think it's a good idea, it's up to *them* to
On 07/12/2021 18:22, Stephen J. Turnbull wrote: prove it. If an architect comes up with a plan to redesign a city centre, would you say to him "Now produce a completely different plan, just to show that your first one wasn't great"? Best wishes Rob Cliffe
Sure, there's some component of "it might interfere with the general facility, or be a small cognitive burden, a warty technical debt, if the general facility is implemented", but that's only a tiny part of why people opposing the proposal oppose it.
On Tue, Dec 07, 2021 at 06:42:18PM +0000, Rob Cliffe via Python-ideas wrote: [The other Stephen]
That's a strawman. The argument is not "Your proposal is good, but not perfect, so we reject it."
That IMO is exactly the argument. It's like saying "I won't buy a car today because in 10/20/50 years time I can probably buy a driverless one".
I think a better analogy for those who reject late-bound defaults but would accept a general deferred evaluation mechanism is: "I won't buy a driverless car today because I have a perfectly good car now, and **driving is not a sufficient burden** that I care for this feature alone. But fully autonomous robots that could drive my car, clean my house, do my chores, now *that's* something I would buy!" [the other Stephen]
if _*you*_ do this other stuff we would likely support it." [My emphasis - RC.]
You want *Chris* to implement deferred-evaluation objects?
Clearly not. It's obvious in context that Stephen is talking about *generic* "you". He's not addressing his comment to Chris. Anyone could read the comment and interpret that "you" as themselves, and respond "What a great idea! I'm going to implement deferred evaluation!". You surely don't imagine that Stephen thinks, or implies, that those who want a generic deferred evaluation feature would reject it if it wasn't done specifically by Chris himself. I know that the Python-Ideas community is changable like the wind and rather mercurial, but we've never yet demanded a feature we want be implemented by a *particular person* or else we will reject it. -- Steve
Rob Cliffe via Python-ideas writes:
On 07/12/2021 18:22, Stephen J. Turnbull wrote:
For values of "hard" == "non-trivial but mostly bikeshedding". I don't think it will be that much harder to get through than Chris's.
And in theory it could be easier: it could be implemented with a new builtin such as "quote_expression" that takes a string, and thus needing no new syntax. I actually don't think that will pass, too clumsy to be of practical use. Although it might fly on the same logic as function annotations did: we add the instance flag that says that "this object is a thunk to be evaluated in the current context" and the code in the interpreter that's needed to use it, and postpone syntax until somebody comes up with a really good proposal.
In fact, we could combine this strategy with Steven d'Aprano's proposal for a thunk object, in which case Chris doesn't need syntax himself. I'm sure he won't like that, but it's an idea.
All this seems like a matter of opinion, on which we shall have to disagree. But ...
"All"? The suggestions about implementation are eminently practical, all are in use in Python already, proposed in accepted PEPs, or proposed by others in this thread for implementing Chris's syntax. The only disagreeable opinions there are about what might be acceptable to other people, but I thought I was pretty conservative on those counts.
And there seems to be no likelihood of anyone tackling it soon.
There's no way to judge that. David Mertz might post a PEP tomorrow for all you know.
Surely you're pulling my leg.
Of course not. I have no reason to believe humor would sway you. > If he had something in the pipeline that would greatly strengthen > his argument, we would have heard about it by now. The point of my argument above is that all the parts are there, "all" we need is a good syntax and a coherent theory of scoping. Those are "flash of inspiration" kinds of thing. It's a good question whether Chris will beat the general deferred advocates to an acceptable syntax, since there are quite a few people who dislike his preferred syntax. I'd give the edge to Chris, since a fair number of people like his idea and I'm sure he'll accept any syntax the SC strongly prefers, and several have been proposed. But it's not a sure thing, since quite a few smart people are completely unexcited by Chris's proposal. The SC might very well go the same way. They might stomp on general deferreds, too -- won't know until we get there.
That's a strawman. The argument is not "Your proposal is good, but not perfect, so we reject it."
That IMO is exactly the argument.
So much for your opinion. I assure you, that is NOT my argument. So if it is anybody's argument, there's no "the" there. > It's like saying "I won't buy a car today because in 10/20/50 years > time I can probably buy a driverless one". I agree that's a good analogy to your mischaracterization of my argument (which mostly has been made by others; important points have been made by David Mertz, Brendan Barnwell, Steven d'Aprano, and likely several others). But it is a mischaracterization, so your analogy is pointless.
Er, what reasons [for opposing the PEP]?
Read the thread. There are a number of them there.
You want *Chris* to implement deferred-evaluation objects?
Sure, I'll take a pony if it's offered. But what matters here is that Chris wants to, if he wants my support and David's. I imagine he considers that a nice-to-have but he'll do without rather than implement the general deferred object. > Hasn't he done enough? No, not enough to get better than -1s from me, David, and some others. Whether he's channeling the SC, or we are, remains to be seen. > You want him to produce a second PEP and a second reference > implementation in competition with his first one? As above. It's been done before, for similar reasons (ie, to make the BDFL, PEP Delegate, or SC happy).
Surely if other people think it's a good idea, it's up to *them* to prove it.
I think you misunderstand how open source projects work. There's always a gatekeeping mechanism in projects that involve more than two or three developeres; in Python it's the SC. (Note: not me or David or Brendan, nor all of us together.) People will do what they consider unnecessary work to get through the gate. Who does the work and how much is done depends on how much each party wants the feature. Whether Chris should do it depends on how much Chris wants *some* kind of deferred evaluation for actual arguments, how much the SC prefers a general deferred to Chris's default- arguments-only deferred, and whether they think having both is a bad idea for some reason. But the "good idea" of general deferreds is only marginally relevant to our -1s. It's those -1s that constitute the main issue for Chris, since they're a noisy signal that the SC might think as we do. > If an architect comes up with a plan to redesign a city centre, > would you say to him "Now produce a completely different plan, just > to show that your first one wasn't great"? No, I wouldn't say that. But I will say you need to come up with analogies that capture your opponents' ideas, instead of strawman arguments. Cheers, Steve
On Thu, Dec 9, 2021 at 4:55 AM Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
But the "good idea" of general deferreds is only marginally relevant to our -1s. It's those -1s that constitute the main issue for Chris, since they're a noisy signal that the SC might think as we do.
Please explain to me *exactly* what your arguments against the current proposal are. At the moment, I am extremely confused as to what people actually object to, and there's endless mischaracterization and accusation happening. Can we actually figure out what people are really saying, and what the problems with this proposal are? NOT that there might potentially be some other proposal, but what the problems with this one are. Argue THIS proposal, not hypothetical other proposals. ChrisA
On Wed, 8 Dec 2021 at 18:09, Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Dec 9, 2021 at 4:55 AM Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:
But the "good idea" of general deferreds is only marginally relevant to our -1s. It's those -1s that constitute the main issue for Chris, since they're a noisy signal that the SC might think as we do.
Please explain to me *exactly* what your arguments against the current proposal are. At the moment, I am extremely confused as to what people actually object to, and there's endless mischaracterization and accusation happening.
Can we actually figure out what people are really saying, and what the problems with this proposal are?
NOT that there might potentially be some other proposal, but what the problems with this one are. Argue THIS proposal, not hypothetical other proposals.
Note that I'm not vehemently -1 on this PEP, but I am against it. So I'm not necessarily one of the people whose response you need and are asking for here, but my views are part of the opposition to the PEP. So here's my problems with this proposal: 1. The problem that the PEP solves simply isn't common enough, or difficult enough to work around, to justify new syntax, plus a second way of defining default values. 2. There's no syntax that has gained consensus, and the objections seem to indicate that there are some relatively fundamental differences of opinion involved. 3. There's no precedent for languages having *both* types of binding behaviour. Sure, late binding is more common, but everyone seems to pick one form and stick with it. 4. It's a solution to one problem in the general "deferred expression" space. If it gets implemented, and deferred expressions are added later, we'll end up with two ways of achieving one result, with one way being strictly better than the other. (Note, for clarity, that's *not* saying that we should wait for something that might never happen, it's saying that IMO the use case here isn't important enough to warrant rushing a partial solution). To be 100% explicit, none of the above are showstopper objections (some, like the choice of syntax, are pretty minor). I'm not arguing that they are. Rather, my problem with the PEP is that we have a number of individually small issues like this, which aren't balanced out by a sufficiently compelling benefit. The PEP isn't *bad*, it's simply not good *enough* (IMO). And it's not obvious how to fix the issue, as there's no clear way to increase the benefit side of the equation. That sucks, as it's a lot of work to write a PEP, and "meh, I'm not convinced" is the worst possible response. But that's how this feels to me. The reason deferred objects keep coming up is because they *do* have a much more compelling benefit - they help in a much broader range of cases. It's fine to say they are a different proposal, and that "but we might get deferred expressions" is a flawed objection (which it is, if that's all the objection consists of). But rejecting that argument doesn't do anything to improve the weak benefits case for late-bound defaults, or to fix the various minor problems that weigh it down. All IMO, of course... Paul
On 08/12/2021 19:27, Paul Moore wrote:
The reason deferred objects keep coming up is because they *do* have a much more compelling benefit - they help in a much broader range of cases.
That may be true. I don't know. Can anyone provide some realistic use cases? I've read the whole thread and I can only recall at most one, viz. the default value is expensive to compute and may not be needed. But that is a good time *not* to use a late-bound default! (The sentinel idiom would be better.) Anything can be used inappropriately, that doesn't make it bad per se. I don't wish to disparage anyone's motives. I am sure all the posts were made sincerely and honestly. But without examples (of how deferred objects would be useful), if *feels to me* (no doubt wrongly) as if people are using a fig leaf to fight against this PEP. Best wishes Rob Cliffe
On Wed, Dec 8, 2021, 2:58 PM Rob Cliffe via Python-ideas
On 08/12/2021 19:27, Paul Moore wrote:
The reason deferred objects keep coming up is because they *do* have a much more compelling benefit - they help in a much broader range of cases.
Can anyone provide some realistic use cases? I've read the whole thread
and I can only recall at most one, viz. the default value is expensive to compute and may not be needed.
https://docs.dask.org/en/stable/delayed.html This library is widely used, extremely powerful, and expressive. Basically, a built-in capability would have every win of Dask Delayed, but decrease the impedance mismatch (which isn't terrible as-is) and avoid the need for external tooling.
On Thu, Dec 9, 2021 at 9:44 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
On Wed, Dec 8, 2021, 2:58 PM Rob Cliffe via Python-ideas
On 08/12/2021 19:27, Paul Moore wrote:
The reason deferred objects keep coming up is because they *do* have a much more compelling benefit - they help in a much broader range of cases.
Can anyone provide some realistic use cases? I've read the whole thread and I can only recall at most one, viz. the default value is expensive to compute and may not be needed.
https://docs.dask.org/en/stable/delayed.html
This library is widely used, extremely powerful, and expressive. Basically, a built-in capability would have every win of Dask Delayed, but decrease the impedance mismatch (which isn't terrible as-is) and avoid the need for external tooling.
Most of that is a *massive* YAGNI as regards function default arguments. We do not need parallel execution just to figure out the length of a list passed as a previous parameter. So you've just added weight to my argument that a generic "delayed" feature is a completely separate proposal, nothing whatsoever to do with PEP 671. ChrisA
On Wed, 8 Dec 2021 at 22:53, Chris Angelico <rosuav@gmail.com> wrote:
Most of that is a *massive* YAGNI as regards function default arguments. We do not need parallel execution just to figure out the length of a list passed as a previous parameter. So you've just added weight to my argument that a generic "delayed" feature is a completely separate proposal, nothing whatsoever to do with PEP 671.
If we concede that delayed expressions are a separate proposal, would you be willing to address the other issues that people have raised? At this point, it seems like the "deferred expressions" debate is distracting everyone from all of the *other* points made by people with reservations about the proposal, which basically come down to "the benefit is limited, and the costs are too high to justify the feature". So far, the responses I've seen to that point mostly seem to come down to "I don't agree, I think the costs are small and the benefits are sufficient". That's not addressing the objections, it's just agreeing to differ¹. At a minimum, the PEP should state the objections fairly, and note that the PEP author disagrees. A PEP isn't a sales pitch, it's a summary of the discussions - so it absolutely should mention that there's been significant opposition to the proposal, which did not get resolved, if that's the reality. Paul ¹ "That's not an argument, it's just contradiction!"
On Thu, Dec 9, 2021 at 10:07 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Wed, 8 Dec 2021 at 22:53, Chris Angelico <rosuav@gmail.com> wrote:
Most of that is a *massive* YAGNI as regards function default arguments. We do not need parallel execution just to figure out the length of a list passed as a previous parameter. So you've just added weight to my argument that a generic "delayed" feature is a completely separate proposal, nothing whatsoever to do with PEP 671.
If we concede that delayed expressions are a separate proposal, would you be willing to address the other issues that people have raised? At this point, it seems like the "deferred expressions" debate is distracting everyone from all of the *other* points made by people with reservations about the proposal, which basically come down to "the benefit is limited, and the costs are too high to justify the feature". So far, the responses I've seen to that point mostly seem to come down to "I don't agree, I think the costs are small and the benefits are sufficient". That's not addressing the objections, it's just agreeing to differ¹.
Part of the problem is that it is really REALLY hard to figure out what the actual objections are. I asked, and the one clear answer I got was one subjective opinion that the cognitive load exceeded the benefit. Great! That's one person's concern. I've responded to that by clarifying parts of the cognitive load problem, and that's about as far as that can go. But if there's nothing more specific than that, what do you want me to respond to? How can I address the objections if the objections are as vague as you're describing?
At a minimum, the PEP should state the objections fairly, and note that the PEP author disagrees. A PEP isn't a sales pitch, it's a summary of the discussions - so it absolutely should mention that there's been significant opposition to the proposal, which did not get resolved, if that's the reality.
Yes, and "significant opposition" doesn't just mean "I don't like this". There's nothing to respond to in that. (Plus, there's significant belligerent support for the proposal, which is even harder to handle.) ChrisA
On Wed, 8 Dec 2021 at 23:18, Chris Angelico <rosuav@gmail.com> wrote:
Part of the problem is that it is really REALLY hard to figure out what the actual objections are. I asked, and the one clear answer I got was one subjective opinion that the cognitive load exceeded the benefit. Great! That's one person's concern. I've responded to that by clarifying parts of the cognitive load problem, and that's about as far as that can go.
Um, what parts of my response were unclear? I gave 4 specific points, Brendan gave 4 more (there wasn't much overlap with mine, either). Multiple people have mentioned that the proposed syntax is confusing. You don't have to respond to everyone individually, and indeed you shouldn't - it's the cumulative effect that matters. Telling 10 people that their concern "is one person's concern" doesn't address the fact that 10 people felt similarly. And honestly, there's only about 10 active participants in this thread, so even 5 people with reservations about the syntax is still "half the people who expressed an opinion". Yes, many of the concerns are somewhat subjective, and many of them are subject to a certain amount of interpretation. That's the nature of this sort of issue. If I said to you that the biggest issue here was that "in the group of people on python-ideas who were motivated enough to get involved in discussions, about half of the participants were arguing against the proposal"¹ would that be a concrete enough objection for you? Count it as my 5th objection, if you like. I know we're not putting the PEP to a vote here, but proposed changes *are* supposed to achieve a certain level of consensus (in the normal course of events - of course the SC can approve anything, even if it's hugely unpopular, that's their privilege). Paul ¹ That's just my gut feeling. Feel free to check the actual numbers and counter my argument with more precise facts.
On Thu, Dec 9, 2021 at 10:35 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Wed, 8 Dec 2021 at 23:18, Chris Angelico <rosuav@gmail.com> wrote:
Part of the problem is that it is really REALLY hard to figure out what the actual objections are. I asked, and the one clear answer I got was one subjective opinion that the cognitive load exceeded the benefit. Great! That's one person's concern. I've responded to that by clarifying parts of the cognitive load problem, and that's about as far as that can go.
Um, what parts of my response were unclear? I gave 4 specific points, Brendan gave 4 more (there wasn't much overlap with mine, either).
Multiple people have mentioned that the proposed syntax is confusing. You don't have to respond to everyone individually, and indeed you shouldn't - it's the cumulative effect that matters. Telling 10 people that their concern "is one person's concern" doesn't address the fact that 10 people felt similarly. And honestly, there's only about 10 active participants in this thread, so even 5 people with reservations about the syntax is still "half the people who expressed an opinion".
I have attempted to explain the syntax. What is confusing? def f(x=spam): ... def f(x=>spam): ... I'm not sure what concerns need to be addressed, because I don't understand the concerns. Maybe I'm just getting caught up on all the side threads about "deferreds are better" and "it should be a magical function instead" and I've lost some of the basics? Feel free to repost a simple concern and I will attempt to respond.
Yes, many of the concerns are somewhat subjective, and many of them are subject to a certain amount of interpretation. That's the nature of this sort of issue. If I said to you that the biggest issue here was that "in the group of people on python-ideas who were motivated enough to get involved in discussions, about half of the participants were arguing against the proposal"¹ would that be a concrete enough objection for you? Count it as my 5th objection, if you like. I know we're not putting the PEP to a vote here, but proposed changes *are* supposed to achieve a certain level of consensus (in the normal course of events - of course the SC can approve anything, even if it's hugely unpopular, that's their privilege).
EVERYONE is arguing against the proposal. Quite frankly, I'm just about ready to throw the whole thing in, because this entire thread has devolved to complaints that are nearly impossible to respond to - or are just repetition of things that ARE responded to in the PEP. Maybe we don't need any new features in Python. Maybe Python 3.10 is already the perfect language, and we should just preserve it in amber. ChrisA
On 2021-12-08 23:39, Chris Angelico wrote:
On Thu, Dec 9, 2021 at 10:35 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Wed, 8 Dec 2021 at 23:18, Chris Angelico <rosuav@gmail.com> wrote:
Part of the problem is that it is really REALLY hard to figure out what the actual objections are. I asked, and the one clear answer I got was one subjective opinion that the cognitive load exceeded the benefit. Great! That's one person's concern. I've responded to that by clarifying parts of the cognitive load problem, and that's about as far as that can go.
Um, what parts of my response were unclear? I gave 4 specific points, Brendan gave 4 more (there wasn't much overlap with mine, either).
Multiple people have mentioned that the proposed syntax is confusing. You don't have to respond to everyone individually, and indeed you shouldn't - it's the cumulative effect that matters. Telling 10 people that their concern "is one person's concern" doesn't address the fact that 10 people felt similarly. And honestly, there's only about 10 active participants in this thread, so even 5 people with reservations about the syntax is still "half the people who expressed an opinion".
I have attempted to explain the syntax. What is confusing?
def f(x=spam): ...
def f(x=>spam): ...
I'm not sure what concerns need to be addressed, because I don't understand the concerns. Maybe I'm just getting caught up on all the side threads about "deferreds are better" and "it should be a magical function instead" and I've lost some of the basics? Feel free to repost a simple concern and I will attempt to respond.
[snip] I haven't been following the thread for some time, but my expectation would be that: def f(x=>spam): ... would behave like: _Missing_ = object() def f(x=_Missing_): if x is _Missing_: x = spam ...
On Thu, Dec 9, 2021 at 12:41 PM MRAB <python@mrabarnett.plus.com> wrote:
On 2021-12-08 23:39, Chris Angelico wrote:
On Thu, Dec 9, 2021 at 10:35 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Wed, 8 Dec 2021 at 23:18, Chris Angelico <rosuav@gmail.com> wrote:
Part of the problem is that it is really REALLY hard to figure out what the actual objections are. I asked, and the one clear answer I got was one subjective opinion that the cognitive load exceeded the benefit. Great! That's one person's concern. I've responded to that by clarifying parts of the cognitive load problem, and that's about as far as that can go.
Um, what parts of my response were unclear? I gave 4 specific points, Brendan gave 4 more (there wasn't much overlap with mine, either).
Multiple people have mentioned that the proposed syntax is confusing. You don't have to respond to everyone individually, and indeed you shouldn't - it's the cumulative effect that matters. Telling 10 people that their concern "is one person's concern" doesn't address the fact that 10 people felt similarly. And honestly, there's only about 10 active participants in this thread, so even 5 people with reservations about the syntax is still "half the people who expressed an opinion".
I have attempted to explain the syntax. What is confusing?
def f(x=spam): ...
def f(x=>spam): ...
I'm not sure what concerns need to be addressed, because I don't understand the concerns. Maybe I'm just getting caught up on all the side threads about "deferreds are better" and "it should be a magical function instead" and I've lost some of the basics? Feel free to repost a simple concern and I will attempt to respond.
[snip]
I haven't been following the thread for some time, but my expectation would be that:
def f(x=>spam): ...
would behave like:
_Missing_ = object()
def f(x=_Missing_): if x is _Missing_: x = spam ...
Yes, broadly so. The differences would be that the signature actually says what the default will be (instead of "<object object at 0xdeadbeef>"), which in turn also means that you can type-check it reliably (for instance, if 'spam' is '[]', then you can show that this argument should always be a list, without having to say "or this specific object"), and there's no need to pollute an outer context with a sentinel value. I prefer to describe it more like: if x was not specified: x = spam even though "was not specified" isn't actually valid Python code. But the nearest equivalent in current code would be a sentinel like that. ChrisA
= (which would mean something syntactical in many cases, just not what is intended)." David Mertz: prefers a keyword like `defer` Ethan Furman: "Yes" Barry Scott: "Yes" Brendan Barnwell: "Yes, it is yet another reason not to do this." Nicholas Cole: "I would actively avoid using this feature and discourage people from using it because: I think that this imposes a significant cognitive burden, not for the simple cases, but when combined with the more advanced function definition syntax. I think
_*Objections to PEP 671 - Summary*_ There seems to be a problem understanding what the objections to PEP 671 are. Chris A wrote: "Part of the problem is that it is really REALLY hard to figure out what the actual objections are. I asked, and the one clear answer I got was one subjective opinion that the cognitive load exceeded the benefit. Great! That's one person's concern. I've responded to that by clarifying parts of the cognitive load problem, and that's about as far as that can go. But if there's nothing more specific than that, what do you want me to respond to? How can I address the objections if the objections are as vague as you're describing?" Well, I have AFAIK re-read every post in the threads and am attempting to summarise the objections to PEP 671 that I have found. Disclaimer: I do NOT claim this is a completely objective survey. I think by now everyone knows on which side of the fence I sit. In some places I have added my own (biased) comments. Nonetheless I have honestly tried to make a fair selection from all the relevant, reasonably important posts. And if anyone thinks that their, or anyone else's objection(s) have been omitted/understated/misrepresented, they are welcome to edit this post and put the result in a new one. Disclaimer: I have not referred to Steven d'Aprano's posts, because I lack the time (ability?) to understand them all. AFAICT he is in favour of something like this PEP, but a bit different. AFAIK these were all the objections that were raised. There may be some overlap or some sub-cases but this is how I've classified them: (A) Don't like the proposed `=>` syntax. (B) Too few use cases to bother making a change to the language. Status quo wins. (C) Some form of "deferred evaluation object" would do everything this PEP does and more, and might be incompatible with it. (D) The (late-binding) default value should be a first-class object that can be accessed and manipulated. (E) Calculation of a "default value" should be done in the function body, not the function header. (F) Concerns that functions using late-bound defaults were harder to wrap. (G) Backward compatibility. (H) No other language supports both early- and late-binding. Paul Moore and Brendan Barnwell were good enough to list their objections when asked. *Paul Moore*: 1. [OBJECTION B] "The problem that the PEP solves simply isn't common enough, or difficult enough to work around, to justify new syntax, plus a second way of defining default values." 2. [OBJECTION A] "There's no syntax that has gained consensus, and the objections seem to indicate that there are some relatively fundamental differences of opinion involved." 3. [OBJECTION H] "There's no precedent for languages having *both* types of binding behaviour. Sure, late binding is more common, but everyone seems to pick one form and stick with it." 4. [OBJECTION C] [paraphrased] deferred expressions are better *Brendan Barnwell*: 1. [OBJECTION B] "The status quo is fine. Using None or another sentinel and checking for it in the body has worked for many years and is not that big a problem. In theory improvement is always possible, but there is no urgency to change anything until we have a proposal with fewer downsides. In addition, as discussed in some posts on this list, not even all cases of None/sentinel defaults will be obviated by this proposal." 2. [OBJECTION A] "Most of the proposed syntaxes make it difficult to visually distinguish the late and early-bound defaults (because they all look similar to a plain equals sign which will still mean a regular early-bound default)." 3. [OBJECTION B AGAIN] "Regardless of the syntax, having the potential for def-time and call-time behavior to be mixed and interleaved in arbitrary ways within the same function signature is confusing." 4. [OBJECTION D] 'Currently anything that is a function default is some kind of Python object that can be inspected, interacted with, and used independently of the function/argument whose default it is. This proposal breaks that assumption. In other words I don't want anything that is "a default" but is not a "default VALUE".' 5. [OBJECTION B AGAIN] "Miscellaneous wrinkles. By this I mean the various sub-discussions about things like what order the late and early defaults should be evaluated in. This is a sort of second-order objection for me, because the objections I gave in my previous message are enough for me to reject the proposal. But even assuming I agreed with the broad outlines, these subsidiary concerns leave enough room for confusion that I would not endorse the proposal. In other words there are too many devils in the details that I feel would lead to difficult-to-reason-about code and traps for the unwary." I now quote the other (most relevant) posts re each of the objections: _*(A) Don't like the proposed `=>` syntax.*__* *_ Chris A asked if the cognitive burden of distinguishing between `=` and '=>' was too great (leading with his chin, but hey). Neil Giradhar: "Yes" André Roberge: "Yes, prefer a keyword" JL: "Yes. Something more distinctive feels better" Barry Scott: [paraphrased] Wouldn't use with => syntax, but would with @name syntax David Mertz: "YES! A few weeks later than the prior long discussion that I read in full, it took a triple take not to read it as this has the potential to make debugging large code-bases much harder." I quote from one of Chris A's replies re a `defer` keyword: 'The trouble is that this actually would be incompatible. If you can defer an expensive calculation and have some "placeholder" value in the variable 'result', then logically, you should be able to have that placeholder as a function default argument, which would be an early-bound default of the placeholder. That is quite different in behaviour from a late-bound default, so if I were to use the word "defer" for late-bound defaults, it would actually prevent the more general proposal.' _*(B) Too few use cases to bother making a change to the language. Status quo wins.*_ potatochowder.com: "Don't find compelling." David Mertz: "In particular, the cases where None will never, in any conceivable circumstances, be a non-sentinel value are at least 98% of all the functions (that have named parameters) I've ever written in Python." Stephen J. Turnbull: "Want to stick with my None sentinel and keep my code consistent." Paul Moore: "I can't think of a realistic case where I'd want to actually use the new feature." _*(C) Some form of "deferred evaluation object" would do everything this PEP does and more, and might be incompatible with it.*_ I think that this has been thrashed out sufficiently that anything I might say here could only stir up more pointless argument (which in fact is still ongoing as I write). _*(D) The (late-binding) default value should be a first-class object that can be accessed and manipulated.*_ Eric V.Smith: [AFAIU, paraphrased] I want the default value to be an object that I can inspect and change. David Mertz: "For the reasons Eric Smith and others have pointed out, I really WANT to keep inspectability of function signatures." Stephen J. Turnbull: "More than any of these issues, the lack of a well-defined, properly introspectable object bothers me." Chris A gave a reply including 'If you consider the equivalent to be a line of code in the function body, then the signature has become MASSIVELY more useful. Instead of simply seeing "x=<object object at 0x7fba1b318690>", you can see "x=>[]" and be able to see what the value would be.' _*(E) Calculation of a "default value" should be done in the function body, not the function header.*__* *_ potatochowder.com: [paraphrased] Computation of default belongs in the function body. Brendan Barnwell: "Just think that computation (as of defaults) should be in the function body, not the signature." I don't find this a compelling argument. Why (other than "Because")? What works, works. Rob Cliffe _*(F) Concerns that functions using late-bound defaults were harder to wrap.*__* *_ Paul Moore raised some concerns. AFAIU Chris A and Steven d'A answered them. I'm not qualified to comment. _*(G) Backward compatibility.*_ Inada Naoki: [paraphrased] Code that explicitly passes None or other sentinel to a function means that we can't change the signature of that function to use a late-bound default. My take: We can use late-bound defaults when we write new functions, but must be very careful changing the signature of existing functions. Rob Cliffe _*OTHER *_On 03/12/2021 Eric V. Smith raised some objections which I don't understand but quote here: "Your version is less friendly to type checking. And it doesn't work with positional-only arguments. How is the sentinel value private information or an implementation detail? It's part of the API. It should be clearly documented. If nothing else, it's can be inspected and discovered." "It is none the less true that default late-bound values cannot be modified. Correct? Early-bound ones can." "This is a tautology. You can't do these things if 671 is accepted because they will defined as not doable by 671. That's a strike against it. My stance is that it should be possible, and a proposal that makes them not possible with late-bound arguments is deficient." "I think it's critical that Signature objects be usable with all types of defaults. And having the default value available as a string isn't very useful, except for displaying help text. It wouldn't be possible to create a new Signature object from an existing Signature object that contains a late-bound argument (say, to create a new Signature with an additional argument). At least I haven't seen how it would be possible, since the PEP makes no mention of Signature objects. Which it definitely should, even if only to say "late-bound arguments are not designed to work with Signature objects". Sentinels are not meaningless. I think you're alienating people every time you suggest they are." to which Chris A replied on the same day "Positional-only args can be done with a list instead of a dict. Not much harder, although less clear what's happening if you have multiple." "Depends on the sentinel. In this example, is the exact value of the sentinel part of the API? _SENTINEL = object() def frobnicate(stuff, extra=_SENTINEL): ... If the sentinel is None, then it may be part of the API. (Of course, if it's a deliberately-chosen string or something, then that's completely different, and then you're not really doing the "optional argument" thing that I'm talking about here, and there's a very real default value.) But when it's an arbitrary sentinel like this, I argue that the precise value is NOT part of the function's API, only that you can pass any object, or not pass an object at all." 'Yes, but I'm asking you to compare late-bound defaults with the "sentinel and replace it in the function" idiom, which is a closer parallel. Can you, externally to the function, change this to use a new empty set instead of a new empty list? No.' "My point is that you *already* cannot do them. My proposal doesn't stop you from doing things you currently can do, it just makes it easier to spell the parts you can do." 'Please revisit your concerns with regard to the "sentinel and replace in the function" idiom, as I've stated in the past few posts. The sentinel itself can be replaced, but that is useless if the function's body is still looking for the old sentinel.' 'They most certainly ARE usable with all types of defaults, but instead of a meaningless "=<object object at 0xasdfqwer>", you get "=[]".' 'Some of them are. Some of them are not. When they are meaningful, PEP 671 does not apply, because they're not the sort of sentinel I'm talking about. Unfortunately, the word "sentinel" means many different things. There are many cases where a sentinel is a truly meaningful and useful value, and in those cases, don't change anything. There are other cases where the sentinel is a technical workaround for the fact that Python currently cannot represent defaults that aren't constant values, and those are meaningless sentinels that can be removed.' ------------------------ Obviously this exchange is of limited use without context, please see the 03/12/2021 actual posts. _*General feedback*_ Neil Giradhar wrote "I think this question is quite the biased sample on python-ideas. Please consider asking this to less advanced python users" Well, Neil, be careful what you wish for. There was a positive response to this PEP from Abdulla Al Kathiri, Jeremiah Vivian, Matt del Valle, Piotr Duda, Abe Dillon, MarylandBall Productions, Andrew Jaffe, Adam Johnson and myself, 9 respondents, as against the few who are arguing vociferously against this PEP. We may or may not be "less advanced Python users", but here we are. And even some of those who were against the PEP didn't absolutely rule out using late-bound defaults themselves (I class the first 2 as roughly neutral and the other 3 as anti): Barry Scott: "Wouldn't use with => syntax, but would with @name syntax" "On a case-by-case basis I might still put defaulting into the body of the function if that made the intent clearer." "I could see me using @file=sys.stdout." Stephen J. Turnbull: "My issues with Chris's proposal are described elsewhere, but I don't really see a problem in principle." André Roberge: 'Currently, I'm thinking "absolutely not". However, I thought the same about the walrus operator and I now miss not being able to use it in a program that includes support for Python 3.6 and where I have literally dozens of places where I would use it if I could.' Paul Moore: "Probably not" "I think that the only thing I might use it for is to make it easier to annotate defaults (as f(a: list[int] => []) rather than as f(a: list[int] | None = None)." "All I'm saying is that the only case when I can imagine using this feature is for when I want a genuinely opaque way of behaving differently if the caller omitted an argument (and using None or a sentinel has been good enough all these years, so it's not exactly a pressing need)." Brendan Barnwell: "I hope not. Realistically I might wind up using it at some point way down the line. I wouldn't start using it immediately. I still almost never use the walrus and only occasionally use f-strings." Brendan, you're really missing out! f-strings are fantastic (especially for debugging)!😁 Get in there! (I was skeptical too, until I started using them.) I hope this post will be useful. Rob Cliffe
On Thu, Dec 9, 2021 at 3:16 PM Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:
Objections to PEP 671 - Summary There seems to be a problem understanding what the objections to PEP 671 are. Chris A wrote: "Part of the problem is that it is really REALLY hard to figure out what the actual objections are. I asked, and the one clear answer I got was one subjective opinion that the cognitive load exceeded the benefit. Great! That's one person's concern. I've responded to that by clarifying parts of the cognitive load problem, and that's about as far as that can go. But if there's nothing more specific than that, what do you want me to respond to? How can I address the objections if the objections are as vague as you're describing?"
Well, I have AFAIK re-read every post in the threads and am attempting to summarise the objections to PEP 671 that I have found.
Thank you. Much appreciated.
Disclaimer: I do NOT claim this is a completely objective survey. I think by now everyone knows on which side of the fence I sit. In some places I have added my own (biased) comments. Nonetheless I have honestly tried to make a fair selection from all the relevant, reasonably important posts. And if anyone thinks that their, or anyone else's objection(s) have been omitted/understated/misrepresented, they are welcome to edit this post and put the result in a new one. Disclaimer: I have not referred to Steven d'Aprano's posts, because I lack the time (ability?) to understand them all. AFAICT he is in favour of something like this PEP, but a bit different.
Unfortunately, neither do I fully understand them, so we're going to have to wait for his clarifications. Or, better, a reference implementation. Until then, let's leave that all aside.
AFAIK these were all the objections that were raised. There may be some overlap or some sub-cases but this is how I've classified them: (A) Don't like the proposed `=>` syntax. (B) Too few use cases to bother making a change to the language. Status quo wins. (C) Some form of "deferred evaluation object" would do everything this PEP does and more, and might be incompatible with it. (D) The (late-binding) default value should be a first-class object that can be accessed and manipulated. (E) Calculation of a "default value" should be done in the function body, not the function header. (F) Concerns that functions using late-bound defaults were harder to wrap. (G) Backward compatibility. (H) No other language supports both early- and late-binding.
(A) I have a few other syntaxes in the PEP, but if syntax is the only issue and I haven't listed someone's preferred syntax, I would be happy to consider others. (B) I've shown some use cases. If you think these aren't of value, no problem, but there's nothing to answer here. (C) This has been asserted without evidence repeatedly. So far, I have yet to see an example of how a deferred-evaluation object could replace default argument handling; the examples have all been massive overkill eg options for parallelism, and entail lots of extra syntax. I'm also not convinced that they'd make late-bound defaults useless, and that hasn't really been explained either. (D) Understood. My response is two-fold: firstly, no other expression in Python is a first-class object, and secondly, the manipulations possible wouldn't be sufficiently general to cover all use-cases. I do provide a textual representation of the default, which in simple situations could be eval'd; in complex situations, nothing external would work anyway. (E) That's purely a matter of opinion, but if that's the case, why aren't *all* defaults done in the body? Why do we have argument defaults in the function header at all? Surely that's of value. It certainly has been to me. (F) Also a matter of opinion, given that *a,**kw is the most common wrapping technique used, and will work reliably. Function signature algebra is a much larger challenge than this. (G) I'm not breaking compatibility in any way. (H) This is true. But if the two syntaxes can be sufficiently similar, the cost should be low, and the feature benefit would be high. Early binding lets you "def f(x=x):" in a loop and capture each x as it goes by. Late binding lets you "def f(x=>[]):" and get a new list every time. Both have their places. If Python had had late binding from the start, and no early binding, we would have developed idioms for early binding (most likely a decorator that captures values or something like that). Both behaviours are sufficiently useful that programmers WILL implement them, language support or not.
I quote from one of Chris A's replies re a `defer` keyword: 'The trouble is that this actually would be incompatible. If you can defer an expensive calculation and have some "placeholder" value in the variable 'result', then logically, you should be able to have that placeholder as a function default argument, which would be an early-bound default of the placeholder. That is quite different in behaviour from a late-bound default, so if I were to use the word "defer" for late-bound defaults, it would actually prevent the more general proposal.'
The incompatibility here, btw, would be using a "defer" keyword to mean late-binding, which isn't the same thing as a "defer expr" keyword that creates a deferred expression. PEP 671 would not be incompatible with a generic deferred-expression concept, as long as they don't use the same word.
(D) The (late-binding) default value should be a first-class object that can be accessed and manipulated.
Eric V.Smith: [AFAIU, paraphrased] I want the default value to be an object that I can inspect and change. David Mertz: "For the reasons Eric Smith and others have pointed out, I really WANT to keep inspectability of function signatures." Stephen J. Turnbull: "More than any of these issues, the lack of a well-defined, properly introspectable object bothers me."
Chris A gave a reply including 'If you consider the equivalent to be a line of code in the function body, then the signature has become MASSIVELY more useful. Instead of simply seeing "x=<object object at 0x7fba1b318690>", you can see "x=>[]" and be able to see what the value would be.'
Correct. In general, expressions are not first-class objects in Python; they only become them when turned into functions or classes (including the special functions used by genexps/comps). We do not have an introspectable, externally-testable, first-class object to represent any other expression: x = 1/y if y else "invalid" There's no object for "1/y". Trying to create one would be a nightmare of subtleties, where assignment expressions would break things, nonlocal variable references would become tricky, etc, etc. Similarly: def f(y): def g(x=1/y): ... return g There's no object for "1/y" here either. As an early-bound default, the expression is simply inlined into the body of f, as part of the construction of the function. Now with late-bound defaults:
def f(y): ... def g(x=>1/y): ... ... return g ... f(4).__defaults_extra__ ('1 / y',)
... you still don't get a first-class object for the expression, but you DO get a string constant that describes it. What is it about this third example that makes the bar so much higher than for the other two? Thank you for this summary. Are there any objections still unanswered? Which of these parts is important enough to add to the PEP? I think I'll be adding a section on introspectability, since it seems to be a hot topic, but I'm not sure about the others. ChrisA
On 2021-12-08 20:55, Chris Angelico wrote:
(H) This is true. But if the two syntaxes can be sufficiently similar, the cost should be low, and the feature benefit would be high. Early binding lets you "def f(x=x):" in a loop and capture each x as it goes by. Late binding lets you "def f(x=>[]):" and get a new list every time. Both have their places.
(The "two syntaxes" here is referring to syntaxes for early and late binding.) I'm actually worried about the reverse. When the two syntaxes are similar, it will be easier to mistake one for the other.
(D) The (late-binding) default value should be a first-class object that can be accessed and manipulated.
Eric V.Smith: [AFAIU, paraphrased] I want the default value to be an object that I can inspect and change. David Mertz: "For the reasons Eric Smith and others have pointed out, I really WANT to keep inspectability of function signatures." Stephen J. Turnbull: "More than any of these issues, the lack of a well-defined, properly introspectable object bothers me."
Chris A gave a reply including 'If you consider the equivalent to be a line of code in the function body, then the signature has become MASSIVELY more useful. Instead of simply seeing "x=<object object at 0x7fba1b318690>", you can see "x=>[]" and be able to see what the value would be.'
Correct. In general, expressions are not first-class objects in Python; they only become them when turned into functions or classes (including the special functions used by genexps/comps). We do not have an introspectable, externally-testable, first-class object to represent any other expression:
x = 1/y if y else "invalid"
There's no object for "1/y". Trying to create one would be a nightmare of subtleties, where assignment expressions would break things, nonlocal variable references would become tricky, etc, etc. Similarly:
As I have stated repeatedly, ternary expressions are not parallel because they do not have a distinct definition-time and call-time. Please stop bringing up the ternary operator case and pretending it is the same as a function. It is not, for reasons that I have already explained several times. To try stating this in yet another way, currently if I have: def f(a=<some code here>) <some code here> must be something that evaluates to a first-class object, and the "argument default" IS that first-class object --- not bytecode to generate it, not some behavior that evaluates the expression, no, the default value is itself an object. This would not be the case for late-bound defaults under the PEP. (Rather, as I phrased it in another post, there would not "be" a late-bound default at all; there would just be some behavior in the function to do some stuff when that argument isn't passed.) -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown