On 1 Dec 2021, at 10:16 AM, Chris Angelico <rosuav@gmail.com> wrote:

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.

*PEP 671: Syntax for late-bound function argument defaults*

Questions, for you all:

1) If this feature existed in Python 3.11 exactly as described, would you use it? Yes I will use it.

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? No, because it would look like a lambda (if the new lambda syntax were approved), indicating this will be evaluated each time the function is run.

(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)

3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

NA

5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I haven’t done it from source. I might try to learn how to do it in the next weekend and give it a try.

I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. I will show this to some of my coworkers who are python experts and I will report back.

Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!

The reference implementation currently has some test failures, which I'm looking into. I'm probably going to make this my personal default Python interpreter for a while, to see how things go.

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UVOQEK... Code of Conduct: http://python.org/psf/codeofconduct/

On Wed, Dec 01, 2021 at 07:47:49AM -0000, Jeremiah Vivian wrote:

...

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? No, it isn't much of a cognitive burden.

You say that now, but if you read function definitions that looked like this: def process(func:List->int=>xs=>expression)->int: ... would you still agree? -- Steve

On Wed, Dec 1, 2021 at 6:58 PM Steven D'Aprano <steve@pearwood.info> wrote:

On Wed, Dec 01, 2021 at 07:47:49AM -0000, Jeremiah Vivian wrote:

...

...

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden? No, it isn't much of a cognitive burden.

You say that now, but if you read function definitions that looked like this:

def process(func:List->int=>xs=>expression)->int: ...

would you still agree?

I'm not sure what that's supposed to mean. Is List->int an annotation, and if so, who's deciding on that syntax? You seem to have used two different arrows in two different ways: List->int # presumably an annotation meaning "function that takes List, returns int" func:ann=>dflt # late-bound default, completely unnecessary here xs=>expression # presumably a lambda function def process(args)->int # return value annotation Why should List->int and xs=>expression use different arrows? Wouldn't it be much more reasonable for them to use the same one, whichever that be? And if that does turn out to be "=>", then yes, I would be looking at changing PEP 671 to recommend := or =: or something, for clarity (but still an equals sign with one other symbol next to it). It's always possible to come up with pathological code. But this is only really as bad as you describe if it has zero spaces in it. Otherwise, it's pretty easy to clarify which parts go where: def process(func: List->int => xs=>expression) -> int: ... Tada, easy grouping. ChrisA

I'll reply to this comment once I've done so. I just realized that this message replied on a comment instead of the thread. Anyways, it's a really impressive implementation (hope this statement doesn't spark some sort of debate).

On 1 Dec 2021, at 12:01 PM, Chris Angelico <rosuav@gmail.com> wrote:

On Wed, Dec 1, 2021 at 6:43 PM Abdulla Al Kathiri <alkathiri.abdulla@gmail.com> wrote:

...

...

On 1 Dec 2021, at 10:16 AM, Chris Angelico <rosuav@gmail.com> wrote: 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else? I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).

Very interesting. That doesn't normally seem like a function default - is the callback ever going to be passed two arguments (self and actual text) such that the default would be ignored? Yeah. Let’s say the callback prints the text on the main window console. I could use the same function to print something on the console not related to the default widget changing text. Maybe another push button that prints literal “WOW”. If I made the second argument a default widget (def callback(self, line_edit=self.line_edit):...) and then call line_edit.text() inside the function, I would be able to pass any other widget that has the method text() but I wouldn’t be able to pass a literal text.

But, hey, if it makes sense in your code to make it a parameter, sure!

...

...

5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :) I haven’t done it from source. I might try to learn how to do it in the next weekend and give it a try.

...

I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative. I will show this to some of my coworkers who are python experts and I will report back.

Thank you! All feedback greatly appreciated.

Building CPython from source can be done by following these instructions:

https://devguide.python.org/setup/

Instead of creating your own clone from the pristine master copy, instead clone my repository at https://github.com/rosuav/cpython and checkout the pep-671 branch. That'll give you my existing reference implementation (it's pretty crummy but it mostly works).

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7VOA7B... Code of Conduct: http://python.org/psf/codeofconduct/

On Wednesday, December 1, 2021 at 1:18:33 AM UTC-5 Chris Angelico wrote:

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.

*PEP 671: Syntax for late-bound function argument defaults*

Questions, for you all:

1) If this feature existed in Python 3.11 exactly as described, would you use it?

No, I would avoid for the sake of readers. Anyway, it can't be used in any public projects until thy drop support for 3.10, which is many years off. Also, I think this question is quite the biased sample on python-ideas. Please consider asking this to less advanced python users, e.g., reddit.com/r/python or learnpython.

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?

(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)

Yes.

3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

No.

5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)

I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative.

Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!

This PEP has a lot of interesting ideas. I still think that none-aware operators (PEP 505) are an easier, more readable general solution to binding calculated default values to arguments succinctly. I think the problems with this idea include: * The caller cannot explicitly ask for default behaviour except by omitting the parameter. This can be very annoying to set up when the parameter values are provided from other places, e.g., if need_x: # do lots of stuff x = whatever else: # do more x = None f(x=x, ...) # easy, but with this PEP, you would have to find a way to remove x from the parameter list. Typically, removing a parameter from a dynamically-created parameter list is hard. * The function code becomes unreadable if the parameter-setting code is long. Having a long piece of code inside the function after "if parameter is None" is just fine. Having none-aware operators would make such code more succinct. * People nearly always avoid writing code in the parameter defaults themselves, and this new practice adds a lot of cognitive load. E.g., people rarely write: def f(x: int = 1+g()) -> None: ... Parameter lists are already busy enough with parameter names, annotations, and defaults. We don't need to encourage this practice. In short, I think this is a creative idea, a great exploration. While optional parameters are common, and some of them have defaults that are calculated inside the function, my feeling is that people will continue to set their values inside the function. Best, Neil

On Wed, Dec 1, 2021 at 8:57 PM Neil Girdhar <mistersheik@gmail.com> wrote:

On Wednesday, December 1, 2021 at 1:18:33 AM UTC-5 Chris Angelico wrote:

...

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.

*PEP 671: Syntax for late-bound function argument defaults*

Questions, for you all:

1) If this feature existed in Python 3.11 exactly as described, would you use it?

No, I would avoid for the sake of readers. Anyway, it can't be used in any public projects until thy drop support for 3.10, which is many years off.

Also, I think this question is quite the biased sample on python-ideas. Please consider asking this to less advanced python users, e.g., reddit.com/r/python or learnpython.

I know it is :) I never expected to get an unbiased sample (from anywhere, least of all here).

This PEP has a lot of interesting ideas. I still think that none-aware operators (PEP 505) are an easier, more readable general solution to binding calculated default values to arguments succinctly. I think the problems with this idea include: * The caller cannot explicitly ask for default behaviour except by omitting the parameter. This can be very annoying to set up when the parameter values are provided from other places, e.g., if need_x: # do lots of stuff x = whatever else: # do more x = None f(x=x, ...) # easy, but with this PEP, you would have to find a way to remove x from the parameter list. Typically, removing a parameter from a dynamically-created parameter list is hard.

You can always do that with *args or **kwargs if that's what you need, and that actually DOES represent the concept of "don't pass this argument". Passing None is a hack that doesn't actually mean "don't pass this argument", and if you're depending on that behaviour, it's been locked in as the function's API, which causes problems if None ever becomes a meaningful non-default value. None-aware operators are solving a different problem. They allow you to accept None, but then simplify the replacement of it with some other value. This proposal allows you to leave None out of the equation altogether. And it only works if None isn't a valid value.

* The function code becomes unreadable if the parameter-setting code is long. Having a long piece of code inside the function after "if parameter is None" is just fine. Having none-aware operators would make such code more succinct.

Again, that only works if None is not a valid parameter value, and PEP 505 doesn't generalize to other sentinels at all. If the code is too long, don't put it into the parameter default. But "if x is None: x = []" can easily be replaced with "x=>[]" and the parameters will actually become shorter. ANY feature can be used badly. You can daisy-chain everything into a gigantic expression with horrendous abuses of lambda functions, but that doesn't mean you should. It also doesn't mean that lambda functions are bad :)

* People nearly always avoid writing code in the parameter defaults themselves, and this new practice adds a lot of cognitive load. E.g., people rarely write: def f(x: int = 1+g()) -> None: ... Parameter lists are already busy enough with parameter names, annotations, and defaults. We don't need to encourage this practice.

They don't become any more noisy by being able to use defaults that aren't constant. The main goal is to put the function header in the function header, and the function body in the function body, instead of having the function body doing part of the work of the header :)

In short, I think this is a creative idea, a great exploration. While optional parameters are common, and some of them have defaults that are calculated inside the function, my feeling is that people will continue to set their values inside the function.

In some cases, they certainly will. And that's not a problem. But for the cases where it's better to put it in the header, it's better to have that option than to restrict it for an arbitrary technical limitation. Some languages don't have ANY function defaults. They simply have "optional arguments", where an omitted optional argument will always and only have a specific null value. Python allows you to show what the default REALLY is, and that's an improvement. I want to be able to show what the default is, even if that is defined in a non-constant way. ChrisA

On Wed, Dec 1, 2021 at 2:17 AM Chris Angelico <rosuav@gmail.com> wrote:

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.

*PEP 671: Syntax for late-bound function argument defaults*

Questions, for you all:

1) If this feature existed in Python 3.11 exactly as described, would you use it?

Currently, I'm thinking "absolutely not".

However, I thought the same about the walrus operator and I now miss not being able to use it in a program that includes support for Python 3.6 and where I have literally dozens of places where I would use it if I could.

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?

Yes. I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

*Perhaps* if a keyword would be used instead of symbols, I might reconsider. I find the emphasis of trying to cram too much information in single lines of code to be really a burden. Many years ago, I argued very unsuccessfully for using a 'where:' code block for annotations. (To this day, I still believe it would make the code much more readable, at the cost of a slight duplication.) Using what is at first glance a cryptic operator like => for late binding is not helping readability, especially when type annotations are thrown in the mix. Aside: at the same time, I can see how using => instead of lambda as a potential win in readability, including for beginners.

5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)

Sorry, I'm not interested enough at this point but, given the amount of work you put into this, I decided that the least I could do is provide feedback rather than be a passive reader. André Roberge

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UVOQEK... Code of Conduct: http://python.org/psf/codeofconduct/

Chris Angelico wrote:

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ Questions, for you all: 1) If this feature existed in Python 3.11 exactly as described, would you use it?

When needed, but I'm uncomfortable with the syntax.

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?

Yes. A lot of arrows with different meanings require the reader of the code to know and remember what each does. Something more distinctive feels better. I'm way too new to Python to have much say, but anyway, arrows make more sense for other uses. Late binding feels to me like it is best served by something more distinctive.

3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?

I'd certainly use it only when I felt it was absolutely necessary. The readability is not aided by utilising syntax to alter behaviour when the altered behaviour is not required. We only quote, escape, etc., when we need to. (Silly example to illustrate, I know.)

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

When I first saw this (very few months ago indeed!!), I wondered about something rather clunkier but more explicit, such as, for example -- ```def foo(x=__late__['bar']): def bar(): #something etc... ``` Ugly though. And I subsequently realised some use cases are not well-served by this. (Particularly when [speculatively] considering some implementation issues.) The point in any case, is that for the occasions when the mechanism is needed, then a clearly explicit way to indicate something is evaluated late, and a tidy place to put the definition of how it's handled (late), seemed to me to be preferable. Of the **cited** examples, I find the `@hi=len(a)` spelling most readable.

5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)

Sadly not yet, but time permitting I will one day figure that out, for sure.

bikeshedding is what we do best!

Should we introduce choice of colour, as we run out of new symbols to use? /jk JL

On Wed, Dec 1, 2021 at 1:18 AM Chris Angelico <rosuav@gmail.com> wrote:

1) If this feature existed in Python 3.11 exactly as described, would you use it?

No. ... except in the sense that as I trainer I have to teach the warts in Python, and would need to warn students they might see that.

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?

YES! A few weeks later than the prior long discussion that I read in full, it took a triple take not to read it as >= (which would mean something syntactical in many cases, just not what is intended).

3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?

I would always recommend against its use if I had any influence on code review.

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

Yes, the delay/later/defer keyword approach is not confusing, and does not preempt a later feature that would actually be worth having. 5) Do you know how to compile CPython from source, and would you be

willing to try this out? Please? :)

I do know how, but it is unlikely I'll have time. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

On Wed, Dec 1, 2021 at 7:51 AM Chris Angelico <rosuav@gmail.com> wrote:

On Wed, Dec 1, 2021 at 10:30 PM André Roberge <andre.roberge@gmail.com> wrote:

...

...

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?

Yes. I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.

Also fair. I'm not a fan of keywords for this sort of thing, since it implies that you could do this:

def f(x=defer []): ...

dflt = defer [] def f(x=dflt): ...

which is a completely different proposal (eg it would be evaluated only when you "touch" that, rather than being guaranteed to be evaluated before the first line of the function body). That's why I want to adorn the equals sign and nothing else.

Shouldn't the PEP contain a rejected idea section where this could be mentioned?

...

...

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

*Perhaps* if a keyword would be used instead of symbols, I might reconsider.

I find the emphasis of trying to cram too much information in single lines of code to be really a burden. Many years ago, I argued very unsuccessfully for using a 'where:' code block for annotations. (To this day, I still believe it would make the code much more readable, at the cost of a slight duplication.) Using what is at first glance a cryptic operator like => for late binding is not helping readability, especially when type annotations are thrown in the mix.

Aside: at the same time, I can see how using => instead of lambda as a potential win in readability, including for beginners.

It's interesting how different people's views go on that sort of thing. It depends a lot on how much people expect to use something. Features you use a lot want to have short notations, features you seldom use are allowed to have longer words.

I rarely use lambda in my own code, and have never written a line of code

anywhere that uses a '=>' operator. If Python had a 'function' keyword instead of 'lambda', I would prefer to keep the function keyword instead of adding => as a symbol. For me, it is not a question of terseness for commonly used features, but one of easing the learning curve. Starting from zero, I do believe that => would be easier to grasp than learning about lambda as a keyword and the syntactic rules to use with it. With function as a keyword, I believe that it is the other way around. No doubt many others will disagree! André Roberge

_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WTU355... Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, Dec 2, 2021 at 1:21 AM Paul Moore <p.f.moore@gmail.com> wrote:

On Wed, 1 Dec 2021 at 06:19, Chris Angelico <rosuav@gmail.com> wrote:

...

3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?

N/A, except to say that when you enumerate the use cases like this, none of them even tempt me to use this feature.

I'm just listing some of the common use-cases, not all of them.

I think that the only thing I might use it for is to make it easier to annotate defaults (as f(a: list[int] => []) rather than as f(a: list[int] | None = None).

That's exactly the goal! It's hugely simpler to say "this is always a list, and defaults to a new empty list" than "this is a list or None, defaults to None, and hey, if it's None, I'll make a new empty list".

So I'll revise my answer to (1) and say that I *might* use this, but only in a way it wasn't intended to be used in, and mostly because I hate how verbose it is to express optional arguments in type annotations. (And the fact that the annotation exposes the sentinel value, even when you want it to be opaque). I hope I don't succumb and do that, though ;-)

That's definitely an intended use-case ("mutable defaults" above); you're focusing on the fact that it improves the annotations, I'm focusing on the fact that it improves documentation and introspection, but it's all the same improvement :)

...

5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)

Sorry, I really don't have time to, in the foreseeable future. If I did have time, one thing I would experiment with is how this interacts with typing and tools like pyright and mypy (yes, I know type checkers would need updating for the new syntax, so that would mostly be a thought experiment) - as I say, I'd expect to annotate a function with an optional list argument defaulting to an empty list as f(a: list[int] => []), which means that __annotations__ needs to distinguish between this case and f(a: list[int]) with no default.

No problem, yup.

Sorry, I don't have any feedback like that. What I can say, though, is I'd find it quite hard to express the question, in the sense that I'd struggle to explain the difference between early and late bound parameters to a non-expert, much less explain why we need both. I'd probably just say "it's short for a default of None and a check" which doesn't really capture the point...

The point is that it's a shortcut for "omitted" rather than "a default of None", but there are many ways to explain it, and I only have one brother who knows enough Python to be able to talk to about this (and I got feedback from him VERY early).

...

Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!

I hope this was useful feedback.

It was. Thank you. ChrisA

On Thu, Dec 2, 2021 at 1:33 AM André Roberge <andre.roberge@gmail.com> wrote:

On Wed, Dec 1, 2021 at 7:51 AM Chris Angelico <rosuav@gmail.com> wrote:

...

On Wed, Dec 1, 2021 at 10:30 PM André Roberge <andre.roberge@gmail.com> wrote:

...

...

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?

Yes. I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.

Also fair. I'm not a fan of keywords for this sort of thing, since it implies that you could do this:

def f(x=defer []): ...

dflt = defer [] def f(x=dflt): ...

which is a completely different proposal (eg it would be evaluated only when you "touch" that, rather than being guaranteed to be evaluated before the first line of the function body). That's why I want to adorn the equals sign and nothing else.

Shouldn't the PEP contain a rejected idea section where this could be mentioned?

Hmm, maybe. It's such a completely different proposal, but it does get asked a few times. If someone could actually put it forward as a full proposal, I'd gladly mention it as an interaction with another PEP. Otherwise, I'll write up a very brief thing on deferred expressions and how they're not what this is about. ChrisA

On Wed, Dec 1, 2021 at 10:12 AM Chris Angelico <rosuav@gmail.com> wrote:

On Thu, Dec 2, 2021 at 12:42 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:

...

...

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

Yes, the delay/later/defer keyword approach is not confusing, and does not preempt a later feature that would actually be worth having.

Do you mean changing the spelling of the existing proposal, or a completely different proposal for deferred objects that are actually objects? Because that is NOT what I mean by a "small change". :)

The spelling of the existing proposal. I.e. if the proposal were: def fun(things: list[int] = defer []) -> int: # ... some implementation I'd be -0 on the idea rather than -100. Yes, my change in attitude is largely because I want *some future PEP* to address the more general situation like: result = defer really_expensive_calculation() if predicate: doubled = result * 2 But I do not think your PEP does (nor even should) include that potential future behavior/syntax. Such a hypothetical future PEP would have a continuity with the syntax of your feature, albeit DEFINITELY need to address many independent concerns/issues that yours does not create. However, even if I assume the mythical future PEP never happens, in terms of readability, a WORD is vastly less confusing than a combination of punctuation that has no obvious or natural interpretation like '=>'. Or rather, I think that spelling is kinda-sorta obvious for the lambda meaning, and the use you want is kinda-sorta similar to a lambda. So I *do* understand how you get there... but it still seems like much too much line noise for a very minimal need. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

On 11/30/21 10:16 PM, Chris Angelico wrote:

*PEP 671: Syntax for late-bound function argument defaults*

Questions, for you all:

1) If this feature existed in Python 3.11 exactly as described, would you use it?

No.

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?

Yes.

3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?

a, b, c

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

Have the token/keyword be at the beginning instead of in the middle. -- ~Ethan~

On Thu, Dec 2, 2021 at 2:40 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:

On Wed, Dec 1, 2021 at 10:12 AM Chris Angelico <rosuav@gmail.com> wrote:

...

On Thu, Dec 2, 2021 at 12:42 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:

...

...

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

Yes, the delay/later/defer keyword approach is not confusing, and does not preempt a later feature that would actually be worth having.

Do you mean changing the spelling of the existing proposal, or a completely different proposal for deferred objects that are actually objects? Because that is NOT what I mean by a "small change". :)

The spelling of the existing proposal. I.e. if the proposal were:

def fun(things: list[int] = defer []) -> int: # ... some implementation

I'd be -0 on the idea rather than -100.

Okay. If it's JUST the spelling, then yes, I'll take that into consideration (though I am still against keywords myself).

Yes, my change in attitude is largely because I want *some future PEP* to address the more general situation like:

result = defer really_expensive_calculation() if predicate: doubled = result * 2

But I do not think your PEP does (nor even should) include that potential future behavior/syntax. Such a hypothetical future PEP would have a continuity with the syntax of your feature, albeit DEFINITELY need to address many independent concerns/issues that yours does not create.

However, even if I assume the mythical future PEP never happens, in terms of readability, a WORD is vastly less confusing than a combination of punctuation that has no obvious or natural interpretation like '=>'. Or rather, I think that spelling is kinda-sorta obvious for the lambda meaning, and the use you want is kinda-sorta similar to a lambda. So I *do* understand how you get there... but it still seems like much too much line noise for a very minimal need.

The trouble is that this actually would be incompatible. If you can defer an expensive calculation and have some "placeholder" value in the variable 'result', then logically, you should be able to have that placeholder as a function default argument, which would be an early-bound default of the placeholder. That is quite different in behaviour from a late-bound default, so if I were to use the word "defer" for late-bound defaults, it would actually prevent the more general proposal. ChrisA

On Thu, Dec 2, 2021 at 4:40 AM Barry Scott <barry@barrys-emacs.org> wrote:

On 1 Dec 2021, at 06:16, Chris Angelico <rosuav@gmail.com> wrote: 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?

yes (a) What does (b) mean? example please. yes (c)

global_default = 500 def do_thing(timeout=>global_default): ... If the global_default timeout changes between function definition and call, omitting timeout will use the updated global. Similarly, you could say "file=>sys.stdout" and if code elsewhere changes sys.stdout, you'll use that. ChrisA

On Thu, Dec 2, 2021 at 8:50 AM Barry Scott <barry@barrys-emacs.org> wrote:

...

On 1 Dec 2021, at 17:59, Chris Angelico <rosuav@gmail.com> wrote:

On Thu, Dec 2, 2021 at 4:40 AM Barry Scott <barry@barrys-emacs.org> wrote:

...

On 1 Dec 2021, at 06:16, Chris Angelico <rosuav@gmail.com> wrote: 3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?

yes (a) What does (b) mean? example please. yes (c)

global_default = 500 def do_thing(timeout=>global_default): ...

...

If the global_default timeout changes between function definition and call, omitting timeout will use the updated global.

Similarly, you could say "file=>sys.stdout" and if code elsewhere changes sys.stdout, you'll use that.

On a case-by-case basis I might still put defaulting into the body of the function if that made the intent clearer.

I could see me using @file=sys.stdout.

That's a simplified version, but you might have the same default shown as get_default_timeout() or Defaults.timeout or something like that. The point is that it might be a simple integer, but it could change at any time, and the default should always be "whatever this name refers to". In any case, it's just one of many use-cases. I was curious what people would use and what they wouldn't. ChrisA

To be honest, I like the `@param=value` syntax. It is sort of easier to read than `param=>value`, though I do not have problems with distinguishing the arrow `=>` from other things.

On Thu, Dec 2, 2021 at 1:30 PM Steven D'Aprano <steve@pearwood.info> wrote:

On Wed, Dec 01, 2021 at 12:26:33PM +0000, Matt del Valle wrote:

...

If anything I think the symmetry between the proposed '=>' syntax and the arrow syntax for lambdas in other languages (potentially even in python in the future) reduces the cognitive burden significantly, given the there is an equivalent symmetry with their semantics (in both cases the code is being evaluated later when something is called).

It may be that behind the scenes the default expression is stored as a callable function which the interpreter calls. (I believe that list comprehensions do something similar.) But that's an implementation detail that can change: it might just as well store the source code as a string, and pass it to eval().

In my reference implementation, there is no object that stores it; it's simply part of the function. A good parallel is the if/else expression: x = float("inf") if z == 0 else y/z Is there an object that represents the if/else expression (or the deferred "y/z" part)? No, although it could be implemented that way if you chose: x = iff(z == 0, lambda: y/z, lambda: float("inf")) But just because you CAN use a function to simulate this behaviour, that doesn't mean that it's inherently a function. So I agree with Steve that the parallel is quite weak. That said, though, I still like the arrow notation, not because of any parallel with a lambda function, but because of the parallel with assignment. (And hey. You're allowed to think of things in any way you like. I'm not forcing you to interpret everything using the same justifications I do.)

Or use some other mechanism that I'm not clever enough to think of, so I shall just call "deepest black magic".

Remind me some time to use some slightly shallower black magic.

There is no *neccessity* for the late-bound default to be a hidden function, and it is certainly not part of the semantics of late-bound defaults. Just the implementation.

If you disagree, and still think that the symmetry is powerful enough to use the same syntax for both lambdas and default arguments, well, how about if we *literally* do that?

def function(spam=expression, # regular default lambda eggs: expression, # late-bound default )

Heh, perhaps the symmetry is not that strong after all :-)

That..... is hilariously confusing. I like it. Just not EVER in the language :) ChrisA

On Wed, Dec 1, 2021, 10:38 PM Steven D'Aprano

"If param is missing **or None**, the default if blah..." I reject Chris' characterisation of this as a hack. There are function parameters where None will *never* in any conceivable circumstances become a valid argument value, and it is safe to use it as a sentinel.

In particular, the cases where None will never, in any conceivable circumstances, be a non-sentinel value are at least 98% of all the functions (that have named parameters) I've ever written in Python. All of this discussion of a syntax change is for, at most, 2% of functions that need a different sentinel.

Steven D'Aprano ""If param is missing **or None**, the default if blah..." I reject Chris' characterisation of this as a hack. There are function parameters where None will *never* in any conceivable circumstances become a valid argument value, and it is safe to use it as a sentinel." Yes, we know *why* the hack works. We're all familiar with it. That doesn't mean it's not a hack. The bottom line is: you *don't actually* want the parameter to default to the value of a sentinel. you *have* to use that hack because you can't express what you want the default to actually be. You're doing something misleading to work around a shortcoming of the language. That's a hack. You have to write something that you don't actually intend. On Wednesday, December 1, 2021 at 9:39:12 PM UTC-6 Steven D'Aprano wrote:

On Wed, Dec 01, 2021 at 05:16:34PM +1100, Chris Angelico wrote:

...

1) If this feature existed in Python 3.11 exactly as described, would you use it?

Yes I would, but probably not as often as counting cases of the "if param is None: ..." idiom might lead you to expect.

The problem is, as Neil also pointed out, that it becomes tricky to explicitly ask for the default behaviour except by leaving the argument out altogether. Not impossible, as Chris mentions elsewhere, you can mess about with `*args` or `**kwargs`, but it is decidedly less convenient, more verbose, and likely to have a performance hit.

So if I were messing about in the interactive interpreter, I would totally use this for the convenience:

def func(L=>[]): ...

but if I were writing a library, I reckon that probably at least half the time I'll stick to the old idiom so I can document the parameter:

"If param is missing **or None**, the default if blah..."

I reject Chris' characterisation of this as a hack. There are function parameters where None will *never* in any conceivable circumstances become a valid argument value, and it is safe to use it as a sentinel.

For example, I have a function that takes a `collation=None` parameter. The collation is a sequence of characters, usually a string. Using None to indicate "use the default collation" will never conflict with some possible future use of "use None as the actual collation" because None is not a sequence of characters.

In this specific case, my function's current signature is:

def itoa(n, base, collation=None, plus='', minus='-', width=0, ):

I could re-write it as:

def itoa(n, base, collation=>_default_collation(base), plus='', minus='-', width=0, ):

but that would lose the ability to explicitly say "use the default collation" by passing None. So I think that, on balance, I would likely stick to the existing idiom for this function.

On the other hand, if None were a valid value, so the signature used a private and undocumented sentinel:

collation=_MISSING

(and I've used that many times in other functions) then I would be likely to swap to this feature and drop the private sentinel.

...

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?

I think you know my opinion on the `=>` syntax :-)

I shall not belabour it further here. I reserve the right to belabour it later on :-)

...

3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?

Any of the above.

...

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

There's lots of Python features that I use even though I don't like the spelling. Not a day goes by that I don't wish we spelled the base class of the object hierarchy "Object", so it were easier to distinguish between Object the base class and object as a generic term for any object.

If the Steering Council loves your `=>` syntax then I would disagree with their decision but still use it.

-- Steve _______________________________________________ Python-ideas mailing list -- python...@python.org To unsubscribe send an email to python-id...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python...@python.org/message/G3V47ST6I... <https://mail.python.org/archives/list/python-ideas@python.org/message/G3V47S...> Code of Conduct: http://python.org/psf/codeofconduct/

Exactly like you. I can read => to mean late-bound or supposedly lambda function and I can distinguish it from the callable typing -> but "@param=value” is definitely better in my eyes. => won’t prevent me from using it for default mutable objects.

On 2 Dec 2021, at 3:45 AM, Jeremiah Vivian <nohackingofkrowten@gmail.com> wrote:

To be honest, I like the `@param=value` syntax. It is sort of easier to read than `param=>value`, though I do not have problems with distinguishing the arrow `=>` from other things.

On 2021-12-01 18:35, Chris Angelico wrote:

In my reference implementation, there is no object that stores it; it's simply part of the function. A good parallel is the if/else expression:

x = float("inf") if z == 0 else y/z

Is there an object that represents the if/else expression (or the deferred "y/z" part)? No, although it could be implemented that way if you chose:

This is not a good parallel. There is nothing deferred there. The entire line is evaluated when it is encountered and you get a result and no part of the if/else expression can ever impact anything else again unless, via some external control flow, execution returns and re-executes the entire line. That is not comparable to a function default, which is STORED and evaluated later independently of the context in which it was originally written (i.e., the function default is re-executed but the function definition itself is not re-executed). The ternary expression vanishes without a trace by the next line, leaving only its evaluated result. There would be no use in being able to access some part of it, since the whole (i.e., the ternary expression) is completely finished by the time you would be able to access it. This is not the case with a function definition. The function definition leaves behind a function object, and that function object needs to "know" about the late-bound default as an independent entity so that it can be evaluated later. It is bad for the function to store that late-bound default only in some private format for its exclusive future use without providing any means for other code to access it as a first-class value. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On Thu, Dec 2, 2021 at 6:12 PM <role.pythonorg-readers@jlassocs.com> wrote:

Brendan Barnwell wrote:

...

...

No. As I mentioned in the earlier thread, I don't support any proposal in which an argument can "have a default" but that default is not a first-class Python object of some sort.

What if a default is a function?

I was inspired by learning Django and saw in models that fields can have a default which is either a regular (early-bound) default such as a first-class Python object as one would expect, *or* a function -- which will be called 'later' when needed.

That prompted me to contemplate a syntax for late-bound defaults, albeit a bit clunky, but I did think it suited a special-case requirement met by late-bound defaults. I still think that littering function arguments throughout all code with large numbers of arrows would make things less readable. Special requirements need special treatment, I'm thinking.

The problem is passing arguments to such a function without it looking like it's being called at definition time.

Also has the same problem of other deferreds, which is: when exactly is it evaluated? def func(stuff, n=>len(stuff)): stuff.append("spam") print(n) func(["spam", "ham"]) What will this print? If it's a function default, it MUST print 2, since len(stuff) is 2 as the function starts. But if it's a deferred object of some sort, then it should probably print 3, since len(stuff) is 3 at the time that n gets printed. Which is it to be? That's why PEP 671 is *not* about generic deferred evaluation. It is specifically about function default arguments, and guarantees that they will be evaluated prior to the first line of code in the function body. ChrisA

On 2021-12-01 23:36, Chris Angelico wrote:

That's exactly why it's such a close parallel. The late-evaluated default is just code, nothing else. It's not "stored" in any way - it is evaluated as part of the function beginning execution.

But it IS stored! There is no way for it to be evaluated without it being stored! I know we're talking past each other here but it is quite obvious that something has to be stored if it is going to be evaluated later. You can say that it is "just code" but that doesn't change the fact that that code has to be stored. You can say that it is just prepended to the function body but that's still storing it. That is still not parallel to a ternary operator in which no part of the expression is EVER re-executed unless control flow causes execution to return to that same source code line and re-execute it as a whole. Actually this raises a question that maybe was answered in the earlier thread but if so I forgot: if a function has a late-bound default, will the code to evaluate it be stored as part of the function's code object? -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On Thu, Dec 2, 2021 at 6:59 PM Brendan Barnwell <brenbarn@brenbarn.net> wrote:

On 2021-12-01 23:36, Chris Angelico wrote:

...

That's exactly why it's such a close parallel. The late-evaluated default is just code, nothing else. It's not "stored" in any way - it is evaluated as part of the function beginning execution.

But it IS stored! There is no way for it to be evaluated without it being stored!

I know we're talking past each other here but it is quite obvious that something has to be stored if it is going to be evaluated later. You can say that it is "just code" but that doesn't change the fact that that code has to be stored. You can say that it is just prepended to the function body but that's still storing it. That is still not parallel to a ternary operator in which no part of the expression is EVER re-executed unless control flow causes execution to return to that same source code line and re-execute it as a whole.

I'm not sure I understand you here. How is the late-bound default "stored" when one side of a ternary is "not stored"?

Actually this raises a question that maybe was answered in the earlier thread but if so I forgot: if a function has a late-bound default, will the code to evaluate it be stored as part of the function's code object?

Yes. To be precise, it is part of the code object's co_code attribute - the bytecode (or wordcode if you prefer) of the function. Here's how a ternary if looks:

...

...

def f(n): ... return 0 if n == 0 else 42/n ... dis.dis(f) 2 0 LOAD_FAST 0 (n) 2 LOAD_CONST 1 (0) 4 COMPARE_OP 2 (==) 6 POP_JUMP_IF_FALSE 6 (to 12) 8 LOAD_CONST 1 (0) 10 RETURN_VALUE >> 12 LOAD_CONST 2 (42) 14 LOAD_FAST 0 (n) 16 BINARY_TRUE_DIVIDE 18 RETURN_VALUE

The "42/n" part is stored in f.__code__.co_code as the part that says "LOAD_CONST 42, LOAD_FAST n, BINARY_TRUE_DIVIDE". It's not an object. It's just code - three instructions. Here's how (in the reference implementation - everything is subject to change) a late-bound default looks:

...

...

def f(x=>[]): print(x) ... dis.dis(f) 1 0 QUERY_FAST 0 (x) 2 POP_JUMP_IF_TRUE 4 (to 8) 4 BUILD_LIST 0 6 STORE_FAST 0 (x) >> 8 LOAD_GLOBAL 0 (print) 10 LOAD_FAST 0 (x) 12 CALL_FUNCTION 1 14 POP_TOP 16 LOAD_CONST 0 (None) 18 RETURN_VALUE

The "=>[]" part is stored in f.__code__.co_code as the part that says "QUERY_FAST x, and if false, BUILD_LIST, STORE_FAST x". It's not an object. It's four instructions in the bytecode. In both cases, no part of the expression is ever re-executed. I'm not understanding the distinction here. Can you explain further please? ChrisA

On Thu, Dec 2, 2021 at 7:31 PM Paul Moore <p.f.moore@gmail.com> wrote:

On Wed, 1 Dec 2021 at 22:27, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:

...

On 2/12/21 4:40 am, Paul Moore wrote:

...

the intended use is that people must supply a list[int] or not supply the argument *at all*.

I don't think this is a style of API that we should be encouraging people to create, because it results in things that are very awkward to wrap.

Hmm, interesting point, I agree with you. It's particularly telling that I got sucked into designing that sort of API, even though I know it's got this problem. I guess that counts as an argument against the late bound defaults proposal - or maybe even two:

1. It's hard (if not impossible) to wrap functions that use late-bound defaults. 2. The feature encourages people to write such unwrappable functions when an alternative formulation that is wrappable is just as good.

(That may actually only be one point - obviously a feature encourages people to use it, and any feature can be over-used. But the point about wrappability stands).

Wrappability when using an arbitrary sentinel looks like this: _SENTINEL = object() def f(value=_SENTINEL): if value is _SENTINEL: value = ... # in another module def wrap_f(): othermodule.f(othermodule._SENTINEL) Or: def wrap_f(): othermodule.f(othermodule.f.__defaults__[0]) I'm not sure either of these is any better than the alternatives. You're reaching into a module to access its internal implementation details. Is that really better than messing with *args? def wrap_f(): args = [42] or [] # pass or don't pass? othermodule.f(*args) It only looks easy when None is used, and even then, it's largely an implementation detail. ChrisA

On Wed, Dec 01, 2021 at 10:50:38PM -0800, Brendan Barnwell wrote:

...

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

No. As I mentioned in the earlier thread, I don't support any proposal in which an argument can "have a default" but that default is not a first-class Python object of some sort.

I don't understand this criticism. Of course the default value will be a first-class Python object of some sort. *Every* value in Python is a first-class object. There are no machine values or unboxed values, and this proposal will not change that. All that this proposal changes is *when* and *how often* the default will be evaluated, not the nature of the value. Status quo: default values are evaluated once, when the def statement is executed. With optional late-binding: default values are evaluated as often as they are needed, when the function is called. But the value will still be an object. I suppose that there will be one other change, relating to introspection of the function. You will no longer be able to inspect the function and see the default values as constants in a cache: >>> (lambda x=1.25: None).__defaults__ (1.25,) Depending on the implementation, you *might* be able to inspect the function and see the default expression as some sort of callable function, or evaluatable code object. (That would be nice.) Or even as a plain old string. All of which are first-class objects. Or it might be that the default expression will be compiled into the body of the function, where is it effectively invisible. So I guess that's a third change: when, how often, and the difference to introspection. -- Steve

On Thu, Dec 2, 2021 at 8:40 PM Steven D'Aprano <steve@pearwood.info> wrote:

Depending on the implementation, you *might* be able to inspect the function and see the default expression as some sort of callable function, or evaluatable code object. (That would be nice.)

Unfortunately not, since the default expression could refer to other parameters, or closure variables, or anything else from the context of the called function. So you won't be able to externally evaluate it.

Or even as a plain old string. All of which are first-class objects.

For documentation purposes, it is indeed available as a plain old string. In colloquial terms, it is the source code for the expression, although technically it's reconstructed from the AST. ( This is approximately as accurate as saying that the repr of an object is its source code. Possibly more accurate, actually.)

Or it might be that the default expression will be compiled into the body of the function, where is it effectively invisible. So I guess that's a third change: when, how often, and the difference to introspection.

Among our changes are when, how often, the difference to introspection, and the ability to externally manipulate the defaults. And nice red uniforms. Although the ability to manipulate them could be considered part of introspection, not sure. I'm still unsure whether this is a cool feature or an utter abomination:

...

...

def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f.__defaults_extra__ = ("n/a",) f(42) You passed x as 42 f() You didn't pass x

(It's an implementation detail and not part of the specification, but if CPython adopts this behaviour, it will become de facto part of the expected behaviour, and someone somewhere will use this deliberately.) ChrisA

On 01/12/2021 15:39, David Mertz, Ph.D. wrote:

However, even if I assume the mythical future PEP never happens, in terms of readability, a WORD is vastly less confusing than a combination of punctuation that has no obvious or natural interpretation like '=>'.  Or rather, I think that spelling is kinda-sorta obvious for the lambda meaning, and the use you want is kinda-sorta similar to a lambda.  So I *do* understand how you get there... but it still seems like much too much line noise for a very minimal need.

Hm.  A word is "vastly less confusing".  OK.  Should Python have been designed with     x assigned y+1 rather than     x = y +1 (note that '=' here does not have its "obvious or natural interpretation", viz. "is equal to"). Should we have     (x becomes y+1) rather than     (x := y+1) Conciseness *is* a virtue, even if it is often trumped by other considerations. Best wishes Rob Cliffe

On 02/12/2021 07:05, Brendan Barnwell wrote:

On 2021-12-01 18:35, Chris Angelico wrote:

...

In my reference implementation, there is no object that stores it; it's simply part of the function. A good parallel is the if/else expression:

x = float("inf") if z == 0 else y/z

Is there an object that represents the if/else expression (or the deferred "y/z" part)? No, although it could be implemented that way if you chose:

    This is not a good parallel.  There is nothing deferred there.  The entire line is evaluated when it is encountered and you get a result and no part of the if/else expression can ever impact anything else again unless, via some external control flow, execution returns and re-executes the entire line.  That is not comparable to a function default, which is STORED and evaluated later independently of the context in which it was originally written (i.e., the function default is re-executed but the function definition itself is not re-executed).

    The ternary expression vanishes without a trace by the next line, leaving only its evaluated result.  There would be no use in being able to access some part of it, It could be useful for debugging. Best wishes Rob Cliffe since the whole (i.e., the ternary expression) is completely finished by the time you would be able to access it.  This is not the case with a function definition.  The function definition leaves behind a function object, and that function object needs to "know" about the late-bound default as an independent entity so that it can be evaluated later.  It is bad for the function to store that late-bound default only in some private format for its exclusive future use without providing any means for other code to access it as a first-class value.

On Thu, Dec 2, 2021 at 3:33 AM Chris Angelico <rosuav@gmail.com> wrote:

...

But it IS stored! There is no way for it to be evaluated

without it

...

being stored!

I'm not sure I understand you here. How is the late-bound default "stored" when one side of a ternary is "not stored"?

...

...

def foo(a): ... b = a + 1 ... print(b) ... foo.__code__ <code object foo at 0x7f167e539710, file "<ipython-input-7-3c44060a0872>",

This seems like dishonest argument. I'm not even sure what point you think it is making. Every time I write a function, everything the function does needs to be STORED. The body is *stored* in the .__code__ attribute. Other things are stored in .__annotations__ and elsewhere. A function is an OBJECT, and everything about it has to be attributes of that object. line 1> A late binding isn't that one thing about a function that never gets stored, but floats in the ether magically ready to operate on a function call by divine intervention. It HAS TO describe *something* attached to the function object, doing *something* by some means. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

On Fri, Dec 3, 2021 at 12:43 AM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:

On Thu, Dec 2, 2021 at 3:33 AM Chris Angelico <rosuav@gmail.com> wrote:

...

...

But it IS stored! There is no way for it to be evaluated without it being stored!

I'm not sure I understand you here. How is the late-bound default "stored" when one side of a ternary is "not stored"?

This seems like dishonest argument. I'm not even sure what point you think it is making.

My point was a response to a claim that a late-bound default fundamentally has to be stored somewhere, yet half of a ternary conditional isn't. My point was that there is no difference, and they are both code. Please examine the surrounding context and respond in more detail, rather than calling my argument "dishonest". ChrisA

On Fri, Dec 3, 2021 at 1:53 AM Steven D'Aprano <steve@pearwood.info> wrote:

On Thu, Dec 02, 2021 at 11:00:33PM +1100, Chris Angelico wrote:

...

On Thu, Dec 2, 2021 at 8:40 PM Steven D'Aprano <steve@pearwood.info> wrote:

...

Depending on the implementation, you *might* be able to inspect the function and see the default expression as some sort of callable function, or evaluatable code object. (That would be nice.)

Unfortunately not, since the default expression could refer to other parameters, or closure variables, or anything else from the context of the called function. So you won't be able to externally evaluate it.

Why not? Functions can do all those things: refer to other variables, or closures, or anything else. You can call functions. Are you sure that this limitation of the default expression is not just a limitation of your implementation?

def f(): a = 1 def f(b, c=>a+b): return c a = 2 return f If there were a function to represent the late-bound default value for c, what parameters should it accept? How would you externally evaluate this? And also: what do you gain by it being a function, other than a lot of unnecessary overhead? And it is potentially a LOT of unnecessary overhead. Consider this edge case: def f(a, b=>c:=len(a)): ... In what context should the name c be bound? If there's a function for the evaluation of b, then that implies making c a closure cell, just for the sake of that. Every reference to c anywhere in the function (not just where it's set to len(a), but anywhere in f()) has to dereference that. It's a massive amount of completely unnecessary overhead AND a difficult question of which parts belong in the closure and which parts belong as parameters, which means that this is nearly impossible to define usefully.

...

I'm still unsure whether this is a cool feature or an utter abomination:

...

...

...

def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f.__defaults_extra__ = ("n/a",) f(42) You passed x as 42 f() You didn't pass x

That is absolutely an abomination. If your implementation has the side-effect that setting a regular early-bound default to Ellipsis makes the parameter unable to retrieve the default, then the implementation is fatally broken.

It absolutely is not a feature.

That's not what the example shows. It shows that changing dunder attributes can do this. I'm not sure why you think that the implementation is as restricted as you imply. The assignment to __defaults_extra__ is kinda significant here :) ChrisA

On Fri, Dec 3, 2021 at 2:11 AM Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:

On 02/12/2021 14:47, Steven D'Aprano wrote:

...

...

...

...

...

def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f.__defaults_extra__ = ("n/a",) f(42) You passed x as 42 f() You didn't pass x That is absolutely an abomination. If your implementation has the side-effect that setting a regular early-bound default to Ellipsis makes

On Thu, Dec 02, 2021 at 11:00:33PM +1100, Chris Angelico wrote: I'm still unsure whether this is a cool feature or an utter abomination: the parameter unable to retrieve the default, then the implementation is fatally broken.

It absolutely is not a feature.

It's backward incompatible:

15:03:04 R:\>python Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.

...

...

...

def f(x=...): ... try: print("You passed x as", x) ... except UnboundLocalError: print("You didn't pass x") ... f() You passed x as Ellipsis

So I must agree with Steven that this should not be a feature.

Clearly I shouldn't post code examples without lots and lots of explanatory comments. https://www.python.org/dev/peps/pep-0671/#implementation-details # REDEFINE THE INTERPRETER'S UNDERSTANDING # OF THE LATE BOUND DEFAULTS FOR THE FUNCTION f.__defaults_extra__ = ("n/a",) There, now it's a bit clearer what's going on. ChrisA

On Wed, Dec 01, 2021 at 09:00:50PM -0800, abed...@gmail.com wrote:

Steven D'Aprano ""If param is missing **or None**, the default if blah..."

The bottom line is:

you *don't actually* want the parameter to default to the value of a sentinel.

Yes I do. I *do* want to be able to give a convenient sentinel value in order to explicitly tell the function "give me the default value".

you *have* to use that hack because you can't express what you want the default to actually be.

The point of having default values is so that the caller doesn't have to express what the default will actually be. If the caller has to express that value, it's not a default, it's a passed-in argument.

You're doing something misleading to work around a shortcoming of the language.

How is it misleading? The parameter is explicitly documented as taking None to have a certain effect. None is behaving here as a convenient, common special constant to trigger a certain behaviour, no different than passing (for example) buffering=-1 to open() to trigger a very complex set of behaviour. (With buffering=-1, the buffer depends on the platform details, and whether the file is binary, text, and whether or not it is a tty.) For historical reasons, probably related to C, the default value for buffering is -1. But it could have just as easily be None, or the string "default", or an enumeration, or some builtin constant. -- Steve

On Fri, Dec 3, 2021 at 4:22 AM Nicholas Cole <nicholas.cole@gmail.com> wrote:

There is nothing that this proposal makes possible that is not already possible with more explicit code.

It's worth noting that "explicit" does not mean "verbose". For instance, this is completely explicit about what it does: x += 1 It does not conceal what it's doing, yet it uses a very compact notation to say "augmented addition". The proposal in question uses an explicit symbol to indicate that the default should be late-bound. In contrast, a less explicit and much worse proposal might be: "If the argument default defines a mutable object, construct a new one every time", so "def f(x=1):" would be early bound and "def f(x=[]):" would be late-bound. This is implicit behaviour, since it's not stated in the code which one is which. ChrisA

On Fri, Dec 3, 2021 at 6:24 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:

What I'm saying is that I want that "thing" to exist. At the time the function is defined, I want there to be a Python object which represents the behavior to be activated at call time if the argument is not passed. In the current proposal there is no such "thing". The function just has behavior melded with its body that does stuff, but there is no addressable "thing" where you can say "if you call the function and the argument isn't passed were are going to take this default-object-whatchamacallit and 'use' it (in some defined way) to get the default value". This is what we already have for early-bound defaults in the function's `__defaults__` attribute.

What would you do with this object? Suppose you have this function: def f(lst=>[], n=>len(lst)): ... What can you usefully do with this hypothetical object representing the default for n? There is no "thing" and no way you could "use" that thing to predict the value of n without first knowing the full context of the function. With early-bound defaults, you can meaningfully inspect them and see what the value is going to be. With late-bound defaults, should the default for lst be an empty list? Should the default for n be 0? Neither is truly correct (but they're wrong in different ways). Do you actually need that object for anything, or is it simply for the sake of arbitrary consistency ("we have objects for early-bound defaults, why can't we have objects for late-bound defaults")? The value that will be assigned to the parameter does not exist until the function is actually called, and may depend on all kinds of things about the function and its context. Until then, there is no value, no object, that can represent it. I put to you the same question I put to David Mertz: how is a late-bound default different from half of a conditional expression? def f(*args): lst = args[0] if len(args) > 0 else [] n = args[1] if len(args) > 1 else len(lst) ... Do the expressions "[]" and "len(lst)" have to have object representations in this form? If not, why should they need object representations when written in the more natural way as "lst=>[]"? ChrisA

On 2021-12-02 00:31, Chris Angelico wrote:

Here's how a ternary if looks:

...

...

...

...

...

>def f(n): ... return 0 if n == 0 else 42/n ... >dis.dis(f) 2 0 LOAD_FAST 0 (n) 2 LOAD_CONST 1 (0) 4 COMPARE_OP 2 (==) 6 POP_JUMP_IF_FALSE 6 (to 12) 8 LOAD_CONST 1 (0) 10 RETURN_VALUE >> 12 LOAD_CONST 2 (42) 14 LOAD_FAST 0 (n) 16 BINARY_TRUE_DIVIDE 18 RETURN_VALUE

The "42/n" part is stored in f.__code__.co_code as the part that says "LOAD_CONST 42, LOAD_FAST n, BINARY_TRUE_DIVIDE". It's not an object. It's just code - three instructions.

Here's how (in the reference implementation - everything is subject to change) a late-bound default looks:

...

...

...

...

...

>def f(x=>[]): print(x) ... >dis.dis(f) 1 0 QUERY_FAST 0 (x) 2 POP_JUMP_IF_TRUE 4 (to 8) 4 BUILD_LIST 0 6 STORE_FAST 0 (x) >> 8 LOAD_GLOBAL 0 (print) 10 LOAD_FAST 0 (x) 12 CALL_FUNCTION 1 14 POP_TOP 16 LOAD_CONST 0 (None) 18 RETURN_VALUE

The "=>[]" part is stored in f.__code__.co_code as the part that says "QUERY_FAST x, and if false, BUILD_LIST, STORE_FAST x". It's not an object. It's four instructions in the bytecode.

In both cases, no part of the expression is ever re-executed. I'm not understanding the distinction here. Can you explain further please?

Your explanation exactly shows how it IS re-executed. I'm not totally clear on this disassembly since this is new behavior, but if I understand right, BUILD_LIST is re-executing the expression `[]` and STORE_FAST is re-assigning it to x. The expression `[]` is syntactically present in the function definition but its execution has been shoved into the function body where it may be re-executed many times (any time the function is called without passing a value). What do you mean when you say it is not re-executed? Is it not the case that `[]` is syntactically present in the `def` line (which is executed only once, and whose bytecode is not shown) yet its implementation (BUILD_LIST) is in the function body, which may be executed many times? How is the BUILD_LIST opcode there not being re-executed on later calls of the function? Perhaps what you are saying is that what is stored is not the literal string "[]" but bytecode that implements it? That distinction is meaningless to me. The point is that you wrote `[]` once, in a line that is executed once (the function definition itself), but the `[]` is executed many times, separately from the function definition. Another way to put it is, again, the examples are not parallel. In your first example the ternary expression is (syntactically) part of the function BODY, so of course it appears in the disassembly. In your second example, the late-bound default is not in the body, it is in the signature. The disassembly is only showing the bytecode of the function body, but the late-bound default is syntactically enmeshed with the function DEFINITION. So I don't want the late-bound default code to be re-executed unless the function is re-defined. Or, if you're going to store that `x = []`, I don't want it to be "stored" by just shoving it in the function body, I want it to be some kind of separate object. Here's an example that may make my point clearer. some_function(x+1, lambda x: x+2) This is our best current approximation to some kind of "late evaluation". The first argument, `x+1`, is evaluated before the function is called, and the function gets only the result. The second argument is also of course evaluated before the function is called, but what is evaluated is a lambda; the `x+2` is not evaluated. But the idea of "evaluate x+2 later" is encapsulated in a function object. Without knowing anything about `some_function`, I still know that there is no way it can ever evaluate `x+2` without going through the interface of that function object. There is no way to pass, as the second argument to `some_function` some kind of amorphous "directive" that says "evaluate x+2 later" without wrapping that directive up into some kind of Python object. Perhaps the fundamental point that I feel you're missing about my position is that a ternary expression does not have a "definition" and a "body" that are executed at separate times. There is just the whole ternary expression and it is evaluated all at once. Thus there can be no parallel with functions, which do have a separation between definition time and call time. Indeed, it's because these are separate that late vs. early binding of defaults is even a meaningful concept for functions (but not for ternary expressions). So what I am saying is if I see a line like this: def f(a=x+1, b@=x+2): The x+2 is syntactically embedded within the `def` line, that is, the function definition (not the body). Thus there are only two kinds of semantics that are going to make me happy: 1) That `x+2`(or any bytecode derived from it, etc.) will never be re-executed unless the program execution again reaches the line with the `def f` (which is what we have with early-bound defaults) 2) A Python object is created that encapsulates the expression `x+2` somehow and defines what it means to "evaluate it in a context" (e.g., by referencing local variables in the scope where it is evaluated) Maybe another way to say this is just "I want any kind of late-bound default to really be an early-bound default whose value is some object that provides a way to evaluate it later". (I'm trying to think of different ways to say this because it seems what I'm saying is not clear to you. :-) -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On 12/2/2021 2:21 PM, Brendan Barnwell wrote:

On 2021-12-02 01:35, Steven D'Aprano wrote:

...

...

...

...

...

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

    No.  As I mentioned in the earlier thread, I don't support any     proposal in which an argument can "have a default" but that default is not a first-class Python object of some sort. I don't understand this criticism.

Of course the default value will be a first-class Python object of some sort.*Every*  value in Python is a first-class object. There are no machine values or unboxed values, and this proposal will not change that.

All that this proposal changes is*when*  and*how often*  the default will be evaluated, not the nature of the value.

    As has happened often in these threads, it seems different people mean different things by "default value".

    What you are calling "the default value" is "a thing that is used at call time if no value is passed for the argument".  What I am calling "the default value" is "a thing that is noted at definition time to be used later if no value is passed for the argument".

    What I'm saying is that I want that "thing" to exist.  At the time the function is defined, I want there to be a Python object which represents the behavior to be activated at call time if the argument is not passed.  In the current proposal there is no such "thing".  The function just has behavior melded with its body that does stuff, but there is no addressable "thing" where you can say "if you call the function and the argument isn't passed were are going to take this default-object-whatchamacallit and 'use' it (in some defined way) to get the default value".  This is what we already have for early-bound defaults in the function's `__defaults__` attribute.

I also have this objection to the proposal (among other concerns). Say I have a function with an early-bound default. I can inspect it and I can change it. One reason to inspect it is so that I can call the function with its default values. This is a form of wrapping the function. I realize "just don't pass that argument when you call the function" will be the response, but I think in good faith you'd have to admit this is more difficult than just passing some default value to a function call. As far as changing the defaults, consider:

...

...

def f(x=3): return x ... f() 3 f.__defaults__=(42,) f() 42

The current PEP design does not provide for this functionality for late-bound defaults. I realize the response will be that code shouldn't need to do these things, but I do not think we should be adding features to python that limit what introspections and runtime modifications user code can do. A classic example of this is PEP 362 function signature objects. I don't think we should be adding parameter types that cannot be represented in a Signature, although of course a Signature might need to be extended to support new features. Signature objects were added for a reason (see the PEP), and I don't think we should just say "well, that's not important for this new feature". Also note that over time we've removed restrictions on Signatures (see, for example, Argument Clinic). So I don't think adding restrictions is the direction we want to go in. Eric

Steven D'Aprano "> ""If param is missing **or None**, the default if blah..."

The bottom line is:

you *don't actually* want the parameter to default to the value of a sentinel.

Yes I do. I *do* want to be able to give a convenient sentinel value in order to explicitly tell the function "give me the default value"." If you want the parameter to default to an empty list, it's categorically more explicit to have the parameter default to an empty list. It's a hack to say "I wan't this to default to an empty list" then have it actually default to a sentinel value that you'll replace with an empty list later on. That's the opposite of explicit. If you actually wanted to have the parameter default to a sentinel value, why would you then overwrite that value? This isn't a very complicated point. Steven D'Aprano "> you *have* to use that hack because you can't express what you want the

default to actually be.

The point of having default values is so that the caller doesn't have to express what the default will actually be. If the caller has to express that value, it's not a default, it's a passed-in argument." None of my comment had anything to do with the caller's perspective. I know how optional parameters work. I'm specifically talking about the programmer's ability to map their intent to code and the ability of the would-be reader of the code (NOT CALLER) to decipher that intent. If my intent is that I want a parameter to default to an empty list, I wouldn't assign it to an object called "monkey_pants" by default then add code to the body of the function to check if it's "monkey_pants". "But wait", I hear you say, "assigning my optional parameter to 'monkey_pants' isn't a hack because I know will *never* in any conceivable circumstances become a valid argument value!" "assigning a value to 'monkey_pants' is how I explicitly communicate tell the function "give me the default value"!" Those are both quotes from you with 'None' replaced with 'monkey_pants'. Steven D'Aprano "How is it misleading?" Your way of having a parameter default to an empty list (or whatever) is by having the parameter default to None. That's how it's misleading. I know how the hack works. I know why the hack works. I know how default parameters work. You can spare trying to explain all that again. What you can't explain is why it's more explicit to have a parameter default to a sentinel value when the actual intent is to have it default to a mutable value. If it were more explicit, then why not replace `def func(x=0)` with `def func(x=SENTINEL): if x is SENTINEL: x = 0`? I'm at a loss for ways to explain my position any clear than that. On Thursday, December 2, 2021 at 9:52:28 AM UTC-6 Steven D'Aprano wrote:

On Wed, Dec 01, 2021 at 09:00:50PM -0800, abed...@gmail.com wrote:

...

Steven D'Aprano ""If param is missing **or None**, the default if blah..."

...

The bottom line is:

you *don't actually* want the parameter to default to the value of a sentinel.

Yes I do. I *do* want to be able to give a convenient sentinel value in order to explicitly tell the function "give me the default value".

...

you *have* to use that hack because you can't express what you want the default to actually be.

The point of having default values is so that the caller doesn't have to express what the default will actually be. If the caller has to express that value, it's not a default, it's a passed-in argument.

...

You're doing something misleading to work around a shortcoming of the language.

How is it misleading? The parameter is explicitly documented as taking None to have a certain effect. None is behaving here as a convenient, common special constant to trigger a certain behaviour, no different than passing (for example) buffering=-1 to open() to trigger a very complex set of behaviour.

(With buffering=-1, the buffer depends on the platform details, and whether the file is binary, text, and whether or not it is a tty.)

For historical reasons, probably related to C, the default value for buffering is -1. But it could have just as easily be None, or the string "default", or an enumeration, or some builtin constant.

-- Steve _______________________________________________ Python-ideas mailing list -- python...@python.org To unsubscribe send an email to python-id...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python...@python.org/message/QGPPY4JLA... <https://mail.python.org/archives/list/python-ideas@python.org/message/QGPPY4...> Code of Conduct: http://python.org/psf/codeofconduct/

"The caller cannot explicitly ask for default behaviour except by omitting the parameter. This can be very annoying to set up when the parameter values are provided from other places, e.g., if need_x: # do lots of stuff x = whatever else: # do more x = None f(x=x, ...) # easy, but with this PEP, you would have to find a way to remove x from the parameter list. Typically, removing a parameter from a dynamically-created parameter list is hard." If removing a parameter from a dynamically-created parameter list is hard, maybe that's a tools problem. Maybe we should consider crafting better tools for such tasks. It also sounds like a lot of people believe the inspectability of callables that use late-bound defaults is a point of contention. Maybe there should be more inspection hooks than described in the implementation details section of the PEP <https://www.python.org/dev/peps/pep-0671/#id8>? On Wednesday, December 1, 2021 at 3:56:35 AM UTC-6 miste...@gmail.com wrote:

On Wednesday, December 1, 2021 at 1:18:33 AM UTC-5 Chris Angelico wrote:

...

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.

*PEP 671: Syntax for late-bound function argument defaults*

Questions, for you all:

1) If this feature existed in Python 3.11 exactly as described, would you use it?

No, I would avoid for the sake of readers. Anyway, it can't be used in any public projects until thy drop support for 3.10, which is many years off.

Also, I think this question is quite the biased sample on python-ideas. Please consider asking this to less advanced python users, e.g., reddit.com/r/python or learnpython.

...

2) Independently: Is the syntactic distinction between "=" and "=>" a cognitive burden?

(It's absolutely valid to say "yes" and "yes", and feel free to say which of those pulls is the stronger one.)

Yes.

...

3) If "yes" to question 1, would you use it for any/all of (a) mutable defaults, (b) referencing things that might have changed, (c) referencing other arguments, (d) something else?

4) If "no" to question 1, is there some other spelling or other small change that WOULD mean you would use it? (Some examples in the PEP.)

No.

...

5) Do you know how to compile CPython from source, and would you be willing to try this out? Please? :)

I'd love to hear, also, from anyone's friends/family who know a bit of Python but haven't been involved in this discussion. If late-bound defaults "just make sense" to people, that would be highly informative.

Any and all comments welcomed. I mean, this is python-ideas after all... bikeshedding is what we do best!

This PEP has a lot of interesting ideas. I still think that none-aware operators (PEP 505) are an easier, more readable general solution to binding calculated default values to arguments succinctly. I think the problems with this idea include: * The caller cannot explicitly ask for default behaviour except by omitting the parameter. This can be very annoying to set up when the parameter values are provided from other places, e.g., if need_x: # do lots of stuff x = whatever else: # do more x = None f(x=x, ...) # easy, but with this PEP, you would have to find a way to remove x from the parameter list. Typically, removing a parameter from a dynamically-created parameter list is hard.

* The function code becomes unreadable if the parameter-setting code is long. Having a long piece of code inside the function after "if parameter is None" is just fine. Having none-aware operators would make such code more succinct.

* People nearly always avoid writing code in the parameter defaults themselves, and this new practice adds a lot of cognitive load. E.g., people rarely write: def f(x: int = 1+g()) -> None: ... Parameter lists are already busy enough with parameter names, annotations, and defaults. We don't need to encourage this practice.

In short, I think this is a creative idea, a great exploration. While optional parameters are common, and some of them have defaults that are calculated inside the function, my feeling is that people will continue to set their values inside the function.

Best,

Neil

On Fri, Dec 3, 2021 at 7:54 AM Eric V. Smith <eric@trueblade.com> wrote:

On 12/2/2021 2:21 PM, Brendan Barnwell wrote:

...

On 2021-12-02 01:35, Steven D'Aprano wrote:

...

...

...

...

>4) If "no" to question 1, is there some other spelling or other small >change that WOULD mean you would use it? (Some examples in the PEP.)

No. As I mentioned in the earlier thread, I don't support any proposal in which an argument can "have a default" but that default is not a first-class Python object of some sort. I don't understand this criticism.

Of course the default value will be a first-class Python object of some sort.*Every* value in Python is a first-class object. There are no machine values or unboxed values, and this proposal will not change that.

All that this proposal changes is*when* and*how often* the default will be evaluated, not the nature of the value.

As has happened often in these threads, it seems different people mean different things by "default value".

What you are calling "the default value" is "a thing that is used at call time if no value is passed for the argument". What I am calling "the default value" is "a thing that is noted at definition time to be used later if no value is passed for the argument".

What I'm saying is that I want that "thing" to exist. At the time the function is defined, I want there to be a Python object which represents the behavior to be activated at call time if the argument is not passed. In the current proposal there is no such "thing". The function just has behavior melded with its body that does stuff, but there is no addressable "thing" where you can say "if you call the function and the argument isn't passed were are going to take this default-object-whatchamacallit and 'use' it (in some defined way) to get the default value". This is what we already have for early-bound defaults in the function's `__defaults__` attribute.

I also have this objection to the proposal (among other concerns).

Say I have a function with an early-bound default. I can inspect it and I can change it. One reason to inspect it is so that I can call the function with its default values. This is a form of wrapping the function. I realize "just don't pass that argument when you call the function" will be the response, but I think in good faith you'd have to admit this is more difficult than just passing some default value to a function call.

1) I want to call this function 2) I may want to not pass this argument 3) Ah, perfect! I will pass this argument with a value of somemod._SENTINEL. Or alternatively: 1) I want to call this function. 2) Prepare a dictionary of arguments. Leave out what I don't want. 3) If I want to pass this argument, add it to the dictionary. This way doesn't require reaching into the function's private information to use a sentinel. Yes, it may be a tad more difficult (though not VERY much), but you're also avoiding binding yourself to what might be an implementation detail.

As far as changing the defaults, consider:

...

...

...

def f(x=3): return x ... f() 3 f.__defaults__=(42,) f() 42

The current PEP design does not provide for this functionality for late-bound defaults.

Remember, though: the true comparison should be something like this: _SENTINEL = object() def f(x=_SENTINEL): if x is _SENTINEL: x = [] return x Can you change that from a new empty list to something else? No. All you can do, by mutating the function's dunders, is change the sentinel, which is actually irrelevant to the function's true behaviour. You cannot change the true default. Consider also this form: default_timeout = 500 def connect(s, timeout=default_timeout): ... def read(s, timeout=default_timeout): ... def write(s, msg, timeout=default_timeout): ... You can now, if you go to some effort, replace the default in every function. Or you can do this, and not go to any effort at all: def read(s, timeout=>default_timeout): ... The true default is now exactly what the function signature says. And if you really want to, you CAN change read.__defaults__ to have an actual early-bound default, which means it will then never check the default timeout. Introspection is no worse in this way than writing out the code longhand. It is significantly better, because even though you can't change it from a latebound default_timeout to a latebound read_timeout, you can at least see the value with external tools. You can't see that if the default is replaced in the body of the function.

I realize the response will be that code shouldn't need to do these things, but I do not think we should be adding features to python that limit what introspections and runtime modifications user code can do.

The response is more that the code CAN'T do these things, by definition. To the extent that you already can, you still can. To the extent that you should be able to, you are still able to. (And more. There are things you're capable of with PEP 671 that you definitely shouldn't do in normal code.)

A classic example of this is PEP 362 function signature objects. I don't think we should be adding parameter types that cannot be represented in a Signature, although of course a Signature might need to be extended to support new features. Signature objects were added for a reason (see the PEP), and I don't think we should just say "well, that's not important for this new feature". Also note that over time we've removed restrictions on Signatures (see, for example, Argument Clinic). So I don't think adding restrictions is the direction we want to go in.

Same again. If you consider the equivalent to be a line of code in the function body, then the signature has become MASSIVELY more useful. Instead of simply seeing "x=<object object at 0x7fba1b318690>", you can see "x=>[]" and be able to see what the value would be. It's primarily human-readable (although you could eval it), but that's still a lot better than seeing a meaningless sentinel. And yes, you can make help() more readable by using a nicely named sentinel, but then you have to go to a lot more effort in your code, worry about pickleability, etc, etc. Using a late-bound default lets you see the true default, not a sentinel. ChrisA

On Fri, Dec 3, 2021 at 9:26 AM Brendan Barnwell <brenbarn@brenbarn.net> wrote:

On 2021-12-02 00:31, Chris Angelico wrote:

...

Here's how a ternary if looks:

...

...

...

...

>>def f(n): ... return 0 if n == 0 else 42/n ... >>dis.dis(f) 2 0 LOAD_FAST 0 (n) 2 LOAD_CONST 1 (0) 4 COMPARE_OP 2 (==) 6 POP_JUMP_IF_FALSE 6 (to 12) 8 LOAD_CONST 1 (0) 10 RETURN_VALUE >> 12 LOAD_CONST 2 (42) 14 LOAD_FAST 0 (n) 16 BINARY_TRUE_DIVIDE 18 RETURN_VALUE

The "42/n" part is stored in f.__code__.co_code as the part that says "LOAD_CONST 42, LOAD_FAST n, BINARY_TRUE_DIVIDE". It's not an object. It's just code - three instructions.

Here's how (in the reference implementation - everything is subject to change) a late-bound default looks:

...

...

...

...

>>def f(x=>[]): print(x) ... >>dis.dis(f) 1 0 QUERY_FAST 0 (x) 2 POP_JUMP_IF_TRUE 4 (to 8) 4 BUILD_LIST 0 6 STORE_FAST 0 (x) >> 8 LOAD_GLOBAL 0 (print) 10 LOAD_FAST 0 (x) 12 CALL_FUNCTION 1 14 POP_TOP 16 LOAD_CONST 0 (None) 18 RETURN_VALUE

The "=>[]" part is stored in f.__code__.co_code as the part that says "QUERY_FAST x, and if false, BUILD_LIST, STORE_FAST x". It's not an object. It's four instructions in the bytecode.

In both cases, no part of the expression is ever re-executed. I'm not understanding the distinction here. Can you explain further please?

Your explanation exactly shows how it IS re-executed. I'm not totally clear on this disassembly since this is new behavior, but if I understand right, BUILD_LIST is re-executing the expression `[]` and STORE_FAST is re-assigning it to x. The expression `[]` is syntactically present in the function definition but its execution has been shoved into the function body where it may be re-executed many times (any time the function is called without passing a value).

Ah, I think I get you. The problem is that code is in the def line but is only executed when the function is called, is that correct? Because the code would be "re-executed" just as much if it were written in the function body. It's executed (at most) once for each call to the function, just like the ternary's side is. I suppose that's a consideration, but it's not nearly as strong in practice as you might think. A lot of people aren't even aware of the difference between compilation time and definition time (even people on this list have made that mistake). Function default args are executed when the function is defined, not when it's called, and that's something that changes with this proposal; but there are many other subtleties to execution order and timing that don't really matter in practice. Perhaps the key point here is to consider function decorators. We could avoid them altogether: def f(): @deco def g(x): ... def h(x): ... h = deco(h) But as well as having the name replication problem, this buries important information down in the body of the surrounding code, rather than putting it at the signature of g/h where it belongs. Even though, semantically, this is actually part of the body of f, we want to be able to read it as part of the signature of g. Logically and conceptually, it is part of the signature. Now compare these two: def f2(): _SENTINEL = object() def g(x=_SENTINEL): if x is _SENTINEL: x = [] ... def h(x=>[]): ... Which one has its signature where its signature belongs? Yes, semantically, the construction of the empty list happens at function call time, not at definition time. But what you're saying is: if there are no args passed, behave as if a new empty list was passed. That's part of the signature. In neither case will you find an object representing the expression [] in the function's signature, because that's not an object, it's an instruction to build an empty list. In the case of g, you can find a meaningless and useless object stashed away in __defaults__, but that doesn't tell you anything about the true behaviour of the function. At least in the case of h, you can find the descriptive string "[]" stashed there, which can tell a human what's happening. ChrisA

On Fri, Dec 3, 2021 at 12:48 PM Steven D'Aprano <steve@pearwood.info> wrote:

On Fri, Dec 03, 2021 at 02:10:12AM +1100, Chris Angelico wrote:

...

...

...

Unfortunately not, since the default expression could refer to other parameters, or closure variables, or anything else from the context of the called function. So you won't be able to externally evaluate it.

Why not? Functions can do all those things: refer to other variables, or closures, or anything else. You can call functions. Are you sure that this limitation of the default expression is not just a limitation of your implementation?

def f(): a = 1 def f(b, c=>a+b): return c a = 2 return f

If there were a function to represent the late-bound default value for c, what parameters should it accept?

I'm not saying that it *must* be a function. It could be a bare code object, that is `eval()`ed. Or something completely new. Dunno.

But you're saying something is impossible, and that seems implausible to me, because things that seems *very similar* are totally possible.

...

How would you externally evaluate this?

inner = f() # Get the inner function. default = inner.__code__.__late_defaults__.wibble[1] # whatever try: value = default() except NameError: # Well, what did you expect to happen? value = eval(default.__code__, globals(), {'a': 101, 'b': 202})

Or something. The point is, rather than dismissing the possibility outright, this should be something we discuss, and carefully consider, before the PEP is complete.

How, with external calling, are you going to know which name references to look up, and where to get their values from? This is already a virtually impossible task in Python, which is why f-strings cannot be implemented as str.format(**something) for any known meaning of "something". You cannot get your *current* variable set, much less the available variables in some other context.

...

And it is potentially a LOT of unnecessary overhead. Consider this edge case:

def f(a, b=>c:=len(a)): ...

In what context should the name c be bound?

With late-bound defaults, the expression is evaluated at function call time, in the scope of f()'s locals. So c would be a local.

In other words, if you're trying to evaluate b's default externally, you have to set c in a context that doesn't even exist yet. Is that correct?

The third obvious answer is that if either the decision or the implementation is really too hard, then make it a syntax error for now, and revisit it in the future.

Everything's perfectly well defined, save that you can't externally evaluate the defaults. You can quite happily use the walrus in a late-bound default, and it'll bind to the function's locals. (Though I don't recommend it. I think that that's going to make for less-readable code than alternatives. Still, it's perfectly legal and well-defined.)

...

It's a massive amount of completely unnecessary overhead AND a difficult question of which parts belong in the closure and which parts belong as parameters, which means that this is nearly impossible to define usefully.

I've given you two useful definitions.

Its not clear what overhead you are worried about.

Accessing variables in cells is almost as fast as accessing locals, but even if they were as slow as globals, premature optimization is the root of all evil. Globals are fast enough.

Or are you worried about the memory overhead of the closures? The extra cost of fetching and calling the functions when evaluating the defaults? None of these things seem to be good reasons to dismiss the idea that default expressions should be independent of the function body.

The problem is that you're trying to reference cell variables in a closure that might not even exist yet, so *just in case* you might have nonlocals, you have to construct a closure... or find an existing one. Ill-defined and inefficient.

...

That's not what the example shows. It shows that changing dunder attributes can do this. I'm not sure why you think that the implementation is as restricted as you imply. The assignment to __defaults_extra__ is kinda significant here :)

Ah, well that is not so clear to people who aren't as immersed in the implementation as you :-)

Maybe, but I did think that the fact that I was assigning to a dunder should be obvious to people who are reading here :)

Messing about with function dunders can do weird shit:

...

...

...

def func(a=1, b=2): ... return a+b ... func.__defaults__ = (1, 2, 3, 4, 5) func() 9

I wouldn't worry about it.

Good, so, not an abomination. (In your opinion. Others are, of course, free to abominate as they please.) ChrisA

On Fri, Dec 03, 2021 at 01:08:50PM +1100, Chris Angelico wrote:

How, with external calling, are you going to know which name references to look up, and where to get their values from?

Isn't the source code available as a string? I will see the names in the expression. Or I will disassemble the code object and inspect its byte-code and/or AST. Or read the documentation. Or maybe the code object will expose those names in a dunder and I can just look at that.

This is already a virtually impossible task in Python, which is why f-strings cannot be implemented as str.format(**something) for any known meaning of "something".

f-strings cannot be implemented as str.format because str.format doesn't evaluate arbitrary expressions.

...

...

f'{1000+len(repr(None))}' '1004'

You cannot get your *current* variable set, much less the available variables in some other context.

Why not? The default expression object can record nonlocal variables in a closure, and you can provide your own globals and/or locals. You're saying "can't, impossible" but I could take your function object, extract the code object, disassemble the code to AST or byte-code, extract the portions that represent the default expression, and evaluate that. So not *impossible*, just bloody inconvenient and annoying. (Well, when I say *I could*, I mean *somebody with mad leet AST or byte- code hacking skillz. Not actually me.)

...

...

And it is potentially a LOT of unnecessary overhead. Consider this edge case:

def f(a, b=>c:=len(a)): ...

In what context should the name c be bound?

With late-bound defaults, the expression is evaluated at function call time, in the scope of f()'s locals. So c would be a local.

In other words, if you're trying to evaluate b's default externally, you have to set c in a context that doesn't even exist yet. Is that correct?

Yes? You seem to be saying that as if it were some fatal flaw in my proposal. mylocals = {} There. Now the context exists. result = eval(default_expression.__code__, globals(), mylocals())

...

The third obvious answer is that if either the decision or the implementation is really too hard, then make it a syntax error for now, and revisit it in the future.

Everything's perfectly well defined, save that you can't externally evaluate the defaults.

You keep saying that, but your arguments are not convincing.

You can quite happily use the walrus in a late-bound default, and it'll bind to the function's locals.

Cool :-)

The problem is that you're trying to reference cell variables in a closure that might not even exist yet, so *just in case* you might have nonlocals, you have to construct a closure... or find an existing one. Ill-defined and inefficient.

That doesn't make sense to me. When you compile the default expression, you know what variable names are assigned to with a walrus expression (if any), you know what names exist in the surrounding nonlocal scope (if any), you know what names are parameters. If the default expression doesn't use walrus, doesn't refer to any nonlocal names, and doesn't refer to any other parameters, why would you construct a closure "just in case"? What would you put in it? -- Steve

On 2021-12-02 15:40, Chris Angelico wrote:

Actually, no. I want to put the default arguments into the signature, and the body in the body. The distinction currently has a technical restriction that means that, in certain circumstances, what belongs in the signature has to be hacked into the body. I'm trying to make it so that those can be put where they belong.

Chris, I know this is probably not your intention, but I feel the discussion is continually muddle by you just saying "default arguments" as if everyone agrees on what those are and the issue is just where to put them. But clearly that is not the case. You seem to take it for granted that "len(x) evaluated when the function is called" "is" a "default argument" in the same sense that an early-bound default like the number 3 is. I do not agree, and it's pretty clear David Mertz does not agree, and I think there are others here who also do not agree. It's not as is there is some pre-existing notion of "default argument" in Python and you are just proposing to add an implementation of it that was left out due to some kind of oversight. Your proposal is CHANGING the idea of what a default argument is. I get that it's natural to refer to your new proposal as "default arguments" but I've now seen a number of messages here where you say "but no we should do X because this is a default argument", taking for granted that what you're talking about is already agreed to be a default argument. (No doubt I and many others are also guilty of similar missteps by saying "default argument" as a shorthand for something or other, and I'll try to be careful about it myself.) By my definition as of now, a default argument has to be an object. Thus what your proposal envisions are not default arguments. You can't just say "I want to do this with default arguments". You need to provide an argument for why we should even consider these to be default arguments at all. Perhaps a way of stating this without reference to arguments is this: you want to put code in the function signature but not have it executed until the function is called. I do not agree with that choice. The function signature and the function body are different syntactic environments with different semantics. Everything in the function signature should be executed when the function is defined (although of course that execution can result in an object which contains deferred behavior itself, like if we pass a function as an argument to another). Only the function body should be executed when the function is called. If we want to provide some way to augment the function body to assign values to missing arguments, I wouldn't rule that out completely, but in my view the function signature is not an available space for that. The function signature is ONLY for things that happen when the function is defined. It is too confusing to have the function signature mix definition-time and call-time behavior. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On Fri, Dec 3, 2021 at 2:24 PM Steven D'Aprano <steve@pearwood.info> wrote:

On Fri, Dec 03, 2021 at 01:08:50PM +1100, Chris Angelico wrote:

...

How, with external calling, are you going to know which name references to look up, and where to get their values from?

Isn't the source code available as a string? I will see the names in the expression.

Or I will disassemble the code object and inspect its byte-code and/or AST. Or read the documentation. Or maybe the code object will expose those names in a dunder and I can just look at that.

...

This is already a virtually impossible task in Python, which is why f-strings cannot be implemented as str.format(**something) for any known meaning of "something".

f-strings cannot be implemented as str.format because str.format doesn't evaluate arbitrary expressions.

...

...

...

f'{1000+len(repr(None))}' '1004'

...

You cannot get your *current* variable set, much less the available variables in some other context.

Why not? The default expression object can record nonlocal variables in a closure, and you can provide your own globals and/or locals.

You're saying "can't, impossible" but I could take your function object, extract the code object, disassemble the code to AST or byte-code, extract the portions that represent the default expression, and evaluate that. So not *impossible*, just bloody inconvenient and annoying.

(Well, when I say *I could*, I mean *somebody with mad leet AST or byte- code hacking skillz. Not actually me.)

When f-strings were first proposed, one of the counter-arguments was basically "just use globals()/locals()". https://www.python.org/dev/peps/pep-0498/#no-use-of-globals-or-locals But you cannot, in a fully general way, solve the problem that way. It is *impossible*. Not just difficult. Impossible. You cannot, in Python as it currently is, look up arbitrary names in any reliable way. This is especially true now that we have assignment expressions, such that evaluating the function default could create a new local or nonlocal name. Not just inconvenient and annoying. So impossible that a str method could not be written to do it. ChrisA

"You've gone from talking about *my* intentions, which you got wrong, to an argument about which is "more explicit"." No. You were the one who brought up explicitness by claiming that using a sentinel value was a way to "explicitly tell the function "give me the default value" (whatever the heck that means). Now that I've deconstructed that argument, you're moving the goal post from "my way is more explicit" to "there is such a thing as *too* explicit, don't ya know?!" and blaming me for shifting the focus to "which is more explicit"... cool. "In the simple example of the default being a new empty list: # Using Chris' preferred syntax, not mine def func(arg=>[]): then I agree" Which is, what? 99% of the use cases? great! "But let's talk about more complex examples: def open(filename, mode='r', buffering=-1): ... How would you make that "more explicit"? def open(filename, mode='r', buffering=>( 1 if 'b' not in mode and os.fdopen(os.open(filename)).isatty() else _guess_system_blocksize() or io.DEFAULT_BUFFER_SIZE ) ): ... Too much information! We're now drowning in explicitness." Then just use the sentinel hack, or write a helper function that has a nice name that says what the default will be and use that. I'm positive there will always be corner cases where logic that's concerned with checking and setting the right values for parameters before the actual business logic starts. "Now how do I, the caller, *explicitly* indicate that I want to use that system-dependent value? I just want a nice, simple value I can explicitly pass as buffering to say "just use the default". with open('myfile.txt', buffering= what? ) as f: I can't. So in this case, your agonisingly explicit default expression forces the caller to be *less* explicit when they call the function." This seems like an exceedingly minor complaint. The obvious answer is: you invoke the default behaviour by not passing the argument. And yes, I know that's less explicit, but that's kinda the whole point of defaults to begin with. It's behavior when the user decides not to specify their own. Nobody's asking "what do I pass to logging.config.fileConfig to explicitly invoke the logging module's default behavior?!" The answer is simply: don't configure logging. It's not rocket science. This want to explicitly pass something to invoke default behavior brings this old Raymond Hetinger talk <https://www.youtube.com/watch?v=OSGv2VnC0go&t=1870s> to mind where he suggests that sometimes passing positional arguments by name can help make your code more readable. I wholeheartedly agree with the practice discussed in Hettinger's talk, but I think your example is completely different. When you see `twitter_search('@obama', False, 20, True)` that naturally invokes the question "what the heck are those arguments??" When you see `with open('myfile.txt') as f` it doesn't raise the question "what is the buffering argument?" I don't know if that's why you think the ability to explicitly invoke default behavior is so important, but I thought I'd address it anyway. "Can we agree that, like salt, sugar, and bike-shedding, sometimes you can have *too much* explicitness in code?" Surely *you* would never agree that there's such a thing as *too much* bike-shedding! Isn't that your raison d'être?! I never said one had to be a Nazi about stamping out every use of the sentinel... er... idiom (if you prefer it to hack). Sure, use it in those rare corner cases where expressing a late-bound default would be so very burdensome. The fact that there are such cases isn't a very good argument against this proposal. It's more like "HA! You can't fix *everything* can you?!". Great. You got me, Steve. *slow claps* "The caller's perspective is important. The caller has to read the function signature (either in the documentation, or if they use `help(function)` in the interpreter, or when their IDE offers up tooltips or argument completion). The caller has to actually use the function. We should consider their perspective." I never said we shouldn't. That's just not what I was talking about at that time and you were misinterpreting my argument. "Nah, I'm kidding of course. But the point I am making is that you are mocking the concept by using a totally silly example. "monkey_pants" indeed, ha ha how very droll. But if you used a more meaningful symbol, like "default", or some enumeration with a meaningful name, or a value which is a standard idiom such as -1, then the idea isn't so ridiculous. And None is one such standard idiom." You missed my point entirely. The point is that the reasons you were giving for sentinels both being explicit and not being a hack don't hold water. Namely: The fact that it's guaranteed to work doesn't make it not a hack, and your inexplicable reasoning that defaulting to a sentinel instead of the actual default value that parameter is supposed to take after all is said and done is somehow more explicit than defaulting to the default value in the first place. Just because it's an idiom, doesn't mean it's actually good. I think the `if __name__ == "__main__":` idiom is a horrible, round-about way of saying "if this file is being run as a script". You might look at that and say "What do you mean? That's exactly what it says!" because you've seen the idiom so many times, just like how you think "x=None" explicitly says "give x the default value", but if you look closely `if __name__ == "__main__":` *doesn't* actually say anything like "if this file is being run as a script", it says a bunch of esoteric and confusing garbage that you've learned to read as "if this file is being run as a script". "> Steven D'Aprano

"How is it misleading?"

Your way of having a parameter default to an empty list (or whatever) is by having the parameter default to None. That's how its misleading.

All you have done is repeat that it is misleading. I'm still no clearer why you think it is misleading" Seriously? If I said "I'm going to pat my head" then proceeded to rub my tummy, would you not think that's a bit misleading? Again, if you look closely, I *didn't* simply repeat that it's misleading. I pointed out the contradiction between intent and code. On Thu, Dec 2, 2021 at 8:57 PM Steven D'Aprano <steve@pearwood.info> wrote:

On Thu, Dec 02, 2021 at 01:20:27PM -0800, abed...@gmail.com wrote:

...

If you want the parameter to default to an empty list, it's categorically more explicit to have the parameter default to an empty list.

You've gone from talking about *my* intentions, which you got wrong, to an argument about which is "more explicit".

In the simple example of the default being a new empty list:

# Using Chris' preferred syntax, not mine def func(arg=>[]):

then I agree: once you know what the syntax means, then it is more explicit at telling the reader that the default value is an empty list. Great! This sort of thing is the primary use-case of the feature.

But let's talk about more complex examples:

def open(filename, mode='r', buffering=-1): ...

How would you make that "more explicit"?

def open(filename, mode='r', buffering=>( 1 if 'b' not in mode and os.fdopen(os.open(filename)).isatty() else _guess_system_blocksize() or io.DEFAULT_BUFFER_SIZE ) ): ...

Too much information! We're now drowning in explicitness.

Now how do I, the caller, *explicitly* indicate that I want to use that system-dependent value? I just want a nice, simple value I can explicitly pass as buffering to say "just use the default".

with open('myfile.txt', buffering= what? ) as f:

I can't. So in this case, your agonisingly explicit default expression forces the caller to be *less* explicit when they call the function.

What the right hand giveth, the left hand taketh away.

Can we agree that, like salt, sugar, and bike-shedding, sometimes you can have *too much* explicitness in code? Sometimes the right level of explicitness is to have an agreed upon symbol that acts as a short, easy to read and write, documented and explicit signal to the function "give me the default value".

And that symbol can be -1, or an enumeration, or a string, or None.

This is not a hack, and its not "implicit".

...

None of my comment had anything to do with the caller's perspective.

The caller's perspective is important. The caller has to read the function signature (either in the documentation, or if they use `help(function)` in the interpreter, or when their IDE offers up tooltips or argument completion). The caller has to actually use the function. We should consider their perspective.

...

If my intent is that I want a parameter to default to an empty list, I wouldn't assign it to an object called "monkey_pants" by default then add code to the body of the function to check if it's "monkey_pants".

Funny you should say that, but *totally by coincidence*, the Bulgarian word for "default" happens to be transliterated into English as *monkeypants*, so if I were Bulgarian that is exactly what I might do! How weird is that?

Nah, I'm kidding of course. But the point I am making is that you are mocking the concept by using a totally silly example. "monkey_pants" indeed, ha ha how very droll. But if you used a more meaningful symbol, like "default", or some enumeration with a meaningful name, or a value which is a standard idiom such as -1, then the idea isn't so ridiculous.

And None is one such standard idiom.

*Especially* when the default value is not some trivially simple expression such as an empty list, but a more complicated expression such as the default buffering used by open files, or in my earlier example of the default collation (a sequence of letters which depends on the other input arguments).

...

Steven D'Aprano "How is it misleading?"

Your way of having a parameter default to an empty list (or whatever) is by having the parameter default to None. That's how it's misleading.

All you have done is repeat that it is misleading. I'm still no clearer why you think it is misleading to document "if the argument is missing or None, the default behaviour is ...".

Is the documentation false? Does the function not do exactly what it says it will do if you pass that symbol?

Let's be more concrete.

Do you feel that using -1 for the buffer size in open() is "misleading"? Do you think that there is ever a possibility that someday Python's file I/O will use a buffer with *literally* a size of -1?

-- Steve _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SZVP4K... Code of Conduct: http://python.org/psf/codeofconduct/

On Fri, Dec 3, 2021 at 3:47 PM Abe Dillon <abedillon@gmail.com> wrote:

This seems like an exceedingly minor complaint. The obvious answer is: you invoke the default behaviour by not passing the argument. And yes, I know that's less explicit, but that's kinda the whole point of defaults to begin with. It's behavior when the user decides not to specify their own. Nobody's asking "what do I pass to logging.config.fileConfig to explicitly invoke the logging module's default behavior?!" The answer is simply: don't configure logging. It's not rocket science.

I'd say it isn't "less explicit". It's actually the ultimate definition of not doing something: you just... don't do it. The same line of explanation is important when working with callback functions: def callback(*a): print("Doing stuff") button.signal_connect("clicked", callback()) Oops, you called the callback instead of passing it to the signal_connect function. What's the solution? DON'T call the callback at this point. How do you NOT call something? Well, how do you call it? You put parentheses after it. So how do you not call it? You don't put the parentheses after it. Python is fairly good at giving you syntactic features that do things, such that you can not do them by omitting them. So I don't think anyone is truly concerned about the pure "don't pass the argument" case. The problem is the "maybe pass the argument" case. That ends up looking like this: c = 42 if cond: func(a, b) else: func(a, b, c) # or args = {} if not cond: args["c"] = 42 func(a, b, **args) # or if you know what the default is: c = some_default if not cond: c = 42 func(a, b, c) The first two will, of course, continue to work. The second one is a fairly clear way to indicate whether or not a parameter should get a value. The problem with the third is that it requires a documented and future-guaranteed default value, published and promised to work, and that simply isn't always possible or practical. So, yes, some forms of default might be a little unclear. And if you're converting an existing codebase, it may be necessary to maintain "if c is None: c = some_default" in the body forever, or until such time as you can remove the support for None as a parameter. But None isn't truly the default value.

"Can we agree that, like salt, sugar, and bike-shedding, sometimes you can have *too much* explicitness in code?"

Surely *you* would never agree that there's such a thing as *too much* bike-shedding! Isn't that your raison d'être?!

I never said one had to be a Nazi about stamping out every use of the sentinel... er... idiom (if you prefer it to hack).

Idiom. Sentinels are extremely important, and PEP 671 isn't proposing to eliminate them. The "hack" part comes when, for technical reasons, a sentinel is used when one isn't logically part of the API. I agree with Steve here: sometimes, the true default is extremely clunky, and it's better to use a sentinel. That's part of why they're never going away (and None will continue to be a popular sentinel). PEP 671 doesn't propose changing every function ever written, because the vast majority of them are perfectly fine as they are.

"The caller's perspective is important. The caller has to read the function signature (either in the documentation, or if they use `help(function)` in the interpreter, or when their IDE offers up tooltips or argument completion). The caller has to actually use the function. We should consider their perspective."

I never said we shouldn't. That's just not what I was talking about at that time and you were misinterpreting my argument.

Yes, absolutely. And that's why the function should be able to put the information in the signature when it's appropriate to, and in the body when it isn't.

Just because it's an idiom, doesn't mean it's actually good. I think the `if __name__ == "__main__":` idiom is a horrible, round-about way of saying "if this file is being run as a script". You might look at that and say "What do you mean? That's exactly what it says!" because you've seen the idiom so many times, just like how you think "x=None" explicitly says "give x the default value", but if you look closely `if __name__ == "__main__":` *doesn't* actually say anything like "if this file is being run as a script", it says a bunch of esoteric and confusing garbage that you've learned to read as "if this file is being run as a script".

TBH I think that the name-is-main idiom isn't too bad, but mainly because I've seen far worse idioms. If someone has a better proposal, I'd be happy to hear it, but the existing idiom isn't horrible. (But you're right. Well-known idioms can indeed be really REALLY bad. Just look at the sorts of things that pre-C99 C code would do.) ChrisA

"# or if you know what the default is: c = some_default if not cond: c = 42 func(a, b, c)" I would argue that, of the three examples; this one is the most problematic because it overly couples your code to an implementation detail of the function and makes it unnecessarily brittle to change. On top of that, in the case of a sentinel; it might even encourage people to circumvent the "guarantees" that your API relies upon. For example, I might make a function where the optional parameter could be anything even a None is valid, so I make my own secret sentinel that, as Steven D'Aprano put it "will *never* in any conceivable circumstances become a valid argument value": _default = object() def func(a, b, c=_default): ... Now the pattern above violates the "nobody will *ever* pass me this" condition because it imports _default. Then some intern sees your code and starts importing _default and using it elsewhere because it's such a neat pattern... "The "hack" part comes when, for technical reasons, a sentinel is used when one isn't logically part of the API." That's my entire point. "TBH I think that the name-is-main idiom isn't too bad" The bad part is a lot of Python newbies see it on day one or two because it's so common and try explaining it without talking about the interpreter and watching their eyes glaze over. You just showed them how to assign values to variables and that unbound variables cause errors and now you're pulling __name__ out of thin air? Where the heck did that come from? Anyway, this is a tangent I promised myself I wouldn't go on... On Thu, Dec 2, 2021 at 11:16 PM Chris Angelico <rosuav@gmail.com> wrote:

On Fri, Dec 3, 2021 at 3:47 PM Abe Dillon <abedillon@gmail.com> wrote:

...

This seems like an exceedingly minor complaint. The obvious answer is: you invoke the default behaviour by not passing the argument. And yes, I know that's less explicit, but that's kinda the whole point of defaults to begin with. It's behavior when the user decides not to specify their own. Nobody's asking "what do I pass to logging.config.fileConfig to explicitly invoke the logging module's default behavior?!" The answer is simply: don't configure logging. It's not rocket science.

I'd say it isn't "less explicit". It's actually the ultimate definition of not doing something: you just... don't do it. The same line of explanation is important when working with callback functions:

def callback(*a): print("Doing stuff")

button.signal_connect("clicked", callback())

Oops, you called the callback instead of passing it to the signal_connect function. What's the solution? DON'T call the callback at this point. How do you NOT call something? Well, how do you call it? You put parentheses after it. So how do you not call it? You don't put the parentheses after it.

Python is fairly good at giving you syntactic features that do things, such that you can not do them by omitting them.

So I don't think anyone is truly concerned about the pure "don't pass the argument" case. The problem is the "maybe pass the argument" case. That ends up looking like this:

c = 42 if cond: func(a, b) else: func(a, b, c)

# or args = {} if not cond: args["c"] = 42 func(a, b, **args)

# or if you know what the default is: c = some_default if not cond: c = 42 func(a, b, c)

The first two will, of course, continue to work. The second one is a fairly clear way to indicate whether or not a parameter should get a value. The problem with the third is that it requires a documented and future-guaranteed default value, published and promised to work, and that simply isn't always possible or practical.

So, yes, some forms of default might be a little unclear. And if you're converting an existing codebase, it may be necessary to maintain "if c is None: c = some_default" in the body forever, or until such time as you can remove the support for None as a parameter. But None isn't truly the default value.

...

"Can we agree that, like salt, sugar, and bike-shedding, sometimes you can have *too much* explicitness in code?"

Surely *you* would never agree that there's such a thing as *too much* bike-shedding! Isn't that your raison d'être?!

I never said one had to be a Nazi about stamping out every use of the sentinel... er... idiom (if you prefer it to hack).

Idiom. Sentinels are extremely important, and PEP 671 isn't proposing to eliminate them. The "hack" part comes when, for technical reasons, a sentinel is used when one isn't logically part of the API.

I agree with Steve here: sometimes, the true default is extremely clunky, and it's better to use a sentinel. That's part of why they're never going away (and None will continue to be a popular sentinel). PEP 671 doesn't propose changing every function ever written, because the vast majority of them are perfectly fine as they are.

...

"The caller's perspective is important. The caller has to read the function signature (either in the documentation, or if they use `help(function)` in the interpreter, or when their IDE offers up tooltips or argument completion). The caller has to actually use the function. We should consider their perspective."

I never said we shouldn't. That's just not what I was talking about at that time and you were misinterpreting my argument.

Yes, absolutely. And that's why the function should be able to put the information in the signature when it's appropriate to, and in the body when it isn't.

...

Just because it's an idiom, doesn't mean it's actually good. I think the `if __name__ == "__main__":` idiom is a horrible, round-about way of saying "if this file is being run as a script". You might look at that and say "What do you mean? That's exactly what it says!" because you've seen the idiom so many times, just like how you think "x=None" explicitly says "give x the default value", but if you look closely `if __name__ == "__main__":` *doesn't* actually say anything like "if this file is being run as a script", it says a bunch of esoteric and confusing garbage that you've learned to read as "if this file is being run as a script".

TBH I think that the name-is-main idiom isn't too bad, but mainly because I've seen far worse idioms. If someone has a better proposal, I'd be happy to hear it, but the existing idiom isn't horrible.

(But you're right. Well-known idioms can indeed be really REALLY bad. Just look at the sorts of things that pre-C99 C code would do.)

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QM2ALI... Code of Conduct: http://python.org/psf/codeofconduct/

On Sat, Dec 4, 2021 at 3:47 AM Andrew Jaffe <a.h.jaffe@gmail.com> wrote:

b) I think another motivation that hasn't been highlighted is teaching -- perhaps surprisingly, I think adding this will make some aspects of teaching *easier*. Mutable defaults are known to be confusing to Python neophytes; I have certainly found this when I have both explictly taught or just had to explain some aspect of python.

Yes, definitely, that is a huge advantage IMO.

c) I would probably use this, but can we confirm something that isn't explicit in the pep:

Does this

def fn(word, num=>len(word)):...

mean the same as this? (except for the default for word):

def fn(word="foo", num=>len(word):...

I can't think of any argument they shouldn't but in fact the former reads funny to me.

Not sure what you mean by "mean the same", but the rule is simple: the expression is allowed to refer to anything in its current context. That includes other parameters. If 'word' has no default, then you cannot call the function without providing a word. Defaults are irrelevant if you get TypeError prior to that point. If you pass a value for "word", then the two will behave identically. Either you pass a value for num and it uses that, or you don't pass a value and it uses the length of word as the default. Things are a little more messy if you refer to parameters to the right of them. In my current reference implementation, this is valid, but will raise UnboundLocalError if those parameters also have late-bound defaults and were not given values. But don't depend on that, and it may become more strictly invalid.

Also, relatedly, I assume that late-bound arguments can refer to names that only exist in the caller's environment, but haven't even been bound at definition time? And that if they are not bound at call time, there is a syntax error?

Not a syntax error. It's like any other name reference; if you refer to something that doesn't exist, you'll get a NameError. You're absolutely welcome - and encouraged - to refer to nonlocal (global or closure) names from your environment. Incidentally, I don't yet have any good examples to back this up, but this sort of thing would be completely valid: class X: def method(self, size=>self.n): ... Technically it's the same thing as "a, size=>len(a)" in that it's referring to an earlier parameter, but to a human, this might be seen as a different and also valid use-case. ChrisA

On Sat, Dec 4, 2021 at 4:48 AM Eric V. Smith <eric@trueblade.com> wrote:

On 12/2/2021 6:36 PM, Chris Angelico wrote:

...

On Fri, Dec 3, 2021 at 7:54 AM Eric V. Smith <eric@trueblade.com> wrote:

...

Say I have a function with an early-bound default. I can inspect it and I can change it. One reason to inspect it is so that I can call the function with its default values. This is a form of wrapping the function. I realize "just don't pass that argument when you call the function" will be the response, but I think in good faith you'd have to admit this is more difficult than just passing some default value to a function call.

1) I want to call this function 2) I may want to not pass this argument 3) Ah, perfect! I will pass this argument with a value of somemod._SENTINEL.

Or alternatively:

1) I want to call this function. 2) Prepare a dictionary of arguments. Leave out what I don't want. 3) If I want to pass this argument, add it to the dictionary.

This way doesn't require reaching into the function's private information to use a sentinel. Yes, it may be a tad more difficult (though not VERY much), but you're also avoiding binding yourself to what might be an implementation detail.

Your version is less friendly to type checking. And it doesn't work with positional-only arguments.

Positional-only args can be done with a list instead of a dict. Not much harder, although less clear what's happening if you have multiple.

How is the sentinel value private information or an implementation detail? It's part of the API. It should be clearly documented. If nothing else, it's can be inspected and discovered.

Depends on the sentinel. In this example, is the exact value of the sentinel part of the API? _SENTINEL = object() def frobnicate(stuff, extra=_SENTINEL): ... If the sentinel is None, then it may be part of the API. (Of course, if it's a deliberately-chosen string or something, then that's completely different, and then you're not really doing the "optional argument" thing that I'm talking about here, and there's a very real default value.) But when it's an arbitrary sentinel like this, I argue that the precise value is NOT part of the function's API, only that you can pass any object, or not pass an object at all.

...

...

As far as changing the defaults, consider:

...

...

...

def f(x=3): return x ... f() 3 f.__defaults__=(42,) f() 42

The current PEP design does not provide for this functionality for late-bound defaults. Remember, though: the true comparison should be something like this:

_SENTINEL = object() def f(x=_SENTINEL): if x is _SENTINEL: x = [] return x

Can you change that from a new empty list to something else? No. All you can do, by mutating the function's dunders, is change the sentinel, which is actually irrelevant to the function's true behaviour. You cannot change the true default. It is none the less true that default late-bound values cannot be modified. Correct? Early-bound ones can.

Yes, but I'm asking you to compare late-bound defaults with the "sentinel and replace it in the function" idiom, which is a closer parallel. Can you, externally to the function, change this to use a new empty set instead of a new empty list? No.

...

...

I realize the response will be that code shouldn't need to do these things, but I do not think we should be adding features to python that limit what introspections and runtime modifications user code can do. The response is more that the code CAN'T do these things, by definition. To the extent that you already can, you still can. To the extent that you should be able to, you are still able to. (And more. There are things you're capable of with PEP 671 that you definitely shouldn't do in normal code.)

This is a tautology. You can't do these things if 671 is accepted because they will defined as not doable by 671. That's a strike against it.

My point is that you *already* cannot do them. My proposal doesn't stop you from doing things you currently can do, it just makes it easier to spell the parts you can do.

My stance is that it should be possible, and a proposal that makes them not possible with late-bound arguments is deficient.

Please revisit your concerns with regard to the "sentinel and replace in the function" idiom, as I've stated in the past few posts. The sentinel itself can be replaced, but that is useless if the function's body is still looking for the old sentinel.

We're going to have to disagree about this. I think it's critical that Signature objects be usable with all types of defaults. And having the default value available as a string isn't very useful, except for displaying help text. It wouldn't be possible to create a new Signature object from an existing Signature object that contains a late-bound argument (say, to create a new Signature with an additional argument). At least I haven't seen how it would be possible, since the PEP makes no mention of Signature objects. Which it definitely should, even if only to say "late-bound arguments are not designed to work with Signature objects".

They most certainly ARE usable with all types of defaults, but instead of a meaningless "=<object object at 0xasdfqwer>", you get "=[]".

Sentinels are not meaningless. I think you're alienating people every time you suggest they are.

Some of them are. Some of them are not. When they are meaningful, PEP 671 does not apply, because they're not the sort of sentinel I'm talking about. Unfortunately, the word "sentinel" means many different things. There are many cases where a sentinel is a truly meaningful and useful value, and in those cases, don't change anything. There are other cases where the sentinel is a technical workaround for the fact that Python currently cannot represent defaults that aren't constant values, and those are meaningless sentinels that can be removed.

In any event, I've expended enough volunteer time discussing this for now, so I'm going to drop off. I'm against another way to specify function argument defaults, and I'm against some of the decisions that PEP 671 has made. I've stated those objections. I'll argue against it further when the SC is asked to consider it.

No problem. I don't expect everyone to agree with me, and if it ever happened, I would wonder why :) ChrisA

On 03/12/2021 19:32, Adam Johnson wrote:

The first unwelcome surprise was:

>>> def func(a=>[]): ... return a ...

>>> import inspect >>> inspect.signature(func).parameters['a'].default Ellipsis

Here the current behaviour of returning `Ellipsis` is very unfortunate, and I think could lead to a lot of head scratching — people wondering why they are getting ellipses in their code, seemingly from nowhere. Sure, it can be noted in the official documentation that `Ellipsis` is used as the indicator of late bound defaults, but third-party resources which aim to explain the uses of `Ellipsis` would (with their current content) leave someone clueless.

+1. This may be a very naive question, apologies if it's nonsense. Instead of Ellipsis, would it be possible to have a built-in LateBound class and use instances of that class instead of Ellipsis? The __str__ method of the inspect.Parameter class could be modified to return something like     "a=>[]" (or whatever syntax is adopted for specifying late-bound defaults) in such cases. The __repr__ and __str__ methods of a LateBound object could return something like, respectively,     "LateBound('[]')"     "[]" I am sure there is code that uses inspect.signature that would be broken, but isn't that inevitable anyway? Best wishes Rob Cliffe

On Sat, Dec 4, 2021 at 8:18 AM Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:

On 03/12/2021 19:32, Adam Johnson wrote:

...

The first unwelcome surprise was:

>>> def func(a=>[]): ... return a ...

>>> import inspect >>> inspect.signature(func).parameters['a'].default Ellipsis

Here the current behaviour of returning `Ellipsis` is very unfortunate, and I think could lead to a lot of head scratching — people wondering why they are getting ellipses in their code, seemingly from nowhere. Sure, it can be noted in the official documentation that `Ellipsis` is used as the indicator of late bound defaults, but third-party resources which aim to explain the uses of `Ellipsis` would (with their current content) leave someone clueless.

+1. This may be a very naive question, apologies if it's nonsense. Instead of Ellipsis, would it be possible to have a built-in LateBound class and use instances of that class instead of Ellipsis? The __str__ method of the inspect.Parameter class could be modified to return something like "a=>[]" (or whatever syntax is adopted for specifying late-bound defaults) in such cases. The __repr__ and __str__ methods of a LateBound object could return something like, respectively, "LateBound('[]')" "[]" I am sure there is code that uses inspect.signature that would be broken, but isn't that inevitable anyway?

That's a possibility for the inspect module. For the core language - and therefore for anything that directly inspects the function's dunders - it's much more efficient to use a well-known object. In the current implementation, I've kept inspect.signature() consistent with the dunders, but if there's good reason to change, I wouldn't be averse to it. But I would need feedback from people who make heavy use of inspect.signature, as I don't do much with it other than the basics of help(). Everything in the reference implementation that isn't part of the PEP should be considered provisional at best :) ChrisA

On Sat, Dec 4, 2021 at 11:59 AM Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:

On 03/12/2021 22:38, Chris Angelico wrote:

...

On Sat, Dec 4, 2021 at 8:18 AM Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:

...

On 03/12/2021 19:32, Adam Johnson wrote:

...

The first unwelcome surprise was:

>>> def func(a=>[]): ... return a ...

>>> import inspect >>> inspect.signature(func).parameters['a'].default Ellipsis

Here the current behaviour of returning `Ellipsis` is very unfortunate, and I think could lead to a lot of head scratching — people wondering why they are getting ellipses in their code, seemingly from nowhere. Sure, it can be noted in the official documentation that `Ellipsis` is used as the indicator of late bound defaults, but third-party resources which aim to explain the uses of `Ellipsis` would (with their current content) leave someone clueless.

+1. This may be a very naive question, apologies if it's nonsense. Instead of Ellipsis, would it be possible to have a built-in LateBound class and use instances of that class instead of Ellipsis? The __str__ method of the inspect.Parameter class could be modified to return something like "a=>[]" (or whatever syntax is adopted for specifying late-bound defaults) in such cases. The __repr__ and __str__ methods of a LateBound object could return something like, respectively, "LateBound('[]')" "[]" I am sure there is code that uses inspect.signature that would be broken, but isn't that inevitable anyway?

That's a possibility for the inspect module. For the core language - and therefore for anything that directly inspects the function's dunders - it's much more efficient to use a well-known object. OK. I'm guessing that by "well-known" you mean pre-existing. Python has a number of built-in singleton objects (None, True, False, Ellipsis).

Mainly by "well-known" I mean "not private to any particular module", so those four you mention are all well-known, but a singleton as part of the inspect module would be a pain, since the core language would have to import that (or it would have to be magically created).

What about adding a new one called LateBound (or other bikeshed colour)?

What would be gained? You would still be able to use LateBound as an early-bound default, so you would still need the same dual check. With inspect.Parameter, you already have a repr and str that show that it's late-bound, exactly like you suggest. What advantage is there from using a dedicated sentinel instead of Ellipsis? It wouldn't be too hard to change the inspect module, but I'd need to know why. ChrisA

On Fri, Dec 03, 2021 at 10:40:42AM +1100, Chris Angelico wrote:

Here's what you get:

...

...

...

def f(lst=>[], n=>len(lst)): ... ... f.__defaults_extra__ ('[]', 'len(lst)')

String representation, but exactly what the default is.

Excellent. And you've just proven that we can evaluate the defaults. >>> a = '[]' # f.__defaults_extra__[0] >>> b = 'len(lst)' # f.__defaults_extra__[1] >>> lst = eval(a, globals()) >>> n = eval(b, globals(), dict(lst=lst)) >>> print(lst, n) [] 0 Worst case scenario, we can eval the string representation, and that should work for the great majority of cases that don't involve nonlocals. But if the defaults are proper code objects, complete with a closure to capture their nonlocal environment, then we should be able to do even better and capture nonlocals as well. Could there be odd corner cases that don't quite work? Say, like a comprehension inside a class body? https://github.com/satwikkansal/wtfpython#-name-resolution-ignoring-class-sc... Oh well. Let's not make the perfect the enemy of the good. -- Steve

On Sat, Dec 4, 2021 at 2:52 PM Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:

On 04/12/2021 01:06, Chris Angelico wrote:

...

On Sat, Dec 4, 2021 at 11:59 AM Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:

...

On 03/12/2021 22:38, Chris Angelico wrote:

...

On Sat, Dec 4, 2021 at 8:18 AM Rob Cliffe via Python-ideas <python-ideas@python.org> wrote:

...

On 03/12/2021 19:32, Adam Johnson wrote:

...

The first unwelcome surprise was:

>>> def func(a=>[]): ... return a ...

>>> import inspect >>> inspect.signature(func).parameters['a'].default Ellipsis

Here the current behaviour of returning `Ellipsis` is very unfortunate, and I think could lead to a lot of head scratching — people wondering why they are getting ellipses in their code, seemingly from nowhere. Sure, it can be noted in the official documentation that `Ellipsis` is used as the indicator of late bound defaults, but third-party resources which aim to explain the uses of `Ellipsis` would (with their current content) leave someone clueless.

+1. This may be a very naive question, apologies if it's nonsense. Instead of Ellipsis, would it be possible to have a built-in LateBound class and use instances of that class instead of Ellipsis? The __str__ method of the inspect.Parameter class could be modified to return something like "a=>[]" (or whatever syntax is adopted for specifying late-bound defaults) in such cases. The __repr__ and __str__ methods of a LateBound object could return something like, respectively, "LateBound('[]')" "[]" I am sure there is code that uses inspect.signature that would be broken, but isn't that inevitable anyway?

That's a possibility for the inspect module. For the core language - and therefore for anything that directly inspects the function's dunders - it's much more efficient to use a well-known object. OK. I'm guessing that by "well-known" you mean pre-existing. Python has a number of built-in singleton objects (None, True, False, Ellipsis).

Mainly by "well-known" I mean "not private to any particular module", so those four you mention are all well-known, but a singleton as part of the inspect module would be a pain, since the core language would have to import that (or it would have to be magically created).

...

What about adding a new one called LateBound (or other bikeshed colour)? What would be gained? You would still be able to use LateBound as an early-bound default, so you would still need the same dual check. I'm struggling here. Yes you could use LateBound as an early-bound default (or as a parameter value to explicitly pass to a function) but ISTM that such usages would be perverse. I've a gut feeling that a solution can be found (to avoid the "Ellipsis from nowhere" problem) but I can't put my finger on it. Maybe explicitly specifying LateBound could be an error, perhaps even a SyntaxError? Help, please!

It would be extremely odd if something could fail due to the precise object chosen. x = LateBound def f(x=x): pass # SyntaxError? Runtime error? Awkward. Problematic. And it wouldn't buy you anything anyway - consumers of inspect.Signature would still have to be aware of this special object, so you still have the same problems. ChrisA

On Sat, Dec 04, 2021 at 03:14:46PM +1100, Chris Angelico wrote:

Lots and lots and lots of potential problems. Consider:

def f(): a = 1 def f(b, x=>a+b): def g(): return x, a, b

Both a and b are closure variables - one because it comes from an outer scope, one because it's used in an inner scope. So to evaluate a+b, you have to look up an existing closure cell, AND construct a new closure cell.

The only way to do that is for the compiled code of a+b to exist entirely within the context of f's code object.

I dispute that is the only way. Let's do a thought experiment. First, we add a new flag to the co_flags field on code objects. Call it the "LB" flag, for late-binding. Second, we make this: def f(b, x=>a+b): ... syntactic sugar for this: def f(b, x=lambda b: a+b): ... except that the lambda has the LB flag set. And third, when the interpreter fetches a default from func.__defaults__, if it is a LB function, it automatically calls that function with the parameters to the left of x (which in this case would be just b). Here's your function, with a couple of returns to make it actually do something: def f(): a = 1 def f(b, x=>a+b): def g(): return x, a, b return g return f We can test that right now (well, almost all of it) with this: def func(): # change of name to distinguish inner and outer f a = 1 def f(b, x=lambda b: a+b): def g(): return x, a, b return g return f and just pretend that x is automatically evaluated by the interpreter. But as a proof of concept, it's enough that we can demonstrate that *we* can manually evaluate it, by calling the lambda. We can call func() to get the inner function f, and call f to get g: >>> f = func() >>> print(f) <function func.<locals>.f at 0x7fc945c41f30> >>> g = f(100) >>> print(g) <function func.<locals>.f.<locals>.g at 0x7fc945e1f520> Calling g works: >>> print(g()) (<function func.<locals>.<lambda> at 0x7fc945c40f70>, 1, 100) with the understanding that the real implementation will have automatically called that lambda, so we would have got 101 instead of the lambda. That step requires interpreter support, so for now we just have to pretend that we get (101, 1, 100) instead of the lambda. But we can demonstrate that calling the lambda works, by manually calling it: >>> x = g()[0] >>> print(x) <function func.<locals>.<lambda> at 0x7fc945c40f70> >>> print(x(100)) # the interpreter knows that b=100 101 Now let's see if we can extract the default and play around with it: >>> default_expression = f.__defaults__[0] >>> print(default_expression) <function func.<locals>.<lambda> at 0x7fc945c40f70> The default expression is just a function (with the new LB flag set). So we can inspect its name, its arguments, its cell variables, etc: >>> default_expression.__closure__ (<cell at 0x7fc945de74f0: int object at 0x7fc94614c0f0>,) We can do anything that we could do with any other other function object. Can we evaluate it? Of course we can. And we can test it with any value we like, we're not limited to the value of b that we originally passed to func(). >>> default_expression(3000) 3001 Of course, if we are in a state of *maximal ignorance* we might have no clue what information is needed to evaluate that default expression: >>> default_expression() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: func.<locals>.<lambda>() missing 1 required positional argument: 'b' Oh look, we get a useful diagnostic message for free! What are we missing? The source code of the original expression, as text. That's pretty easy too: the compiler knows the source, it can cram it into the default expression object: >>> default_expression.__expression__ = 'a+b' Introspection tools like help() can learn to look for that. What else are we missing? A cool repr. >>> print(default_expression) # Simulated. <late bound default expression a+b> We can probably come up with a better repr, and a better name than "late bound default expression". We already have other co_flags that change the repr: 32 GENERATOR 128 COROUTINE 256 ITERABLE_COROUTINE so we need a name that is at least as cool as "generator" or "coroutine". Summary of changes: * add a new co_flag with a cool name better than "LB"; * add an `__expression__` dunder to hold the default expression; (possibly missing for regular functions -- we don't necessarily need *every* function to have this dunder) * change the repr of LB functions to display the expression; * teach the interpreter to compile late-bound defaults into one of these LB functions, including the source expression; * teach the interpreter that when retrieving default values from the function's `__defaults__`, if they are a LB function, it must call the function and use its return result as the actual default value; * update help() and other introspection tools to handle these LB functions; but if any tools don't get updated, you still get a useful result with an informative repr. -- Steve

On 4 Dec 2021, at 09:44, Steven D'Aprano <steve@pearwood.info> wrote:

On Sat, Dec 04, 2021 at 03:14:46PM +1100, Chris Angelico wrote:

...

Lots and lots and lots of potential problems. Consider:

def f(): a = 1 def f(b, x=>a+b): def g(): return x, a, b

Both a and b are closure variables - one because it comes from an outer scope, one because it's used in an inner scope. So to evaluate a+b, you have to look up an existing closure cell, AND construct a new closure cell.

The only way to do that is for the compiled code of a+b to exist entirely within the context of f's code object.

I dispute that is the only way. Let's do a thought experiment.

There are many possible implementation of the late bound idea that could create an object/default expression. But is it reasonable to bother with that added complexity/maintenance burden for a first implementation. And maybe no one will care enough to ever implement the ability to modify the code of a late bound variables expression as a separate object later. I think I understand the argument as being along the lines of for early bound defaults they can be inspected and modified. Therefore being able to do the same for late bound defaults must be implemented. I'm not convinced that that is reasonable to require is implemented. If python had always had late bound defaults, as it is with most languages in the survey posted earlier in this thread, would that have been implemented as an object/expression? Maybe, but I doubt it. Summary: I agree it's not impossible, I do not agree that it's needed. Barry

On 2021-12-04 03:50, Chris Angelico wrote:

On Sat, Dec 4, 2021 at 8:48 PM Steven D'Aprano <steve@pearwood.info> wrote:

...

And third, when the interpreter fetches a default from func.__defaults__, if it is a LB function, it automatically calls that function with the parameters to the left of x (which in this case would be just b).

Plausible. Okay.

What this does mean, though, is that there are "magic objects" that cannot be used like other objects. Consider:

Your proposal also has the same problem, since it involves "magic functions" that do not have usable values for their argument defaults, instead having some kind of Ellipsis two-step. It's all a matter of what you consider magic.

Great. So now we have some magnificently magical behaviour in the language, which will have some nice sharp edge cases, but which nobody will ever notice. Totally. I'm sure. Plus, we pay a performance price in any function that makes use of argument references, not just for the late-bound default, but in the rest of the code. We also need to have these special functions that get stored as separate code objects.

All to buy what, exactly? The ability to manually synthesize an equivalent parameter value, as long as there's no assignment expressions, no mutation, no other interactions, etc, etc, etc? That's an awful lot of magic for not a lot of benefit.

I would consider most of what you say here an accurate description of your own proposal. :-) Now we have magnificently magical behavior in the language, which will take expressions in the function signature and behind the scenes "inline" them into the function body. We also need to have these special function arguments that do NOT get stored as separate objects, unlike ordinary function arguments. All to buy what, exactly? The ability to write something in the function signature that we can already write in the body, and that quite naturally belongs in the body, because it is executed when the function is called, not when it is defined.

I *really* don't like the idea that some types of object will be executed instead of being used, just because they have a flag set. That strikes me as the sort of thing that should be incredibly scary, but since I can't think of any specific reasons, I just have to call it "extremely off-putting".

I *really* don't like the idea that some types of argument will be inlined into the function body instead of being stored as first-class values like other `__defaults__`, just because there happens to be this one extra character next to the equals sign in the function signature. That strikes me as the sort of thing that should be incredibly scary. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On Sun, Dec 5, 2021 at 5:29 AM Barry Scott <barry@barrys-emacs.org> wrote:

...

On 1 Dec 2021, at 06:16, Chris Angelico <rosuav@gmail.com> wrote:

I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/ with some additional information about the reference implementation, and some clarifications elsewhere.

(I suspect that there was a reply that I should be replying to but, cannot find one appropriate)

I have a lot of code that exploits the fact that passing an explicit None will cause the early bound default idiom to set the default for me.

def inner(timestamp=None): if timestamp is None: timestamp = time.time() do_stuff...

def outer(timestamp=None): inner(timestamp=timestamp)

outer can in an idiomatic way have inner default timestamp and not have to know what that means.

If you need outer() to be able to have a value that means "use the default", then there are three options: 1) Don't pass timestamp at all. In simple cases where it will only and always specify the default, this is fine. 2) Define a sentinel that is indeed part of your API. 3) Use *args or **kwargs to choose whether to pass it or not (best if there are multiple of them). You can continue to use the existing system of "if none, do this", or you can flip it around and have the sentinel as a special token within your code: def inner(timestamp=>time.time()): if timestamp is None: timestamp = time.time() Depends on how important this feature is outside of your own helper functions. (I would probably not do this for None specifically - if it's purely internal, I'm more likely to use a dedicated local sentinel object.) But as soon as there are two or three arguments that "might have to be passed, might not", it's far more readable to use kwargs to pass just the ones you want. def outer(**kwargs): inner(**kwargs) That way, if something changes in inner(), you don't have to worry about breaking your caller's API.

With late bound I cannot do this without more complex pattern of building an arg list.

What if passing None still worked? I know the argument that there are more sentinels then None.

def inner(timestamp=>time.time()) do_stuff...

def outer(timestamp=None): inner(timestamp=timestamp)

The code in inner that decides to when to allow the default could check for timestamp being missing or arg present and None.

Would the lack of support for other sentinels out weight the simple way to get the default applied?

None is most assuredly not going to trigger a late-bound default. Python is not JavaScript :) ChrisA

On Sun, Dec 5, 2021 at 11:34 AM Steven D'Aprano <steve@pearwood.info> wrote:

On Sat, Dec 04, 2021 at 10:50:14PM +1100, Chris Angelico wrote:

...

...

syntactic sugar for this:

def f(b, x=lambda b: a+b): ...

except that the lambda has the LB flag set.

Okay. So the references to 'a' and 'b' here are one more level of function inside the actual function we're defining, which means you're paying the price of nonlocals just to be able to late-evaluate defaults. Not a deal-breaker, but that is a notable cost (every reference to them inside the function will be slower).

How much slower? By my tests:

- access to globals is 25% more expensive than access to locals; - access to globals is 19% more expensive than nonlocals; - and nonlocals are 6% more expensive than locals.

Or if you do the calculation the other way (the percentages don't match because the denominators are different):

- locals are 20% faster than globals; - and 5% faster than nonlocals; - nonlocals are 16% faster than globals.

Premature optimization is the root of all evil.

We would be better off spending effort making nonlocals faster for everyone than throwing out desirable features and a cleaner design just to save 5% on a microbenchmark.

Fair, but the desirable feature can be achieved without this cost, and IMO your design isn't cleaner than the one I'm already using, and 5% is a lot for no benefit.

[...]

...

What this does mean, though, is that there are "magic objects" that cannot be used like other objects.

NotImplemented says hello :-)

Good point. Still, I don't think we want more magic like that.

And if you still think that we should care, we can come up with a more complex trigger condition:

- the parameter was flagged as using a late-default; - AND the default is a LB function.

Problem solved. Now you can use LB functions as early-bound defaults, and all it costs is to record and check a flag for each parameter. Is it worth it? Dunno.

Uhh.... so..... the parameter has to be flagged AND the value has to be flagged? My current proposal just flags the parameter. So I ask again: what are you gaining by this change? You've come right back to where you started, and added extra costs and requirements, all for.... what?

[...]

...

...

The default expression is just a function (with the new LB flag set). So we can inspect its name, its arguments, its cell variables, etc:

>>> default_expression.__closure__ (<cell at 0x7fc945de74f0: int object at 0x7fc94614c0f0>,)

We can do anything that we could do with any other other function object.

Yup. As long as it doesn't include any assignment expressions, or anything else that would behave differently.

I don't get what you mean here. Functions with the walrus operator are still just functions that we can introspect:

...

...

...

f = lambda a, b: (len(w:=str(a))+b)*w f('spam', 2) 'spamspamspamspamspamspam' f.__code__ <code object <lambda> at 0x7fc945e07c00, file "<stdin>", line 1>

What sort of "behave differently" do you think would prevent us from introspecting the function object? "Differently" from what?

Wrapping it in a function means the walrus would assign in that function's context, not the outer function. I think it'd be surprising if this works: def f(x=>(a:=1)+a): # default is 2 but this doesn't: def g(x=>(a:=1), y=>a): # default is UnboundLocalError It's not a showstopper, but it is most definitely surprising. The obvious solution is to say that, in this context, a is a nonlocal. But this raises a new problem: The function object, when created, MUST know its context. A code object says "this is a nonlocal", and a function object says "when I'm called, this is my context". Which means you can't have a function object that gets called externally, because it's the code, not the function, that is what you need here. And that means it's not directly executable, but it needs a context. So, once again, we come right back around to what I have already: code that you can't lift out and call externally. The difference is that, by your proposal, there's a lot more overhead, for the benefit of maybe under some very specific circumstances being able to synthesize the result.

...

We also need to have these special functions that get stored as separate code objects.

That's not a cost, that's a feature. Seriously. We're doing that so that we can introspect them individually, not just as the source string, but as actual callable objects that can be:

- introspected;

- tested;

- monkey-patched and modified in place (to the degree that any function can be modified, which is not a lot);

- copied or replaced with a new function.

Testing is probably the big one. Test frameworks will soon develop a way to let you write tests to confirm that your late bound defaults do what you expect them to do.

That's trivial for `arg=[]` expressions, but for complex expressions in complex functions, being able to isolate them for testing is a big plus.

I'm still not convinced that it's as useful as you say. Compare these append-and-return functions: def build1(value, lst=None): if lst is None: lst = [] lst.append(value) return lst _SENTINEL = object() def build2(value, lst=_SENTINEL): if lst is _SENTINEL: lst = [] lst.append(value) return lst def hidden_sentinel(): _SENTINEL = object() def build3(value, lst=_SENTINEL): if lst is _SENTINEL: lst = [] lst.append(value) return lst return build3 build3 = hidden_sentinel() def build4(value, *optional_lst): if len(optional_lst) > 1: raise TypeError("too many args") if not optional_lst: optional_lst = [[]] optional_lst[0].append(value) return optional_lst[0] def build5(value, lst=>[]): lst.append(value) return lst (Have I missed any other ways to do this?) In which of them can you introspect the []? Three of them have an externally-visible sentinel, but you can't usefully change it in any way. You can look at it, and you'll see None or "<object object at 0xYourCapitalCity>", but that doesn't tell you that you'll get a new empty list. In which of them do you have a thing you can call that will generate an equivalent empty list? In which can you monkey-patch or modify-in-place the []? Not a big deal though, you admit that there wouldn't be much monkey-patching in any form. But you get none whatsoever with other idioms. In which of these can you separately test the []? Do any current test frameworks have a way to let you write tests to make sure that these do what you expect? In which of these can you copy the [] into another context, or replace the [] with something else? Why are these such major problems for build5 if they're not for the other four? ChrisA

On Sun, Dec 5, 2021 at 3:08 PM Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:

Barry Scott writes:

...

There are many possible implementation of the late bound idea that could create an object/default expression. But is it reasonable to bother with that added complexity/maintenance burden for a first implementation.

Yes. If you don't do it, you'll have backward compatibility issues or technical debt.

I'm not saying that's a compelling argument here, except that one of the main alleged problems is that users don't understand mutable defaults. So adding more and more layers of support for default arguments is making matters worse, I suspect. (Remember, they're going to be reading "arg=None" and "@arg=[]" for a long long time.)

This one is Worth Doing Right the first time, I think. And IMO David Mertz is right: doing it right means a more general deferred-evaluation object (not to be confused with Deferreds that need to be queried about their value).

If you think that deferred evaluation objects are the right way to do it, then write up a proposal to compete with PEP 671. In my opinion, it is a completely independent idea, which has its own merit, and which is not a complete replacement for late-bound defaults; the two could coexist in Python simultaneously, or either one could be accepted without the other, or we could continue to have neither. Yes, there's some overlap in the problems they solve, just as there's overlap between PEP 671 and PEP 661 on named sentinels; but there's also overlap between plenty of other language features, and we don't deem try/finally or context managers to be useless because of the other. ChrisA

Brendan Barnwell writes:

The ability to write something in the function signature that we can already write in the body, and that quite naturally belongs in the body, because it is executed when the function is called, not when it is defined.

I'm basically in sympathy with your conclusion, but I don't think it's useful to prejudice the argument by saying it *naturally belongs* in the body. Some languages quite naturally support thunks/blocks (Ruby) or even go so far as to equate code to data (Lisp), and execute that code in lieu of what appear to be variable references. But maybe it's *Pythonic* to necessarily place the source code in the body? I can't say that.

...

I *really* don't like the idea that some types of object will be executed instead of being used, just because they have a flag set.

From a syntactic point of view, that's how Ruby blocks work. Closer to home, that's how properties work. And in the end, *all* objects are accessed by executing code. This is a distinction without a difference, except in our heads. I wouldn't want to be asked to explain the dividing line between objects that were "just used" and objects that were "produced by code that was executed instead of being just used".

I *really* don't like the idea that some types of argument will be inlined into the function body instead of being stored as first-class values like other `__defaults__`, just because there happens to be this one extra character next to the equals sign in the function signature. That strikes me as the sort of thing that should be incredibly scary.

Properties have *no* visible syntax if they're imported from a module. Properties are extremely useful, and we all use them all the time without noticing or caring. I see no reason in principle why the same kind of feature wouldn't be useful and just as invisible and just as "natural" for local or global variables -- or callable parameters, as long as properly restricted. Chris's proposal is nothing if not restricted! :-) My issues with Chris's proposal are described elsewhere, but I don't really see a problem in principle.

Chris Angelico writes:

On Sun, Dec 5, 2021 at 3:08 PM Stephen J. Turnbull <stephenjturnbull@gmail.com> wrote:

...

This one is Worth Doing Right the first time, I think. And IMO David Mertz is right: doing it right means a more general deferred-evaluation object (not to be confused with Deferreds that need to be queried about their value).

If you think that deferred evaluation objects are the right way to do it, then write up a proposal to compete with PEP 671.

That's not your call, I'm afraid. "Keep the status quo" is always a viable option, regardless of other options. And other things equal, it's the preferred option.

In my opinion, it is a completely independent idea,

You're welcome to your opinion, of course. But if you want to claim that's a reason for implementing your proposal, you need to support it. You also need to explain why the additional potential complexity of a third kind of default argument (evaluated at definition, evaluated during call, evaluated when referenced) isn't a big problem.

which is not a complete replacement for late-bound defaults;

Why not? If we have such objects, we could simply specify that in the case where such an object is specified as a function parameter default, it is evaluated in the same environment as your late-bound defaults. So we can have your schsemantics if that's what we want. On the other hand, it might turn out that 90% of the time, it doesn't matter if it's evaluated as part of the calling process, 9% of the time the natural place to evaluate it is at the point of first use, and 1% of the time it should be evaluated before the function body proper is entered. In that case the late-bound default would be of some utility, but is it worth it? David's "infinitesimal utility" argument seems likely to apply. What else would such a deferred-evaluation object be unable to do that your late-bound default can do?

we could continue to have neither.

That's where I am. More than any of these issues, the lack of a well- defined, properly introspectable object bothers me. In 2021, we should provide that.