I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
(It's absolutely valid to say "yes" and "yes", and feel free to say
which of those pulls is the stronger one.)
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
Any and all comments welcomed. I mean, this is python-ideas after
all... bikeshedding is what we do best!
The reference implementation currently has some test failures, which
I'm looking into. I'm probably going to make this my personal default
Python interpreter for a while, to see how things go.
ChrisA
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Yes I will use it.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
No, because it would look like a lambda (if the new lambda syntax were approved), indicating this will be evaluated each time the function is run.
(It's absolutely valid to say "yes" and "yes", and feel free to say
which of those pulls is the stronger one.)
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
I haven’t done it from source. I might try to learn how to do it in the next weekend and give it a try.
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
I will show this to some of my coworkers who are python experts and I will report back.
Any and all comments welcomed. I mean, this is python-ideas after
all... bikeshedding is what we do best!
The reference implementation currently has some test failures, which
I'm looking into. I'm probably going to make this my personal default
Python interpreter for a while, to see how things go.
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
I would definitely use it.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
No, it isn't much of a cognitive burden.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
Probably all options other than (d).
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
Yes I do know how to, and I'm gonna try it out soon enough. I'll reply to this comment once I've done so.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
No, it isn't much of a cognitive burden.
You say that now, but if you read function definitions that looked
like this:
def process(func:List->int=>xs=>expression)->int:
...
would you still agree?
--
Steve
On 1 Dec 2021, at 10:16 AM, Chris Angelico wrote:
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).
Very interesting. That doesn't normally seem like a function default -
is the callback ever going to be passed two arguments (self and actual
text) such that the default would be ignored?
But, hey, if it makes sense in your code to make it a parameter, sure!
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
I haven’t done it from source. I might try to learn how to do it in the next weekend and give it a try.
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
I will show this to some of my coworkers who are python experts and I will report back.
Thank you! All feedback greatly appreciated.
Building CPython from source can be done by following these instructions:
https://devguide.python.org/setup/
Instead of creating your own clone from the pristine master copy,
instead clone my repository at https://github.com/rosuav/cpython and
checkout the pep-671 branch. That'll give you my existing reference
implementation (it's pretty crummy but it mostly works).
ChrisA
I'm not sure what that's supposed to mean. Is List->int an annotation,
and if so, who's deciding on that syntax? You seem to have used two
different arrows in two different ways:
List->int # presumably an annotation meaning "function that takes
List, returns int"
func:ann=>dflt # late-bound default, completely unnecessary here
xs=>expression # presumably a lambda function
def process(args)->int # return value annotation
Why should List->int and xs=>expression use different arrows? Wouldn't
it be much more reasonable for them to use the same one, whichever
that be? And if that does turn out to be "=>", then yes, I would be
looking at changing PEP 671 to recommend := or =: or something, for
clarity (but still an equals sign with one other symbol next to it).
It's always possible to come up with pathological code. But this is
only really as bad as you describe if it has zero spaces in it.
Otherwise, it's pretty easy to clarify which parts go where:
def process(func: List->int => xs=>expression) -> int:
...
Tada, easy grouping.
ChrisA
(To clarify: by "that", I mean "the hypothetical line of code", not
the question you're asking. I'm not sure what the code example is
supposed to mean.)
ChrisA
I'll reply to this comment once I've done so.
I just realized that this message replied on a comment instead of the thread. Anyways, it's a really impressive implementation (hope this statement doesn't spark some sort of debate).
I'll reply to this comment once I've done so.
I just realized that this message replied on a comment instead of the thread. Anyways, it's a really impressive implementation (hope this statement doesn't spark some sort of debate).
I'm.... going to take that at face value and interpret it as a
compliment. Thank you. :)
If you intended that to be taken differently, then yeah, you're
probably not wrong there too - it is, in a sense, quite... impressive.
ChrisA
On 1 Dec 2021, at 10:16 AM, Chris Angelico wrote:
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).
Very interesting. That doesn't normally seem like a function default -
is the callback ever going to be passed two arguments (self and actual
text) such that the default would be ignored?
Yeah. Let’s say the callback prints the text on the main window console. I could use the same function to print something on the console not related to the default widget changing text. Maybe another push button that prints literal “WOW”. If I made the second argument a default widget (def callback(self, line_edit=self.line_edit):...) and then call line_edit.text() inside the function, I would be able to pass any other widget that has the method text() but I wouldn’t be able to pass a literal text.
But, hey, if it makes sense in your code to make it a parameter, sure!
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
I haven’t done it from source. I might try to learn how to do it in the next weekend and give it a try.
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
I will show this to some of my coworkers who are python experts and I will report back.
Thank you! All feedback greatly appreciated.
Building CPython from source can be done by following these instructions:
Instead of creating your own clone from the pristine master copy,
instead clone my repository at https://github.com/rosuav/cpython and
checkout the pep-671 branch. That'll give you my existing reference
implementation (it's pretty crummy but it mostly works).
On 1 Dec 2021, at 10:16 AM, Chris Angelico wrote:
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
I will definitely use it for default mutable collections like list, set, dictionary etc. I will also use it to reference things that might have changed. For example, when making callbacks to GUI push buttons, I find myself at the start of the function/callback to be fetching the values from other widgets so we can do something with them. Now those values can be directly passed as late-bound defaults from their respective widgets (e.g., def callback(self, text1 => self.line_edit.text()): …).
Very interesting. That doesn't normally seem like a function default -
is the callback ever going to be passed two arguments (self and actual
text) such that the default would be ignored?
Yeah. Let’s say the callback prints the text on the main window console. I could use the same function to print something on the console not related to the default widget changing text. Maybe another push button that prints literal “WOW”. If I made the second argument a default widget (def callback(self, line_edit=self.line_edit):...) and then call line_edit.text() inside the function, I would be able to pass any other widget that has the method text() but I wouldn’t be able to pass a literal text.
Ahh gotcha. Then yes, you have a function that takes up to two
parameters (or one if you don't count self), and yes, omitting the
second argument will have the same effect as fetching the text from
that control. So, sounds like a great use for it!
ChrisA
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
No, I would avoid for the sake of readers. Anyway, it can't be used in any
public projects until thy drop support for 3.10, which is many years off.
Also, I think this question is quite the biased sample on python-ideas.
Please consider asking this to less advanced python users, e.g.,
reddit.com/r/python or learnpython.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
Any and all comments welcomed. I mean, this is python-ideas after
all... bikeshedding is what we do best!
This PEP has a lot of interesting ideas. I still think that none-aware
operators (PEP 505) are an easier, more readable general solution to
binding calculated default values to arguments succinctly. I think the
problems with this idea include:
* The caller cannot explicitly ask for default behaviour except by omitting
the parameter. This can be very annoying to set up when the parameter
values are provided from other places, e.g.,
if need_x:
# do lots of stuff
x = whatever
else:
# do more
x = None
f(x=x, ...) # easy, but with this PEP, you would have to find a way to
remove x from the parameter list. Typically, removing a parameter from a
dynamically-created parameter list is hard.
* The function code becomes unreadable if the parameter-setting code is
long. Having a long piece of code inside the function after "if parameter
is None" is just fine. Having none-aware operators would make such code
more succinct.
* People nearly always avoid writing code in the parameter defaults
themselves, and this new practice adds a lot of cognitive load. E.g.,
people rarely write:
def f(x: int = 1+g()) -> None: ...
Parameter lists are already busy enough with parameter names, annotations,
and defaults. We don't need to encourage this practice.
In short, I think this is a creative idea, a great exploration. While
optional parameters are common, and some of them have defaults that are
calculated inside the function, my feeling is that people will continue to
set their values inside the function.
Best,
Neil
Is List->int an annotation,
and if so, who's deciding on that syntax? You seem to have used two
different arrows in two different ways:
Exactly my point. Beyond your arrow syntax for default parameters, and
the existing return annotation use, there are two other hypothetical
proposals for arrow syntax on the table:
- using `->` as an alias for typing.Callable;
- using `=>` as a more compact lambda;
See here:
https://lwn.net/Articles/847960/
I rate the first one (the typing.Callable alias) as extremely likely. It
is hard to annotate parameters which are functions, and the typing
community is actively working on making annotations easier. So my gut
feeling is that it is only a matter of time before we have annotations
like `func:List->int`. And probably closer to 3.11 than 3.20 :-)
The second, the compact lambda, I rate as only moderately likely. Guido
seems to like it, but whether somebody writes a PEP and the Steering
Council accepts it is uncertain. But it is something that does keep
coming up, and I think we should consider it as a possibility.
Even if `=>` doesn't get used for compact lambda, it is an obvious
syntax to use for *something* and so we should consider how late-binding
will look, not just with existing syntax, but also with *likely* future
syntax.
(When you build a road, you don't plan for just the population you have
now, but also the population you are likely to have in the future.)
I don't expect anyone to take into account *arbitrary* and unpredictable
future syntax, but it is wise to take into account future syntax that
we're already planning or that seems likely.
func:ann=>dflt # late-bound default, completely unnecessary here
Come on Chris, how can you say that it is "completely unnecessary"?
Unless that's an admission that late-bound defaults are all
unnecessary... *wink*
There is a difference between these two:
def func(arg=lambda a: expression):
...
and this:
def func(arg=None):
if arg is None:
arg = lambda a: expression
...
therefore there will be a difference between:
def func(arg=lambda a: expression):
def func(arg=>lambda a: expression):
If nothing else, in the first case (early binding) you get the same
function object every time. In the second, you get a freshly made
function object each time. Since function objects are mutable (they have
a writable `__dict__` that's a visible difference even if the bodies are
identical. And they may not be.
But even if it is unnecessary, it will still be permitted, just as we
will be able to write:
# Now this *actually is* totally unnecessary use of late-binding
def func(arg=>None):
xs=>expression # presumably a lambda function
def process(args)->int # return value annotation
Why should List->int and xs=>expression use different arrows?
Because they do different things. To avoid confusion.
Chris, you wrote the PEP for the walrus operator. Why should the
assignment operator use a different symbol from the assignment
statement? Same reason that typing.Callable and lambda will likely use
different arrows.
Anyway, if you disagree, take it up with Guido, it was his suggestion to
use different arrows :-P
Wouldn't
it be much more reasonable for them to use the same one, whichever
that be? And if that does turn out to be "=>", then yes, I would be
looking at changing PEP 671 to recommend := or =: or something, for
clarity (but still an equals sign with one other symbol next to it).
Oh great, now you're going to conflict with walrus...
def process(obj:Union[T:=something, List[T]]:=func(x:=expression)+x)->T:
We ought to at least try to avoid clear and obvious conflicts between
new and existing syntax.
Using `:=` is even worse than `=>`, and `=:` is just *begging* to
confuse newcomers "why do we have THREE different assignment symbols?"
It's always possible to come up with pathological code. But this is
only really as bad as you describe if it has zero spaces in it.
Otherwise, it's pretty easy to clarify which parts go where:
"My linter complains about spaces around operators! Take them out!"
Or maybe we should put more in? Spaces are optional.
def process(func : List -> int => xs => expression) -> int:
...
I'm not saying that an experienced Pythonista who is a careful reader
can't work out what the meaning is. It's not *ambiguous* syntax to a
careful reader. But it's *confusing* syntax to somebody who may not be
as careful and experienced as you, or is coding late at night, or in a
hurry, or coding while tired and emotional or distracted.
We should prefer to avoid confusable syntax when we can. The interpreter
can always disambiguate assignment as an expression from assignment as a
statement, but we still chose to make it easy on the human reader by
using distinct symbols.
def process(arg>=expression)
would be totally unambiguous to the intepreter, but I trust you would
reject that because it reads like "greater than" to the human reader.
--
Steve
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
No, I would avoid for the sake of readers. Anyway, it can't be used in any public projects until thy drop support for 3.10, which is many years off.
Also, I think this question is quite the biased sample on python-ideas. Please consider asking this to less advanced python users, e.g., reddit.com/r/python or learnpython.
I know it is :) I never expected to get an unbiased sample (from
anywhere, least of all here).
This PEP has a lot of interesting ideas. I still think that none-aware operators (PEP 505) are an easier, more readable general solution to binding calculated default values to arguments succinctly. I think the problems with this idea include:
* The caller cannot explicitly ask for default behaviour except by omitting the parameter. This can be very annoying to set up when the parameter values are provided from other places, e.g.,
if need_x:
# do lots of stuff
x = whatever
else:
# do more
x = None
f(x=x, ...) # easy, but with this PEP, you would have to find a way to remove x from the parameter list. Typically, removing a parameter from a dynamically-created parameter list is hard.
You can always do that with *args or **kwargs if that's what you need,
and that actually DOES represent the concept of "don't pass this
argument". Passing None is a hack that doesn't actually mean "don't
pass this argument", and if you're depending on that behaviour, it's
been locked in as the function's API, which causes problems if None
ever becomes a meaningful non-default value.
None-aware operators are solving a different problem. They allow you
to accept None, but then simplify the replacement of it with some
other value. This proposal allows you to leave None out of the
equation altogether. And it only works if None isn't a valid value.
* The function code becomes unreadable if the parameter-setting code is long. Having a long piece of code inside the function after "if parameter is None" is just fine. Having none-aware operators would make such code more succinct.
Again, that only works if None is not a valid parameter value, and PEP
505 doesn't generalize to other sentinels at all.
If the code is too long, don't put it into the parameter default. But
"if x is None: x = []" can easily be replaced with "x=>[]" and the
parameters will actually become shorter.
ANY feature can be used badly. You can daisy-chain everything into a
gigantic expression with horrendous abuses of lambda functions, but
that doesn't mean you should. It also doesn't mean that lambda
functions are bad :)
* People nearly always avoid writing code in the parameter defaults themselves, and this new practice adds a lot of cognitive load. E.g., people rarely write:
def f(x: int = 1+g()) -> None: ...
Parameter lists are already busy enough with parameter names, annotations, and defaults. We don't need to encourage this practice.
They don't become any more noisy by being able to use defaults that
aren't constant. The main goal is to put the function header in the
function header, and the function body in the function body, instead
of having the function body doing part of the work of the header :)
In short, I think this is a creative idea, a great exploration. While optional parameters are common, and some of them have defaults that are calculated inside the function, my feeling is that people will continue to set their values inside the function.
In some cases, they certainly will. And that's not a problem. But for
the cases where it's better to put it in the header, it's better to
have that option than to restrict it for an arbitrary technical
limitation.
Some languages don't have ANY function defaults. They simply have
"optional arguments", where an omitted optional argument will always
and only have a specific null value. Python allows you to show what
the default REALLY is, and that's an improvement. I want to be able to
show what the default is, even if that is defined in a non-constant
way.
ChrisA
Exactly my point. Beyond your arrow syntax for default parameters, and
the existing return annotation use, there are two other hypothetical
proposals for arrow syntax on the table:
- using `->` as an alias for typing.Callable;
- using `=>` as a more compact lambda;
That confuses me. Why would they use different arrows?
If we have a compact lambda syntax, why can't actual functions be used
as annotations to represent functions? I believe there's plans to have
[str] mean List[str], which makes a lot of sense. Why not have "lambda
str: int" or "str=>int" be an annotation too?
But okay. Supposing that annotations have to use one arrow and lambda
functions use another, then yes, this is a good reason to use "=:" for
late-bound defaults. It's not that big a change.
If nothing else, in the first case (early binding) you get the same
function object every time. In the second, you get a freshly made
function object each time. Since function objects are mutable (they have
a writable `__dict__` that's a visible difference even if the bodies are
identical. And they may not be.
Yyyyyes. Okay. There is a technical way in which you might want this
alternate behaviour. Is that REALLY something you're planning on doing
- having a function which takes another function as an argument, and
which behaves differently based on whether it's given the same
function every time or multiple different functions with the same
behaviour?
But even if it is unnecessary, it will still be permitted, just as we
will be able to write:
# Now this *actually is* totally unnecessary use of late-binding
def func(arg=>None):
Yes. That one is absolutely unnecessary, since there is no way you'll
ever get back a different result. Except that you could then mutate
func's dunders and change the behaviour. So in a purely technical
sense, nothing can be called "unnecessary".
Now, in real terms: late-binding a lambda function with no default
arguments seems like a pretty pointless thing to do. I stand by my
original statement :) And I also stand by my original statement that
proper use of the space bar can not only improve readability, it can
also help you find a date with space girls and space guys... or maybe
that last part only works on Mars.
xs=>expression # presumably a lambda function
def process(args)->int # return value annotation
Why should List->int and xs=>expression use different arrows?
Because they do different things. To avoid confusion.
Chris, you wrote the PEP for the walrus operator. Why should the
assignment operator use a different symbol from the assignment
statement? Same reason that typing.Callable and lambda will likely use
different arrows.
Anyway, if you disagree, take it up with Guido, it was his suggestion to
use different arrows :-P
Fair enough, but typing also started out with List[str] and is now
looking at using list[str] and, I think, [str], because it's more
consistent :)
Wouldn't
it be much more reasonable for them to use the same one, whichever
that be? And if that does turn out to be "=>", then yes, I would be
looking at changing PEP 671 to recommend := or =: or something, for
clarity (but still an equals sign with one other symbol next to it).
Oh great, now you're going to conflict with walrus...
We ought to at least try to avoid clear and obvious conflicts between
new and existing syntax.
Using `:=` is even worse than `=>`, and `=:` is just *begging* to
confuse newcomers "why do we have THREE different assignment symbols?"
If you look at every way the equals sign is used in Python grammar,
there are a lot of subtle differences. The difference between "f(x=1)"
and "x=1" doesn't bother people, and those are exactly the same
symbol. There are only so many symbols that we can type on everyone's
keyboards.
But you're right, and that's why I would very much like to reuse an
existing symbol rather than create new ones.
It's always possible to come up with pathological code. But this is
only really as bad as you describe if it has zero spaces in it.
Otherwise, it's pretty easy to clarify which parts go where:
Or maybe we should put more in? Spaces are optional.
def process(func : List -> int => xs => expression) -> int:
...
I'm not saying that an experienced Pythonista who is a careful reader
can't work out what the meaning is. It's not *ambiguous* syntax to a
careful reader. But it's *confusing* syntax to somebody who may not be
as careful and experienced as you, or is coding late at night, or in a
hurry, or coding while tired and emotional or distracted.
We should prefer to avoid confusable syntax when we can. The interpreter
can always disambiguate assignment as an expression from assignment as a
statement, but we still chose to make it easy on the human reader by
using distinct symbols.
def process(arg>=expression)
would be totally unambiguous to the intepreter, but I trust you would
reject that because it reads like "greater than" to the human reader.
Actually that, to me, is no different from the other options - it's
still an arrow (but now more like a funnel than an arrow, which is a
bit weird). It's no more a greater-than than "=>" is.
ChrisA
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Currently, I'm thinking "absolutely not".
However, I thought the same about the walrus operator and I now miss not
being able to use it in a program that includes support for Python 3.6 and
where I have literally dozens of places where I would use it if I could.
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
*Perhaps* if a keyword would be used instead of symbols, I might reconsider.
I find the emphasis of trying to cram too much information in single lines
of code to be really a burden. Many years ago, I argued very unsuccessfully
for using a 'where:' code block for annotations. (To this day, I still
believe it would make the code much more readable, at the cost of a slight
duplication.) Using what is at first glance a cryptic operator like => for
late binding is not helping readability, especially when type annotations
are thrown in the mix.
Aside: at the same time, I can see how using => instead of lambda as a
potential win in readability, including for beginners.
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
Sorry, I'm not interested enough at this point but, given the amount of
work you put into this, I decided that the least I could do is provide
feedback rather than be a passive reader.
André Roberge
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Currently, I'm thinking "absolutely not".
However, I thought the same about the walrus operator and I now miss not being able to use it in a program that includes support for Python 3.6 and where I have literally dozens of places where I would use it if I could.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
Yes.
I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.
Also fair. I'm not a fan of keywords for this sort of thing, since it
implies that you could do this:
def f(x=defer []): ...
dflt = defer []
def f(x=dflt): ...
which is a completely different proposal (eg it would be evaluated
only when you "touch" that, rather than being guaranteed to be
evaluated before the first line of the function body). That's why I
want to adorn the equals sign and nothing else.
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
*Perhaps* if a keyword would be used instead of symbols, I might reconsider.
I find the emphasis of trying to cram too much information in single lines of code to be really a burden. Many years ago, I argued very unsuccessfully for using a 'where:' code block for annotations. (To this day, I still believe it would make the code much more readable, at the cost of a slight duplication.) Using what is at first glance a cryptic operator like => for late binding is not helping readability, especially when type annotations are thrown in the mix.
Aside: at the same time, I can see how using => instead of lambda as a potential win in readability, including for beginners.
It's interesting how different people's views go on that sort of
thing. It depends a lot on how much people expect to use something.
Features you use a lot want to have short notations, features you
seldom use are allowed to have longer words.
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
Sorry, I'm not interested enough at this point but, given the amount of work you put into this, I decided that the least I could do is provide feedback rather than be a passive reader.
That's absolutely fine. I don't by any means expect everyone to be
able or willing to compile CPython. Feedback is absolutely
appreciated, and I asked the question with full expectation of getting
a few "No" responses :)
Thank you for taking the time to respond. Thoughtful feedback is
incredibly helpful.
ChrisA
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
When needed, but I'm uncomfortable with the syntax.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
Yes. A lot of arrows with different meanings require the reader of the code to know and remember what each does. Something more distinctive feels better.
I'm way too new to Python to have much say, but anyway, arrows make more sense for other uses. Late binding feels to me like it is best served by something more distinctive.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
I'd certainly use it only when I felt it was absolutely necessary. The readability is not aided by utilising syntax to alter behaviour when the altered behaviour is not required. We only quote, escape, etc., when we need to. (Silly example to illustrate, I know.)
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
When I first saw this (very few months ago indeed!!), I wondered about something rather clunkier but more explicit, such as, for example --
```def foo(x=__late__['bar']):
def bar():
#something
etc...
```
Ugly though. And I subsequently realised some use cases are not well-served by this. (Particularly when [speculatively] considering some implementation issues.)
The point in any case, is that for the occasions when the mechanism is needed, then a clearly explicit way to indicate something is evaluated late, and a tidy place to put the definition of how it's handled (late), seemed to me to be preferable.
Of the **cited** examples, I find the `@hi=len(a)` spelling most readable.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
Nope, I don't think this is an undue cognitive burden. If anything I think
the symmetry between the proposed '=>' syntax and the arrow syntax for
lambdas in other languages (potentially even in python in the future)
reduces the cognitive burden significantly, given the there is an
equivalent symmetry with their semantics (in both cases the code is being
evaluated later when something is called).
Steven gave the following example of a function signature that would be
difficult to visually parse if this proposal and arrow lambdas were
accepted:
def process(func:List->int=>xs=>expression)->int:
And while I agree that it does sort of stop you in your tracks when you see
this, I think there are a couple of reasons why this is not as big of a
problem as it appears.
Firstly, I think if you're writing this sort of code, you can improve the
legibility a lot by using appropriate spacing and adding parentheses around
the lambda arguments (even if they're unnecessary for a lambda with only a
single argument, like in Steven's example):
def process(func: (List) -> int => (xs) => expression) -> int:
Personally I find this much easier to visually parse since I'm used to
seeing lambdas in the form:
(*args) => expr
So my brain just kind of groups `(List) -> int` and `(xs) => expression`
into atomic chunks fairly effortlessly. And this is before even mentioning
the *massive* cognitive assistance provided by syntax highlighting :)
But at this point I would also like to point out that code like this is
probably going to be vanishingly rare in the wild. The majority of
use-cases for late-bound defaults as far as I can tell are going to be for
mutable defaults (and maybe some references to early-bound arguments). I
don't like the idea of compromising on what is a very clean and intuitive
syntax in 99% of cases to cater to the hypothetical 1% of cases where it
admittedly takes a bit more cognitive effort to parse.
I agree with the statement Chris made above where he mentioned that the
same symbols sometimes have different meanings based on context, and that's
okay. For example:
def some_func(some_arg: int) -> int: ...
def another_func(another_arg: int = some_func(some_arg=3)) -> int: ...
In the second function definition above you've got the '=' symbol pulling
double-duty both as declaring an argument for a function def and as a
keyword arg for a function call in the same statement. I think this is
perfectly fine, and I think the same thing is true of the => symbol being
used for both lambdas and late-bound arguments, especially given they are
semantically related.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
(a) is the primary use-case I can see myself using this feature for. I'm
not sure what you mean by (b). I can also definitely see some situations
where (c) would be dead useful, though I don't think this would come up in
the sort of code I write as much as (a). Still, when the use-case for (c)
did present itself, I would be *extremely* grateful to have late-bound
defaults to reach for.
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Currently, I'm thinking "absolutely not".
However, I thought the same about the walrus operator and I now miss not
being able to use it in a program that includes support for Python 3.6 and
where I have literally dozens of places where I would use it if I could.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
Yes.
I really think that using a keyword like defer, or from_calling_scope
;-), would significantly reduce the cognitive burden.
Also fair. I'm not a fan of keywords for this sort of thing, since it
implies that you could do this:
def f(x=defer []): ...
dflt = defer []
def f(x=dflt): ...
which is a completely different proposal (eg it would be evaluated
only when you "touch" that, rather than being guaranteed to be
evaluated before the first line of the function body). That's why I
want to adorn the equals sign and nothing else.
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
*Perhaps* if a keyword would be used instead of symbols, I might
reconsider.
I find the emphasis of trying to cram too much information in single
lines of code to be really a burden. Many years ago, I argued very
unsuccessfully for using a 'where:' code block for annotations. (To this
day, I still believe it would make the code much more readable, at the cost
of a slight duplication.) Using what is at first glance a cryptic operator
like => for late binding is not helping readability, especially when type
annotations are thrown in the mix.
Aside: at the same time, I can see how using => instead of lambda as a
potential win in readability, including for beginners.
It's interesting how different people's views go on that sort of
thing. It depends a lot on how much people expect to use something.
Features you use a lot want to have short notations, features you
seldom use are allowed to have longer words.
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
Sorry, I'm not interested enough at this point but, given the amount of
work you put into this, I decided that the least I could do is provide
feedback rather than be a passive reader.
That's absolutely fine. I don't by any means expect everyone to be
able or willing to compile CPython. Feedback is absolutely
appreciated, and I asked the question with full expectation of getting
a few "No" responses :)
Thank you for taking the time to respond. Thoughtful feedback is
incredibly helpful.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
YES!
A few weeks later than the prior long discussion that I read in full, it
took a triple take not to read it as >= (which would mean something
syntactical in many cases, just not what is intended).
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
I would always recommend against its use if I had any influence on code
review.
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
Yes, the delay/later/defer keyword approach is not confusing, and does not
preempt a later feature that would actually be worth having.
5) Do you know how to compile CPython from source, and would you be
I do know how, but it is unlikely I'll have time.
--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Probably not. Mainly because I don't have any real use for it rather
than because I have any inherent problem with it (but see below, the
more I thought about it the more uncomfortable with it I became).
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
N/A, except to say that when you enumerate the use cases like this,
none of them even tempt me to use this feature. I think that the only
thing I might use it for is to make it easier to annotate defaults (as
f(a: list[int] => []) rather than as f(a: list[int] | None = None).
So I'll revise my answer to (1) and say that I *might* use this, but
only in a way it wasn't intended to be used in, and mostly because I
hate how verbose it is to express optional arguments in type
annotations. (And the fact that the annotation exposes the sentinel
value, even when you want it to be opaque). I hope I don't succumb and
do that, though ;-)
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
Sorry, I really don't have time to, in the foreseeable future. If I
did have time, one thing I would experiment with is how this interacts
with typing and tools like pyright and mypy (yes, I know type checkers
would need updating for the new syntax, so that would mostly be a
thought experiment) - as I say, I'd expect to annotate a function with
an optional list argument defaulting to an empty list as f(a:
list[int] => []), which means that __annotations__ needs to
distinguish between this case and f(a: list[int]) with no default.
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
Sorry, I don't have any feedback like that. What I can say, though, is
I'd find it quite hard to express the question, in the sense that I'd
struggle to explain the difference between early and late bound
parameters to a non-expert, much less explain why we need both. I'd
probably just say "it's short for a default of None and a check" which
doesn't really capture the point...
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
Yes.
I really think that using a keyword like defer, or from_calling_scope
;-), would significantly reduce the cognitive burden.
Also fair. I'm not a fan of keywords for this sort of thing, since it
implies that you could do this:
def f(x=defer []): ...
dflt = defer []
def f(x=dflt): ...
which is a completely different proposal (eg it would be evaluated
only when you "touch" that, rather than being guaranteed to be
evaluated before the first line of the function body). That's why I
want to adorn the equals sign and nothing else.
Shouldn't the PEP contain a rejected idea section where this could be
mentioned?
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
*Perhaps* if a keyword would be used instead of symbols, I might
reconsider.
I find the emphasis of trying to cram too much information in single
lines of code to be really a burden. Many years ago, I argued very
unsuccessfully for using a 'where:' code block for annotations. (To this
day, I still believe it would make the code much more readable, at the cost
of a slight duplication.) Using what is at first glance a cryptic operator
like => for late binding is not helping readability, especially when type
annotations are thrown in the mix.
Aside: at the same time, I can see how using => instead of lambda as a
potential win in readability, including for beginners.
It's interesting how different people's views go on that sort of
thing. It depends a lot on how much people expect to use something.
Features you use a lot want to have short notations, features you
seldom use are allowed to have longer words.
I rarely use lambda in my own code, and have never written a line of code
anywhere that uses a '=>' operator.
If Python had a 'function' keyword instead of 'lambda', I would prefer to
keep the function keyword instead of adding => as a symbol. For me, it is
not a question of terseness for commonly used features, but one of easing
the learning curve. Starting from zero, I do believe that => would be
easier to grasp than learning about lambda as a keyword and the syntactic
rules to use with it. With function as a keyword, I believe that it is the
other way around. No doubt many others will disagree!
André Roberge
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
Yes, the delay/later/defer keyword approach is not confusing, and does not preempt a later feature that would actually be worth having.
Do you mean changing the spelling of the existing proposal, or a
completely different proposal for deferred objects that are actually
objects? Because that is NOT what I mean by a "small change". :)
ChrisA
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
N/A, except to say that when you enumerate the use cases like this,
none of them even tempt me to use this feature.
I'm just listing some of the common use-cases, not all of them.
I think that the only
thing I might use it for is to make it easier to annotate defaults (as
f(a: list[int] => []) rather than as f(a: list[int] | None = None).
That's exactly the goal! It's hugely simpler to say "this is always a
list, and defaults to a new empty list" than "this is a list or None,
defaults to None, and hey, if it's None, I'll make a new empty list".
So I'll revise my answer to (1) and say that I *might* use this, but
only in a way it wasn't intended to be used in, and mostly because I
hate how verbose it is to express optional arguments in type
annotations. (And the fact that the annotation exposes the sentinel
value, even when you want it to be opaque). I hope I don't succumb and
do that, though ;-)
That's definitely an intended use-case ("mutable defaults" above);
you're focusing on the fact that it improves the annotations, I'm
focusing on the fact that it improves documentation and introspection,
but it's all the same improvement :)
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
Sorry, I really don't have time to, in the foreseeable future. If I
did have time, one thing I would experiment with is how this interacts
with typing and tools like pyright and mypy (yes, I know type checkers
would need updating for the new syntax, so that would mostly be a
thought experiment) - as I say, I'd expect to annotate a function with
an optional list argument defaulting to an empty list as f(a:
list[int] => []), which means that __annotations__ needs to
distinguish between this case and f(a: list[int]) with no default.
Sorry, I don't have any feedback like that. What I can say, though, is
I'd find it quite hard to express the question, in the sense that I'd
struggle to explain the difference between early and late bound
parameters to a non-expert, much less explain why we need both. I'd
probably just say "it's short for a default of None and a check" which
doesn't really capture the point...
The point is that it's a shortcut for "omitted" rather than "a default
of None", but there are many ways to explain it, and I only have one
brother who knows enough Python to be able to talk to about this (and
I got feedback from him VERY early).
I think that the only
thing I might use it for is to make it easier to annotate defaults (as
f(a: list[int] => []) rather than as f(a: list[int] | None = None).
Why not `f(a: Optional[list[int]] = None)`?
I'm not counting characters, but that form seems to express the intention
better than either of the others IMHO.
--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
Yes.
I really think that using a keyword like defer, or from_calling_scope ;-), would significantly reduce the cognitive burden.
Also fair. I'm not a fan of keywords for this sort of thing, since it
implies that you could do this:
def f(x=defer []): ...
dflt = defer []
def f(x=dflt): ...
which is a completely different proposal (eg it would be evaluated
only when you "touch" that, rather than being guaranteed to be
evaluated before the first line of the function body). That's why I
want to adorn the equals sign and nothing else.
Shouldn't the PEP contain a rejected idea section where this could be mentioned?
Hmm, maybe. It's such a completely different proposal, but it does get
asked a few times. If someone could actually put it forward as a full
proposal, I'd gladly mention it as an interaction with another PEP.
Otherwise, I'll write up a very brief thing on deferred expressions
and how they're not what this is about.
ChrisA
I think that the only
thing I might use it for is to make it easier to annotate defaults (as
f(a: list[int] => []) rather than as f(a: list[int] | None = None).
Why not `f(a: Optional[list[int]] = None)`?
I'm not counting characters, but that form seems to express the intention better than either of the others IMHO.
"a: list[int] => []" fully expresses the intention IMO, but yes,
Optional is usually better than "| None". That's just a minor spelling
problem though.
ChrisA
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
Yes, the delay/later/defer keyword approach is not confusing, and does
not preempt a later feature that would actually be worth having.
Do you mean changing the spelling of the existing proposal, or a
completely different proposal for deferred objects that are actually
objects? Because that is NOT what I mean by a "small change". :)
The spelling of the existing proposal. I.e. if the proposal were:
def fun(things: list[int] = defer []) -> int:
# ... some implementation
I'd be -0 on the idea rather than -100.
Yes, my change in attitude is largely because I want *some future PEP* to
address the more general situation like:
result = defer really_expensive_calculation()
if predicate:
doubled = result * 2
But I do not think your PEP does (nor even should) include that potential
future behavior/syntax. Such a hypothetical future PEP would have a
continuity with the syntax of your feature, albeit DEFINITELY need to
address many independent concerns/issues that yours does not create.
However, even if I assume the mythical future PEP never happens, in terms
of readability, a WORD is vastly less confusing than a combination of
punctuation that has no obvious or natural interpretation like '=>'. Or
rather, I think that spelling is kinda-sorta obvious for the lambda
meaning, and the use you want is kinda-sorta similar to a lambda. So I
*do* understand how you get there... but it still seems like much too much
line noise for a very minimal need.
--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
I think that the only
thing I might use it for is to make it easier to annotate defaults (as
f(a: list[int] => []) rather than as f(a: list[int] | None = None).
Why not `f(a: Optional[list[int]] = None)`?
I'm not counting characters, but that form seems to express the intention better than either of the others IMHO.
If None were a valid argument, and I was using an opaque sentinel,
Optional doesn't work, and exposing the type of the sentinel is not
what I intend (as it's invalid to explicitly supply the sentinel
value).
Also, Optional[list[int]] doesn't express the intent accurately - the
intended use is that people must supply a list[int] or not supply the
argument *at all*. Optional allows them to supply None as well.
As I say, I don't consider this an intended use case for the feature,
because what I'm actually discussing here is optional arguments and
sentinels, which is a completely different feature. All I'm saying is
that the only case when I can imagine using this feature is for when I
want a genuinely opaque way of behaving differently if the caller
omitted an argument (and using None or a sentinel has been good enough
all these years, so it's not exactly a pressing need).
Let's just go back to the basic point, which is that I can't think of
a realistic case where I'd want to actually use the new feature.
Paul
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
No, but it will be cognitive burden with shorthand lambda proposed
syntax, for example
def x(a: (b, c)=>c):
is annotation for a (b, c) or maybe (b, c)=>c
(It's absolutely valid to say "yes" and "yes", and feel free to say
which of those pulls is the stronger one.)
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
While I answered yes to question 1, personally I would prefer not
adding new syntax, but switching completly to late defaults (requiring
future import for some next versions)
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
Yes, the delay/later/defer keyword approach is not confusing, and does not preempt a later feature that would actually be worth having.
Do you mean changing the spelling of the existing proposal, or a
completely different proposal for deferred objects that are actually
objects? Because that is NOT what I mean by a "small change". :)
The spelling of the existing proposal. I.e. if the proposal were:
Yes, my change in attitude is largely because I want *some future PEP* to address the more general situation like:
result = defer really_expensive_calculation()
if predicate:
doubled = result * 2
But I do not think your PEP does (nor even should) include that potential future behavior/syntax. Such a hypothetical future PEP would have a continuity with the syntax of your feature, albeit DEFINITELY need to address many independent concerns/issues that yours does not create.
However, even if I assume the mythical future PEP never happens, in terms of readability, a WORD is vastly less confusing than a combination of punctuation that has no obvious or natural interpretation like '=>'. Or rather, I think that spelling is kinda-sorta obvious for the lambda meaning, and the use you want is kinda-sorta similar to a lambda. So I *do* understand how you get there... but it still seems like much too much line noise for a very minimal need.
The trouble is that this actually would be incompatible. If you can
defer an expensive calculation and have some "placeholder" value in
the variable 'result', then logically, you should be able to have that
placeholder as a function default argument, which would be an
early-bound default of the placeholder. That is quite different in
behaviour from a late-bound default, so if I were to use the word
"defer" for late-bound defaults, it would actually prevent the more
general proposal.
ChrisA
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
(It's absolutely valid to say "yes" and "yes", and feel free to say
which of those pulls is the stronger one.)
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
yes (a)
What does (b) mean? example please.
yes (c)
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
Any and all comments welcomed. I mean, this is python-ideas after
all... bikeshedding is what we do best!
The reference implementation currently has some test failures, which
I'm looking into. I'm probably going to make this my personal default
Python interpreter for a while, to see how things go.
On 1 Dec 2021, at 06:16, Chris Angelico wrote:
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
yes (a)
What does (b) mean? example please.
yes (c)
global_default = 500
def do_thing(timeout=>global_default): ...
If the global_default timeout changes between function definition and
call, omitting timeout will use the updated global.
Similarly, you could say "file=>sys.stdout" and if code elsewhere
changes sys.stdout, you'll use that.
ChrisA
On 1 Dec 2021, at 06:16, Chris Angelico wrote:
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
yes (a)
What does (b) mean? example please.
yes (c)
If the global_default timeout changes between function definition and
call, omitting timeout will use the updated global.
Similarly, you could say "file=>sys.stdout" and if code elsewhere
changes sys.stdout, you'll use that.
On a case-by-case basis I might still put defaulting into the body
of the function if that made the intent clearer.
I could see me using @file=sys.stdout.
Barry
On 1 Dec 2021, at 06:16, Chris Angelico wrote:
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
yes (a)
What does (b) mean? example please.
yes (c)
If the global_default timeout changes between function definition and
call, omitting timeout will use the updated global.
Similarly, you could say "file=>sys.stdout" and if code elsewhere
changes sys.stdout, you'll use that.
On a case-by-case basis I might still put defaulting into the body
of the function if that made the intent clearer.
I could see me using @file=sys.stdout.
That's a simplified version, but you might have the same default shown
as get_default_timeout() or Defaults.timeout or something like that.
The point is that it might be a simple integer, but it could change at
any time, and the default should always be "whatever this name refers
to".
In any case, it's just one of many use-cases. I was curious what
people would use and what they wouldn't.
ChrisA
the
intended use is that people must supply a list[int] or not supply the
argument *at all*.
I don't think this is a style of API that we should be encouraging
people to create, because it results in things that are very
awkward to wrap.
--
Greg
To be honest, I like the `@param=value` syntax. It is sort of easier to read than `param=>value`, though I do not have problems with distinguishing the arrow `=>` from other things.
If anything I think
the symmetry between the proposed '=>' syntax and the arrow syntax for
lambdas in other languages (potentially even in python in the future)
reduces the cognitive burden significantly, given the there is an
equivalent symmetry with their semantics (in both cases the code is being
evaluated later when something is called).
There is not as much symmetry as you might think between a hypothetical
lambda arrow and the proposed late-bound default arrow.
arg => arg + 1 # lambda
arg=>expr # late-bound default
The first case is (or could be some day...) an actual expression that
returns a function object, which we will explicitly call at some point.
Or at least pass the function to another function, which will call it.
But the late bound default is not an expression, it is a declaration. It
declares the default value used for arg. We don't have to call anything
to get access to the default value. It just shows up when we access the
parameter without providing an argument for it.
We certainly don't need to call arg explicitly to evaluate the default.
It may be that behind the scenes the default expression is stored as a
callable function which the interpreter calls. (I believe that list
comprehensions do something similar.) But that's an implementation
detail that can change: it might just as well store the source code as a
string, and pass it to eval().
Or use some other mechanism that I'm not clever enough to think of, so I
shall just call "deepest black magic".
There is no *neccessity* for the late-bound default to be a hidden
function, and it is certainly not part of the semantics of late-bound
defaults. Just the implementation.
If you disagree, and still think that the symmetry is powerful enough to
use the same syntax for both lambdas and default arguments, well, how
about if we *literally* do that?
def function(spam=expression, # regular default
lambda eggs: expression, # late-bound default
)
Heh, perhaps the symmetry is not that strong after all :-)
--
Steve
If anything I think
the symmetry between the proposed '=>' syntax and the arrow syntax for
lambdas in other languages (potentially even in python in the future)
reduces the cognitive burden significantly, given the there is an
equivalent symmetry with their semantics (in both cases the code is being
evaluated later when something is called).
It may be that behind the scenes the default expression is stored as a
callable function which the interpreter calls. (I believe that list
comprehensions do something similar.) But that's an implementation
detail that can change: it might just as well store the source code as a
string, and pass it to eval().
In my reference implementation, there is no object that stores it;
it's simply part of the function. A good parallel is the if/else
expression:
x = float("inf") if z == 0 else y/z
Is there an object that represents the if/else expression (or the
deferred "y/z" part)? No, although it could be implemented that way if
you chose:
x = iff(z == 0, lambda: y/z, lambda: float("inf"))
But just because you CAN use a function to simulate this behaviour,
that doesn't mean that it's inherently a function.
So I agree with Steve that the parallel is quite weak. That said,
though, I still like the arrow notation, not because of any parallel
with a lambda function, but because of the parallel with assignment.
(And hey. You're allowed to think of things in any way you like. I'm
not forcing you to interpret everything using the same justifications
I do.)
There is no *neccessity* for the late-bound default to be a hidden
function, and it is certainly not part of the semantics of late-bound
defaults. Just the implementation.
If you disagree, and still think that the symmetry is powerful enough to
use the same syntax for both lambdas and default arguments, well, how
about if we *literally* do that?
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Yes I would, but probably not as often as counting cases of the "if
param is None: ..." idiom might lead you to expect.
The problem is, as Neil also pointed out, that it becomes tricky to
explicitly ask for the default behaviour except by leaving the argument
out altogether. Not impossible, as Chris mentions elsewhere, you can
mess about with `*args` or `**kwargs`, but it is decidedly less
convenient, more verbose, and likely to have a performance hit.
So if I were messing about in the interactive interpreter, I would
totally use this for the convenience:
def func(L=>[]): ...
but if I were writing a library, I reckon that probably at least half
the time I'll stick to the old idiom so I can document the parameter:
"If param is missing **or None**, the default if blah..."
I reject Chris' characterisation of this as a hack. There are function
parameters where None will *never* in any conceivable circumstances
become a valid argument value, and it is safe to use it as a sentinel.
For example, I have a function that takes a `collation=None` parameter.
The collation is a sequence of characters, usually a string. Using None
to indicate "use the default collation" will never conflict with some
possible future use of "use None as the actual collation" because None
is not a sequence of characters.
In this specific case, my function's current signature is:
def itoa(n, base,
collation=None,
plus='',
minus='-',
width=0,
):
I could re-write it as:
def itoa(n, base,
collation=>_default_collation(base),
plus='',
minus='-',
width=0,
):
but that would lose the ability to explicitly say "use the default
collation" by passing None. So I think that, on balance, I would
likely stick to the existing idiom for this function.
On the other hand, if None were a valid value, so the signature used a
private and undocumented sentinel:
collation=_MISSING
(and I've used that many times in other functions) then I would be
likely to swap to this feature and drop the private sentinel.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
There's lots of Python features that I use even though I don't like the
spelling. Not a day goes by that I don't wish we spelled the base class
of the object hierarchy "Object", so it were easier to distinguish
between Object the base class and object as a generic term for any
object.
If the Steering Council loves your `=>` syntax then I would disagree
with their decision but still use it.
--
Steve
"If param is missing **or None**, the default if blah..."
I reject Chris' characterisation of this as a hack. There are function
parameters where None will *never* in any conceivable circumstances become
a valid argument value, and it is safe to use it as a sentinel.
In particular, the cases where None will never, in any conceivable
circumstances, be a non-sentinel value are at least 98% of all the
functions (that have named parameters) I've ever written in Python.
All of this discussion of a syntax change is for, at most, 2% of functions
that need a different sentinel.
I like the => syntax and would use it probably 80% of the time for mutable
defaults.
I don't think it causes cognitive load.
I especially disagree with the argument that it looks like arbitrary
symbols and adds more 'line noise' than a keyword. My eye picks it up as a
distinct arrow rather than a random assembly of punctuation. Maybe it's not
as hard as people claim to visually parse strings of punctuation that make
little pictures ¯\_(ツ)_/¯
I also disagree that it makes Python harder to teach or learn. You *have
to* teach students about early binding anyway. That's the hard part.
Explaining how parameter binding works so that they can avoid the pit traps
is the hard part. Then you either show them a verbose and clumsy work
around of checking for a sentinel or you show them a cute little arrow.
I think people are missing the overall point of doing in the function
header what belongs in the function header so that it doesn't spill into
the body.
My favorite alternative is ?= if people think => and -> are getting
overly loaded. What I really don't like is @param=[] because it puts the
emphasis on the parameter name rather than the act of binding. Not only
does it make it look like @param is a special kind of variable, it also
mimics the *args and **kwargs syntax which makes them seem related.
On Wed, Dec 1, 2021, 8:36 PM Chris Angelico wrote:
If anything I think
the symmetry between the proposed '=>' syntax and the arrow syntax for
lambdas in other languages (potentially even in python in the future)
reduces the cognitive burden significantly, given the there is an
equivalent symmetry with their semantics (in both cases the code is
It may be that behind the scenes the default expression is stored as a
callable function which the interpreter calls. (I believe that list
comprehensions do something similar.) But that's an implementation
detail that can change: it might just as well store the source code as a
string, and pass it to eval().
In my reference implementation, there is no object that stores it;
it's simply part of the function. A good parallel is the if/else
expression:
x = float("inf") if z == 0 else y/z
Is there an object that represents the if/else expression (or the
deferred "y/z" part)? No, although it could be implemented that way if
you chose:
x = iff(z == 0, lambda: y/z, lambda: float("inf"))
But just because you CAN use a function to simulate this behaviour,
that doesn't mean that it's inherently a function.
So I agree with Steve that the parallel is quite weak. That said,
though, I still like the arrow notation, not because of any parallel
with a lambda function, but because of the parallel with assignment.
(And hey. You're allowed to think of things in any way you like. I'm
not forcing you to interpret everything using the same justifications
I do.)
There is no *neccessity* for the late-bound default to be a hidden
function, and it is certainly not part of the semantics of late-bound
defaults. Just the implementation.
If you disagree, and still think that the symmetry is powerful enough to
use the same syntax for both lambdas and default arguments, well, how
about if we *literally* do that?
Steven D'Aprano
""If param is missing **or None**, the default if blah..."
I reject Chris' characterisation of this as a hack. There are function
parameters where None will *never* in any conceivable circumstances
become a valid argument value, and it is safe to use it as a sentinel."
Yes, we know *why* the hack works. We're all familiar with it. That doesn't
mean it's not a hack.
The bottom line is:
you *don't actually* want the parameter to default to the value of a
sentinel.
you *have* to use that hack because you can't express what you want the
default to actually be.
You're doing something misleading to work around a shortcoming of the
language.
That's a hack.
You have to write something that you don't actually intend.
On Wednesday, December 1, 2021 at 9:39:12 PM UTC-6 Steven D'Aprano wrote:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Yes I would, but probably not as often as counting cases of the "if
param is None: ..." idiom might lead you to expect.
The problem is, as Neil also pointed out, that it becomes tricky to
explicitly ask for the default behaviour except by leaving the argument
out altogether. Not impossible, as Chris mentions elsewhere, you can
mess about with `*args` or `**kwargs`, but it is decidedly less
convenient, more verbose, and likely to have a performance hit.
So if I were messing about in the interactive interpreter, I would
totally use this for the convenience:
def func(L=>[]): ...
but if I were writing a library, I reckon that probably at least half
the time I'll stick to the old idiom so I can document the parameter:
"If param is missing **or None**, the default if blah..."
I reject Chris' characterisation of this as a hack. There are function
parameters where None will *never* in any conceivable circumstances
become a valid argument value, and it is safe to use it as a sentinel.
For example, I have a function that takes a `collation=None` parameter.
The collation is a sequence of characters, usually a string. Using None
to indicate "use the default collation" will never conflict with some
possible future use of "use None as the actual collation" because None
is not a sequence of characters.
In this specific case, my function's current signature is:
but that would lose the ability to explicitly say "use the default
collation" by passing None. So I think that, on balance, I would
likely stick to the existing idiom for this function.
On the other hand, if None were a valid value, so the signature used a
private and undocumented sentinel:
collation=_MISSING
(and I've used that many times in other functions) then I would be
likely to swap to this feature and drop the private sentinel.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
There's lots of Python features that I use even though I don't like the
spelling. Not a day goes by that I don't wish we spelled the base class
of the object hierarchy "Object", so it were easier to distinguish
between Object the base class and object as a generic term for any
object.
If the Steering Council loves your `=>` syntax then I would disagree
with their decision but still use it.
2% of functions is a lot of functions. We're talking about a language
that's been around 30 years. The feature set is pretty mature. If it were
lacking features that would improve a much larger percent of code for so
long, I don't think it would be as popular. It's not like the next PIP is
going to be as big as the for-loop.
On Wednesday, December 1, 2021 at 9:56:40 PM UTC-6 David Mertz, Ph.D. wrote:
"If param is missing **or None**, the default if blah..."
I reject Chris' characterisation of this as a hack. There are function
parameters where None will *never* in any conceivable circumstances become
a valid argument value, and it is safe to use it as a sentinel.
In particular, the cases where None will never, in any conceivable
circumstances, be a non-sentinel value are at least 98% of all the
functions (that have named parameters) I've ever written in Python.
All of this discussion of a syntax change is for, at most, 2% of functions
that need a different sentinel.
Exactly like you. I can read => to mean late-bound or supposedly lambda function and I can distinguish it from the callable typing -> but "@param=value” is definitely better in my eyes. => won’t prevent me from using it for default mutable objects.
To be honest, I like the `@param=value` syntax. It is sort of easier to read than `param=>value`, though I do not have problems with distinguishing the arrow `=>` from other things.
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
I hope not. :-)
Realistically I might wind up using it at some point way down the line.
I wouldn't start using it immediately. I still almost never use the
walrus and only occasionally use f-strings.
You didn't ask how people would feel about READING this rather than
writing it, but what I would do is get really annoyed at seeing it in
code as people start to use it and confuse me and others.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
No. As I mentioned in the earlier thread, I don't support any proposal
in which an argument can "have a default" but that default is not a
first-class Python object of some sort.
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
No, I don't.
I know I said this before, but I really hope this change is not
adopted. It is to me a classic example of adding significant complexity
to the language and reducing readability for only a very small benefit
in expressiveness.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
In my reference implementation, there is no object that stores it;
it's simply part of the function. A good parallel is the if/else
expression:
x = float("inf") if z == 0 else y/z
Is there an object that represents the if/else expression (or the
deferred "y/z" part)? No, although it could be implemented that way if
you chose:
This is not a good parallel. There is nothing deferred there. The
entire line is evaluated when it is encountered and you get a result and
no part of the if/else expression can ever impact anything else again
unless, via some external control flow, execution returns and
re-executes the entire line. That is not comparable to a function
default, which is STORED and evaluated later independently of the
context in which it was originally written (i.e., the function default
is re-executed but the function definition itself is not re-executed).
The ternary expression vanishes without a trace by the next line,
leaving only its evaluated result. There would be no use in being able
to access some part of it, since the whole (i.e., the ternary
expression) is completely finished by the time you would be able to
access it. This is not the case with a function definition. The
function definition leaves behind a function object, and that function
object needs to "know" about the late-bound default as an independent
entity so that it can be evaluated later. It is bad for the function to
store that late-bound default only in some private format for its
exclusive future use without providing any means for other code to
access it as a first-class value.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
No. As I mentioned in the earlier thread, I don't support any proposal
in which an argument can "have a default" but that default is not a
first-class Python object of some sort.
What if a default is a function?
I was inspired by learning Django and saw in models that fields can have a default which is either a regular (early-bound) default such as a first-class Python object as one would expect, *or* a function -- which will be called 'later' when needed.
That prompted me to contemplate a syntax for late-bound defaults, albeit a bit clunky, but I did think it suited a special-case requirement met by late-bound defaults. I still think that littering function arguments throughout all code with large numbers of arrows would make things less readable. Special requirements need special treatment, I'm thinking.
The problem is passing arguments to such a function without it looking like it's being called at definition time.
No. As I mentioned in the earlier thread, I don't support any proposal
in which an argument can "have a default" but that default is not a
first-class Python object of some sort.
What if a default is a function?
I was inspired by learning Django and saw in models that fields can have a default which is either a regular (early-bound) default such as a first-class Python object as one would expect, *or* a function -- which will be called 'later' when needed.
That prompted me to contemplate a syntax for late-bound defaults, albeit a bit clunky, but I did think it suited a special-case requirement met by late-bound defaults. I still think that littering function arguments throughout all code with large numbers of arrows would make things less readable. Special requirements need special treatment, I'm thinking.
The problem is passing arguments to such a function without it looking like it's being called at definition time.
Also has the same problem of other deferreds, which is: when exactly
is it evaluated?
def func(stuff, n=>len(stuff)):
stuff.append("spam")
print(n)
func(["spam", "ham"])
What will this print? If it's a function default, it MUST print 2,
since len(stuff) is 2 as the function starts. But if it's a deferred
object of some sort, then it should probably print 3, since len(stuff)
is 3 at the time that n gets printed. Which is it to be?
That's why PEP 671 is *not* about generic deferred evaluation. It is
specifically about function default arguments, and guarantees that
they will be evaluated prior to the first line of code in the function
body.
ChrisA
In my reference implementation, there is no object that stores it;
it's simply part of the function. A good parallel is the if/else
expression:
x = float("inf") if z == 0 else y/z
Is there an object that represents the if/else expression (or the
deferred "y/z" part)? No, although it could be implemented that way if
you chose:
This is not a good parallel. There is nothing deferred there. The
entire line is evaluated when it is encountered and you get a result and
no part of the if/else expression can ever impact anything else again
unless, via some external control flow, execution returns and
re-executes the entire line. That is not comparable to a function
default, which is STORED and evaluated later independently of the
context in which it was originally written (i.e., the function default
is re-executed but the function definition itself is not re-executed).
The ternary expression vanishes without a trace by the next line,
leaving only its evaluated result. There would be no use in being able
to access some part of it, since the whole (i.e., the ternary
expression) is completely finished by the time you would be able to
access it. This is not the case with a function definition. The
function definition leaves behind a function object, and that function
object needs to "know" about the late-bound default as an independent
entity so that it can be evaluated later. It is bad for the function to
store that late-bound default only in some private format for its
exclusive future use without providing any means for other code to
access it as a first-class value.
That's exactly why it's such a close parallel. The late-evaluated
default is just code, nothing else. It's not "stored" in any way - it
is evaluated as part of the function beginning execution.
There is no "first class object" for a late-evaluated default any more
than there is one for the "y/z" part of the ternary.
ChrisA
That's exactly why it's such a close parallel. The late-evaluated
default is just code, nothing else. It's not "stored" in any way - it
is evaluated as part of the function beginning execution.
But it IS stored! There is no way for it to be evaluated without it
being stored!
I know we're talking past each other here but it is quite obvious that
something has to be stored if it is going to be evaluated later. You
can say that it is "just code" but that doesn't change the fact that
that code has to be stored. You can say that it is just prepended to
the function body but that's still storing it. That is still not
parallel to a ternary operator in which no part of the expression is
EVER re-executed unless control flow causes execution to return to that
same source code line and re-execute it as a whole.
Actually this raises a question that maybe was answered in the earlier
thread but if so I forgot: if a function has a late-bound default, will
the code to evaluate it be stored as part of the function's code object?
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
the
intended use is that people must supply a list[int] or not supply the
argument *at all*.
I don't think this is a style of API that we should be encouraging
people to create, because it results in things that are very
awkward to wrap.
Hmm, interesting point, I agree with you. It's particularly telling
that I got sucked into designing that sort of API, even though I know
it's got this problem. I guess that counts as an argument against the
late bound defaults proposal - or maybe even two:
1. It's hard (if not impossible) to wrap functions that use late-bound defaults.
2. The feature encourages people to write such unwrappable functions
when an alternative formulation that is wrappable is just as good.
(That may actually only be one point - obviously a feature encourages
people to use it, and any feature can be over-used. But the point
about wrappability stands).
Paul
That's exactly why it's such a close parallel. The late-evaluated
default is just code, nothing else. It's not "stored" in any way - it
is evaluated as part of the function beginning execution.
But it IS stored! There is no way for it to be evaluated without it
being stored!
I know we're talking past each other here but it is quite obvious that
something has to be stored if it is going to be evaluated later. You
can say that it is "just code" but that doesn't change the fact that
that code has to be stored. You can say that it is just prepended to
the function body but that's still storing it. That is still not
parallel to a ternary operator in which no part of the expression is
EVER re-executed unless control flow causes execution to return to that
same source code line and re-execute it as a whole.
I'm not sure I understand you here. How is the late-bound default
"stored" when one side of a ternary is "not stored"?
Actually this raises a question that maybe was answered in the earlier
thread but if so I forgot: if a function has a late-bound default, will
the code to evaluate it be stored as part of the function's code object?
Yes. To be precise, it is part of the code object's co_code attribute
- the bytecode (or wordcode if you prefer) of the function.
Here's how a ternary if looks:
The "42/n" part is stored in f.__code__.co_code as the part that says
"LOAD_CONST 42, LOAD_FAST n, BINARY_TRUE_DIVIDE". It's not an object.
It's just code - three instructions.
Here's how (in the reference implementation - everything is subject to
change) a late-bound default looks:
The "=>[]" part is stored in f.__code__.co_code as the part that says
"QUERY_FAST x, and if false, BUILD_LIST, STORE_FAST x". It's not an
object. It's four instructions in the bytecode.
In both cases, no part of the expression is ever re-executed. I'm not
understanding the distinction here. Can you explain further please?
ChrisA
the
intended use is that people must supply a list[int] or not supply the
argument *at all*.
I don't think this is a style of API that we should be encouraging
people to create, because it results in things that are very
awkward to wrap.
Hmm, interesting point, I agree with you. It's particularly telling
that I got sucked into designing that sort of API, even though I know
it's got this problem. I guess that counts as an argument against the
late bound defaults proposal - or maybe even two:
1. It's hard (if not impossible) to wrap functions that use late-bound defaults.
2. The feature encourages people to write such unwrappable functions
when an alternative formulation that is wrappable is just as good.
(That may actually only be one point - obviously a feature encourages
people to use it, and any feature can be over-used. But the point
about wrappability stands).
Actually, Chris - does functools.wraps work properly in your
implementation when wrapping functions with late-bound defaults?
the
intended use is that people must supply a list[int] or not supply the
argument *at all*.
I don't think this is a style of API that we should be encouraging
people to create, because it results in things that are very
awkward to wrap.
Hmm, interesting point, I agree with you. It's particularly telling
that I got sucked into designing that sort of API, even though I know
it's got this problem. I guess that counts as an argument against the
late bound defaults proposal - or maybe even two:
1. It's hard (if not impossible) to wrap functions that use late-bound defaults.
2. The feature encourages people to write such unwrappable functions
when an alternative formulation that is wrappable is just as good.
(That may actually only be one point - obviously a feature encourages
people to use it, and any feature can be over-used. But the point
about wrappability stands).
Wrappability when using an arbitrary sentinel looks like this:
_SENTINEL = object()
def f(value=_SENTINEL):
if value is _SENTINEL: value = ...
# in another module
def wrap_f():
othermodule.f(othermodule._SENTINEL)
Or:
def wrap_f():
othermodule.f(othermodule.f.__defaults__[0])
I'm not sure either of these is any better than the alternatives.
You're reaching into a module to access its internal implementation
details. Is that really better than messing with *args?
def wrap_f():
args = [42] or [] # pass or don't pass?
othermodule.f(*args)
It only looks easy when None is used, and even then, it's largely an
implementation detail.
ChrisA
Yes, it does. There are two things happening here which, from a
technical standpoint, are actually orthogonal; the function *call* is
simply using *a,**kw notation, so it's passing along all the
positional and keyword parameters untouched; but the function
*documentation* just says "hey look over there for the real
signature".
When you do "inner = wraps(f)(inner)", what happens is that the
function's name and qualname get updated, and then __wrapped__ gets
added as a pointer saying "hey, assume that I have the signature of
that guy over there". There's unfortunately no way to merge signatures
(you can't do something like "def f(*a, timeout=500, **kw):" and then
use the timeout parameter in your wrapper and pass the rest on -
help(f) will just show the base function's signature), and nothing is
actually aware of the underlying details.
But on the plus side, this DOES work correctly.
Documentation for late-bound defaults is done by a (compile-time)
string snapshot of the AST that defined the default, so it's about as
accurate as for early-bound defaults. One thing I'm not 100% happy
with in the reference implementation is that the string for help() is
stored on the function object, but the behaviour of the late-bound
default is inherent to the code object. I'm not sure of a way around
that, but it also doesn't seem terribly problematic in practice.
ChrisA
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
No. As I mentioned in the earlier thread, I don't support any
proposal in which an argument can "have a default" but that default is not
a first-class Python object of some sort.
I don't understand this criticism.
Of course the default value will be a first-class Python object of some
sort. *Every* value in Python is a first-class object. There are no
machine values or unboxed values, and this proposal will not change
that.
All that this proposal changes is *when* and *how often* the default
will be evaluated, not the nature of the value.
Status quo: default values are evaluated once, when the def statement
is executed.
With optional late-binding: default values are evaluated as often as
they are needed, when the function is called. But the value will still
be an object.
I suppose that there will be one other change, relating to introspection
of the function. You will no longer be able to inspect the function and
see the default values as constants in a cache:
>>> (lambda x=1.25: None).__defaults__
(1.25,)
Depending on the implementation, you *might* be able to inspect the
function and see the default expression as some sort of callable
function, or evaluatable code object. (That would be nice.) Or even as a
plain old string. All of which are first-class objects. Or it might be
that the default expression will be compiled into the body of the
function, where is it effectively invisible. So I guess that's a third
change: when, how often, and the difference to introspection.
--
Steve
1. It's hard (if not impossible) to wrap functions that use late-bound
defaults.
I don't understand that argument.
We can implement late-bound defaults right now, using the usual sentinel
jiggery-pokery.
def func(spam, eggs, cheese=None, aardvark=42):
if cheese is None: ... # You know the drill.
We can wrap functions like this. You don't generally even care what the
sentinel is. A typical wrapper function looks something like this:
@functools.wraps(func)
def inner(*args, **kwargs):
# wrapper implementation
func(*args, **kwargs)
with appropriate pre-processing or post-processing as needed.
Why would argument unpacking to call the wrapped function stop working?
--
Steve
Depending on the implementation, you *might* be able to inspect the
function and see the default expression as some sort of callable
function, or evaluatable code object. (That would be nice.)
Unfortunately not, since the default expression could refer to other
parameters, or closure variables, or anything else from the context of
the called function. So you won't be able to externally evaluate it.
Or even as a plain old string. All of which are first-class objects.
For documentation purposes, it is indeed available as a plain old
string. In colloquial terms, it is the source code for the expression,
although technically it's reconstructed from the AST. ( This is
approximately as accurate as saying that the repr of an object is its
source code. Possibly more accurate, actually.)
Or it might be
that the default expression will be compiled into the body of the
function, where is it effectively invisible. So I guess that's a third
change: when, how often, and the difference to introspection.
Among our changes are when, how often, the difference to
introspection, and the ability to externally manipulate the defaults.
And nice red uniforms.
Although the ability to manipulate them could be considered part of
introspection, not sure.
I'm still unsure whether this is a cool feature or an utter abomination:
def f(x=...):
... try: print("You passed x as", x)
... except UnboundLocalError: print("You didn't pass x")
...
f.__defaults_extra__ = ("n/a",)
f(42)
You passed x as 42
f()
You didn't pass x
(It's an implementation detail and not part of the specification, but
if CPython adopts this behaviour, it will become de facto part of the
expected behaviour, and someone somewhere will use this deliberately.)
ChrisA
We ought to at least try to avoid clear and obvious conflicts between
new and existing syntax.
Using `:=` is even worse than `=>`, and `=:` is just *begging* to
confuse newcomers "why do we have THREE different assignment symbols?"
You can't have it both ways. Either you re-use `:=`, or you add a third
symbol.
(There would be no actual conflict with the walrus operator.)
Best wishes
Rob Cliffe
However, even if I assume the mythical future PEP never happens, in
terms of readability, a WORD is vastly less confusing than a
combination of punctuation that has no obvious or natural
interpretation like '=>'. Or rather, I think that spelling is
kinda-sorta obvious for the lambda meaning, and the use you want is
kinda-sorta similar to a lambda. So I *do* understand how you get
there... but it still seems like much too much line noise for a very
minimal need.
Hm. A word is "vastly less confusing". OK. Should Python have been
designed with
x assigned y+1
rather than
x = y +1
(note that '=' here does not have its "obvious or natural
interpretation", viz. "is equal to").
Should we have
(x becomes y+1)
rather than
(x := y+1)
Conciseness *is* a virtue, even if it is often trumped by other
considerations.
Best wishes
Rob Cliffe
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Yes I would, but probably not as often as counting cases of the "if
param is None: ..." idiom might lead you to expect.
The problem is, as Neil also pointed out, that it becomes tricky to
explicitly ask for the default behaviour except by leaving the argument
out altogether. Not impossible, as Chris mentions elsewhere, you can
mess about with `*args` or `**kwargs`, but it is decidedly less
convenient, more verbose, and likely to have a performance hit.
So if I were messing about in the interactive interpreter, I would
totally use this for the convenience:
def func(L=>[]): ...
but if I were writing a library, I reckon that probably at least half
the time I'll stick to the old idiom so I can document the parameter:
"If param is missing **or None**, the default if blah..."
I reject Chris' characterisation of this as a hack. There are function
parameters where None will *never* in any conceivable circumstances
become a valid argument value, and it is safe to use it as a sentinel.
There are also circumstances when you start off thinking that None will
never be a valid argument value, but later you have to cater for it.
Now it does start to look as if you used a hack.
Best wishes
Rob Cliffe
In my reference implementation, there is no object that stores it;
it's simply part of the function. A good parallel is the if/else
expression:
x = float("inf") if z == 0 else y/z
Is there an object that represents the if/else expression (or the
deferred "y/z" part)? No, although it could be implemented that way if
you chose:
This is not a good parallel. There is nothing deferred there.
The entire line is evaluated when it is encountered and you get a
result and no part of the if/else expression can ever impact anything
else again unless, via some external control flow, execution returns
and re-executes the entire line. That is not comparable to a function
default, which is STORED and evaluated later independently of the
context in which it was originally written (i.e., the function default
is re-executed but the function definition itself is not re-executed).
The ternary expression vanishes without a trace by the next line,
leaving only its evaluated result. There would be no use in being
able to access some part of it,
It could be useful for debugging.
Best wishes
Rob Cliffe
since the whole (i.e., the ternary expression) is completely finished
by the time you would be able to access it. This is not the case with
a function definition. The function definition leaves behind a
function object, and that function object needs to "know" about the
late-bound default as an independent entity so that it can be
evaluated later. It is bad for the function to store that late-bound
default only in some private format for its exclusive future use
without providing any means for other code to access it as a
first-class value.
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Yes, when appropriate.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
(It's absolutely valid to say "yes" and "yes", and feel free to say
which of those pulls is the stronger one.)
Yes. Any way PEP 671 is implemented adds to the Python learning curve
and the cognitive burden, by definition. I don't see how it is
logically possible to answer "No" to this question.
But IMHO it is well worth it.
Providing only one of early-bound and late-bound defaults (and arguably
On 01/12/2021 06:16, Chris Angelico wrote:
the less useful one) is a deficiency in Python. Providing both would be
useful.
New features are added to the language because they are thought to be
useful. That means that people have to learn about them (even if they
don't write them themselves). That's life.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
I can imagine using it for (a), (b) and (c). Nothing else springs to
mind at the moment.
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
I answered "yes" to question 1, but I'm not letting that stop me from
reiterating that I think the `=>' arrow is the wrong way round.😁 Hence
my preference for `:=` or `=:`.
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
Any and all comments welcomed. I mean, this is python-ideas after
all... bikeshedding is what we do best!
The reference implementation currently has some test failures, which
I'm looking into. I'm probably going to make this my personal default
Python interpreter for a while, to see how things go.
ChrisA
I agree with 3 things that Abe Dillon said in 3 separate posts:
(1) What I really don't like is @param=[] because it puts the emphasis
on the parameter name rather than the act of binding. Not only does it
make it look like @param is a special kind of variable, it also mimics
the *args and **kwargs syntax which makes them seem related.
(2) Yes, we know *why* the hack works. We're all familiar with it. That
doesn't mean it's not a hack.
The bottom line is:
you *don't actually* want the parameter to default to the value of a
sentinel.
you *have* to use that hack because you can't express what you want the
default to actually be.
You're doing something misleading to work around a shortcoming of the
language.
That's a hack.
You have to write something that you don't actually intend.
(3) 2% of functions is a lot of functions.
Best wishes
Rob Cliffe
def foo(a):
... b = a + 1
... print(b)
...
foo.__code__
This seems like dishonest argument. I'm not even sure what point you think
it is making.
Every time I write a function, everything the function does needs to be
STORED. The body is *stored* in the .__code__ attribute. Other things are
stored in .__annotations__ and elsewhere. A function is an OBJECT, and
everything about it has to be attributes of that object.
line 1>
A late binding isn't that one thing about a function that never gets
stored, but floats in the ether magically ready to operate on a function
call by divine intervention. It HAS TO describe *something* attached to
the function object, doing *something* by some means.
--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
Depending on the implementation, you *might* be able to inspect the
function and see the default expression as some sort of callable
function, or evaluatable code object. (That would be nice.)
Unfortunately not, since the default expression could refer to other
parameters, or closure variables, or anything else from the context of
the called function. So you won't be able to externally evaluate it.
Why not? Functions can do all those things: refer to other variables, or
closures, or anything else. You can call functions. Are you sure that
this limitation of the default expression is not just a limitation of
your implementation?
def f(x=...):
... try: print("You passed x as", x)
... except UnboundLocalError: print("You didn't pass x")
...
f.__defaults_extra__ = ("n/a",)
f(42)
You passed x as 42
f()
You didn't pass x
That is absolutely an abomination. If your implementation has the
side-effect that setting a regular early-bound default to Ellipsis makes
the parameter unable to retrieve the default, then the implementation is
fatally broken.
It absolutely is not a feature.
--
Steve
But it IS stored! There is no way for it to be evaluated without it
being stored!
I'm not sure I understand you here. How is the late-bound default
"stored" when one side of a ternary is "not stored"?
This seems like dishonest argument. I'm not even sure what point you think it is making.
My point was a response to a claim that a late-bound default
fundamentally has to be stored somewhere, yet half of a ternary
conditional isn't. My point was that there is no difference, and they
are both code. Please examine the surrounding context and respond in
more detail, rather than calling my argument "dishonest".
ChrisA
def f(x=...):
... try: print("You passed x as", x)
... except UnboundLocalError: print("You didn't pass x")
...
f.__defaults_extra__ = ("n/a",)
f(42)
You passed x as 42
f()
You didn't pass x
That is absolutely an abomination. If your implementation has the
side-effect that setting a regular early-bound default to Ellipsis makes
On Thu, Dec 02, 2021 at 11:00:33PM +1100, Chris Angelico wrote:
I'm still unsure whether this is a cool feature or an utter abomination:
the parameter unable to retrieve the default, then the implementation is
fatally broken.
It absolutely is not a feature.
It's backward incompatible:
15:03:04 R:\>python
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32
bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
Depending on the implementation, you *might* be able to inspect the
function and see the default expression as some sort of callable
function, or evaluatable code object. (That would be nice.)
Unfortunately not, since the default expression could refer to other
parameters, or closure variables, or anything else from the context of
the called function. So you won't be able to externally evaluate it.
Why not? Functions can do all those things: refer to other variables, or
closures, or anything else. You can call functions. Are you sure that
this limitation of the default expression is not just a limitation of
your implementation?
def f():
a = 1
def f(b, c=>a+b): return c
a = 2
return f
If there were a function to represent the late-bound default value for
c, what parameters should it accept? How would you externally evaluate
this? And also: what do you gain by it being a function, other than a
lot of unnecessary overhead?
And it is potentially a LOT of unnecessary overhead. Consider this edge case:
def f(a, b=>c:=len(a)): ...
In what context should the name c be bound? If there's a function for
the evaluation of b, then that implies making c a closure cell, just
for the sake of that. Every reference to c anywhere in the function
(not just where it's set to len(a), but anywhere in f()) has to
dereference that.
It's a massive amount of completely unnecessary overhead AND a
difficult question of which parts belong in the closure and which
parts belong as parameters, which means that this is nearly impossible
to define usefully.
def f(x=...):
... try: print("You passed x as", x)
... except UnboundLocalError: print("You didn't pass x")
...
f.__defaults_extra__ = ("n/a",)
f(42)
You passed x as 42
f()
You didn't pass x
That is absolutely an abomination. If your implementation has the
side-effect that setting a regular early-bound default to Ellipsis makes
the parameter unable to retrieve the default, then the implementation is
fatally broken.
It absolutely is not a feature.
That's not what the example shows. It shows that changing dunder
attributes can do this. I'm not sure why you think that the
implementation is as restricted as you imply. The assignment to
__defaults_extra__ is kinda significant here :)
ChrisA
My favorite alternative is ?= if people think => and -> are getting
overly loaded. What I really don't like is @param=[] because it puts the
emphasis on the parameter name rather than the act of binding. Not only
does it make it look like @param is a special kind of variable, it also
mimics the *args and **kwargs syntax which makes them seem related.
But it *is* a special kind of parameter: it is a parameter that uses
late binding for the default value, instead of early binding.
Putting the emphasis on the parameter name is entirely appropriate.
Late bound parameters don't have a different sort of binding to other
parameters. There aren't two kinds of binding:
1. Parameters with no default, and parameters with early bound default,
use the same old type of name binding that other local variables use
(local slots in CPython, maybe a dict for other implementations);
2. and parameters with late bound defaults use a different sort of name
binding, and we need a different syntax to reflect that.
That is wrong: there is only one sort of binding in Python (although it
can have a few different storage mechanisms: slots, cells and dict
namespaces, maybe even others). The storage mechanism is irrelevant.
It's all just name binding, and the implementation details are handled
by the interpreter.
By the time the function body is entered, all the parameters have been
bound to a value. (If there are parameters that don't have a value, the
interpreter raises an exception and you never enter the function body.)
It doesn't matter where those values came from, whether they were passed
in by the caller, or early bound defaults loaded from the cache, or late
bound defaults freshly evaluated. The value is bound to the parameter in
exactly the same way, and you cannot determine where that value came
from.
--
Steve
def f(x=...):
... try: print("You passed x as", x)
... except UnboundLocalError: print("You didn't pass x")
...
f.__defaults_extra__ = ("n/a",)
f(42)
You passed x as 42
f()
You didn't pass x
That is absolutely an abomination. If your implementation has the
side-effect that setting a regular early-bound default to Ellipsis makes
On Thu, Dec 02, 2021 at 11:00:33PM +1100, Chris Angelico wrote:
I'm still unsure whether this is a cool feature or an utter abomination:
the parameter unable to retrieve the default, then the implementation is
fatally broken.
It absolutely is not a feature.
It's backward incompatible:
15:03:04 R:\>python
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32
bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
def f(x=...):
... try: print("You passed x as", x)
... except UnboundLocalError: print("You didn't pass x")
...
f()
You passed x as Ellipsis
So I must agree with Steven that this should not be a feature.
Clearly I shouldn't post code examples without lots and lots of
explanatory comments.
https://www.python.org/dev/peps/pep-0671/#implementation-details
# REDEFINE THE INTERPRETER'S UNDERSTANDING
# OF THE LATE BOUND DEFAULTS FOR THE FUNCTION
f.__defaults_extra__ = ("n/a",)
There, now it's a bit clearer what's going on.
ChrisA
My favorite alternative is ?= if people think => and -> are getting
overly loaded. What I really don't like is @param=[] because it puts the
emphasis on the parameter name rather than the act of binding. Not only
does it make it look like @param is a special kind of variable, it also
mimics the *args and **kwargs syntax which makes them seem related.
But it *is* a special kind of parameter: it is a parameter that uses
late binding for the default value, instead of early binding.
Putting the emphasis on the parameter name is entirely appropriate.
Late bound parameters don't have a different sort of binding to other
parameters. There aren't two kinds of binding:
1. Parameters with no default, and parameters with early bound default,
use the same old type of name binding that other local variables use
(local slots in CPython, maybe a dict for other implementations);
2. and parameters with late bound defaults use a different sort of name
binding, and we need a different syntax to reflect that.
That is wrong: there is only one sort of binding in Python (although it
can have a few different storage mechanisms: slots, cells and dict
namespaces, maybe even others). The storage mechanism is irrelevant.
It's all just name binding, and the implementation details are handled
by the interpreter.
In these examples, is the name binding any different?
x = 1
x += 1
(x := 1)
for x in [1]: pass
They use different equals signs, but once the assignment happens, it's
the exact same name binding.
But exactly what IS different about them? Is the name 'x' different?
The value 1? The name binding itself?
By the time the function body is entered, all the parameters have been
bound to a value. (If there are parameters that don't have a value, the
interpreter raises an exception and you never enter the function body.)
It doesn't matter where those values came from, whether they were passed
in by the caller, or early bound defaults loaded from the cache, or late
bound defaults freshly evaluated. The value is bound to the parameter in
exactly the same way, and you cannot determine where that value came
from.
Yes, that is correct. And the parameter is the exact same kind of
thing either way, too, so adorning the parameter name is wrong too.
And the expression being evaluated is still just an expression, so
adorning THAT is wrong too. There's nothing about late-bound defaults
that is fundamentally different from the rest of Python, so there's no
obvious "this is the bit to adorn" thing.
In contrast, *args and **kwargs actually change the parameter that
gets received: one gives a sequence, the other a mapping, whereas
leaving off both markers gives you a single value.
ChrisA
you *have* to use that hack because you can't express what you want the
default to actually be.
The point of having default values is so that the caller doesn't have to
express what the default will actually be. If the caller has to express
that value, it's not a default, it's a passed-in argument.
You're doing something misleading to work around a shortcoming of the
language.
How is it misleading? The parameter is explicitly documented as taking
None to have a certain effect. None is behaving here as a convenient,
common special constant to trigger a certain behaviour, no different
than passing (for example) buffering=-1 to open() to trigger a very
complex set of behaviour.
(With buffering=-1, the buffer depends on the platform details, and
whether the file is binary, text, and whether or not it is a tty.)
For historical reasons, probably related to C, the default value for
buffering is -1. But it could have just as easily be None, or the string
"default", or an enumeration, or some builtin constant.
--
Steve
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
I would actively avoid using this feature and discourage people from
using it because:
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
I think that this imposes a significant cognitive burden, not for the
simple cases, but when combined with the more advanced function
definition syntax. I think this has the potential to make debugging
large code-bases much harder.
There is nothing that this proposal makes possible that is not already
possible with more explicit code.
There is nothing that this proposal makes possible that is not already
possible with more explicit code.
It's worth noting that "explicit" does not mean "verbose". For
instance, this is completely explicit about what it does:
x += 1
It does not conceal what it's doing, yet it uses a very compact
notation to say "augmented addition". The proposal in question uses an
explicit symbol to indicate that the default should be late-bound.
In contrast, a less explicit and much worse proposal might be: "If the
argument default defines a mutable object, construct a new one every
time", so "def f(x=1):" would be early bound and "def f(x=[]):" would
be late-bound. This is implicit behaviour, since it's not stated in
the code which one is which.
ChrisA
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
No. As I mentioned in the earlier thread, I don't support any
proposal in which an argument can "have a default" but that default is not
a first-class Python object of some sort.
I don't understand this criticism.
Of course the default value will be a first-class Python object of some
sort.*Every* value in Python is a first-class object. There are no
machine values or unboxed values, and this proposal will not change
that.
All that this proposal changes is*when* and*how often* the default
will be evaluated, not the nature of the value.
As has happened often in these threads, it seems different people mean
different things by "default value".
What you are calling "the default value" is "a thing that is used at
call time if no value is passed for the argument". What I am calling
"the default value" is "a thing that is noted at definition time to be
used later if no value is passed for the argument".
What I'm saying is that I want that "thing" to exist. At the time the
function is defined, I want there to be a Python object which represents
the behavior to be activated at call time if the argument is not passed.
In the current proposal there is no such "thing". The function just
has behavior melded with its body that does stuff, but there is no
addressable "thing" where you can say "if you call the function and the
argument isn't passed were are going to take this
default-object-whatchamacallit and 'use' it (in some defined way) to get
the default value". This is what we already have for early-bound
defaults in the function's `__defaults__` attribute.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
What I'm saying is that I want that "thing" to exist. At the time the
function is defined, I want there to be a Python object which represents
the behavior to be activated at call time if the argument is not passed.
In the current proposal there is no such "thing". The function just
has behavior melded with its body that does stuff, but there is no
addressable "thing" where you can say "if you call the function and the
argument isn't passed were are going to take this
default-object-whatchamacallit and 'use' it (in some defined way) to get
the default value". This is what we already have for early-bound
defaults in the function's `__defaults__` attribute.
What would you do with this object? Suppose you have this function:
def f(lst=>[], n=>len(lst)):
...
What can you usefully do with this hypothetical object representing
the default for n? There is no "thing" and no way you could "use" that
thing to predict the value of n without first knowing the full context
of the function.
With early-bound defaults, you can meaningfully inspect them and see
what the value is going to be. With late-bound defaults, should the
default for lst be an empty list? Should the default for n be 0?
Neither is truly correct (but they're wrong in different ways).
Do you actually need that object for anything, or is it simply for the
sake of arbitrary consistency ("we have objects for early-bound
defaults, why can't we have objects for late-bound defaults")?
The value that will be assigned to the parameter does not exist until
the function is actually called, and may depend on all kinds of things
about the function and its context. Until then, there is no value, no
object, that can represent it.
I put to you the same question I put to David Mertz: how is a
late-bound default different from half of a conditional expression?
def f(*args):
lst = args[0] if len(args) > 0 else []
n = args[1] if len(args) > 1 else len(lst)
...
Do the expressions "[]" and "len(lst)" have to have object
representations in this form? If not, why should they need object
representations when written in the more natural way as "lst=>[]"?
ChrisA
There is nothing that this proposal makes possible that is not already
possible with more explicit code.
It's worth noting that "explicit" does not mean "verbose". For
instance, this is completely explicit about what it does:
x += 1
It does not conceal what it's doing, yet it uses a very compact
notation to say "augmented addition". The proposal in question uses an
explicit symbol to indicate that the default should be late-bound.
In contrast, a less explicit and much worse proposal might be: "If the
argument default defines a mutable object, construct a new one every
time", so "def f(x=1):" would be early bound and "def f(x=[]):" would
be late-bound. This is implicit behaviour, since it's not stated in
the code which one is which.
A bit of an aside but I find it interesting that you pick += for the
example here because there is very much an implicit behaviour with +=
that it mutates in-place or not depending on the mutability of the
object in question (a property that is invisible in the code). For x
+= 1 you can guess that x is a number and remember that all the
standard number types are immutable. Where you have e.g. mutable and
immutable versions of a type though there is no way to know just by
looking at the augmented assignment statement itself:
Is this implicitness a problem in practice? Usually it isn't but very
occasionally it gives the kind of bug that can send someone banging
their head against a wall for a long time.
--
Oscar
The "42/n" part is stored in f.__code__.co_code as the part that says
"LOAD_CONST 42, LOAD_FAST n, BINARY_TRUE_DIVIDE". It's not an object.
It's just code - three instructions.
Here's how (in the reference implementation - everything is subject to
change) a late-bound default looks:
The "=>[]" part is stored in f.__code__.co_code as the part that says
"QUERY_FAST x, and if false, BUILD_LIST, STORE_FAST x". It's not an
object. It's four instructions in the bytecode.
In both cases, no part of the expression is ever re-executed. I'm not
understanding the distinction here. Can you explain further please?
Your explanation exactly shows how it IS re-executed. I'm not totally
clear on this disassembly since this is new behavior, but if I
understand right, BUILD_LIST is re-executing the expression `[]` and
STORE_FAST is re-assigning it to x. The expression `[]` is
syntactically present in the function definition but its execution has
been shoved into the function body where it may be re-executed many
times (any time the function is called without passing a value).
What do you mean when you say it is not re-executed? Is it not the
case that `[]` is syntactically present in the `def` line (which is
executed only once, and whose bytecode is not shown) yet its
implementation (BUILD_LIST) is in the function body, which may be
executed many times? How is the BUILD_LIST opcode there not being
re-executed on later calls of the function?
Perhaps what you are saying is that what is stored is not the literal
string "[]" but bytecode that implements it? That distinction is
meaningless to me. The point is that you wrote `[]` once, in a line
that is executed once (the function definition itself), but the `[]` is
executed many times, separately from the function definition.
Another way to put it is, again, the examples are not parallel. In
your first example the ternary expression is (syntactically) part of the
function BODY, so of course it appears in the disassembly. In your
second example, the late-bound default is not in the body, it is in the
signature. The disassembly is only showing the bytecode of the function
body, but the late-bound default is syntactically enmeshed with the
function DEFINITION. So I don't want the late-bound default code to be
re-executed unless the function is re-defined. Or, if you're going to
store that `x = []`, I don't want it to be "stored" by just shoving it
in the function body, I want it to be some kind of separate object.
Here's an example that may make my point clearer.
some_function(x+1, lambda x: x+2)
This is our best current approximation to some kind of "late
evaluation". The first argument, `x+1`, is evaluated before the
function is called, and the function gets only the result. The second
argument is also of course evaluated before the function is called, but
what is evaluated is a lambda; the `x+2` is not evaluated. But the idea
of "evaluate x+2 later" is encapsulated in a function object. Without
knowing anything about `some_function`, I still know that there is no
way it can ever evaluate `x+2` without going through the interface of
that function object. There is no way to pass, as the second argument
to `some_function` some kind of amorphous "directive" that says
"evaluate x+2 later" without wrapping that directive up into some kind
of Python object.
Perhaps the fundamental point that I feel you're missing about my
position is that a ternary expression does not have a "definition" and a
"body" that are executed at separate times. There is just the whole
ternary expression and it is evaluated all at once. Thus there can be
no parallel with functions, which do have a separation between
definition time and call time. Indeed, it's because these are separate
that late vs. early binding of defaults is even a meaningful concept for
functions (but not for ternary expressions).
So what I am saying is if I see a line like this:
def f(a=x+1, b@=x+2):
The x+2 is syntactically embedded within the `def` line, that is, the
function definition (not the body). Thus there are only two kinds of
semantics that are going to make me happy:
1) That `x+2`(or any bytecode derived from it, etc.) will never be
re-executed unless the program execution again reaches the line with the
`def f` (which is what we have with early-bound defaults)
2) A Python object is created that encapsulates the expression `x+2`
somehow and defines what it means to "evaluate it in a context" (e.g.,
by referencing local variables in the scope where it is evaluated)
Maybe another way to say this is just "I want any kind of late-bound
default to really be an early-bound default whose value is some object
that provides a way to evaluate it later". (I'm trying to think of
different ways to say this because it seems what I'm saying is not clear
to you. :-)
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
Nicholas Cole
"There is nothing that this proposal makes possible that is not already
possible with more explicit code."
There's nothing any of Python's syntax makes possible that is not already
possible with Brainfuck or any other language that's Turing complete. The
current hacks used to get around the lack of late-bound optional parameters
aren't more explicit. They just require more code. If the coder's intent is
to have an optional parameter default to an empty list, the most EXPLICIT
way to encode that intent would be to have the optional parameter default
to an empty list. It's categorically LESS EXPLICIT to bind the parameter to
None and add boilerplate code to the body of the function to correct that.
On Thursday, December 2, 2021 at 11:20:43 AM UTC-6 Nicholas Cole wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
I would actively avoid using this feature and discourage people from
using it because:
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
I think that this imposes a significant cognitive burden, not for the
simple cases, but when combined with the more advanced function
definition syntax. I think this has the potential to make debugging
large code-bases much harder.
4) If "no" to question 1, is there some other spelling or other
small
change that WOULD mean you would use it? (Some examples in the
PEP.)
No. As I mentioned in the earlier thread, I don't support any
proposal in which an argument can "have a default" but that
default is not
a first-class Python object of some sort.
I don't understand this criticism.
Of course the default value will be a first-class Python object of some
sort.*Every* value in Python is a first-class object. There are no
machine values or unboxed values, and this proposal will not change
that.
All that this proposal changes is*when* and*how often* the default
will be evaluated, not the nature of the value.
As has happened often in these threads, it seems different people
mean different things by "default value".
What you are calling "the default value" is "a thing that is used
at call time if no value is passed for the argument". What I am
calling "the default value" is "a thing that is noted at definition
time to be used later if no value is passed for the argument".
What I'm saying is that I want that "thing" to exist. At the time
the function is defined, I want there to be a Python object which
represents the behavior to be activated at call time if the argument
is not passed. In the current proposal there is no such "thing". The
function just has behavior melded with its body that does stuff, but
there is no addressable "thing" where you can say "if you call the
function and the argument isn't passed were are going to take this
default-object-whatchamacallit and 'use' it (in some defined way) to
get the default value". This is what we already have for early-bound
defaults in the function's `__defaults__` attribute.
I also have this objection to the proposal (among other concerns).
Say I have a function with an early-bound default. I can inspect it and
I can change it. One reason to inspect it is so that I can call the
function with its default values. This is a form of wrapping the
function. I realize "just don't pass that argument when you call the
function" will be the response, but I think in good faith you'd have to
admit this is more difficult than just passing some default value to a
function call.
As far as changing the defaults, consider:
def f(x=3): return x
...
f()
3
f.__defaults__=(42,)
f()
42
The current PEP design does not provide for this functionality for
late-bound defaults.
I realize the response will be that code shouldn't need to do these
things, but I do not think we should be adding features to python that
limit what introspections and runtime modifications user code can do.
A classic example of this is PEP 362 function signature objects. I don't
think we should be adding parameter types that cannot be represented in
a Signature, although of course a Signature might need to be extended to
support new features. Signature objects were added for a reason (see the
PEP), and I don't think we should just say "well, that's not important
for this new feature". Also note that over time we've removed
restrictions on Signatures (see, for example, Argument Clinic). So I
don't think adding restrictions is the direction we want to go in.
Eric
def f(*args):
lst = args[0] if len(args) > 0 else []
n = args[1] if len(args) > 1 else len(lst)
Although such is obviously not your intention, I think you have provided a
stronger argument against this feature/PEP than any other I've seen raised
so far. Well, maybe just against the specific implementation you have in
mind.
You are correct, of course, that the second form does not provide
inspectability for `lst` and `n`. Well, it does, but only in a fairly
contorted way of disassembling f.__code__.co_code. Or maybe with an
equally indirect look at the parse tree or something. The body of a
function is very specifically A BODY.
What your proposal/implementation does is put things into the function
signature that are simultaneously excluded from direct inspectability as
function attributes. Python, like almost all programming languages, makes a
pretty clear distinction between function signatures and function bodies.
You propose to remove that useful distinction, or at least weaken it.
For the reasons Eric Smith and others have pointed out, I really WANT to
keep inspectability of function signatures.
--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
you *don't actually* want the parameter to default to the value of a
sentinel.
Yes I do. I *do* want to be able to give a convenient sentinel value in
order to explicitly tell the function "give me the default value"."
If you want the parameter to default to an empty list, it's categorically
more explicit to have the parameter default to an empty list. It's a hack
to say "I wan't this to default to an empty list" then have it actually
default to a sentinel value that you'll replace with an empty list later
on. That's the opposite of explicit. If you actually wanted to have the
parameter default to a sentinel value, why would you then overwrite that
value? This isn't a very complicated point.
Steven D'Aprano
"> you *have* to use that hack because you can't express what you want the
The point of having default values is so that the caller doesn't have to
express what the default will actually be. If the caller has to express
that value, it's not a default, it's a passed-in argument."
None of my comment had anything to do with the caller's perspective. I know
how optional parameters work. I'm specifically talking about the
programmer's ability to map their intent to code and the ability of the
would-be reader of the code (NOT CALLER) to decipher that intent.
If my intent is that I want a parameter to default to an empty list, I
wouldn't assign it to an object called "monkey_pants" by default then add
code to the body of the function to check if it's "monkey_pants".
"But wait", I hear you say, "assigning my optional parameter to
'monkey_pants' isn't a hack because I know will *never* in any conceivable
circumstances become a valid argument value!"
"assigning a value to 'monkey_pants' is how I explicitly communicate tell
the function "give me the default value"!"
Those are both quotes from you with 'None' replaced with 'monkey_pants'.
Steven D'Aprano
"How is it misleading?"
Your way of having a parameter default to an empty list (or whatever) is by
having the parameter default to None. That's how it's misleading. I know
how the hack works. I know why the hack works. I know how default
parameters work. You can spare trying to explain all that again. What you
can't explain is why it's more explicit to have a parameter default to a
sentinel value when the actual intent is to have it default to a mutable
value. If it were more explicit, then why not replace `def func(x=0)` with
`def func(x=SENTINEL): if x is SENTINEL: x = 0`?
I'm at a loss for ways to explain my position any clear than that.
On Thursday, December 2, 2021 at 9:52:28 AM UTC-6 Steven D'Aprano wrote:
you *have* to use that hack because you can't express what you want the
default to actually be.
The point of having default values is so that the caller doesn't have to
express what the default will actually be. If the caller has to express
that value, it's not a default, it's a passed-in argument.
You're doing something misleading to work around a shortcoming of the
language.
How is it misleading? The parameter is explicitly documented as taking
None to have a certain effect. None is behaving here as a convenient,
common special constant to trigger a certain behaviour, no different
than passing (for example) buffering=-1 to open() to trigger a very
complex set of behaviour.
(With buffering=-1, the buffer depends on the platform details, and
whether the file is binary, text, and whether or not it is a tty.)
For historical reasons, probably related to C, the default value for
buffering is -1. But it could have just as easily be None, or the string
"default", or an enumeration, or some builtin constant.
overly loaded. What I really don't like is @param=[] because it puts the
emphasis on the parameter name rather than the act of binding. Not only
does it make it look like @param is a special kind of variable, it also
mimics the *args and **kwargs syntax which makes them seem related.
But it *is* a special kind of parameter: it is a parameter that uses
late binding for the default value, instead of early binding.
Putting the emphasis on the parameter name is entirely appropriate.
Late bound parameters don't have a different sort of binding to other
parameters. There aren't two kinds of binding:"
Again: the important part is the *binding*, not the name of the parameter.
Right now, the order of reading something like `def load_tweets(...
hash_tags: List[str] = None, ...)` goes like this:
1) there's a parameter named 'hash_tags'
2) it's a list of strings
3) it has a default value (bound at function declaration)
4) the default value is None
With `def load_tweets(... @hash_tags: List[str] = [], ...)` it goes like:
1) There's some parameter bound late
2) It's a parameter named 'hash_tags'
3) It's a list of strings
4) It has a default value (of course because of information we learned 3
steps ago)
5) The default value is an empty list
With `def load_tweets(... hash_tags: List[str] => [], ...)` it goes like:
1) there's a parameter named 'hash_tags'
2) it's a list of strings
3) it has a default value (bound at call-time)
4) the default value is None
Notice how it makes way more sense to put related parts closer together
rather than saying "here's a little information. Now put a pin in that:
we'll get back to it later on".
To drive the point home: The @ and = signs are connected, but not visually,
which is confusing. You couldn't write @param without an '=' somewhere down
the line:
`def load_tweets(... @hash_tags, ...) # Error!`
I probably should have been more clear about what I meant when I said "it
make[s] it look like @param is a special kind of variable, it also mimics
the *args and **kwargs syntax which makes them seem related." Of course the
way the parameter is bound *is* special making it, in the ways you pointed
out, a "special kind of variable", however, again: the specialness is all
about the binding of the variable, and: unlike *args and **kwargs, @param
doesn't imply any specific type (in the 'type(kwargs)' sense) for the
variable. We know that args and kwargs are an iterable and mapping,
respectively.
The other question is: how important is the information that a parameter is
late-binding. I've always been against the coding standard of making enums
and constants ALL_CAPS because it makes code shouty and obnoxious just as
it makes any other communication shouty and obnoxious. It doesn't seem like
the fact that something is an enum or constant is important enough to
warrant anything more than some syntax coloring from my IDE (if that).
People always point out that the fact that something is an enum or constant
is, in fact, information, and how else would they possibly communicate that
information? The answer is: you communicate it by declaring an enum or
constant and stop shouting at me! There's other information I you could put
in your variable names that most sane people don't. You don't always append
the name of the class of an object to the variable's name. You say
'hash_tags' not 'hash_tags_list_of_strings'.
In the same vein, I also don't see the need here to make a big fuss about
the binding behavior of a parameter such that it *must* be up-front ahead
of the name, annotation, and other binding info. It may seem like a big
deal now because it's new and scary so we must prefix it with
"HERE_BE_DRAGONS!!!", but years down the line when late-binding params are
common place, it'll be a wart.
On Thursday, December 2, 2021 at 9:16:33 AM UTC-6 Steven D'Aprano wrote:
My favorite alternative is ?= if people think => and -> are getting
overly loaded. What I really don't like is @param=[] because it puts the
emphasis on the parameter name rather than the act of binding. Not only
does it make it look like @param is a special kind of variable, it also
mimics the *args and **kwargs syntax which makes them seem related.
But it *is* a special kind of parameter: it is a parameter that uses
late binding for the default value, instead of early binding.
Putting the emphasis on the parameter name is entirely appropriate.
Late bound parameters don't have a different sort of binding to other
parameters. There aren't two kinds of binding:
1. Parameters with no default, and parameters with early bound default,
use the same old type of name binding that other local variables use
(local slots in CPython, maybe a dict for other implementations);
2. and parameters with late bound defaults use a different sort of name
binding, and we need a different syntax to reflect that.
That is wrong: there is only one sort of binding in Python (although it
can have a few different storage mechanisms: slots, cells and dict
namespaces, maybe even others). The storage mechanism is irrelevant.
It's all just name binding, and the implementation details are handled
by the interpreter.
By the time the function body is entered, all the parameters have been
bound to a value. (If there are parameters that don't have a value, the
interpreter raises an exception and you never enter the function body.)
It doesn't matter where those values came from, whether they were passed
in by the caller, or early bound defaults loaded from the cache, or late
bound defaults freshly evaluated. The value is bound to the parameter in
exactly the same way, and you cannot determine where that value came
from.
"The caller cannot explicitly ask for default behaviour except by omitting
the parameter. This can be very annoying to set up when the parameter
values are provided from other places, e.g.,
if need_x:
# do lots of stuff
x = whatever
else:
# do more
x = None
f(x=x, ...) # easy, but with this PEP, you would have to find a way to
remove x from the parameter list. Typically, removing a parameter from a
dynamically-created parameter list is hard."
If removing a parameter from a dynamically-created parameter list is hard,
maybe that's a tools problem. Maybe we should consider crafting better
tools for such tasks. It also sounds like a lot of people believe the
inspectability of callables that use late-bound defaults is a point of
contention. Maybe there should be more inspection hooks than described in the
implementation details section of the PEP
https://www.python.org/dev/peps/pep-0671/#id8?
On Wednesday, December 1, 2021 at 3:56:35 AM UTC-6 miste...@gmail.com wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
No, I would avoid for the sake of readers. Anyway, it can't be used in
any public projects until thy drop support for 3.10, which is many years
off.
Also, I think this question is quite the biased sample on python-ideas.
Please consider asking this to less advanced python users, e.g.,
reddit.com/r/python or learnpython.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
Any and all comments welcomed. I mean, this is python-ideas after
all... bikeshedding is what we do best!
This PEP has a lot of interesting ideas. I still think that none-aware
operators (PEP 505) are an easier, more readable general solution to
binding calculated default values to arguments succinctly. I think the
problems with this idea include:
* The caller cannot explicitly ask for default behaviour except by
omitting the parameter. This can be very annoying to set up when the
parameter values are provided from other places, e.g.,
if need_x:
# do lots of stuff
x = whatever
else:
# do more
x = None
f(x=x, ...) # easy, but with this PEP, you would have to find a way to
remove x from the parameter list. Typically, removing a parameter from a
dynamically-created parameter list is hard.
* The function code becomes unreadable if the parameter-setting code is
long. Having a long piece of code inside the function after "if parameter
is None" is just fine. Having none-aware operators would make such code
more succinct.
* People nearly always avoid writing code in the parameter defaults
themselves, and this new practice adds a lot of cognitive load. E.g.,
people rarely write:
def f(x: int = 1+g()) -> None: ...
Parameter lists are already busy enough with parameter names, annotations,
and defaults. We don't need to encourage this practice.
In short, I think this is a creative idea, a great exploration. While
optional parameters are common, and some of them have defaults that are
calculated inside the function, my feeling is that people will continue to
set their values inside the function.
There is nothing that this proposal makes possible that is not already
possible with more explicit code.
It's worth noting that "explicit" does not mean "verbose". For
instance, this is completely explicit about what it does:
x += 1
It does not conceal what it's doing, yet it uses a very compact
notation to say "augmented addition". The proposal in question uses an
explicit symbol to indicate that the default should be late-bound.
In contrast, a less explicit and much worse proposal might be: "If the
argument default defines a mutable object, construct a new one every
time", so "def f(x=1):" would be early bound and "def f(x=[]):" would
be late-bound. This is implicit behaviour, since it's not stated in
the code which one is which.
A bit of an aside but I find it interesting that you pick += for the
example here because there is very much an implicit behaviour with +=
that it mutates in-place or not depending on the mutability of the
object in question (a property that is invisible in the code). For x
+= 1 you can guess that x is a number and remember that all the
standard number types are immutable. Where you have e.g. mutable and
immutable versions of a type though there is no way to know just by
looking at the augmented assignment statement itself:
Is this implicitness a problem in practice? Usually it isn't but very
occasionally it gives the kind of bug that can send someone banging
their head against a wall for a long time.
It's only as implicit as every other operator. For instance, when you
write "x + y", that might call type(x).__add__(y), and it might call
type(y).__radd__(x). Is that implicit behaviour? It's very clearly
defined (as is when each will happen).
If there is no __iadd__ method, then += will fall back on + and =.
I think you're confusing "implicit behaviour" with "polymorphic
behaviour", which is a strong tenet of pretty much every modern
object-oriented programming language. The precise behaviour depends on
the types of the objects involved. That's not a problem; it's a
spectacularly useful feature!
And yes, sometimes complexity leads to banging heads on walls. If
there's some weird bug in an __iadd__ method, it can be annoyingly
difficult to track down. But ultimately, it's not that difficult to
figure out exactly what a line of code does. (It's just impractical to
do that simultaneously for every line of code.)
ChrisA
>4) If "no" to question 1, is there some other spelling or other
small
>change that WOULD mean you would use it? (Some examples in the
PEP.)
No. As I mentioned in the earlier thread, I don't support any
proposal in which an argument can "have a default" but that
default is not
a first-class Python object of some sort.
I don't understand this criticism.
Of course the default value will be a first-class Python object of some
sort.*Every* value in Python is a first-class object. There are no
machine values or unboxed values, and this proposal will not change
that.
All that this proposal changes is*when* and*how often* the default
will be evaluated, not the nature of the value.
As has happened often in these threads, it seems different people
mean different things by "default value".
What you are calling "the default value" is "a thing that is used
at call time if no value is passed for the argument". What I am
calling "the default value" is "a thing that is noted at definition
time to be used later if no value is passed for the argument".
What I'm saying is that I want that "thing" to exist. At the time
the function is defined, I want there to be a Python object which
represents the behavior to be activated at call time if the argument
is not passed. In the current proposal there is no such "thing". The
function just has behavior melded with its body that does stuff, but
there is no addressable "thing" where you can say "if you call the
function and the argument isn't passed were are going to take this
default-object-whatchamacallit and 'use' it (in some defined way) to
get the default value". This is what we already have for early-bound
defaults in the function's `__defaults__` attribute.
I also have this objection to the proposal (among other concerns).
Say I have a function with an early-bound default. I can inspect it and
I can change it. One reason to inspect it is so that I can call the
function with its default values. This is a form of wrapping the
function. I realize "just don't pass that argument when you call the
function" will be the response, but I think in good faith you'd have to
admit this is more difficult than just passing some default value to a
function call.
1) I want to call this function
2) I may want to not pass this argument
3) Ah, perfect! I will pass this argument with a value of somemod._SENTINEL.
Or alternatively:
1) I want to call this function.
2) Prepare a dictionary of arguments. Leave out what I don't want.
3) If I want to pass this argument, add it to the dictionary.
This way doesn't require reaching into the function's private
information to use a sentinel. Yes, it may be a tad more difficult
(though not VERY much), but you're also avoiding binding yourself to
what might be an implementation detail.
def f(x=3): return x
...
f()
3
f.__defaults__=(42,)
f()
42
The current PEP design does not provide for this functionality for
late-bound defaults.
Remember, though: the true comparison should be something like this:
_SENTINEL = object()
def f(x=_SENTINEL):
if x is _SENTINEL: x = []
return x
Can you change that from a new empty list to something else? No. All
you can do, by mutating the function's dunders, is change the
sentinel, which is actually irrelevant to the function's true
behaviour. You cannot change the true default.
Consider also this form:
default_timeout = 500
def connect(s, timeout=default_timeout): ...
def read(s, timeout=default_timeout): ...
def write(s, msg, timeout=default_timeout): ...
You can now, if you go to some effort, replace the default in every
function. Or you can do this, and not go to any effort at all:
def read(s, timeout=>default_timeout): ...
The true default is now exactly what the function signature says. And
if you really want to, you CAN change read.__defaults__ to have an
actual early-bound default, which means it will then never check the
default timeout.
Introspection is no worse in this way than writing out the code
longhand. It is significantly better, because even though you can't
change it from a latebound default_timeout to a latebound
read_timeout, you can at least see the value with external tools. You
can't see that if the default is replaced in the body of the function.
I realize the response will be that code shouldn't need to do these
things, but I do not think we should be adding features to python that
limit what introspections and runtime modifications user code can do.
The response is more that the code CAN'T do these things, by
definition. To the extent that you already can, you still can. To the
extent that you should be able to, you are still able to. (And more.
There are things you're capable of with PEP 671 that you definitely
shouldn't do in normal code.)
A classic example of this is PEP 362 function signature objects. I don't
think we should be adding parameter types that cannot be represented in
a Signature, although of course a Signature might need to be extended to
support new features. Signature objects were added for a reason (see the
PEP), and I don't think we should just say "well, that's not important
for this new feature". Also note that over time we've removed
restrictions on Signatures (see, for example, Argument Clinic). So I
don't think adding restrictions is the direction we want to go in.
Same again. If you consider the equivalent to be a line of code in the
function body, then the signature has become MASSIVELY more useful.
Instead of simply seeing "x=
def f(*args):
lst = args[0] if len(args) > 0 else []
n = args[1] if len(args) > 1 else len(lst)
Although such is obviously not your intention, I think you have provided a stronger argument against this feature/PEP than any other I've seen raised so far. Well, maybe just against the specific implementation you have in mind.
You are correct, of course, that the second form does not provide inspectability for `lst` and `n`. Well, it does, but only in a fairly contorted way of disassembling f.__code__.co_code. Or maybe with an equally indirect look at the parse tree or something. The body of a function is very specifically A BODY.
What your proposal/implementation does is put things into the function signature that are simultaneously excluded from direct inspectability as function attributes. Python, like almost all programming languages, makes a pretty clear distinction between function signatures and function bodies. You propose to remove that useful distinction, or at least weaken it.
Actually, no. I want to put the default arguments into the signature,
and the body in the body. The distinction currently has a technical
restriction that means that, in certain circumstances, what belongs in
the signature has to be hacked into the body. I'm trying to make it so
that those can be put where they belong.
String representation, but exactly what the default is. Any tool that
inspects a signature (notably, help() etc) will be able to access
this.
Now write the function the other way, and show me how easy it is to
determine the behaviour when arguments are omitted. What can you
access?
ChrisA
The "42/n" part is stored in f.__code__.co_code as the part that says
"LOAD_CONST 42, LOAD_FAST n, BINARY_TRUE_DIVIDE". It's not an object.
It's just code - three instructions.
Here's how (in the reference implementation - everything is subject to
change) a late-bound default looks:
The "=>[]" part is stored in f.__code__.co_code as the part that says
"QUERY_FAST x, and if false, BUILD_LIST, STORE_FAST x". It's not an
object. It's four instructions in the bytecode.
In both cases, no part of the expression is ever re-executed. I'm not
understanding the distinction here. Can you explain further please?
Your explanation exactly shows how it IS re-executed. I'm not totally
clear on this disassembly since this is new behavior, but if I
understand right, BUILD_LIST is re-executing the expression `[]` and
STORE_FAST is re-assigning it to x. The expression `[]` is
syntactically present in the function definition but its execution has
been shoved into the function body where it may be re-executed many
times (any time the function is called without passing a value).
Ah, I think I get you. The problem is that code is in the def line but
is only executed when the function is called, is that correct? Because
the code would be "re-executed" just as much if it were written in the
function body. It's executed (at most) once for each call to the
function, just like the ternary's side is.
I suppose that's a consideration, but it's not nearly as strong in
practice as you might think. A lot of people aren't even aware of the
difference between compilation time and definition time (even people
on this list have made that mistake). Function default args are
executed when the function is defined, not when it's called, and
that's something that changes with this proposal; but there are many
other subtleties to execution order and timing that don't really
matter in practice.
Perhaps the key point here is to consider function decorators. We
could avoid them altogether:
def f():
@deco
def g(x): ...
def h(x): ...
h = deco(h)
But as well as having the name replication problem, this buries
important information down in the body of the surrounding code, rather
than putting it at the signature of g/h where it belongs. Even though,
semantically, this is actually part of the body of f, we want to be
able to read it as part of the signature of g. Logically and
conceptually, it is part of the signature. Now compare these two:
def f2():
_SENTINEL = object()
def g(x=_SENTINEL):
if x is _SENTINEL: x = []
...
def h(x=>[]):
...
Which one has its signature where its signature belongs? Yes,
semantically, the construction of the empty list happens at function
call time, not at definition time. But what you're saying is: if there
are no args passed, behave as if a new empty list was passed. That's
part of the signature.
In neither case will you find an object representing the expression []
in the function's signature, because that's not an object, it's an
instruction to build an empty list. In the case of g, you can find a
meaningless and useless object stashed away in __defaults__, but that
doesn't tell you anything about the true behaviour of the function. At
least in the case of h, you can find the descriptive string "[]"
stashed there, which can tell a human what's happening.
ChrisA
Unfortunately not, since the default expression could refer to other
parameters, or closure variables, or anything else from the context of
the called function. So you won't be able to externally evaluate it.
Why not? Functions can do all those things: refer to other variables, or
closures, or anything else. You can call functions. Are you sure that
this limitation of the default expression is not just a limitation of
your implementation?
def f():
a = 1
def f(b, c=>a+b): return c
a = 2
return f
If there were a function to represent the late-bound default value for
c, what parameters should it accept?
I'm not saying that it *must* be a function. It could be a bare code
object, that is `eval()`ed. Or something completely new. Dunno.
But you're saying something is impossible, and that seems implausible to
me, because things that seems *very similar* are totally possible.
inner = f() # Get the inner function.
default = inner.__code__.__late_defaults__.wibble[1] # whatever
try:
value = default()
except NameError:
# Well, what did you expect to happen?
value = eval(default.__code__, globals(), {'a': 101, 'b': 202})
Or something. The point is, rather than dismissing the possibility
outright, this should be something we discuss, and carefully consider,
before the PEP is complete.
And also: what do you gain by it being a function, other than a
lot of unnecessary overhead?
Nicer introspection.
Brendan goes from strongly opposed to the PEP to its biggest and most
tireless supporter *wink*
Cleaner separation of concerns: the defaults get handled independently
of the function body.
Plays nice with other tools that (say) use byte-code manipulation on
the function body.
And it is potentially a LOT of unnecessary overhead. Consider this edge case:
def f(a, b=>c:=len(a)): ...
In what context should the name c be bound?
The obvious (which is not necessarily correct) answer is, the same scope
that the expression is evaluated in, unless its declared global or
nonlocal. (Which is only relevant if the function f() is nested in
another function.)
With regular defaults, the expression is evaluated at function
definition time, and c gets bound to the surrounding scope.
With late-bound defaults, the expression is evaluated at function call
time, in the scope of f()'s locals. So c would be a local.
Or the other obvious answer is that c will always be in the surrounding
scope, for both early and late bound defaults.
Consider the existence of walrus expressions in comprehensions:
def demo(a, b=((w:=i**2)*str(w) for i in range(5))):
... return b
...
it = demo(None)
next(it)
''
next(it)
'1'
next(it)
'4444'
w
4
So there is precedent for having function-like entities (in this case, a
comprehension) exposing their walrus variables in the surrounding scope.
The third obvious answer is that if either the decision or the
implementation is really too hard, then make it a syntax error for now,
and revisit it in the future.
If there's a function for
the evaluation of b, then that implies making c a closure cell, just
for the sake of that. Every reference to c anywhere in the function
(not just where it's set to len(a), but anywhere in f()) has to
dereference that.
Okay. Is this a problem?
If it really is a problem, then make it a syntax error to use walrus
expressions inside late bound defaults.
It's a massive amount of completely unnecessary overhead AND a
difficult question of which parts belong in the closure and which
parts belong as parameters, which means that this is nearly impossible
to define usefully.
I've given you two useful definitions.
Its not clear what overhead you are worried about.
Accessing variables in cells is almost as fast as accessing locals, but
even if they were as slow as globals, premature optimization is the root
of all evil. Globals are fast enough.
Or are you worried about the memory overhead of the closures? The extra
cost of fetching and calling the functions when evaluating the defaults?
None of these things seem to be good reasons to dismiss the idea that
default expressions should be independent of the function body.
"Using a walrus expression in the default expression will make your
function 3% slower and 1% larger, so therefore we must not make the
default expression an introspectable code object..."
That's not what the example shows. It shows that changing dunder
attributes can do this. I'm not sure why you think that the
implementation is as restricted as you imply. The assignment to
__defaults_extra__ is kinda significant here :)
Ah, well that is not so clear to people who aren't as immersed in the
implementation as you :-)
Messing about with function dunders can do weird shit:
Unfortunately not, since the default expression could refer to other
parameters, or closure variables, or anything else from the context of
the called function. So you won't be able to externally evaluate it.
Why not? Functions can do all those things: refer to other variables, or
closures, or anything else. You can call functions. Are you sure that
this limitation of the default expression is not just a limitation of
your implementation?
def f():
a = 1
def f(b, c=>a+b): return c
a = 2
return f
If there were a function to represent the late-bound default value for
c, what parameters should it accept?
I'm not saying that it *must* be a function. It could be a bare code
object, that is `eval()`ed. Or something completely new. Dunno.
But you're saying something is impossible, and that seems implausible to
me, because things that seems *very similar* are totally possible.
inner = f() # Get the inner function.
default = inner.__code__.__late_defaults__.wibble[1] # whatever
try:
value = default()
except NameError:
# Well, what did you expect to happen?
value = eval(default.__code__, globals(), {'a': 101, 'b': 202})
Or something. The point is, rather than dismissing the possibility
outright, this should be something we discuss, and carefully consider,
before the PEP is complete.
How, with external calling, are you going to know which name
references to look up, and where to get their values from? This is
already a virtually impossible task in Python, which is why f-strings
cannot be implemented as str.format(**something) for any known meaning
of "something". You cannot get your *current* variable set, much less
the available variables in some other context.
The third obvious answer is that if either the decision or the
implementation is really too hard, then make it a syntax error for now,
and revisit it in the future.
Everything's perfectly well defined, save that you can't externally
evaluate the defaults. You can quite happily use the walrus in a
late-bound default, and it'll bind to the function's locals. (Though I
don't recommend it. I think that that's going to make for
less-readable code than alternatives. Still, it's perfectly legal and
well-defined.)
It's a massive amount of completely unnecessary overhead AND a
difficult question of which parts belong in the closure and which
parts belong as parameters, which means that this is nearly impossible
to define usefully.
I've given you two useful definitions.
Its not clear what overhead you are worried about.
Accessing variables in cells is almost as fast as accessing locals, but
even if they were as slow as globals, premature optimization is the root
of all evil. Globals are fast enough.
Or are you worried about the memory overhead of the closures? The extra
cost of fetching and calling the functions when evaluating the defaults?
None of these things seem to be good reasons to dismiss the idea that
default expressions should be independent of the function body.
The problem is that you're trying to reference cell variables in a
closure that might not even exist yet, so *just in case* you might
have nonlocals, you have to construct a closure... or find an existing
one. Ill-defined and inefficient.
That's not what the example shows. It shows that changing dunder
attributes can do this. I'm not sure why you think that the
implementation is as restricted as you imply. The assignment to
__defaults_extra__ is kinda significant here :)
Ah, well that is not so clear to people who aren't as immersed in the
implementation as you :-)
Maybe, but I did think that the fact that I was assigning to a dunder
should be obvious to people who are reading here :)
If you want the parameter to default to an empty list, it's categorically
more explicit to have the parameter default to an empty list.
You've gone from talking about *my* intentions, which you got wrong, to
an argument about which is "more explicit".
In the simple example of the default being a new empty list:
# Using Chris' preferred syntax, not mine
def func(arg=>[]):
then I agree: once you know what the syntax means, then it is more
explicit at telling the reader that the default value is an empty list.
Great! This sort of thing is the primary use-case of the feature.
But let's talk about more complex examples:
def open(filename, mode='r', buffering=-1): ...
How would you make that "more explicit"?
def open(filename, mode='r', buffering=>(
1 if 'b' not in mode
and os.fdopen(os.open(filename)).isatty()
else _guess_system_blocksize() or io.DEFAULT_BUFFER_SIZE
)
): ...
Too much information! We're now drowning in explicitness.
Now how do I, the caller, *explicitly* indicate that I want to use that
system-dependent value? I just want a nice, simple value I can
explicitly pass as buffering to say "just use the default".
with open('myfile.txt', buffering= what? ) as f:
I can't. So in this case, your agonisingly explicit default expression
forces the caller to be *less* explicit when they call the function.
What the right hand giveth, the left hand taketh away.
Can we agree that, like salt, sugar, and bike-shedding, sometimes you
can have *too much* explicitness in code? Sometimes the right level of
explicitness is to have an agreed upon symbol that acts as a short, easy
to read and write, documented and explicit signal to the function "give
me the default value".
And that symbol can be -1, or an enumeration, or a string, or None.
This is not a hack, and its not "implicit".
None of my comment had anything to do with the caller's perspective.
The caller's perspective is important. The caller has to read the
function signature (either in the documentation, or if they use
`help(function)` in the interpreter, or when their IDE offers up
tooltips or argument completion). The caller has to actually use the
function. We should consider their perspective.
If my intent is that I want a parameter to default to an empty list, I
wouldn't assign it to an object called "monkey_pants" by default then add
code to the body of the function to check if it's "monkey_pants".
Funny you should say that, but *totally by coincidence*, the Bulgarian
word for "default" happens to be transliterated into English as
*monkeypants*, so if I were Bulgarian that is exactly what I might do!
How weird is that?
Nah, I'm kidding of course. But the point I am making is that you are
mocking the concept by using a totally silly example. "monkey_pants"
indeed, ha ha how very droll. But if you used a more meaningful symbol,
like "default", or some enumeration with a meaningful name, or a value
which is a standard idiom such as -1, then the idea isn't so ridiculous.
And None is one such standard idiom.
*Especially* when the default value is not some trivially simple
expression such as an empty list, but a more complicated expression such
as the default buffering used by open files, or in my earlier example of
the default collation (a sequence of letters which depends on the
other input arguments).
Your way of having a parameter default to an empty list (or whatever) is by
having the parameter default to None. That's how it's misleading.
All you have done is repeat that it is misleading. I'm still no clearer
why you think it is misleading to document "if the argument is missing
or None, the default behaviour is ...".
Is the documentation false? Does the function not do exactly what it
says it will do if you pass that symbol?
Let's be more concrete.
Do you feel that using -1 for the buffer size in open() is "misleading"?
Do you think that there is ever a possibility that someday Python's file
I/O will use a buffer with *literally* a size of -1?
--
Steve
How, with external calling, are you going to know which name
references to look up, and where to get their values from?
Isn't the source code available as a string? I will see the names in the
expression.
Or I will disassemble the code object and inspect its byte-code and/or
AST. Or read the documentation. Or maybe the code object will expose
those names in a dunder and I can just look at that.
This is already a virtually impossible task in Python, which is why
f-strings cannot be implemented as str.format(**something) for any
known meaning of "something".
f-strings cannot be implemented as str.format because str.format doesn't
evaluate arbitrary expressions.
You cannot get your *current* variable set, much less
the available variables in some other context.
Why not? The default expression object can record nonlocal variables in
a closure, and you can provide your own globals and/or locals.
You're saying "can't, impossible" but I could take your function object,
extract the code object, disassemble the code to AST or byte-code,
extract the portions that represent the default expression, and evaluate
that. So not *impossible*, just bloody inconvenient and annoying.
(Well, when I say *I could*, I mean *somebody with mad leet AST or byte-
code hacking skillz. Not actually me.)
And it is potentially a LOT of unnecessary overhead. Consider this edge case:
def f(a, b=>c:=len(a)): ...
In what context should the name c be bound?
With late-bound defaults, the expression is evaluated at function call
time, in the scope of f()'s locals. So c would be a local.
In other words, if you're trying to evaluate b's default externally,
you have to set c in a context that doesn't even exist yet. Is that
correct?
Yes? You seem to be saying that as if it were some fatal flaw in my
proposal.
mylocals = {}
There. Now the context exists.
result = eval(default_expression.__code__, globals(), mylocals())
The third obvious answer is that if either the decision or the
implementation is really too hard, then make it a syntax error for now,
and revisit it in the future.
Everything's perfectly well defined, save that you can't externally
evaluate the defaults.
You keep saying that, but your arguments are not convincing.
The problem is that you're trying to reference cell variables in a
closure that might not even exist yet, so *just in case* you might
have nonlocals, you have to construct a closure... or find an existing
one. Ill-defined and inefficient.
That doesn't make sense to me. When you compile the default expression,
you know what variable names are assigned to with a walrus expression
(if any), you know what names exist in the surrounding nonlocal scope
(if any), you know what names are parameters.
If the default expression doesn't use walrus, doesn't refer to any
nonlocal names, and doesn't refer to any other parameters, why would you
construct a closure "just in case"? What would you put in it?
--
Steve
Same again. If you consider the equivalent to be a line of code in the
function body, then the signature has become MASSIVELY more useful.
Instead of simply seeing "x=
You: "it is impossible to evaluate the late-bound default outside of the
function!"
Also you: "Just eval the string."
You can't have it both ways :-)
--
Steve
Actually, no. I want to put the default arguments into the signature,
and the body in the body. The distinction currently has a technical
restriction that means that, in certain circumstances, what belongs in
the signature has to be hacked into the body. I'm trying to make it so
that those can be put where they belong.
Chris, I know this is probably not your intention, but I feel the
discussion is continually muddle by you just saying "default arguments"
as if everyone agrees on what those are and the issue is just where to
put them. But clearly that is not the case. You seem to take it for
granted that "len(x) evaluated when the function is called" "is" a
"default argument" in the same sense that an early-bound default like
the number 3 is. I do not agree, and it's pretty clear David Mertz does
not agree, and I think there are others here who also do not agree.
It's not as is there is some pre-existing notion of "default argument"
in Python and you are just proposing to add an implementation of it that
was left out due to some kind of oversight. Your proposal is CHANGING
the idea of what a default argument is. I get that it's natural to
refer to your new proposal as "default arguments" but I've now seen a
number of messages here where you say "but no we should do X because
this is a default argument", taking for granted that what you're talking
about is already agreed to be a default argument. (No doubt I and many
others are also guilty of similar missteps by saying "default argument"
as a shorthand for something or other, and I'll try to be careful about
it myself.)
By my definition as of now, a default argument has to be an object.
Thus what your proposal envisions are not default arguments. You can't
just say "I want to do this with default arguments". You need to
provide an argument for why we should even consider these to be default
arguments at all.
Perhaps a way of stating this without reference to arguments is this:
you want to put code in the function signature but not have it executed
until the function is called. I do not agree with that choice. The
function signature and the function body are different syntactic
environments with different semantics. Everything in the function
signature should be executed when the function is defined (although of
course that execution can result in an object which contains deferred
behavior itself, like if we pass a function as an argument to another).
Only the function body should be executed when the function is called.
If we want to provide some way to augment the function body to assign
values to missing arguments, I wouldn't rule that out completely, but in
my view the function signature is not an available space for that. The
function signature is ONLY for things that happen when the function is
defined. It is too confusing to have the function signature mix
definition-time and call-time behavior.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
Ah, I think I get you. The problem is that code is in the def line but
is only executed when the function is called, is that correct? Because
the code would be "re-executed" just as much if it were written in the
function body. It's executed (at most) once for each call to the
function, just like the ternary's side is.
Hmmm, is that really the root of our disagreement? That's surprising. :-)
Anyway, yes. The function body and the function signature are two
different things. Function definition time and function call time are
two different things.
I suppose that's a consideration, but it's not nearly as strong in
practice as you might think. A lot of people aren't even aware of the
difference between compilation time and definition time (even people
on this list have made that mistake). Function default args are
executed when the function is defined, not when it's called, and
that's something that changes with this proposal; but there are many
other subtleties to execution order and timing that don't really
matter in practice.
That is a surprising statement to me. I consider the distinction
between definition time and call time to be quite fundamental to Python.
At a minimum one has to understand that the function body isn't
executed when the function is defined (i.e., you have to call it later
to "activate" the defined behavior). This is something I've seen
beginner programmers struggle with.
Understanding the difference between def time and call time is also
important for understanding how decorators work. It's why most
decorators have a structure that is confusing to many newcomers, where
the decorator itself is a function that defines another function and
then returns it. It has to be this way because of the distinction
between the time the decorated function is defined (which is when the
decorator is called) and the time it is called (which is when the "inner
function" that the decorator defines is called).
Or, to put it bluntly, I think people just need to understand the
difference between def time and call time to understand Python's
semantics. Of course even experienced programmers may sometimes get
mixed up in a particular case, but I fully reject the idea that this
distinction is some kind of obscure technicality that we can ignore or
blur by including call-time behavior in the def-time signature. The
distinction is fundamental to how functions work in Python.
Perhaps the key point here is to consider function decorators. We
could avoid them altogether:
def f():
@deco
def g(x): ...
def h(x): ...
h = deco(h)
But as well as having the name replication problem, this buries
important information down in the body of the surrounding code, rather
than putting it at the signature of g/h where it belongs. Even though,
semantically, this is actually part of the body of f, we want to be
able to read it as part of the signature of g. Logically and
conceptually, it is part of the signature. Now compare these two:
Decorators aren't part of the signature of the function they decorate.
deco is not part of the signature of g. A decorator is sort of like a
"transform" applied to the function after it is defined. It's true we
have decorators and like them because they allow us to put that
transform up front rather than at the bottom, but that's not because of
anything about signatures. It's just because some transforms are nice
to know about up front.
def f2():
_SENTINEL = object()
def g(x=_SENTINEL):
if x is _SENTINEL: x = []
...
def h(x=>[]):
...
Which one has its signature where its signature belongs? Yes,
semantically, the construction of the empty list happens at function
call time, not at definition time. But what you're saying is: if there
are no args passed, behave as if a new empty list was passed. That's
part of the signature.
Again, I disagree. Any behavior that happens at call time is not part
of the signature. The signature specifies the function's parameter
names and optionally maps any or all of them to objects to be used in
case they are not passed at call time.
In neither case will you find an object representing the expression []
in the function's signature, because that's not an object, it's an
instruction to build an empty list. In the case of g, you can find a
meaningless and useless object stashed away in __defaults__, but that
doesn't tell you anything about the true behaviour of the function. At
least in the case of h, you can find the descriptive string "[]"
stashed there, which can tell a human what's happening.
You say "In the case of g, you can find a meaningless and useless
object stashed away in __defaults__, but that doesn't tell you anything
about the true behaviour of the function." Yes. And that's fine. As
I've mentioned in other messages in these threads, I am totally fine
with the idea that there is stuff in the function signature that does
not completely characterize the function's behavior. The place for the
function author to explain what the functions defaults mean is in
documentation, and the place for the user to learn what the defaults
mean is also the documentation.
I think again that another area of our disagreement is the relationship
between the function signature and "the true behavior of the function".
To me the signature is to the "true behavior" as the cover of a thick
tome to its contents. The "true behavior" of a function can be
arbitrarily complex in myriad ways which we can never dream of
communicating via the signature. The signature cannot hope to even
begin to convey more than an echo of a whisper of a shadowy reflection
of "the true behavior of the function". We should not worry ourselves
in the slightest about the fact that "the true behavior of the function"
may not be transparently represented in the signature. Sure, we should
choose good names for the parameters, but that's about all I expect from
a function signature.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
How, with external calling, are you going to know which name
references to look up, and where to get their values from?
Isn't the source code available as a string? I will see the names in the
expression.
Or I will disassemble the code object and inspect its byte-code and/or
AST. Or read the documentation. Or maybe the code object will expose
those names in a dunder and I can just look at that.
This is already a virtually impossible task in Python, which is why
f-strings cannot be implemented as str.format(**something) for any
known meaning of "something".
f-strings cannot be implemented as str.format because str.format doesn't
evaluate arbitrary expressions.
You cannot get your *current* variable set, much less
the available variables in some other context.
Why not? The default expression object can record nonlocal variables in
a closure, and you can provide your own globals and/or locals.
You're saying "can't, impossible" but I could take your function object,
extract the code object, disassemble the code to AST or byte-code,
extract the portions that represent the default expression, and evaluate
that. So not *impossible*, just bloody inconvenient and annoying.
(Well, when I say *I could*, I mean *somebody with mad leet AST or byte-
code hacking skillz. Not actually me.)
When f-strings were first proposed, one of the counter-arguments was
basically "just use globals()/locals()".
https://www.python.org/dev/peps/pep-0498/#no-use-of-globals-or-locals
But you cannot, in a fully general way, solve the problem that way. It
is *impossible*. Not just difficult. Impossible. You cannot, in Python
as it currently is, look up arbitrary names in any reliable way. This
is especially true now that we have assignment expressions, such that
evaluating the function default could create a new local or nonlocal
name.
Not just inconvenient and annoying. So impossible that a str method
could not be written to do it.
ChrisA
Same again. If you consider the equivalent to be a line of code in the
function body, then the signature has become MASSIVELY more useful.
Instead of simply seeing "x=
You: "it is impossible to evaluate the late-bound default outside of the
function!"
Also you: "Just eval the string."
You can't have it both ways :-)
I'm saying that there's a certain level of machine-readability to it,
in the same way that a repr has that. You cannot, with full
generality, eval a repr to get the same result; yet I don't think
you'd call it a purely-human-readable representation of something.
What you get in the docs for a late-bound default is more
machine-readable than a repr (you could parse it to AST and expect to
get the same AST that originally compiled the code), which is a far
cry from "just human readable", but you can't actually eval it to
reliably reconstruct the default.
Please can we focus on whether the proposal is useful, and not on some
throwaway comment where I took a shorthand in clarifying how something
can be interpreted? No? Dangit, I must be on python-ideas :)
ChrisA
"You've gone from talking about *my* intentions, which you got wrong, to
an argument about which is "more explicit"."
No. You were the one who brought up explicitness by claiming that using a
sentinel value was a way to "explicitly tell the function "give me the
default value" (whatever the heck that means). Now that I've deconstructed
that argument, you're moving the goal post from "my way is more explicit"
to "there is such a thing as *too* explicit, don't ya know?!" and blaming
me for shifting the focus to "which is more explicit"... cool.
"In the simple example of the default being a new empty list:
# Using Chris' preferred syntax, not mine
def func(arg=>[]):
then I agree"
Which is, what? 99% of the use cases? great!
"But let's talk about more complex examples:
def open(filename, mode='r', buffering=-1): ...
How would you make that "more explicit"?
def open(filename, mode='r', buffering=>(
1 if 'b' not in mode
and os.fdopen(os.open(filename)).isatty()
else _guess_system_blocksize() or io.DEFAULT_BUFFER_SIZE
)
): ...
Too much information! We're now drowning in explicitness."
Then just use the sentinel hack, or write a helper function that has a nice
name that says what the default will be and use that. I'm positive there
will always be corner cases where logic that's concerned with checking and
setting the right values for parameters before the actual business logic
starts.
"Now how do I, the caller, *explicitly* indicate that I want to use that
system-dependent value? I just want a nice, simple value I can
explicitly pass as buffering to say "just use the default".
with open('myfile.txt', buffering= what? ) as f:
I can't. So in this case, your agonisingly explicit default expression
forces the caller to be *less* explicit when they call the function."
This seems like an exceedingly minor complaint. The obvious answer is: you
invoke the default behaviour by not passing the argument. And yes, I know
that's less explicit, but that's kinda the whole point of defaults to begin
with. It's behavior when the user decides not to specify their own.
Nobody's asking "what do I pass to logging.config.fileConfig to explicitly
invoke the logging module's default behavior?!" The answer is simply: don't
configure logging. It's not rocket science.
This want to explicitly pass something to invoke default behavior brings this
old Raymond Hetinger talk
https://www.youtube.com/watch?v=OSGv2VnC0go&t=1870s to mind where he
suggests that sometimes passing positional arguments by name can help make
your code more readable. I wholeheartedly agree with the practice discussed
in Hettinger's talk, but I think your example is completely different.
When you see `twitter_search('@obama', False, 20, True)` that naturally
invokes the question "what the heck are those arguments??"
When you see `with open('myfile.txt') as f` it doesn't raise the question
"what is the buffering argument?"
I don't know if that's why you think the ability to explicitly invoke
default behavior is so important, but I thought I'd address it anyway.
"Can we agree that, like salt, sugar, and bike-shedding, sometimes you
can have *too much* explicitness in code?"
Surely *you* would never agree that there's such a thing as *too much*
bike-shedding! Isn't that your raison d'être?!
I never said one had to be a Nazi about stamping out every use of the
sentinel... er... idiom (if you prefer it to hack). Sure, use it in those
rare corner cases where expressing a late-bound default would be so very
burdensome. The fact that there are such cases isn't a very good argument
against this proposal. It's more like "HA! You can't fix *everything* can
you?!". Great. You got me, Steve. *slow claps*
"The caller's perspective is important. The caller has to read the
function signature (either in the documentation, or if they use
`help(function)` in the interpreter, or when their IDE offers up
tooltips or argument completion). The caller has to actually use the
function. We should consider their perspective."
I never said we shouldn't. That's just not what I was talking about at that
time and you were misinterpreting my argument.
"Nah, I'm kidding of course. But the point I am making is that you are
mocking the concept by using a totally silly example. "monkey_pants"
indeed, ha ha how very droll. But if you used a more meaningful symbol,
like "default", or some enumeration with a meaningful name, or a value
which is a standard idiom such as -1, then the idea isn't so ridiculous.
And None is one such standard idiom."
You missed my point entirely. The point is that the reasons you were giving
for sentinels both being explicit and not being a hack don't hold water.
Namely: The fact that it's guaranteed to work doesn't make it not a hack,
and your inexplicable reasoning that defaulting to a sentinel instead of
the actual default value that parameter is supposed to take after all is
said and done is somehow more explicit than defaulting to the default value
in the first place.
Just because it's an idiom, doesn't mean it's actually good. I think the `if
__name__ == "__main__":` idiom is a horrible, round-about way of saying "if
this file is being run as a script". You might look at that and say "What
do you mean? That's exactly what it says!" because you've seen the idiom so
many times, just like how you think "x=None" explicitly says "give x the
default value", but if you look closely `if __name__ == "__main__":`
*doesn't* actually say anything like "if this file is being run as a
script", it says a bunch of esoteric and confusing garbage that you've
learned to read as "if this file is being run as a script".
"> Steven D'Aprano
Your way of having a parameter default to an empty list (or whatever) is
by
having the parameter default to None. That's how its misleading.
All you have done is repeat that it is misleading. I'm still no clearer
why you think it is misleading"
Seriously? If I said "I'm going to pat my head" then proceeded to rub my
tummy, would you not think that's a bit misleading?
Again, if you look closely, I *didn't* simply repeat that it's misleading.
I pointed out the contradiction between intent and code.
On Thu, Dec 2, 2021 at 8:57 PM Steven D'Aprano wrote:
If you want the parameter to default to an empty list, it's
categorically
more explicit to have the parameter default to an empty list.
You've gone from talking about *my* intentions, which you got wrong, to
an argument about which is "more explicit".
In the simple example of the default being a new empty list:
# Using Chris' preferred syntax, not mine
def func(arg=>[]):
then I agree: once you know what the syntax means, then it is more
explicit at telling the reader that the default value is an empty list.
Great! This sort of thing is the primary use-case of the feature.
But let's talk about more complex examples:
def open(filename, mode='r', buffering=-1): ...
How would you make that "more explicit"?
def open(filename, mode='r', buffering=>(
1 if 'b' not in mode
and os.fdopen(os.open(filename)).isatty()
else _guess_system_blocksize() or io.DEFAULT_BUFFER_SIZE
)
): ...
Too much information! We're now drowning in explicitness.
Now how do I, the caller, *explicitly* indicate that I want to use that
system-dependent value? I just want a nice, simple value I can
explicitly pass as buffering to say "just use the default".
with open('myfile.txt', buffering= what? ) as f:
I can't. So in this case, your agonisingly explicit default expression
forces the caller to be *less* explicit when they call the function.
What the right hand giveth, the left hand taketh away.
Can we agree that, like salt, sugar, and bike-shedding, sometimes you
can have *too much* explicitness in code? Sometimes the right level of
explicitness is to have an agreed upon symbol that acts as a short, easy
to read and write, documented and explicit signal to the function "give
me the default value".
And that symbol can be -1, or an enumeration, or a string, or None.
None of my comment had anything to do with the caller's perspective.
The caller's perspective is important. The caller has to read the
function signature (either in the documentation, or if they use
`help(function)` in the interpreter, or when their IDE offers up
tooltips or argument completion). The caller has to actually use the
function. We should consider their perspective.
If my intent is that I want a parameter to default to an empty list, I
wouldn't assign it to an object called "monkey_pants" by default then
add
code to the body of the function to check if it's "monkey_pants".
Funny you should say that, but *totally by coincidence*, the Bulgarian
word for "default" happens to be transliterated into English as
*monkeypants*, so if I were Bulgarian that is exactly what I might do!
How weird is that?
Nah, I'm kidding of course. But the point I am making is that you are
mocking the concept by using a totally silly example. "monkey_pants"
indeed, ha ha how very droll. But if you used a more meaningful symbol,
like "default", or some enumeration with a meaningful name, or a value
which is a standard idiom such as -1, then the idea isn't so ridiculous.
And None is one such standard idiom.
*Especially* when the default value is not some trivially simple
expression such as an empty list, but a more complicated expression such
as the default buffering used by open files, or in my earlier example of
the default collation (a sequence of letters which depends on the
other input arguments).
Your way of having a parameter default to an empty list (or whatever) is
by
having the parameter default to None. That's how it's misleading.
All you have done is repeat that it is misleading. I'm still no clearer
why you think it is misleading to document "if the argument is missing
or None, the default behaviour is ...".
Is the documentation false? Does the function not do exactly what it
says it will do if you pass that symbol?
Let's be more concrete.
Do you feel that using -1 for the buffer size in open() is "misleading"?
Do you think that there is ever a possibility that someday Python's file
I/O will use a buffer with *literally* a size of -1?
Actually, no. I want to put the default arguments into the signature,
and the body in the body. The distinction currently has a technical
restriction that means that, in certain circumstances, what belongs in
the signature has to be hacked into the body. I'm trying to make it so
that those can be put where they belong.
Chris, I know this is probably not your intention, but I feel the
discussion is continually muddle by you just saying "default arguments"
as if everyone agrees on what those are and the issue is just where to
put them. But clearly that is not the case. You seem to take it for
granted that "len(x) evaluated when the function is called" "is" a
"default argument" in the same sense that an early-bound default like
the number 3 is. I do not agree, and it's pretty clear David Mertz does
not agree, and I think there are others here who also do not agree.
It's not as is there is some pre-existing notion of "default argument"
in Python and you are just proposing to add an implementation of it that
was left out due to some kind of oversight. Your proposal is CHANGING
the idea of what a default argument is. I get that it's natural to
refer to your new proposal as "default arguments" but I've now seen a
number of messages here where you say "but no we should do X because
this is a default argument", taking for granted that what you're talking
about is already agreed to be a default argument. (No doubt I and many
others are also guilty of similar missteps by saying "default argument"
as a shorthand for something or other, and I'll try to be careful about
it myself.)
Some functions most assuredly DO have a pre-existing notion of
"default argument". Some do not. Allow me to give a few examples:
def getattr(object, name, default):
Omitting the third argument does not have a default. It will cause
different behaviour (getattr will raise an exception).
def dict.get(key, default=None):
Omitting the last argument behaves exactly as if you passed None. This
has a default; the function behaves identically whether you pass an
argument or not, and you can determine what the behaviour would be if
you do. (These two are a little confusing in that they use the name
"default", but the fact is that default arguments are often used for
defaults, surprise surprise. So I'm going to be reusing that word a
lot.)
def open(file, mode='r', encoding=????):
Omitting mode is exactly the same as passing 'r'. Omitting encoding is
exactly the same as passing locale.getpreferredencoding(False). Both
of these have real defaults.
def bisect.bisect(a, x, lo=0, hi=len(a)):
Omitting lo is exactly the same as passing 0. Omitting hi is exactly
the same as passing len(a). Both of these have real defaults.
For parameters that do not have defaults, PEP 671 has nothing to say.
Continue doing what you're already doing (whether that's a sentinel,
or *args, or whatever), as there's nothing needing to be changed.
For parameters whose defaults are simple constants, PEP 671 also has
nothing to say. Continue defining them as early-bound defaults, and
the behaviour will not change.
The difference comes with those arguments whose true default is
calculated in some way, or is dynamic. These really truly do have
default values, and there is absolutely no behavioural difference
between hi=len(a) and omitting the hi parameter to bisect(). Due to a
technical limitation, though, the documentation for bisect() has to
explain this in the docstring rather than the function signature.
Would this code pass review?
def spaminate(msg, times=None):
if times is None: times = 50
...
Unless there's a very VERY good reason for using None as the default,
shouldn't it use 50 in the signature? It would be a clearer
declaration of intent: omitting this argument is the same as passing
50.
By my definition as of now, a default argument has to be an object.
Thus what your proposal envisions are not default arguments. You can't
just say "I want to do this with default arguments". You need to
provide an argument for why we should even consider these to be default
arguments at all.
Yes, that is a current, and technical, limitation. Consider this:
https://en.wikipedia.org/wiki/Default_argument#Evaluation
Python is *notable* for always evaluating default arguments just once.
The concept of default args, as described there and as understood by
anyone that I have asked (mainly students, which I admit isn't an
unbiased sample, but nothing is), is that they should be evaluated at
call time.
My argument for why they should be considered default arguments is
that, aside from Python being unable to represent them, they *are*
precisely what default arguments are.
Suppose you're on the ECMAScript committee, and someone proposes that
the language support bignums. (That's not truly hypothetical,
incidentally - I saw references to such a proposal.) Would you argue
that 9007199254740993 is not really a number, on the basis that
current ECMAScript cannot represent it? Would you force someone to
prove that it is really a number? It's only special because of
implementation restrictions.
Perhaps a way of stating this without reference to arguments is this:
you want to put code in the function signature but not have it executed
until the function is called. I do not agree with that choice. The
function signature and the function body are different syntactic
environments with different semantics. Everything in the function
signature should be executed when the function is defined (although of
course that execution can result in an object which contains deferred
behavior itself, like if we pass a function as an argument to another).
That's reasonable, but I disagree from a perspective of practicality:
logically and usefully, it is extremely helpful to be able to describe
arguments that way. Passing positional parameters is approximately
equivalent to:
def func(*args):
"""func(a, b, c=1, d=a+b)"""
a, args = args
b, args = args
if args: c, args = args
else: c = 1
if args: d, args = args
else: d = a + b
If you try to explain it to someone who's learning about default
argument values, do you first have to explain that they get evaluated
and recorded somewhere, and these indescribable values are what's
actually assigned? Or would you describe it mostly like this (or as
"if c is omitted: c = 1", which comes to the same thing)?
Argument default snapshotting is an incredibly helpful performance
advantage, but it isn't inherent to the very concept of default
arguments.
Only the function body should be executed when the function is called.
If we want to provide some way to augment the function body to assign
values to missing arguments, I wouldn't rule that out completely, but in
my view the function signature is not an available space for that. The
function signature is ONLY for things that happen when the function is
defined. It is too confusing to have the function signature mix
definition-time and call-time behavior.
The trouble is that the function signature MUST be the one and only
place where you figure out whether the argument is optional or
mandatory. Otherwise there is an endless sea of debugging nightmares -
just ask anyone who works regularly in JavaScript, where all arguments
are optional and will default to the special value 'undefined' if
omitted.
So you'd need to separate "this is optional" from "if omitted, do
this". That might be a possibility, but it would need two separate
compiler features:
def func(a, b, c=1, d=?):
if unset d: d = a+b
where "unset N" is a unary operator which returns True if the name
would raise, False if not. I'm not proposing that, but if someone
wants to, I would be happy to share my reference implementation, since
90% of the code is actually there :)
ChrisA
This seems like an exceedingly minor complaint. The obvious answer is: you invoke the default behaviour by not passing the argument. And yes, I know that's less explicit, but that's kinda the whole point of defaults to begin with. It's behavior when the user decides not to specify their own. Nobody's asking "what do I pass to logging.config.fileConfig to explicitly invoke the logging module's default behavior?!" The answer is simply: don't configure logging. It's not rocket science.
I'd say it isn't "less explicit". It's actually the ultimate
definition of not doing something: you just... don't do it. The same
line of explanation is important when working with callback functions:
def callback(*a):
print("Doing stuff")
button.signal_connect("clicked", callback())
Oops, you called the callback instead of passing it to the
signal_connect function. What's the solution? DON'T call the callback
at this point. How do you NOT call something? Well, how do you call
it? You put parentheses after it. So how do you not call it? You don't
put the parentheses after it.
Python is fairly good at giving you syntactic features that do things,
such that you can not do them by omitting them.
So I don't think anyone is truly concerned about the pure "don't pass
the argument" case. The problem is the "maybe pass the argument" case.
That ends up looking like this:
c = 42
if cond: func(a, b)
else: func(a, b, c)
# or
args = {}
if not cond: args["c"] = 42
func(a, b, **args)
# or if you know what the default is:
c = some_default
if not cond: c = 42
func(a, b, c)
The first two will, of course, continue to work. The second one is a
fairly clear way to indicate whether or not a parameter should get a
value. The problem with the third is that it requires a documented and
future-guaranteed default value, published and promised to work, and
that simply isn't always possible or practical.
So, yes, some forms of default might be a little unclear. And if
you're converting an existing codebase, it may be necessary to
maintain "if c is None: c = some_default" in the body forever, or
until such time as you can remove the support for None as a parameter.
But None isn't truly the default value.
"Can we agree that, like salt, sugar, and bike-shedding, sometimes you
can have *too much* explicitness in code?"
Surely *you* would never agree that there's such a thing as *too much* bike-shedding! Isn't that your raison d'être?!
I never said one had to be a Nazi about stamping out every use of the sentinel... er... idiom (if you prefer it to hack).
Idiom. Sentinels are extremely important, and PEP 671 isn't proposing
to eliminate them. The "hack" part comes when, for technical reasons,
a sentinel is used when one isn't logically part of the API.
I agree with Steve here: sometimes, the true default is extremely
clunky, and it's better to use a sentinel. That's part of why they're
never going away (and None will continue to be a popular sentinel).
PEP 671 doesn't propose changing every function ever written, because
the vast majority of them are perfectly fine as they are.
"The caller's perspective is important. The caller has to read the
function signature (either in the documentation, or if they use
`help(function)` in the interpreter, or when their IDE offers up
tooltips or argument completion). The caller has to actually use the
function. We should consider their perspective."
I never said we shouldn't. That's just not what I was talking about at that time and you were misinterpreting my argument.
Yes, absolutely. And that's why the function should be able to put the
information in the signature when it's appropriate to, and in the body
when it isn't.
Just because it's an idiom, doesn't mean it's actually good. I think the `if __name__ == "__main__":` idiom is a horrible, round-about way of saying "if this file is being run as a script". You might look at that and say "What do you mean? That's exactly what it says!" because you've seen the idiom so many times, just like how you think "x=None" explicitly says "give x the default value", but if you look closely `if __name__ == "__main__":` *doesn't* actually say anything like "if this file is being run as a script", it says a bunch of esoteric and confusing garbage that you've learned to read as "if this file is being run as a script".
TBH I think that the name-is-main idiom isn't too bad, but mainly
because I've seen far worse idioms. If someone has a better proposal,
I'd be happy to hear it, but the existing idiom isn't horrible.
(But you're right. Well-known idioms can indeed be really REALLY bad.
Just look at the sorts of things that pre-C99 C code would do.)
ChrisA
So I don't think anyone is truly concerned about the pure "don't pass
the argument" case. The problem is the "maybe pass the argument" case.
That ends up looking like this:
c = 42
if cond: func(a, b)
else: func(a, b, c)
[...]
Yes, that's exactly it!
I agree with the rest of your post here.
--
Steve
"# or if you know what the default is:
c = some_default
if not cond: c = 42
func(a, b, c)"
I would argue that, of the three examples; this one is the most problematic
because it overly couples your code to an implementation detail of the
function and makes it unnecessarily brittle to change.
On top of that, in the case of a sentinel; it might even encourage people
to circumvent the "guarantees" that your API relies upon.
For example, I might make a function where the optional parameter could be
anything even a None is valid, so I make my own secret sentinel that, as
Steven D'Aprano put it
"will *never* in any conceivable circumstances become a valid argument
value":
_default = object()
def func(a, b, c=_default): ...
Now the pattern above violates the "nobody will *ever* pass me this"
condition because it imports _default. Then some intern sees your code and
starts importing _default and using it elsewhere because it's such a neat
pattern...
"The "hack" part comes when, for technical reasons,
a sentinel is used when one isn't logically part of the API."
That's my entire point.
"TBH I think that the name-is-main idiom isn't too bad"
The bad part is a lot of Python newbies see it on day one or two because
it's so common and try explaining it without talking about the interpreter
and watching their eyes glaze over. You just showed them how to assign
values to variables and that unbound variables cause errors and now you're
pulling __name__ out of thin air? Where the heck did that come from?
Anyway, this is a tangent I promised myself I wouldn't go on...
On Thu, Dec 2, 2021 at 11:16 PM Chris Angelico wrote:
This seems like an exceedingly minor complaint. The obvious answer is:
you invoke the default behaviour by not passing the argument. And yes, I
know that's less explicit, but that's kinda the whole point of defaults to
begin with. It's behavior when the user decides not to specify their own.
Nobody's asking "what do I pass to logging.config.fileConfig to explicitly
invoke the logging module's default behavior?!" The answer is simply: don't
configure logging. It's not rocket science.
I'd say it isn't "less explicit". It's actually the ultimate
definition of not doing something: you just... don't do it. The same
line of explanation is important when working with callback functions:
def callback(*a):
print("Doing stuff")
button.signal_connect("clicked", callback())
Oops, you called the callback instead of passing it to the
signal_connect function. What's the solution? DON'T call the callback
at this point. How do you NOT call something? Well, how do you call
it? You put parentheses after it. So how do you not call it? You don't
put the parentheses after it.
Python is fairly good at giving you syntactic features that do things,
such that you can not do them by omitting them.
So I don't think anyone is truly concerned about the pure "don't pass
the argument" case. The problem is the "maybe pass the argument" case.
That ends up looking like this:
c = 42
if cond: func(a, b)
else: func(a, b, c)
# or
args = {}
if not cond: args["c"] = 42
func(a, b, **args)
# or if you know what the default is:
c = some_default
if not cond: c = 42
func(a, b, c)
The first two will, of course, continue to work. The second one is a
fairly clear way to indicate whether or not a parameter should get a
value. The problem with the third is that it requires a documented and
future-guaranteed default value, published and promised to work, and
that simply isn't always possible or practical.
So, yes, some forms of default might be a little unclear. And if
you're converting an existing codebase, it may be necessary to
maintain "if c is None: c = some_default" in the body forever, or
until such time as you can remove the support for None as a parameter.
But None isn't truly the default value.
"Can we agree that, like salt, sugar, and bike-shedding, sometimes you
can have *too much* explicitness in code?"
Surely *you* would never agree that there's such a thing as *too much*
bike-shedding! Isn't that your raison d'être?!
I never said one had to be a Nazi about stamping out every use of the
sentinel... er... idiom (if you prefer it to hack).
Idiom. Sentinels are extremely important, and PEP 671 isn't proposing
to eliminate them. The "hack" part comes when, for technical reasons,
a sentinel is used when one isn't logically part of the API.
I agree with Steve here: sometimes, the true default is extremely
clunky, and it's better to use a sentinel. That's part of why they're
never going away (and None will continue to be a popular sentinel).
PEP 671 doesn't propose changing every function ever written, because
the vast majority of them are perfectly fine as they are.
"The caller's perspective is important. The caller has to read the
function signature (either in the documentation, or if they use
`help(function)` in the interpreter, or when their IDE offers up
tooltips or argument completion). The caller has to actually use the
function. We should consider their perspective."
I never said we shouldn't. That's just not what I was talking about at
that time and you were misinterpreting my argument.
Yes, absolutely. And that's why the function should be able to put the
information in the signature when it's appropriate to, and in the body
when it isn't.
Just because it's an idiom, doesn't mean it's actually good. I think the
`if __name__ == "__main__":` idiom is a horrible, round-about way of saying
"if this file is being run as a script". You might look at that and say
"What do you mean? That's exactly what it says!" because you've seen the
idiom so many times, just like how you think "x=None" explicitly says "give
x the default value", but if you look closely `if __name__ == "__main__":`
*doesn't* actually say anything like "if this file is being run as a
script", it says a bunch of esoteric and confusing garbage that you've
learned to read as "if this file is being run as a script".
TBH I think that the name-is-main idiom isn't too bad, but mainly
because I've seen far worse idioms. If someone has a better proposal,
I'd be happy to hear it, but the existing idiom isn't horrible.
(But you're right. Well-known idioms can indeed be really REALLY bad.
Just look at the sorts of things that pre-C99 C code would do.)
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
Well, I prefer to repurpose the walrus.
I think that is mostly because of my experience with Mathematica/Wolfram
in which "=" means "bind now" and ":=" means "bind later" in a similar
way to the proposal here. I understand there is possible ambiguity both
with the walrus itself and with annotations, but to me it is less
confusing than =>, especially in the face of its possible use as a new
spelling for lambda, which might be a pretty common use-case for late
binding.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
a) for sure; b) possibly; c) likely; d) who knows?
More on these:
b) I think another motivation that hasn't been highlighted is teaching
-- perhaps surprisingly, I think adding this will make some aspects of
teaching *easier*. Mutable defaults are known to be confusing to Python
neophytes; I have certainly found this when I have both explictly taught
or just had to explain some aspect of python.
Yes, I know that understanding this behaviour is exactly part of
learning the meaning of "binding" in Python but nonetheless being able
to contrast the two of these would actually be useful in making the
distinction.
c) I would probably use this, but can we confirm something that isn't
explicit in the pep:
Does this
def fn(word, num=>len(word)):...
mean the same as this? (except for the default for word):
def fn(word="foo", num=>len(word):...
I can't think of any argument they shouldn't but in fact the former
reads funny to me.
Also, relatedly, I assume that late-bound arguments can refer to names
that only exist in the caller's environment, but haven't even been bound
at definition time? And that if they are not bound at call time, there
is a syntax error?
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
I am not a regular contributor so perhaps I already fit into this
category; see also my comment about teaching above.
Any and all comments welcomed. I mean, this is python-ideas after
all... bikeshedding is what we do best!
The reference implementation currently has some test failures, which
I'm looking into. I'm probably going to make this my personal default
Python interpreter for a while, to see how things go.
b) I think another motivation that hasn't been highlighted is teaching
-- perhaps surprisingly, I think adding this will make some aspects of
teaching *easier*. Mutable defaults are known to be confusing to Python
neophytes; I have certainly found this when I have both explictly taught
or just had to explain some aspect of python.
c) I would probably use this, but can we confirm something that isn't
explicit in the pep:
Does this
def fn(word, num=>len(word)):...
mean the same as this? (except for the default for word):
def fn(word="foo", num=>len(word):...
I can't think of any argument they shouldn't but in fact the former
reads funny to me.
Not sure what you mean by "mean the same", but the rule is simple: the
expression is allowed to refer to anything in its current context.
That includes other parameters.
If 'word' has no default, then you cannot call the function without
providing a word. Defaults are irrelevant if you get TypeError prior
to that point.
If you pass a value for "word", then the two will behave identically.
Either you pass a value for num and it uses that, or you don't pass a
value and it uses the length of word as the default.
Things are a little more messy if you refer to parameters to the right
of them. In my current reference implementation, this is valid, but
will raise UnboundLocalError if those parameters also have late-bound
defaults and were not given values. But don't depend on that, and it
may become more strictly invalid.
Also, relatedly, I assume that late-bound arguments can refer to names
that only exist in the caller's environment, but haven't even been bound
at definition time? And that if they are not bound at call time, there
is a syntax error?
Not a syntax error. It's like any other name reference; if you refer
to something that doesn't exist, you'll get a NameError.
You're absolutely welcome - and encouraged - to refer to nonlocal
(global or closure) names from your environment.
Incidentally, I don't yet have any good examples to back this up, but
this sort of thing would be completely valid:
class X:
def method(self, size=>self.n): ...
Technically it's the same thing as "a, size=>len(a)" in that it's
referring to an earlier parameter, but to a human, this might be seen
as a different and also valid use-case.
ChrisA
Say I have a function with an early-bound default. I can inspect it and
I can change it. One reason to inspect it is so that I can call the
function with its default values. This is a form of wrapping the
function. I realize "just don't pass that argument when you call the
function" will be the response, but I think in good faith you'd have to
admit this is more difficult than just passing some default value to a
function call.
1) I want to call this function
2) I may want to not pass this argument
3) Ah, perfect! I will pass this argument with a value of somemod._SENTINEL.
Or alternatively:
1) I want to call this function.
2) Prepare a dictionary of arguments. Leave out what I don't want.
3) If I want to pass this argument, add it to the dictionary.
This way doesn't require reaching into the function's private
information to use a sentinel. Yes, it may be a tad more difficult
(though not VERY much), but you're also avoiding binding yourself to
what might be an implementation detail.
Your version is less friendly to type checking. And it doesn't work with
positional-only arguments.
How is the sentinel value private information or an implementation
detail? It's part of the API. It should be clearly documented. If
nothing else, it's can be inspected and discovered.
def f(x=3): return x
...
f()
3
f.__defaults__=(42,)
f()
42
The current PEP design does not provide for this functionality for
late-bound defaults.
Remember, though: the true comparison should be something like this:
_SENTINEL = object()
def f(x=_SENTINEL):
if x is _SENTINEL: x = []
return x
Can you change that from a new empty list to something else? No. All
you can do, by mutating the function's dunders, is change the
sentinel, which is actually irrelevant to the function's true
behaviour. You cannot change the true default.
It is none the less true that default late-bound values cannot be
modified. Correct? Early-bound ones can.
Consider also this form:
You can now, if you go to some effort, replace the default in every
function. Or you can do this, and not go to any effort at all:
def read(s, timeout=>default_timeout): ...
The true default is now exactly what the function signature says. And
if you really want to, you CAN change read.__defaults__ to have an
actual early-bound default, which means it will then never check the
default timeout.
Introspection is no worse in this way than writing out the code
longhand. It is significantly better, because even though you can't
change it from a latebound default_timeout to a latebound
read_timeout, you can at least see the value with external tools. You
can't see that if the default is replaced in the body of the function.
I realize the response will be that code shouldn't need to do these
things, but I do not think we should be adding features to python that
limit what introspections and runtime modifications user code can do.
The response is more that the code CAN'T do these things, by
definition. To the extent that you already can, you still can. To the
extent that you should be able to, you are still able to. (And more.
There are things you're capable of with PEP 671 that you definitely
shouldn't do in normal code.)
This is a tautology. You can't do these things if 671 is accepted
because they will defined as not doable by 671. That's a strike against it.
My stance is that it should be possible, and a proposal that makes them
not possible with late-bound arguments is deficient.
A classic example of this is PEP 362 function signature objects. I don't
think we should be adding parameter types that cannot be represented in
a Signature, although of course a Signature might need to be extended to
support new features. Signature objects were added for a reason (see the
PEP), and I don't think we should just say "well, that's not important
for this new feature". Also note that over time we've removed
restrictions on Signatures (see, for example, Argument Clinic). So I
don't think adding restrictions is the direction we want to go in.
Same again. If you consider the equivalent to be a line of code in the
function body, then the signature has become MASSIVELY more useful.
Instead of simply seeing "x=
We're going to have to disagree about this. I think it's critical that
Signature objects be usable with all types of defaults. And having the
default value available as a string isn't very useful, except for
displaying help text. It wouldn't be possible to create a new Signature
object from an existing Signature object that contains a late-bound
argument (say, to create a new Signature with an additional argument).
At least I haven't seen how it would be possible, since the PEP makes no
mention of Signature objects. Which it definitely should, even if only
to say "late-bound arguments are not designed to work with Signature
objects".
Sentinels are not meaningless. I think you're alienating people every
time you suggest they are.
In any event, I've expended enough volunteer time discussing this for
now, so I'm going to drop off. I'm against another way to specify
function argument defaults, and I'm against some of the decisions that
PEP 671 has made. I've stated those objections. I'll argue against it
further when the SC is asked to consider it.
Eric
Say I have a function with an early-bound default. I can inspect it and
I can change it. One reason to inspect it is so that I can call the
function with its default values. This is a form of wrapping the
function. I realize "just don't pass that argument when you call the
function" will be the response, but I think in good faith you'd have to
admit this is more difficult than just passing some default value to a
function call.
1) I want to call this function
2) I may want to not pass this argument
3) Ah, perfect! I will pass this argument with a value of somemod._SENTINEL.
Or alternatively:
1) I want to call this function.
2) Prepare a dictionary of arguments. Leave out what I don't want.
3) If I want to pass this argument, add it to the dictionary.
This way doesn't require reaching into the function's private
information to use a sentinel. Yes, it may be a tad more difficult
(though not VERY much), but you're also avoiding binding yourself to
what might be an implementation detail.
Your version is less friendly to type checking. And it doesn't work with
positional-only arguments.
Positional-only args can be done with a list instead of a dict. Not
much harder, although less clear what's happening if you have
multiple.
How is the sentinel value private information or an implementation
detail? It's part of the API. It should be clearly documented. If
nothing else, it's can be inspected and discovered.
Depends on the sentinel. In this example, is the exact value of the
sentinel part of the API?
_SENTINEL = object()
def frobnicate(stuff, extra=_SENTINEL):
...
If the sentinel is None, then it may be part of the API. (Of course,
if it's a deliberately-chosen string or something, then that's
completely different, and then you're not really doing the "optional
argument" thing that I'm talking about here, and there's a very real
default value.) But when it's an arbitrary sentinel like this, I argue
that the precise value is NOT part of the function's API, only that
you can pass any object, or not pass an object at all.
def f(x=3): return x
...
f()
3
f.__defaults__=(42,)
f()
42
The current PEP design does not provide for this functionality for
late-bound defaults.
Remember, though: the true comparison should be something like this:
_SENTINEL = object()
def f(x=_SENTINEL):
if x is _SENTINEL: x = []
return x
Can you change that from a new empty list to something else? No. All
you can do, by mutating the function's dunders, is change the
sentinel, which is actually irrelevant to the function's true
behaviour. You cannot change the true default.
It is none the less true that default late-bound values cannot be
modified. Correct? Early-bound ones can.
Yes, but I'm asking you to compare late-bound defaults with the
"sentinel and replace it in the function" idiom, which is a closer
parallel. Can you, externally to the function, change this to use a
new empty set instead of a new empty list? No.
I realize the response will be that code shouldn't need to do these
things, but I do not think we should be adding features to python that
limit what introspections and runtime modifications user code can do.
The response is more that the code CAN'T do these things, by
definition. To the extent that you already can, you still can. To the
extent that you should be able to, you are still able to. (And more.
There are things you're capable of with PEP 671 that you definitely
shouldn't do in normal code.)
This is a tautology. You can't do these things if 671 is accepted
because they will defined as not doable by 671. That's a strike against it.
My point is that you *already* cannot do them. My proposal doesn't
stop you from doing things you currently can do, it just makes it
easier to spell the parts you can do.
My stance is that it should be possible, and a proposal that makes them
not possible with late-bound arguments is deficient.
Please revisit your concerns with regard to the "sentinel and replace
in the function" idiom, as I've stated in the past few posts. The
sentinel itself can be replaced, but that is useless if the function's
body is still looking for the old sentinel.
We're going to have to disagree about this. I think it's critical that
Signature objects be usable with all types of defaults. And having the
default value available as a string isn't very useful, except for
displaying help text. It wouldn't be possible to create a new Signature
object from an existing Signature object that contains a late-bound
argument (say, to create a new Signature with an additional argument).
At least I haven't seen how it would be possible, since the PEP makes no
mention of Signature objects. Which it definitely should, even if only
to say "late-bound arguments are not designed to work with Signature
objects".
They most certainly ARE usable with all types of defaults, but instead
of a meaningless "=
Sentinels are not meaningless. I think you're alienating people every
time you suggest they are.
Some of them are. Some of them are not. When they are meaningful, PEP
671 does not apply, because they're not the sort of sentinel I'm
talking about.
Unfortunately, the word "sentinel" means many different things. There
are many cases where a sentinel is a truly meaningful and useful
value, and in those cases, don't change anything. There are other
cases where the sentinel is a technical workaround for the fact that
Python currently cannot represent defaults that aren't constant
values, and those are meaningless sentinels that can be removed.
In any event, I've expended enough volunteer time discussing this for
now, so I'm going to drop off. I'm against another way to specify
function argument defaults, and I'm against some of the decisions that
PEP 671 has made. I've stated those objections. I'll argue against it
further when the SC is asked to consider it.
No problem. I don't expect everyone to agree with me, and if it ever
happened, I would wonder why :)
ChrisA
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
Presented in isolation, like that, no — however I do feel that the
distinguishing character is the at the wrong side of the equals.
Default values may start with a prefix operator (`+`, `-`, `~`), thus
it could be possible to incorrectly interpret the `>` as some sort of
quote/defer prefix operator (or just difficult to spot) when additional
whitespace is lacking. In other words, I think these look a little too
similar:
def func(arg=-default): ...
def func(arg=>default): ...
Additionally `=>` would conflict with the proposed alternate lambda
syntax, both cognitively and syntactically — assuming the `=>` form
would be valid everywhere that a lambda expression is currently
(without requiring additional enclosing parentheses). The following is
legal syntax:
def func(arg: lambda x: x = 42): ...
# for clarification:
# func.__defaults__ == (42,)
# func.__annotations__ == {'arg': }
It doesn't look promising to place the marker for late bound defaults
on other side of the equals either — causing a syntactical conflict
with comparison operators or assignment operator (or cognitive
conflict augmented assignment) depending on the choice of character.
This leads me to favour the `@param=default` style and although I
agree with Abe Dillon that this somewhat mimics the `*args` and
`**kwds` syntax, I don't see this parallel as a negative. We already
have some variation of late binding in parameter lists, where? `*args`
and `**kwds`: both are rebound upon each call of the function.
Another odd (though not useful) similarity with the current proposal is
that function objects also lack attributes containing some kind of
special representation of the `*args` and `**kwds` parameter defaults
(i.e. the empty tuple & dict). One **cannot** successfully perform
something akin to the following:
def func(**kwds):
return kwds
func.__kwds_dict_default__ = {'keyword_one': 1}
assert func() == {'keyword_one': 1}
Just as with the proposal one cannot modify the method(s) of calculation
used to obtain the late bound default(s) once a function is defined.
I don't know that I have a strong preference for the specific marker
character, but I quite like how `@param=default` could be understood
as "at each (call) `param` defaults to `default`".
---
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
Likely all three, maybe all four. A combination of (b) & (c) could be
particularly useful with methods since one of those other arguments is
`self`, for example:
class IO:
def truncate(self, position=>self.tell()): ...
---
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
I have.
The first unwelcome surprise was:
>>> def func(a=>[]):
... return a
...
>>> import inspect
>>> inspect.signature(func).parameters['a'].default
Ellipsis
Here the current behaviour of returning `Ellipsis` is very unfortunate,
and I think could lead to a lot of head scratching — people wondering
why they are getting ellipses in their code, seemingly from nowhere.
Sure, it can be noted in the official documentation that `Ellipsis` is
used as the indicator of late bound defaults, but third-party resources
which aim to explain the uses of `Ellipsis` would (with their current
content) leave someone clueless.
Additionally I don't think it's too unreasonable an expectation that,
for a function with no required parameters, either of the following (or
something similar) should be equivalent to calling `func()`:
pos_only_args, kwds = [], {}
for name, param in inspect.signature(func).parameters.items():
if param.default is param.empty:
continue
elif param.kind is param.POSITIONAL_ONLY:
pos_only_args.append(param.default)
else:
kwds[name] = param.default
func(*pos_only_args, **kwds)
# or, by direct access to the dunders
func(*func.__defaults__, **func.__kwdefaults__)
The presence of the above if statement's first branch (which was
technically unnecessary, since we established for the purpose of this
example all arguments of `func` are optional / have non-empty defaults)
hints that perhaps `inspect.Parameter` should grow another sentinel
attribute similar to `Parameter.empty` — perhaps `Parameter.late_bound`
— to be set as the `default` attribute of applicable `Parameter`
instances (if not also to be used as the sentinel in `__defaults__` &
`__kwdefaults__`, instead of `Ellipsis`).
Even if the above were implemented, then only way to indicate that the
late bound default should be used would still be by omission of that
argument. Thus, if we combine a late bound default with positional-only
arguments e.g.:
def func(a=>[], b=0, /): ...
It then becomes impossible to programmatically use the given late bound
default for `a` whilst passing a value for `b`. Sure, in this simplistic
case one can manually pass an empty list, but in general — for the same
reasons that it could be "impossible" to evaluate a late bound default
from another context — it would be impossible to manually compute a
replacement value exactly equivalent to the default.
Honestly the circumstances where one may wish to define a function such
as that above seem limited — but it'd be a shame if reverting to use of
a sentinel were required, just in order to have a guaranteed way of
forcing the default behaviour.
Here the current behaviour of returning `Ellipsis` is very unfortunate,
and I think could lead to a lot of head scratching — people wondering
why they are getting ellipses in their code, seemingly from nowhere.
Sure, it can be noted in the official documentation that `Ellipsis` is
used as the indicator of late bound defaults, but third-party resources
which aim to explain the uses of `Ellipsis` would (with their current
content) leave someone clueless.
+1.
This may be a very naive question, apologies if it's nonsense.
Instead of Ellipsis, would it be possible to have a built-in LateBound
class and use instances of that class instead of Ellipsis?
The __str__ method of the inspect.Parameter class could be modified to
return something like
"a=>[]"
(or whatever syntax is adopted for specifying late-bound defaults) in
such cases.
The __repr__ and __str__ methods of a LateBound object could return
something like, respectively,
"LateBound('[]')"
"[]"
I am sure there is code that uses inspect.signature that would be
broken, but isn't that inevitable anyway?
Best wishes
Rob Cliffe
Here the current behaviour of returning `Ellipsis` is very unfortunate,
and I think could lead to a lot of head scratching — people wondering
why they are getting ellipses in their code, seemingly from nowhere.
Sure, it can be noted in the official documentation that `Ellipsis` is
used as the indicator of late bound defaults, but third-party resources
which aim to explain the uses of `Ellipsis` would (with their current
content) leave someone clueless.
Yes. Unfortunately, since there is fundamentally no object that can be
valid here, this kind of thing WILL happen. So when you see Ellipsis
in a default, you have to do one more check to figure out whether it's
a late-bound default, or an actual early-bound Ellipsis:
Ellipsis is less likely as a default than, say, None, so this will
come up fairly rarely. When it does, anything that's unaware of
late-bound defaults will see Ellipsis, and everything else will do a
second lookup.
(I could have the default show the extra instead, but that would lead
to other confusing behaviour.)
Additionally I don't think it's too unreasonable an expectation that,
for a function with no required parameters, either of the following (or
something similar) should be equivalent to calling `func()`:
pos_only_args, kwds = [], {}
for name, param in inspect.signature(func).parameters.items():
if param.default is param.empty:
continue
elif param.kind is param.POSITIONAL_ONLY:
pos_only_args.append(param.default)
else:
kwds[name] = param.default
func(*pos_only_args, **kwds)
# or, by direct access to the dunders
func(*func.__defaults__, **func.__kwdefaults__)
The problem is that then, parameters with late-bound defaults would
look like mandatory parameters. The solution is another check after
seeing if the default is empty:
if param.default is ... and param.extra: continue
The presence of the above if statement's first branch (which was
technically unnecessary, since we established for the purpose of this
example all arguments of `func` are optional / have non-empty defaults)
hints that perhaps `inspect.Parameter` should grow another sentinel
attribute similar to `Parameter.empty` — perhaps `Parameter.late_bound`
— to be set as the `default` attribute of applicable `Parameter`
instances (if not also to be used as the sentinel in `__defaults__` &
`__kwdefaults__`, instead of `Ellipsis`).
Ah, I guess you didn't see .extra then. Currently the only possible
meanings for extra are None and a string, and neither has meaning
unless the default is Ellipsis; it's possible that, in the future,
other alternate defaults will be implemented, which is why I didn't
call it "late_bound". But it has the same functionality.
Even if the above were implemented, then only way to indicate that the
late bound default should be used would still be by omission of that
argument. Thus, if we combine a late bound default with positional-only
arguments e.g.:
def func(a=>[], b=0, /): ...
It then becomes impossible to programmatically use the given late bound
default for `a` whilst passing a value for `b`. Sure, in this simplistic
case one can manually pass an empty list, but in general — for the same
reasons that it could be "impossible" to evaluate a late bound default
from another context — it would be impossible to manually compute a
replacement value exactly equivalent to the default.
That's already the case. How would you call this function with a value
for b and no value for a? When you make positional-only arguments, you
are expecting that they will be passed from left to right. That's just
how parameters work.
I don't consider this to be a problem in practice.
Honestly the circumstances where one may wish to define a function such
as that above seem limited — but it'd be a shame if reverting to use of
a sentinel were required, just in order to have a guaranteed way of
forcing the default behaviour.
If you actually need to be able to specify b without specifying a,
then there are several options:
1) Use a sentinel. If it's part of your API, then it's not a hack. You
might want to use something like None, or maybe a sentinel string like
"new", but it's hard to judge with toy examples; in realistic
examples, there's often a good choice.
2) Allow keyword arguments. That's exactly what they're for: to allow
you to specify some arguments out of order.
3) Redefine the function so the first argument is list_or_count, such
that func(0) is interpreted as omitting a and passing b. This is
usually a messy API, but there are a few functions where it works (eg
range(), and things that work similarly eg random.randrange).
Personally, I'd be inclined to option 2, but it depends a lot on the
API you're building.
ChrisA
Here the current behaviour of returning `Ellipsis` is very unfortunate,
and I think could lead to a lot of head scratching — people wondering
why they are getting ellipses in their code, seemingly from nowhere.
Sure, it can be noted in the official documentation that `Ellipsis` is
used as the indicator of late bound defaults, but third-party resources
which aim to explain the uses of `Ellipsis` would (with their current
content) leave someone clueless.
+1.
This may be a very naive question, apologies if it's nonsense.
Instead of Ellipsis, would it be possible to have a built-in LateBound
class and use instances of that class instead of Ellipsis?
The __str__ method of the inspect.Parameter class could be modified to
return something like
"a=>[]"
(or whatever syntax is adopted for specifying late-bound defaults) in
such cases.
The __repr__ and __str__ methods of a LateBound object could return
something like, respectively,
"LateBound('[]')"
"[]"
I am sure there is code that uses inspect.signature that would be
broken, but isn't that inevitable anyway?
That's a possibility for the inspect module. For the core language -
and therefore for anything that directly inspects the function's
dunders - it's much more efficient to use a well-known object. In the
current implementation, I've kept inspect.signature() consistent with
the dunders, but if there's good reason to change, I wouldn't be
averse to it. But I would need feedback from people who make heavy use
of inspect.signature, as I don't do much with it other than the basics
of help().
Everything in the reference implementation that isn't part of the PEP
should be considered provisional at best :)
ChrisA
Here the current behaviour of returning `Ellipsis` is very unfortunate,
and I think could lead to a lot of head scratching — people wondering
why they are getting ellipses in their code, seemingly from nowhere.
Sure, it can be noted in the official documentation that `Ellipsis` is
used as the indicator of late bound defaults, but third-party resources
which aim to explain the uses of `Ellipsis` would (with their current
content) leave someone clueless.
+1.
This may be a very naive question, apologies if it's nonsense.
Instead of Ellipsis, would it be possible to have a built-in LateBound
class and use instances of that class instead of Ellipsis?
The __str__ method of the inspect.Parameter class could be modified to
return something like
"a=>[]"
(or whatever syntax is adopted for specifying late-bound defaults) in
such cases.
The __repr__ and __str__ methods of a LateBound object could return
something like, respectively,
"LateBound('[]')"
"[]"
I am sure there is code that uses inspect.signature that would be
broken, but isn't that inevitable anyway?
That's a possibility for the inspect module. For the core language -
and therefore for anything that directly inspects the function's
dunders - it's much more efficient to use a well-known object.
OK.
I'm guessing that by "well-known" you mean pre-existing. Python has a
number of built-in singleton objects (None, True, False, Ellipsis).
What about adding a new one called LateBound (or other bikeshed colour)?
Best wishes
Rob Cliffe
In the
current implementation, I've kept inspect.signature() consistent with
the dunders, but if there's good reason to change, I wouldn't be
averse to it. But I would need feedback from people who make heavy use
of inspect.signature, as I don't do much with it other than the basics
of help().
Everything in the reference implementation that isn't part of the PEP
should be considered provisional at best :)
Here the current behaviour of returning `Ellipsis` is very unfortunate,
and I think could lead to a lot of head scratching — people wondering
why they are getting ellipses in their code, seemingly from nowhere.
Sure, it can be noted in the official documentation that `Ellipsis` is
used as the indicator of late bound defaults, but third-party resources
which aim to explain the uses of `Ellipsis` would (with their current
content) leave someone clueless.
+1.
This may be a very naive question, apologies if it's nonsense.
Instead of Ellipsis, would it be possible to have a built-in LateBound
class and use instances of that class instead of Ellipsis?
The __str__ method of the inspect.Parameter class could be modified to
return something like
"a=>[]"
(or whatever syntax is adopted for specifying late-bound defaults) in
such cases.
The __repr__ and __str__ methods of a LateBound object could return
something like, respectively,
"LateBound('[]')"
"[]"
I am sure there is code that uses inspect.signature that would be
broken, but isn't that inevitable anyway?
That's a possibility for the inspect module. For the core language -
and therefore for anything that directly inspects the function's
dunders - it's much more efficient to use a well-known object.
OK.
I'm guessing that by "well-known" you mean pre-existing. Python has a
number of built-in singleton objects (None, True, False, Ellipsis).
Mainly by "well-known" I mean "not private to any particular module",
so those four you mention are all well-known, but a singleton as part
of the inspect module would be a pain, since the core language would
have to import that (or it would have to be magically created).
What about adding a new one called LateBound (or other bikeshed colour)?
What would be gained? You would still be able to use LateBound as an
early-bound default, so you would still need the same dual check.
With inspect.Parameter, you already have a repr and str that show that
it's late-bound, exactly like you suggest. What advantage is there
from using a dedicated sentinel instead of Ellipsis?
It wouldn't be too hard to change the inspect module, but I'd need to know why.
ChrisA
As has happened often in these threads, it seems different people
mean different things by "default value".
What you are calling "the default value" is "a thing that is used at
call time if no value is passed for the argument". What I am calling
"the default value" is "a thing that is noted at definition time to be
used later if no value is passed for the argument".
Right-o, I think I get it now!
So you are referring to the default *expression*, and you would like it
to be an introspectable and maybe even modifiable object, rather than
compiled directly in the function body.
So given
def func(arg=len(seq)+1)
you want there to be an actual object representing the expression
`len(seq)+1`. An executable object you can poke at, inspect, replace and
evaluate, analogous to the way functions already have closures, cells,
and code objects rather than just compiling the whole kit and kaboodle
into one big binary blob.
I'm with you on this. That would be my preference too.
But I don't think it would be a deal-breaker for me if there wasn't.
I'll push hard for the "independent code object" feature, but if it is
impossible as Chris argues, then what cannot be, cannot be.
But speaking as somebody with little understanding of the CPython
internals, I find it implausible that it is *impossible*. But I have no
solid grasp of just how difficult it might be. I can sketch a picture of
what I think should happen during the function call process:
1. initialise a namespace "ns" for the function (slots for each local,
including the function parameters);
2. populate the local slots for each parameter passed by the caller,
including early-bound defaults as needed;
3. for each of the late-bound defaults required:
- fetch its code object;
- exec it with globals set to the function's globals and locals
set to the ns namespace (or a copy of it?) as created in step 1;
- if need be, update the real ns from values in the ns-copy;
4. and enter the body of the function.
I presume that steps 1, 2 and 4 are already what the interpreter does,
or something very close to it. So only 3 is new, and that doesn't seem
obviously impossible. It sounds quite like step 4, only using a
different code object.
In that case, the code objects for the defaults can be independently
executed, provided we pass in an appropriate globals and locals
namespace.
So with the serene self-confidence of somebody who knows that they will
never have to do this themselves, I think I can say that none of this
sounds hard :-)
--
Steve
String representation, but exactly what the default is.
Excellent. And you've just proven that we can evaluate the defaults.
>>> a = '[]' # f.__defaults_extra__[0]
>>> b = 'len(lst)' # f.__defaults_extra__[1]
>>> lst = eval(a, globals())
>>> n = eval(b, globals(), dict(lst=lst))
>>> print(lst, n)
[] 0
Worst case scenario, we can eval the string representation, and that
should work for the great majority of cases that don't involve
nonlocals.
But if the defaults are proper code objects, complete with a closure to
capture their nonlocal environment, then we should be able to do even
better and capture nonlocals as well.
Could there be odd corner cases that don't quite work? Say, like a
comprehension inside a class body?
https://github.com/satwikkansal/wtfpython#-name-resolution-ignoring-class-sc...
Oh well. Let's not make the perfect the enemy of the good.
--
Steve
Here the current behaviour of returning `Ellipsis` is very unfortunate,
and I think could lead to a lot of head scratching — people wondering
why they are getting ellipses in their code, seemingly from nowhere.
Sure, it can be noted in the official documentation that `Ellipsis` is
used as the indicator of late bound defaults, but third-party resources
which aim to explain the uses of `Ellipsis` would (with their current
content) leave someone clueless.
+1.
This may be a very naive question, apologies if it's nonsense.
Instead of Ellipsis, would it be possible to have a built-in LateBound
class and use instances of that class instead of Ellipsis?
The __str__ method of the inspect.Parameter class could be modified to
return something like
"a=>[]"
(or whatever syntax is adopted for specifying late-bound defaults) in
such cases.
The __repr__ and __str__ methods of a LateBound object could return
something like, respectively,
"LateBound('[]')"
"[]"
I am sure there is code that uses inspect.signature that would be
broken, but isn't that inevitable anyway?
That's a possibility for the inspect module. For the core language -
and therefore for anything that directly inspects the function's
dunders - it's much more efficient to use a well-known object.
OK.
I'm guessing that by "well-known" you mean pre-existing. Python has a
number of built-in singleton objects (None, True, False, Ellipsis).
Mainly by "well-known" I mean "not private to any particular module",
so those four you mention are all well-known, but a singleton as part
of the inspect module would be a pain, since the core language would
have to import that (or it would have to be magically created).
What about adding a new one called LateBound (or other bikeshed colour)?
What would be gained? You would still be able to use LateBound as an
early-bound default, so you would still need the same dual check.
I'm struggling here. Yes you could use LateBound as an early-bound
default (or as a parameter value to explicitly pass to a function) but
ISTM that such usages would be perverse.
I've a gut feeling that a solution can be found (to avoid the "Ellipsis
from nowhere" problem) but I can't put my finger on it. Maybe explicitly
specifying LateBound could be an error, perhaps even a SyntaxError?
Help, please!
With inspect.Parameter, you already have a repr and str that show that
it's late-bound, exactly like you suggest. What advantage is there
from using a dedicated sentinel instead of Ellipsis?
It wouldn't be too hard to change the inspect module, but I'd need to know why.
Here the current behaviour of returning `Ellipsis` is very unfortunate,
and I think could lead to a lot of head scratching — people wondering
why they are getting ellipses in their code, seemingly from nowhere.
Sure, it can be noted in the official documentation that `Ellipsis` is
used as the indicator of late bound defaults, but third-party resources
which aim to explain the uses of `Ellipsis` would (with their current
content) leave someone clueless.
+1.
This may be a very naive question, apologies if it's nonsense.
Instead of Ellipsis, would it be possible to have a built-in LateBound
class and use instances of that class instead of Ellipsis?
The __str__ method of the inspect.Parameter class could be modified to
return something like
"a=>[]"
(or whatever syntax is adopted for specifying late-bound defaults) in
such cases.
The __repr__ and __str__ methods of a LateBound object could return
something like, respectively,
"LateBound('[]')"
"[]"
I am sure there is code that uses inspect.signature that would be
broken, but isn't that inevitable anyway?
That's a possibility for the inspect module. For the core language -
and therefore for anything that directly inspects the function's
dunders - it's much more efficient to use a well-known object.
OK.
I'm guessing that by "well-known" you mean pre-existing. Python has a
number of built-in singleton objects (None, True, False, Ellipsis).
Mainly by "well-known" I mean "not private to any particular module",
so those four you mention are all well-known, but a singleton as part
of the inspect module would be a pain, since the core language would
have to import that (or it would have to be magically created).
What about adding a new one called LateBound (or other bikeshed colour)?
What would be gained? You would still be able to use LateBound as an
early-bound default, so you would still need the same dual check.
I'm struggling here. Yes you could use LateBound as an early-bound
default (or as a parameter value to explicitly pass to a function) but
ISTM that such usages would be perverse.
I've a gut feeling that a solution can be found (to avoid the "Ellipsis
from nowhere" problem) but I can't put my finger on it. Maybe explicitly
specifying LateBound could be an error, perhaps even a SyntaxError?
Help, please!
It would be extremely odd if something could fail due to the precise
object chosen.
x = LateBound
def f(x=x): pass # SyntaxError? Runtime error?
Awkward. Problematic. And it wouldn't buy you anything anyway -
consumers of inspect.Signature would still have to be aware of this
special object, so you still have the same problems.
ChrisA
String representation, but exactly what the default is.
Excellent. And you've just proven that we can evaluate the defaults.
>>> a = '[]' # f.__defaults_extra__[0]
>>> b = 'len(lst)' # f.__defaults_extra__[1]
>>> lst = eval(a, globals())
>>> n = eval(b, globals(), dict(lst=lst))
>>> print(lst, n)
[] 0
Worst case scenario, we can eval the string representation, and that
should work for the great majority of cases that don't involve
nonlocals.
Yes, those awkward cases that involve nonlocals. A pity about those.
But hey, if that's not a problem to you, then sure, go ahead, just
eval it. The string is there for you to use.
But if the defaults are proper code objects, complete with a closure to
capture their nonlocal environment, then we should be able to do even
better and capture nonlocals as well.
Could there be odd corner cases that don't quite work? Say, like a
comprehension inside a class body?
Oh well. Let's not make the perfect the enemy of the good.
Lots and lots and lots of potential problems. Consider:
def f():
a = 1
def f(b, x=>a+b):
def g(): return x, a, b
Both a and b are closure variables - one because it comes from an
outer scope, one because it's used in an inner scope. So to evaluate
a+b, you have to look up an existing closure cell, AND construct a new
closure cell.
The only way to do that is for the compiled code of a+b to exist
entirely within the context of f's code object. Which means that there
isn't really anything to usefully pull out and evaluate, since it
can't be executed until you've set up a stack frame to call f, and
once you've done that, you're basically just... calling f anyway.
If you still dispute that it's impossible, you're absolutely welcome
to write your own reference implementation. I've argued this point
with as much detail as I can, and short of pointing stuff out in the
code, I can't do anything more. If you want to keep on saying "but of
COURSE it's possible", then go ahead, prove it to me.
ChrisA
Lots and lots and lots of potential problems. Consider:
def f():
a = 1
def f(b, x=>a+b):
def g(): return x, a, b
Both a and b are closure variables - one because it comes from an
outer scope, one because it's used in an inner scope. So to evaluate
a+b, you have to look up an existing closure cell, AND construct a new
closure cell.
The only way to do that is for the compiled code of a+b to exist
entirely within the context of f's code object.
I dispute that is the only way. Let's do a thought experiment.
First, we add a new flag to the co_flags field on code objects. Call it
the "LB" flag, for late-binding.
Second, we make this:
def f(b, x=>a+b): ...
syntactic sugar for this:
def f(b, x=lambda b: a+b): ...
except that the lambda has the LB flag set.
And third, when the interpreter fetches a default from
func.__defaults__, if it is a LB function, it automatically calls that
function with the parameters to the left of x (which in this case
would be just b).
Here's your function, with a couple of returns to make it actually do
something:
def f():
a = 1
def f(b, x=>a+b):
def g(): return x, a, b
return g
return f
We can test that right now (well, almost all of it) with this:
def func(): # change of name to distinguish inner and outer f
a = 1
def f(b, x=lambda b: a+b):
def g(): return x, a, b
return g
return f
and just pretend that x is automatically evaluated by the interpreter.
But as a proof of concept, it's enough that we can demonstrate that *we*
can manually evaluate it, by calling the lambda.
We can call func() to get the inner function f, and call f to get g:
>>> f = func()
>>> print(f)
>>> g = f(100)
>>> print(g)
Calling g works:
>>> print(g())
(, 1, 100)
with the understanding that the real implementation will have
automatically called that lambda, so we would have got 101 instead of
the lambda. That step requires interpreter support, so for now we just
have to pretend that we get
(101, 1, 100)
instead of the lambda. But we can demonstrate that calling the lambda
works, by manually calling it:
>>> x = g()[0]
>>> print(x)
>>> print(x(100)) # the interpreter knows that b=100
101
Now let's see if we can extract the default and play around with it:
>>> default_expression = f.__defaults__[0]
>>> print(default_expression)
The default expression is just a function (with the new LB flag set). So
we can inspect its name, its arguments, its cell variables, etc:
>>> default_expression.__closure__
(,)
We can do anything that we could do with any other other function object.
Can we evaluate it? Of course we can. And we can test it with any value
we like, we're not limited to the value of b that we originally passed
to func().
>>> default_expression(3000)
3001
Of course, if we are in a state of *maximal ignorance* we might have no
clue what information is needed to evaluate that default expression:
>>> default_expression()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: func.<locals>.<lambda>() missing 1 required positional argument: 'b'
Oh look, we get a useful diagnostic message for free!
What are we missing? The source code of the original expression, as
text. That's pretty easy too: the compiler knows the source, it can cram
it into the default expression object:
>>> default_expression.__expression__ = 'a+b'
Introspection tools like help() can learn to look for that.
What else are we missing? A cool repr.
>>> print(default_expression) # Simulated.
We can probably come up with a better repr, and a better name than "late
bound default expression". We already have other co_flags that change
the repr:
32 GENERATOR
128 COROUTINE
256 ITERABLE_COROUTINE
so we need a name that is at least as cool as "generator" or
"coroutine".
Summary of changes:
* add a new co_flag with a cool name better than "LB";
* add an `__expression__` dunder to hold the default expression;
(possibly missing for regular functions -- we don't necessarily
need *every* function to have this dunder)
* change the repr of LB functions to display the expression;
* teach the interpreter to compile late-bound defaults into one of
these LB functions, including the source expression;
* teach the interpreter that when retrieving default values from
the function's `__defaults__`, if they are a LB function, it
must call the function and use its return result as the actual
default value;
* update help() and other introspection tools to handle
these LB functions; but if any tools don't get updated,
you still get a useful result with an informative repr.
--
Steve
Lots and lots and lots of potential problems. Consider:
def f():
a = 1
def f(b, x=>a+b):
def g(): return x, a, b
Both a and b are closure variables - one because it comes from an
outer scope, one because it's used in an inner scope. So to evaluate
a+b, you have to look up an existing closure cell, AND construct a new
closure cell.
The only way to do that is for the compiled code of a+b to exist
entirely within the context of f's code object.
I dispute that is the only way. Let's do a thought experiment.
First, we add a new flag to the co_flags field on code objects. Call it
the "LB" flag, for late-binding.
Second, we make this:
def f(b, x=>a+b): ...
syntactic sugar for this:
def f(b, x=lambda b: a+b): ...
except that the lambda has the LB flag set.
Okay. So the references to 'a' and 'b' here are one more level of
function inside the actual function we're defining, which means you're
paying the price of nonlocals just to be able to late-evaluate
defaults. Not a deal-breaker, but that is a notable cost (every
reference to them inside the function will be slower).
And third, when the interpreter fetches a default from
func.__defaults__, if it is a LB function, it automatically calls that
function with the parameters to the left of x (which in this case
would be just b).
Plausible. Okay.
What this does mean, though, is that there are "magic objects" that
cannot be used like other objects. Consider:
def make_printer(dflt):
def func(x=dflt):
print("x is", x)
return func
Will make_printer behave the same way for all objects? Clearly the
expectation is that it will display the repr of whichever object is
passed to func, or if none is, whichever object is passed to
make_printer. But if you pass it a function with the magic LB flag
set, it will *execute* that function. I don't like the idea that some
objects will be invisibly different like that.
Here's your function, with a couple of returns to make it actually do
something:
def f():
a = 1
def f(b, x=>a+b):
def g(): return x, a, b
return g
return f
We can test that right now (well, almost all of it) with this:
def func(): # change of name to distinguish inner and outer f
a = 1
def f(b, x=lambda b: a+b):
def g(): return x, a, b
return g
return f
and just pretend that x is automatically evaluated by the interpreter.
But as a proof of concept, it's enough that we can demonstrate that *we*
can manually evaluate it, by calling the lambda.
Okay, sure. It's a bit hard to demo it (since it has to ONLY do that
magic if the arg was omitted), but sure, we can pretend.
We can call func() to get the inner function f, and call f to get g:
>>> f = func()
>>> print(f)
>>> g = f(100)
>>> print(g)
Calling g works:
>>> print(g())
(, 1, 100)
with the understanding that the real implementation will have
automatically called that lambda, so we would have got 101 instead of
the lambda. That step requires interpreter support, so for now we just
have to pretend that we get
(101, 1, 100)
instead of the lambda. But we can demonstrate that calling the lambda
works, by manually calling it:
>>> x = g()[0]
>>> print(x)
>>> print(x(100)) # the interpreter knows that b=100
101
Now let's see if we can extract the default and play around with it:
Can we evaluate it? Of course we can. And we can test it with any value
we like, we're not limited to the value of b that we originally passed
to func().
>>> default_expression(3000)
3001
Of course, if we are in a state of *maximal ignorance* we might have no
clue what information is needed to evaluate that default expression:
Oh look, we get a useful diagnostic message for free!
What are we missing? The source code of the original expression, as
text. That's pretty easy too: the compiler knows the source, it can cram
it into the default expression object:
>>> default_expression.__expression__ = 'a+b'
Introspection tools like help() can learn to look for that.
What else are we missing? A cool repr.
>>> print(default_expression) # Simulated.
We can probably come up with a better repr, and a better name than "late
bound default expression". We already have other co_flags that change
the repr:
32 GENERATOR
128 COROUTINE
256 ITERABLE_COROUTINE
so we need a name that is at least as cool as "generator" or
"coroutine".
* add a new co_flag with a cool name better than "LB";
* add an `__expression__` dunder to hold the default expression;
(possibly missing for regular functions -- we don't necessarily
need *every* function to have this dunder)
* change the repr of LB functions to display the expression;
* teach the interpreter to compile late-bound defaults into one of
these LB functions, including the source expression;
* teach the interpreter that when retrieving default values from
the function's `__defaults__`, if they are a LB function, it
must call the function and use its return result as the actual
default value;
* update help() and other introspection tools to handle
these LB functions; but if any tools don't get updated,
you still get a useful result with an informative repr.
Great. So now we have some magnificently magical behaviour in the
language, which will have some nice sharp edge cases, but which nobody
will ever notice. Totally. I'm sure. Plus, we pay a performance price
in any function that makes use of argument references, not just for
the late-bound default, but in the rest of the code. We also need to
have these special functions that get stored as separate code objects.
All to buy what, exactly? The ability to manually synthesize an
equivalent parameter value, as long as there's no assignment
expressions, no mutation, no other interactions, etc, etc, etc? That's
an awful lot of magic for not a lot of benefit.
I *really* don't like the idea that some types of object will be
executed instead of being used, just because they have a flag set.
That strikes me as the sort of thing that should be incredibly scary,
but since I can't think of any specific reasons, I just have to call
it "extremely off-putting".
But hey. Go ahead and build a reference implementation. I'll compile
it and give it a whirl.
ChrisA
Lots and lots and lots of potential problems. Consider:
def f():
a = 1
def f(b, x=>a+b):
def g(): return x, a, b
Both a and b are closure variables - one because it comes from an
outer scope, one because it's used in an inner scope. So to evaluate
a+b, you have to look up an existing closure cell, AND construct a new
closure cell.
The only way to do that is for the compiled code of a+b to exist
entirely within the context of f's code object.
I dispute that is the only way. Let's do a thought experiment.
There are many possible implementation of the late bound idea that could create an object/default expression.
But is it reasonable to bother with that added complexity/maintenance burden for a first implementation.
And maybe no one will care enough to ever implement the ability to modify the code of a late bound
variables expression as a separate object later.
I think I understand the argument as being along the lines of
for early bound defaults they can be inspected and modified.
Therefore being able to do the same for late bound defaults must be implemented.
I'm not convinced that that is reasonable to require is implemented.
If python had always had late bound defaults, as it is with most languages in the survey
posted earlier in this thread, would that have been implemented as an object/expression?
Maybe, but I doubt it.
Summary: I agree it's not impossible, I do not agree that it's needed.
Barry
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
(I suspect that there was a reply that I should be replying to but, cannot find one appropriate)
I have a lot of code that exploits the fact that passing an explicit None will cause the early bound default idiom to set the default for me.
def inner(timestamp=None):
if timestamp is None:
timestamp = time.time()
do_stuff...
def outer(timestamp=None):
inner(timestamp=timestamp)
outer can in an idiomatic way have inner default timestamp and not have to know what that means.
With late bound I cannot do this without more complex pattern of building an arg list.
What if passing None still worked? I know the argument that there are more sentinels then None.
def inner(timestamp=>time.time())
do_stuff...
def outer(timestamp=None):
inner(timestamp=timestamp)
The code in inner that decides to when to allow the default could check for timestamp being
missing or arg present and None.
Would the lack of support for other sentinels out weight the simple way to get the default applied?
Barry
And third, when the interpreter fetches a default from
func.__defaults__, if it is a LB function, it automatically calls that
function with the parameters to the left of x (which in this case
would be just b).
Plausible. Okay.
What this does mean, though, is that there are "magic objects" that
cannot be used like other objects. Consider:
Your proposal also has the same problem, since it involves "magic
functions" that do not have usable values for their argument defaults,
instead having some kind of Ellipsis two-step. It's all a matter of
what you consider magic.
Great. So now we have some magnificently magical behaviour in the
language, which will have some nice sharp edge cases, but which nobody
will ever notice. Totally. I'm sure. Plus, we pay a performance price
in any function that makes use of argument references, not just for
the late-bound default, but in the rest of the code. We also need to
have these special functions that get stored as separate code objects.
All to buy what, exactly? The ability to manually synthesize an
equivalent parameter value, as long as there's no assignment
expressions, no mutation, no other interactions, etc, etc, etc? That's
an awful lot of magic for not a lot of benefit.
I would consider most of what you say here an accurate description of
your own proposal. :-)
Now we have magnificently magical behavior in the language, which will
take expressions in the function signature and behind the scenes
"inline" them into the function body. We also need to have these
special function arguments that do NOT get stored as separate objects,
unlike ordinary function arguments. All to buy what, exactly? The
ability to write something in the function signature that we can already
write in the body, and that quite naturally belongs in the body, because
it is executed when the function is called, not when it is defined.
I *really* don't like the idea that some types of object will be
executed instead of being used, just because they have a flag set.
That strikes me as the sort of thing that should be incredibly scary,
but since I can't think of any specific reasons, I just have to call
it "extremely off-putting".
I *really* don't like the idea that some types of argument will be
inlined into the function body instead of being stored as first-class
values like other `__defaults__`, just because there happens to be this
one extra character next to the equals sign in the function signature.
That strikes me as the sort of thing that should be incredibly scary.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
And third, when the interpreter fetches a default from
func.__defaults__, if it is a LB function, it automatically calls that
function with the parameters to the left of x (which in this case
would be just b).
Plausible. Okay.
What this does mean, though, is that there are "magic objects" that
cannot be used like other objects. Consider:
Your proposal also has the same problem, since it involves "magic
functions" that do not have usable values for their argument defaults,
instead having some kind of Ellipsis two-step. It's all a matter of
what you consider magic.
My proposal allows any object to be used as a function default
argument. There's a minor technical difference that means that there's
a second lookup if you use Ellipsis, but you can still use Ellipsis
just fine.
There are no objects that will behave differently if used in this way.
EVERY object can be a function default argument. Steve's proposal has
some objects (functions with the LB flag set) actually behave
differently - they *will not behave correctly* if used in this way.
This is a restriction placed on the rest of the language.
Great. So now we have some magnificently magical behaviour in the
language, which will have some nice sharp edge cases, but which nobody
will ever notice. Totally. I'm sure. Plus, we pay a performance price
in any function that makes use of argument references, not just for
the late-bound default, but in the rest of the code. We also need to
have these special functions that get stored as separate code objects.
All to buy what, exactly? The ability to manually synthesize an
equivalent parameter value, as long as there's no assignment
expressions, no mutation, no other interactions, etc, etc, etc? That's
an awful lot of magic for not a lot of benefit.
I would consider most of what you say here an accurate description of
your own proposal. :-)
That's highly unfair. No, I won't let that pass. Please retract or
justify that statement. You are quoting the conclusion of a lengthy
post in which I show significant magic in Steve's proposal,
contrasting it with mine which has much clearer behaviour, and you
then say that my proposal has the same magic. Frankly, that is not a
reasonable assertion, and I take offense.
Now we have magnificently magical behavior in the language, which will
take expressions in the function signature and behind the scenes
"inline" them into the function body. We also need to have these
special function arguments that do NOT get stored as separate objects,
unlike ordinary function arguments. All to buy what, exactly? The
ability to write something in the function signature that we can already
write in the body, and that quite naturally belongs in the body, because
it is executed when the function is called, not when it is defined.
You assert that it "belongs in the body", but only because Python
currently doesn't allow it to be anywhere else. Other languages have
this exact information in the function signature. This is a much
larger distinction than what Steve shows, which is the exact same
feature but with these magic callables.
I *really* don't like the idea that some types of object will be
executed instead of being used, just because they have a flag set.
That strikes me as the sort of thing that should be incredibly scary,
but since I can't think of any specific reasons, I just have to call
it "extremely off-putting".
I *really* don't like the idea that some types of argument will be
inlined into the function body instead of being stored as first-class
values like other `__defaults__`, just because there happens to be this
one extra character next to the equals sign in the function signature.
That strikes me as the sort of thing that should be incredibly scary.
You're still being highly offensive here. There's a HUGE difference
between these two assertions. Steve's proposal makes some objects
*behave differently when used in existing features*. It would be like
creating a new type of string which, when printed out, would eval
itself. That proposal wouldn't fly, and it's why f-strings are most
assuredly NOT first-class objects.
Why is it such a big deal for these function default expressions to
not be first-class objects? None of these are first-class either:
print(f"An f-string's {x+y} subexpressions")
print(x/y if y else "An if/else expression's sides")
assign(x.y[42], "An assignment target")
We don't have a problem with these being unable to be externally
referenced, manipulated, etc, as first-class objects. Why is it a
problem to be unable to refer to "new empty list" as some sort of
object when used like this?
def f(x=>[]): ...
Can you explain why it is necessary? And then, after that, explain why
you claim that Steve's proposal, which makes some objects *not even
work in early-bound defaults*, is just as magical?
ChrisA
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
(I suspect that there was a reply that I should be replying to but, cannot find one appropriate)
I have a lot of code that exploits the fact that passing an explicit None will cause the early bound default idiom to set the default for me.
def inner(timestamp=None):
if timestamp is None:
timestamp = time.time()
do_stuff...
outer can in an idiomatic way have inner default timestamp and not have to know what that means.
If you need outer() to be able to have a value that means "use the
default", then there are three options:
1) Don't pass timestamp at all. In simple cases where it will only and
always specify the default, this is fine.
2) Define a sentinel that is indeed part of your API.
3) Use *args or **kwargs to choose whether to pass it or not (best if
there are multiple of them).
You can continue to use the existing system of "if none, do this", or
you can flip it around and have the sentinel as a special token within
your code:
def inner(timestamp=>time.time()):
if timestamp is None: timestamp = time.time()
Depends on how important this feature is outside of your own helper
functions. (I would probably not do this for None specifically - if
it's purely internal, I'm more likely to use a dedicated local
sentinel object.)
But as soon as there are two or three arguments that "might have to be
passed, might not", it's far more readable to use kwargs to pass just
the ones you want.
def outer(**kwargs):
inner(**kwargs)
That way, if something changes in inner(), you don't have to worry
about breaking your caller's API.
Okay. So the references to 'a' and 'b' here are one more level of
function inside the actual function we're defining, which means you're
paying the price of nonlocals just to be able to late-evaluate
defaults. Not a deal-breaker, but that is a notable cost (every
reference to them inside the function will be slower).
How much slower? By my tests:
- access to globals is 25% more expensive than access to locals;
- access to globals is 19% more expensive than nonlocals;
- and nonlocals are 6% more expensive than locals.
Or if you do the calculation the other way (the percentages don't match
because the denominators are different):
- locals are 20% faster than globals;
- and 5% faster than nonlocals;
- nonlocals are 16% faster than globals.
Premature optimization is the root of all evil.
We would be better off spending effort making nonlocals faster for
everyone than throwing out desirable features and a cleaner design just
to save 5% on a microbenchmark.
[...]
What this does mean, though, is that there are "magic objects" that
cannot be used like other objects.
NotImplemented says hello :-)
You are correct that one cannot use a LB function as a standard, early
bound default without triggering the "evaluate this at call time"
behaviour. If we're happy with this behaviour, it would need to be
documented for people to ignore *wink*
There's precedence though. You cannot overload an operator method to
return NotImplemented without triggering the special "your object
doesn't support this operator" behaviour.
And there are two obvious workarounds:
1. Just pass the LB function in as an explicit argument. The trigger
only operates when looking up a default, not on every access to a
function.
2. Or you can wrap the LB function you actually want to be the default
in a late-bound expression that returns that function.
And if you still think that we should care, we can come up with a more
complex trigger condition:
- the parameter was flagged as using a late-default;
- AND the default is a LB function.
Problem solved. Now you can use LB functions as early-bound defaults,
and all it costs is to record and check a flag for each parameter. Is it
worth it? Dunno.
[...]
Great. So now we have some magnificently magical behaviour in the
language, which will have some nice sharp edge cases, but which nobody
will ever notice. Totally. I'm sure.
NotImplemented. Document it and move on.
There are two work-arounds for those who care. And if you still think it
matters, you can record a flag for each parameter recording whether it
actually used a late-bound default or not.
Plus, we pay a performance price
in any function that makes use of argument references, not just for
the late-bound default, but in the rest of the code.
Using a late-bound default doesn't turn every local variable in your
function into a cell variable. For any function that does a meaningful
amount of work, the cost of making one or two parameters into cell
variables instead of local variables is negligible.
At worst, if you do *no other work at all*, it's a cost of about 5% on
two-fifths of bugger-all. But if your function does a lot of real work,
the difference between using cell variables instead of locals is going
to be insignificant compared to ~~the power of the Force~~ the rest of
the work done in the function.
And if you have some unbelievably critical function that you need to
optimize up the wahzoo?
def func(a, b=None):
if b is None:
# Look ma, no cell variables!
b = expression
Python trades off convenience for speed and safety all the time.
This will just be another such example. You want the convenience of a
late-bound default? Use this feature. You want it to be 3ns faster?
Use the old "if arg is None" idiom.
Or write your code in C, and make it 5000000000ns faster.
We also need to
have these special functions that get stored as separate code objects.
That's not a cost, that's a feature. Seriously. We're doing that so that
we can introspect them individually, not just as the source string, but
as actual callable objects that can be:
- introspected;
- tested;
- monkey-patched and modified in place (to the degree that any function
can be modified, which is not a lot);
- copied or replaced with a new function.
Testing is probably the big one. Test frameworks will soon develop a way
to let you write tests to confirm that your late bound defaults do what
you expect them to do.
That's trivial for `arg=[]` expressions, but for complex expressions in
complex functions, being able to isolate them for testing is a big plus.
--
Steve
Okay. So the references to 'a' and 'b' here are one more level of
function inside the actual function we're defining, which means you're
paying the price of nonlocals just to be able to late-evaluate
defaults. Not a deal-breaker, but that is a notable cost (every
reference to them inside the function will be slower).
How much slower? By my tests:
- access to globals is 25% more expensive than access to locals;
- access to globals is 19% more expensive than nonlocals;
- and nonlocals are 6% more expensive than locals.
Or if you do the calculation the other way (the percentages don't match
because the denominators are different):
- locals are 20% faster than globals;
- and 5% faster than nonlocals;
- nonlocals are 16% faster than globals.
Premature optimization is the root of all evil.
We would be better off spending effort making nonlocals faster for
everyone than throwing out desirable features and a cleaner design just
to save 5% on a microbenchmark.
Fair, but the desirable feature can be achieved without this cost, and
IMO your design isn't cleaner than the one I'm already using, and 5%
is a lot for no benefit.
And if you still think that we should care, we can come up with a more
complex trigger condition:
- the parameter was flagged as using a late-default;
- AND the default is a LB function.
Problem solved. Now you can use LB functions as early-bound defaults,
and all it costs is to record and check a flag for each parameter. Is it
worth it? Dunno.
Uhh.... so..... the parameter has to be flagged AND the value has to
be flagged? My current proposal just flags the parameter. So I ask
again: what are you gaining by this change? You've come right back to
where you started, and added extra costs and requirements, all for....
what?
f = lambda a, b: (len(w:=str(a))+b)*w
f('spam', 2)
'spamspamspamspamspamspam'
f.__code__
What sort of "behave differently" do you think would prevent us from
introspecting the function object? "Differently" from what?
Wrapping it in a function means the walrus would assign in that
function's context, not the outer function. I think it'd be surprising
if this works:
def f(x=>(a:=1)+a): # default is 2
but this doesn't:
def g(x=>(a:=1), y=>a): # default is UnboundLocalError
It's not a showstopper, but it is most definitely surprising.
The obvious solution is to say that, in this context, a is a nonlocal.
But this raises a new problem: The function object, when created, MUST
know its context. A code object says "this is a nonlocal", and a
function object says "when I'm called, this is my context". Which
means you can't have a function object that gets called externally,
because it's the code, not the function, that is what you need here.
And that means it's not directly executable, but it needs a context.
So, once again, we come right back around to what I have already: code
that you can't lift out and call externally. The difference is that,
by your proposal, there's a lot more overhead, for the benefit of
maybe under some very specific circumstances being able to synthesize
the result.
We also need to
have these special functions that get stored as separate code objects.
That's not a cost, that's a feature. Seriously. We're doing that so that
we can introspect them individually, not just as the source string, but
as actual callable objects that can be:
- introspected;
- tested;
- monkey-patched and modified in place (to the degree that any function
can be modified, which is not a lot);
- copied or replaced with a new function.
Testing is probably the big one. Test frameworks will soon develop a way
to let you write tests to confirm that your late bound defaults do what
you expect them to do.
That's trivial for `arg=[]` expressions, but for complex expressions in
complex functions, being able to isolate them for testing is a big plus.
I'm still not convinced that it's as useful as you say. Compare these
append-and-return functions:
def build1(value, lst=None):
if lst is None: lst = []
lst.append(value)
return lst
_SENTINEL = object()
def build2(value, lst=_SENTINEL):
if lst is _SENTINEL: lst = []
lst.append(value)
return lst
def hidden_sentinel():
_SENTINEL = object()
def build3(value, lst=_SENTINEL):
if lst is _SENTINEL: lst = []
lst.append(value)
return lst
return build3
build3 = hidden_sentinel()
def build4(value, *optional_lst):
if len(optional_lst) > 1: raise TypeError("too many args")
if not optional_lst: optional_lst = [[]]
optional_lst[0].append(value)
return optional_lst[0]
def build5(value, lst=>[]):
lst.append(value)
return lst
(Have I missed any other ways to do this?)
In which of them can you introspect the []? Three of them have an
externally-visible sentinel, but you can't usefully change it in any
way. You can look at it, and you'll see None or "
There are many possible implementation of the late bound idea that
could create an object/default expression.
But is it reasonable to bother with that added
complexity/maintenance burden for a first implementation.
Yes. If you don't do it, you'll have backward compatibility issues or
technical debt.
I'm not saying that's a compelling argument here, except that one of
the main alleged problems is that users don't understand mutable
defaults. So adding more and more layers of support for default
arguments is making matters worse, I suspect. (Remember, they're
going to be reading "arg=None" and "@arg=[]" for a long long time.)
This one is Worth Doing Right the first time, I think. And IMO David
Mertz is right: doing it right means a more general deferred-evaluation
object (not to be confused with Deferreds that need to be queried
about their value).
And maybe no one will care enough to ever implement the ability to
modify the code of a late bound variables expression as a separate
object later.
Hear! Hear! That's exactly how I feel about *this* proposal! With
all due respect to Chris and Steve who have done great work
advocating, implementing, and clarifying the proposal, IAGNU (I am
gonna not use). Too much muscle memory, and more important, existing
code whose style I want to be consistent and don't wanna mess with
because it works, around "arg=None".
There are many possible implementation of the late bound idea that
could create an object/default expression.
But is it reasonable to bother with that added
complexity/maintenance burden for a first implementation.
Yes. If you don't do it, you'll have backward compatibility issues or
technical debt.
I'm not saying that's a compelling argument here, except that one of
the main alleged problems is that users don't understand mutable
defaults. So adding more and more layers of support for default
arguments is making matters worse, I suspect. (Remember, they're
going to be reading "arg=None" and "@arg=[]" for a long long time.)
This one is Worth Doing Right the first time, I think. And IMO David
Mertz is right: doing it right means a more general deferred-evaluation
object (not to be confused with Deferreds that need to be queried
about their value).
If you think that deferred evaluation objects are the right way to do
it, then write up a proposal to compete with PEP 671. In my opinion,
it is a completely independent idea, which has its own merit, and
which is not a complete replacement for late-bound defaults; the two
could coexist in Python simultaneously, or either one could be
accepted without the other, or we could continue to have neither. Yes,
there's some overlap in the problems they solve, just as there's
overlap between PEP 671 and PEP 661 on named sentinels; but there's
also overlap between plenty of other language features, and we don't
deem try/finally or context managers to be useless because of the
other.
ChrisA
(It's absolutely valid to say "yes" and "yes", and feel free to say
which of those pulls is the stronger one.)
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
That depends on what you mean by "use." I wouldn't *write* code that
uses it (I can't find many (if any) cases of (a), (b), or (c) in my
code), but I would have to *read* other people's code that does.
FWIW, the PEP doesn't mention mutability or mutable values at all.
Also FWIW, I still think that if you're doing (b) or (c), then you're
*not* doing default values anymore, you're moving pieces of the logic or
the design into the wrong place. One example of (b) goes something like
this:
def write_to_log(event, time=>current_time()):
actually_write_to_log(event, time)
IOW, default to the current time, but allow the caller to specify a some
other time instead. Maybe I'm old school, or overly pedantic, but IMO,
those are two different use cases, and there should be two separate
functions (potentially with separate authorization and/or notations in
the log, or maybe I've spent too much time deciphering badly designed
logs and log entries). *Maybe* a better example would be something like
this:
def write_to_log(event, id=>generate_appropriate_uuid()):
actually_write_to_log(event, id)
but I would still personally rather (for testability and maintainability
reasons) write two functions, even (or perhaps especially) if they both
called a common lower-level function to do the actual work.
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
Yes, and no. (Seriously: Apparently, I don't create APIs, in any
language, that would/could/might benefit from late binding default
values. What would I be trying?)
The ability to write something in the function signature that we
can already write in the body, and that quite naturally belongs in
the body, because it is executed when the function is called, not
when it is defined.
I'm basically in sympathy with your conclusion, but I don't think it's
useful to prejudice the argument by saying it *naturally belongs* in
the body. Some languages quite naturally support thunks/blocks (Ruby)
or even go so far as to equate code to data (Lisp), and execute that
code in lieu of what appear to be variable references. But maybe
it's *Pythonic* to necessarily place the source code in the body? I
can't say that.
I *really* don't like the idea that some types of object will be
executed instead of being used, just because they have a flag
set.
From a syntactic point of view, that's how Ruby blocks work. Closer
to home, that's how properties work. And in the end, *all* objects
are accessed by executing code. This is a distinction without a
difference, except in our heads. I wouldn't want to be asked to
explain the dividing line between objects that were "just used" and
objects that were "produced by code that was executed instead of being
just used".
I *really* don't like the idea that some types of argument will be
inlined into the function body instead of being stored as first-class
values like other `__defaults__`, just because there happens to be this
one extra character next to the equals sign in the function signature.
That strikes me as the sort of thing that should be incredibly
scary.
Properties have *no* visible syntax if they're imported from a module.
Properties are extremely useful, and we all use them all the time
without noticing or caring. I see no reason in principle why the same
kind of feature wouldn't be useful and just as invisible and just as
"natural" for local or global variables -- or callable parameters, as
long as properly restricted. Chris's proposal is nothing if not
restricted! :-)
My issues with Chris's proposal are described elsewhere, but I don't
really see a problem in principle.
Also FWIW, I still think that if you're doing (b) or (c), then you're
*not* doing default values anymore, you're moving pieces of the logic or
the design into the wrong place. One example of (b) goes something like
this:
Very very common use-case for that:
https://pyauth.github.io/pyotp/#time-based-otps
The vast majority of calls are going to leave the time parameter at
the default. (The one I linked to has separate "at" and "now"
functions, but combining them makes very good sense.)
ChrisA
This one is Worth Doing Right the first time, I think. And IMO David
Mertz is right: doing it right means a more general deferred-evaluation
object (not to be confused with Deferreds that need to be queried
about their value).
If you think that deferred evaluation objects are the right way to do
it, then write up a proposal to compete with PEP 671.
That's not your call, I'm afraid. "Keep the status quo" is always a
viable option, regardless of other options. And other things equal,
it's the preferred option.
In my opinion, it is a completely independent idea,
You're welcome to your opinion, of course. But if you want to claim
that's a reason for implementing your proposal, you need to support
it. You also need to explain why the additional potential complexity
of a third kind of default argument (evaluated at definition,
evaluated during call, evaluated when referenced) isn't a big problem.
which is not a complete replacement for late-bound defaults;
Why not? If we have such objects, we could simply specify that in the
case where such an object is specified as a function parameter
default, it is evaluated in the same environment as your late-bound
defaults. So we can have your schsemantics if that's what we want.
On the other hand, it might turn out that 90% of the time, it doesn't
matter if it's evaluated as part of the calling process, 9% of the
time the natural place to evaluate it is at the point of first use,
and 1% of the time it should be evaluated before the function body
proper is entered. In that case the late-bound default would be of
some utility, but is it worth it? David's "infinitesimal utility"
argument seems likely to apply.
What else would such a deferred-evaluation object be unable to do that
your late-bound default can do?
That's where I am. More than any of these issues, the lack of a well-
defined, properly introspectable object bothers me. In 2021, we
should provide that.
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
(I suspect that there was a reply that I should be replying to but, cannot find one appropriate)
I have a lot of code that exploits the fact that passing an explicit None will cause the early bound default idiom to set the default for me.
def inner(timestamp=None):
if timestamp is None:
timestamp = time.time()
do_stuff...
outer can in an idiomatic way have inner default timestamp and not have to know what that means.
If you need outer() to be able to have a value that means "use the
default", then there are three options:
1) Don't pass timestamp at all. In simple cases where it will only and
always specify the default, this is fine.
2) Define a sentinel that is indeed part of your API.
3) Use *args or **kwargs to choose whether to pass it or not (best if
there are multiple of them).
You can continue to use the existing system of "if none, do this", or
you can flip it around and have the sentinel as a special token within
your code:
def inner(timestamp=>time.time()):
if timestamp is None: timestamp = time.time()
And, obviously, if you end up needing the write the explicit check for None there is no
advantage to using late bound default.
Depends on how important this feature is outside of your own helper
functions. (I would probably not do this for None specifically - if
it's purely internal, I'm more likely to use a dedicated local
sentinel object.)
But as soon as there are two or three arguments that "might have to be
passed, might not", it's far more readable to use kwargs to pass just
the ones you want.
def outer(**kwargs):
inner(**kwargs)
That way, if something changes in inner(), you don't have to worry
about breaking your caller's API.
Yes that's a good point. Use the *kwargs style to pass down stuff.
The code in inner that decides to when to allow the default could check for timestamp being
missing or arg present and None.
Would the lack of support for other sentinels out weight the simple way to get the default applied?
None is most assuredly not going to trigger a late-bound default.
Are you state that this is because in most of the cases where I might think that I need this behaviour there are better patterns to use like *kwargs?
Is that worth stating in the PEP in the rejected ideas?
def inner(timestamp=>time.time()):
if timestamp is None: timestamp = time.time()
And, obviously, if you end up needing the write the explicit check for None there is no
advantage to using late bound default.
Hmm, I wouldn't say NO advantage - it still puts the primary and
meaningful default in the signature, but then has a special case for
backward compatibility, or for that one caller where it makes sense,
or whatever. But yes, far less advantage when you actually have None
as part of your API. The main point of late-bound defaults is to
remove None from the API.
Yes that's a good point. Use the *kwargs style to pass down stuff.
That's normally the recommendation anyway. It's safe against
additional parameters being added, it's safe against the defaults
changing, it clearly says "this passes on its arguments unchanged";
the only problem is that the help for the function doesn't adequately
show what parameters it can actually accept. And that's a fundamental
problem - look at these docs:
https://docs.python.org/3/library/subprocess.html#subprocess.run
"""The arguments shown above are merely the most common ones,
described below in Frequently Used Arguments (hence the use of
keyword-only notation in the abbreviated signature)."""
The docs aren't restricted to what can be implemented in an actual
function signature, yet it's still most effective to just toss in
"**other_popen_kwargs" at the end.
(That said, though: it would be rather nice to be able to do algebra
with function signatures. For instance, you could say "my signature is
that function's kwargs plus frobnosticate=42" or "my signature is that
function's kwargs minus stdin". But that's a topic for another thread
or another day.)
None is most assuredly not going to trigger a late-bound default.
Are you state that this is because in most of the cases where I might think that I need this behaviour there are better patterns to use like *kwargs?
Is that worth stating in the PEP in the rejected ideas?
I don't think so, because None doesn't mean "omit this argument". It
is a perfectly valid value. There's also no need to say that object()
won't trigger late-bound defaults, or 0, or anything else. The only
way to cause a default argument to be evaluated is to not pass the
argument - as is already the case.
Yeah :) I say this because, in JavaScript, there is fundamentally no
difference between passing the special value 'undefined' (kinda like
None, although there's also null as a separate value) and not passing
the argument at all, which means that...
function foo(x="hello") {console.log("x is " + x);}
foo(undefined);
foo(foo.any_unknown_attr);
will print "x is hello" twice. I don't want that :) And that's why
there is, by definition, no value that will cause a function to think
that an argument wasn't passed.
ChrisA
There are many possible implementation of the late bound idea that
could create an object/default expression. But is it reasonable to
bother with that added complexity/maintenance burden for a first
implementation.
I don't think we can conclude that factoring out the late-bound defaults
into their own routines is "added complexity/maintenance burden".
Usually, factoring stand-alone code into its own routines *reduces* the
maintenance burden, it doesn't increase it. You can test chunks of code
in isolation. You can refactor it. It is easier to analyse and
understand.
The ability to introspect code is not a "burden", it is a feature that
reduces the difficulty of testing and maintaining code.
That's why we have functions in the first place, instead of having one
giant ball of mud.
https://exceptionnotfound.net/big-ball-of-mud-the-daily-software-anti-patter...
Chris wants to throw the late-bound defaults into the body of the
function because that's what we are currently forced to do to emulate
late-bound defaults. But we are designing the feature from scratch, and
we shouldn't be bound by the limitations of the existing idiom.
Encapsulation is good. Putting late-bound expressions in their own
function so that they can be tested is a good thing, not a problem to be
avoided. Most of us have, from time to time, already moved the
late bound default into its own function just to make it easy to test in
isolation:
def func(arg=None):
if arg is None:
arg = _implementation()
# and now handle arg
With my strategy, we get that isolation for almost for free. There is no
extra effort needed on the programmer's side, no new name or top-level
function needed.
Will people take advantage of it? Of course people won't bother if the
default is `[]` but beyond a certain level of complexity, people will
want to test the default expressions in isolation, to be sure that it
does the right thing.
--
Steve
Also FWIW, I still think that if you're doing (b) or (c), then you're
*not* doing default values anymore, you're moving pieces of the logic or
the design into the wrong place. One example of (b) goes something like
this:
I agree. *Not* conflating timestamps and event IDs is a good thing!
APIs and libraries like that are making my point: the very notion of
"overriding the current time" is a bad one. The notion of "defaulting
to the current time" might be okay, in some systems, until it isn't.
Any feature can be abused, but I don't think we should be using bad
designs and bad APIs to justify the feature in the first place.
The vast majority of calls are going to leave the time parameter at
the default. (The one I linked to has separate "at" and "now"
functions, but combining them makes very good sense.)
I disagree. Combining/conflating the time an event occurred and the
time it's actually logged doesn't make sense at all. Or maybe I've
spent too much time rummaging through logs from concurrent and parallel
systems.
Oh, wait, we're veering off topic, but you like you said, this is Python
Ideas! ;-)
There are many possible implementation of the late bound idea that
could create an object/default expression. But is it reasonable to
bother with that added complexity/maintenance burden for a first
implementation.
Chris wants to throw the late-bound defaults into the body of the
function because that's what we are currently forced to do to emulate
late-bound defaults. But we are designing the feature from scratch, and
we shouldn't be bound by the limitations of the existing idiom.
Not quite true. I want to have them syntactically as part of the body
of the function, and semantically as part of the function call. As a
function begins executing, a ton of stuff happens, including
allocating arguments to parameters, and providing default values for
optional parameters that weren't passed. All I want to change is the
way that defaults can be provided.
Tell me, what's the first bytecode instruction in the function g here?
def f(x):
print(x)
return lambda: x
In current CPython, it's not "LOAD_GLOBAL print". It's "MAKE_CELL x".
Conceptually, that's part of the framework of setting up the function
- setting up its local variables, providing a place for nonlocals. But
it's a bytecode at the start of the function's code object. (I'm not
sure when that became a bytecode instruction. It wasn't one in Python
2.7.) CPython bytecode is an implementation detail, and the fact that
there's bytecode to do this or that is not part of the language
definition.
Encapsulation is good. Putting late-bound expressions in their own
function so that they can be tested is a good thing, not a problem to be
avoided. Most of us have, from time to time, already moved the
late bound default into its own function just to make it easy to test in
isolation:
def func(arg=None):
if arg is None:
arg = _implementation()
# and now handle arg
With my strategy, we get that isolation for almost for free. There is no
extra effort needed on the programmer's side, no new name or top-level
function needed.
And if you want an _implementation function for external testing,
you're still welcome to do that.
def func(arg=>_implementation()):
...
No magic, just perfectly normal coding practices. If something's
important enough to test separately, refactor it into a private
function so you can call it. Why should this happen automatically for
late-bound argument defaults? From what I can tell, the only syntactic
constructs which form separate code objects are those which need them
for namespacing purposes: class blocks, nested functions, genexps,
comprehensions. Default argument values are not separate namespaces,
so they shouldn't need dedicated stack frames, dedicated code objects,
or anything like that.
Will people take advantage of it? Of course people won't bother if the
default is `[]` but beyond a certain level of complexity, people will
want to test the default expressions in isolation, to be sure that it
does the right thing.
Beyond a certain level of complexity, the fact that argument defaults
are in the signature will naturally pressure people to refactor them
into their own functions. I don't think we'll see people putting
multiline argument default expressions directly into the function
header.
ChrisA
Also FWIW, I still think that if you're doing (b) or (c), then you're
*not* doing default values anymore, you're moving pieces of the logic or
the design into the wrong place. One example of (b) goes something like
this:
I agree. *Not* conflating timestamps and event IDs is a good thing!
APIs and libraries like that are making my point: the very notion of
"overriding the current time" is a bad one. The notion of "defaulting
to the current time" might be okay, in some systems, until it isn't.
Time-based OTPs are using timestamps. That's what they do. Defaulting
to the current time is *precisely* how most 2FA systems work. Being
able to override the time is useful primarily for testing. So for the
TOTP case, I would say that "timestamp=>time.time()" is the perfect
way to spell it.
The vast majority of calls are going to leave the time parameter at
the default. (The one I linked to has separate "at" and "now"
functions, but combining them makes very good sense.)
I disagree. Combining/conflating the time an event occurred and the
time it's actually logged doesn't make sense at all. Or maybe I've
spent too much time rummaging through logs from concurrent and parallel
systems.
Oh, wait, we're veering off topic, but you like you said, this is Python
Ideas! ;-)
I don't know why you'd have something in a logger that lets you
configure the time, but my guess would be that it's the same thing:
you can unit-test the logger with consistent inputs. For instance:
def format_log_line(event, time=>current_time(), host=>get_host()):
return ...
# shorthand, obv you'd be using a proper testing framework
assert format_log_line({...}, time=1638717131, host="example") == "..."
TBH, I think that defaulting to "event happened right now" is about as
good a default as you'll ever get. In some situations you'll know when
the event happened... but honestly, I'd rather know when the log line
happened too. So if I have an event with an inbuilt timestamp, I'll
incorporate that into the *body* of the log line, and still have the
logger add its own timestamp.
But maybe I've spent too much time rummaging through logs from buggy systems.
ChrisA
Steven gave the following example of a function signature that would be
difficult to visually parse if this proposal and arrow lambdas were
accepted:
def process(func:List->int=>xs=>expression)->int:
And while I agree that it does sort of stop you in your tracks when you see
this, I think there are a couple of reasons why this is not as big of a
problem as it appears.
I totally agree with you that, for even moderately experienced
Pythonistas, it is possible to parse that. It's not literally ambiguous
syntax that cannot be resolved. I am confident that the parser will be
able to work it out just fine :-)
I care more about the poor reader who may not be a moderately
experienced Pythonista, and will be trying to work out why there are two
different arrow symbols with four different meanings:
* function return annotation
* typing.Callable type annotation
* lambda alternative syntax
* late-bound defaults
I'm sure that people will learn the many uses of the arrow symbols.
After all, people learn Perl and APL :-) Closer to home, we also learn
all the many different uses of the star symbol `*`.
Yes, spaces will help the reader *parse* the pieces of the function
signature. But spaces doesn't help the reader decipher and remember the
different meanings of the arrows. Also the use of extra spaces goes
against the usual style guides that we *don't* use spaces between the
colon and the annotation, or the equals sign and the default.
For the sake of discussion, I've been using Chris' arrow symbol, but
that doesn't mean I've warmed to it. Aside from the other issues, it is
the wrong way around:
parameter => expression
implies moving the parameter into the expression, which is the
wrong way around. The expression moves into the parameter.
E.g. in R, you can write assignment with an equals sign, or an arrow,
but the arrow points from the value to the variable:
> x <- 1
> 2 -> y
> c(x, y)
[1] 1 2
I've never seen a language or pseudo-code that does assignment with the
arrow pointing from the variable to the value. Does anyone know of any?
--
Steve
And if you still think that we should care, we can come up with a more
complex trigger condition:
- the parameter was flagged as using a late-default;
- AND the default is a LB function.
Problem solved. Now you can use LB functions as early-bound defaults,
and all it costs is to record and check a flag for each parameter. Is it
worth it? Dunno.
Uhh.... so..... the parameter has to be flagged AND the value has to
be flagged? My current proposal just flags the parameter.
*shrug* Is that a problem? Okay, then we can just flag the parameter,
and use a plain ol' function with no special co_flag, although we do
surely still want to give it an attribute `__expression__` to hold the
source expression.
So I ask again: what are you gaining by this change?
Separation of concerns. Testing. Cleaner design (the body of the
function contains only the byte code from the body of the function).
Testing. Introspection. Testing. You don't have to hack the byte-code of
the function in order to monkey-patch defaults. Testing.
In case it wasn't obvious, I think that testing is important :-)
Other people might think of other uses.
[...]
What sort of "behave differently" do you think would prevent us from
introspecting the function object? "Differently" from what?
Wrapping it in a function means the walrus would assign in that
function's context, not the outer function.
Unless the variable was a cell variable of the outer function. Which I
think that they need to be. Didn't you already decide that walrus
assignment has to be in the context of the function?
That's not a rhetorical question, I genuinely don't remember.
But this raises a new problem: The function object, when created, MUST
know its context. A code object says "this is a nonlocal", and a
function object says "when I'm called, this is my context". Which
means you can't have a function object that gets called externally,
because it's the code, not the function, that is what you need here.
And that means it's not directly executable, but it needs a context.
Sorry Chris, I don't understand what you are trying to say here. If I
take what you are saying literally, I would take you as trying to say
that closures can never be executed. But they clearly can be, and I know
that you know that. So obviously I am misunderstanding you.
I don't understand why you think that we can't take one of those
Late-Bound (LB) functions, which will be a closure, and call it like we
can call any other closure.
By the time we have access to the LB functions, the owner function will
exist, so there shouldn't be any problem with the context not existing.
So, once again, we come right back around to what I have already: code
that you can't lift out and call externally. The difference is that,
by your proposal, there's a lot more overhead, for the benefit of
maybe under some very specific circumstances being able to synthesize
the result.
Surely it is the other way around? If you are correct, there are some
extremely specific cicumstances involving the use of walrus operator in
two different default expressions such that you cannot call one of the
two functions, but the 99.99% of straight-forward non-walrus, non-tricky
default expressions will work perfectly fine as independently callable
functions.
I'm still not convinced that it's as useful as you say. Compare these
append-and-return functions:
def build1(value, lst=None):
if lst is None: lst = []
[...]
In which of them can you introspect the []?
"Here are lots of ways to solve a problem that don't let you test or
introspect part of your code. Therefore we shouldn't add a feature that
will let us more easily test and introspect that part of the code."
Yes Chris, I know that emulated late-bound defaults are currently
effectively invisible at runtime (short of byte-code hacking). That
weakness of the status quo is why you have written this PEP.
That invisibility is not a feature we should duplicate in late-bound
defaults, but an argument for introducing defaults that overcome that
weakness.
Right now, your PEP adds a new feature that gives us *zero* new
functionality. There is nothing your PEP allows us to do that we cannot
already do. It adds no new functionality. It's just syntactic sugar for
"if arg is missing: return expression". You compile that code (or
something very close to it) into the body of the function.
The only wins are (1) when people stumble into the "mutable default"
gotcha, it is easy to tell them how to fix it (but not much easier than
the status quo) and (2) a small increase in clarity, since we will be
able to write the default expression directly in the signature instead
of hiding behind a sentinel.
That's not nothing, but we've seen on this list a fair number of people
who think that the increase in clarity is not enough to justify the
feature. Do you think the Steering Council will consider those two small
wins sufficient?
But with a different design, we have not just the increase in clarity,
but also open the door to a ton of new possibilities that come about
thanks to the separation of those default expressions into their own
objects. I don't know whether that will be enough to sway the Steering
Council. They might hate my idea even more. But feedback on this list
suggests that your design's lack of introspectability is putting people
off the idea.
It may be that this feature will never be used for anything more complex
than `param=>[]`. In which case, okay, I concede, moving the default
expression into its own code unit (a function? something else?) is
overkill.
But I don't think that people will only use this for such trivial
expressions. Maybe 60% of the cases will be just a literal list display
or dict display, and maybe another 30% will be a simple call to an
existing function, like `time.ctime()`.
But its the last 10% of non-trivial, interesting expressions where
people will want to do more than just inspect the string. And having the
expressions as first-class code units will open up those new
opportunities that we currently lack.
--
Steve
What sort of "behave differently" do you think would prevent us from
introspecting the function object? "Differently" from what?
Wrapping it in a function means the walrus would assign in that
function's context, not the outer function.
Unless the variable was a cell variable of the outer function. Which I
think that they need to be. Didn't you already decide that walrus
assignment has to be in the context of the function?
That's not a rhetorical question, I genuinely don't remember.
Yes, it does, and that won't be the case if you create an invisible
nested function.
But this raises a new problem: The function object, when created, MUST
know its context. A code object says "this is a nonlocal", and a
function object says "when I'm called, this is my context". Which
means you can't have a function object that gets called externally,
because it's the code, not the function, that is what you need here.
And that means it's not directly executable, but it needs a context.
Sorry Chris, I don't understand what you are trying to say here. If I
take what you are saying literally, I would take you as trying to say
that closures can never be executed. But they clearly can be, and I know
that you know that. So obviously I am misunderstanding you.
Closures cannot be executed without a context. Consider:
def f(x=lambda: (a:=[])):
if isinstance(x, FunctionType): x = x()
print(a)
Here's the problem: The name 'a' should be in the context of f, but
that context *does not exist* until f starts executing. As an
early-bound default, like this, it's not going to work, because the
function object has to be fully constructed at function definition
time.
If you try to do this sort of thing with a magical function that is
synthesized as part of late-bound default handling, you'll still need
to figure out when f's context gets formed. Normally, it won't get
formed until the stack frame for f gets constructed - which is when
arguments get assigned to parameters, etc, etc. Trying to call that
inner function without first giving it a context won't work.
And if all you have is a code object without a function, what's the
use? How is it better than just having code in the function itself?
I don't understand why you think that we can't take one of those
Late-Bound (LB) functions, which will be a closure, and call it like we
can call any other closure.
By the time we have access to the LB functions, the owner function will
exist, so there shouldn't be any problem with the context not existing.
The function will exist. The instance of it won't. Consider this basic
demo of closures:
def counter():
n = 1
def incr():
nonlocal n
n += 1
return n
return incr
Prior to calling counter(), you can disassemble incr's bytecode and
see what it does. But you can't call incr. Its context does not yet
exist. Until you call counter, there is actually no function for incr,
only the code object. And while it is possible to eval() a code
object...
eval(counter.__code__.co_consts[2])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: code object passed to eval() may not contain free variables
... you can't if it has nonlocals, because it needs a context. So if
there's any possibility of nonlocals in latebound argument defaults,
you won't be able to call this externally, which leaves me wondering:
why all this hassle if it isn't even going to work?
So, once again, we come right back around to what I have already: code
that you can't lift out and call externally. The difference is that,
by your proposal, there's a lot more overhead, for the benefit of
maybe under some very specific circumstances being able to synthesize
the result.
Surely it is the other way around? If you are correct, there are some
extremely specific cicumstances involving the use of walrus operator in
two different default expressions such that you cannot call one of the
two functions, but the 99.99% of straight-forward non-walrus, non-tricky
default expressions will work perfectly fine as independently callable
functions.
To be honest, 99.99% of late-bound defaults would probably work just
fine if you simply eval their texts. But that doesn't justify having
functions either.
I'm still not convinced that it's as useful as you say. Compare these
append-and-return functions:
def build1(value, lst=None):
if lst is None: lst = []
[...]
In which of them can you introspect the []?
"Here are lots of ways to solve a problem that don't let you test or
introspect part of your code. Therefore we shouldn't add a feature that
will let us more easily test and introspect that part of the code."
Yes Chris, I know that emulated late-bound defaults are currently
effectively invisible at runtime (short of byte-code hacking). That
weakness of the status quo is why you have written this PEP.
That invisibility is not a feature we should duplicate in late-bound
defaults, but an argument for introducing defaults that overcome that
weakness.
Okay, but you're asking for a LOT of complexity for a very small part
of the feature.
Right now, your PEP adds a new feature that gives us *zero* new
functionality. There is nothing your PEP allows us to do that we cannot
already do. It adds no new functionality. It's just syntactic sugar for
"if arg is missing: return expression". You compile that code (or
something very close to it) into the body of the function.
MOST new features give us zero new functionality. That's what it means
for a language to be Turing-complete. But "if arg is missing:" doesn't
currently exist, nor does "this argument is optional and has no
default", so you can't quite write this currently, and that's why we
need sentinels.
The only wins are (1) when people stumble into the "mutable default"
gotcha, it is easy to tell them how to fix it (but not much easier than
the status quo) and (2) a small increase in clarity, since we will be
able to write the default expression directly in the signature instead
of hiding behind a sentinel.
That's not nothing, but we've seen on this list a fair number of people
who think that the increase in clarity is not enough to justify the
feature. Do you think the Steering Council will consider those two small
wins sufficient?
I'm hoping so. That's the main thrust of what I'm doing. Clarity, and
an easier way to make mutable defaults working. And more intuitive
behaviour in some situations. Clarity, mutable defaults, and intuitive
behaviour. And nice red uniforms.
But with a different design, we have not just the increase in clarity,
but also open the door to a ton of new possibilities that come about
thanks to the separation of those default expressions into their own
objects. I don't know whether that will be enough to sway the Steering
Council. They might hate my idea even more. But feedback on this list
suggests that your design's lack of introspectability is putting people
off the idea.
Or it's suggesting that people on this list love to argue. I don't
think we're surprised by that. :)
(And I don't have a problem with it.)
It may be that this feature will never be used for anything more complex
than `param=>[]`. In which case, okay, I concede, moving the default
expression into its own code unit (a function? something else?) is
overkill.
But I don't think that people will only use this for such trivial
expressions. Maybe 60% of the cases will be just a literal list display
or dict display, and maybe another 30% will be a simple call to an
existing function, like `time.ctime()`.
But its the last 10% of non-trivial, interesting expressions where
people will want to do more than just inspect the string. And having the
expressions as first-class code units will open up those new
opportunities that we currently lack.
If you want to write up a competing reference implementation, go
ahead. I don't think it will be as easy as you claim. And I'm not
going to mention the possibility in the PEP *without* a reference
implementation, nor do I intend to write one myself (since I dispute
its usefulness).
Will the benefits outweigh the costs? I don't know. But I think not,
and that the much simpler proposal will be better.
ChrisA
As a function begins executing, a ton of stuff happens, including
allocating arguments to parameters, and providing default values for
optional parameters that weren't passed.
And neither of those things are part of the function body. They are
set up *before* the function's code object is called.
All I want to change is the way that defaults can be provided.
Right. And the community reaction on this mailing list is underwhelming.
Apart from Abe, who is rather vigourly in favour of the PEP as it
stands, I'm not really sure this PEP has much support as it stands now.
And one of the most frequent issues raised is that **all** you want to
do is change the way that the late defaults can be provided.
You're the PEP author, but maybe you should consider listening to
community feedback? Just sayin'.
CPython bytecode is an implementation detail, and the fact that
there's bytecode to do this or that is not part of the language
definition.
Bytecode can and does change.
I'm talking about the language definition: late bound defaults should be
compiled into their own publicly accessible callable functions. I don't
care what bytecode is emitted to do that. Whatever it is, it will change
in the future, without changing the Python semantics one whit.
You've just made a good argument against your own PEP. We don't need new
syntax for late-bound defaults, just use the perfectly normal coding
practices that have worked fine for 30 years.
If something's
important enough to test separately, refactor it into a private
function so you can call it. Why should this happen automatically for
late-bound argument defaults?
Sure. And why should early-bound defaults be available in a publicly
accessible dunder attribute, when they could be compiled directly into
the function code object?
Hell, the entire code object, and most of the function object, could
be compiled into one great big opaque ball of mud. Aside from standard
attributes inherited from object, all the function really needs is a
name and a binary blob of everything else.
But that's not the sort of language we have.
--
Steve
I want to have them syntactically as part of the body
of the function, and semantically as part of the function call.
Then you'll love the status quo, because that's exactly what we have
now! *wink*
Aaaaaand that's what I get for typing emails at 2AM. LOL.
Syntactically as part of the signature of the function, is what I
meant to say. Thanks for the correction.
As a function begins executing, a ton of stuff happens, including
allocating arguments to parameters, and providing default values for
optional parameters that weren't passed.
And neither of those things are part of the function body. They are
set up *before* the function's code object is called.
All I want to change is the way that defaults can be provided.
Right. And the community reaction on this mailing list is underwhelming.
Apart from Abe, who is rather vigourly in favour of the PEP as it
stands, I'm not really sure this PEP has much support as it stands now.
And one of the most frequent issues raised is that **all** you want to
do is change the way that the late defaults can be provided.
You're the PEP author, but maybe you should consider listening to
community feedback? Just sayin'.
Believe you me, I am listening. And quite frankly, the tone of this
list is sounding like "shut up, go away, don't do anything, because
there are other proposals that nobody can be bothered writing up, but
if they existed, they'd be way better than what you're doing".
Not exactly encouraging, since nobody is willing to do the work of
writing up proposals, but is plenty willing to criticize.
CPython bytecode is an implementation detail, and the fact that
there's bytecode to do this or that is not part of the language
definition.
Bytecode can and does change.
I'm talking about the language definition: late bound defaults should be
compiled into their own publicly accessible callable functions. I don't
care what bytecode is emitted to do that. Whatever it is, it will change
in the future, without changing the Python semantics one whit.
Yes. So why is it a problem for the default's evaluation to be part of
the function's bytecode? I don't understand why this is such a big
hassle.
You've just made a good argument against your own PEP. We don't need new
syntax for late-bound defaults, just use the perfectly normal coding
practices that have worked fine for 30 years.
Hmm.... you mean that "arg=>_implementation()" works currently?
You're complaining that complicated argument defaults can't be
separately tested, and that we need magic to do the refactoring for us
and make them testable. I say that, if it needs to be separately
tested, refactor it, and if it's simple enough to not be worth
refactoring, it doesn't get separately tested.
That's an argument against your "break it into a separate magical
subfunction" proposal, but not against the PEP itself.
Sure. And why should early-bound defaults be available in a publicly
accessible dunder attribute, when they could be compiled directly into
the function code object?
Because they're part of the FUNCTION, not the CODE. There's a big
difference. Consider:
for i in range(10):
def func(i=i): ...
There is only one code object. There are multiple function objects,
each with their own early-bound defaults. By definition, since the
defaults are evaluated as part of the function definition, the code to
evaluate that MUST be part of the surrounding context, not the
function being defined.
And it's not introspectable at the moment, only the resulting value
is. You are asking for far more than the language currently offers in
any related context.
Hell, the entire code object, and most of the function object, could
be compiled into one great big opaque ball of mud. Aside from standard
attributes inherited from object, all the function really needs is a
name and a binary blob of everything else.
But that's not the sort of language we have.
Indeed. We have all kinds of useful introspection tools. And
late-bound defaults already have quite a few introspection features,
most notably a source-code-reconstruction of the expression. That's
more than you get for early-bound defaults.
(In discussing this, I have noted a hole in the specification: it
would be nice to have a way to see, on the code object, which args
have late-bound defaults. Consider that a specification bug to be
fixed.)
You're setting the bar for PEP 671 a long way higher than literally
everything else in the language... and yet you refuse to code your own
reference implementation for the better version that you propose. Any
plans to put some code behind your words?
ChrisA
On Wed, Dec 01, 2021 at 12:26:33PM +0000, Matt del Valle wrote:
I'm sure that people will learn the many uses of the arrow symbols.
After all, people learn Perl and APL :-) Closer to home, we also learn
all the many different uses of the star symbol `*`.
Not to mention the half-dozen or so uses of colon (introduce a suite,
slicing, dictionary displays, annotations, lambdas). Anybody still have
a problem with them? Raise your hand and I'll give you your money back.😁
Best wishes
Rob Cliffe
(That said, though: it would be rather nice to be able to do algebra
with function signatures. For instance, you could say "my signature is
that function's kwargs plus frobnosticate=42" or "my signature is that
function's kwargs minus stdin". But that's a topic for another thread
or another day.)
Heck, or even " my signature is that other function's signature" -- that is
what passing *args, **kwargs does, but you have to look at the
implementation to know.
As it happens, right now, someone on my team is trying out an
implementation that uses inspect to grab the signature of superclass
methods so that we can have a complete function signature without repeating
ourselves all over the place. Not sure that's a good idea, but it would be
cool if there were a standard and reliable way to do that.
But yes, topic for another day.
None is most assuredly not going to trigger a late-bound default.
I don't think so, because None doesn't mean "omit this argument". It
is a perfectly valid value. There's also no need to say that object()
won't trigger late-bound defaults, or 0, or anything else. The only
way to cause a default argument to be evaluated is to not pass the
argument - as is already the case.
But I'd like to see as a (perhaps rejected) idea is to have a new sentinel
that does mean undefined.Sure there could be (rare) cases where you would
need to have it a valid value, but then maybe you can't use late-bound
defaults in that case.
This is very different from None, because it would be new, so no one is
already using it for anything else. And sure, folks could choose to use it
inappropriately, but consenting adults and all that.
Yeah :) I say this because, in JavaScript, there is fundamentally no
difference between passing the special value 'undefined' (kinda like
None, although there's also null as a separate value) and not passing
the argument at all, which means that...
function foo(x="hello") {console.log("x is " + x);}
foo(undefined);
foo(foo.any_unknown_attr);
will print "x is hello" twice. I don't want that :)
Sure, that's ugly, but isn't the real problem here that
foo(foo.any_unknown_attr) doesn't raise an Exception? Would we have that
same issue in Python?
e.g., doesn't Javascript already have:
foo();
foo(foo.any_unknown_attr);
lead to the same thing? whereas in Python, that would raise, yes?
Or is there something special about undefined that I'm missing?
(sorry, I don't really "get" Javascript)
I personally think more standard special purpose sentinels would be a good
idea, though I understand the arguments made against that in previous
discussions. But this is a little different, because late-bound defaults
are a new thing, so we don't already have a body of code using None, or
anything else for "use the late bound default".
-CHB
--
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
And quite frankly, the tone of this list is sounding like "shut up, go
away, don't do anything, because there are other proposals that nobody can
be bothered writing up, but if they existed, they'd be way better than what
you're doing". Not exactly encouraging, since nobody is willing to do the
work of writing up proposals, but is plenty willing to criticize.
I'll write up my proposal:
"Keep the status quo"
All done.
I admit I may be a overly vociferous in my opposition to this particular
change. But I also think your tone has been rather consistently pugnacious,
and a bit combative, in dismissing objections or disagreements.
I know you genuinely wish to improve Python, and believe this PEP would do
so. But I think you've become attached to your idea in a way that becomes
non-constructive and unlikely to garner support.
For example, your careful exclusion of alternative ideas from the PEP feels
less than forthcoming. It almost feels like you are trying to hide the
alternatives from the SC. Obviously, many of them read this list, and all
of them will think of basically the same ideas others have suggested on
their own anyway.
I first discussed the idea of a "generalized deferred object/type" on this
list at least two years ago, probably more than three (I haven't looked
through archives lately to be sure the dates). The idea got some vague
interest, but I was too lazy, or too busy, or whatever, to write an actual
PEP or implementation.
It's fine to criticize my inaction in advancing the more general idea. But
the result of my failing isn't "therefore PEP 671 should be adopted" as you
keep claiming. It's just that I haven't done the work to flesh out the
encompassing idea that would cover late-binding as a minor aspect.
As an analogy, PEP 275 was written in 2001 and rejected/neglected. PEP 3103
was rejected in 2006. The very simple case switch on literals was thought
not to be broad enough to change Python syntax, despite being a construct
in numerous other programming languages.
Then in 2020, PEP 622 was written, widely discussed and refined, and
adopted. PEP 622 does EVERYTHING that PEP 275 would have, 19 years earlier,
and even with pretty much the same syntax. But it also does MUCH more as
well, and hence was thought to be worth a syntax change.
PEP 671 is very much the same. It does something worthwhile. But it does
vastly less than needed to warrant new syntax and semantics. I hope it
takes less than 19 years, but a generalized deferred construct is worth
waiting for.
(That said, though: it would be rather nice to be able to do algebra
with function signatures. For instance, you could say "my signature is
that function's kwargs plus frobnosticate=42" or "my signature is that
function's kwargs minus stdin". But that's a topic for another thread
or another day.)
Heck, or even " my signature is that other function's signature" -- that is what passing *args, **kwargs does, but you have to look at the implementation to know.
Ah, if it's absolutely exactly "that other function's signature", then
set func.__wrapped__ to the other function (that's what
@functools.wraps does to make the signature work). But that's the only
option. You can't do modifications in this way:
def func(*args, **kwargs, frobnosticate=42):
...
basefunc(**args, **kwargs)
def func(*args, **kwargs):
if "stdin" in kwargs:
stdin = kwargs.pop("stdin")
...
basefunc(*args, **kwargs)
You have to do all-or-nothing at the moment. I don't know of a good
way to make this happen, but if someone has a brilliant idea, I'd love
to hear one.
As it happens, right now, someone on my team is trying out an implementation that uses inspect to grab the signature of superclass methods so that we can have a complete function signature without repeating ourselves all over the place. Not sure that's a good idea, but it would be cool if there were a standard and reliable way to do that.
If you mean that it's chasing all the way up the class hierarchy,
building the signature piece by piece, then you're right, there's no
easy way to do that at the moment. Signature algebra would allow you
to do that - you'd have each function say "that function, but these
changes" - but how you specify that is the hard part.
None is most assuredly not going to trigger a late-bound default.
I don't think so, because None doesn't mean "omit this argument". It
is a perfectly valid value. There's also no need to say that object()
won't trigger late-bound defaults, or 0, or anything else. The only
way to cause a default argument to be evaluated is to not pass the
argument - as is already the case.
But I'd like to see as a (perhaps rejected) idea is to have a new sentinel that does mean undefined.Sure there could be (rare) cases where you would need to have it a valid value, but then maybe you can't use late-bound defaults in that case.
This is very different from None, because it would be new, so no one is already using it for anything else. And sure, folks could choose to use it inappropriately, but consenting adults and all that.
I think that would be more confusion than it's worth. Having an object
with no value is a means of major insanity. I'll give a specific
example of something that frustrated me from JavaScript; this kind of
API is very common:
// If "foo" is present, remove it. Otherwise, add it
classList.toggle("foo")
// Add "foo"
classList.toggle("foo", 1)
// Remove "foo"
classList.toggle("foo", 0)
These are defined using the standard truthiness rules: any true value
will add, any false value will remove. And if you don't pass anything
at all, it toggles.
Next up, consider this:
state = {
"title": "some title",
"is_open": true/false,
"etc": etc,
}
// Check to see if the thing is open or not
if (state.is_open) {...} else {...}
Cool. Nice and easy. Both of these use the truthiness rules. Seems
pretty normal, right?
Unfortunately, there is ONE special value which doesn't behave the
same way: undefined. In an 'if' statement, undefined is falsy, so
you'd go into the 'else' clause. Just like None in Python, just like
every language with a concept of truthiness/falsiness. But in the
toggle() call, undefined is indistinguishable from not passing the
argument at all. So instead of removing, it will toggle.
That specific issue cost me some debugging time, because I didn't even
think to ask the question "why is state.is_open undefined instead of
false" - because in every other way, undefined did indeed behave as if
it were false.
Yeah :) I say this because, in JavaScript, there is fundamentally no
difference between passing the special value 'undefined' (kinda like
None, although there's also null as a separate value) and not passing
the argument at all, which means that...
function foo(x="hello") {console.log("x is " + x);}
foo(undefined);
foo(foo.any_unknown_attr);
will print "x is hello" twice. I don't want that :)
Sure, that's ugly, but isn't the real problem here that foo(foo.any_unknown_attr) doesn't raise an Exception? Would we have that same issue in Python?
Well, true, that particular part of it is handled by that. But in
Python, you might get something from a dictionary using
foo.get("any_unknown_key") and you'll still get back None. What's
important is that in Python, instead of getting back a null value that
behaves as if you didn't pass anything at all, you get back a value
that is a real thing in and of itself. None isn't the absence of a
value - it is a value that has real meaning. (And real attributes,
though not many of them.)
lead to the same thing? whereas in Python, that would raise, yes?
Or is there something special about undefined that I'm missing?
(sorry, I don't really "get" Javascript)
There are two halves to that second example: firstly, that
foo.any_unknown_attr is undefined rather than being an error; and
secondly, that passing undefined to a function is indistinguishable
from not passing an argument. You can get undefined from all kinds of
sources, and it's often used in places where Python would use None, so
that part isn't a major problem. The problem is that, in a function
call, it ceases to be a value, and becomes a non-value.
(And don't worry. Nobody really "gets" JavaScript. We just use it
anyway, since it's the thing that browsers are most comfortable with.)
I personally think more standard special purpose sentinels would be a good idea, though I understand the arguments made against that in previous discussions. But this is a little different, because late-bound defaults are a new thing, so we don't already have a body of code using None, or anything else for "use the late bound default".
Hmm, the problem with a multitude of standard special-purpose
sentinels is that, inevitably, you need something that lets you
enumerate all standard sentinels, and also say "nothing to see here".
So you'll end up needing your own private sentinel. The standard ones
don't end up buying you much, unless they have meaning to the language
itself (as NotImplemented does).
ChrisA
And quite frankly, the tone of this list is sounding like "shut up, go away, don't do anything, because there are other proposals that nobody can be bothered writing up, but if they existed, they'd be way better than what you're doing". Not exactly encouraging, since nobody is willing to do the work of writing up proposals, but is plenty willing to criticize.
I'll write up my proposal:
"Keep the status quo"
All done.
PEP 671 is very much the same. It does something worthwhile. But it does vastly less than needed to warrant new syntax and semantics. I hope it takes less than 19 years, but a generalized deferred construct is worth waiting for.
You: "Keep the status quo, all done"
Also you: "Let's wait for something better"
That's what I take issue with. You are simultaneously telling me that
this proposal is bad because there's another proposal that would be
better, AND saying that you don't want to push for any other proposal.
So you're welcome to keep waiting. Meanwhile, I'm going to try to
actually accomplish something NOW, not wait for some hypothetical
future.
ChrisA
You: "Keep the status quo, all done"
Also you: "Let's wait for something better"
Now is better than never.
Although never is often better than *right* now.
Neither of which says "the distant and uncertain future of an
unwritten proposal is better than either now or never". If you JUST
said "keep the status quo, all done", that is a reasonable and
consistent position (which I disagree with, but I fully respect). But
you then taint your claims with this statement that the reason for
keeping the status quo is not that the status quo is better, but that
there is a hypothetical idea that might be even better. That's not
"now is better than never". That's not "never is better than right
now" either.
I think it's time for me to drop this entire topic for a while. This
list is getting more and more noisy and I'm going to stop contributing
to that.
ChrisA
And quite frankly, the tone of this list is sounding like "shut
up, go away, don't do anything, because there are other proposals
that nobody can be bothered writing up, but if they existed,
they'd be way better than what you're doing". Not exactly
encouraging, since nobody is willing to do the work of writing up
proposals, but is plenty willing to criticize.
I'll write up my proposal:
"Keep the status quo"
All done.
I admit I may be a overly vociferous in my opposition to this
particular change. But I also think your tone has been rather
consistently pugnacious, and a bit combative, in dismissing objections
or disagreements.
I know you genuinely wish to *improve Python*, and believe this PEP
*would* do so. But I think you've become attached to your idea in a
way that becomes non-constructive and unlikely to garner support.
For example, your careful exclusion of alternative ideas from the PEP
feels less than forthcoming. It almost feels like you are trying to
hide the alternatives from the SC. Obviously, many of them read this
list, and all of them will think of basically the same ideas others
have suggested on their own anyway.
I am sure Chris A can answer for himself, but IMO the above is, frankly,
insulting. It almost feels like you have run out of arguments and are
resorting to a personal attack.
I first discussed the idea of a "generalized deferred object/type" on
this list at least two years ago, probably more than three (I haven't
looked through archives lately to be sure the dates). The idea got
some vague interest, but I was too lazy, or too busy, or whatever, to
write an actual PEP or implementation.
It's fine to criticize my inaction in advancing the more general idea.
But the result of my failing isn't "therefore PEP 671 should be
adopted" as you keep claiming. It's just that I haven't done the work
to flesh out the encompassing idea that would cover late-binding as a
minor aspect.
Nobody has attempted (or at least completed) a PEP, never mind an
implementation, of a "generalized deferred object/type", in the last N
years or decades. And no reason to suppose that anyone will in the next
N years or decades. (I am sure it is very difficult.) And I think it
is fair to say that opinion is mixed on the benefits of such a
On 05/12/2021 18:37, David Mertz, Ph.D. wrote:
proposal. I think it is also fair to say that such a proposal need not
be incompatible with PEP 671.
Meanwhile we have a completed PEP and implementation (though obviously
changes might still be made) that "*would*" "*improve Python*".
As an analogy, PEP 275 was written in 2001 and rejected/neglected. PEP
3103 was rejected in 2006. The very simple case switch on literals was
thought not to be broad enough to change Python syntax, despite being
a construct in numerous other programming languages.
Then in 2020, PEP 622 was written, widely discussed and refined, and
adopted. PEP 622 does EVERYTHING that PEP 275 would have, 19 years
earlier, and even with pretty much the same syntax. But it also does
MUCH more as well, and hence was thought to be worth a syntax change.
That is a fair point which, I admit, I struggle somewhat to refute. Let
me say:
It would carry more weight if we had a time-line, even a vague one,
for implementing a "generalized deferred object/type". Nobody seems to
be able and prepared to do the work. Without that, Python could wait
indefinitely for this (PEP 671's) improvement.
In my very personal, subjective opinion, PEP 622 feels more like a
luxury. Anything that can be done with it could be done with
pre-existing syntax. Whereas late-bound defaults feels more like a ...
hm, "necessity" is obviously the wrong word, I think I mean "basic
feature". (And AFAIU can not really be implemented with existing syntax
- well, Turing-complete and all that, but I hope you know what I mean.)
As witness, 12 of the 16 languages that Steven d'Aprano surveyed provide
it. YMMV.
Best wishes
Rob Cliffe
PEP 671 is very much the same. It does something worthwhile. But it
does vastly less than needed to warrant new syntax and semantics. I
hope it takes less than 19 years, but a generalized deferred construct
is worth waiting for.
I agree. *Not* conflating timestamps and event IDs is a good thing!
APIs and libraries like that are making my point: the very notion of
"overriding the current time" is a bad one. The notion of "defaulting
to the current time" might be okay, in some systems, until it isn't.
Time-based OTPs are using timestamps. That's what they do. Defaulting
to the current time is *precisely* how most 2FA systems work. Being
able to override the time is useful primarily for testing. So for the
TOTP case, I would say that "timestamp=>time.time()" is the perfect
way to spell it.
If time-based OTPs use timestamps, then why is there a timestamp
parameter at all? "Current time" is part of the function, not part of
the API.
Testing functions like that is another problem; adding parameters to the
API does not solve it. IMO, a better solution is a test harness that
can provide a known "current time" value.
[...]
I don't know why you'd have something in a logger that lets you
configure the time, but my guess would be that it's the same thing:
you can unit-test the logger with consistent inputs. For instance:
It's not a question of configuring *the* time, it's a question of
recognizing that there's more than one time: the time the event
occurred is different from the time the event is logged. Yes, in many
cases in many systems, it's a difference without a distinction. In
other systems, timestamps added by loggers are wholly irrelevant.
Out of curiosity, why did you make host a late-binding parameter?
# shorthand, obv you'd be using a proper testing framework
assert format_log_line({...}, time=1638717131, host="example") == "..."
TBH, I think that defaulting to "event happened right now" is about as
good a default as you'll ever get. In some situations you'll know when
the event happened... but honestly, I'd rather know when the log line
happened too. So if I have an event with an inbuilt timestamp, I'll
incorporate that into the *body* of the log line, and still have the
logger add its own timestamp.
But maybe I've spent too much time rummaging through logs from buggy systems.
In time-critical code, I'm not going to waste resources (time, memory,
CPU cycles) formatting a log entry. The event occurred and has a
timestamp; formatting and logging will happen in a whole different
context (another thread? another CPU? another OS? the O&M system at
the time the user asks to look at the logs?). I'm not denying that
there are times you want both timestamps; I'm denying that you can
always conflate them without losing important information.
I'm sticking to my story, which is no doubt a product of the sorts of
systems I've built (and debugged, and not built): the apparent use case
of "defaulting to a value that changes, like the current time or a
function-generated ID" is conflating the logic of the function with that
function's API.
I read PEP 671 today, and I feel PEP 671 is not as useful as someone expected.
For example, PEP 671 has this example:
def bisect_right(a, x, lo=0, hi=>len(a), *, key=None):
But I think we can not change the signature for backward
compatibility. For example,
def search(self, x, lo=0, hi=None):
return bisect_right(self._data, x, lo=lo, hi=hi)
If we just change the signature of bisect_right, this wrapper method
will be broken.
So bisect_right should support None for several versions and emit
frustrating DeprecationWarning.
I don't think this change has good cost/performance.
Additionally, this example illustrates that PEP 671 is not wrapper
functions friendly.
If the wrapped functions uses PEP 671, wrapper functions should:
* Copy & paste all default expressions, or
* But default expression may contain module private variables...
* Use **kwds and hide real signatures.
I have not read all of PEP 671 threads so I am sorry if this is
already discussed.
But this topic is not covered in current PEP 671 yet.
Generally speaking, I don't want to add anything to Python language
that makes Python more complex.
But if I chose one PEP, I prefer PEP 505 than PEP 671.
PEP 505 can be used for default parameters (e.g. `hi ??= len(a)`) and
many other places.
I feel it has far better benefit / language complexity ratio.
Regards,
--
Inada Naoki
I first discussed the idea of a "generalized deferred object/type" on this
list at least two years ago, probably more than three (I haven't looked
through archives lately to be sure the dates). The idea got some vague
interest, but I was too lazy, or too busy, or whatever, to write an actual
PEP or implementation.
I don’t think a full PEP is required at this point, but throughout this
discussion, the idea of a deferred object being a better solution to this
same problem has been brought up.
My sense it that some folks think if we want late-bound defaults, we really
should just do deferred objects.
But I honestly don’t get it. My idea of a deferred object would be quite
different that this, would not be a great replacement for this, and could
quite happily co-exist with this idea. Clearly I’m missing something.
So I think a paragraph or two explaining what is meant by defers objects,
and how they’d be a better way to address late-bound defaults would be very
helpful to the conversation.
It's fine to criticize my inaction in advancing the more general idea. But
the result of my failing isn't "therefore PEP 671 should be adopted" as you
keep claiming.
Of course not— but you (and I honk others) have used the idea of the
possibility of some future deferred construct being a reason to reject
this idea. So you have some obligation to explain.
PEP 671 is very much the same. It does something worthwhile. But it does
vastly less than needed to warrant new syntax and semantics.
I’m personally ambivalent about that at this point, but you can make that
case without referring to some possible new feature.
-CHB
--
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
Nobody has attempted (or at least completed) a PEP, never mind an
implementation, of a "generalized deferred object/type", in the last N
years or decades.
Haskell anything. Ruby blocks. Closer to home, properties and closures.
So I don't think the object part is that hard, it's the syntax and
semantics that's devilish. The evaluation syntax doesn't seem hard:
you just reference the identifier the object is bound to. The
creation syntax, though, probably needs to be concise, maybe even
familiar, as Chris's proposal is.
For the semantics, just in the case of defining an alternative to PEP
671, consider: if you just stuff a generalized deferred expression
object into a formal argument's default, you still have to solve the
problems that Chris's proposal does: where do you get the appropriate
namespace to extract variable values from? and when do you do it? I
do admire the elegance of Chris's solution to those questions, but if
I were to fit a generalized deferred into that role, I might not adopt
the "evaluate deferreds just before entering the function body proper"
strategy.
And then there's the question of whether referencing the deferred
"collapses the waveform" into an object and binds it to the original
identifier (as in Chris's proposal, but for more general contexts), or
whether the deferred gets reevaluated each time in the current context
(as Ruby blocks and Python properties usually do).
Steve
Nobody has attempted (or at least completed) a PEP, never mind an
implementation, of a "generalized deferred object/type", in the last N
years or decades.
Haskell anything. Ruby blocks. Closer to home, properties and closures.
So I don't think the object part is that hard, it's the syntax and
semantics that's devilish. The evaluation syntax doesn't seem hard:
you just reference the identifier the object is bound to. The
creation syntax, though, probably needs to be concise, maybe even
familiar, as Chris's proposal is.
For the semantics, just in the case of defining an alternative to PEP
671, consider: if you just stuff a generalized deferred expression
object into a formal argument's default, you still have to solve the
problems that Chris's proposal does: where do you get the appropriate
namespace to extract variable values from? and when do you do it? I
do admire the elegance of Chris's solution to those questions, but if
I were to fit a generalized deferred into that role, I might not adopt
the "evaluate deferreds just before entering the function body proper"
strategy.
And then there's the question of whether referencing the deferred
"collapses the waveform" into an object and binds it to the original
identifier (as in Chris's proposal, but for more general contexts), or
whether the deferred gets reevaluated each time in the current context
(as Ruby blocks and Python properties usually do).
Steve
I think you're making my point.
You're saying that the object part isn't that hard, but other parts of
it are. Which overall means that it is a hard problem. And there seems
to be no likelihood of anyone tackling it soon. (If anyone is working
on it now, or planning to soon, please correct me.)
I can't see the point of rejecting something that provides a tangible
benefit, now, because some fictitious vapourware *might*, one day,
provide another way of doing it.
Best wishes
Rob Cliffe
There are no objects that will behave differently if used in this way.
EVERY object can be a function default argument. Steve's proposal has
some objects (functions with the LB flag set) actually behave
differently - they *will not behave correctly* if used in this way.
This is a restriction placed on the rest of the language.
Its a restriction in name only.
I've already given you two work-arounds for the restriction, and one
solution to that (non-)problem. We can change the trigger condition if
its really an issue. But if we don't, there are still ways to get the
result that you want.
How many ways are there to remove the restriction that operator dunders
cannot return NotImplemented as a first-class value? Zero.
Another precedence is the `__new__` dunder. Classes can return anything
they like from the constructor, but if you return an instance of the
class, it will automatically trigger calling the `__init__` method,
whether you want it to or not. You probably see that as a feature.
In my "late-bound protocol", the fact that a function with the "late
bound" flag triggers calling the function is likewise a feature, not a
bug.
Unlike NotImplemented and `__new__`, there are at least two work-arounds
for the rare case where you don't want that.
And as I said, if this is really a show-stopper, we can change the
trigger condition. There are pros and cons and I'm not wedded to one way
or the other.
You assert that it "belongs in the body", but only because Python
currently doesn't allow it to be anywhere else. Other languages have
this exact information in the function signature.
This argument about where the evaluation of the default expression is
surreal.
Brendan argues in favour of the status quo, because he thinks that the
evaluation of the default expression "belongs in the body".
You (Chris) argue in favour of your PEP, where the bytecode of the
default expression is inlined into the function's body, because you
insist that it belongs in the body. You justify that claim by making
spurious arguments that it is "impossible" to do otherwise (your term,
not mine). But it isn't impossible, and I shall justify that claim in
another post. A bold claim for somebody who knows nothing about the C
implementation :-)
The bottom line is, you both are arguing at each other because you
both want the default expression to be executed in the body of the
function o_O
Everyone please chill a bit.
Taking offence at vigourous but still respectful intellectual
disagreement is not kind or welcoming and most of all it is not
*helpful*. Brendan has not accused you of molesting children or
betraying a position of trust. He hasn't even suggested you have bad
breath. He has merely disagreed with your interpretation of a proposed
programming language feature. Is it really worth taking offence over
that?
And if you think Brendan is not being *respectful* because his opinion
is so obviously *wrong* (in your view), well, fine. Suppose you are
right. Does it matter? Should you challenge him to pistols at dawn for
besmirching your honour?
Chris, I know this has been a long, hard PEP for you. Here is a feature
that people in the Python community say they want, you write a PEP
proposing it, and you get nothing but negativity. Ouch :-(
Well, not nothing but negativity. There has been some support. And I
didn't spend six hours researching and writing up evaluation strategies
of nearly two dozen languages because I'm trying to sabotage your
proposal. You and I may disagree with some details of this PEP, but I'm
supporting it and want it to be the best late-bound feature it can be.
--
Steve
Nobody has attempted (or at least completed) a PEP, never mind an
implementation, of a "generalized deferred object/type", in the last N
years or decades.
Haskell anything. Ruby blocks. Closer to home, properties and closures.
So I don't think the object part is that hard, it's the syntax and
semantics that's devilish.
At one level, it's trivial. A deferred expression is `lambda:
expression`. Evaluating it is `deferred_expr()`.
What's not at all obvious is the requirements beyond that - what do
people *actually* want that isn't covered by this. The most obvious
answer is that they don't want to have to check for a deferred
expression and explicitly evaluate it, which triggers the question,
when do they want the language to evaluate it for them? "Every time"
doesn't work, because then you can't treat deferred expressions as
first class objects - they keep disappearing on you ;-)
So IMO it's the *requirements* that are hard. Maybe that's just me
using different words for the same thing you were saying, but to me,
the distinction is important. People throw around the term "deferred
object", but everyone seems to think that everyone else understands
what they mean by that term, and yet no-one will give a precise
definition.
We can't have a PEP or an implementation until we know what we're
proposing/implementing.
I don't intend to champion a "deferred objects" proposal, but I do
think that they (whatever they are) would be a better (more general)
solution than late-bound arguments. So here's a possible minimal
definition of what a "deferred object" is. It takes the view that
explicitly requesting the evaluation of a deferred is OK, but people
don't want to have to check it's a deferred before evaluating.
1. `defer EXPR` creates a "deferred object", that is semantically
identical to `lambda: EXPR`, except that it isn't a callable, instead
it's a new type of object.
2. `undefer EXPR` is exactly the same as `EXPR`, except that if `EXPR`
evaluates to a deferred object, it gets called (in the sense of it
being equivalent to a lambda which can be called).
Here's a prototype implementation, and a demonstration of how it would
be used to implement late bound arguments. Please note, I understand
that the syntax here is horrible. That's exactly the point, this needs
language support to be non-horrible. That's what a "deferred
expression" proposal would provide.
# Explicitly creating Deferred objects is horrible, this is the bit
that *really* needs language support
class Deferred:
def __init__(self, callable):
self.callable = callable
# This could easily be a builtin function (or an operator if people
prefer syntax) once we have deferred objects.
def undefer(expr):
if isinstance(expr, Deferred):
return expr.callable()
return expr
x = 12
# def f(a=defer x):
def f(a=Deferred(lambda: x)):
a = undefer(a)
return a
assert f(9) == 9
assert f() == 12
x = 8
assert f() == 8
assert f(9) == 9
If anyone wants to take this and make a *proper* deferred object
proposal out of it, then please do so. If not, then at a minimum I
think this offers something vaguely concrete to discuss regarding the
"why deferred objects are a more general solution to the late bound
argument" question.
Paul
You assert that it "belongs in the body", but only because Python
currently doesn't allow it to be anywhere else. Other languages have
this exact information in the function signature.
This argument about where the evaluation of the default expression is
surreal.
Brendan argues in favour of the status quo, because he thinks that the
evaluation of the default expression "belongs in the body".
You (Chris) argue in favour of your PEP, where the bytecode of the
default expression is inlined into the function's body, because you
insist that it belongs in the body. You justify that claim by making
spurious arguments that it is "impossible" to do otherwise (your term,
not mine). But it isn't impossible, and I shall justify that claim in
another post. A bold claim for somebody who knows nothing about the C
implementation :-)
I am arguing that it belongs in the *header*. I think I once mistyped
that and said "body" by mistake, but I have never intentionally
advocated for that (and you called me out on it promptly for a
correction). Default expressions, whether evaluated at definition time
or call time, belong in the header.
With definition-time expressions, their code is inlined into the body
of the surrounding context. With call-time expressions, their code is
inlined into the body of the function. Neither of these is part of the
signature, but that's because the signature doesn't itself have any
code.
The bottom line is, you both are arguing at each other because you
both want the default expression to be executed in the body of the
function o_O
You're conflating implementation with syntax. I want it to be
implemented in the body, because that's clean and efficient *for
implementation*. I want it to be written in the signature, because
that's where function defaults belong.
You're asking for it to be written in the signature, same as I am, but
then to have it executed... somewhere else. But still in basically the
same place.
Taking offence at vigourous but still respectful intellectual
disagreement is not kind or welcoming and most of all it is not
*helpful*. Brendan has not accused you of molesting children or
betraying a position of trust. He hasn't even suggested you have bad
breath. He has merely disagreed with your interpretation of a proposed
programming language feature. Is it really worth taking offence over
that?
He apologized privately and we have settled that matter. Let's drop it.
Chris, I know this has been a long, hard PEP for you. Here is a feature
that people in the Python community say they want, you write a PEP
proposing it, and you get nothing but negativity. Ouch :-(
Well, not nothing but negativity. There has been some support. And I
didn't spend six hours researching and writing up evaluation strategies
of nearly two dozen languages because I'm trying to sabotage your
proposal. You and I may disagree with some details of this PEP, but I'm
supporting it and want it to be the best late-bound feature it can be.
Thank you, I do appreciate that. Still, it does get somewhat tiresome
when there is a dogpile of negativity about the proposal, much of it
unbacked by any sort of code, and just declaring that what I've said
doesn't work or isn't ideal or should be left off because of a
hypothetical future feature. Very tiresome.
ChrisA
Here's a prototype implementation, and a demonstration of how it would
be used to implement late bound arguments. Please note, I understand
that the syntax here is horrible. That's exactly the point, this needs
language support to be non-horrible. That's what a "deferred
expression" proposal would provide.
# Explicitly creating Deferred objects is horrible, this is the bit
that *really* needs language support
class Deferred:
def __init__(self, callable):
self.callable = callable
# This could easily be a builtin function (or an operator if people
prefer syntax) once we have deferred objects.
def undefer(expr):
if isinstance(expr, Deferred):
return expr.callable()
return expr
x = 12
# def f(a=defer x):
def f(a=Deferred(lambda: x)):
a = undefer(a)
return a
If anyone wants to take this and make a *proper* deferred object
proposal out of it, then please do so. If not, then at a minimum I
think this offers something vaguely concrete to discuss regarding the
"why deferred objects are a more general solution to the late bound
argument" question.
The reason I consider this to be an independent proposal, and NOT a
mechanism for late-bound defaults, is this problem:
def f(lst, n=>len(lst)):
lst.append(1)
print(n)
f([10, 20, 30])
A late-bound default should print 3. A deferred expression should
print 4. They're not a more general solution to the same question;
they're a solution to a different question that has some overlap in
what it can achieve. A None-coalescing operator would also have some
overlap with each of the above, but it is, again, not the same thing.
ChrisA
At one level, it's trivial. A deferred expression is `lambda:
expression`. Evaluating it is `deferred_expr()`.
[...]
We can't have a PEP or an implementation until we know what we're
proposing/implementing.
Indeed. I have been working on, well, dabbling with, for three or four
years now, and I'm still not entirely sure what I'm proposing. But, yes,
I agree, it most likely will involve an explicit "undefer" or evaluate
step.
It would probably be easier if we were designing a language from
scratch, like Haskell, rather than trying to retrofit the concept into
an existing language.
--
Steve
The reason I consider this to be an independent proposal, and NOT a
mechanism for late-bound defaults, is this problem:
def f(lst, n=>len(lst)):
lst.append(1)
print(n)
f([10, 20, 30])
A late-bound default should print 3. A deferred expression should
print 4.
Not according to R's model for late-bound default values, which are
performed at need.
"Late evaluation of defaults" can cover a lot of different semantics. We
can choose which semantics we want.
--
Steve
The reason I consider this to be an independent proposal, and NOT a
mechanism for late-bound defaults, is this problem:
def f(lst, n=>len(lst)):
lst.append(1)
print(n)
f([10, 20, 30])
A late-bound default should print 3. A deferred expression should
print 4. They're not a more general solution to the same question;
they're a solution to a different question that has some overlap in
what it can achieve. A None-coalescing operator would also have some
overlap with each of the above, but it is, again, not the same thing.
As I said, no-one is being clear about what they mean by "deferred
expressions". My strawman had an explicit syntax for "undeferring" for
precisely this reason, it lets the programmer decide whether to
undefer before or after the append.
Most of the objections to deferred expressions that I've seen seem to
involve this confusion - objectors assume that evaluation happens
"magically" and then object to the fact that the place they want the
evaluation to happen doesn't match with the place they assume the
magic would occur. I see this as more of an argument that implicit
evaluation is a non-starter, and therefore deferred expressions should
be explicitly evaluated.
Paul
Closures cannot be executed without a context. Consider:
def f(x=lambda: (a:=[])):
if isinstance(x, FunctionType): x = x()
print(a)
Here's the problem: The name 'a' should be in the context of f,
Not in the code as you show it. The a in the signature is local to the
lambda, and the a in the print() call is a global. The code exactly as
shown works fine, in exactly that way.
So I'm going to assume that you meant:
def f(x=>(a:=[]))
which compiles to a separate `lambda:(a:=[])`, as per my suggestion.
but that context *does not exist* until f starts executing.
If this really cannot be solved, then we could prohibit walrus operators
in late bound defaults. If something is too hard to get right, we can
prohibit it until such time (if ever) that we work out how to do it. For
many years, we had a restriction that you could have try...except and
you could have try...finally but you couldn't have try...except...
finally in a single block.
(And then we solved that technical limitation.)
Or we make them global, like they are for early bound defaults. Its not
*mandatory* that walrus bindings in the default expression go into the
function locals. They could go into the function's surrounding scope
(usually globals). That's a perfectly acceptable solution too.
Or... be bold. Think outside the box. "Do not go where the path may
lead, go instead where there is no path and leave a trail." To boldly go
where no science fiction show has gone before, and other cliches.
If we can't execute the expression without the context existing, we make
it exist. Details to follow in another post.
This paragraph confused me for the longest time, because as an
early-bound default, it does work. If you put `return x` after the
print, you get the expected result:
this is a global
[]
If you try to do this sort of thing with a magical function that is
synthesized as part of late-bound default handling, you'll still need
to figure out when f's context gets formed. Normally, it won't get
formed until the stack frame for f gets constructed - which is when
arguments get assigned to parameters, etc, etc. Trying to call that
inner function without first giving it a context won't work.
If you want to write up a competing reference implementation, go
ahead. I don't think it will be as easy as you claim. And I'm not
going to mention the possibility in the PEP *without* a reference
implementation, nor do I intend to write one myself (since I dispute
its usefulness).
It is the job of the PEP to describe rejected alternatives. If you
choose to disregard my clearly superior suggestion *wink* but don't
explain why you are rejecting it, then the PEP is not doing its job.
--
Steve
If we can't execute the expression without the context existing, we make
it exist. Details to follow in another post.
Here is some handwavy pseudo-code for setting up the context prior to
calling a function "func":
code = func.__code__
locals = [NIL pointer for _ in range(code.co_nlocals)]
cells = [cell() for _ in range(len(code.co_cellvars))]
assign args, kwargs, early defaults to locals
# make a frame
frame = Frame()
frame.f_back = current frame
frame.f_lasti = 0
frame.f_locals = locals + cells + list(func.__closure__)
frame.f_code = code
and then the interpreter does its funky thang with the frame and the
function is executed.
But here's the thing... there is nothing that says that the f_code has
to have come from func.__code__. It just needs to be code that expects
the same environment (locals, cell, etc) as the frame is set up for.
So here's how we could run the late bound default expression alone:
1. Ensure that the default expression code object has the same
environment (locals, cells etc) as the function that owns it;
this could be a copy, or it could be an reference back to the
owner.
2. Set up the frame, as above, for the function that owns the
expression.
3. But instead of setting f_code to the owner's code object, we set
it to the default expression's code object. They share the same
environment (cells, etc) so that's safe.
4. And then just run the function.
And to run the owner function:
1. Set up the frame.
2. Run each of the default expressions as above.
3. Then run the owner function.
Obviously this is very handwavy. But I am confident that it demonstrates
that the idea of factoring default expressions out into their own code
objects is not "impossible".
--
Steve
You (Chris) argue in favour of your PEP, where the bytecode of the
default expression is inlined into the function's body, because you
insist that it belongs in the body. You justify that claim by making
spurious arguments that it is "impossible" to do otherwise (your term,
not mine). But it isn't impossible, and I shall justify that claim in
another post. A bold claim for somebody who knows nothing about the C
implementation :-)
I am arguing that it belongs in the *header*.
The *source code* of the expression. But you are inlining its *bytecode*
into the function body.
With definition-time expressions, their code is inlined into the body
of the surrounding context. With call-time expressions, their code is
inlined into the body of the function.
If we can't execute the expression without the context existing, we make
it exist. Details to follow in another post.
Here is some handwavy pseudo-code for setting up the context prior to
calling a function "func":
code = func.__code__
locals = [NIL pointer for _ in range(code.co_nlocals)]
cells = [cell() for _ in range(len(code.co_cellvars))]
assign args, kwargs, early defaults to locals
# make a frame
frame = Frame()
frame.f_back = current frame
frame.f_lasti = 0
frame.f_locals = locals + cells + list(func.__closure__)
frame.f_code = code
and then the interpreter does its funky thang with the frame and the
function is executed.
But here's the thing... there is nothing that says that the f_code has
to have come from func.__code__. It just needs to be code that expects
the same environment (locals, cell, etc) as the frame is set up for.
Okay. So you want to break out a separate code object from the main
body, but it has to expect the exact same environment as the main
body.
Which means you cannot replace it with anything else, you cannot call
it externally without constructing the exact same environment, and
frankly, I cannot for the life of me see what you gain above just
having it part of the body. You keep arguing in favour of having it be
somehow separate, but to what end?
Please. Go make a reference implementation. At least then we'll know
what's possible and what's not.
ChrisA
Closures cannot be executed without a context. Consider:
def f(x=lambda: (a:=[])):
if isinstance(x, FunctionType): x = x()
print(a)
Here's the problem: The name 'a' should be in the context of f, but
that context*does not exist* until f starts executing.
Frankly, I would consider this another disadvantage of late-bound
arguments as defined under your proposal. I do not want argument
defaults to be able to have the side effect of creating additional local
variables in the function. (There is also the question of whether they
could assign in this manner to names already used by other arguments, so
that one argument's default could potentially override the default of
another.)
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
Closures cannot be executed without a context. Consider:
def f(x=lambda: (a:=[])):
if isinstance(x, FunctionType): x = x()
print(a)
Here's the problem: The name 'a' should be in the context of f, but
that context*does not exist* until f starts executing.
Frankly, I would consider this another disadvantage of late-bound
arguments as defined under your proposal. I do not want argument
defaults to be able to have the side effect of creating additional local
variables in the function. (There is also the question of whether they
could assign in this manner to names already used by other arguments, so
that one argument's default could potentially override the default of
another.)
Unless it's otherwise denied, all valid forms of expression should be
valid in an argument default. They are already valid in early-bound
defaults:
Since early-bound defaults are evaluated at definition time, the
assignment happens in the outer function. If anything, it's more
logical for it to be a part of the inner function, but for early-bound
defaults, that's not possible.
Though I wouldn't recommend that people actually *do* this sort of
thing. Legal doesn't mean recommended. But if someone does, it needs
to behave logically and correctly.
ChrisA
You're saying that the object part isn't that hard, but other parts of
it are.
For values of "hard" == "non-trivial but mostly bikeshedding". I
don't think it will be that much harder to get through than Chris's.
And in theory it could be easier: it could be implemented with a new
builtin such as "quote_expression" that takes a string, and thus
needing no new syntax. I actually don't think that will pass, too
clumsy to be of practical use. Although it might fly on the same
logic as function annotations did: we add the instance flag that says
that "this object is a thunk to be evaluated in the current context"
and the code in the interpreter that's needed to use it, and postpone
syntax until somebody comes up with a really good proposal.
In fact, we could combine this strategy with Steven d'Aprano's
proposal for a thunk object, in which case Chris doesn't need syntax
himself. I'm sure he won't like that, but it's an idea.
I can't see the point of rejecting something that provides a
tangible benefit, now, because some fictitious vapourware *might*,
one day, provide another way of doing it.
That's a strawman. The argument is not "Your proposal is good, but
not perfect, so we reject it." The basic argument is "Your proposal
isn't good enough to deserve syntax for reasons given elsewhere, but
if you do this other stuff we would likely support it." Sure, there's
some component of "it might interfere with the general facility, or be
a small cognitive burden, a warty technical debt, if the general
facility is implemented", but that's only a tiny part of why people
opposing the proposal oppose it.
You're saying that the object part isn't that hard, but other parts of
it are.
For values of "hard" == "non-trivial but mostly bikeshedding". I
don't think it will be that much harder to get through than Chris's.
And in theory it could be easier: it could be implemented with a new
builtin such as "quote_expression" that takes a string, and thus
needing no new syntax. I actually don't think that will pass, too
clumsy to be of practical use. Although it might fly on the same
logic as function annotations did: we add the instance flag that says
that "this object is a thunk to be evaluated in the current context"
and the code in the interpreter that's needed to use it, and postpone
syntax until somebody comes up with a really good proposal.
In fact, we could combine this strategy with Steven d'Aprano's
proposal for a thunk object, in which case Chris doesn't need syntax
himself. I'm sure he won't like that, but it's an idea.
All this seems like a matter of opinion, on which we shall have to
disagree. But ...
And there seems to be no likelihood of anyone tackling it soon.
There's no way to judge that. David Mertz might post a PEP tomorrow
for all you know.
Surely you're pulling my leg. If he had something in the pipeline that
would greatly strengthen his argument, we would have heard about it by now.
I can't see the point of rejecting something that provides a
tangible benefit, now, because some fictitious vapourware *might*,
one day, provide another way of doing it.
That's a strawman. The argument is not "Your proposal is good, but
not perfect, so we reject it."
That IMO is exactly the argument. It's like saying "I won't buy a car
today because in 10/20/50 years time I can probably buy a driverless one".
The basic argument is "Your proposal
isn't good enough to deserve syntax for reasons given elsewhere,
Er, what reasons?
but
if_*you*_ do this other stuff we would likely support it." [My emphasis - RC.]
You want *Chris* to implement deferred-evaluation objects? Hasn't he
done enough? You want him to produce a second PEP and a second
reference implementation in competition with his first one? (He has on
several occasions mentioned problems that he sees with the idea.)
Surely if other people think it's a good idea, it's up to *them* to
On 07/12/2021 18:22, Stephen J. Turnbull wrote:
prove it. If an architect comes up with a plan to redesign a city
centre, would you say to him "Now produce a completely different plan,
just to show that your first one wasn't great"?
Best wishes
Rob Cliffe
Sure, there's
some component of "it might interfere with the general facility, or be
a small cognitive burden, a warty technical debt, if the general
facility is implemented", but that's only a tiny part of why people
opposing the proposal oppose it.
That's a strawman. The argument is not "Your proposal is good, but
not perfect, so we reject it."
That IMO is exactly the argument. It's like saying "I won't buy a car
today because in 10/20/50 years time I can probably buy a driverless one".
I think a better analogy for those who reject late-bound defaults but
would accept a general deferred evaluation mechanism is:
"I won't buy a driverless car today because I have a perfectly good car
now, and **driving is not a sufficient burden** that I care for this
feature alone. But fully autonomous robots that could drive my car,
clean my house, do my chores, now *that's* something I would buy!"
[the other Stephen]
if _*you*_ do this other stuff we would likely support it."
[My emphasis - RC.]
You want *Chris* to implement deferred-evaluation objects?
Clearly not. It's obvious in context that Stephen is talking about
*generic* "you". He's not addressing his comment to Chris.
Anyone could read the comment and interpret that "you" as themselves,
and respond "What a great idea! I'm going to implement deferred
evaluation!". You surely don't imagine that Stephen thinks, or implies,
that those who want a generic deferred evaluation feature would reject
it if it wasn't done specifically by Chris himself.
I know that the Python-Ideas community is changable like the wind and
rather mercurial, but we've never yet demanded a feature we want be
implemented by a *particular person* or else we will reject it.
--
Steve
For values of "hard" == "non-trivial but mostly bikeshedding". I
don't think it will be that much harder to get through than Chris's.
And in theory it could be easier: it could be implemented with a new
builtin such as "quote_expression" that takes a string, and thus
needing no new syntax. I actually don't think that will pass, too
clumsy to be of practical use. Although it might fly on the same
logic as function annotations did: we add the instance flag that says
that "this object is a thunk to be evaluated in the current context"
and the code in the interpreter that's needed to use it, and postpone
syntax until somebody comes up with a really good proposal.
In fact, we could combine this strategy with Steven d'Aprano's
proposal for a thunk object, in which case Chris doesn't need syntax
himself. I'm sure he won't like that, but it's an idea.
All this seems like a matter of opinion, on which we shall have to
disagree. But ...
"All"? The suggestions about implementation are eminently practical,
all are in use in Python already, proposed in accepted PEPs, or
proposed by others in this thread for implementing Chris's syntax.
The only disagreeable opinions there are about what might be
acceptable to other people, but I thought I was pretty conservative on
those counts.
Of course not. I have no reason to believe humor would sway you.
> If he had something in the pipeline that would greatly strengthen
> his argument, we would have heard about it by now.
The point of my argument above is that all the parts are there, "all"
we need is a good syntax and a coherent theory of scoping. Those are
"flash of inspiration" kinds of thing. It's a good question whether
Chris will beat the general deferred advocates to an acceptable
syntax, since there are quite a few people who dislike his preferred
syntax. I'd give the edge to Chris, since a fair number of people
like his idea and I'm sure he'll accept any syntax the SC strongly
prefers, and several have been proposed. But it's not a sure thing,
since quite a few smart people are completely unexcited by Chris's
proposal. The SC might very well go the same way. They might stomp
on general deferreds, too -- won't know until we get there.
So much for your opinion. I assure you, that is NOT my argument. So
if it is anybody's argument, there's no "the" there.
> It's like saying "I won't buy a car today because in 10/20/50 years
> time I can probably buy a driverless one".
I agree that's a good analogy to your mischaracterization of my
argument (which mostly has been made by others; important points have
been made by David Mertz, Brendan Barnwell, Steven d'Aprano, and
likely several others). But it is a mischaracterization, so your
analogy is pointless.
You want *Chris* to implement deferred-evaluation objects?
Sure, I'll take a pony if it's offered. But what matters here is that
Chris wants to, if he wants my support and David's. I imagine he
considers that a nice-to-have but he'll do without rather than
implement the general deferred object.
> Hasn't he done enough?
No, not enough to get better than -1s from me, David, and some others.
Whether he's channeling the SC, or we are, remains to be seen.
> You want him to produce a second PEP and a second reference
> implementation in competition with his first one?
As above. It's been done before, for similar reasons (ie, to make the
BDFL, PEP Delegate, or SC happy).
Surely if other people think it's a good idea, it's up to *them* to
prove it.
I think you misunderstand how open source projects work. There's
always a gatekeeping mechanism in projects that involve more than two
or three developeres; in Python it's the SC. (Note: not me or David
or Brendan, nor all of us together.) People will do what they
consider unnecessary work to get through the gate.
Who does the work and how much is done depends on how much each party
wants the feature. Whether Chris should do it depends on how much
Chris wants *some* kind of deferred evaluation for actual arguments,
how much the SC prefers a general deferred to Chris's default-
arguments-only deferred, and whether they think having both is a bad
idea for some reason.
But the "good idea" of general deferreds is only marginally relevant
to our -1s. It's those -1s that constitute the main issue for Chris,
since they're a noisy signal that the SC might think as we do.
> If an architect comes up with a plan to redesign a city centre,
> would you say to him "Now produce a completely different plan, just
> to show that your first one wasn't great"?
No, I wouldn't say that. But I will say you need to come up with
analogies that capture your opponents' ideas, instead of strawman
arguments.
Cheers,
Steve
But the "good idea" of general deferreds is only marginally relevant
to our -1s. It's those -1s that constitute the main issue for Chris,
since they're a noisy signal that the SC might think as we do.
Please explain to me *exactly* what your arguments against the current
proposal are. At the moment, I am extremely confused as to what people
actually object to, and there's endless mischaracterization and
accusation happening.
Can we actually figure out what people are really saying, and what the
problems with this proposal are?
NOT that there might potentially be some other proposal, but what the
problems with this one are. Argue THIS proposal, not hypothetical
other proposals.
ChrisA
But the "good idea" of general deferreds is only marginally relevant
to our -1s. It's those -1s that constitute the main issue for Chris,
since they're a noisy signal that the SC might think as we do.
Please explain to me *exactly* what your arguments against the current
proposal are. At the moment, I am extremely confused as to what people
actually object to, and there's endless mischaracterization and
accusation happening.
Can we actually figure out what people are really saying, and what the
problems with this proposal are?
NOT that there might potentially be some other proposal, but what the
problems with this one are. Argue THIS proposal, not hypothetical
other proposals.
Note that I'm not vehemently -1 on this PEP, but I am against it. So
I'm not necessarily one of the people whose response you need and are
asking for here, but my views are part of the opposition to the PEP.
So here's my problems with this proposal:
1. The problem that the PEP solves simply isn't common enough, or
difficult enough to work around, to justify new syntax, plus a second
way of defining default values.
2. There's no syntax that has gained consensus, and the objections
seem to indicate that there are some relatively fundamental
differences of opinion involved.
3. There's no precedent for languages having *both* types of binding
behaviour. Sure, late binding is more common, but everyone seems to
pick one form and stick with it.
4. It's a solution to one problem in the general "deferred expression"
space. If it gets implemented, and deferred expressions are added
later, we'll end up with two ways of achieving one result, with one
way being strictly better than the other. (Note, for clarity, that's
*not* saying that we should wait for something that might never
happen, it's saying that IMO the use case here isn't important enough
to warrant rushing a partial solution).
To be 100% explicit, none of the above are showstopper objections
(some, like the choice of syntax, are pretty minor). I'm not arguing
that they are. Rather, my problem with the PEP is that we have a
number of individually small issues like this, which aren't balanced
out by a sufficiently compelling benefit. The PEP isn't *bad*, it's
simply not good *enough* (IMO). And it's not obvious how to fix the
issue, as there's no clear way to increase the benefit side of the
equation. That sucks, as it's a lot of work to write a PEP, and "meh,
I'm not convinced" is the worst possible response. But that's how this
feels to me.
The reason deferred objects keep coming up is because they *do* have a
much more compelling benefit - they help in a much broader range of
cases. It's fine to say they are a different proposal, and that "but
we might get deferred expressions" is a flawed objection (which it is,
if that's all the objection consists of). But rejecting that argument
doesn't do anything to improve the weak benefits case for late-bound
defaults, or to fix the various minor problems that weigh it down.
All IMO, of course...
Paul
But the "good idea" of general deferreds is only marginally relevant
to our -1s. It's those -1s that constitute the main issue for Chris,
since they're a noisy signal that the SC might think as we do.
Please explain to me *exactly* what your arguments against the current
proposal are. At the moment, I am extremely confused as to what people
actually object to, and there's endless mischaracterization and
accusation happening.
Can we actually figure out what people are really saying, and what the
problems with this proposal are?
1. The status quo is fine. Using None or another sentinel and checking
for it in the body has worked for many years and is not that big a
problem. In theory improvement is always possible, but there is no
urgency to change anything until we have a proposal with fewer
downsides. In addition, as discussed in some posts on this list, not
even all cases of None/sentinel defaults will be obviated by this proposal.
2. Most of the proposed syntaxes make it difficult to visually
distinguish the late and early-bound defaults (because they all look
similar to a plain equals sign which will still mean a regular
early-bound default).
3. Regardless of the syntax, having the potential for def-time and
call-time behavior to be mixed and interleaved in arbitrary ways within
the same function signature is confusing.
4. Currently anything that is a function default is some kind of Python
object that can be inspected, interacted with, and used independently of
the function/argument whose default it is. This proposal breaks that
assumption. In other words I don't want anything that is "a default"
but is not a "default VALUE".
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
Closures cannot be executed without a context. Consider:
def f(x=lambda: (a:=[])):
if isinstance(x, FunctionType): x = x()
print(a)
Here's the problem: The name 'a' should be in the context of f, but
that context*does not exist* until f starts executing.
Frankly, I would consider this another disadvantage of late-bound
arguments as defined under your proposal. I do not want argument
defaults to be able to have the side effect of creating additional local
variables in the function. (There is also the question of whether they
could assign in this manner to names already used by other arguments, so
that one argument's default could potentially override the default of
another.)
Unless it's otherwise denied, all valid forms of expression should be
valid in an argument default. They are already valid in early-bound
defaults:
That's true. I guess what I really mean is "I still think the walrus
was a bad idea". :-)
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
But the "good idea" of general deferreds is only marginally relevant
to our -1s. It's those -1s that constitute the main issue for Chris,
since they're a noisy signal that the SC might think as we do.
Please explain to me *exactly* what your arguments against the current
proposal are. At the moment, I am extremely confused as to what people
actually object to, and there's endless mischaracterization and
accusation happening.
Can we actually figure out what people are really saying, and what the
problems with this proposal are?
There is one other that I forgot to include in my last message
5. Miscellaneous wrinkles. By this I mean the various sub-discussions
about things like what order the late and early defaults should be
evaluated in. This is a sort of second-order objection for me, because
the objections I gave in my previous message are enough for me to reject
the proposal. But even assuming I agreed with the broad outlines, these
subsidiary concerns leave enough room for confusion that I would not
endorse the proposal. In other words there are too many devils in the
details that I feel would lead to difficult-to-reason-about code and
traps for the unwary.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
The reason deferred objects keep coming up is because they *do* have a
much more compelling benefit - they help in a much broader range of
cases.
That may be true. I don't know.
Can anyone provide some realistic use cases? I've read the whole thread
and I can only recall at most one, viz. the default value is expensive
to compute and may not be needed. But that is a good time *not* to use
a late-bound default! (The sentinel idiom would be better.) Anything
can be used inappropriately, that doesn't make it bad per se.
I don't wish to disparage anyone's motives. I am sure all the posts
were made sincerely and honestly. But without examples (of how deferred
objects would be useful), if *feels to me* (no doubt wrongly) as if
people are using a fig leaf to fight against this PEP.
Best wishes
Rob Cliffe
Please explain to me *exactly* what your arguments against the current
proposal are. At the moment, I am extremely confused as to what people
actually object to,
and I can only recall at most one, viz. the default value is expensive to
compute and may not be needed.
https://docs.dask.org/en/stable/delayed.html
This library is widely used, extremely powerful, and expressive. Basically,
a built-in capability would have every win of Dask Delayed, but decrease
the impedance mismatch (which isn't terrible as-is) and avoid the need for
external tooling.
Can anyone provide some realistic use cases? I've read the whole thread and I can only recall at most one, viz. the default value is expensive to compute and may not be needed.
This library is widely used, extremely powerful, and expressive. Basically, a built-in capability would have every win of Dask Delayed, but decrease the impedance mismatch (which isn't terrible as-is) and avoid the need for external tooling.
Most of that is a *massive* YAGNI as regards function default
arguments. We do not need parallel execution just to figure out the
length of a list passed as a previous parameter. So you've just added
weight to my argument that a generic "delayed" feature is a completely
separate proposal, nothing whatsoever to do with PEP 671.
ChrisA
Thank you David.
But AIUI (i.e. practically not at all) Dask is about parallel computing,
which is not the same thing as deferred evaluation, though doubtless
they overlap. Again AIUI, parallel computing is mainly useful when you
have multiple cores or multiple computers.
Can anyone give examples (in Python pseudo-code perhaps) showing how
*deferred evaluation* would be useful for a concrete task? (Solving an
equation. Drawing a graph. Analysing a document. Manufacturing a
widget. Planning a journey. Firing a missile. Anything! You name it.)
Best wishes
Rob Cliffe
On 08/12/2021 22:40, David Mertz, Ph.D. wrote:
On Wed, Dec 8, 2021, 2:58 PM Rob Cliffe via Python-ideas
On 08/12/2021 19:27, Paul Moore wrote:
> The reason deferred objects keep coming up is because they *do*
have a much more compelling benefit - they help in a much broader
range of cases.
Can anyone provide some realistic use cases? I've read the whole
thread and I can only recall at most one, viz. the default value
is expensive to compute and may not be needed.
This library is widely used, extremely powerful, and expressive.
Basically, a built-in capability would have every win of Dask Delayed,
but decrease the impedance mismatch (which isn't terrible as-is) and
avoid the need for external tooling.
The reason deferred objects keep coming up is because they *do* have a
much more compelling benefit - they help in a much broader range of
cases.
That may be true. I don't know.
Can anyone provide some realistic use cases?
Of what? Deferred expressions? I ask because the rest of your post
seems to only be thinking in terms of argument defaults, when the
point I'm trying to make is that deferred expressions have uses
outside of that situation.
Honestly, I don't have particular examples off the top of my head.
It's not me that's arguing for deferred objects. I probably should
have worded that sentence as "The reason deferred objects keep coming
up is because the people interested in them are claiming that they
*do* have a much more compelling benefit - they help in a much broader
range of cases."
In the context of the current discussion about late-bound defaults, I
already said that deferred expressions could reasonably be declared
not relevant, and it wouldn't affect my core complaint, which is that
the benefit doesn't justify the costs. But certainly if someone does
propose introducing deferred expressions, I'd expect them to explain
the benefits, and I would expect that (a) one benefit would be that
they handle all of the cases that late-bound defaults cover, and (b)
there are further benefits in areas outside default values. That's
what I mean when I say that deferred expressions are a superset of the
functionality of late-bound defaults.
I've read the whole thread
and I can only recall at most one, viz. the default value is expensive
to compute and may not be needed. But that is a good time *not* to use
a late-bound default! (The sentinel idiom would be better.) Anything
can be used inappropriately, that doesn't make it bad per se.
I don't wish to disparage anyone's motives. I am sure all the posts
were made sincerely and honestly. But without examples (of how deferred
objects would be useful), if *feels to me* (no doubt wrongly) as if
people are using a fig leaf to fight against this PEP.
I agree. But you're only responding to the last paragraph of my post.
Everything else I said was explaining my reservations over the
late-bound defaults proposal, and I *explicitly* said that those
reservations stand independent of any deferred expression proposal.
Honestly, it may feel to you that people are using weak arguments to
fight the PEP, but to me it feels like supporters of the PEP are
ignoring all of the *other* objections and trying to make the argument
entirely about deferred expressions. I guess we're both mistaken in
our feelings ;-)
Paul
Most of that is a *massive* YAGNI as regards function default
arguments. We do not need parallel execution just to figure out the
length of a list passed as a previous parameter. So you've just added
weight to my argument that a generic "delayed" feature is a completely
separate proposal, nothing whatsoever to do with PEP 671.
If we concede that delayed expressions are a separate proposal, would
you be willing to address the other issues that people have raised? At
this point, it seems like the "deferred expressions" debate is
distracting everyone from all of the *other* points made by people
with reservations about the proposal, which basically come down to
"the benefit is limited, and the costs are too high to justify the
feature". So far, the responses I've seen to that point mostly seem to
come down to "I don't agree, I think the costs are small and the
benefits are sufficient". That's not addressing the objections, it's
just agreeing to differ¹.
At a minimum, the PEP should state the objections fairly, and note
that the PEP author disagrees. A PEP isn't a sales pitch, it's a
summary of the discussions - so it absolutely should mention that
there's been significant opposition to the proposal, which did not get
resolved, if that's the reality.
Paul
¹ "That's not an argument, it's just contradiction!"
But AIUI (i.e. practically not at all) Dask is about parallel computing,
which is not the same thing as deferred evaluation, though doubtless they
overlap. Again AIUI, parallel computing is mainly useful when you have
multiple cores or multiple computers.
Much of Dask is about parallelism. But Dask Delayed really isn't. I mean,
yes it's a good adjunct to actual parallelism, but much of the benefit is
independent.
In particular, in Dask delayed—much as in a thoroughly lazy language like
Haskell—you can express a graph of interrelated computations that you might
POTENTIALLY perform.
There are many times when expressing those dependencies is useful, even
before you know which, if any, of them will actually need to be performed.
The site I linked as many more fleshed out examples, but suppose I have
this dataflow relationship:
A -> B -> C -> D -> E
Each of those letters name some expensive computation (or maybe expensive
I/O, or both).
In a particular run of our program, we might determine that we need the
data created by B. But in that particular run, we never wind up using C, D
or E. Of course, a different run, based on different conditions, will
actually need E.
In this simplest possible DAG, I've deliberately avoided any possible
parallelism. Every step entirely depends on the one before it. But delayed
compution can still be useful. Of course, when the DAG has branches, often
operating on branches can often be usefully parallelized (but that's still
not required for laziness to remain useful.
Most of that is a *massive* YAGNI as regards function default
arguments. We do not need parallel execution just to figure out the
length of a list passed as a previous parameter. So you've just added
weight to my argument that a generic "delayed" feature is a completely
separate proposal, nothing whatsoever to do with PEP 671.
If we concede that delayed expressions are a separate proposal, would
you be willing to address the other issues that people have raised? At
this point, it seems like the "deferred expressions" debate is
distracting everyone from all of the *other* points made by people
with reservations about the proposal, which basically come down to
"the benefit is limited, and the costs are too high to justify the
feature". So far, the responses I've seen to that point mostly seem to
come down to "I don't agree, I think the costs are small and the
benefits are sufficient". That's not addressing the objections, it's
just agreeing to differ¹.
Part of the problem is that it is really REALLY hard to figure out
what the actual objections are. I asked, and the one clear answer I
got was one subjective opinion that the cognitive load exceeded the
benefit. Great! That's one person's concern. I've responded to that by
clarifying parts of the cognitive load problem, and that's about as
far as that can go.
But if there's nothing more specific than that, what do you want me to
respond to? How can I address the objections if the objections are as
vague as you're describing?
At a minimum, the PEP should state the objections fairly, and note
that the PEP author disagrees. A PEP isn't a sales pitch, it's a
summary of the discussions - so it absolutely should mention that
there's been significant opposition to the proposal, which did not get
resolved, if that's the reality.
Yes, and "significant opposition" doesn't just mean "I don't like
this". There's nothing to respond to in that.
(Plus, there's significant belligerent support for the proposal, which
is even harder to handle.)
ChrisA
On Wed, Dec 8, 2021, 5:55 PM Rob Cliffe via Python-ideas
But AIUI (i.e. practically not at all) Dask is about parallel
computing, which is not the same thing as deferred evaluation,
though doubtless they overlap. Again AIUI, parallel computing is
mainly useful when you have multiple cores or multiple computers.
Much of Dask is about parallelism. But Dask Delayed really isn't. I
mean, yes it's a good adjunct to actual parallelism, but much of the
benefit is independent.
In particular, in Dask delayed—much as in a thoroughly lazy language
like Haskell—you can express a graph of interrelated computations that
you might POTENTIALLY perform.
There are many times when expressing those dependencies is useful,
even before you know which, if any, of them will actually need to be
performed. The site I linked as many more fleshed out examples, but
suppose I have this dataflow relationship:
A -> B -> C -> D -> E
Each of those letters name some expensive computation (or maybe
expensive I/O, or both).
In a particular run of our program, we might determine that we need
the data created by B. But in that particular run, we never wind up
using C, D or E. Of course, a different run, based on different
conditions, will actually need E.
In this simplest possible DAG, I've deliberately avoided any possible
parallelism. Every step entirely depends on the one before it. But
delayed compution can still be useful. Of course, when the DAG has
branches, often operating on branches can often be usefully
parallelized (but that's still not required for laziness to remain useful.
This is all abstract. You give no clue to what your application is or
what it is meant to do. Please, may I refer you to my previous post:
"/Can anyone give examples (in Python pseudo-code perhaps) showing
how *deferred evaluation* would be useful for a concrete task? (Solving
an equation. Drawing a graph. Analysing a document. Manufacturing a
widget. Planning a journey. Firing a missile. *Anything!* *You* name
it./)"
David? Anybody??
Best wishes
Rob Cliffe
Part of the problem is that it is really REALLY hard to figure out
what the actual objections are. I asked, and the one clear answer I
got was one subjective opinion that the cognitive load exceeded the
benefit. Great! That's one person's concern. I've responded to that by
clarifying parts of the cognitive load problem, and that's about as
far as that can go.
Um, what parts of my response were unclear? I gave 4 specific points,
Brendan gave 4 more (there wasn't much overlap with mine, either).
Multiple people have mentioned that the proposed syntax is confusing.
You don't have to respond to everyone individually, and indeed you
shouldn't - it's the cumulative effect that matters. Telling 10 people
that their concern "is one person's concern" doesn't address the fact
that 10 people felt similarly. And honestly, there's only about 10
active participants in this thread, so even 5 people with reservations
about the syntax is still "half the people who expressed an opinion".
Yes, many of the concerns are somewhat subjective, and many of them
are subject to a certain amount of interpretation. That's the nature
of this sort of issue. If I said to you that the biggest issue here
was that "in the group of people on python-ideas who were motivated
enough to get involved in discussions, about half of the participants
were arguing against the proposal"¹ would that be a concrete enough
objection for you? Count it as my 5th objection, if you like. I know
we're not putting the PEP to a vote here, but proposed changes *are*
supposed to achieve a certain level of consensus (in the normal course
of events - of course the SC can approve anything, even if it's hugely
unpopular, that's their privilege).
Paul
¹ That's just my gut feeling. Feel free to check the actual numbers
and counter my argument with more precise facts.
But the "good idea" of general deferreds is only marginally relevant
to our -1s. It's those -1s that constitute the main issue for Chris,
since they're a noisy signal that the SC might think as we do.
Please explain to me *exactly* what your arguments against the current
proposal are. At the moment, I am extremely confused as to what people
actually object to, and there's endless mischaracterization and
accusation happening.
Can we actually figure out what people are really saying, and what the
problems with this proposal are?
1. The status quo is fine. Using None or another sentinel and
checking for it in the body has worked for many years and is not that
big a problem. In theory improvement is always possible, but there is
no urgency to change anything until we have a proposal with fewer
downsides. In addition, as discussed in some posts on this list, not
even all cases of None/sentinel defaults will be obviated by this
proposal.
2. Most of the proposed syntaxes make it difficult to visually
distinguish the late and early-bound defaults (because they all look
similar to a plain equals sign which will still mean a regular
early-bound default).
3. Regardless of the syntax, having the potential for def-time and
call-time behavior to be mixed and interleaved in arbitrary ways
within the same function signature is confusing.
4. Currently anything that is a function default is some kind of
Python object that can be inspected, interacted with, and used
independently of the function/argument whose default it is. This
proposal breaks that assumption. In other words I don't want anything
that is "a default" but is not a "default VALUE".
Brandon sums up my objections here, except in #4 I'd make it "... some
kind of Python object that can be _created_, interacted with, and used
independently ...".
Someone asked what the use case for a "deferred object" would be. I hate
that name, but I'll stick with it here for the time being. Had they
existed, I would have found some way to use them for
dataclasses.fields's default_factory parameter. Instead, I had to use a
zero-argument callable with an unfriendly name.
Image a world where you could create a "deferred object" with
back-ticks, and that would remember the context where it was created.
Then you could have:
@dataclasses.dataclass
class A:
a: int = 0
b: list=`[]`
Instead of:
@dataclasses.dataclass
class A:
a: int = 0
b: list=dataclasses.field(default_factory=list)
Eric
Part of the problem is that it is really REALLY hard to figure out
what the actual objections are. I asked, and the one clear answer I
got was one subjective opinion that the cognitive load exceeded the
benefit. Great! That's one person's concern. I've responded to that by
clarifying parts of the cognitive load problem, and that's about as
far as that can go.
Um, what parts of my response were unclear? I gave 4 specific points,
Brendan gave 4 more (there wasn't much overlap with mine, either).
Multiple people have mentioned that the proposed syntax is confusing.
You don't have to respond to everyone individually, and indeed you
shouldn't - it's the cumulative effect that matters. Telling 10 people
that their concern "is one person's concern" doesn't address the fact
that 10 people felt similarly. And honestly, there's only about 10
active participants in this thread, so even 5 people with reservations
about the syntax is still "half the people who expressed an opinion".
I have attempted to explain the syntax. What is confusing?
def f(x=spam): ...
def f(x=>spam): ...
I'm not sure what concerns need to be addressed, because I don't
understand the concerns. Maybe I'm just getting caught up on all the
side threads about "deferreds are better" and "it should be a magical
function instead" and I've lost some of the basics? Feel free to
repost a simple concern and I will attempt to respond.
Yes, many of the concerns are somewhat subjective, and many of them
are subject to a certain amount of interpretation. That's the nature
of this sort of issue. If I said to you that the biggest issue here
was that "in the group of people on python-ideas who were motivated
enough to get involved in discussions, about half of the participants
were arguing against the proposal"¹ would that be a concrete enough
objection for you? Count it as my 5th objection, if you like. I know
we're not putting the PEP to a vote here, but proposed changes *are*
supposed to achieve a certain level of consensus (in the normal course
of events - of course the SC can approve anything, even if it's hugely
unpopular, that's their privilege).
EVERYONE is arguing against the proposal. Quite frankly, I'm just
about ready to throw the whole thing in, because this entire thread
has devolved to complaints that are nearly impossible to respond to -
or are just repetition of things that ARE responded to in the PEP.
Maybe we don't need any new features in Python. Maybe Python 3.10 is
already the perfect language, and we should just preserve it in amber.
ChrisA
There are tens of concrete examples at the link I gave, and hundreds more
you can find easily by searching on Dask Delayed. This feels more like
trying to believe a contrary than seeking understanding.
Here's a concrete example that I wrote last summer. I wanted to write a
similar program in a bunch of programming languages to learn those
languages. From long ago, I had a Python implementation (which I improved
quite a lot through the exercise, as well).
https://github.com/DavidMertz/LanguagePractice
What the programs do is identify any duplicate files in a filesystem tree
(i.e. perhaps among millions of files, often with different names but same
content).
The basic idea is that a hash like SHA1 serves as a fingerprint of
contents. However, the main speedup potential is in NOT computing the hash
when files are either hardlinks or soft links to the same underlying inode.
I/O nowadays is more of a hit than CPU cycles, but the concept applies
either way.
Essentially the same technique is used in all the languages. But in the
Haskell case, it is NECESSARY to express this as deferred computation. I
don't want Python to be like Haskell, which was in most ways the most
difficult to work with.
However, it would be interesting and expressive to write a Python version
based around Dask Delayed... Or around a generalized "deferred" construct
in Python 3.13, maybe. I'm pretty sure it could be shorter and more
readable thereby.
On Wed, Dec 8, 2021, 6:28 PM Rob Cliffe via Python-ideas <
python-ideas@python.org> wrote:
But AIUI (i.e. practically not at all) Dask is about parallel computing,
which is not the same thing as deferred evaluation, though doubtless they
overlap. Again AIUI, parallel computing is mainly useful when you have
multiple cores or multiple computers.
Much of Dask is about parallelism. But Dask Delayed really isn't. I mean,
yes it's a good adjunct to actual parallelism, but much of the benefit is
independent.
In particular, in Dask delayed—much as in a thoroughly lazy language like
Haskell—you can express a graph of interrelated computations that you might
POTENTIALLY perform.
There are many times when expressing those dependencies is useful, even
before you know which, if any, of them will actually need to be performed.
The site I linked as many more fleshed out examples, but suppose I have
this dataflow relationship:
A -> B -> C -> D -> E
Each of those letters name some expensive computation (or maybe expensive
I/O, or both).
In a particular run of our program, we might determine that we need the
data created by B. But in that particular run, we never wind up using C, D
or E. Of course, a different run, based on different conditions, will
actually need E.
In this simplest possible DAG, I've deliberately avoided any possible
parallelism. Every step entirely depends on the one before it. But delayed
compution can still be useful. Of course, when the DAG has branches, often
operating on branches can often be usefully parallelized (but that's still
not required for laziness to remain useful.
This is all abstract. You give no clue to what your application is or
what it is meant to do. Please, may I refer you to my previous post:
"*Can anyone give examples (in Python pseudo-code perhaps) showing
how *deferred evaluation* would be useful for a concrete task? (Solving an
equation. Drawing a graph. Analysing a document. Manufacturing a
widget. Planning a journey. Firing a missile. Anything! You name it.*
)"
Part of the problem is that it is really REALLY hard to figure out
what the actual objections are. I asked, and the one clear answer I
got was one subjective opinion that the cognitive load exceeded the
benefit. Great! That's one person's concern. I've responded to that by
clarifying parts of the cognitive load problem, and that's about as
far as that can go.
Um, what parts of my response were unclear? I gave 4 specific points,
Brendan gave 4 more (there wasn't much overlap with mine, either).
Multiple people have mentioned that the proposed syntax is confusing.
You don't have to respond to everyone individually, and indeed you
shouldn't - it's the cumulative effect that matters. Telling 10 people
that their concern "is one person's concern" doesn't address the fact
that 10 people felt similarly. And honestly, there's only about 10
active participants in this thread, so even 5 people with reservations
about the syntax is still "half the people who expressed an opinion".
I have attempted to explain the syntax. What is confusing?
def f(x=spam): ...
def f(x=>spam): ...
I'm not sure what concerns need to be addressed, because I don't
understand the concerns. Maybe I'm just getting caught up on all the
side threads about "deferreds are better" and "it should be a magical
function instead" and I've lost some of the basics? Feel free to
repost a simple concern and I will attempt to respond.
[snip]
I haven't been following the thread for some time, but my expectation
would be that:
def f(x=>spam):
...
would behave like:
_Missing_ = object()
def f(x=_Missing_):
if x is _Missing_:
x = spam
...
There are tens of concrete examples at the link I gave, and hundreds more you can find easily by searching on Dask Delayed. This feels more like trying to believe a contrary than seeking understanding.
Here's a concrete example that I wrote last summer. I wanted to write a similar program in a bunch of programming languages to learn those languages. From long ago, I had a Python implementation (which I improved quite a lot through the exercise, as well).
What the programs do is identify any duplicate files in a filesystem tree (i.e. perhaps among millions of files, often with different names but same content).
The basic idea is that a hash like SHA1 serves as a fingerprint of contents. However, the main speedup potential is in NOT computing the hash when files are either hardlinks or soft links to the same underlying inode. I/O nowadays is more of a hit than CPU cycles, but the concept applies either way.
Essentially the same technique is used in all the languages. But in the Haskell case, it is NECESSARY to express this as deferred computation. I don't want Python to be like Haskell, which was in most ways the most difficult to work with.
However, it would be interesting and expressive to write a Python version based around Dask Delayed... Or around a generalized "deferred" construct in Python 3.13, maybe. I'm pretty sure it could be shorter and more readable thereby.
The basic and obvious way to write that is a simple dictionary lookup.
It's not particularly hard to recognize inode numbers without a
deferred/delayed construct. And this is still arguing for their
benefit in the wider language, with no indication of how it's better
for default arguments. This is a MASSIVE amount of overhead for simple
cases of "x=>[]" or similar.
ChrisA
Part of the problem is that it is really REALLY hard to figure out
what the actual objections are. I asked, and the one clear answer I
got was one subjective opinion that the cognitive load exceeded the
benefit. Great! That's one person's concern. I've responded to that by
clarifying parts of the cognitive load problem, and that's about as
far as that can go.
Um, what parts of my response were unclear? I gave 4 specific points,
Brendan gave 4 more (there wasn't much overlap with mine, either).
Multiple people have mentioned that the proposed syntax is confusing.
You don't have to respond to everyone individually, and indeed you
shouldn't - it's the cumulative effect that matters. Telling 10 people
that their concern "is one person's concern" doesn't address the fact
that 10 people felt similarly. And honestly, there's only about 10
active participants in this thread, so even 5 people with reservations
about the syntax is still "half the people who expressed an opinion".
I have attempted to explain the syntax. What is confusing?
def f(x=spam): ...
def f(x=>spam): ...
I'm not sure what concerns need to be addressed, because I don't
understand the concerns. Maybe I'm just getting caught up on all the
side threads about "deferreds are better" and "it should be a magical
function instead" and I've lost some of the basics? Feel free to
repost a simple concern and I will attempt to respond.
[snip]
I haven't been following the thread for some time, but my expectation
would be that:
def f(x=>spam):
...
would behave like:
_Missing_ = object()
def f(x=_Missing_):
if x is _Missing_:
x = spam
...
Yes, broadly so. The differences would be that the signature actually
says what the default will be (instead of "
That's correct Chris!
No matter how much you try to change the topic, I was asked how a general
deferred concept would be useful and gave an example.
As the example itself shows, obviously there ARE other ways one could do it
(as I did in the code linked). And there are ways it could be improved by a
general deferred object as well.
It is true that an awkward syntax to move a function body into a signature
isn't the same thing. That's one of MANY reasons, all clearly stated
numerous times, why I and most others who have discussed this oppose this
PEP.
An obvious reason to oppose it is that it is a much worse way of writing an
existing sentinel check.
On Wed, Dec 8, 2021, 9:09 PM Chris Angelico wrote:
Here's a concrete example that I wrote last summer. I wanted to write a
similar program in a bunch of programming languages to learn those
languages. From long ago, I had a Python implementation (which I improved
quite a lot through the exercise, as well).
The basic idea is that a hash like SHA1 serves as a fingerprint of
contents. However, the main speedup potential is in NOT computing the hash
when files are either hardlinks or soft links to the same underlying inode.
I/O nowadays is more of a hit than CPU cycles, but the concept applies
either way.
Essentially the same technique is used in all the languages. But in the
Haskell case, it is NECESSARY to express this as deferred computation. I
don't want Python to be like Haskell, which was in most ways the most
difficult to work with.
However, it would be interesting and expressive to write a Python
version based around Dask Delayed... Or around a generalized "deferred"
construct in Python 3.13, maybe. I'm pretty sure it could be shorter and
more readable thereby.
The basic and obvious way to write that is a simple dictionary lookup.
It's not particularly hard to recognize inode numbers without a
deferred/delayed construct. And this is still arguing for their
benefit in the wider language, with no indication of how it's better
for default arguments. This is a MASSIVE amount of overhead for simple
cases of "x=>[]" or similar.
No matter how much you try to change the topic, I was asked how a general deferred concept would be useful and gave an example.
I've never denied that it would be useful in general, just that it has
nothing to do with PEP 671.
So far, I've not seen any examples that have anything to do with
function default arguments. Yet you claim that PEP 671 should be
rejected on this basis.
Easy! Look at EVERY Python function that uses a sentinel. They all get a
little bit worse under your proposal.
On Wed, Dec 8, 2021, 9:54 PM Chris Angelico
At a minimum, the PEP should state the objections fairly, and note
that the PEP author disagrees. A PEP isn't a sales pitch, it's a
summary of the discussions - so it absolutely should mention that
there's been significant opposition to the proposal, which did not get
resolved, if that's the reality.
Yes, and "significant opposition" doesn't just mean "I don't like
this". There's nothing to respond to in that.
Well, in a way it does. In another message on this thread you wrote:
I have attempted to explain the syntax. What is confusing?
def f(x=spam): ...
def f(x=>spam): ...
I'm not sure what concerns need to be addressed, because I don't
understand the concerns.
Speaking for myself, I would agree that there is nothing for you to
"address" per se. The problem, though, is that if half the people in
the discussion think it is confusing, then that in itself constitutes a
case that it is confusing and maybe we shouldn't add it. It's possible
this is why this discussion is kind of going in circles, because a lot
of it does come down to a somewhat subjective judgment about what is
"too confusing". But that doesn't mean that the subjective disagreement
about the relative costs and benefits doesn't exist, or that there is
some external source of truth about whether it "really is" confusing.
Your example there is very simple. But for me a lot of it comes down
to this:
def f(a=[], b@={}, c=some_function(a, b), d@=other_function(a, b, c)):
The PEP envisions a world in which you can have a list of arguments
like that, alternating back and forth between early and late binding,
and they can refer to each other in arbitrary ways, and the semantics
will differ from one argument to another (in that, e.g., for c the
arguments a and b will be evaluated in the enclosing scope, whereas for
d they are evaluated in the function scope). And yet none of the
late-bound defaults will exist as objects or be accessible or creatable
in any way apart from in function signatures, so all of this complexity,
insofar as it exists, is necessarily going to be crammed entirely into
function signatures, and also provide no benefit anywhere else.
Now certainly it is possible to write confusing code now, and on some
level any language that allows arbitrary nesting of expressions allows
arbitrarily confusing code. But from my perspective, the point is that
introducing this entire new "way" for things to be confusing just isn't
worth it for the benefit this PEP provides. Of course we can say "well
people could write confusing code like that but they
won't/shouldn't/etc.". But to my mind there is no reason to even give
them the chance. There would need to be a lot more than just "I don't
have to do an if-None check" to justify this new "surface area" of
complexity.
I agree with you that it is unclear how you would "address" this
objection, but that doesn't mean it's not a valid objection. To me it
is sort of an "irreducible" objection, in the sense that I don't
currently see how it could possibly be addressed and therefore I am
against the PEP outright. Maybe someone could come up with some
alternative mechanism for late-bound defaults that I would think was
fine, but as of now it's hard for me to envision what that could
possibly be. From where I stand it is like someone told me they were
going to put a horse into a suitcase; unless it turns out the suitcase
has a false bottom, or someone comes up with a kind of horse or suitcase
I've never seen before, it just fundamentally will not fit.
To touch on the "deferred" issue, I will also say it's quite possible
that someone might come up with a proposal for deferred evaluation and I
would still oppose it because I thought it was too confusing. That's
really neither here nor there. It's not that I oppose this PEP because
I think we can get some great deferred-eval PEP instead; it's just that
the benefit this PEP provides is too small to justify the complexity.
Now, I can add to that that in theory (in theory!) one reason I might
look differently on a deferred-eval proposal is that such a proposal
could provide benefits OUTSIDE of function signatures. Because your
proposal focuses only on function signatures, it can provide no benefit
outside of function signatures, and (in my view) there simply isn't
enough potential benefit available within function signatures to
outweigh the possibility of confusion that I outlined above. The
cost-benefit account of function signatures is overdrawn.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
Please explain to me *exactly* what your arguments against the current
proposal are.
No, thank you. They're all in the thread. Managing the thread is
your job as proponent.
In brief, my issues are introspection, adding syntax, the particular
syntax "=>". If you were proposing a general deferred type and
special syntax for default arguments, yes, I'd be more sympathetic to
a proposal implemented that way, but I don't depend on that for my
evaluation of the proposal as too limited to clear the bar for new
syntax even if that were the case.
On the other hand, I don't have a problem with the Pythonicity of your
proposal, specifically, I'm not sure why some opponents have talked
about "magic". In my opinion, your proposal is perfectly in line with
the Zen aphorisms about complexity and ease of understanding
implementations.
[T]here's endless mischaracterization and accusation happening.
It's not exactly nice to say this since you're the only fish who must
swim in this barrel, but you're not innocent of those issues. It's
happening on both sides, and you're not responsible for the majority
of it. However, a more flexible attitude on your part in
*understanding* others' claims would help a lot because you are the
central figure here. You say "you're wrong" all too quickly in this
thread. David was (to me) surprisingly direct about this, but as the
idiom goes, "he's not wrong".
Note: understanding != agreement, this thread does not display your
normal attitude, and "wrong" is not the other element of a doubleton.
With sincere respect,
Steve
= (which would mean something syntactical in many cases, just not what
is intended)."
David Mertz: prefers a keyword like `defer`
Ethan Furman: "Yes"
Barry Scott: "Yes"
Brendan Barnwell: "Yes, it is yet another reason not to do this."
Nicholas Cole: "I would actively avoid using this feature and
discourage people from using it because: I think that this imposes a
significant cognitive burden, not for the simple cases, but when
combined with the more advanced function definition syntax. I think
_*Objections to PEP 671 - Summary*_
There seems to be a problem understanding what the objections to PEP 671
are. Chris A wrote:
"Part of the problem is that it is really REALLY hard to figure out what
the actual objections are. I asked, and the one clear answer I got was
one subjective opinion that the cognitive load exceeded the benefit.
Great! That's one person's concern. I've responded to that by clarifying
parts of the cognitive load problem, and that's about as far as that can
go. But if there's nothing more specific than that, what do you want me
to respond to? How can I address the objections if the objections are as
vague as you're describing?"
Well, I have AFAIK re-read every post in the threads and am attempting
to summarise the objections to PEP 671 that I have found.
Disclaimer: I do NOT claim this is a completely objective survey. I
think by now everyone knows on which side of the fence I sit. In some
places I have added my own (biased) comments. Nonetheless I have
honestly tried to make a fair selection from all the relevant,
reasonably important posts. And if anyone thinks that their, or anyone
else's objection(s) have been omitted/understated/misrepresented, they
are welcome to edit this post and put the result in a new one.
Disclaimer: I have not referred to Steven d'Aprano's posts, because I
lack the time (ability?) to understand them all. AFAICT he is in favour
of something like this PEP, but a bit different.
AFAIK these were all the objections that were raised. There may be some
overlap or some sub-cases but this is how I've classified them:
(A) Don't like the proposed `=>` syntax.
(B) Too few use cases to bother making a change to the language.
Status quo wins.
(C) Some form of "deferred evaluation object" would do everything
this PEP does and more, and might be incompatible with it.
(D) The (late-binding) default value should be a first-class object
that can be accessed and manipulated.
(E) Calculation of a "default value" should be done in the function
body, not the function header.
(F) Concerns that functions using late-bound defaults were harder
to wrap.
(G) Backward compatibility.
(H) No other language supports both early- and late-binding.
Paul Moore and Brendan Barnwell were good enough to list their
objections when asked.
*Paul Moore*:
1. [OBJECTION B] "The problem that the PEP solves simply isn't
common enough, or difficult enough to work around, to justify new
syntax, plus a second way of defining default values."
2. [OBJECTION A] "There's no syntax that has gained consensus, and
the objections seem to indicate that there are some relatively
fundamental differences of opinion involved."
3. [OBJECTION H] "There's no precedent for languages having *both*
types of binding behaviour. Sure, late binding is more common, but
everyone seems to pick one form and stick with it."
4. [OBJECTION C] [paraphrased] deferred expressions are better
*Brendan Barnwell*:
1. [OBJECTION B] "The status quo is fine. Using None or another
sentinel and checking for it in the body has worked for many years and
is not that big a problem. In theory improvement is always possible,
but there is no urgency to change anything until we have a proposal with
fewer downsides. In addition, as discussed in some posts on this list,
not even all cases of None/sentinel defaults will be obviated by this
proposal."
2. [OBJECTION A] "Most of the proposed syntaxes make it difficult
to visually distinguish the late and early-bound defaults (because they
all look similar to a plain equals sign which will still mean a regular
early-bound default)."
3. [OBJECTION B AGAIN] "Regardless of the syntax, having the
potential for def-time and call-time behavior to be mixed and
interleaved in arbitrary ways within the same function signature is
confusing."
4. [OBJECTION D] 'Currently anything that is a function default is
some kind of Python object that can be inspected, interacted with,
and used independently of the function/argument whose default it
is. This proposal breaks that assumption. In other words I don't want
anything that is "a default" but is not a "default VALUE".'
5. [OBJECTION B AGAIN] "Miscellaneous wrinkles. By this I mean the
various sub-discussions about things like what order the late and early
defaults should be evaluated in. This is a sort of second-order
objection for me, because the objections I gave in my previous message
are enough for me to reject the proposal. But even assuming I agreed
with the broad outlines, these subsidiary concerns leave enough room for
confusion that I would not endorse the proposal. In other words there
are too many devils in the details that I feel would lead to
difficult-to-reason-about code and traps for the unwary."
I now quote the other (most relevant) posts re each of the objections:
_*(A) Don't like the proposed `=>` syntax.*__*
*_
Chris A asked if the cognitive burden of distinguishing between `=` and
'=>' was too great (leading with his chin, but hey).
Neil Giradhar: "Yes"
André Roberge: "Yes, prefer a keyword"
JL: "Yes. Something more distinctive feels better"
Barry Scott: [paraphrased] Wouldn't use with => syntax, but would
with @name syntax
David Mertz: "YES! A few weeks later than the prior long
discussion that I read in full, it took a triple take not to read it as
this has the potential to make debugging large code-bases much harder."
I quote from one of Chris A's replies re a `defer` keyword:
'The trouble is that this actually would be incompatible. If you can
defer an expensive calculation and have some "placeholder" value in the
variable 'result', then logically, you should be able to have that
placeholder as a function default argument, which would be an
early-bound default of the placeholder. That is quite different in
behaviour from a late-bound default, so if I were to use the word
"defer" for late-bound defaults, it would actually prevent the more
general proposal.'
_*(B) Too few use cases to bother making a change to the language.
Status quo wins.*_
potatochowder.com: "Don't find compelling."
David Mertz: "In particular, the cases where None will never, in
any conceivable circumstances, be a non-sentinel value are at least 98%
of all the functions (that have named parameters) I've ever written in
Python."
Stephen J. Turnbull: "Want to stick with my None sentinel and keep
my code consistent."
Paul Moore: "I can't think of a realistic case where I'd want to
actually use the new feature."
_*(C) Some form of "deferred evaluation object" would do everything this
PEP does and more, and might be incompatible with it.*_
I think that this has been thrashed out sufficiently that anything
I might say here could only stir up more pointless argument (which in
fact is still ongoing as I write).
_*(D) The (late-binding) default value should be a first-class object
that can be accessed and manipulated.*_
Eric V.Smith: [AFAIU, paraphrased] I want the default value to be
an object that I can inspect and change.
David Mertz: "For the reasons Eric Smith and others have pointed
out, I really WANT to keep inspectability of function signatures."
Stephen J. Turnbull: "More than any of these issues, the lack of a
well-defined, properly introspectable object bothers me."
Chris A gave a reply including 'If you consider the equivalent to be a
line of code in the function body, then the signature has become
MASSIVELY more useful. Instead of simply seeing "x=
Let me go back to the top and answer the original questions, then offer a few thoughts that have been germinating in my head through this discussion.
Chris Angelico wrote:
I've just updated PEP 671 https://www.python.org/dev/peps/pep-0671/
with some additional information about the reference implementation,
and some clarifications elsewhere.
*PEP 671: Syntax for late-bound function argument defaults*
Questions, for you all:
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Yes, but reluctantly. I take advantage of the features the language I'm using offers, even if I dislike them. And I greatly dislike this and would rather not have it for reasons described below.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
(It's absolutely valid to say "yes" and "yes", and feel free to say
which of those pulls is the stronger one.)
Yes. When a single punctuation character distinguishes between two closely related but significantly different behaviors, that is a large cognitive burden because it slows down code reading. I am used to scanning through unfamiliar code quickly, as I frequently go through the code of third-party modules, and if this becomes a thing, I would have to stop and slow down to check whether argument defaults are being bound early or late to comprehend the code I'm reading. I don't want that.
3) If "yes" to question 1, would you use it for any/all of (a) mutable
defaults, (b) referencing things that might have changed, (c)
referencing other arguments, (d) something else?
Whatever appropriate uses arise. Most likely that would be mutable defaults like `[]` and `{}`. Referencing other arguments might be useful as well. I don't see any use for (b) in my code.
I'd love to hear, also, from anyone's friends/family who know a bit of
Python but haven't been involved in this discussion. If late-bound
defaults "just make sense" to people, that would be highly
informative.
Late-bound defaults absolutely "just make sense". Way more sense than early-bound to me. This is reflected in the review of languages that Steven D'Aprano did a few days ago, where 80% of languages with syntax for default argument values use late binding, and all that bind early restrict default values to constants. Thus, when non-constant values are allowed as default values, current Python is entirely unique (among that list) in doing early binding. Early binding of non-constants has too many "gotcha"s and no real benefits, and any decent language being developed today ought to use late binding.
Unfortunately, we can't remake the past. Python is stuck with the wart of early binding. Note that on that list of languages Steven produced, **none** offer both early and late binding. Arguably, offering both and expecting users to grok which is which is *worse* than the current wart of a single behavior that works in one way and is not difficult to work around. And therefore I believe that Python should not in any case offer both behaviors simultaneously.
My preferences to resolve this are, in order:
1. Introduce `from __future__ import late_default`. When present, argument defaults in that file are late bound instead of early bound. After a suitable deprecation period, make the future statement the default behavior. Then Python will comply with best practices demonstrated by Steven's language review. I have not done any analysis, but I believe based on intuition that any breakage in libraries and scripts stemming from this would be relatively easy to fix, and most existing code should just work (in particular, the common existing usage of a sentinel as a default with an `is None` or `is sentinel` check in the body would not break and could be migrated to the new behavior at leisure). If true, it would result in minimal fuss for maximum benefit.
2. If a future statement and behavior change is deemed too disruptive, then keep early binding, do not introduce late binding, and introduce a new use for the keyword `pass` to represent an absent argument. Under this idea, `pass` would be accepted as an expression in the following three contexts, and exactly nowhere else: a) as a complete argument to a call, b) as a complete "value" for an argument default, and c) in the expressions `x is pass` and `x is not pass`, but only when both `x` is a parameter to the immediately enclosing function and the default value of that parameter is `pass`. This way, `pass` acts as a sentinel that isn't a valid value in any other context, which would solve the issue of when `None` is a valid value.
3. If both of those are undesired, then keep the wart in the status quo, and don't add late binding in any form.
1. Introduce `from __future__ import late_default`. When present, argument defaults in that file are late bound instead of early bound. After a suitable deprecation period, make the future statement the default behavior. Then Python will comply with best practices demonstrated by Steven's language review. I have not done any analysis, but I believe based on intuition that any breakage in libraries and scripts stemming from this would be relatively easy to fix, and most existing code should just work (in particular, the common existing usage of a sentinel as a default with an `is None` or `is sentinel` check in the body would not break and could be migrated to the new behavior at leisure). If true, it would result in minimal fuss for maximum benefit.
IMO this is strictly worse than supporting both alternatives with
syntactic differences. The language still needs to support both,
programmers still need to comprehend both, but instead of being able
to distinguish "def f(x=[]):" from "def f(x=>[]):", you have to go
look at the top of the file to see which way around it is. To the
extent that the distinction needs to be visible, it needs to be
visible at the function's definition, not at the top of the file.
2. If a future statement and behavior change is deemed too disruptive, then keep early binding, do not introduce late binding, and introduce a new use for the keyword `pass` to represent an absent argument. Under this idea, `pass` would be accepted as an expression in the following three contexts, and exactly nowhere else: a) as a complete argument to a call, b) as a complete "value" for an argument default, and c) in the expressions `x is pass` and `x is not pass`, but only when both `x` is a parameter to the immediately enclosing function and the default value of that parameter is `pass`. This way, `pass` acts as a sentinel that isn't a valid value in any other context, which would solve the issue of when `None` is a valid value.
This is a good idea that desperately needs good syntax. I don't like
"pass" used in this way. It's perfectly implementable but only if
someone can figure out how to write it.
(I'd define it as "the default is for the variable to be unbound" and
"if the variable is unbound". That makes very good sense and would
work within the language.)
Unfortunately this still has several of the problems that argument
defaults are supposed to solve. It means that you can mark a parameter
as optional, but you get no information about what it would be if
omitted. That's just as bad as the current sentinel option, with the
only advantage being that there's no sentinel.
Option 2 might actually make a good extension beyond PEP 671, but it's
not a replacement for it.
ChrisA
An obvious reason to oppose it is that it is a much worse way of writing an existing sentinel check.
This is what I want to see an example of.
They do? Please go into more detail. Please explain how this becomes worse:
def f(a:Optional[List]=None):
if a is None: a = []
... # use a
# becoming
def f(a:List=>[]):
... # use a
The hyperbolic assertion that EVERY function that uses a sentinel will
get worse is clearly false, since not every function would be
rewritten in this way. But even if every function that gets rewritten
is worse, you haven't shown how that's the case.
Please. I am BEGGING you for actual arguments here. I want something
that I can usefully respond to. Why do we have to go round and round
and round with nothing??
ChrisA
The PEP envisions a world in which you can have a list of arguments
like that, alternating back and forth between early and late binding,
and they can refer to each other in arbitrary ways, and the semantics
will differ from one argument to another (in that, e.g., for c the
arguments a and b will be evaluated in the enclosing scope, whereas for
d they are evaluated in the function scope). And yet none of the
late-bound defaults will exist as objects or be accessible or creatable
in any way apart from in function signatures, so all of this complexity,
insofar as it exists, is necessarily going to be crammed entirely into
function signatures, and also provide no benefit anywhere else.
As with every other complex construction, it can be misused. It's
certainly possible to use a list comprehension to replace any sort of
loop, no matter what the purpose; but for the most part, people don't
do that.
It is the simplest cases which are the most useful, and also the least
likely to cause confusion.
Remember, though: The comparison should be to a function that looks like this:
def f(a=[], b=_SENTINEL1, c=_SENTINEL2, d=_SENTINEL3):
if b is _SENTINEL1: b = {}
if c is _SENTINEL2: c = some_function(a, b)
if d is _SENTINEL3: d = other_function(a, b, c)
If you find the long-hand form more readable, use the long-hand form!
It's not going away. But the introspectability is no better or worse
for these two. The late-bound defaults "{}", "some_function(a, b)",
and "other_function(a, b, c)" do not exist as objects here. Using PEP
671's syntax, they would at least exist as string constants, allowing
you to visually see what would happen (and, for instance, see that in
help() and inspect.signature).
And of course, there will be cases where you mix and match:
def f(a=[], b=>{}, c=_SENTINEL, d=_SENTINEL): ...
Which is also perfectly acceptable. Sentinels don't have to be
abolished from the language.
Is there benefit in replacing just one simple case with something that
would be more readable, more program-comprehensible, etc, etc, even if
the others stay as they are?
ChrisA
1. Introduce `from __future__ import late_default`. When present,
argument defaults in that file are late bound instead of early bound. After
a suitable deprecation period, make the future statement the default
behavior. Then Python will comply with best practices demonstrated by
Steven's language review. I have not done any analysis, but I believe based
on intuition that any breakage in libraries and scripts stemming from this
would be relatively easy to fix, and most existing code should just work
(in particular, the common existing usage of a sentinel as a default with
an `is None` or `is sentinel` check in the body would not break and could
be migrated to the new behavior at leisure). If true, it would result in
minimal fuss for maximum benefit.
IMO this is strictly worse than supporting both alternatives with
syntactic differences. The language still needs to support both,
programmers still need to comprehend both, but instead of being able
to distinguish "def f(x=[]):" from "def f(x=>[]):", you have to go
look at the top of the file to see which way around it is. To the
extent that the distinction needs to be visible, it needs to be
visible at the function's definition, not at the top of the file.
Is it really worse? Yes, it's a burden during the transition period, but
the burden (both of managing both behaviors, and of grokking early-bound
mutable defaults) eventually goes away after a couple versions and adoption
time. After that, only one needs to be taught and understood. In contrast,
adding a syntax for late binding while keeping early binding means you have
to teach and comprehend both behaviors forever. I'd much, much rather have
a temporary burden than a permanent burden.
2. If a future statement and behavior change is deemed too disruptive,
then keep early binding, do not introduce late binding, and introduce a new
use for the keyword `pass` to represent an absent argument. Under this
idea, `pass` would be accepted as an expression in the following three
contexts, and exactly nowhere else: a) as a complete argument to a call, b)
as a complete "value" for an argument default, and c) in the expressions `x
is pass` and `x is not pass`, but only when both `x` is a parameter to the
immediately enclosing function and the default value of that parameter is
`pass`. This way, `pass` acts as a sentinel that isn't a valid value in any
other context, which would solve the issue of when `None` is a valid value.
This is a good idea that desperately needs good syntax. I don't like
"pass" used in this way. It's perfectly implementable but only if
someone can figure out how to write it.
(I'd define it as "the default is for the variable to be unbound" and
"if the variable is unbound". That makes very good sense and would
work within the language.)
Unfortunately this still has several of the problems that argument
defaults are supposed to solve. It means that you can mark a parameter
as optional, but you get no information about what it would be if
omitted. That's just as bad as the current sentinel option, with the
only advantage being that there's no sentinel.
Option 2 might actually make a good extension beyond PEP 671, but it's
not a replacement for it.
I never said it was a replacement. On the contrary, I explicitly said in
the first sentence of Option 2 to "do not introduce late binding". Period.
I am firmly against having both behaviors baked into the language forever.
If we cannot change the default behavior to late binding via a future
statement and a deprecation period, then we should not introduce late
binding at all. Python should have one permanent behavior, and should not
support both at once unless as part of a temporary transition to a new
default behavior.
In other words, the status quo with all of its warts is highly preferable
to me over PEP 671, however it's spelled.
I stand as follows:
My Option 1 (future statement, deprecation of early binding, and permanent
switch to late binding): +1
My Option 2 (syntax for an unbound argument): +0.4
Status quo (no changes at all): +0.3
PEP 671 or any other proposal to introduce a different spelling for late
binding (thereby supporting both at once forever): -1
Objections to PEP 671 - Summary
There seems to be a problem understanding what the objections to PEP 671 are. Chris A wrote:
"Part of the problem is that it is really REALLY hard to figure out what the actual objections are. I asked, and the one clear answer I got was one subjective opinion that the cognitive load exceeded the benefit. Great! That's one person's concern. I've responded to that by clarifying parts of the cognitive load problem, and that's about as far as that can go. But if there's nothing more specific than that, what do you want me to respond to? How can I address the objections if the objections are as vague as you're describing?"
Well, I have AFAIK re-read every post in the threads and am attempting to summarise the objections to PEP 671 that I have found.
Disclaimer: I do NOT claim this is a completely objective survey. I think by now everyone knows on which side of the fence I sit. In some places I have added my own (biased) comments. Nonetheless I have honestly tried to make a fair selection from all the relevant, reasonably important posts. And if anyone thinks that their, or anyone else's objection(s) have been omitted/understated/misrepresented, they are welcome to edit this post and put the result in a new one.
Disclaimer: I have not referred to Steven d'Aprano's posts, because I lack the time (ability?) to understand them all. AFAICT he is in favour of something like this PEP, but a bit different.
Unfortunately, neither do I fully understand them, so we're going to
have to wait for his clarifications. Or, better, a reference
implementation. Until then, let's leave that all aside.
AFAIK these were all the objections that were raised. There may be some overlap or some sub-cases but this is how I've classified them:
(A) Don't like the proposed `=>` syntax.
(B) Too few use cases to bother making a change to the language. Status quo wins.
(C) Some form of "deferred evaluation object" would do everything this PEP does and more, and might be incompatible with it.
(D) The (late-binding) default value should be a first-class object that can be accessed and manipulated.
(E) Calculation of a "default value" should be done in the function body, not the function header.
(F) Concerns that functions using late-bound defaults were harder to wrap.
(G) Backward compatibility.
(H) No other language supports both early- and late-binding.
(A) I have a few other syntaxes in the PEP, but if syntax is the only
issue and I haven't listed someone's preferred syntax, I would be
happy to consider others.
(B) I've shown some use cases. If you think these aren't of value, no
problem, but there's nothing to answer here.
(C) This has been asserted without evidence repeatedly. So far, I have
yet to see an example of how a deferred-evaluation object could
replace default argument handling; the examples have all been massive
overkill eg options for parallelism, and entail lots of extra syntax.
I'm also not convinced that they'd make late-bound defaults useless,
and that hasn't really been explained either.
(D) Understood. My response is two-fold: firstly, no other expression
in Python is a first-class object, and secondly, the manipulations
possible wouldn't be sufficiently general to cover all use-cases. I do
provide a textual representation of the default, which in simple
situations could be eval'd; in complex situations, nothing external
would work anyway.
(E) That's purely a matter of opinion, but if that's the case, why
aren't *all* defaults done in the body? Why do we have argument
defaults in the function header at all? Surely that's of value. It
certainly has been to me.
(F) Also a matter of opinion, given that *a,**kw is the most common
wrapping technique used, and will work reliably. Function signature
algebra is a much larger challenge than this.
(G) I'm not breaking compatibility in any way.
(H) This is true. But if the two syntaxes can be sufficiently similar,
the cost should be low, and the feature benefit would be high. Early
binding lets you "def f(x=x):" in a loop and capture each x as it goes
by. Late binding lets you "def f(x=>[]):" and get a new list every
time. Both have their places.
If Python had had late binding from the start, and no early binding,
we would have developed idioms for early binding (most likely a
decorator that captures values or something like that). Both
behaviours are sufficiently useful that programmers WILL implement
them, language support or not.
I quote from one of Chris A's replies re a `defer` keyword:
'The trouble is that this actually would be incompatible. If you can defer an expensive calculation and have some "placeholder" value in the variable 'result', then logically, you should be able to have that placeholder as a function default argument, which would be an early-bound default of the placeholder. That is quite different in behaviour from a late-bound default, so if I were to use the word "defer" for late-bound defaults, it would actually prevent the more general proposal.'
The incompatibility here, btw, would be using a "defer" keyword to
mean late-binding, which isn't the same thing as a "defer expr"
keyword that creates a deferred expression. PEP 671 would not be
incompatible with a generic deferred-expression concept, as long as
they don't use the same word.
(D) The (late-binding) default value should be a first-class object that can be accessed and manipulated.
Eric V.Smith: [AFAIU, paraphrased] I want the default value to be an object that I can inspect and change.
David Mertz: "For the reasons Eric Smith and others have pointed out, I really WANT to keep inspectability of function signatures."
Stephen J. Turnbull: "More than any of these issues, the lack of a well-defined, properly introspectable object bothers me."
Chris A gave a reply including 'If you consider the equivalent to be a line of code in the function body, then the signature has become MASSIVELY more useful. Instead of simply seeing "x=", you can see "x=>[]" and be able to see what the value would be.'
Correct. In general, expressions are not first-class objects in
Python; they only become them when turned into functions or classes
(including the special functions used by genexps/comps). We do not
have an introspectable, externally-testable, first-class object to
represent any other expression:
x = 1/y if y else "invalid"
There's no object for "1/y". Trying to create one would be a nightmare
of subtleties, where assignment expressions would break things,
nonlocal variable references would become tricky, etc, etc. Similarly:
def f(y):
def g(x=1/y):
...
return g
There's no object for "1/y" here either. As an early-bound default,
the expression is simply inlined into the body of f, as part of the
construction of the function. Now with late-bound defaults:
... you still don't get a first-class object for the expression, but
you DO get a string constant that describes it. What is it about this
third example that makes the bar so much higher than for the other
two?
Thank you for this summary. Are there any objections still unanswered?
Which of these parts is important enough to add to the PEP? I think
I'll be adding a section on introspectability, since it seems to be a
hot topic, but I'm not sure about the others.
ChrisA
Remember, though: The comparison should be to a function that looks like this:
def f(a=[], b=_SENTINEL1, c=_SENTINEL2, d=_SENTINEL3):
if b is _SENTINEL1: b = {}
if c is _SENTINEL2: c = some_function(a, b)
if d is _SENTINEL3: d = other_function(a, b, c)
If you find the long-hand form more readable, use the long-hand form!
It's not going away. But the introspectability is no better or worse
for these two. The late-bound defaults "{}", "some_function(a, b)",
and "other_function(a, b, c)" do not exist as objects here. Using PEP
671's syntax, they would at least exist as string constants, allowing
you to visually see what would happen (and, for instance, see that in
help() and inspect.signature).
I don't want to get bogged down in terminology but I am becoming
increasingly frustrated by you using the term "default" both for things
that are values and things that are not, as if there is no difference
between them. There are no late-bound defaults here, in the sense that
I mean, which as I said before has to do with default VALUES. There is
just code in the function body that does stuff. I am fine with code in
a function body doing stuff, but that is the purview of the function and
not the argument. An individual ARGUMENT having a default VALUE is not
the same as the FUNCTION defining BEHAVIOR to deal with a missing value
for an argument.
Your discussion of this point (as I interpret it, at least) continues
to take it for granted that it is perfectly fine to move stuff between
the function signature and the function body and those are somehow the
same thing. They are not. Currently in Python there is nothing that
can be used as a default argument this way:
def f(a=<some expression here>):
. . .that cannot also be done this way:
obj = <some expression here>
def f(a=obj):
The reason I want the function defaults to "exist as objects" is to
maintain this consistency. There is no need for random code in the
function body to be accessible from outside, because, well, it's in the
function body! It's not in the signature! If you want to change that,
okay, but I feel in the PEP and your discussion of it you are not fully
acknowledging that this is actually how functions work in Python now,
and thus the PEP would break some existing assumptions and bring about a
nontrivial change in how code can be refactored. (This again may be why
there is disagreement about "how confusing" the proposed change would
be. Part of what I mean by "confusing" is that it requires changing
assumptions like the one I mentioned.)
Is there benefit in replacing just one simple case with something that
would be more readable, more program-comprehensible, etc, etc, even if
the others stay as they are?
There might be, but that's not what this PEP is, because it is not
restricted to simple cases. And even if there were such benefit, it's
not at all clear to me that it would justify the increase in the
complexity of the language. Your example above, as awkward as it is,
doesn't require the reader to know anything about Python that hasn't
changed in 10+ years. In that sense it is more readable than a new
alternative which requires the reader to learn the meaning of new syntax.
I'll also point out that you misread my original example, which was:
def f(a=[], b@={}, c=some_function(a, b), d@=other_function(a, b, c)):
Note that c is early-bound here, whereas in your example you have
changed it to "late-bound" (aka "behavior inside the function"). I
realize this was probably just a thinko, but perhaps it also gently
illustrates my point that peril lies in allowing early and late-bound
defaults to mix within the same signature. It's not always trivial to
remember which arguments are which. :-)
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
(I started writing this this morning, and there’s been quite a bit more
discussion since, but I don’t think it’s completely too late)
Thanks Paul (and others that have commented). All reasonable
considerations, and of course what the SC will be thinking about. However,
a key question.
space. If it gets implemented, and deferred expressions are added
later, we'll end up with two ways of achieving one result, with one
way being strictly better than the other.
This is what confuses me. Yes, the evaluation of late-bound defaults is
technically “deferred” until the function is called, but it is a very
specific use case, with the time and namespace(s) of evaluation clearly
defined. I presume a “general deferred expression” object would provide
some flexibility on both those counts, or it wouldn’t be a general solution
at all. Sure, a general purpose deferred expression could be used to
simulate late-bound defaults, but you would still need to write extra code
or have special syntax or meaning for late bound defaults. Or have them be
poorly defined.
Could someone making this argument please propose a possible deferred
expression symtax, and how it would be used for late-bound defaults?
Otherwise, it seems completely irrelevant to the topic at hand.
Example:
Let’s say that we have:
x = deferred exp
Where deferred is a keyword, and exp is an expression which is not
evaluated right away.
So: when does exp get evaluated? Maybe when you reference x?
OK, now:
y = x
Will result in exp being evaluated.
But what namespace does it use? The one where it’s evaluated? Or the one
where it was defined? Pick one.
OK, now we have:
def fun(x = deferred exp):
Now when does expr get evaluated? If it’s when it’s used, then either the
result depends on what’s changed in the namespace where the function is
defined, or the result depends on what’s changed it the function’s
namespace.
So:
def fun(n, x = deferred n**2):
y = x
Gives a different value for y than
def fun(n, x = deferred n**2):
n += 2
y = x
Or n is looked up
In the global namespace, and that could lead to some real WTF moments.
Which makes defining the default in the signature pretty pointless— reading
the signature wouldn’t give you any new info.
So then we’re back to saying that a deferred expression always gets
evaluated in the function namespace, and at the beginning of the function.
Which is, well, pretty much what the PEP is proposing.
Maybe a deferred expression gets evaluated when specifically asked to:
y = realize x
OK, but then when you look at a function signature, you’ll have no idea
when the exp will be realized — so no idea what it will do. So, back to
having to explain it in the docstring, like we do for the sentinel
approach.
I’m sure you all think I’ve made a straw man here — something poorly
designed so I can shoot it down, but I’ve honestly done my best.
If some one can come up with a general deferred expression approach that
can also be used clearly and simply as a late bound default—please let us
know!
-CHB
--
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
(H) This is true. But if the two syntaxes can be sufficiently similar,
the cost should be low, and the feature benefit would be high. Early
binding lets you "def f(x=x):" in a loop and capture each x as it goes
by. Late binding lets you "def f(x=>[]):" and get a new list every
time. Both have their places.
(The "two syntaxes" here is referring to syntaxes for early and late
binding.) I'm actually worried about the reverse. When the two
syntaxes are similar, it will be easier to mistake one for the other.
(D) The (late-binding) default value should be a first-class object that can be accessed and manipulated.
Eric V.Smith: [AFAIU, paraphrased] I want the default value to be an object that I can inspect and change.
David Mertz: "For the reasons Eric Smith and others have pointed out, I really WANT to keep inspectability of function signatures."
Stephen J. Turnbull: "More than any of these issues, the lack of a well-defined, properly introspectable object bothers me."
Chris A gave a reply including 'If you consider the equivalent to be a line of code in the function body, then the signature has become MASSIVELY more useful. Instead of simply seeing "x=", you can see "x=>[]" and be able to see what the value would be.'
Correct. In general, expressions are not first-class objects in
Python; they only become them when turned into functions or classes
(including the special functions used by genexps/comps). We do not
have an introspectable, externally-testable, first-class object to
represent any other expression:
x = 1/y if y else "invalid"
There's no object for "1/y". Trying to create one would be a nightmare
of subtleties, where assignment expressions would break things,
nonlocal variable references would become tricky, etc, etc. Similarly:
As I have stated repeatedly, ternary expressions are not parallel
because they do not have a distinct definition-time and call-time.
Please stop bringing up the ternary operator case and pretending it is
the same as a function. It is not, for reasons that I have already
explained several times.
To try stating this in yet another way, currently if I have:
def f(a=<some code here>)
<some code here> must be something that evaluates to a first-class
object, and the "argument default" IS that first-class object --- not
bytecode to generate it, not some behavior that evaluates the
expression, no, the default value is itself an object. This would not
be the case for late-bound defaults under the PEP. (Rather, as I
phrased it in another post, there would not "be" a late-bound default at
all; there would just be some behavior in the function to do some stuff
when that argument isn't passed.)
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
My Option 1 (future statement, deprecation of early binding, and permanent
switch to late binding): +
I have no authority about this at all, and a negligible amount of
influence, but I’ve been around Python a long time:
That is not going to happen.
Way too much of a breaking change, and early bound defaults are useful, and
awkward to replicate. It was not an accident.
-CHB
--
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
(H) This is true. But if the two syntaxes can be sufficiently similar,
the cost should be low, and the feature benefit would be high. Early
binding lets you "def f(x=x):" in a loop and capture each x as it goes
by. Late binding lets you "def f(x=>[]):" and get a new list every
time. Both have their places.
(The "two syntaxes" here is referring to syntaxes for early and late
binding.) I'm actually worried about the reverse. When the two
syntaxes are similar, it will be easier to mistake one for the other.
For the most part, there won't be any confusion. If the default is a
simple value, it's going to be early-bound (eg "def
list.pop(index=-1):"). If it's a mutable that needs to be constructed,
it'll be late-bound. The distinction will be important if it's looked
up from another namespace ("def
fetch(timeout=some.default.timeout):"), but even then, the distinction
is whether the default is affected by future changes or not; the two
forms are functionally very similar.
There's no object for "1/y". Trying to create one would be a nightmare
of subtleties, where assignment expressions would break things,
nonlocal variable references would become tricky, etc, etc. Similarly:
As I have stated repeatedly, ternary expressions are not parallel
because they do not have a distinct definition-time and call-time.
Please stop bringing up the ternary operator case and pretending it is
the same as a function. It is not, for reasons that I have already
explained several times.
To try stating this in yet another way, currently if I have:
def f(a=<some code here>)
<some code here> must be something that evaluates to a first-class
object, and the "argument default" IS that first-class object --- not
bytecode to generate it, not some behavior that evaluates the
expression, no, the default value is itself an object. This would not
be the case for late-bound defaults under the PEP. (Rather, as I
phrased it in another post, there would not "be" a late-bound default at
all; there would just be some behavior in the function to do some stuff
when that argument isn't passed.)
The VALUE is a first-class object - that's the result of evaluating
the expression. With early-bound defaults, that's the only thing that
gets saved - not the expression, just the resulting value. (Which can
be seen if you do something like "def f(x=0x100):", which will show
the default as 256.)
Remember, a late-bound default is most similar to this code:
def f(a=<optional>):
if a was omitted: a = <some code here>
And in that form, the code isn't available as a first-class object.
That's why I say that it is parallel to every other partial expression
in Python. Until you evaluate it, there is no first-class object
representing it. (A code object comes close, but it's more than just
an expression - it also depends on its context. A function requires
even more context.)
Suppose you wanted, for a function with an early-bound default, to see
the actual expression that got you there. This can be synthesized in
some cases (eg an enum with a good repr), but in the general case, how
can you figure out that expression? Could you reevaluate it? Suppose
you wanted to make a decorator like this:
@reevaluate_defaults_every_call
def f(x=[], y=func(), z=x.length + y): ...
Could you, using early-bound defaults, get hold of the expressions?
Are they first-class objects?
ChrisA
Remember, though: The comparison should be to a function that looks like this:
def f(a=[], b=_SENTINEL1, c=_SENTINEL2, d=_SENTINEL3):
if b is _SENTINEL1: b = {}
if c is _SENTINEL2: c = some_function(a, b)
if d is _SENTINEL3: d = other_function(a, b, c)
If you find the long-hand form more readable, use the long-hand form!
It's not going away. But the introspectability is no better or worse
for these two. The late-bound defaults "{}", "some_function(a, b)",
and "other_function(a, b, c)" do not exist as objects here. Using PEP
671's syntax, they would at least exist as string constants, allowing
you to visually see what would happen (and, for instance, see that in
help() and inspect.signature).
I don't want to get bogged down in terminology but I am becoming
increasingly frustrated by you using the term "default" both for things
that are values and things that are not, as if there is no difference
between them.
That's absolutely correct: I am using the term "default" for anything
that provides a default for an optional argument that was omitted. In
some cases, they are default values. In other cases, they are default
expressions. If your docstring says "omitting d will use the length of
a", then the default for d is len(a).
There are no late-bound defaults here, in the sense that
I mean, which as I said before has to do with default VALUES. There is
just code in the function body that does stuff. I am fine with code in
a function body doing stuff, but that is the purview of the function and
not the argument. An individual ARGUMENT having a default VALUE is not
the same as the FUNCTION defining BEHAVIOR to deal with a missing value
for an argument.
In a technical sense, the default value for b is _SENTINEL1, but would
you describe that in the docstring, or would you say that omitting b
would use a new empty dictionary? You're getting bogged down, not in
terminology, but in mechanics. At an abstract level, the default for
that argument is whatever would be used if the argument is omitted.
Your discussion of this point (as I interpret it, at least) continues
to take it for granted that it is perfectly fine to move stuff between
the function signature and the function body and those are somehow the
same thing. They are not. Currently in Python there is nothing that
can be used as a default argument this way:
def f(a=<some expression here>):
. . .that cannot also be done this way:
obj = <some expression here>
def f(a=obj):
That is correct, and that is a current limitation. Is it a fundamental?
Up until very recently, there was nothing in Python that could be used here:
obj = <some expression here>
that would also have the effect of:
x = 42
Was that a fundamental limitation? It changed. A feature was added,
and what had previously been impossible became possible. If a feature
is rejected simply because it makes something possible that previously
wasn't, then no proposal should ever be accepted.
To justify this, please explain WHY it is so important for defaults to
all be objects. Not just "that's how they are now", but why that is an
important feature.
If you want to change that,
okay, but I feel in the PEP and your discussion of it you are not fully
acknowledging that this is actually how functions work in Python now,
and thus the PEP would break some existing assumptions and bring about a
nontrivial change in how code can be refactored.
I'm definitely changing how functions work. Otherwise I wouldn't be
proposing anything. Yes, I want it to be possible for argument
defaults to no longer be objects. I consider this a feature, NOT a
flaw. The counter-argument has always just been "but that's how it
is".
(This again may be why
there is disagreement about "how confusing" the proposed change would
be. Part of what I mean by "confusing" is that it requires changing
assumptions like the one I mentioned.)
Right. And I put it to you that it won't actually be very confusing
after all. The most common cases will simply behave as expected - in
fact, they will behave MORE as expected than they currently do.
Note that c is early-bound here, whereas in your example you have
changed it to "late-bound" (aka "behavior inside the function"). I
realize this was probably just a thinko, but perhaps it also gently
illustrates my point that peril lies in allowing early and late-bound
defaults to mix within the same signature. It's not always trivial to
remember which arguments are which. :-)
Oh, sorry, I didn't know whether that was a typo on your part or a
mistake on mine. With synthetic examples like this, it's not always
easy to tell. In real code, that would be unlikely to be an issue (and
honestly, the distinction between a and b here would be highly unusual
in a single function).
There is one small aspect of mixing that has some technical
consequences, but I'm declaring it to be undefined behaviour:
currently, the late-bound default for b would be allowed to refer to
c, and it would succeed. Similarly, b's default could refer to d, but
only if d is provided by the caller, and not if it's using its
default. (Otherwise you'd get UnboundLocalError.) But I am absolutely
okay with a future change, or a different Python implementation,
declaring those to be errors. Other than that, the rule is simple:
parameters get initialized from left to right.
ChrisA
My Option 1 (future statement, deprecation of early binding, and permanent switch to late binding): +
I have no authority about this at all, and a negligible amount of influence, but I’ve been around Python a long time:
That is not going to happen.
Way too much of a breaking change, and early bound defaults are useful, and awkward to replicate. It was not an accident.
I agree that it's a massively breaking change, but we could easily
have a system of early-bound defaults. Behold:
import functools
def lock_kwdefaults(**kw1):
def wrapper(f):
@functools.wraps(f)
def inner(*a, **kw2):
return f(*a, **{**kw2, **kw1})
return inner
return wrapper
@lock_kwdefaults(a=[])
def foo(val, a=>[], b=>[]):
a.append(val)
b.append(val)
print(a, b)
foo(1)
foo(2)
foo(3)
It would be an idiom no worse than we currently have. Either form can
implement the other form.
But there's no way we're going to do a complete migration to
late-bound. I'm definitely not advocating for that, and I highly doubt
the SC would approve any such proposal. (If nothing else, it would
impose a pointless run-time cost on all functions with defaults.)
ChrisA
To try stating this in yet another way, currently if I have:
def f(a=<some code here>)
<some code here> must be something that evaluates to a first-class
object, and the "argument default" IS that first-class object --- not
bytecode to generate it, not some behavior that evaluates the
expression, no, the default value is itself an object. This would not
be the case for late-bound defaults under the PEP. (Rather, as I
phrased it in another post, there would not "be" a late-bound default at
all; there would just be some behavior in the function to do some stuff
when that argument isn't passed.)
The VALUE is a first-class object - that's the result of evaluating
the expression. With early-bound defaults, that's the only thing that
gets saved - not the expression, just the resulting value. (Which can
be seen if you do something like "def f(x=0x100):", which will show
the default as 256.)
Right, but that's what I'm saying. To me it is not a default unless
there is a value that gets saved. Otherwise it is just behavior in the
function.
Remember, a late-bound default is most similar to this code:
def f(a=<optional>):
if a was omitted: a = <some code here>
And in that form, the code isn't available as a first-class object.
That's why I say that it is parallel to every other partial expression
in Python. Until you evaluate it, there is no first-class object
representing it. (A code object comes close, but it's more than just
an expression - it also depends on its context. A function requires
even more context.)
Yes, but that's the point. To me that code is quite a different matter
from "a late-bound default" as I conceive it. I get the impression that
you really do see that code as "a late-bound default" but to me it is
not at all. It just behavior in the function. It's true that the
result is to assign a certain value to the variable, but that alone
doesn't make it "a default" to me.
I mean, maybe it would help if I say it this way. To me here's kind of
how a function works:
1. A function is distinct from other kinds of expression in that some
things happen when you define it, and it also "saves" some things for
later when you call it.
2. When you define it, it saves two things: some code to be run when
it's called (i.e., the function body) and some values to be used if some
arguments aren't provided. (It probably saves some other stuff too like
the docstring but these are the relevant ones for our purposes.) It
stores those values with a mapping to their corresponding arguments
(that is if you do `def f(a=1, b=2)` it stores that 1 goes with a and 2
with b).
3. Those values that are saved to be used later are the argument
defaults. That's it. The only thing that can "be an argument default"
is a thing that is saved when the function is defined and is (maybe)
retrieved later when it's called. Everything that isn't a value is
BEHAVIOR. You can do other things to the function at def time (like
replace it with another one using a decorator, effectively augmenting it
somehow) but argument defaults are values, they're not behavior.
From that perspective, there is all the difference in the world between
what we currently have, which you apparently think of as sort of a
"manual" late-bound default, and a real late-bound default, which would
be a value that is stored at def time. If the effect of writing the
signature a certain way (e.g., `=>[]` instead of `=[]`) is not to store
a value but to somehow manipulate the bytecode of the function body, I
don't consider that an argument default; it's a behavioral modification
more akin to a decorator that wraps the original function.
Part of the reason I feel this way is because what we currently have is
in no way restricted to specifying default values. What if I have this:
def f(a=<optional>):
if a was omitted and random.random() < 0.5: a = <some code here>
. . . or perhaps more realistically:
def f(a=<optional>, b=<optional>, c=<optional>):
if a was omitted and 0 < b <= 5 and not c.some_boolean_attr:
a = <some code here>
Now what "is the default"? Is there one? There is no clear
distinction between code in the function body that defines a "late-bound
default" and code that just does something else. In the former case the
behavior is random. In the latter case it may be that you can say in
English what the "default" is, but I don't consider that an ARGUMENT
default. It may be default BEHAVIOR of the FUNCTION to decide in a
certain way what to assign to that local variable, but that's not a
default "of the argument", it's part of the function's defined behavior
like anything else. In order for me to consider it an argument default,
it has to have some independent status as a "thing" that is individually
associated with the argument, not simply rolled into the bytecode of the
function as a whole. For instance, this function has default behavior too:
def f(a=<optional>):
if a was omitted:
download_a_file_from_the_internet()
else:
dont_download_anything()
But the behavior doesn't suddenly become "a default" just because the
code happens to assign a value to a.
In my conception you can't specify an argument default by means of
modifications to the function body, because the function body is
arbitrary code that can do anything. The different between "an argument
default" and "stuff that the function does as part of its behavior" is
that the argument default is segmented out and has its own independent
existence.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
Remember, though: The comparison should be to a function that looks like this:
def f(a=[], b=_SENTINEL1, c=_SENTINEL2, d=_SENTINEL3):
if b is _SENTINEL1: b = {}
if c is _SENTINEL2: c = some_function(a, b)
if d is _SENTINEL3: d = other_function(a, b, c)
If you find the long-hand form more readable, use the long-hand form!
It's not going away. But the introspectability is no better or worse
for these two. The late-bound defaults "{}", "some_function(a, b)",
and "other_function(a, b, c)" do not exist as objects here. Using PEP
671's syntax, they would at least exist as string constants, allowing
you to visually see what would happen (and, for instance, see that in
help() and inspect.signature).
I don't want to get bogged down in terminology but I am becoming
increasingly frustrated by you using the term "default" both for things
that are values and things that are not, as if there is no difference
between them.
That's absolutely correct: I am using the term "default" for anything
that provides a default for an optional argument that was omitted. In
some cases, they are default values. In other cases, they are default
expressions. If your docstring says "omitting d will use the length of
a", then the default for d is len(a).
Your definition is somewhat circular, because you say that a default is
"anything that provides a default". But that says "default" again. So
what is a default?
By your definition, any arbitrary code inside a function body that
eventually assigns something to an argument name is a default. (It is
not clear to me whether you would consider some code a default if it may
or may not assign a value to an argument, depending on some conditions.)
So I don't agree with that definition. That can be default BEHAVIOR,
but it is function behavior; it is not an argument default.
There are no late-bound defaults here, in the sense that
I mean, which as I said before has to do with default VALUES. There is
just code in the function body that does stuff. I am fine with code in
a function body doing stuff, but that is the purview of the function and
not the argument. An individual ARGUMENT having a default VALUE is not
the same as the FUNCTION defining BEHAVIOR to deal with a missing value
for an argument.
In a technical sense, the default value for b is _SENTINEL1, but would
you describe that in the docstring, or would you say that omitting b
would use a new empty dictionary? You're getting bogged down, not in
terminology, but in mechanics. At an abstract level, the default for
that argument is whatever would be used if the argument is omitted.
I don't agree. At an abstract level, there is no clear dividing line
between what you call an argument default and just "arbitrary behavior
of the function". What if "what would be used" depends on random
numbers or data from some external source?
Or, again, what you are describing is not an argument default (in my
conception). It may be the BEHAVIOR of the function to do a certain
thing (like use a certain value in place of an omitted argument) but
unless that behavior is segmented and associated with the argument
itself (not merely part of the function's code flow) I don't consider it
an argument default.
As for the docstring, yes, I might well mention _SENTINEL1 in the
docstring. I certainly wouldn't see anything wrong with that. That's
what the default is. Again, the function may USE that value in some
way, but that doesn't mean that's not what the default is; it just means
the function conditions its behavior on its argument value, as any
function may do on any argument value, omitted or not. I get the
impression you think that in a case like that the default "really is"
something else defined in the body of the function, but again I
disagree. The default really is _SENTINEL1. Conceptually we may
understand that the function will use that value in a special way, but
that is not really any different than understanding that passing "r" to
open() will open the file for reading while passing "w" will open it for
writing. It's just that to know how to use a function you need to know
more than the default values of the arguments; you need to know what
they MEAN, and (at least with current technology :-) we have no way of
deriving that from the source code.
You're quite right that "at an abstract level" it may be the case that
the default behavior is to do a certain thing, but I guess one way to
state my position would be that I think that is TOO abstract of a level
to worry about representing in code. At an abstract level I may say
"this function computes the number of paths of length N between the
given nodes in the given graph", but I don't expect that to be mentioned
in the signature or automatically provided in the docstring. I would
certainly WRITE it in the docstring, but I don't expect Python to deduce
that "abstract" level of meaning from code annotations and write that
docstring for me.
In other words, I think mechanics is the right level to be at here. We
cannot hope to capture the abstract level that you're describing, and I
think doing so will just muddle matters. At an abstract level we say
""this function computes the number of paths of length N between the
given nodes in the given graph" but what we write is `def n_paths(graph,
node1, node2)`. I don't see any reason we need to be able to write
`len(x)` in the function signature just because at that abstract level
we think of it as something that may be computed later. This is
especially so because, as I mentioned above, there is no clear line
separating "code that we can write in a function to assign a default
value to an argument" and "code we can write in a function for other
purposes" --- and thus there is no way to distinguish behavior that is
"tied" to a particular argument from just code that uses any old
combination of values it wants.
We write code in terms of instrumental units which necessarily are at a
slightly more concrete level than the purely abstract or conceptual
realm of "what this function does". For instance, objects (which, until
now, every function argument, default or not, is). I don't see any
reason why late-bound defaults should be represented in code in a way
that attempts to capture this abstract level when other aspects of
functions are not and cannot be.
To justify this, please explain WHY it is so important for defaults to
all be objects. Not just "that's how they are now", but why that is an
important feature.
Because I'm used to reasoning about Python code in terms of operations
on objects, and so are a lot of other people. Everything I or anyone
else currently needs to know about how functions and their arguments
work in Python can be thought of in terms of objects. Why add a new
complication? I mean, okay, maybe that is really just saying "that's
how they are now", although it's more like "right now defaults are part
of the big set of things that are objects and this change would peel
them off and create a new type of thing".
But apart from that, I think part of what makes Python a nice language
is the way that many language functions are represented in terms of
objects, for instance the iterator and descriptor protocols. The idea
of the object as a locus of functionality --- that the way you "do
something" (like loop or access an attribute) is represented as "get an
object representing the functionality and call certain methods on it"
--- gives unity to many Python features. It's true that's a pretty
abstract reason, but I think it's a legit one.
Also, let's remember that burden of evidence is really the other way
around here. Can you really explain WHY it is so important for
late-bound defaults to be represented with special syntax of the type
you propose? Not only can you not rely on "that's how they are now"
(because they're not), but in fact you must overcome the reverse
argument, namely that people have been doing pretty well with just
early-bound defaults for decades now. In other words, even if it is not
particularly important for defaults to be objects, it may still be more
important than being able to write a "late-bound default" (aka "behavior
in the function body") in the signature.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
Your definition is somewhat circular, because you say that a
default is "anything that provides a default". But that says "default"
again. So what is a default?
This seems somewhat disingenuous. A "default" is anything that provides
a value for a parameter when it is omitted from the call.
Andrew
I have attempted to explain the syntax. What is confusing?
def f(x=spam): ...
def f(x=>spam): ...
Are we using the term "confusing" in different ways? I'm saying that
it's easy to confuse the two forms = and =>, as they look very
similar, and in many (nearly all?) cases, do exactly the same
(although I assume => would be slower, as I doubt the compiler would
be able to tell that it could optimise away an unnecessary late
binding).
It's not that I find the syntax or semantics hard to understand, but
that the two forms are easily confused with each other. And it's not
that I can't see which is which on a careful reading. It's in a
maintenance context where I imagine the issues - a PR that includes a
typo using = instead of =>, which doesn't get picked up in code review
because the reviewer missed the typo. Or a function that uses a
default of =[] when =>[] was intended, and gets a subtle timing error
that the tests don't pick up and reviewers misread as "using that late
binding syntax" rather than thinking "=[] is a known gotcha".
Note that the same argument applies for a lot of alternative spellings
as well: :=, >=, =:, ?=, all have the same problem. It's more the
structure that's the issue. And yes, I know people don't confuse a+b
and a-b. This is a people problem, so it can't be dealt with by
applying a set of objective, mechanical rules on what works and what
doesn't. But that doesn't mean it's all opinion. User interface design
(which is what this is, essentially) is *hard*, unfortunately.
I suspect your next question will be "what do you expect me to do
about that?" And I'll be honest, I don't know. It's your proposal, not
mine. Reconsider some of the other proposals for syntax? Explore
approaches other than "the syntax is deliberately similar to the
existing early-bind syntax" which is stated as a design principle in
the PEP? Note the concern in the PEP, with some concrete details on
what proportion of people in the discussion believed it would be an
issue, and state that you don't consider it a significant risk?
Sometimes the colour of the bikeshed *is* important. But you get to pick.
I'm not sure what concerns need to be addressed, because I don't
understand the concerns.
Fair, but you can state them, surely? Even if you're just copying what
people say into the PEP, and noting that these are open issues that
the PEP author cannot address without further explanation of the
issue, that's better than having nothing (and having the same
objections raised over and over again). At the moment the PEP doesn't
even *have* an "open issues" section.
Maybe I'm just getting caught up on all the
side threads about "deferreds are better" and "it should be a magical
function instead" and I've lost some of the basics? Feel free to
repost a simple concern and I will attempt to respond.
Well, I did that when you asked the last time. Maybe you don't think
my concerns are "simple", but I don't know what to do about that - if
you are saying my concerns "aren't stated simply enough" for a
response, I'm out of ideas for how to proceed. If my ideas were that
simple to state, they'd be simple to fix, and I wouldn't be worried
about them. I could explain any one of my objections in more detail.
But how would a paragraph or two of explanation "simplify" things?
Yes, many of the concerns are somewhat subjective, and many of them
are subject to a certain amount of interpretation. That's the nature
of this sort of issue. If I said to you that the biggest issue here
was that "in the group of people on python-ideas who were motivated
enough to get involved in discussions, about half of the participants
were arguing against the proposal"¹ would that be a concrete enough
objection for you? Count it as my 5th objection, if you like. I know
we're not putting the PEP to a vote here, but proposed changes *are*
supposed to achieve a certain level of consensus (in the normal course
of events - of course the SC can approve anything, even if it's hugely
unpopular, that's their privilege).
So your response to my concern that opinion is divided on the PEP, is
to say that actually, no-one likes it? I get that you're frustrated,
but that doesn't seem useful.
Quite frankly, I'm just
about ready to throw the whole thing in, because this entire thread
has devolved to complaints that are nearly impossible to respond to -
or are just repetition of things that ARE responded to in the PEP.
OK, well I've given my reservations (and note that I have repeatedly
used the terms "reservations" and "concerns" rather than "objections",
and that's deliberate). I won't continue to discuss or clarify them
unless you have specific questions, as I feel that doing so will just
increase your frustration here, and I don't want to do that. But if
you *do* feel there's merit in trying to address points that have been
raised, feel free to pick one of my points and ask more detailed
questions, if you think that would help.
I don't think it's true that everyone objects, though. There are some
posters who support the proposal enthusiastically. And yes, there's a
lot of debate, but it feels to me like it's mostly trying to be
constructive, but people are getting frustrated because they can't get
their point across.
Maybe we don't need any new features in Python. Maybe Python 3.10 is
already the perfect language, and we should just preserve it in amber.
I assume that's frustration speaking, because no-one's saying that.
Sure, the number of changes that meet the bar for inclusion has gone
down. The bar is higher when you're the world's most popular
programming language, after all. And fixing imperfections that people
have survived with for years can be a hard sell (I still have hope
that someday we'll get a better spelling for lambda, though!) But if
we give up on all innovation as a result, we won't be the most popular
language for long :-(
Paul
We seem to be arguing in circles and talking past each other here about
nomenclature:
f(arg=>dflt)
What does it matter if we call it "a default" or "just behaviour in the
function" or "a Jabberwocky" or "<your name here>"?
The RELEVANT question should be "Is it useful"?
Best wishes
Rob Cliffe
On 09/12/2021 08:06, Brendan Barnwell wrote:
To try stating this in yet another way, currently if I have:
def f(a=<some code here>)
<some code here> must be something that evaluates to a
first-class
object, and the "argument default" IS that first-class object --- not
bytecode to generate it, not some behavior that evaluates the
expression, no, the default value is itself an object. This would not
be the case for late-bound defaults under the PEP. (Rather, as I
phrased it in another post, there would not "be" a late-bound
default at
all; there would just be some behavior in the function to do some stuff
when that argument isn't passed.)
The VALUE is a first-class object - that's the result of evaluating
the expression. With early-bound defaults, that's the only thing that
gets saved - not the expression, just the resulting value. (Which can
be seen if you do something like "def f(x=0x100):", which will show
the default as 256.)
Right, but that's what I'm saying. To me it is not a default
unless there is a value that gets saved. Otherwise it is just
behavior in the function.
Remember, a late-bound default is most similar to this code:
def f(a=<optional>):
if a was omitted: a = <some code here>
And in that form, the code isn't available as a first-class object.
That's why I say that it is parallel to every other partial expression
in Python. Until you evaluate it, there is no first-class object
representing it. (A code object comes close, but it's more than just
an expression - it also depends on its context. A function requires
even more context.)
Yes, but that's the point. To me that code is quite a different
matter from "a late-bound default" as I conceive it. I get the
impression that you really do see that code as "a late-bound default"
but to me it is not at all. It just behavior in the function. It's
true that the result is to assign a certain value to the variable, but
that alone doesn't make it "a default" to me.
I mean, maybe it would help if I say it this way. To me here's
kind of how a function works:
1. A function is distinct from other kinds of expression in that some
things happen when you define it, and it also "saves" some things for
later when you call it.
2. When you define it, it saves two things: some code to be run when
it's called (i.e., the function body) and some values to be used if
some arguments aren't provided. (It probably saves some other stuff
too like the docstring but these are the relevant ones for our
purposes.) It stores those values with a mapping to their
corresponding arguments (that is if you do `def f(a=1, b=2)` it stores
that 1 goes with a and 2 with b).
3. Those values that are saved to be used later are the argument
defaults. That's it. The only thing that can "be an argument
default" is a thing that is saved when the function is defined and is
(maybe) retrieved later when it's called. Everything that isn't a
value is BEHAVIOR. You can do other things to the function at def
time (like replace it with another one using a decorator, effectively
augmenting it somehow) but argument defaults are values, they're not
behavior.
From that perspective, there is all the difference in the world
between what we currently have, which you apparently think of as sort
of a "manual" late-bound default, and a real late-bound default, which
would be a value that is stored at def time. If the effect of writing
the signature a certain way (e.g., `=>[]` instead of `=[]`) is not to
store a value but to somehow manipulate the bytecode of the function
body, I don't consider that an argument default; it's a behavioral
modification more akin to a decorator that wraps the original function.
Part of the reason I feel this way is because what we currently
have is in no way restricted to specifying default values. What if I
have this:
def f(a=<optional>):
if a was omitted and random.random() < 0.5: a = <some code here>
. . . or perhaps more realistically:
def f(a=<optional>, b=<optional>, c=<optional>):
if a was omitted and 0 < b <= 5 and not c.some_boolean_attr:
a = <some code here>
Now what "is the default"? Is there one? There is no clear
distinction between code in the function body that defines a
"late-bound default" and code that just does something else. In the
former case the behavior is random. In the latter case it may be that
you can say in English what the "default" is, but I don't consider
that an ARGUMENT default. It may be default BEHAVIOR of the FUNCTION
to decide in a certain way what to assign to that local variable, but
that's not a default "of the argument", it's part of the function's
defined behavior like anything else. In order for me to consider it
an argument default, it has to have some independent status as a
"thing" that is individually associated with the argument, not simply
rolled into the bytecode of the function as a whole. For instance,
this function has default behavior too:
def f(a=<optional>):
if a was omitted:
download_a_file_from_the_internet()
else:
dont_download_anything()
But the behavior doesn't suddenly become "a default" just because
the code happens to assign a value to a.
In my conception you can't specify an argument default by means of
modifications to the function body, because the function body is
arbitrary code that can do anything. The different between "an
argument default" and "stuff that the function does as part of its
behavior" is that the argument default is segmented out and has its
own independent existence.
(I started writing this this morning, and there’s been quite a bit more discussion since, but I don’t think it’s completely too late)
Your email client is doing "white on white" again. You should try to
get that fixed :-(
I agree, without an actual proposal for how deferred expressions work,
it's impossible to be specific about what might be affected. And
holding one proposal hostage to something that might never happen
isn't reasonable. So I'm happy to class a lot of the discussion about
deferred expressions as off-topic.
My point was more specific, though - that whenever deferred
expressions have been discussed in the past (and they do come up every
so often) it's fairly clear that one case they would be expected to
handle would be calculating function defaults at call time (at least,
that's my impression). So if they ever do get implemented, they would
render this PEP obsolete. I'm not aware of any case where we've added
a language feature knowing that people were interested in something
strictly more general. Indeed, switch statements were rejected for
years because they "weren't general enough", without any specific more
general feature being on the table.
So my objection is that I don't see why this particular PEP warrants
an exception to that precedent, and why in this specific case, "never
is often better than *right* now" doesn't apply. What's changed to
make late bound defaults worth fixing right now? That's *not* a
rhetorical question - I'm happy if someone can tell me, if now is the
right time to do this, then what's different about getting this in
3.11, as opposed to 3.8, or 3.14? And I'm fine with an imprecise
answer here - assignment expressions got in mostly because "they've
been asked about for years, we have the momentum right now, and Guido
wants to do it". If the argument here is similarly imprecise, that's
fine - a precise argument would be *stronger*, but a lot of the
concerns being raised are imprecise, I'm not expecting to hold the
responses to a standard the concerns don't achieve. But I would like a
response.
Paul
PS While I'm posting, a *huge* thanks to Rob Cliffe for preparing and
posting that summary of the concerns. That was extremely helpful.
To try stating this in yet another way, currently if I have:
def f(a=<some code here>)
<some code here> must be something that evaluates to a first-class
object, and the "argument default" IS that first-class object --- not
bytecode to generate it, not some behavior that evaluates the
expression, no, the default value is itself an object. This would not
be the case for late-bound defaults under the PEP. (Rather, as I
phrased it in another post, there would not "be" a late-bound default at
all; there would just be some behavior in the function to do some stuff
when that argument isn't passed.)
The VALUE is a first-class object - that's the result of evaluating
the expression. With early-bound defaults, that's the only thing that
gets saved - not the expression, just the resulting value. (Which can
be seen if you do something like "def f(x=0x100):", which will show
the default as 256.)
Right, but that's what I'm saying. To me it is not a default unless
there is a value that gets saved. Otherwise it is just behavior in the
function.
Remember, a late-bound default is most similar to this code:
def f(a=<optional>):
if a was omitted: a = <some code here>
And in that form, the code isn't available as a first-class object.
That's why I say that it is parallel to every other partial expression
in Python. Until you evaluate it, there is no first-class object
representing it. (A code object comes close, but it's more than just
an expression - it also depends on its context. A function requires
even more context.)
Yes, but that's the point. To me that code is quite a different matter
from "a late-bound default" as I conceive it. I get the impression that
you really do see that code as "a late-bound default" but to me it is
not at all. It just behavior in the function. It's true that the
result is to assign a certain value to the variable, but that alone
doesn't make it "a default" to me.
Fair enough. To me, it's all defaults; or rather, the only thing that
is truly an aspect of the function is whether the parameter is
mandatory or optional.
In a sense, that's all you need. You could write every function to
simply have mandatory parameters and optional parameters, and then
have everything done as "behaviour in the function". Having the open
mode default to "r" is really just the function's behaviour - if you
omit the parameter, it's going to do this.
Function default arguments are a convenience for the common cases, and
also an aid to documentation. For instance:
str.encode(self, /, encoding='utf-8', errors='strict')
If you just call s.encode(), you know exactly what it'll do. And if
you call s.encode("ISO-8859-1"), you know that it'll assume strict
error handling. This is a good thing. But in a sense, we could just
write it as:
str.encode(self, /, [encoding], [errors])
and leave the rest in the body.
What defines what belongs in the body and what belongs in the signature?
1. A function is distinct from other kinds of expression in that some
things happen when you define it, and it also "saves" some things for
later when you call it.
A lot of things. It knows its context, for instance. A function isn't
just a block of code or an expression - it's a thing that exists in a
particular world.
2. When you define it, it saves two things: some code to be run when
it's called (i.e., the function body) and some values to be used if some
arguments aren't provided. (It probably saves some other stuff too like
the docstring but these are the relevant ones for our purposes.) It
stores those values with a mapping to their corresponding arguments
(that is if you do `def f(a=1, b=2)` it stores that 1 goes with a and 2
with b).
A lot is saved when you compile it, which happens before it's defined.
At definition time, I believe that all it has to do is gather the
previously-saved things, prepare the defaults, and save the context
(closure cells) if required.
(At a technical level, positional and pos-or-kwd arguments are stored
in a tuple, kwonly are stored in a dict. But same difference.)
3. Those values that are saved to be used later are the argument
defaults. That's it. The only thing that can "be an argument default"
is a thing that is saved when the function is defined and is (maybe)
retrieved later when it's called. Everything that isn't a value is
BEHAVIOR. You can do other things to the function at def time (like
replace it with another one using a decorator, effectively augmenting it
somehow) but argument defaults are values, they're not behavior.
That clearly defines the way things currently are. Is that actually
how things must be, or only how it is?
From that perspective, there is all the difference in the world between
what we currently have, which you apparently think of as sort of a
"manual" late-bound default, and a real late-bound default, which would
be a value that is stored at def time. If the effect of writing the
signature a certain way (e.g., `=>[]` instead of `=[]`) is not to store
a value but to somehow manipulate the bytecode of the function body, I
don't consider that an argument default; it's a behavioral modification
more akin to a decorator that wraps the original function.
If it is storing a value, it's manipulating the bytecode of the
surrounding function. Code has to go somewhere.
Part of the reason I feel this way is because what we currently have is
in no way restricted to specifying default values. What if I have this:
def f(a=<optional>):
if a was omitted and random.random() < 0.5: a = <some code here>
. . . or perhaps more realistically:
def f(a=<optional>, b=<optional>, c=<optional>):
if a was omitted and 0 < b <= 5 and not c.some_boolean_attr:
a = <some code here>
Now what "is the default"? Is there one? There is no clear
distinction between code in the function body that defines a "late-bound
default" and code that just does something else. In the former case the
behavior is random. In the latter case it may be that you can say in
English what the "default" is, but I don't consider that an ARGUMENT
default.
And in cases like these, it probably shouldn't be put in the default,
because it is just behaviour. Of course you *could* cram that into the
function signature, but it probably doesn't belong. Hard to say,
though, without a real example.
Some things are function behaviour. Others are argument defaults. The
ones that are argument defaults should go in the signature; the ones
that aren't should go in the body. At the moment, there's a technical
limitation that means that "new empty list" cannot be spelled as an
argument default, and therefore we need workarounds. That's the only
part that I want to change.
Remember, when the walrus operator was introduced, it wasn't meant to
replace all assignment. When list comprehensions were brought in, they
weren't meant to replace all lists. When match/case was added to the
language, it wasn't intended to supplant all if/elif trees. In each
case, a feature exists for the situations where it is more expressive
than the alternatives, and for other cases, don't use it.
It may be default BEHAVIOR of the FUNCTION to decide in a
certain way what to assign to that local variable, but that's not a
default "of the argument", it's part of the function's defined behavior
like anything else. In order for me to consider it an argument default,
it has to have some independent status as a "thing" that is individually
associated with the argument, not simply rolled into the bytecode of the
function as a whole. For instance, this function has default behavior too:
def f(a=<optional>):
if a was omitted:
download_a_file_from_the_internet()
else:
dont_download_anything()
But the behavior doesn't suddenly become "a default" just because the
code happens to assign a value to a.
Correct. Behaviour doesn't become an argument default. But argument
defaults can be crammed into function behaviour if, for a technical
reason, they can't be spelled as defaults. If it's wrong to cram
behaviour into the signature, isn't it just as wrong to stuff the
default into the body of the function?
In my conception you can't specify an argument default by means of
modifications to the function body, because the function body is
arbitrary code that can do anything. The different between "an argument
default" and "stuff that the function does as part of its behavior" is
that the argument default is segmented out and has its own independent
existence.
Precisely. That is exactly the distinction. Of course, when there are
technical limitations, sometimes things have to go into other places,
but ideally, the argument defaults should be segmented out and given
their correct position in the signature.
ChrisA
I have attempted to explain the syntax. What is confusing?
def f(x=spam): ...
def f(x=>spam): ...
Are we using the term "confusing" in different ways? I'm saying that
it's easy to confuse the two forms = and =>, as they look very
similar, and in many (nearly all?) cases, do exactly the same
(although I assume => would be slower, as I doubt the compiler would
be able to tell that it could optimise away an unnecessary late
binding).
Are = and := confusing? They are also very similar, and they have some
subtle distinctions. Is it a problem for two different operators to
look similar when they do very similar things?
Argument defaults look like assignment because it makes sense for them
to look like assignment. That's not an accident.
It's not that I find the syntax or semantics hard to understand, but
that the two forms are easily confused with each other. And it's not
that I can't see which is which on a careful reading. It's in a
maintenance context where I imagine the issues - a PR that includes a
typo using = instead of =>, which doesn't get picked up in code review
because the reviewer missed the typo. Or a function that uses a
default of =[] when =>[] was intended, and gets a subtle timing error
that the tests don't pick up and reviewers misread as "using that late
binding syntax" rather than thinking "=[] is a known gotcha".
If this is accepted, it will become a standard idiom to see "=>[]" or
"=>{}" or whatever, and seeing "=[]" will continue to look wrong. I
think you're underestimating people's ability to understand code.
But maybe we ARE using the term "confusing" in different ways. Maybe
I'm trying to respond to completely the wrong thing. If the concern is
that the new syntax is "confusing" but I'm interpreting that word
wrongly, please clarify.
I'm not sure what concerns need to be addressed, because I don't
understand the concerns.
Fair, but you can state them, surely? Even if you're just copying what
people say into the PEP, and noting that these are open issues that
the PEP author cannot address without further explanation of the
issue, that's better than having nothing (and having the same
objections raised over and over again). At the moment the PEP doesn't
even *have* an "open issues" section.
No, I can't, because every time I try to pin down actual issues, they
slip away. I can't nail jelly to the PEP.
So your response to my concern that opinion is divided on the PEP, is
to say that actually, no-one likes it? I get that you're frustrated,
but that doesn't seem useful.
The problem is that even people who claim to be in support of it are
arguing against it. That's what makes it incredibly hard to figure out
what concerns are still open.
Maybe we don't need any new features in Python. Maybe Python 3.10 is
already the perfect language, and we should just preserve it in amber.
I assume that's frustration speaking, because no-one's saying that.
Sure, the number of changes that meet the bar for inclusion has gone
down. The bar is higher when you're the world's most popular
programming language, after all. And fixing imperfections that people
have survived with for years can be a hard sell (I still have hope
that someday we'll get a better spelling for lambda, though!) But if
we give up on all innovation as a result, we won't be the most popular
language for long :-(
Exactly. And this particular issue is fixing an issue that people
point to as a gotcha - "be careful of mutable default arguments" or
even "mutable values don't work in argument defaults" (which I see all
too often). With a good response of "they behave that way if you write
=>[] instead of =[]", it still looks and feels like an argument
default, but it has the slightly different behaviour that people
expect, and all is well.
ChrisA
Are = and := confusing? They are also very similar, and they have some
subtle distinctions. Is it a problem for two different operators to
look similar when they do very similar things?
Maybe. I don't use assignment expressions - I don't really have any
good uses for them. And no project I maintain is able to use them yet
as we support older Python versions. So I'm honestly not sure. I'd be
more concerned about confusing := and ==, as both can appear in the
same expression. But again I don't know.
And I don't know *for certain* about =>. It's a concern, not a
showstopper. No-one has yet (as far as I can recall) shown real-life
cases, such as a PR against an existing project that changes code to
use the new syntax. There's no requirement on a PEP to do that much
work up front, so I'm not demanding that you do. But I'll remain
concerned about confusability until I see some "real world" evidence
like that. And like with assignment expressions, the PEP might still
get accepted even though I'm concerned.
Argument defaults look like assignment because it makes sense for them
to look like assignment. That's not an accident.
I didn't say it was. But there comes a point when too many things all
look like X because they are similar to X, and we run out of easily
distinguishable variations. Are we at that point yet? I don't know.
Might we be? In my opinion, yes we might. And the PEP should document
*your* view on the question, which can be "I think it's fine, but
everyone disagrees with me", or "the majority of people on the list
think it's OK", or "there were concerns expressed but they were all
theoretical and I believe the benefits of this proposal outweigh the
theoretical risk". Or whatever. It's your PEP, not mine. Your view is
what should be in there.
If this is accepted, it will become a standard idiom to see "=>[]" or
"=>{}" or whatever, and seeing "=[]" will continue to look wrong. I
think you're underestimating people's ability to understand code.
I think you're (at a minimum) overestimating *my* ability to spot
mistakes like this. I'm certain that I've missed similar things in
reviews in the past. Maybe I'm not typical. Maybe you're thinking more
of people writing their own code, and less about maintainers reviewing
externally submitted code. Or maybe projects will define standards and
good practices that prohibit over-use of late-bound defaults, or
prefer None sentinels where possible, or just impose a general "keep
it simple" rule and use that to minimise the risk of confusion.
And maybe people will end up cargo culting "you use => for mutable
values like [] or {}, and = for everything else" and not actually
understanding when values get bound at all. And people do the right
thing by accident, so everything is fine (sort of).
But maybe we ARE using the term "confusing" in different ways. Maybe
I'm trying to respond to completely the wrong thing. If the concern is
that the new syntax is "confusing" but I'm interpreting that word
wrongly, please clarify.
I think your responses above are addressing what I mean by
"confusing". But your responses are essentially that you don't agree
with me. That's fine, and I'm not demanding that we have to agree
here. But to be a fair reflection of the discussion, the PEP should
note this concern and be clear that there was disagreement with your
position. That is literally all I am asking here.
If the PEP ends up containing a section with a list of concerns people
have, and your statement that you don't agree with them, that's 100%
fine. Maybe it makes the PEP less persuasive. I don't have a problem
with that - after all, I'm the one arguing that the status quo is
fine. You're the one arguing that the change is worth making, so
presumably you believe the section in the PEP that describes the
benefits of this change outweighs this.
I'm not sure what concerns need to be addressed, because I don't
understand the concerns.
Fair, but you can state them, surely? Even if you're just copying what
people say into the PEP, and noting that these are open issues that
the PEP author cannot address without further explanation of the
issue, that's better than having nothing (and having the same
objections raised over and over again). At the moment the PEP doesn't
even *have* an "open issues" section.
No, I can't, because every time I try to pin down actual issues, they
slip away. I can't nail jelly to the PEP.
Rob Cliffe did an excellent summary. Just copy and paste that. If you
really don't understand any of the points in spite of his summary and
everyone's comments, just note in the PEP that this was the objection
and it's not been addressed because you didn't understand what the
problem was. And move on.
So your response to my concern that opinion is divided on the PEP, is
to say that actually, no-one likes it? I get that you're frustrated,
but that doesn't seem useful.
The problem is that even people who claim to be in support of it are
arguing against it. That's what makes it incredibly hard to figure out
what concerns are still open.
OK. I don't claim to be in support of the PEP, but I'm only mildly
against it (I prefer the status quo). I'll be satisfied if Rob's list
of people's concerns is noted in the PEP. You can say these haven't
been addressed because you couldn't get a clear understanding of what
the precise issue was. I'm fine with that. Or you can say that the
concern is noted, but you don't plan on changing the PEP because you
don't believe the issue is significant enough. Or whatever, as long as
the issue is noted and there's a response. Do that and I'll have
nothing further to say. I'll still prefer the status quo, but you
don't have to persuade me, you just have to persuade the SC, and as
long as the points I've made are noted for the SC's consideration,
that's all I ask.
Exactly. And this particular issue is fixing an issue that people
point to as a gotcha - "be careful of mutable default arguments" or
even "mutable values don't work in argument defaults" (which I see all
too often). With a good response of "they behave that way if you write
=>[] instead of =[]", it still looks and feels like an argument
default, but it has the slightly different behaviour that people
expect, and all is well.
And people don't all agree. So what? Present your case, represent the
objections fairly, and that's good enough.
As someone said (Jonathan Goble, I think) if we were designing Python
now, we'd quite likely have picked late bound defaults as the only
behaviour. Early binding has some unfortunate warts. But we didn't,
and we're not talking about changing that decision. So the question is
whether the existing warts are better or worse than any warts a new
solution would have (multiple types of binding behaviour, confusable
syntax, etc - you know the drill, you've heard it way too many times
now).
You think the fix is better than the status quo. Some people here
don't (you yourself said "everyone", I think you were being
pessimistic, but I think "at least half" is probably fair). Pass the
facts to the SC and let them decide. Or drop the idea, if you feel
it's not going to succeed.
Paul
On Sun, Dec 5, 2021 at 10:38 AM David Mertz, Ph.D.
wrote:
I first discussed the idea of a "generalized deferred object/type"
on this list at least two years ago, probably more than three (I
haven't looked through archives lately to be sure the dates). The
idea got some vague interest, but I was too lazy, or too busy, or
whatever, to write an actual PEP or implementation.
I don’t think a full PEP is required at this point, but throughout
this discussion, the idea of a deferred object being a better solution
to this same problem has been brought up.
My sense it that some folks think if we want late-bound defaults, we
really should just do deferred objects.
But I honestly don’t get it. My idea of a deferred object would be
quite different that this, would not be a great replacement for this,
and could quite happily co-exist with this idea. Clearly I’m missing
something.
I've used this example before.
Let's assume backticks create a "deferred object" (a name I hate, but
I'll continue to use in this discussion). I realize backticks won't fly,
I'm using them deliberately so as to not argue about the syntax here.
If I want to specify a late bound dataclasses default parameter,
currently I have to say:
@dataclasses.dataclass
class A:
my_list: list = dataclasses.field(default_factory=list)
What I'd like to be able to say:
@dataclasses.dataclass
class A:
my_list: list = `[]`
In the class A, before @dataclass is called, I want A.my_list to be a
"deferred object" that I could then use when @dataclass is generating
__init__(). Exactly how and when the "deferred object" would get
evaluated is debatable, but not so important for the sake of this
discussion. Suffice it to say that it would either be explicitly or
implicitly evaluated at the start of __init__.
I think you can see that this would benefit from similar functionality
to late-bound parameters, and that if we had this more general mechanism
that late-bound parameters could use the same underlying mechanism.
Had these "deferred objects" existed when I designed dataclasses, I
would have used them instead of the clunky default_factory. PEP 671 does
not help with this use case, where a late-bound parameter isn't
specified in a function definition. I need the late-bound parameter to
be stored in an object I can refer to later.
Eric
I’m still not clear if this is a disagreement about something more than
terminology, but as I understand it, other languages that have non-constant
defaults use late binding, and call them “defaults”.
It seems to be a well accepted term.
-CHB
On Thu, Dec 9, 2021 at 12:45 AM Brendan Barnwell
wrote:
def f(a=[], b=_SENTINEL1, c=_SENTINEL2, d=_SENTINEL3):
if b is _SENTINEL1: b = {}
if c is _SENTINEL2: c = some_function(a, b)
if d is _SENTINEL3: d = other_function(a, b, c)
If you find the long-hand form more readable, use the long-hand form!
It's not going away. But the introspectability is no better or worse
for these two. The late-bound defaults "{}", "some_function(a, b)",
and "other_function(a, b, c)" do not exist as objects here. Using PEP
671's syntax, they would at least exist as string constants, allowing
you to visually see what would happen (and, for instance, see that in
help() and inspect.signature).
I don't want to get bogged down in terminology but I am becoming
increasingly frustrated by you using the term "default" both for things
that are values and things that are not, as if there is no difference
between them.
That's absolutely correct: I am using the term "default" for anything
that provides a default for an optional argument that was omitted. In
some cases, they are default values. In other cases, they are default
expressions. If your docstring says "omitting d will use the length of
a", then the default for d is len(a).
Your definition is somewhat circular, because you say that a
default is
"anything that provides a default". But that says "default" again. So
what is a default?
By your definition, any arbitrary code inside a function body that
eventually assigns something to an argument name is a default. (It is
not clear to me whether you would consider some code a default if it may
or may not assign a value to an argument, depending on some conditions.)
So I don't agree with that definition. That can be default BEHAVIOR,
but it is function behavior; it is not an argument default.
There are no late-bound defaults here, in the sense that
I mean, which as I said before has to do with default VALUES. There is
just code in the function body that does stuff. I am fine with code in
a function body doing stuff, but that is the purview of the function and
not the argument. An individual ARGUMENT having a default VALUE is not
the same as the FUNCTION defining BEHAVIOR to deal with a missing value
for an argument.
In a technical sense, the default value for b is _SENTINEL1, but would
you describe that in the docstring, or would you say that omitting b
would use a new empty dictionary? You're getting bogged down, not in
terminology, but in mechanics. At an abstract level, the default for
that argument is whatever would be used if the argument is omitted.
I don't agree. At an abstract level, there is no clear dividing
line
between what you call an argument default and just "arbitrary behavior
of the function". What if "what would be used" depends on random
numbers or data from some external source?
Or, again, what you are describing is not an argument default (in
my
conception). It may be the BEHAVIOR of the function to do a certain
thing (like use a certain value in place of an omitted argument) but
unless that behavior is segmented and associated with the argument
itself (not merely part of the function's code flow) I don't consider it
an argument default.
As for the docstring, yes, I might well mention _SENTINEL1 in the
docstring. I certainly wouldn't see anything wrong with that. That's
what the default is. Again, the function may USE that value in some
way, but that doesn't mean that's not what the default is; it just means
the function conditions its behavior on its argument value, as any
function may do on any argument value, omitted or not. I get the
impression you think that in a case like that the default "really is"
something else defined in the body of the function, but again I
disagree. The default really is _SENTINEL1. Conceptually we may
understand that the function will use that value in a special way, but
that is not really any different than understanding that passing "r" to
open() will open the file for reading while passing "w" will open it for
writing. It's just that to know how to use a function you need to know
more than the default values of the arguments; you need to know what
they MEAN, and (at least with current technology :-) we have no way of
deriving that from the source code.
You're quite right that "at an abstract level" it may be the case
that
the default behavior is to do a certain thing, but I guess one way to
state my position would be that I think that is TOO abstract of a level
to worry about representing in code. At an abstract level I may say
"this function computes the number of paths of length N between the
given nodes in the given graph", but I don't expect that to be mentioned
in the signature or automatically provided in the docstring. I would
certainly WRITE it in the docstring, but I don't expect Python to deduce
that "abstract" level of meaning from code annotations and write that
docstring for me.
In other words, I think mechanics is the right level to be at
here. We
cannot hope to capture the abstract level that you're describing, and I
think doing so will just muddle matters. At an abstract level we say
""this function computes the number of paths of length N between the
given nodes in the given graph" but what we write is `def n_paths(graph,
node1, node2)`. I don't see any reason we need to be able to write
`len(x)` in the function signature just because at that abstract level
we think of it as something that may be computed later. This is
especially so because, as I mentioned above, there is no clear line
separating "code that we can write in a function to assign a default
value to an argument" and "code we can write in a function for other
purposes" --- and thus there is no way to distinguish behavior that is
"tied" to a particular argument from just code that uses any old
combination of values it wants.
We write code in terms of instrumental units which necessarily are
at a
slightly more concrete level than the purely abstract or conceptual
realm of "what this function does". For instance, objects (which, until
now, every function argument, default or not, is). I don't see any
reason why late-bound defaults should be represented in code in a way
that attempts to capture this abstract level when other aspects of
functions are not and cannot be.
To justify this, please explain WHY it is so important for defaults to
all be objects. Not just "that's how they are now", but why that is an
important feature.
Because I'm used to reasoning about Python code in terms of
operations
on objects, and so are a lot of other people. Everything I or anyone
else currently needs to know about how functions and their arguments
work in Python can be thought of in terms of objects. Why add a new
complication? I mean, okay, maybe that is really just saying "that's
how they are now", although it's more like "right now defaults are part
of the big set of things that are objects and this change would peel
them off and create a new type of thing".
But apart from that, I think part of what makes Python a nice
language
is the way that many language functions are represented in terms of
objects, for instance the iterator and descriptor protocols. The idea
of the object as a locus of functionality --- that the way you "do
something" (like loop or access an attribute) is represented as "get an
object representing the functionality and call certain methods on it"
--- gives unity to many Python features. It's true that's a pretty
abstract reason, but I think it's a legit one.
Also, let's remember that burden of evidence is really the other
way
around here. Can you really explain WHY it is so important for
late-bound defaults to be represented with special syntax of the type
you propose? Not only can you not rely on "that's how they are now"
(because they're not), but in fact you must overcome the reverse
argument, namely that people have been doing pretty well with just
early-bound defaults for decades now. In other words, even if it is not
particularly important for defaults to be objects, it may still be more
important than being able to write a "late-bound default" (aka "behavior
in the function body") in the signature.
--
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
Your email client is doing "white on white" again. You should try to
get that fixed :-(
Aarrgg— it was iPhone “dark mode” — white on black for me. Why the heck it
preserves the text color (and not the background) is beyond me. I hope this
works now.
My point was more specific, though - that whenever deferred
expressions have been discussed in the past (and they do come up every
so often) it's fairly clear that one case they would be expected to
handle would be calculating function defaults at call time (at least,
that's my impression). So if they ever do get implemented, they would
render this PEP obsolete.
But that’s exactly my question. I can see how a general purpose deferred
expression system could be used for the implementation of late-bound
defaults. But I can’t see how one would be both generally useful, and clear
and simple when used for late bound defaults.
Which is why I don’t think this PEP would be rendered obsolete— maybe the
reference implementation would be, but syntax specific to late bound
defaults would still be helpful.
This is different than the switch-case example, where pattern matching is a
clear superset.
-CHB
--
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
But I honestly don’t get it. My idea of a deferred object would be
quite different that this, would not be a great replacement for this,
and could quite happily co-exist with this idea. Clearly I’m missing
something.
I've used this example before.
Let's assume backticks create a "deferred object" (a name I hate, but
I'll continue to use in this discussion). I realize backticks won't
fly, I'm using them deliberately so as to not argue about the syntax here.
If I want to specify a late bound dataclasses default parameter,
currently I have to say:
@dataclasses.dataclass
class A:
my_list: list = dataclasses.field(default_factory=list)
What I'd like to be able to say:
@dataclasses.dataclass
class A:
my_list: list = `[]`
In the class A, before @dataclass is called, I want A.my_list to be a
"deferred object" that I could then use when @dataclass is generating
__init__(). Exactly how and when the "deferred object" would get
evaluated is debatable, but not so important for the sake of this
discussion. Suffice it to say that it would either be explicitly or
implicitly evaluated at the start of __init__.
I think you can see that this would benefit from similar functionality
to late-bound parameters, and that if we had this more general
mechanism that late-bound parameters could use the same underlying
mechanism.
And in case I wasn't clear: to get the late-bound parameter
functionality using this syntax, you'd use:
def foo(my_list = `[]`):
That's why I think we should have a larger concept that just late-bound
parameters: I think there's a general concept here that can be extended
beyond parameters. And that's why I thing not restricting it to a
function-definition-only syntax is important: we should produce a syntax
that can be used in more places than just functions. This is why I think
we need to decide on this larger scope before accepting the narrow
function-definition-only syntax: if we decide to add "deferred objects"
later, we'd have two ways to specify late-bound parameters [0].
Eric
[0]: Or arguments, I can never remember which is which: someone needs to
invent a memorable mnemonic device.
Had these "deferred objects" existed when I designed dataclasses, I
would have used them instead of the clunky default_factory. PEP 671
does not help with this use case, where a late-bound parameter isn't
specified in a function definition. I need the late-bound parameter to
be stored in an object I can refer to later.
But I honestly don’t get it. My idea of a deferred object would be quite different that this, would not be a great replacement for this, and could quite happily co-exist with this idea. Clearly I’m missing something.
I've used this example before.
Let's assume backticks create a "deferred object" (a name I hate, but I'll continue to use in this discussion). I realize backticks won't fly, I'm using them deliberately so as to not argue about the syntax here.
If I want to specify a late bound dataclasses default parameter, currently I have to say:
@dataclasses.dataclass
class A:
my_list: list = dataclasses.field(default_factory=list)
What I'd like to be able to say:
@dataclasses.dataclass
class A:
my_list: list = `[]`
In the class A, before @dataclass is called, I want A.my_list to be a "deferred object" that I could then use when @dataclass is generating __init__(). Exactly how and when the "deferred object" would get evaluated is debatable, but not so important for the sake of this discussion. Suffice it to say that it would either be explicitly or implicitly evaluated at the start of __init__.
I think you can see that this would benefit from similar functionality to late-bound parameters, and that if we had this more general mechanism that late-bound parameters could use the same underlying mechanism.
And in case I wasn't clear: to get the late-bound parameter functionality using this syntax, you'd use:
def foo(my_list = `[]`):
That's why I think we should have a larger concept that just late-bound parameters: I think there's a general concept here that can be extended beyond parameters. And that's why I thing not restricting it to a function-definition-only syntax is important: we should produce a syntax that can be used in more places than just functions. This is why I think we need to decide on this larger scope before accepting the narrow function-definition-only syntax: if we decide to add "deferred objects" later, we'd have two ways to specify late-bound parameters [0].
It's larger than argument defaults, but also smaller:
def bisect(a, hi=`len(a)`):
Would this work by your theory? Remember that it must still be
possible to pass a simple number (eg bisect(stuff, 5)), so the default
needs to coalesce to an actual value immediately. The overlap with
late-bound defaults is the simple case of mutable objects that you
want to freshly construct every time, but ultimately, that's not
hugely different from a lambda function:
stuff = defaultdict(lambda: [])
stuff = defaultdict(`[]`)
The only difference might be that you could use isinstance to
distinguish a deferred expression from some other sort of function.
I'm not sure whether that would be significant enough for dataclasses
to take advantage of.
(Another theoretical difference is that a deferred expression is
parsed in the context of its *usage* rather than its *definition*, but
that would break all manner of things in Python and is quite
impractical.)
ChrisA
[0]: Or arguments, I can never remember which is which: someone needs
to invent a memorable mnemonic device.
Pass Arguments
Accept Parameters
(I don't know. I just made that up. For some reason, the fact that
functions have "formal parameters" stuck with me (to me?) from
somewhere, and then I can put the pieces together to figure out where
the arguments must come from. And you can't unsee it now; sorry.)
[0]: Or arguments, I can never remember which is which: someone needs
to invent a memorable mnemonic device.
Pass Arguments
Accept Parameters
(I don't know. I just made that up. For some reason, the fact that
functions have "formal parameters" stuck with me (to me?) from
somewhere, and then I can put the pieces together to figure out where
the arguments must come from. And you can't unsee it now; sorry.)
Formal parameters are the placeholders - the variable names that receive values.
Arguments or "actual parameters" are what got used when the function got called.
An optional parameter can have a default argument which is used when
there is no argument passed. In some languages, a parameter can be
optional without a default, in which case it won't have a value, but
in current Python, defaults and optionality always go together.
Whether 'self' counts as a parameter, and whether the object to the
left of the dot counts as an argument, depends on your point of view.
:)
ChrisA
--
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
That's why I think we should have a larger concept that just late-bound
parameters: I think there's a general concept here that can be extended
beyond parameters.
One advantage of Chris's preferred syntax is that as a default in a
function's parameter list it could do whatever it needs to do there,
while in an executable context, it could return the object, as :=
does. That's a bit inconsistent and quite ugly, I guess, and it might
lead to code like
def foo(x=>[]):
a = bar()
return x
foo(x=>[a])
which I think would be rather confusing. Whether it would be
acceptable is a question of how often it would be used as an
assignment expression rather than as an assignment statement.
I think I've just convinced myself this is a non-starter, but I'll
leave it in as documentation that defining the syntax is a nontrivial
issue.
later, we'd have two ways to specify late-bound parameters [0].
We might want that anyway. One of the aspects of Chris's proposal is
that late-bound parameters get resolved to objects in a well-specified
(if complicated) order at call time, before entering the function
body. In
foo(x=othermodule.bar())
the compiler cannot know that x's value is a "deferred object" (aside:
how about "autoevaluated expression object", presumably with an
abbreviated form such as "autoeval" for normal use). I don't see how
we can guarantee order of evaluation without syntax for late binding
of parameters. I think the predictability is a potentially valuable
aspect of Chris's proposal that might justify special syntax for
late-bound defaults of formal arguments.[1]
[0]: Or arguments, I can never remember which is which: someone needs to
invent a memorable mnemonic device.
C: argv and argc refer to the actual arguments, not the parameter list.
I just use "formal" vs. "actual" to modify "argument".
Footnotes:
[1] I write "potential" not to deprecate it but because I have no
particular use case for late-bound defaults that demands them, rather
than the sentinel idiom. So I have no sense whether this
predictability is a big deal.
That's why I think we should have a larger concept that just late-bound
parameters: I think there's a general concept here that can be extended
beyond parameters.
One advantage of Chris's preferred syntax is that as a default in a
function's parameter list it could do whatever it needs to do there,
while in an executable context, it could return the object, as :=
does. That's a bit inconsistent and quite ugly, I guess, and it might
lead to code like
def foo(x=>[]):
a = bar()
return x
foo(x=>[a])
which I think would be rather confusing. Whether it would be
acceptable is a question of how often it would be used as an
assignment expression rather than as an assignment statement.
I'm not sure what that last line would mean. My proposal doesn't
change the call site in any way, so I'm trying to figure out what you
mean by that call. Are you saying that x=>[a] would be an assignment
statement that sets x to an unevaluated expression? If so, that's
independent of the default argument. If it's a special way to pass
keyword arguments to a function, which passes unevaluated expressions
that are collapsed into values upon usage, then this would be
extremely interesting, but highly bizarre behaviour. It would also
require significant changes to the way that nonlocal names are looked
up (or would be restricted in what nonlocals it can refer to, possibly
none at all).
later, we'd have two ways to specify late-bound parameters [0].
We might want that anyway. One of the aspects of Chris's proposal is
that late-bound parameters get resolved to objects in a well-specified
(if complicated) order at call time, before entering the function
body. In
foo(x=othermodule.bar())
the compiler cannot know that x's value is a "deferred object" (aside:
how about "autoevaluated expression object", presumably with an
abbreviated form such as "autoeval" for normal use). I don't see how
we can guarantee order of evaluation without syntax for late binding
of parameters. I think the predictability is a potentially valuable
aspect of Chris's proposal that might justify special syntax for
late-bound defaults of formal arguments.[1]
Indeed. But here's the question: If a deferred object is to be a
replacement for default_factory, then it must by definition be able to
be stored for later. So it can't be autoevaluated unless there's some
mechanism for delaying the autoevaluation. This seems like an
incredibly messy approach.
The use-cases for deferred evaluation differ based on whether you need
it to be evaluated once or multiple times, whether you want names to
be looked up in the caller's or the callee's context, etc, etc, etc.
What would have overlap with argument defaults isn't the same thing
that would be useful for dataclasses. So there would need to be
multiple variants, or multiple ways to use them.
ChrisA
The overlap with late-bound defaults is the simple case of mutable
objects that you want to freshly construct every time, but
ultimately, that's not hugely different from a lambda function:
Of course it's hugely different from a lambda function. It will be
evaluated at the time of reference, whereas a lambda function will
not, it won't be evaluated until called. (This means that to access a
deferred object without evaluating it, a separate API will be needed,
sort of the dual of function call.)
(Another theoretical difference is that a deferred expression is
parsed in the context of its *usage* rather than its *definition*, but
that would break all manner of things in Python and is quite
impractical.)
I'm a little confused by "theoretical" and "parsed". I guess by
"theoretical" you mean that this is a design choice, and by "parsed in
the context" you mean that the expression could be represented in the
deferred object as a string, an AST, or a code object. Please
confirm.
This isn't about your proposal, it's about more general syntax. Not
everything being discussed is about your proposal, and I suspect one
reason you have trouble figuring out what other people are talking
about is that you persistently try to force everything into that
context.
My proposal doesn't change the call site in any way, so I'm trying
to figure out what you mean by that call. Are you saying that
x=>[a] would be an assignment statement that sets x to an
unevaluated expression?
No, I'm saying it would be an assignment expression (like :=), but
that's what it would do. But that binding would be ignored, and the
value passed into the function in the usual way. A keyword argument
would be set with the even uglier x = (x=>[a]).
It would also require significant changes to the way that nonlocal
names are looked up (or would be restricted in what nonlocals it
can refer to, possibly none at all).
All of which should clue you in that that's probably not what I'm
talking about, especially when I explicitly wrote "while in an
executable context, it could return the object, as := does".
Indeed. But here's the question: If a deferred object is to be a
replacement for default_factory, then it must by definition be able to
be stored for later. So it can't be autoevaluated unless there's some
mechanism for delaying the autoevaluation. This seems like an
incredibly messy approach.
Such an object is not evaluated when created; it's evaluated when
referenced, like a descriptor. Descriptors are messy, too, but
very useful.
Yup. It's an object, those use-cases can "almost certainly" (IMHO,
but I suspect David and Eric agree) be distinguished by attributes set
on the object, or auxiliary APIs for special use-cases where you want
the unevaluated object. I suspect that most folks who want "deferred
objects" haven't really thought about this issue, but have focused on
their immediate applications.
What would have overlap with argument defaults isn't the same thing
that would be useful for dataclasses.
If you say so, but forgive me if I table your comment and wait for
Eric to weigh in on requirements related to dataclasses. In any case,
if the default is such a deferred object x, I'm pretty sure that doing
x=x before entering the function body is equivalent to your proposal.
I see no particular reason why we couldn't have that rule for
"deferred objects" created in "function signature scope".
Steve
The overlap with late-bound defaults is the simple case of mutable
objects that you want to freshly construct every time, but
ultimately, that's not hugely different from a lambda function:
Of course it's hugely different from a lambda function. It will be
evaluated at the time of reference, whereas a lambda function will
not, it won't be evaluated until called. (This means that to access a
deferred object without evaluating it, a separate API will be needed,
sort of the dual of function call.)
So it's a lambda function that gets called the moment you touch it in any way.
(Another theoretical difference is that a deferred expression is
parsed in the context of its *usage* rather than its *definition*, but
that would break all manner of things in Python and is quite
impractical.)
I'm a little confused by "theoretical" and "parsed". I guess by
"theoretical" you mean that this is a design choice, and by "parsed in
the context" you mean that the expression could be represented in the
deferred object as a string, an AST, or a code object. Please
confirm.
What I mean is that I don't know whether you intend it one way or the
other, so I don't know whether it's an actual difference in your
proposal, or something that could in theory be.
If name lookups in these temporary expressions have to refer to names
in the target function, not in their current context, it causes all
kinds of problems. Is that your intention? Otherwise, what is x there?
ChrisA
This isn't about your proposal, it's about more general syntax. Not
everything being discussed is about your proposal, and I suspect one
reason you have trouble figuring out what other people are talking
about is that you persistently try to force everything into that
context.
Yes, it's silly of me to think of everything in a PEP 671 thread as if
it's about argument defaults. Carrying on.
The reason I thought it might be about arg defaults is that you did
also show an arg default in the exact same block. That kinda sullies
the waters a bit. Actually, a lot. If you want to talk about deferred
expressions, can you restrict it to one example rather than two? It's
hard to parse, especially with all the one-letter names.
My proposal doesn't change the call site in any way, so I'm trying
to figure out what you mean by that call. Are you saying that
x=>[a] would be an assignment statement that sets x to an
unevaluated expression?
No, I'm saying it would be an assignment expression (like :=), but
that's what it would do. But that binding would be ignored, and the
value passed into the function in the usual way. A keyword argument
would be set with the even uglier x = (x=>[a]).
uhhh.... I'm lost. Are you saying that "x=>" is a magic token that
doesn't actually assign to x, but it just means that the thing that
follows it is a deferred expression? If so, why not a keyword like
"defer"?
It would also require significant changes to the way that nonlocal
names are looked up (or would be restricted in what nonlocals it
can refer to, possibly none at all).
All of which should clue you in that that's probably not what I'm
talking about, especially when I explicitly wrote "while in an
executable context, it could return the object, as := does".
What's an executable context though? I don't understand.
Indeed. But here's the question: If a deferred object is to be a
replacement for default_factory, then it must by definition be able to
be stored for later. So it can't be autoevaluated unless there's some
mechanism for delaying the autoevaluation. This seems like an
incredibly messy approach.
Such an object is not evaluated when created; it's evaluated when
referenced, like a descriptor. Descriptors are messy, too, but
very useful.
What would have overlap with argument defaults isn't the same thing
that would be useful for dataclasses.
If you say so, but forgive me if I table your comment and wait for
Eric to weigh in on requirements related to dataclasses. In any case,
if the default is such a deferred object x, I'm pretty sure that doing
x=x before entering the function body is equivalent to your proposal.
I see no particular reason why we couldn't have that rule for
"deferred objects" created in "function signature scope".
I *really* don't like that idea. If you want to propose that kind of
thing, where it magically changes when you look at it, then go ahead,
but that has nothing whatsoever to do with PEP 671, as it doesn't
answer the same use-cases, is largely orthogonal, can be better
explained without argument defaults being involved at all, and happens
at the call site rather than the function signature. If I'm
understanding your proposal correctly, using one of these deferreds as
an argument default would look like "def f(x=x=>[]):", which, as well
as being weirdly ugly, wouldn't even be incompatible with the proposal
I'm making.
ChrisA
So [a "deferred object" is] a lambda function that gets called the
moment you touch it in any way.
I'll take that as a question, though you present it like a fact. It's
a code object plus "other stuff" that gets called automatically when
dereferencing a name in most contexts (like a descriptor but works
with "plain" identifiers rather than attribute names). Other cases
I'm not sure about, and I don't know whether they can be resolved in a
way *generally* useful enough to make them worth adding. For example,
let a and b[0] both refer to a "deferred object".
1. The statement x = a will evaluate the contained expression.
2. I don't know when x = b[0] will evaluate the contained expression.
3. I don't know when f(a) will evaluate the contained expression.
4. There will be APIs such as "isdeferred(x)" and "unevalled x" that
do not evaluate the contained expression.
I think my answers are "2. before binding x" and "3. as in PEP 671",
but I'm somewhat unsure of both, since I don't know David's and Eric's
use cases. (FWIW, the alternatives I had in mind were "2. when x is
dereferenced" and "3. when a is dereferenced in the body of f". There
may be others.)
The "other stuff" mentioned above similarly depends on use cases that
I don't know. I suspect that "use cases I don't know" are
characteristic of everyone's reluctance to do a "generic deferred
evaluation" PEP. I get the feeling there are a number of them.
(Another theoretical difference is that a deferred expression is
parsed in the context of its *usage* rather than its *definition*, but
that would break all manner of things in Python and is quite
impractical.)
I'm a little confused by "theoretical" and "parsed".
What I mean is that I don't know whether you intend it one way or the
other, so I don't know whether it's an actual difference in your
proposal, or something that could in theory be.
Do you not realize that "you" is plural, and "proposal" is not unique
at this point? In particular, I was asking about you channelling
Eric. I was not making any statement about my own proposal (which is
starting to gel but until a day or so ago was 100% nonexistent), or
Eric's for that matter.
I still don't understand why time of parsing matters. Do you mean
time of compilation?
If name lookups in these temporary expressions have to refer to
names in the target function, not in their current context, it
causes all kinds of problems.
That's what Ruby blocks do, and Rubyists love them. They certainly
don't think blocks have huge problems.
Alternatively, it would certainly be possible to do as Common Lisp
does, and provide for optional closure over creation-time arguments as
well as access to namespaces active at evaluation time via explicit
APIs. Sure, all this would be complex, and maybe "too complicated to
be a good idea". On the other hand, there may be other use cases that
can take advantage of evaluation of a simple reference to z after
z = defer foo(*args) besides the case of late binding of function
arguments. If those use cases are sufficiently compatible and
compelling, it may be possible to define the implicit APIs and add
explicit APIs as needed in the future.
Is that your intention? Otherwise, what is x there?
I don't understand why you are pushing so hard for these details, when
I doubt any of the advocates of a more general deferred evaluation
facility have communicated with each other yet, and the details
probably vary substantially. And even David has acknowledged that he
doesn't have a worked-out proposal at the moment, only a wisp of a
threat to write one.
It's reasonable to say, as you have, "my proposal is the only one on
the table." That's true (and acknowledged by all who prefer to wait
for a generic deferred evaluation proposal), but existence of a
proposal is a necessary condition for adoption, not sufficient. If
nonexistence of a competing proposal is not the point you're trying to
make, I'm not sure what you're after in this subthread.
Please don't waste your time answering any implicit questions in this
post. If you find them interesting, go right ahead, but I'm not
asking for answers from you, I'm trying to provide answers to your
questions as I understand them. (David, Eric, and any lurkers may
have different answers, though!)
Regards,
Steve
BTW, did you intend for this to be entirely off-list?
Nope, and apologies to all, but at least it's given me the opportunity
to correct a typo & do some slight reformatting. Here's it is:
On Thu, 9 Dec 2021 at 07:25, Adam Johnson wrote:
Here the current behaviour of returning `Ellipsis` is very unfortunate,
and I think could lead to a lot of head scratching — people wondering
why they are getting ellipses in their code, seemingly from nowhere.
Sure, it can be noted in the official documentation that `Ellipsis` is
used as the indicator of late bound defaults, but third-party resources
which aim to explain the uses of `Ellipsis` would (with their current
content) leave someone clueless.
Yes. Unfortunately, since there is fundamentally no object that can be
valid here, this kind of thing WILL happen. So when you see Ellipsis
in a default, you have to do one more check to figure out whether it's
a late-bound default, or an actual early-bound Ellipsis...
My discomfort is that any code that doesn't do that extra check will
continue to function, but incorrectly operate under the assumption that
`Ellipsis` was the actual intended value. I wouldn't go so far as to say
this is outright backwards-incompatible, but perhaps
'backwards-misleading'.
When attempting to inspect a late-bound default I'd much rather an
exception were raised than return value that, as far as any existing
machinery is concerned, could be valid. (More on this thought later...)
Additionally I don't think it's too unreasonable an expectation that,
for a function with no required parameters, either of the following (or
something similar) should be equivalent to calling `func()`:
pos_only_args, kwds = [], {}
for name, param in inspect.signature(func).parameters.items():
if param.default is param.empty:
continue
elif param.kind is param.POSITIONAL_ONLY:
pos_only_args.append(param.default)
else:
kwds[name] = param.default
func(*pos_only_args, **kwds)
# or, by direct access to the dunders
func(*func.__defaults__, **func.__kwdefaults__)
The problem is that then, parameters with late-bound defaults would
look like mandatory parameters. The solution is another check after
seeing if the default is empty:
if param.default is ... and param.extra: continue
In some situations, though, late-bound defaults do essentially become
mandatory. Picking an example you posted yourself (when demonstrating
that not using the functions own context could be surprising):
def g(x=>(a:=1), y=>a): ...
In your implementation `a` is local to `g` and gets bound to `1` when no
argument is supplied for `x` and the default is evaluated, however
**supplying an argument for `x` leaves `a` unbound**. Therefore, unless
`y` is also supplied, the function immediately throws an
`UnboundLocalError` when attempting to get the default for `y`.
With the current implementation it is possible to avoid this issue, but
it's fairly ugly — especially if calculating the value for `a` has side
effects:
def g(
x => (a:=next(it)),
y => locals()['a'] if 'a' in locals() else next(it),
): ...
# or, if `a` is needed within the body of `g`
def g(
x => (a:=next(it)),
y => locals()['a'] if 'a' in locals() else (a:=next(it)),
): ...
The presence of the above if statement's first branch (which was
technically unnecessary, since we established for the purpose of this
example all arguments of `func` are optional / have non-empty defaults)
hints that perhaps `inspect.Parameter` should grow another sentinel
attribute similar to `Parameter.empty` — perhaps `Parameter.late_bound`
— to be set as the `default` attribute of applicable `Parameter`
instances (if not also to be used as the sentinel in `__defaults__` &
`__kwdefaults__`, instead of `Ellipsis`).
Ah, I guess you didn't see .extra then. Currently the only possible
meanings for extra are None and a string, and neither has meaning
unless the default is Ellipsis; it's possible that, in the future,
other alternate defaults will be implemented, which is why I didn't
call it "late_bound". But it has the same functionality.
Correct, I did not initially see `.extra`.
Since the value of `.default` was potentially valid (not _obviously_
wrong, like `Parameter.empty`), there was nothing to prompt me to look
elsewhere.
As above, even though **I** now know `.extra` exists, pre-PEP-671 code
doesn't and will proceed to give misleading values until updated.
Even if the above were implemented, then only way to indicate that the
late bound default should be used would still be by omission of that
argument. Thus, if we combine a late bound default with positional-only
arguments e.g.:
def func(a=>[], b=0, /): ...
It then becomes impossible to programmatically use the given late bound
default for `a` whilst passing a value for `b`. Sure, in this simplistic
case one can manually pass an empty list, but in general — for the same
reasons that it could be "impossible" to evaluate a late bound default
from another context — it would be impossible to manually compute a
replacement value exactly equivalent to the default.
That's already the case. How would you call this function with a value
for b and no value for a?
You're quite right, I couldn't call the function with **no** value for
`a`, but (at present, with early-bound defaults) I can call the function
with the exact object that's used as the default — by pulling it from
`func.__defaults__` (likely directly, if I'm at the REPL — otherwise via
`inspect.signature`).
---
Spending some time thinking about my issues with the current
implementation and your exchanges with Steven D'Aprano regarding using
semi-magical objects within `__defaults__` / `__kwdefaults__` to contain
the code for calculating the defaults, I had an idea about a potential
alternate approach.
As it stands, any object is valid within `__defaults__` /
`__kwdefaults__` and none has intrinsic 'magical' meaning. Therefore,
unless that were to change, there's no valid value you could use
**within** them to indicate a late-bound default — that includes
Steven's use of flagged code objects and your use of `Ellipsis` alike
(again, pre-existing code doesn't know to look at `__defaults_extra__` /
`__kwdefaults_extra__` / `inspect.Parameter.extra` to prove whether
`Ellipsis`, or any other value, is present only as a placeholder).
However, to our advantage, current code also assumes that `__defaults__`
and `__kwdefaults__` are, respectively, a tuple and a dict (or `None`) —
what if that were no longer true in the case of functions with
late-bound defaults?
Instead, one (or both, as appropriate) could be replaced by a callable
with the same parameter list as the main function. Upon calling the main
function, the `__defaults__` / `__kwdefaults__` would automatically be
called (with the same arguments as the function) in order to supply the
default values.
Consequently, existing code designed for handling the collection of
default values as tuple/dict pair would raise an exception when
attempting to iterate or subscript a callable value that was passed
instead. Therefore preventing incorrect conclusions about the default
values from being drawn.
Furthermore, this would make calculated default values become accessible
via manually calling `__defaults__` / `__kwdefaults__`.
This is a (somewhat) basic example to hopefully demonstrate what I'm
thinking:
>>> def func(a => [], b=0, /, *, c=1): ...
...
>>> # callable since default of `a` is late-bound
>>> defaults = func.__defaults__()
>>> defaults
([], 0)
>>>
>>> # only set to a callable when necessary
>>> func.__kwdefaults__
{'c': 1}
>>>
>>> # equivalent to passing only `b`
>>> func(func.__defaults__()[0], b)
A slight wrinkle with this idea is that when late-binding is present
defaults and keyword defaults may be defined interdependently, yet are
normally are stored (and thus accessed) separately — therefore care must
be taken. For example:
Finally (and hopefully not buried by the rest of this message), Chris, a
heads-up that your reference implementation currently has `=>` as
separate `=` & `>` tokens (thus whitespace is valid **between** them) —
i.e. probably not what you intend.
This isn't about your proposal, it's about more general syntax. Not
everything being discussed is about your proposal, and I suspect one
reason you have trouble figuring out what other people are talking
about is that you persistently try to force everything into that
context.
Yes, it's silly of me to think of everything in a PEP 671 thread as if
it's about argument defaults. Carrying on.
Silly, no, I would say "human", but either way I believe it is
impeding *your* understanding, and almost nobody else's.
Of course, the *thread* is generally about argument defaults, but
"everything" in it is not specifically about defaults. In *this*
subthread Eric was arguing for waiting for a facility for *generic*
deferral of expression evaluation, and I was trying (unsuccessfully)
to see if your syntax for defaults could be extended to the more
generic idea.[1] Elsewhere in the thread, you often ask about others'
ideas for such a facility, instead of saying "that's off-topic, you
have my proposal, let's keep discussion strictly to that" or
alternatively, "nobody claims that's more than vaporware, I say now is
better than never, nothing to see here, move on." In that sense, yes,
you can treat everything in this thread as being about argument
defaults by cutting short any other discussion (or just ignoring it).
There's nothing wrong with doing that -- but you did not. Instead you
talk about being confused, not understanding the suggested
alternatives, and you ask about them. In that context, it's on you to
try to channel others' thinking rather than demand that they channel
your confusion.
If you're not in *this* subthread to understand alternative ideas
(again, *there is nothing wrong with ending this subthread here*), I
have nothing further to say in it. If you are, you need to calm down
and start asking questions that specify what you want to know rather
than adding a question mark to a grunt as in
Footnotes:
[1] I think that's important because elsewhere I suggested that
defaults for actual arguments are a sufficiently important use case to
deserve separate syntax from the generic evaluation-deferring syntax
if needed.
This isn't about your proposal, it's about more general syntax. Not
everything being discussed is about your proposal, and I suspect one
reason you have trouble figuring out what other people are talking
about is that you persistently try to force everything into that
context.
Yes, it's silly of me to think of everything in a PEP 671 thread as if
it's about argument defaults. Carrying on.
Silly, no, I would say "human", but either way I believe it is
impeding *your* understanding, and almost nobody else's.
If you're not in *this* subthread to understand alternative ideas
(again, *there is nothing wrong with ending this subthread here*), I
have nothing further to say in it. If you are, you need to calm down
and start asking questions that specify what you want to know rather
than adding a question mark to a grunt as in
By "alternative ideas", do you mean "alternative ways to implement
argument defaults", or "completely different ideas that have
absolutely nothing to do with argument defaults"? Because if it's the
latter, please, change the subject line so it isn't confusing. We can
have all manner of completely independent discussions happening at
once, and there's no problem. But if you mean "alternative ways to
implement argument defaults (and a bunch of other stuff too)", which
is what seemed to be the case when people said that PEP 671 should be
rescinded in favour of a more generic system, is it really wrong of me
to try to think of how this affects PEP 671?
Are you, or are you not, asking me to change or retract PEP 671? Does
your proposal in any way reflect upon argument defaults? Genuine
question. I am utterly, completely, Fblthp-level lost here.
ChrisA
Of course, the *thread* is generally about argument defaults, but
"everything" in it is not specifically about defaults. In *this*
subthread Eric was arguing for waiting for a facility for *generic*
deferral of expression evaluation, and I was trying (unsuccessfully)
to see if your syntax for defaults could be extended to the more
generic idea.[1] Elsewhere in the thread, you often ask about others'
ideas for such a facility, instead of saying "that's off-topic, you
have my proposal, let's keep discussion strictly to that" or
alternatively, "nobody claims that's more than vaporware, I say now is
better than never, nothing to see here, move on." In that sense, yes,
you can treat everything in this thread as being about argument
defaults by cutting short any other discussion (or just ignoring it).
There's nothing wrong with doing that -- but you did not. Instead you
talk about being confused, not understanding the suggested
alternatives, and you ask about them.*In that context, it's on you to try to channel others' thinking
rather than demand that they channel your confusion.*
That makes no sense to me. If people have alternative proposals, it's
up to them to propose them. And to propose them as clearly and
explicitly as possible. (Yes, I understand that writing a fully
detailed spec is not easy or simple.) It's not Chris A's job to try to
clarify what *he thinks they mean*, which is what you appear to be
saying (if not, what *are* you saying by "channel others' thinking"?).
Best wishes
Rob Cliffe
By "alternative ideas", do you mean "alternative ways to implement
argument defaults",
As you just phrased it, yes. I do not know how I can make that more
obvious.
If you didn't mean what you wrote so generally, but really mean
"concrete, ready for copy-and-paste into PEP 671", then no.[1] And I
don't know how to make that more obvious, either.
Until you understand that, I think this subthread is a dead letter.
Footnotes:
[1] Steven d'Aprano has a proposal that as I understand it mitigates
most of my concrete reservations, but I worry that it would not be
generalizable if and when a generic facility for representing and
handling evaluation-deferred expressions is added.
By "alternative ideas", do you mean "alternative ways to implement
argument defaults",
As you just phrased it, yes. I do not know how I can make that more
obvious.
So if you ARE talking about argument defaults, then how is it wrong of
me to try to interpret them in the context of argument defaults? Still
confused.
ChrisA
That makes no sense to me. If people have alternative proposals, it's
up to them to propose them. And to propose them as clearly and
explicitly as possible. (Yes, I understand that writing a fully
detailed spec is not easy or simple.) It's not Chris A's job to try to
clarify what *he thinks they mean*, which is what you appear to be
saying (if not, what *are* you saying by "channel others' thinking"?).
It is the PEP author's job to clearly, accurately and fairly discuss any
rejected ideas:
https://www.python.org/dev/peps/pep-0001/#what-belongs-in-a-successful-pep
Obviously there is a lot of wiggle-room here, and it is ultimately
subjective. And the PEP doesn't need to make the case for the
alternatives. But it should mention them, and note that they are
rejected, and give reasons why they were rejected.
Objective reasons are stronger than subjective, but sometimes the only
reason is "because I don't like it".
It seems to me that Chris is determined to push forward with his ideas
for default expressions, regardless of the mostly-negative feedback on
Python-Ideas. I'm also unsure whether he has tried to get any feedback
on other forums, such as the Python mailing list, Discuss, Reddit, etc.
This is his perogative, of course. Ultimately the PEP author is
proposing his vision for the language, and if others fail to convince
him that another vision is better, he should push forward with his ideas
as they stand, and hope the Steering Council will see things his way.
According to my count, the discussion for this PEP has now reached 610
messages over various threads. The most prolific poster is Chris
himself, with over 34% of the posts (209); followed by me with 12% (75
posts including this one).
There have been 48 people who have commented, with a mean of 12.7
posts per person and median of 2.5. 13 people have posted 10 times or
more, and 15 people have posted just once.
I think that this shows that:
(1) this discussion on Python-Ideas is dominated by a few people;
(2) nevertheless, there has been a moderately large number of people who
cared enough to post;
(3) if we haven't reached a consensus after 600+ posts and two months,
we probably aren't going to.
Unless Chris wants to try to attract some fresh blood into the
discussion by taking it to another forum (say, Reddit's r/python, or
Discuss) to see if there are any new ideas and/or consensus, my
personal feeling is that this thread is now bogged in the mud and making
no progress. Those who like it like it, those who don't don't.
--
Steve
It seems to me that Chris is determined to push forward with his ideas
for default expressions, regardless of the mostly-negative feedback on
Python-Ideas. I'm also unsure whether he has tried to get any feedback
on other forums, such as the Python mailing list, Discuss, Reddit, etc.
That's largely because I am having extreme difficulty pinning down
what the other ideas are, whether they are even alternates to what I'm
proposing, whether I'm wrong for even thinking that the post was about
default argument expressions, and I don't even know what else.
You're right, this thread is bogged down. I'm abandoning it for the
time being. Maybe some day in the future, it'll be worth reopening
this discussion.
For now, I don't think we're getting anywhere.
ChrisA
If people have alternative proposals, it's up to them to propose
them.
I don't understand why you focus on proposals that don't exist yet.
Both alternatives that I can imagine have been proposed.
One alternative proposal is Steven d'Aprano's "put the thunk in an
object, execute it in the same place, and the rest is PEP 671" idea.
The other is "keep the status quo." Both are viable. "Keep the
status quo" is the usual outcome for the first proposal to address an
issue via new syntax.
All the rest is rationale for supporting one proposal or another.
It's not Chris A's job to try to clarify what *he thinks they mean*
I don't think it's an obligation on Chris, I think it's to his
advantage. See Paul Moore's post on why Chris should at least include
a list of reservations in PEP 671, and then consider whether it would
help Chris's case if he understands them well enough to dispose of
them efficiently if the SC asks.
You seem to think that if there's no alternative that addresses the
issue we all see, the PEP should be adopted by default. That simply
isn't the case in Python -- do nothing is the default. The Loyal
Opposition is not trying to crush the PEP (we can't, anyway), we are
trying to explain why we don't like it. And it will make the PEP more
likely to succeed if Chris can address any reservations the SC asks
about succinctly and effectively.
Cheers,
Steve
This may be a poor choice of time to wade into PEP 671 discussion,
since the consensus seems to be that the thread has exhausted its
usefulness. But there is one specific aspect of the discussion which
(it seems to me, though I may have missed some of the hundreds of
emails) has gotten inadequate discussion, and deserves better
treatment in the PEP itself.
On Wed, Dec 8, 2021 at 10:02 PM Chris Angelico wrote:
(F) Concerns that functions using late-bound defaults were harder to wrap.
...
(F) Also a matter of opinion, given that *a,**kw is the most common
wrapping technique used, and will work reliably. Function signature
algebra is a much larger challenge than this.
I don't think this is a fair dismissal of the concern. Taken broadly,
function "wrapping" is extremely common, in the sense that what many
(most?) functions do is call other functions, and there is a wide
spectrum of "wrapping" from "pure wrapper that does not change the
signature at all" through "non-trivial wrapper that has a different
signature but requires some of the same arguments with the same
semantics" all the way to "not a wrapper at all because it uses the
called function for a very small portion of what it does and shares no
signature with it."
In any case where the same argument with the same semantics needs to
be passed through multiple layers of a function call chain (and again,
my experience is this is quite common in real world code; I can
collect some data on this if anyone finds this assertion
unconvincing), non-trivial argument defaults are painful. One is faced
with two unappealing options: either duplicate the non-trivial default
(in which case you have code duplication and more places to update on
any change), or give up entirely on introspectable/legible signatures
and use `*args, **kwargs`. I don't think it is true in real code that
the latter is "the most common form of wrapping," and more importantly
I think it is a poor last resort that we should not encourage. For a
PEP that largely stakes its value proposition on "function signatures
should be more useful," it seems pretty strange to promote `*args,
**kwargs` (which makes the signature entirely useless to both humans
and tooling) as an acceptable solution to the wrapping problem.
This is already a problem with early-bound defaults. I think that it
is a big enough problem to consider non-trivial argument defaults an
anti-pattern in general, and to consider sentinels in place of
non-trivial defaults to be preferable API design, rather than a hack
or workaround.
But late-bound defaults make the problem significantly worse, by
encouraging more non-trivial argument defaults to be stuffed into
signatures rather than calculated in the body, including some that
can't be duplicated at all by a wrapper, therefore leaving `*args,
**kwargs` the only option.
So for me, any example where PEP 671 allows a non-trivial default
(including e.g. `len(a)`) to be stuffed into the signature directly is
a strike against PEP 671, not a use case in favor of it.
This leaves the only value proposition of PEP 671 as simple mutable
defaults like `=>[]` and `=>{}`. These values are trivial enough that
it's not a problem to just duplicate them in wrapping cases, and I'd
have no problem in principle enabling their use as argument defaults;
they are effectively parallel to a default of `None` or `0` or `-1`.
And I agree that the inability to do this with early-bound defaults is
an unfortunate wart. But I'm not sure PEP 671 provides enough value
here to justify its complexity. If we could make `=[]` and `={}`
somehow "do what newbies expect" while preserving clear, predictable
semantics, that would have value. But requiring it to be `=>[]` means
that a newbie still has to first understand that `[]` and `{}` are
special somehow and different syntax is needed to use them as
defaults. It seems like this understanding is the largest barrier;
once it has been crossed, is it really that much worse to do `x=None`
in the signature and `x = x or []` in the body? (And perhaps improve
this a bit further with PEP 505.)
Carl
Thank you for this, Carl. This is an excellent point, and well
articulated. I agree it's important.
I also agree that this should be addressed in the PEP. If nothing else,
it needs to be listed in a "Discussion" section, or similar. I think
pointing to your email would be a good start.
Eric
On 12/12/2021 2:07 PM, Carl Meyer wrote:
This may be a poor choice of time to wade into PEP 671 discussion,
since the consensus seems to be that the thread has exhausted its
usefulness. But there is one specific aspect of the discussion which
(it seems to me, though I may have missed some of the hundreds of
emails) has gotten inadequate discussion, and deserves better
treatment in the PEP itself.
On Wed, Dec 8, 2021 at 10:02 PM Chris Angelico wrote:
Objections to PEP 671 - Summary
...
(F) Concerns that functions using late-bound defaults were harder to wrap.
...
(F) Also a matter of opinion, given that *a,**kw is the most common
wrapping technique used, and will work reliably. Function signature
algebra is a much larger challenge than this.
I don't think this is a fair dismissal of the concern. Taken broadly,
function "wrapping" is extremely common, in the sense that what many
(most?) functions do is call other functions, and there is a wide
spectrum of "wrapping" from "pure wrapper that does not change the
signature at all" through "non-trivial wrapper that has a different
signature but requires some of the same arguments with the same
semantics" all the way to "not a wrapper at all because it uses the
called function for a very small portion of what it does and shares no
signature with it."
In any case where the same argument with the same semantics needs to
be passed through multiple layers of a function call chain (and again,
my experience is this is quite common in real world code; I can
collect some data on this if anyone finds this assertion
unconvincing), non-trivial argument defaults are painful. One is faced
with two unappealing options: either duplicate the non-trivial default
(in which case you have code duplication and more places to update on
any change), or give up entirely on introspectable/legible signatures
and use `*args, **kwargs`. I don't think it is true in real code that
the latter is "the most common form of wrapping," and more importantly
I think it is a poor last resort that we should not encourage. For a
PEP that largely stakes its value proposition on "function signatures
should be more useful," it seems pretty strange to promote `*args,
**kwargs` (which makes the signature entirely useless to both humans
and tooling) as an acceptable solution to the wrapping problem.
This is already a problem with early-bound defaults. I think that it
is a big enough problem to consider non-trivial argument defaults an
anti-pattern in general, and to consider sentinels in place of
non-trivial defaults to be preferable API design, rather than a hack
or workaround.
But late-bound defaults make the problem significantly worse, by
encouraging more non-trivial argument defaults to be stuffed into
signatures rather than calculated in the body, including some that
can't be duplicated at all by a wrapper, therefore leaving `*args,
**kwargs` the only option.
So for me, any example where PEP 671 allows a non-trivial default
(including e.g. `len(a)`) to be stuffed into the signature directly is
a strike against PEP 671, not a use case in favor of it.
This leaves the only value proposition of PEP 671 as simple mutable
defaults like `=>[]` and `=>{}`. These values are trivial enough that
it's not a problem to just duplicate them in wrapping cases, and I'd
have no problem in principle enabling their use as argument defaults;
they are effectively parallel to a default of `None` or `0` or `-1`.
And I agree that the inability to do this with early-bound defaults is
an unfortunate wart. But I'm not sure PEP 671 provides enough value
here to justify its complexity. If we could make `=[]` and `={}`
somehow "do what newbies expect" while preserving clear, predictable
semantics, that would have value. But requiring it to be `=>[]` means
that a newbie still has to first understand that `[]` and `{}` are
special somehow and different syntax is needed to use them as
defaults. It seems like this understanding is the largest barrier;
once it has been crossed, is it really that much worse to do `x=None`
in the signature and `x = x or []` in the body? (And perhaps improve
this a bit further with PEP 505.)
I don't think this is a fair dismissal of the concern. Taken broadly,
function "wrapping" is extremely common, in the sense that what many
(most?) functions do is call other functions, and there is a wide
spectrum of "wrapping" from "pure wrapper that does not change the
signature at all" through "non-trivial wrapper that has a different
signature but requires some of the same arguments with the same
semantics" all the way to "not a wrapper at all because it uses the
called function for a very small portion of what it does and shares no
signature with it."
In any case where the same argument with the same semantics needs to
be passed through multiple layers of a function call chain (and again,
my experience is this is quite common in real world code; I can
collect some data on this if anyone finds this assertion
unconvincing), non-trivial argument defaults are painful. One is faced
with two unappealing options: either duplicate the non-trivial default
(in which case you have code duplication and more places to update on
any change), or give up entirely on introspectable/legible signatures
and use `*args, **kwargs`.
I think you have identified a real pain point, function wrapping, but it
is a much bigger pain point that just function defaults, and it won't be
made appreciably worse by late-bound defaults.
Over the years, I have written about this pain point a number of times.
It is much more significant than just defaults: it is hits parameter
naming, and order, and annotations.
The problem is that we have no good notation to say that one function
inherits its signature (possibly with changes) from another function.
This is especially noticable during rapid experimental development,
where the signature of functions might be changing rapidly, but it
occurs in stable code too. Even if the signature is changing, the
wrapper nevertheless has to be aware of the signature, and duplicate it.
The idea that this problem is unique, or especially acute, for function
defaults is false. It is a problem across the board:
- if the *order of parameters* changes, the wrapping function must
also change, unless it exclusively uses keyword arguments;
- if the *parameter names* change, the wrapping function likewise
must also change, unless it exclusively uses positional arguments;
- if the *parameter type declarations* change, the wrapping function
should also change, lest the reader get confused, and static or
runtime type checkers flag the code as wrong.
Or to put it more succinctly:
The wrapper must be aware of the wrapped signature in order to duplicate
it. Function signatures are code, and this is a violation of DRY.
If anything, the "default argument" problem is the *least* issue here,
not the most, because the wrapper can, at least sometimes, just omit the
parameter with a default.
The only solution to this right now is to use `*args, **kwargs`.
If that hurts introspection, then that demonstrates a weakness in our
introspection tools that needs to be fixed.
If you try to reduce the problem by removing defaults, or annotations,
or only using keyword arguments, or only using positional arguments, not
only does it not solve the problem, but the solution is worse than the
problem being solved.
But if we focus only on one tiny little corner of the problem space,
complex defaults (whether early or late), there is one other possible
mitigation that is out of scope for this PEP but perhaps we could
consider it. Namely a Javascript-like undefined value that the
interpreter can use as an explicit signal to "treat this as a missing
argument and use the default".
But undefined has its own problems to, and its not clear to me that this
would be any more of a solution to the tight coupling between wrapper
and wrapped functions problem than any of the other non-solutions are.
In any case, coming back to this PEP:
- late defaults do not make the wrapper problem appreciably worse;
- the wrapper problem has something of a solution, `*args, **kwargs`,
but our tooling needs to be better at handling that;
- that tooling weakness is independent of this PEP and should be fixed
regardless of this PEP;
- arguably, the wrapper problem may reduce the scope for people to use
late defaults, since they might prefer to use None or some other
sentinel, but it doesn't eliminate it.
--
Steve
I don't think this is a fair dismissal of the concern. Taken broadly,
function "wrapping" is extremely common, in the sense that what many
(most?) functions do is call other functions, and there is a wide
spectrum of "wrapping" from "pure wrapper that does not change the
signature at all" through "non-trivial wrapper that has a different
signature but requires some of the same arguments with the same
semantics" all the way to "not a wrapper at all because it uses the
called function for a very small portion of what it does and shares no
signature with it."
In any case where the same argument with the same semantics needs to
be passed through multiple layers of a function call chain (and again,
my experience is this is quite common in real world code; I can
collect some data on this if anyone finds this assertion
unconvincing), non-trivial argument defaults are painful. One is faced
with two unappealing options: either duplicate the non-trivial default
(in which case you have code duplication and more places to update on
any change), or give up entirely on introspectable/legible signatures
and use `*args, **kwargs`.
I think you have identified a real pain point, function wrapping, but it
is a much bigger pain point that just function defaults, and it won't be
made appreciably worse by late-bound defaults.
Over the years, I have written about this pain point a number of times.
It is much more significant than just defaults: it is hits parameter
naming, and order, and annotations.
The problem is that we have no good notation to say that one function
inherits its signature (possibly with changes) from another function.
This is especially noticable during rapid experimental development,
where the signature of functions might be changing rapidly, but it
occurs in stable code too. Even if the signature is changing, the
wrapper nevertheless has to be aware of the signature, and duplicate it.
The idea that this problem is unique, or especially acute, for function
defaults is false. It is a problem across the board:
- if the *order of parameters* changes, the wrapping function must
also change, unless it exclusively uses keyword arguments;
- if the *parameter names* change, the wrapping function likewise
must also change, unless it exclusively uses positional arguments;
- if the *parameter type declarations* change, the wrapping function
should also change, lest the reader get confused, and static or
runtime type checkers flag the code as wrong.
Or to put it more succinctly:
The wrapper must be aware of the wrapped signature in order to duplicate
it. Function signatures are code, and this is a violation of DRY.
If anything, the "default argument" problem is the *least* issue here,
not the most, because the wrapper can, at least sometimes, just omit the
parameter with a default.
The only solution to this right now is to use `*args, **kwargs`.
If that hurts introspection, then that demonstrates a weakness in our
introspection tools that needs to be fixed.
If you try to reduce the problem by removing defaults, or annotations,
or only using keyword arguments, or only using positional arguments, not
only does it not solve the problem, but the solution is worse than the
problem being solved.
But if we focus only on one tiny little corner of the problem space,
complex defaults (whether early or late), there is one other possible
mitigation that is out of scope for this PEP but perhaps we could
consider it. Namely a Javascript-like undefined value that the
interpreter can use as an explicit signal to "treat this as a missing
argument and use the default".
But undefined has its own problems to, and its not clear to me that this
would be any more of a solution to the tight coupling between wrapper
and wrapped functions problem than any of the other non-solutions are.
[snip]
Hmm.
What about something like this as a bit of syntax:
def my_decorator(f):
@wraps
def wrapper(from my_decorator):
return f(from my_decorator)
return wrapper
The idea is that in a function's parameter list it would pick up the
signature and in a function call it would pick up the arguments.
The idea is that in a function's parameter list it would pick up the
signature and in a function call it would pick up the arguments.
In the simple case, that's already done. Just use *a,**kw, and the
wraps(f) decorator will mark it as "use the signature of that function
over there".
Unfortunately, that's only the simple case. You can't say "use the
signature of that function, but I also accept this arg". You can't say
"use the signature of that function, except that I don't accept this
arg". But it would be theoretically possible to do exactly that using
basically the inspect module, and it would be an extremely useful
feature for many situations, especially if it can be automated in some
way (eg if it notices "def f(spam, *a, **kw)" and adds spam to the
parameter list).
ChrisA
I think you have identified a real pain point, function wrapping, but it
is a much bigger pain point that just function defaults, and it won't be
made appreciably worse by late-bound defaults.
Unfortunately I don't think either of those things is true.
Over the years, I have written about this pain point a number of times.
It is much more significant than just defaults: it is hits parameter
naming, and order, and annotations.
The problem is that we have no good notation to say that one function
inherits its signature (possibly with changes) from another function.
This is especially noticable during rapid experimental development,
where the signature of functions might be changing rapidly, but it
occurs in stable code too. Even if the signature is changing, the
wrapper nevertheless has to be aware of the signature, and duplicate it.
The idea that this problem is unique, or especially acute, for function
defaults is false. It is a problem across the board:
- if the *order of parameters* changes, the wrapping function must
also change, unless it exclusively uses keyword arguments;
- if the *parameter names* change, the wrapping function likewise
must also change, unless it exclusively uses positional arguments;
- if the *parameter type declarations* change, the wrapping function
should also change, lest the reader get confused, and static or
runtime type checkers flag the code as wrong.
Or to put it more succinctly:
The wrapper must be aware of the wrapped signature in order to duplicate
it. Function signatures are code, and this is a violation of DRY.
I think in fact the default-arguments problem is significantly _more_
pervasive and significant than the other issues you've identified. One
reason is the "spectrum of function wrapping" I mentioned earlier.
Parameter order and often names are only relevant for wrappers that
are attempting to exactly (or near-exactly) imitate the signature of
the wrapped function, e.g. so they can be directly substituted for it.
But there is a much broader set of functions (in the "middle" of the
spectrum) that are not attempting to exactly mirror the signature of
any called function, and yet must thread some particular defaulted
argument through to it. Perhaps these should not be called "wrappers,"
and we should distinguish the broader "defaulted-argument-threading
problem" from the narrower "function wrapping" problem.
(I think also argument names are a much lesser problem since a)
renaming things is a common operation in refactoring code, and many
tools already have good support for it and b) failure to rename one
argument when you rename the other leads to a discrepancy in
signature, but not a bug in behavior. Forgetting to change one
non-trivial default value when you change another one can easily lead
to silent bugs. Similarly forgetting to change a matching type will
easily be caught by a type checker, rather than silently leading to
wrong behavior.)
The only solution to this right now is to use `*args, **kwargs`.
In fact, sticking to simple default argument values is an easy and
very effective solution to the defaulted-argument-threading problem,
and valuable in many cases where no other part of the function
wrapping problem is relevant.
If that hurts introspection, then that demonstrates a weakness in our
introspection tools that needs to be fixed.
No objection to hypothetical future improvements in introspection
tools! But it is nevertheless the case that today a `*args, **kwargs`
signature is much inferior for almost any use, in any tooling I've
ever seen. Even in a hypothetical future with better tooling it will
always remain less legible to a reader of the source code without
tooling assistance, and there will always be plenty such readers of
any code. So it remains contradictory for a PEP that claims to improve
function signatures to recommend the use of the least legible form of
signature available.
[Skipping discussion of undefined. It's an interesting proposal with
its own set of pros and cons, but I think it's too far off topic from
PEP 671 and I don't want to digress.]
- late defaults do not make the wrapper problem appreciably worse;
I don't think this is true. Aside from the question of whether they
encourage more use of complex defaults (numerous examples already
given in favor of the PEP certainly suggest so!), late defaults make
the problem qualitatively worse by introducing a new category of
complex defaults that may be entirely impossible for a "wrapping"
function to duplicate, even if they would prefer duplication over the
other options. (The wrapping function could probably substitute a
sentinel in these cases and then _also_ duplicate the logic of the
complex default in its body, but it's not a great argument for the
feature that its use will force related APIs back to sentinels, plus
still needing duplicated logic, whereas if the function had just used
a sentinel instead of the complex late-bound default there would be no
duplication and no signature inconsistency.)
- arguably, the wrapper problem may reduce the scope for people to use
late defaults, since they might prefer to use None or some other
sentinel, but it doesn't eliminate it.
Of course people can still choose to use sentinels even if late
defaults are an option. But that's not the relevant question. We add
features and complexity to the language only if on the whole they will
tend to improve code, not just because "well, this feature won't force
people to make their code any worse." If sentinels are in general a
better option than complex defaults, then complex defaults should not
be used as an example of the benefits of the PEP, and we should focus
on whether it actually solves the `[]` and `{}` problem, which (unlike
complex defaults being calculated inside the function body) is an
actual problem. Unfortunately I don't think the PEP solves that
problem effectively either, because after the PEP `=[]` wil continue
to be every bit as much the newbie foot-gun that silently does an
unexpected thing as it is today.
Carl
I stumbled upon PEP 671 again today, and for what it's worth I fully agree with everything said here.
For the same reasons as you listed, I am generally opposed to PEP 671. Wrapping functions in one way or another is extremely common and this PEP will make a problem which is currently super small much bigger - the inability to have a function's defaults apply without messing with `*args` and `**kwargs`.
The reason this probably isn't brought up a lot is because of the existence of None. A wrapping function can set its default to None matching the wrapped function's default. If the wrapped function were to use PEP 671 this would cause the wrapping function to need to go to dynamic `*args` and `**kwargs` to match this behaviour.
JavaScript users can use `undefined` for this; if you pass `undefined` to a parameter in JavaScript, the default will be applied. I think that Python should have a similar mechanic (but that's another discussion) which would need to be added before PEP 671. Together that adds up to being years away (because of dependencies minimum Python requirement, etc) and at that point I would rather see the current status quo being upheld and PEP 671 deferred.
Tl;Dr PEP 671 makes things worse without other additions
I stumbled upon PEP 671 again today, and for what it's worth I fully agree with everything said here.
For the same reasons as you listed, I am generally opposed to PEP 671. Wrapping functions in one way or another is extremely common and this PEP will make a problem which is currently super small much bigger - the inability to have a function's defaults apply without messing with `*args` and `**kwargs`.
The reason this probably isn't brought up a lot is because of the existence of None. A wrapping function can set its default to None matching the wrapped function's default. If the wrapped function were to use PEP 671 this would cause the wrapping function to need to go to dynamic `*args` and `**kwargs` to match this behaviour.
JavaScript users can use `undefined` for this; if you pass `undefined` to a parameter in JavaScript, the default will be applied. I think that Python should have a similar mechanic (but that's another discussion) which would need to be added before PEP 671. Together that adds up to being years away (because of dependencies minimum Python requirement, etc) and at that point I would rather see the current status quo being upheld and PEP 671 deferred.
Tl;Dr PEP 671 makes things worse without other additions
Then propose your own change that fixes everything! PEP 671 basically
died because the exact same arguments kept getting rehashed. Instead
of just hating on my proposal, *make your own*. I don't have exclusive
rights to suggest changes to the way argument passing is done.
Figure out the semantics you're trying to suggest, decide on a syntax,
and put a proposal forward. Yes, it's more work than just one email
saying that PEP 671 sucks, but it would also be a lot more useful.
ChrisA
I couldn't figure out the best place in the reply tree to post this, so replying to the OP, answering the questions, taking into account other discussion that has happened.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
No, but I feel there is some cognitive burden with the distinction between that and other arrow notations that we have now and will likely have later.
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
Technically this is not applicable since I would use it anyway, but…
I would slightly prefer any one of the alternative syntaxes. At first, I was not liking the '@' prefix idea because the '@' is separated from the default expression that it is conceptually associated with. That option does have a strong redeeming aspect though, which is that I think it might be the easiest to read.
5) Do you know how to compile CPython from source, and would you be
willing to try this out? Please? :)
Sure. I don't think I need to try it to know that I would appreciate it though, unless I were to find that it is buggy or something.
One thing was not clear to me from the current PEP 671 text.
When that is used in a method, what is the closure for the expressions? Would/should assignments in the class definition be available or only global variables in the module and local variables in the function (if applicable) in which the class definition happens?
One thing was not clear to me from the current PEP 671 text.
When that is used in a method, what is the closure for the expressions? Would/should assignments in the class definition be available or only global variables in the module and local variables in the function (if applicable) in which the class definition happens?
It's exactly the same as if the code got executed inside the
function's body. It has access to the function's locals, but not to
class scope (other than through cls.X or self.X).
ChrisA
Ah and since previous parameters can be referenced, and `self` or `cls` is the first argument to any method, that is always available to default value expressions. Correct?
Ah and since previous parameters can be referenced, and `self` or `cls` is the first argument to any method, that is always available to default value expressions. Correct?
To clarify my statement about readability of the '@' prefix option…
I think that its meaning is less clear if one doesn't already know what the syntax means. I think the code would be easier to skim, however, using that option after one does know its meaning.
My favorite options are '@' or '?=' (tied), followed by ':=' followed by '=>'.
I stumbled upon PEP 671 again today, and for what it's worth I fully agree
with everything said here.
For the same reasons as you listed, I am generally opposed to PEP 671.
Wrapping functions in one way or another is extremely common and this PEP
will make a problem which is currently super small much bigger - the
inability to have a function's defaults apply without messing with `*args`
and `**kwargs`.
The reason this probably isn't brought up a lot is because of the
existence of None. A wrapping function can set its default to None matching
the wrapped function's default. If the wrapped function were to use PEP 671
this would cause the wrapping function to need to go to dynamic `*args` and
`**kwargs` to match this behaviour.
JavaScript users can use `undefined` for this; if you pass `undefined` to
a parameter in JavaScript, the default will be applied. I think that Python
should have a similar mechanic (but that's another discussion) which would
need to be added before PEP 671. Together that adds up to being years away
(because of dependencies minimum Python requirement, etc) and at that point
I would rather see the current status quo being upheld and PEP 671 deferred.
Tl;Dr PEP 671 makes things worse without other additions
This has been proposed many times. You can check the mailing list history.
Such proposals have been even less popular then PEP 671, since it requires
a new keyword, which is generally avoided at nearly all costs, and requires
it either be restricted to only being used in defs, or will just end up
like None where people are passing it as arguments, which defeats the
purpose.
You may not like PEP-671, but it at least provides a feasible solution.
Using a new special parameter is never going to fly.
I used to prefer `:=` but coming back to this topic after a long
interval I am happy with `=>` and perhaps I even like it more, Chris.😁
The PEP status is "Draft". What are the chances of something happening
any time soon, i.e. the PEP being considered by the Steering Committee?
Or is it still too controversial, or is there some other obstacle?
In case it's not clear, I support the PEP. I think it plugs an obvious gap.
Best wishes
Rob Cliffe
On 13/06/2022 09:38, Steve Jorgensen wrote:
To clarify my statement about readability of the '@' prefix option…
I think that its meaning is less clear if one doesn't already know what the syntax means. I think the code would be easier to skim, however, using that option after one does know its meaning.
I used to prefer `:=` but coming back to this topic after a long
interval I am happy with `=>` and perhaps I even like it more, Chris.😁
The PEP status is "Draft". What are the chances of something happening
any time soon, i.e. the PEP being considered by the Steering Committee?
Or is it still too controversial, or is there some other obstacle?
In case it's not clear, I support the PEP. I think it plugs an obvious gap.
The main obstacle is that I got weary of responding to emails on it,
put it on the back burner, and let the topic go cold.
To revive it, what I'd recommend is: Go through the PEP and figure out
everything that *you* would need to see done before *you* would
approve it, supposing that you were a member of the SC. Then either
suggest wording changes, or at least list your concerns, and we can
reopen the discussion.
ChrisA
Could this be the behaviour of passing in an Ellipsis? e.g.
def foo(defaults_to_one=1):
return defaults_to_one
assert foo(...) == foo()
def bar(something=...):
return foo(something)
assert bar() == foo()
def baz(arg): # no defaults
return arg
assert baz(...) == ...
The only place that I am aware of the Ellipsis being used is in index notation (numpy).
So this would have likely an impact on __getitem__ or the slice object.
*Alternatively* a subtype of Ellipses specifically for when used in argument defaults DefaultEllipsis (or equivalent):
def foo(x=...):
return x
assert isinstance(foo(), EllipsisType)
assert foo() != Ellipsis
assert isinstance(foo(...), EllipsisType)
assert foo(...) == Ellipsis
It isn't clear to me whether your question is a request for clarification
(does the PEP mean this...?) or a request for a change in behaviour
(could you change the PEP to do this...?).
Why would you want to type `foo(...)` when you could just type `foo()`?
The only place that I am aware of the Ellipsis being used is in index notation (numpy).
So this would have likely an impact on __getitem__ or the slice object.
Ellipsis has been around for over twenty years so we have to assume it
would have an impact on thousands of programs. We don't just care about
famous, popular libraries like numpy, we care about breaking little
scripts used by one person too.
--
Steve
The only place that I am aware of the Ellipsis being used is in index notation (numpy).
So this would have likely an impact on __getitem__ or the slice object.
*Alternatively* a subtype of Ellipses specifically for when used in argument defaults DefaultEllipsis (or equivalent):
def foo(x=...):
return x
No, because Ellipsis is a very real object and could be passed as a
parameter at any time. Although, since it's unusual, it's actually a
slightly deoptimized case in my reference implementation - or rather,
whenever a function observes that it's been passed Ellipsis, it checks
to see if the parameter was actually omitted.
Fundamentally, Python does not have any value that indicates the lack
of a value. And having worked with a number of JavaScript APIs where
undefined is used as "absent", I don't want to inflict that on anyone.
(For instance, classList.toggle("some-class", X) will add the class if
X is truthy, remove it if it's falsy, but... toggle if X is undefined.
Which is otherwise falsy. Extremely annoying to try to track down.)
The standard way in Python to indicate "no value" is to raise an
exception instead of returning, but that doesn't work for parameters.
You can use sequence unpacking to achieve that sort of effect, though.
foo(*([] if x is ... else [x]))
This will skip the parameter if x is Ellipsis. Obviously, you use this
only when you know you'll never use Ellipsis as an actual value,
though, so this can't be done in the language, but it could be useful.
ChrisA
This has been proposed many times. You can check the mailing list history.
Such proposals have been even less popular then PEP 671, since it requires
a new keyword, which is generally avoided at nearly all costs,
Now that Python is using a PEG parser, adding a soft keyword is no big
deal. We could use a named keyword:
def spam(arg = defer default_expression):
pass
without affecting code that used "defer" as a variable or function name.
We could even write:
def spam(defer = defer defer()): ...
where the same word "defer" refers to a parameter, a soft keyword, and a
function call, all in the same function signature. Needless to say one
should not make a habit of this. But it would be allowed.
--
Steve
I used to prefer `:=` but coming back to this topic after a long
interval I am happy with `=>` and perhaps I even like it more, Chris.😁
The PEP status is "Draft". What are the chances of something happening
any time soon, i.e. the PEP being considered by the Steering Committee?
There's no Sponsor, so it isn't being considered by the SC. That much is
objectively true.
Beyond that, the following is all my personal opinion, and should not be
taken as definitive or official in any way. Importantly, I have *not*
read back through the entire thread to refresh my memory. However, I
have re-read the PEP in detail.
There's no consensus that this feature is worth the added complexity, or
even what the semantics are. The PEP punts on the semantics, saying that
the behaviour may vary across implementations.
There's no consensus on the syntax, which may not matter, the Steering
Council can make the final decision if necessary. But with at least four
options in the PEP it would be good to narrow it down a bit. No soft
keywords have been considered.
In my opinion, there are weaknesses in the PEP:
- lack of any reference to previous discussions;
- no attempt to gather feedback from other forums;
- no review of languages that offer choice of early or late binding;
- little attempt to justify why this is better than the status quo; the
PEP seems to take the position that it is self-evident that Python
needs this feature, rather than being a balanced document setting out
both pros and cons;
- little or no attempt in the PEP to answer objections;
- examples are all chosen to show the feature in the best possible
light, rather than to show both the good and bad; (e.g. no examples
show the parameter with annotations)
- failure to acknowledge that at least one of the suggested syntaxes
is visually ambiguous with existing syntax.
E.g. this would be legal with the PEP's second choice of spelling:
def func(spam, eggs:=(x:=spam)):
Even if the parser can distinguish the two uses of `:=` there, its
awfully cryptic. In and of itself, that's not necessarily a fatal flaw
(e.g. slicing) but the benefits have to outweigh the negatives, and the
PEP should be a balanced discussion of both.
--
Steve
As well as all the matters Steven raises, I continue to dislike the
proposal for the same reason I did on earlier rounds. I believe a general
"deferred computation" mechanism is useful, but that one limited to the
context of function parameters does more harm than good is scoped narrowly
to that single use. I keyword version might bridge that gap by introducing
"later" or "defer" or "delay" in a narrow context, but not foreclosing its
later use more broadly.
On Wed, Jun 15, 2022 at 8:38 AM Steven D'Aprano wrote:
I used to prefer `:=` but coming back to this topic after a long
interval I am happy with `=>` and perhaps I even like it more, Chris.😁
The PEP status is "Draft". What are the chances of something happening
any time soon, i.e. the PEP being considered by the Steering Committee?
There's no Sponsor, so it isn't being considered by the SC. That much is
objectively true.
Beyond that, the following is all my personal opinion, and should not be
taken as definitive or official in any way. Importantly, I have *not*
read back through the entire thread to refresh my memory. However, I
have re-read the PEP in detail.
There's no consensus that this feature is worth the added complexity, or
even what the semantics are. The PEP punts on the semantics, saying that
the behaviour may vary across implementations.
There's no consensus on the syntax, which may not matter, the Steering
Council can make the final decision if necessary. But with at least four
options in the PEP it would be good to narrow it down a bit. No soft
keywords have been considered.
In my opinion, there are weaknesses in the PEP:
- lack of any reference to previous discussions;
- no attempt to gather feedback from other forums;
- no review of languages that offer choice of early or late binding;
- little attempt to justify why this is better than the status quo; the
PEP seems to take the position that it is self-evident that Python
needs this feature, rather than being a balanced document setting out
both pros and cons;
- little or no attempt in the PEP to answer objections;
- examples are all chosen to show the feature in the best possible
light, rather than to show both the good and bad; (e.g. no examples
show the parameter with annotations)
- failure to acknowledge that at least one of the suggested syntaxes
is visually ambiguous with existing syntax.
E.g. this would be legal with the PEP's second choice of spelling:
def func(spam, eggs:=(x:=spam)):
Even if the parser can distinguish the two uses of `:=` there, its
awfully cryptic. In and of itself, that's not necessarily a fatal flaw
(e.g. slicing) but the benefits have to outweigh the negatives, and the
PEP should be a balanced discussion of both.
--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
Please. This has been many times by several people already. No-one is
going to change their mind on this by now. There's no point in
rehashing it and adding noise to the thread.
Best wishes
Rob Cliffe
On 15/06/2022 13:43, David Mertz, Ph.D. wrote:
As well as all the matters Steven raises, I continue to dislike the
proposal for the same reason I did on earlier rounds. I believe a
general "deferred computation" mechanism is useful, but that one
limited to the context of function parameters does more harm than good
is scoped narrowly to that single use. I keyword version might bridge
that gap by introducing "later" or "defer" or "delay" in a narrow
context, but not foreclosing its later use more broadly.
On Wed, Jun 15, 2022 at 8:38 AM Steven D'Aprano
wrote:
On Tue, Jun 14, 2022 at 11:59:44AM +0100, Rob Cliffe via
Python-ideas wrote:
> I used to prefer `:=` but coming back to this topic after a long
> interval I am happy with `=>` and perhaps I even like it more,
Chris.😁
> The PEP status is "Draft". What are the chances of something
happening
> any time soon, i.e. the PEP being considered by the Steering
Committee?
There's no Sponsor, so it isn't being considered by the SC. That
much is
objectively true.
Beyond that, the following is all my personal opinion, and should
not be
taken as definitive or official in any way. Importantly, I have *not*
read back through the entire thread to refresh my memory. However, I
have re-read the PEP in detail.
There's no consensus that this feature is worth the added
complexity, or
even what the semantics are. The PEP punts on the semantics,
saying that
the behaviour may vary across implementations.
There's no consensus on the syntax, which may not matter, the
Steering
Council can make the final decision if necessary. But with at
least four
options in the PEP it would be good to narrow it down a bit. No soft
keywords have been considered.
In my opinion, there are weaknesses in the PEP:
- lack of any reference to previous discussions;
- no attempt to gather feedback from other forums;
- no review of languages that offer choice of early or late binding;
- little attempt to justify why this is better than the status
quo; the
PEP seems to take the position that it is self-evident that Python
needs this feature, rather than being a balanced document
setting out
both pros and cons;
- little or no attempt in the PEP to answer objections;
- examples are all chosen to show the feature in the best possible
light, rather than to show both the good and bad; (e.g. no examples
show the parameter with annotations)
- failure to acknowledge that at least one of the suggested syntaxes
is visually ambiguous with existing syntax.
E.g. this would be legal with the PEP's second choice of spelling:
def func(spam, eggs:=(x:=spam)):
Even if the parser can distinguish the two uses of `:=` there, its
awfully cryptic. In and of itself, that's not necessarily a fatal
flaw
(e.g. slicing) but the benefits have to outweigh the negatives,
and the
PEP should be a balanced discussion of both.
--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
Please. This has been many times by several people already. No-one is going to change their mind on this by now. There's no point in rehashing it and adding noise to the thread.
To be fair, the only real point in re-opening the discussion at all is
to determine if anyone has changed their mind. That said, IMO it's
unlikely that enough time has passed for that to have happened, so
it's unlikely that anything productive will come from this new thread.
Paul
There's no consensus that this feature is worth the added complexity, or
even what the semantics are. The PEP punts on the semantics, saying that
the behaviour may vary across implementations.
Excuse me? I left one or two things open-ended, where they're bad code
and I'm not going to lock the language into supporting them just
because the reference implementation happens to be able to, but
"punts"? That's a bit much. The semantics are QUITE specific.
There's no consensus on the syntax, which may not matter, the Steering
Council can make the final decision if necessary. But with at least four
options in the PEP it would be good to narrow it down a bit. No soft
keywords have been considered.
"""Choice of spelling. While this document specifies a single syntax
`name=>expression`..."""
The PEP specifies *one* option.
- no attempt to gather feedback from other forums;
- no review of languages that offer choice of early or late binding;
- little attempt to justify why this is better than the status quo; the
PEP seems to take the position that it is self-evident that Python
needs this feature, rather than being a balanced document setting out
both pros and cons;
- little or no attempt in the PEP to answer objections;
- examples are all chosen to show the feature in the best possible
light, rather than to show both the good and bad; (e.g. no examples
show the parameter with annotations)
- failure to acknowledge that at least one of the suggested syntaxes
is visually ambiguous with existing syntax.
E.g. this would be legal with the PEP's second choice of spelling:
def func(spam, eggs:=(x:=spam)):
Even if the parser can distinguish the two uses of `:=` there, its
awfully cryptic. In and of itself, that's not necessarily a fatal flaw
(e.g. slicing) but the benefits have to outweigh the negatives, and the
PEP should be a balanced discussion of both.
And that's not the recommended syntax anyway.
Please, if you're going to criticize the document, *at least* have the
decency to check your facts.
You are the PRIMARY reason that I gave up pushing this earlier. I am
within a very short span of blocking you in my mail client and
refusing to respond to your emails, because it feels like a massive
waste of my time trying.
ChrisA
Please. This has been many times by several people already. No-one is
going to change their mind on this by now. There's no point in
rehashing it and adding noise to the thread.
Rob, there's no rule that only "people who support this PEP" are allowed
to comment. If it is okay for you to say you like this PEP even more now
than previously, it is okay for David to say that his opinion hasn't
changed. Especially since David even pointed out one potential change
which might lead him to support the PEP, or at least shift to "neutral".
--
Steve
There's no consensus that this feature is worth the added complexity, or
even what the semantics are. The PEP punts on the semantics, saying that
the behaviour may vary across implementations.
Excuse me? I left one or two things open-ended, where they're bad code
and I'm not going to lock the language into supporting them just
because the reference implementation happens to be able to, but
"punts"? That's a bit much. The semantics are QUITE specific.
Under the Specification section, the PEP explicitly refers to
behaviour which "may fail, may succeed", and different behaviour which
is "Highly likely to give an error", and states "Using names of later
arguments should not be relied upon, and while this MAY work in some
Python implementations, it should be considered dubious".
So, yes, the PEP *punts* on the semantics of the feature, explicitly
leaving the specification implementation-dependent.
There's no consensus on the syntax, which may not matter, the Steering
Council can make the final decision if necessary. But with at least four
options in the PEP it would be good to narrow it down a bit. No soft
keywords have been considered.
"""Choice of spelling. While this document specifies a single syntax
`name=>expression`..."""
The PEP specifies *one* option.
The part of the sentence you replaced with an ellipsis says "alternate
spellings are similarly plausible." The very next sentence says
"Open for consideration are the following"
and a couple of paragraphs later you even explicitly refer to a
second proof of concept implemention.
The PEP has a preferred syntax, as is its right, but it lists three
alternatives still under consideration.
--
Steve
There's no consensus that this feature is worth the added complexity, or
even what the semantics are. The PEP punts on the semantics, saying that
the behaviour may vary across implementations.
Excuse me? I left one or two things open-ended, where they're bad code
and I'm not going to lock the language into supporting them just
because the reference implementation happens to be able to, but
"punts"? That's a bit much. The semantics are QUITE specific.
Under the Specification section, the PEP explicitly refers to
behaviour which "may fail, may succeed", and different behaviour which
is "Highly likely to give an error", and states "Using names of later
arguments should not be relied upon, and while this MAY work in some
Python implementations, it should be considered dubious".
So, yes, the PEP *punts* on the semantics of the feature, explicitly
leaving the specification implementation-dependent.
One very very specific aspect of it is left undefined. Are you really
bothered by that? I don't understand how you can dare to write a
single line of code, given how many other things are actually not
specified. Have you ever written a __del__ method? Python *does not
guarantee* when it will be called. Wow! Python is hopelessly
implementation-dependent, there's no WAY this should ever be used!
And then you take this tiny point where I left it open to implementers
to choose, and you say that the PEP "punts on the semantics" as if the
entire specification is in doubt.
I'm trying as hard as I can to believe that you're arguing in good
faith, but it's getting less and less plausible.
ChrisA
Is there anything that I can do, as a random Python user to help move this to the next stage? I'm happy to go along with whatever the preponderance of responses here seem to think in terms of which syntax choice is best. Although I have a slight preference, all of the options seem decent to me.
I am definitely in favor of having the PEP accepted and implemented.
I've been scolded that I'm not allowed to post unless I support the PEP.
Nonetheless, I reiterate that I oppose it. There is no "preponderance" of
support, but perhaps a slim majority of the small number who have commented
(5 vs 3, I think).
On Thu, Jun 16, 2022, 10:38 PM Steve Jorgensen wrote:
Is there anything that I can do, as a random Python user to help move this
to the next stage? I'm happy to go along with whatever the preponderance of
responses here seem to think in terms of which syntax choice is best.
Although I have a slight preference, all of the options seem decent to me.
Nonetheless, I reiterate that I oppose it. There is no "preponderance"
of support, but perhaps a slim majority of the small number who have
commented (5 vs 3, I think).
On Thu, Jun 16, 2022, 10:38 PM Steve Jorgensen
wrote:
Is there anything that I can do, as a random Python user to help
move this to the next stage? I'm happy to go along with whatever
the preponderance of responses here seem to think in terms of
which syntax choice is best. Although I have a slight preference,
all of the options seem decent to me.
Please. This has been many times by several people already. No-one is
going to change their mind on this by now. There's no point in
rehashing it and adding noise to the thread.
Rob, there's no rule that only "people who support this PEP" are allowed
to comment. If it is okay for you to say you like this PEP even more now
On Wed, Jun 15, 2022 at 01:58:28PM +0100, Rob Cliffe via Python-ideas wrote:
than previously, it is okay for David to say that his opinion hasn't
changed.
Of course I wasn't saying that some people aren't allowed to comment.
What I was objecting to was the rehashing of old arguments which have
already been done to death. I refrained, at the time, from offering
counter-arguments, because those have been done to death too.
Especially since David even pointed out one potential change
which might lead him to support the PEP, or at least shift to "neutral".
You are quite right Steven, and perhaps I was too hasty, didn't pay
enough attention to David's suggestion, and failed to address it. Well,
let me address it now. I am afraid I can't do so without repeating some
of my old arguments and thus doing exactly what I asked David not to do,
but he has made a suggestion and it merits being answered. Thank you
for contributing to this thread, David.
"I believe a general "deferred computation" mechanism is useful, but
that one limited to the context of function parameters does more harm
than good is scoped narrowly to that single use. I keyword version
might bridge that gap by introducing "later" or "defer" or "delay" in a
narrow context, but not foreclosing its later use more broadly." [DM]
The bar for adding a new hard keyword to Python is very high.
The suggestion is to add a new keyword to a PEP which absolutely doesn't
need it, on the grounds that it **might** (**not** would - we can't know
without a spec) give compatibility with some fictional vapourware which
- for all people keep talking about it - hasn't happened in years, isn't
happening (AFAIK nobody is working on it), doesn't have anything
remotely close to even an outline specification (people disagree as to
what it should do), very likely never will happen, and at best won't
happen for years.
And if it does, what's the **worst** that could happen? We would have
two ways of writing the same thing! Terrible! Well, we already have
that in effect (yes, I know they generate different compiled code):
x = 42
(x := 42)
{}
dict() # similarly for list, tuple
and I expect there are a host of other examples I can't bring to mind at
the moment,
and the sky hasn't fallen.
I'm sorry, but if that's what it takes to make David or anyone else less
opposed to the PEP, it ain't worth it. Not remotely close.
So It could be a soft keyword, you say? OK, AFAICS that could work.
But let me note in passing that if we consider 4 possible syntaxes:
def func(later parm = default): # Number 1
def func(parm later = default): # Number 2
def func(parm = later default): # Number 3
def func(parm = default later): # Number 4
Number 3, which some might find the most attractive, won't work. Consider:
def f(x = later -y):
Is that a late-bound default of -y? Bad luck; it's already legal syntax
for an early-bound default of `later` minus `y`.
AFAICS the others are possible.
If you have some other syntax in mind, David, please specify.
But (and AFAIK this has not been said before) let's think about this
supposed "consistency". Let's suppose we have a bright-and-shiny,
up-and-running delayed evaluation scheme. I imagine (and again, if this
is not what you have in mind, David, please specify) it would allow you
to write something like
x = expression later
which would assign to x some entity, a "deferred evaluation object"
(DEO), probably containing a code object, which would allow the
expression to be evaluated at some future time (and possibly in some
different scope, but let's not go there now). On the principle of least
surprise, one would want "expression later" to be a Python expression,
and to behave as far as possible in the same way in different contexts.
So you could write stuff like
x = y = expression later
someTuple = (expression1 later, expression2 later)
and so on.
(It's not relevant to my point, but I can't imagine wanting syntax like
Number 1 or Number 2 above:
later x = expression # value of expression to be calculated and
assigned to x at some future time
What would these mean, if legal:
later x = y = expression
x = later y = expression
later x = later y = expression
).
But when I use that same syntax *in a function signature*:
def f(x = expression later):
and then call f(), I do **not** (in most cases) want the default value
of x to be a DEO. I want it evaluated NOW, at function call time. So
where is the consistency? There isn't any. It's a myth. Of course, you
could have a rule that 'later' behaved differently when applied to
function parameter defaults, but that would just be adding confusion by
using the same syntax in two different ways. Making two different
things look the same. In short, late-bound defaults and deferred
evaluation are **not** the same thing (nor is one a subset of the other):
Late-bound defaults are meant to be evaluated at function call
time (and in particular, not some way down in the function body when the
parameter gets used).
DEOs are meant to be evaluated ... well, I don't know when, but
presumably there would be a mechanism (possibly implicit) for saying
"please evaluate this now".
And there's readability. Compare these two syntaxes:
def foo(bar=>[], baz=>timenow(), bam=>inc(baz)):
def for(bar=[] later, baz=timenow() later, bam=inc(baz) later):
Not only is the second one longer and repetitious, it adds more
**words**. People are used to distinguishing between words and symbols
and do it easily. In the first example, all the words are parameter
names or part of their default values. Cluttering the signature with
more words makes it harder, visually, to pick out what is what.
Case rests.
Best wishes
Rob Cliffe
Is there anything that I can do, as a random Python user to help move
this to the next stage?
If you think the PEP is as complete and persuasive as possible right
now, you can offer moral support and encouragement. Or you can suggest
some improvements, and see whether the PEP author agrees.
It is up to the PEP author to decide whether, in his opinion, the PEP is
sufficiently complete to move forward, or whether it needs more work.
Other options include leaving it deferred/incomplete, to withdraw it, or
solicit for somebody to take it over.
If the PEP author abandons it, you could ask to take it over, or you
could write your own competing PEP as an alternative.
If the author decides to move forward, he needs to ask for a core dev to
sponsor it. Assuming he gets one, that will start the next round of
debate, followed by a request to the Steering Council to make a decision
whether to accept it as is, demand some changes, or reject it.
--
Steve
Under the Specification section, the PEP explicitly refers to
behaviour which "may fail, may succeed", and different behaviour which
is "Highly likely to give an error", and states "Using names of later
arguments should not be relied upon, and while this MAY work in some
Python implementations, it should be considered dubious".
So, yes, the PEP *punts* on the semantics of the feature, explicitly
leaving the specification implementation-dependent.
One very very specific aspect of it is left undefined. Are you really
bothered by that?
Yes.
This is not just some minor, trivial implementation issue, it cuts right
to the core of this feature's semantics:
* Which arguments can a late-bound parameter access?
* When the late-bound default is evaluated, what is the name resolution
rule? (Which variables from which scopes will be seen?)
These are fundamental details related to the meaning of code, not relatively
minor details such as the timing of when a destructor will run.
If we have:
```
items = ['spam', 'eggs']
def frob(n=>len(items), items=[]):
print(n)
```
we cannot even tell whether `frob()` will print 0 or 2 or raise an
exception.
I described this underspecification as a weakness of the PEP. As I said
at the time, that was my opinion. As the PEP author, of course it is
your perogative to leave the semantics of this feature underspecified,
hoping that the Steering Council will be happy with implementation-
dependent semantics.
For the benefit of other people reading this, in case it isn't clear,
let me try to explain what the issue is.
When late-bound defaults are simulated with the `is None` trick, we
write:
```
def frob(n=None, items=[]):
# If we enter the body of the function,
# items is guaranteed to have a value.
if n is None:
n = len(items)
print(n)
```
and there is never any doubt about the scoping rules for `len(items)`.
It always refers to the parameter `items`, never to the variable in the
surrounding scope, and because that parameter is guaranteed to be bound
to a value, so the simulated default `len(items)` cannot fail with
NameError. We can reason about the code's meaning very easily.
If we want "real" late-bound defaults to match that behaviour,
`n=>len(items)` must evaluate `len(items)` *after* items is bound
to a value, even though items occurs to the right of n.
Under the PEP though, this behaviour is underspecified. The PEP
describes this case as implementation dependent. Any of the following
behaviours would be legal when `frob()` is called:
* n=>len(items) evaluates the parameter `items`, *after* it gets
bound to the default of [], and so n=0 (that is, it has the same
semantics as the status quo);
* n=>len(items) evaluates the parameter `items`, but it isn't bound
to a value yet (because `items` occurs to the right of n), and so
evaluating the default raises (presumably) UnboundLocalError;
* n=>len(items) evaluates the variable items from the surrounding scope,
and so evaluates to n=2; if no such variable exists, it will presumably
raise NameError.
With the behaviour unspecified, we can't predict whether the above
frob() example is legal or what it will do if it is. It could vary not
only between CPython and other Pythons, but from one version of CPython
and another.
--
Steve
Under the Specification section, the PEP explicitly refers to
behaviour which "may fail, may succeed", and different behaviour which
is "Highly likely to give an error", and states "Using names of later
arguments should not be relied upon, and while this MAY work in some
Python implementations, it should be considered dubious".
So, yes, the PEP *punts* on the semantics of the feature, explicitly
leaving the specification implementation-dependent.
One very very specific aspect of it is left undefined. Are you really
bothered by that?
Yes.
This is not just some minor, trivial implementation issue, it cuts right
to the core of this feature's semantics:
* Which arguments can a late-bound parameter access?
Definitely those that come before it. Potentially some of those that
come after it, but I'm not mandating that.
* When the late-bound default is evaluated, what is the name resolution
rule? (Which variables from which scopes will be seen?)
Exactly the same as any other code executed in the function. I really
don't see why this is so surprising to you; it is *exactly the normal
behaviour of Python code*.
These are fundamental details related to the meaning of code, not relatively
minor details such as the timing of when a destructor will run.
They're exactly as fundamental, actually. The 'with' statement was
added to the language specifically because it was critically important
to clean things up in ways that CPython does, but Python does not
guarantee.
In my reference implementation, more is available than the PEP
guarantees. That's all.
I described this underspecification as a weakness of the PEP. As I said
at the time, that was my opinion. As the PEP author, of course it is
your perogative to leave the semantics of this feature underspecified,
hoping that the Steering Council will be happy with implementation-
dependent semantics.
It's a lot less implementation-dependent than you seem to think.
For the benefit of other people reading this, in case it isn't clear,
let me try to explain what the issue is.
When late-bound defaults are simulated with the `is None` trick, we
write:
```
def frob(n=None, items=[]):
# If we enter the body of the function,
# items is guaranteed to have a value.
if n is None:
n = len(items)
print(n)
```
and there is never any doubt about the scoping rules for `len(items)`.
It always refers to the parameter `items`, never to the variable in the
surrounding scope, and because that parameter is guaranteed to be bound
to a value, so the simulated default `len(items)` cannot fail with
NameError. We can reason about the code's meaning very easily.
This is exactly the same. I don't understand what's confusing here.
If we want "real" late-bound defaults to match that behaviour,
`n=>len(items)` must evaluate `len(items)` *after* items is bound
to a value, even though items occurs to the right of n.
Yes, and the ONLY part that is underspecified is the order. It is
perfectly legal for it to be implemented in either of these ways:
def frob():
n = len(items)
items = []
def frob():
items = []
n = len(items)
And I would consider code that depends on specifically one or the
other to be bad code, just like code that depends on __del__ methods
being called.
What wording could I add to the PEP to make this more obvious?
Under the PEP though, this behaviour is underspecified. The PEP
describes this case as implementation dependent. Any of the following
behaviours would be legal when `frob()` is called:
* n=>len(items) evaluates the parameter `items`, *after* it gets
bound to the default of [], and so n=0 (that is, it has the same
semantics as the status quo);
* n=>len(items) evaluates the parameter `items`, but it isn't bound
to a value yet (because `items` occurs to the right of n), and so
evaluating the default raises (presumably) UnboundLocalError;
* n=>len(items) evaluates the variable items from the surrounding scope,
and so evaluates to n=2; if no such variable exists, it will presumably
raise NameError.
No, this makes no sense whatsoever. In Python, a parameter is
(effectively) assigned to within the function, and therefore *any*
reference to it *must* refer to the local, not to any surrounding
scope. Late-bound defaults do not change this fundamental.
With the behaviour unspecified, we can't predict whether the above
frob() example is legal or what it will do if it is. It could vary not
only between CPython and other Pythons, but from one version of CPython
and another.
That is correct. This issue ONLY happens if a late-bound default
refers to an early-bound argument that comes to the right of it in the
argument list, and the ONLY possible results are UnboundLocalError and
getting the value.
Please, provide some better wording for the PEP, something that would
have convinced you that this is the case. I am getting very tired of
you not reading my posts, and continuing to argue from ignorance.
ChrisA
If we have:
```
items = ['spam', 'eggs']
def frob(n=>len(items), items=[]):
print(n)
```
we cannot even tell whether `frob()` will print 0 or 2 or raise an
exception.
It will either print 0 or raise UnboundLocalError. There is no
circumstance in which it will legally print 2.
Under the PEP though, this behaviour is underspecified. The PEP
describes this case as implementation dependent. Any of the following
behaviours would be legal when `frob()` is called:
* n=>len(items) evaluates the parameter `items`, *after* it gets
bound to the default of [], and so n=0 (that is, it has the same
semantics as the status quo);
* n=>len(items) evaluates the parameter `items`, but it isn't bound
to a value yet (because `items` occurs to the right of n), and so
evaluating the default raises (presumably) UnboundLocalError;
* n=>len(items) evaluates the variable items from the surrounding scope,
and so evaluates to n=2; if no such variable exists, it will presumably
raise NameError.
No, this makes no sense whatsoever. In Python, a parameter is
(effectively) assigned to within the function, and therefore *any*
reference to it *must* refer to the local, not to any surrounding
scope. Late-bound defaults do not change this fundamental.
I understand this is unambiguous, but it is nonetheless potentially
confusing: normal, immediate-evaluation arguments do, of course, have
access to enclosing scope, and so one might be led to believe that this
is still possible.
With the behaviour unspecified, we can't predict whether the above
frob() example is legal or what it will do if it is. It could vary not
only between CPython and other Pythons, but from one version of CPython
and another.
That is correct. This issue ONLY happens if a late-bound default
refers to an early-bound argument that comes to the right of it in the
argument list, and the ONLY possible results are UnboundLocalError and
getting the value.
Is there a *reason* why you are leaving this unspecified? To put it more
baldly, is there any reason (e.g., difficulty of parsing?) why allowing
these "forward" references should *not* be allowed? It seems that
"n=>len(items), items=[]" might be an important use case.
Andrew
def for(bar=[] later, baz=timenow() later, bam=inc(baz) later):
Not only is the second one longer and repetitious, it adds more **words**.
People are used to distinguishing between words and symbols and do it
easily.
You have just articulated why I find type annotations so hard to read.
Back OT: add annotations to the examples, and it gets worse. I like it or
not, I suspect annotations are becoming “standard”
-CHB
--
Christopher Barker, PhD (Chris)
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
Is there a *reason* why you are leaving this unspecified? To put it more
baldly, is there any reason (e.g., difficulty of parsing?) why allowing
these "forward" references should *not* be allowed? It seems that
"n=>len(items), items=[]" might be an important use case.
Am I right in thinking the key issue here is that => is *not* used for
"items"? So
def frob(n=>len(items), items=[]):
print(n)
items.append(1)
gets very complicated to reason about. What does this print?
frob()
frob()
frob(items=[1,2,3,4,5])
frob(3, [])
frob()
frob(3)
frob()
Even if someone *can* provide an answer, I'd be reluctant to accept
that any answer could be described as "intuitive". And "well, don't do
that" is just ducking the question - in essentially the same way as
"it's implementation defined" does...
Paul
If we have:
```
items = ['spam', 'eggs']
def frob(n=>len(items), items=[]):
print(n)
```
we cannot even tell whether `frob()` will print 0 or 2 or raise an
exception.
It will either print 0 or raise UnboundLocalError. There is no
circumstance in which it will legally print 2.
Under the PEP though, this behaviour is underspecified. The PEP
describes this case as implementation dependent. Any of the following
behaviours would be legal when `frob()` is called:
* n=>len(items) evaluates the parameter `items`, *after* it gets
bound to the default of [], and so n=0 (that is, it has the same
semantics as the status quo);
* n=>len(items) evaluates the parameter `items`, but it isn't bound
to a value yet (because `items` occurs to the right of n), and so
evaluating the default raises (presumably) UnboundLocalError;
* n=>len(items) evaluates the variable items from the surrounding scope,
and so evaluates to n=2; if no such variable exists, it will presumably
raise NameError.
No, this makes no sense whatsoever. In Python, a parameter is
(effectively) assigned to within the function, and therefore *any*
reference to it *must* refer to the local, not to any surrounding
scope. Late-bound defaults do not change this fundamental.
I understand this is unambiguous, but it is nonetheless potentially
confusing: normal, immediate-evaluation arguments do, of course, have
access to enclosing scope, and so one might be led to believe that this
is still possible.
That's because immediate-evaluation is like this:
_default = SOME_EXPRESSION
def func(n=None):
if n is None: n = _default
Whereas late evaluation is like this:
def func(n=None):
if n is None: n = SOME_EXPRESSION
Unfortunately, there's not going to be any way to resolve this. The
entire point of this feature is to be able to do things that can't be
done with early evaluation, and that includes referring to other
arguments, so it fundamentally has to be done in the function's scope.
Maybe it would have been convenient for Python to define that function
defaults are *always* evaluated in the function's scope, but (a) that
ship has well and truly sailed, and (b) I'm not sure that that would
be better anyway - the current behaviour lets you do an easy snapshot
by writing something like "i=i", so it has the same name on the inside
that it has on the outside.
For the most part, it's not a problem; scopes are nested, so you can
happily refer to a name in an enclosing scope. Exceptions include
names that are shadowed, class-level names (but you'll usually be able
to write "self.X" or "cls.X" so it's just a difference of spelling),
and possibly some quirks of closures, although most of those are a
consequence of timing rather than scoping.
With the behaviour unspecified, we can't predict whether the above
frob() example is legal or what it will do if it is. It could vary not
only between CPython and other Pythons, but from one version of CPython
and another.
That is correct. This issue ONLY happens if a late-bound default
refers to an early-bound argument that comes to the right of it in the
argument list, and the ONLY possible results are UnboundLocalError and
getting the value.
Is there a *reason* why you are leaving this unspecified? To put it more
baldly, is there any reason (e.g., difficulty of parsing?) why allowing
these "forward" references should *not* be allowed? It seems that
"n=>len(items), items=[]" might be an important use case.
Yes. The alternative is that I make it
"reference-implementation-defined", and I've seen so much of that that
I don't want to lock that in. Just because the way I happen to have
implemented it allows for the late-bound defaults to refer to
early-bound arguments to their right, I don't want to lock the
language into behaving that way forever; conversely, I don't want to
have a non-compliant reference implementation based on a definition of
"all arguments are assigned left to right", which is much cleaner and
simpler to describe, but a lot harder to implement. I want the
language to be open to the cleaner definition, while permitting the
"arguments are assigned left to right in two stages" implementation as
well.
That's the only distinction though. And if you simply place all
late-bound defaults to the right of all early-bound defaults, there
won't be any problem, ever.
ChrisA
Is there a *reason* why you are leaving this unspecified? To put it more
baldly, is there any reason (e.g., difficulty of parsing?) why allowing
these "forward" references should *not* be allowed? It seems that
"n=>len(items), items=[]" might be an important use case.
Am I right in thinking the key issue here is that => is *not* used for
"items"? So
gets very complicated to reason about. What does this print?
There are several ways to make this clearly sane.
# Clearly UnboundLocalError
def frob(n=>len(items), items=>[]):
# Clearly correct behaviour
def frob(items=[], n=>len(items)):
def frob(items=>[], n=>len(items)):
The only way for it to be confusing is to have => on one argument and
then = on a subsequent argument, *and* to have the earlier one refer
to the later one.
Even if someone *can* provide an answer, I'd be reluctant to accept
that any answer could be described as "intuitive". And "well, don't do
that" is just ducking the question - in essentially the same way as
"it's implementation defined" does...
But "don't do that" is a perfectly reasonable response to other kinds
of bad code, like messing up your spacing:
x = 1+2 * 3+4
Is this intuitive? Some people will think that x should be 21, but the
actual answer is 11. Python won't stop you from doing this, but style
guides absolutely should.
In the same way, I would strongly recommend that style guides frown
upon referring to arguments later in the parameter list, even if it
happens to be legal. I'm just not mandating that the language check
for this and artificially block it.
ChrisA
For comparison, please look at the PEP for the statistics module. Steve
wrote both PEP and the standard library module. In my opinion the PEP is
deficient in its description of core semantics, and I don't see a reference
implementation prior to acceptance.
https://peps.python.org/pep-0450/
Of course, that was nearly 10 years ago. The recently discussed problems
with type conversion in the statistics module together with what's missing
in PEP 450 together support Steve's request that the PEP discussed in this
thread be improved, so that we better avoid future problems.
--
Jonathan
Maybe... I'm not sure I see this as *that* much more obvious, although
I concede that the left-to-right evaluation rule implies it (it feels
like a mathematician's use of "obvious" - which quite often isn't ;-))
Using assignment expressions in argument defaults is well-defined but
not necessarily obvious in a similar way (to me, at least).
The only way for it to be confusing is to have => on one argument and
then = on a subsequent argument, *and* to have the earlier one refer
to the later one.
For you, maybe. I assert that the forms above *are* confusing for me.
You're welcome to explain them to me, like you have, and maybe I'll
now remember the logic for the future, but as a data point, I stand by
my statement that these were confusing to me when I encountered them
fresh. Feel free to state that there's not *enough* cases of people
being confused by the semantics to outweigh the benefits, but it feels
to me that there are a few people claiming confusion here, and simply
saying "you shouldn't be confused, it's obvious" isn't really
addressing the point.
Even if someone *can* provide an answer, I'd be reluctant to accept
that any answer could be described as "intuitive". And "well, don't do
that" is just ducking the question - in essentially the same way as
"it's implementation defined" does...
But "don't do that" is a perfectly reasonable response to other kinds
of bad code, like messing up your spacing:
x = 1+2 * 3+4
Is this intuitive? Some people will think that x should be 21, but the
actual answer is 11. Python won't stop you from doing this, but style
guides absolutely should.
But that's not the same as you leaving the behaviour implementation
defined. In the case of operator precedence, there *is* a well-defined
answer, but the spacing doesn't match that interpretation. But in the
case of
frob(n=>len(items), items=())
you're refusing to give a well-defined semantics, and then saying that
people shouldn't do that. But unlike spacing of expressions, the order
of arguments is *important* - it is part of the API of frob that the
first positional argument is n, so "just swap the arguments" is a
semantic change. So how should people get the ("obvious") intended
behaviour? Abandon the new syntax and go back to using None as a
default? That seems a shame, given that (as I understand it) your
reference implementation works exactly as I'd want.
In the same way, I would strongly recommend that style guides frown
upon referring to arguments later in the parameter list, even if it
happens to be legal. I'm just not mandating that the language check
for this and artificially block it.
You're not *just* recommending this for style guides, you're also
explicitly stating that you refuse to assign semantics to it.
Anyway, all of this is just my opinion. I'm not trying to persuade you
that you're wrong, just to point out that others see things
differently. It's up to you what you do with that information. Change
the PEP or don't, put it back into deferred status or submit it. I'm
not the decision maker here, just a community member whose feedback
will (hopefully) be considered by the SC when making the decision if
the PEP comes to them.
Paul
Um, I didn't see that as any more obvious than the original example. I
guess I can see it's UnboundLocalError, but honestly that's not
obvious to me.
Question: Is this obvious?
def f():
x, x[0] = [2], 3
print(x)
def boom():
x[0], x = 3, [2]
# raises UnboundLocalError
I understand that left-to-right evaluation is something that has to be
learned (and isn't 100% true - operator precedence is a thing too),
but at very least, if it isn't *obvious*, it should at least be
*unsurprising* if you then get UnboundLocalError.
Maybe... I'm not sure I see this as *that* much more obvious, although
I concede that the left-to-right evaluation rule implies it (it feels
like a mathematician's use of "obvious" - which quite often isn't ;-))
Using assignment expressions in argument defaults is well-defined but
not necessarily obvious in a similar way (to me, at least).
When you say "assignment expressions", do you mean "default
expressions", or are you referring to the walrus operator? There's a
lot of other potentially-surprising behaviour if you mix assignment
expressions in with this, because of the difference of scope. It's the
sort of thing that can definitely be figured out, but I would advise
against it.
def frob(items=>[], n=>len(items:=[])):
This will reassign items to be an empty list if n is omitted.
Obviously that's bad code, but in general, I think assignment
expressions inside default expressions are likely to be very
surprising :)
The only way for it to be confusing is to have => on one argument and
then = on a subsequent argument, *and* to have the earlier one refer
to the later one.
For you, maybe. I assert that the forms above *are* confusing for me.
You're welcome to explain them to me, like you have, and maybe I'll
now remember the logic for the future, but as a data point, I stand by
my statement that these were confusing to me when I encountered them
fresh.
Then let's leave aside the term "obvious" and just go for
"unsurprising". If you write code and get UnboundLocalError, will you
be surprised that it doesn't work? If you write code and it works,
will you be surprised with the result you got?
Once you learn the basic idea of left-to-right evaluation, it should
be possible to try things out and get unsurprising results. That's
what I'm hoping for.
Feel free to state that there's not *enough* cases of people
being confused by the semantics to outweigh the benefits, but it feels
to me that there are a few people claiming confusion here, and simply
saying "you shouldn't be confused, it's obvious" isn't really
addressing the point.
Part of the problem is that one person seems to think that Python will
completely change its behaviour, and he's spreading misinformation.
Ignore him, look just at the proposal itself, and tell me if it's
still confusing.
Even if someone *can* provide an answer, I'd be reluctant to accept
that any answer could be described as "intuitive". And "well, don't do
that" is just ducking the question - in essentially the same way as
"it's implementation defined" does...
But "don't do that" is a perfectly reasonable response to other kinds
of bad code, like messing up your spacing:
x = 1+2 * 3+4
Is this intuitive? Some people will think that x should be 21, but the
actual answer is 11. Python won't stop you from doing this, but style
guides absolutely should.
But that's not the same as you leaving the behaviour implementation
defined. In the case of operator precedence, there *is* a well-defined
answer, but the spacing doesn't match that interpretation.
That IS the concern when people are talking about what's "intuitive" though.
you're refusing to give a well-defined semantics, and then saying that
people shouldn't do that. But unlike spacing of expressions, the order
of arguments is *important* - it is part of the API of frob that the
first positional argument is n, so "just swap the arguments" is a
semantic change. So how should people get the ("obvious") intended
behaviour? Abandon the new syntax and go back to using None as a
default? That seems a shame, given that (as I understand it) your
reference implementation works exactly as I'd want.
The only two possible behaviours are:
1) It does the single obvious thing: n defaults to the length of
items, and items defaults to an empty tuple.
2) It raises UnboundLocalError if you omit n.
To be quite honest, I can't think of any non-toy examples where the
defaults would be defined backwards, like this. (Keyword-only
arguments can of course be reordered as required, as their order isn't
part of the API.) But if there is one out there, then yes, you would
probably need to go back to using None; or, you can rely on
implementation-specific permission and do it anyway. It's not like
Steven's constant panic-fear that "undefined behaviour" literally
means the Python interpreter could choose to melt down your computer.
There are *two* options, no more, no less, for what is legal.
In the same way, I would strongly recommend that style guides frown
upon referring to arguments later in the parameter list, even if it
happens to be legal. I'm just not mandating that the language check
for this and artificially block it.
You're not *just* recommending this for style guides, you're also
explicitly stating that you refuse to assign semantics to it.
It's unfair to say that I "refuse to assign semantics" as if I'm
permitting literally any behaviour. All I'm doing is saying that the
UnboundLocalError is optional, *at this stage*. There have been far
less-defined semantics that have remained in the language for a long
time, or cases where something has changed in behaviour over time
despite not being explicitly stated as implementation-defined. Is this
legal?
def f():
x = 1
global x
Does Python mandate whether this is legal or not? If so, how far back
in Python's history has it been defined?
The semantics, if this code is legal, are obvious: the name x must
always refer to the global, including in the assignment above it. If
it's not legal, you get an exception, not an interpreter crash, not
your hard drive getting wiped, and not a massive electric shock to the
programmer.
Would you prefer that I simply mandate that it be permitted, and then
a future version of Python changes it to be an exception? Or the other
way around? Because I could do that. Maybe it would reduce the
arguments. Pun intended, and I am not apologizing for it.
ChrisA
but at very least, if it isn't *obvious*, it should at least be
*unsurprising* if you then get UnboundLocalError.
+1 I think this (== "obvious in hindsight") is probably a better
criterion than "obvious" (from the definition) when an error *will* be
raised. (That's as a general principle. Ie, it doesn't need to be
obvious that you will *get* feedback from the system when you do
something, but it should be obvious what the feedback means when you
*do* get it.)
Steve
I understand that left-to-right evaluation is something that has to be
learned (and isn't 100% true - operator precedence is a thing too),
but at very least, if it isn't *obvious*, it should at least be
*unsurprising* if you then get UnboundLocalError.
Why? Are you saying I can't be surprised by the details of rules that
I don't often have a need to understand in detail?
I fear we're getting off-topic here, though. I'm not arguing that
anything here isn't well-defined, just that it's not obvious *to me*.
And I'm not even "arguing" that, I'm simply stating it as an observed
fact about how I initially reacted to the quoted example. It's you who
is stating that the frob case is "clearly" UnboundLocalError, and all
I'm saying is that's not "clear" to me, even if it is a consequence of
the rules in the PEP. And actually, I could argue that the PEP would
benefit from some clarification to make that consequence clearer - but
I don't feel that you're likely to be particularly receptive to that
statement. In case you are, consider that as written, the PEP says
that the *defaults* are evaluated left to right in the function's
runtime scope, but it doesn't say when the parameter names are
introduced in that scope - prior to this PEP there was no need to
define that detail, as nothing could happen before the names were
introduced at the start of the scope. If you accept that
clarification, can you accept that the current text isn't as clear as
it might be?
Maybe... I'm not sure I see this as *that* much more obvious, although
I concede that the left-to-right evaluation rule implies it (it feels
like a mathematician's use of "obvious" - which quite often isn't ;-))
Using assignment expressions in argument defaults is well-defined but
not necessarily obvious in a similar way (to me, at least).
When you say "assignment expressions", do you mean "default
expressions", or are you referring to the walrus operator? There's a
lot of other potentially-surprising behaviour if you mix assignment
expressions in with this, because of the difference of scope. It's the
sort of thing that can definitely be figured out, but I would advise
against it.
I meant the walrus operator, and that's my point. There's a lot of
not-immediately-obvious interactions here. Even if we don't include
default expressions, I'd argue that the behaviour is non-obvious:
def f(a=(b:=12)):
... print(a, b)
...
f()
12 12
b
12
I assume (possibly naïvely) that this is defined in the language spec,
though, as it's existing behaviour. But when you add in default
expressions, you need to be sure that the various interactions are
well-defined. Note that at this point, I'm not even talking about
"obvious", simply the bare minimum of "if I write this supposedly
legal code, does the PEP explain what it does?"
This will reassign items to be an empty list if n is omitted.
Obviously that's bad code, but in general, I think assignment
expressions inside default expressions are likely to be very
surprising :)
def f(a=(b:=12), b=9):
... print(a, b)
...
f()
12 9
b
12
Would
def frob(n=>len(items:=[]), items=>[1,2]):
...
reassign items if n is omitted? Or would it assign the *global* items
and then shadow it with a local for the parameter? Can you point to
the explanation in the PEP that covers this? And even if you can, are
you trying to claim that the behaviour is "obvious"?
Then let's leave aside the term "obvious" and just go for
"unsurprising". If you write code and get UnboundLocalError, will you
be surprised that it doesn't work? If you write code and it works,
will you be surprised with the result you got?
As I noted above, "surprising" is no different. I can easily be
surprised by well-defined behaviour. I'm not arguing that there's no
explanation for why a particular construct works the way that it does,
just that the behaviour may not be intuitive to people even if it is a
consequence of the rules. I'm arguing that the behaviour fails an "is
this easy to teach" criterion, not "is this logically consistent".
Once you learn the basic idea of left-to-right evaluation, it should
be possible to try things out and get unsurprising results. That's
what I'm hoping for.
Get "explainable" results, yes. But I thought Python was supposed to
aspire to more than that, and match how people thought about things.
"Executable pseudocode" and all that.
Feel free to state that there's not *enough* cases of people
being confused by the semantics to outweigh the benefits, but it feels
to me that there are a few people claiming confusion here, and simply
saying "you shouldn't be confused, it's obvious" isn't really
addressing the point.
Part of the problem is that one person seems to think that Python will
completely change its behaviour, and he's spreading misinformation.
Ignore him, look just at the proposal itself, and tell me if it's
still confusing.
OK, if this is going to boil down to you asserting that the only
problems here are with "one person" then I don't think it's worth
continuing. I am not simply parroting "misinformation spread by that
one person" (and you've made it very obvious already who that
individual is, so please try to keep your personal problem with them
out of your discussions with me). If you're not willing to accept my
comments as feedback given in my own right, then it's you who is
shutting down discussion here, and I don't see much point in trying to
provide a good-faith response to you.
1) It does the single obvious thing: n defaults to the length of
items, and items defaults to an empty tuple.
2) It raises UnboundLocalError if you omit n.
To be quite honest, I can't think of any non-toy examples where the
defaults would be defined backwards, like this.
If that's the case, then what is the downside of picking one?
Personally, I have a nagging feeling that I could find a non-toy
example, but it's not that important to me. What I'm arguing is that
there's no point in not picking a behaviour. You're saying you don't
want to lock other implementations into the particular behaviour you
choose - but you also don't have an example of where that would be a
problem, so we're *both* arguing hypotheticals here.
It's not like
Steven's constant panic-fear that "undefined behaviour" literally
means the Python interpreter could choose to melt down your computer.
Oh, please. If that's the only way in which you can imagine
implementation-defined behaviour being an issue, then you've lived a
pretty sheltered life. How about "My code works on Python 3.12 but not
on 3.13, because the behaviour in this case changed with no warning"?
Sure, the PEP (and presumably the docs) said "don't do that", but you
said above that people experiment and work out behaviour from those
experiments. So breaking their code because they did precisely that
seems at best pretty harsh.
There are *two* options, no more, no less, for what is legal.
Nope, there are two that you consider acceptable behaviour. And I
don't disagree with you. But what's so magical about two? Why not have
just one that's legal. Because people might disagree with your choice?
You're the PEP author, let them. Or are you worried that this single
point could cause the PEP to fail?
You're not *just* recommending this for style guides, you're also
explicitly stating that you refuse to assign semantics to it.
It's unfair to say that I "refuse to assign semantics" as if I'm
permitting literally any behaviour.
Don't put words into my mouth. You have stated that you won't require
a particular behaviour. That's refusing to assign semantics. If it
makes you feel better I'll concede that you're not allowing
*arbitrary* semantics.
By the way, a lot of this debate could be solved incredibly easily by
writing the PEP in terms of code equivalence:
def fn(p1=>e1, p2=>e2, p3=e3):
body
behaves the same as
def fn(p1=(_d1:=object()), p2=(_d2:=object()), p3=e3):
if p1 is _d1:
p1 = e1
if p2 is _d2:
p2 = e2
There's probably some details to flesh out, but that's precise and
well-defined. Debates over whether the resulting behaviour is
"obvious" or "intuitive" can then take place against a background
where everyone agrees what will happen (and can experiment with real
code to see if they are comfortable with it).
All I'm doing is saying that the
UnboundLocalError is optional, *at this stage*. There have been far
less-defined semantics that have remained in the language for a long
time, or cases where something has changed in behaviour over time
despite not being explicitly stated as implementation-defined. Is this
legal?
def f():
x = 1
global x
Does Python mandate whether this is legal or not? If so, how far back
in Python's history has it been defined?
*Shrug*. There was never a PEP about it, I suspect, and the behaviour
was probably defined a long time before Python was the most popular
language in the world. It would be nice if we still had the freedom
that we did back then. Sadly, we don't. Maybe some people are *too*
cautious nowadays. It's entirely possible I'm one of them. That's why
we have the SC - if you're confident that your proposal is solid in
spite of people like me complaining about edge cases, then submit it.
I'll trust the SC's judgement.
The semantics, if this code is legal, are obvious: the name x must
always refer to the global, including in the assignment above it. If
it's not legal, you get an exception, not an interpreter crash, not
your hard drive getting wiped, and not a massive electric shock to the
programmer.
Sigh. You have a very narrow view of "obvious". I can think of other
equally "obvious" interpretations. I won't list them because you'll
just accuse me of being contrary.
But I will say that I tried that code and you get an exception. But
interestingly, it's a *syntax* error (name assigned before global
declaration), not a *runtime* exception. I genuinely don't know which
you intended to suggest would be the obvious behaviour...
Would you prefer that I simply mandate that it be permitted, and then
a future version of Python changes it to be an exception? Or the other
way around? Because I could do that. Maybe it would reduce the
arguments. Pun intended, and I am not apologizing for it.
lol, I'm always up for a good pun :-)
Are you still talking about the global example? Because I'd prefer you
left that part of the language alone. And if you're talking about PEP
671, you know my answer (I'd prefer you permit it and define what it
does, so it can't change in future).
Paul
Um, I didn't see that as any more obvious than the original example. I
guess I can see it's UnboundLocalError, but honestly that's not
obvious to me.
Question: Is this obvious?
def f():
x, x[0] = [2], 3
print(x)
def boom():
x[0], x = 3, [2]
# raises UnboundLocalError
No. I'm not sure what point you're trying to make here?
The point is that many things can be unobvious, including aspects of
important features that we make good use of all the time. But they are
consistent, which means that, once you try it and run into a problem,
you should be able to see *why* it's a problem.
(This particular example is another case of LTR evaluation - or to be
more precise, LTR assignment - and while I wouldn't do it in a simple
statement like this, I certainly have made use of it in a 'for' loop.)
I understand that left-to-right evaluation is something that has to be
learned (and isn't 100% true - operator precedence is a thing too),
but at very least, if it isn't *obvious*, it should at least be
*unsurprising* if you then get UnboundLocalError.
Why? Are you saying I can't be surprised by the details of rules that
I don't often have a need to understand in detail?
My point is that "unsurprising", while a much weaker criterion than
"obvious", should be quite attainable. If you try the above two pieces
of code, you'll quickly find that one of them works and one doesn't,
and from the exceptions you get, the rule should be fairly clear.
In case you are, consider that as written, the PEP says
that the *defaults* are evaluated left to right in the function's
runtime scope, but it doesn't say when the parameter names are
introduced in that scope - prior to this PEP there was no need to
define that detail, as nothing could happen before the names were
introduced at the start of the scope. If you accept that
clarification, can you accept that the current text isn't as clear as
it might be?
I actually don't accept that clarification, because nothing has
changed. At what point in this function do the names get introduced to
the scope?
def spam(x, y=1, *, z=2):
ham = [x, y, z]
They are all "introduced", if that term even has meaning, at the very
instant that the scope begins to exist. The name 'ham' isn't
introduced to the scope at a subsequent point. There are languages
that work this way (and it can be very convenient when used
correctly), but Python is not one of them.
Late-bound defaults do not affect this in any way. A function
parameter, like any other local, is local for the entire scope of the
function. It doesn't "become local" part way through.
Do I need to state this in the PEP? Are there other parts of Python's
semantics which need to be restated in the PEP too? Which parts,
despite not changing, are now going to be brought into question?
I meant the walrus operator, and that's my point. There's a lot of
not-immediately-obvious interactions here. Even if we don't include
default expressions, I'd argue that the behaviour is non-obvious:
def f(a=(b:=12)):
... print(a, b)
...
f()
12 12
b
12
I assume (possibly naïvely) that this is defined in the language spec,
though, as it's existing behaviour. But when you add in default
expressions, you need to be sure that the various interactions are
well-defined.
They absolutely are well-defined. Almost certainly not useful, but
well-defined. The right hand side of either "a=EXPR" or "a=>EXPR" is
simply evaluated as an ordinary expression; the only difference is
whether it's evaluated at function definition time and in function
definition context, or at function invocation time and in the context
of the function itself.
def f(a=(b:=12), b=9):
... print(a, b)
...
f()
12 9
b
12
Since this is an early-bound default, it can be considered like this:
_default = (b:=12)
def f(a=None, b=9):
if a is None: a = _default
print(a, b)
And then it should be unsurprising that b becomes 12 in the
surrounding scope, paralleling a's default value, and b defaults to 9
in the function's context.
reassign items if n is omitted? Or would it assign the *global* items
and then shadow it with a local for the parameter? Can you point to
the explanation in the PEP that covers this? And even if you can, are
you trying to claim that the behaviour is "obvious"?
Since these are both late-bound defaults, they can be considered like this:
def frob(n=None, items=None):
if n is None: n = len(items:=[])
if items is None: items = [1, 2]
...
Under the "Specification" section, the PEP says:
"""Multiple late-bound arguments are evaluated from left to right, and
can refer to previously-defined values."""
Everything hinges on this left-to-right evaluation. The entire
expression, including the assignment, is evaluated, and then you move
on to the next one.
(Of course, in the actual proposal, None isn't special like this. But
from the perspective of assignment semantics, the longhand forms are
broadly equivalent, and should be read more as a mythical "if items is
not assigned:" syntax.)
Then let's leave aside the term "obvious" and just go for
"unsurprising". If you write code and get UnboundLocalError, will you
be surprised that it doesn't work? If you write code and it works,
will you be surprised with the result you got?
As I noted above, "surprising" is no different. I can easily be
surprised by well-defined behaviour. I'm not arguing that there's no
explanation for why a particular construct works the way that it does,
just that the behaviour may not be intuitive to people even if it is a
consequence of the rules. I'm arguing that the behaviour fails an "is
this easy to teach" criterion, not "is this logically consistent".
Okay. So what's the threshold then? I've tried to make this logically
consistent, not only with itself, but with *every other place in
Python where assignment happens*. It's always left-to-right.
Once you learn the basic idea of left-to-right evaluation, it should
be possible to try things out and get unsurprising results. That's
what I'm hoping for.
Get "explainable" results, yes. But I thought Python was supposed to
aspire to more than that, and match how people thought about things.
"Executable pseudocode" and all that.
I'm sorry that Python already doesn't live up to this expectation, but
there's nothing I can do about that. Ultimately, everything has to
have defined semantics, even the weird edge cases, and this is
definitely an edge case.
If this feature were implemented, I doubt that people would often see
examples like this outside of test suites. Referring to arguments
out-of-order simply isn't a normal thing that programmers want to do,
because it makes for a confusing API. We are debating the teachability
of something that is usually going to be irrelevant, because most use
of this will be trivially simple to understand:
def f(items, n=>len(items)):
This will Just Work, and there's no backwards evaluation to worry
about. It's ONLY when you put the arguments the other way around that
evaluation order even becomes significant. This is no different from
many other parts of Python, where the order of evaluation is defined,
but usually irrelevant.
Feel free to state that there's not *enough* cases of people
being confused by the semantics to outweigh the benefits, but it feels
to me that there are a few people claiming confusion here, and simply
saying "you shouldn't be confused, it's obvious" isn't really
addressing the point.
Part of the problem is that one person seems to think that Python will
completely change its behaviour, and he's spreading misinformation.
Ignore him, look just at the proposal itself, and tell me if it's
still confusing.
OK, if this is going to boil down to you asserting that the only
problems here are with "one person" then I don't think it's worth
continuing. I am not simply parroting "misinformation spread by that
one person" (and you've made it very obvious already who that
individual is, so please try to keep your personal problem with them
out of your discussions with me). If you're not willing to accept my
comments as feedback given in my own right, then it's you who is
shutting down discussion here, and I don't see much point in trying to
provide a good-faith response to you.
If you can show me a way in which this proposal isn't consistent with
the rest of Python, then I'll address that.
1) It does the single obvious thing: n defaults to the length of
items, and items defaults to an empty tuple.
2) It raises UnboundLocalError if you omit n.
So why not pick one?
For the same reason that Python didn't just "pick one" about things
like __del__ invocation time: it constrains language implementations
unnecessarily.
To be quite honest, I can't think of any non-toy examples where the
defaults would be defined backwards, like this.
If that's the case, then what is the downside of picking one?
Even if the situation never came up outside of toy examples, the
language would be forced to go through hoops to implement it.
Whichever semantic form was chosen, it would likely be suboptimal for
some implementation.
Maybe down the track, it would be able to be more rigorously defined.
That's happened before, plenty of times. It's much harder to change
the definition than to tighten up something that wasn't fully
specified, because people won't have been depending on it.
Personally, I have a nagging feeling that I could find a non-toy
example, but it's not that important to me. What I'm arguing is that
there's no point in not picking a behaviour. You're saying you don't
want to lock other implementations into the particular behaviour you
choose - but you also don't have an example of where that would be a
problem, so we're *both* arguing hypotheticals here.
I actually do have an example, except that it was just a previous
version of my reference implementation, where I tried to implement
perfect left-to-right evaluation (as opposed to two-pass). It was
incredibly messy. But maybe in the future, someone will be able to
make a much better one, and then it would be worth using it.
Iteration order of Python's dictionaries had, for years, been
completely unspecified. Then hash randomization came along, and
iterating over a dictionary of strings became actually random. And
then iteration order became defined, not because someone felt like the
specification should have 'just chosen', but because an
*implementation* made it worthwhile.
It's easy for you to say that there's "no point in not picking", but
believe you me, there is plenty of point, otherwise I would have cut
off all these debates by simply locking in the two-pass behaviour of
the reference implementation.
It's not like
Steven's constant panic-fear that "undefined behaviour" literally
means the Python interpreter could choose to melt down your computer.
Oh, please. If that's the only way in which you can imagine
implementation-defined behaviour being an issue, then you've lived a
pretty sheltered life. How about "My code works on Python 3.12 but not
on 3.13, because the behaviour in this case changed with no warning"?
It seems to be the assumption that he has. Ask him some time about C's
concept of undefined behaviour, and then see if you can understand why
he's so vitriolic about my proposal.
Sure, the PEP (and presumably the docs) said "don't do that", but you
said above that people experiment and work out behaviour from those
experiments. So breaking their code because they did precisely that
seems at best pretty harsh.
Things DO change. Generally, Python tries to avoid breaking changes,
and especially, changes where there's no way to "straddle" your code
(if 3.13 breaks your code in some way, but the fixed version works
just as well as the original on 3.12, then you can push out the fix
without worrying about 3.12 now breaking your code). In this
particular situation, the absolute worst-case option is that you
forfeit the benefits of this feature and go with the sentinel object:
_UNSPECIFIED = object()
def foo(n=_UNSPECIFIED, items=()):
if n is _UNSPECIFIED: n = len(items)
So even if this does start to become a problem in the future, people
can, without materially changing their APIs, write code that uses this
out-of-order evaluation. But I would still like to see a non-toy
example where this would even come up.
There are *two* options, no more, no less, for what is legal.
Nope, there are two that you consider acceptable behaviour. And I
don't disagree with you. But what's so magical about two? Why not have
just one that's legal. Because people might disagree with your choice?
You're the PEP author, let them. Or are you worried that this single
point could cause the PEP to fail?
Let me rephrase. According to the specification in the PEP, these are
the only two behaviours which are considered compliant.
Python implementations are not permitted to, in the face of
out-of-order parameter references, do completely arbitrary things like
assigning 42 to all parameters.
What's so magical about two? Nothing. They're just the only two
behaviours that are consistent with the rest of the document.
You're not *just* recommending this for style guides, you're also
explicitly stating that you refuse to assign semantics to it.
It's unfair to say that I "refuse to assign semantics" as if I'm
permitting literally any behaviour.
Don't put words into my mouth. You have stated that you won't require
a particular behaviour. That's refusing to assign semantics. If it
makes you feel better I'll concede that you're not allowing
*arbitrary* semantics.
I'm not sure what you mean by putting words in your mouth, but the
part inside the quotation marks was literally words from your
preceding comment. You did indeed say that.
By the way, a lot of this debate could be solved incredibly easily by
writing the PEP in terms of code equivalence:
def fn(p1=>e1, p2=>e2, p3=e3):
body
behaves the same as
def fn(p1=(_d1:=object()), p2=(_d2:=object()), p3=e3):
if p1 is _d1:
p1 = e1
if p2 is _d2:
p2 = e2
The trouble is, it's not 100% equivalent. It's good enough for a post
here, but it needs a lot of caveats. Generally speaking, using => is
*broadly equivalent* to this sort of check, but I can't do what PEP
380 did for the "yield from" statement here and define its semantics
entirely, because Python simply doesn't have a way to leave something
unassigned and then check if it's been assigned to.
But yes, if you want some example equivalencies, I could add those. (I
would simply assign before the def statement though, rather than
muddying the waters with assignment expressions. People will be
confused enough without wondering what the scope of those is.)
There's probably some details to flesh out, but that's precise and
well-defined. Debates over whether the resulting behaviour is
"obvious" or "intuitive" can then take place against a background
where everyone agrees what will happen (and can experiment with real
code to see if they are comfortable with it).
Well, I did write a reference implementation, so if people want to
experiment with real code, they absolutely can.
All I'm doing is saying that the
UnboundLocalError is optional, *at this stage*. There have been far
less-defined semantics that have remained in the language for a long
time, or cases where something has changed in behaviour over time
despite not being explicitly stated as implementation-defined. Is this
legal?
def f():
x = 1
global x
Does Python mandate whether this is legal or not? If so, how far back
in Python's history has it been defined?
*Shrug*. There was never a PEP about it, I suspect, and the behaviour
was probably defined a long time before Python was the most popular
language in the world. It would be nice if we still had the freedom
that we did back then. Sadly, we don't. Maybe some people are *too*
cautious nowadays. It's entirely possible I'm one of them. That's why
we have the SC - if you're confident that your proposal is solid in
spite of people like me complaining about edge cases, then submit it.
I'll trust the SC's judgement.
The behaviour was actually fully legal until quite recently (it did
issue a warning, but most people have those turned off). It didn't get
a PEP, and it was just a small note in the What's New under "smaller
changes to the language".
Behaviour DOES change. Is it so bad to have advance warning that
something might change? Because if people prefer it, I could
absolutely lock in one definition of this, knowing full well that the
next release might want to reverse that decision.
The semantics, if this code is legal, are obvious: the name x must
always refer to the global, including in the assignment above it. If
it's not legal, you get an exception, not an interpreter crash, not
your hard drive getting wiped, and not a massive electric shock to the
programmer.
Sigh. You have a very narrow view of "obvious". I can think of other
equally "obvious" interpretations. I won't list them because you'll
just accuse me of being contrary.
I'll grant you that other languages DO have completely different
semantics here, but in Python, a name is what it is throughout a
function; there is not a single circumstance where you can refer to a
name in two different scopes at once. The nearest to that is tricks
like "def f(x=x):" where something is evaluated in a different
context, but it is still completely unambiguous. (Okay, it might be
only *technically* unambiguous when you mix comprehensions and
assignment expressions, but they got special-cased to make them less
surprising.)
But I will say that I tried that code and you get an exception. But
interestingly, it's a *syntax* error (name assigned before global
declaration), not a *runtime* exception. I genuinely don't know which
you intended to suggest would be the obvious behaviour...
Yes, that's because the global statement is a syntactic feature, not
an executable one. But you may note that I never said it was obvious
that it had to be SyntaxError; only that it had to be an error (or to
refer to the global). The distinction between syntax errors (parse
time), function definition runtime errors, and function invocation
runtime errors, is much more subtle, and I don't expect people to be
able to intuit which one anything should be.
Would you prefer that I simply mandate that it be permitted, and then
a future version of Python changes it to be an exception? Or the other
way around? Because I could do that. Maybe it would reduce the
arguments. Pun intended, and I am not apologizing for it.
Are you still talking about the global example? Because I'd prefer you
left that part of the language alone. And if you're talking about PEP
671, you know my answer (I'd prefer you permit it and define what it
does, so it can't change in future).
But I don't want to force it to not change in the future. In any case,
the future can't be fully mandated like that. So your options are:
1) Lock in the semantics now, and if in the future it changes, then it
breaks people's code
2) Provide two options that implementations can choose, and if in the
future only one is legal, people's code should still have been
compatible with both
Which would you prefer? I am completely open to the first option, but
I just think it's unfair to future people's code to have it break,
when I could have given them fair warning that this shouldn't be done.
ChrisA
1) It does the single obvious thing: n defaults to the length of
items, and items defaults to an empty tuple.
2) It raises UnboundLocalError if you omit n.
...
Would you prefer that I simply mandate that it be permitted, and then
a future version of Python changes it to be an exception? Or the other
way around? Because I could do that. Maybe it would reduce the
arguments. Pun intended, and I am not apologizing for it.
A third option would be that it's a syntax error for you to define such
a function. This would be my preferred approach. This way, you can
always assign semantics in the future. Sure, there are some cases you
might want to support with the as-specified undefined behavior, but I
don't think that's a good design.
If CPython 3.x works a certain way (especially if there's a test for
it), you can be sure that other implementations will work the same way,
and you can be sure that you can never change the behavior in the
future. No amount of "but we said it was undefined" will allow us to
change such behavior.
It would be like 3.7 saying "yeah, we said dicts are ordered in 3.6 but
you shouldn't count on it, so we're going to make them unordered in
3.7". It would just never happen.
That all said, I'm still -1 on this PEP, for reasons I won't rehash.
Eric
Um, I didn't see that as any more obvious than the original example. I
guess I can see it's UnboundLocalError, but honestly that's not
obvious to me.
Question: Is this obvious?
def f():
x, x[0] = [2], 3
print(x)
def boom():
x[0], x = 3, [2]
# raises UnboundLocalError
Personally, I don't care so much whether the behavior is "obvious" or
even "confusing" as about whether it's defined. Both of those examples
have defined behavior in Python. That's better than having undefined
behavior where either of them might do different things depending on the
implementation.
It's unfair to say that I "refuse to assign semantics" as if I'm
permitting literally any behaviour. All I'm doing is saying that the
UnboundLocalError is optional, *at this stage*. There have been far
less-defined semantics that have remained in the language for a long
time, or cases where something has changed in behaviour over time
despite not being explicitly stated as implementation-defined. Is this
legal?
def f():
x = 1
global x
Does Python mandate whether this is legal or not? If so, how far back
in Python's history has it been defined?
If it doesn't mandate it, then it would be better if it did. In my
view, none of your arguments about "some things are undefined" are
actually providing any positive support for the idea that the behavior
in question should remained undefined in the PEP.
I think the PEP would benefit from a fully explicit definition of
exactly when and how the late-bound defaults would be evaluated. For
instance, by demonstrating an "unrolling" into something paralleling
existing Python code. Like:
def f(a=>items[0], items=[]):
# body
is equivalent to:
def f(a=FakeDefault, items=[]):
a = items[0]
# body
. . . or whatever. I mean, it's up to you as the PEP author to decide
what semantics you want. But I think the PEP should be fully explicit
about the order of evaluation of everything. I don't see any benefit to
leaving it ambiguous.
This is all academic to me, however, since even if you did that I still
wouldn't support the PEP for various more basic reasons that I've
mentioned in the earlier iteration of this thread.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
Um, I didn't see that as any more obvious than the original example. I
guess I can see it's UnboundLocalError, but honestly that's not
obvious to me.
Question: Is this obvious?
def f():
x, x[0] = [2], 3
print(x)
def boom():
x[0], x = 3, [2]
# raises UnboundLocalError
Personally, I don't care so much whether the behavior is "obvious" or
even "confusing" as about whether it's defined. Both of those examples
have defined behavior in Python. That's better than having undefined
behavior where either of them might do different things depending on the
implementation.
It's unfair to say that I "refuse to assign semantics" as if I'm
permitting literally any behaviour. All I'm doing is saying that the
UnboundLocalError is optional, *at this stage*. There have been far
less-defined semantics that have remained in the language for a long
time, or cases where something has changed in behaviour over time
despite not being explicitly stated as implementation-defined. Is this
legal?
def f():
x = 1
global x
Does Python mandate whether this is legal or not? If so, how far back
in Python's history has it been defined?
If it doesn't mandate it, then it would be better if it did. In my
view, none of your arguments about "some things are undefined" are
actually providing any positive support for the idea that the behavior
in question should remained undefined in the PEP.
I've just pushed a change to the wording. Let's see if it makes a difference.
I think the PEP would benefit from a fully explicit definition of
exactly when and how the late-bound defaults would be evaluated. For
instance, by demonstrating an "unrolling" into something paralleling
existing Python code. Like:
def f(a=>items[0], items=[]):
# body
is equivalent to:
def f(a=FakeDefault, items=[]):
a = items[0]
# body
. . . or whatever. I mean, it's up to you as the PEP author to decide
what semantics you want. But I think the PEP should be fully explicit
about the order of evaluation of everything. I don't see any benefit to
leaving it ambiguous.
Read the latest version and tell me if it still sounds ambiguous.
This is all academic to me, however, since even if you did that I still
wouldn't support the PEP for various more basic reasons that I've
mentioned in the earlier iteration of this thread.
And there we have it. People are complaining loudly, but then ALSO
saying that they don't support the proposal anyway. Why are you
bothering to debate this if you've already made your decision?
ChrisA
The suggestion is to add a new keyword to a PEP which absolutely doesn't
need it,
*shrug*
The match...case statement didn't "need" keywords either, we could have
picked symbols instead if we wanted to look like APL. Remember that
keywords have advantages as well as disadvantages. Given the existence
of community support for keywords, the PEP should make the case that
symbols are better in this case.
Even if that's only "a majority prefer symbols".
on the grounds that it **might** (**not** would - we can't know
without a spec) give compatibility with some fictional vapourware which
- for all people keep talking about it - hasn't happened in years, isn't
happening (AFAIK nobody is working on it), doesn't have anything
remotely close to even an outline specification (people disagree as to
what it should do), very likely never will happen, and at best won't
happen for years.
I think that is broadly accurate. Harsh but fair: nobody has a concrete
plan for generalising "defer" keyword would do. It is still vapourware.
[...]
Late-bound defaults are meant to be evaluated at function
call time (and in particular, not some way down in the function body when
the parameter gets used).
Not necessarily.
I don't recall if this has been raised in this thread before, but it is
possible to delay the evaluation of the default value until it is
actually needed. I believe that this is how Haskell operates pretty much
everywhere. (Haskell experts: do I have that correct?)
I expect Chris will be annoyed at me raising this, but one way of
implementing this would be to introduce a generalised "lazy evaluation"
mechanism, similar to what Haskell does, rather than special-casing
late-bound defaults. Then late-bound defaults just use the same
mechanism, and syntax, as lazily evaluated values anywhere else.
I expect that this is the point that David is making: don't introduce
syntax for a special case that will be obsolete in (mumble mumble...)
releases.
David's point would be stronger if he could point to a concrete plan to
introduce lazy evaluation in Python. The Zen of Python gives us some
hints:
Now is better than never.
Although never is often better than *right* now.
which possibly suggests that the Zen was written by elves:
"I hear it is unwise to seek the council of elves, for they will answer
with yes and no."
Chris may choose to reject this generalised lazy evaluation idea, but if
so it needs to go into a Rejected Ideas section. Or he may decide that
actually having a generalised lazy evaluation idea is *brilliant* and
much nicer than making defaults a special case.
(I think that the Zen has something to say about special cases too.)
This raises another choice: should lazy defaults be evaluated before
entering the body of the function, or at the point where the parameter
is used? Which would be more useful?
# `defer n=len(items)`
def func(items=[], n=>len(items)):
items.append("Hello")
print(n)
func()
Printing 1 would require a generalised lazy mechanism, but printing 0
is independent of the mechanism. As it stands, the PEP requires 0. Which
would be better or more useful?
I guess Chris will say 0 and David will say 1, but I might be wrong
about either of them.
One way or the other, these are the sorts of questions that the
discussion is supposed to work out, and the PEP is supposed to
reference. There are well over 600 emails in this thread and the
Steering Council should not be expected to read the whole thing, the PEP
is supposed to be an honest and fair summary of alternatives and
rejected ideas.
Chris is welcome to push for a particular proposal. That is the purpose
of the PEP process. He is also supposed to give dissenting arguments and
alternatives fair airing in the PEP itself, even if only in a Rejected
Ideas section.
--
Steve
I expect Chris will be annoyed at me raising this, but one way of
implementing this would be to introduce a generalised "lazy evaluation"
mechanism, similar to what Haskell does, rather than special-casing
late-bound defaults. Then late-bound defaults just use the same
mechanism, and syntax, as lazily evaluated values anywhere else.
Go ahead. Implement it. It actually is NOT sufficient, for a number of
reasons, which I have laid out in previous posts.
Generic "lazy evaluation" is sufficient to do *some* of what can be
done with late-bound argument defaults, but it is not sufficient for
everything, unless it is a nightmare that basically involves eval'ing
a text string in the surrounding context, with full ability to create
and reference any name bindings. Python does not support this concept.
Chris may choose to reject this generalised lazy evaluation idea, but if
so it needs to go into a Rejected Ideas section. Or he may decide that
actually having a generalised lazy evaluation idea is *brilliant* and
much nicer than making defaults a special case.
It's an almost completely orthogonal proposal. I used to have a
reference to it in the PEP but removed it because it was unhelpful.
This raises another choice: should lazy defaults be evaluated before
entering the body of the function, or at the point where the parameter
is used? Which would be more useful?
That's one of the problems. Generic lazy evaluation should be
processed at some point where the parameter is used, but late-bound
defaults are evaluated as the function begins. They are orthogonal.
Printing 1 would require a generalised lazy mechanism, but printing 0
is independent of the mechanism. As it stands, the PEP requires 0. Which
would be better or more useful?
Probably both. If someone wants to propose a generic deferred
evaluation feature, I would be happy to discuss it, but it's not a
replacement for PEP 671, nor is PEP 671 a replacement for it.
Chris is welcome to push for a particular proposal. That is the purpose
of the PEP process. He is also supposed to give dissenting arguments and
alternatives fair airing in the PEP itself, even if only in a Rejected
Ideas section.
At what point is an unrelated proposal a "rejected idea"? How
different does it have to be before it doesn't help to have it in that
section?
You're welcome to keep on arguing for the sake of arguing, but you're
not actually accomplishing anything by it. Especially since you're
rehashing the exact same complaints that you raised previously, and
which I responded, exactly the same way, at the time.
ChrisA
I think the PEP would benefit from a fully explicit definition of
exactly when and how the late-bound defaults would be evaluated. For
instance, by demonstrating an "unrolling" into something paralleling
existing Python code. Like:
def f(a=>items[0], items=[]):
# body
is equivalent to:
def f(a=FakeDefault, items=[]):
a = items[0]
# body
This is all academic to me, however, since even if you did
I read it before your update. Is the version up now the updated one?
I'm guessing it is, because of the remark about "implementations may
choose to do this in two separate passes".
But I don't see where something like what I showed there is given as a
specification. I do see the "How to Teach This" section, which has an
example similar to mine. But that pretty clearly isn't specifying the
behavior. It still talks about "broadly equivalent" and a "rule of
thumb", which are more ways of saying things other than a fully explicit
and normative specification of the behavior.
that I still
wouldn't support the PEP for various more basic reasons that I've
mentioned in the earlier iteration of this thread.
And there we have it. People are complaining loudly, but then ALSO
saying that they don't support the proposal anyway. Why are you
bothering to debate this if you've already made your decision?
I didn't intend to initially, but as the discussion continued I figured
since everyone else was restating their opinions there's no reason I
can't do so as well. :-) Also, to avoid people coming back later and
saying that some kind of consensus emerged in the second round because
no one objected, etc.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
I think the PEP would benefit from a fully explicit definition of
exactly when and how the late-bound defaults would be evaluated. For
instance, by demonstrating an "unrolling" into something paralleling
existing Python code. Like:
def f(a=>items[0], items=[]):
# body
is equivalent to:
def f(a=FakeDefault, items=[]):
a = items[0]
# body
That IS in the PEP. Have you read it?
I read it before your update. Is the version up now the updated one?
I'm guessing it is, because of the remark about "implementations may
choose to do this in two separate passes".
But I don't see where something like what I showed there is given as a
specification. I do see the "How to Teach This" section, which has an
example similar to mine. But that pretty clearly isn't specifying the
behavior. It still talks about "broadly equivalent" and a "rule of
thumb", which are more ways of saying things other than a fully explicit
and normative specification of the behavior.
"Broadly equivalent" is the best you're going to get, though. The
entire point of this proposal is that it isn't possible to do a
perfect job without language support. So the best you'll get is
something with a lot of caveats, or something that is just "basically
like this".
And your example has the exact same limitations. There's no
"FakeDefault" that can behave like that. Is "broadly equivalent" such
a bad thing?
ChrisA
This raises another choice: should lazy defaults be evaluated before
entering the body of the function, or at the point where the parameter
is used? Which would be more useful?
That's one of the problems. Generic lazy evaluation should be
processed at some point where the parameter is used, but late-bound
defaults are evaluated as the function begins. They are orthogonal.
That sounds like an *extremely* good statement to make in the
"Rejected suggestions" section of the PEP, explaining why the PEP's
proposal and a "lazy evaluation" proposal are different.
At what point is an unrelated proposal a "rejected idea"? How
different does it have to be before it doesn't help to have it in that
section?
At the point where people repeatedly offer it as an alternative to the
PEP. If it's so unrelated as to make no sense at all, just add it as
"XXX: rejected as it solves a different problem" or similar. But your
response above is better for the lazy evaluation suggestion, and
explains why you think it's unrelated.
You're welcome to keep on arguing for the sake of arguing, but you're
not actually accomplishing anything by it. Especially since you're
rehashing the exact same complaints that you raised previously, and
which I responded, exactly the same way, at the time.
*Everyone* is just rehashing the same comments by this point. People
are piling in because of a fear that if they don't, someone will claim
that we now have consensus, not because they have anything new to add.
IIRC, someone early in this thread even said something along the lines
of "we seem to be reaching a consensus", but I can't find the
reference now.
I'd suggest that Chris either mark the PEP as withdrawn, to make it
explicit that he has no plans to submit it, or submit it to the SC
(ideally with a note for transparency saying that there was no
consensus on python-ideas). Nobody is really benefiting from repeating
this discussion over and over.
Paul
And there we have it. People are complaining loudly, but then ALSO
saying that they don't support the proposal anyway. Why are you
bothering to debate this if you've already made your decision?
I can't speak for Brendan, but I have two reasons for discussing
despite being broadly in sympathy with David Mertz's reasons for
opposing the proposal:
1. Often enough I've disagreed with a proposal, only to see it
implemented. Of course, I want the least bad version from my
point of view.
2. More altruistically, even though I disagree with a proposal, I
would be ashamed if I thought there was a way to improve a
proposal, and didn't mention it in the hope of sabotaging the
proposal's chances by leaving it less than it could be.
That doesn't mean in either case I'm right, of course, but at least I
can *try*. :-)
Regards,
Steve
The match...case statement didn't "need" keywords either, we could have
picked symbols instead if we wanted to look like APL. Remember that
keywords have advantages as well as disadvantages. Given the existence
of community support for keywords, the PEP should make the case that
symbols are better in this case.
The argument I would make is that in the presence of type annotations
a keyword adds a lot of verbosity, and is possibly confusable (ie,
ignorable) as a type component, and that that is sufficient argument
for a symbol rather than a keyword.
This raises another choice: should lazy defaults be evaluated before
entering the body of the function, or at the point where the parameter
is used? Which would be more useful?
<img src=both_both_both.gif/>
Chris makes the point that we've been through this before so I won't
belabor that point. Both are potentially useful. Chris wants syntax
for the common pattern
def foo(arg_with_new_empty_list_default=None):
if arg_with_new_empty_list_default is None:
arg_with_new_empty_list_default = []
# do stuff
and variants (eg, where the default is a computation expensive enough
that you don't want to do it if the argument is never defaulted). I
don't find that burdensome enough to want syntax, and I can't guess
how quickly I'd start using it if available, that probably depends on
how often the community seems to be using it (ie, in the code I'm
reading).
I can't really guess how useful the "use point" version would be.
It's not a pattern I've used, I use a zero-argument function very
occasionally but I can't recall a case where I used a lambda (lambdas
are kinda annoying). Adding the parens at the call is easy to forget,
but not that huge a burden either. The only use case I can think of
offhand is where the default is such an expensive computation that you
don't want it done unless its result is actually used, and it might
not be:
def foo(flag, arg=None, *otherargs):
if flag:
if arg is None:
arg = expensive_function
return arg(*other_args)
else:
return a_constant
On the one hand, it seems likely that a very expensive function is
expensive enough to avoid at the cost of an obscure default and an if
in the definition of foo. On the other the whole scenario seems
rather contrived and not worth syntax.
I don't find that burdensome enough to want syntax, and I can't guess
how quickly I'd start using it if available, that probably depends on
how often the community seems to be using it (ie, in the code I'm
reading).
To be honest, I think this is the real flaw with the proposal. It's
somewhat attractive in theory, adding
a short form for something which people do tend to write out
"longhand" at the moment. But the saving is
relatively small, and there are a number of vaguely annoying edge
cases that probably don't come
up often, but overall just push the proposal into the "annoyingly
complicated" area. The net result is that
we get something that *might* help with a minor annoyance, but the
cost in (theoretical, but necessary)
complexity is just a bit too high.
Paul
I think the example that Steven gave, and Stephen approximately repeats is
good.
def func(items=[], n=later len(items)):
items.append("Hello")
print(n)
func()
I guess Chris will say 0 and David will say 1, but I might be wrong about
either of them.
This is correct. And even though using a (soft) keyword like this gets me
to -0, the semantics I want indeed are different. I only want the binding
to be evaluated when it is referenced. If it never gets referenced, the
compassion time asked side effects are skipped.
Of course, the first line of the function body could be `n = n`. But adding
that line isn't so much different from starting with `if n is None: n = ...
`
As to Stephen's comments on not having used the "evaluate on reference"
pattern, that's pretty much not having used dask.deferred.
The difference is that with Dask (or Haskell) everything stays lazy until
you explicitly call `.compute()` on something in the DAG of operations. I'd
prefer not to need that.
But then my not-a-proposal would need a way to have "a reference that isn't
a reference". I think the same keywords works.
def func(items=[], n=later len(items)):
items.append("Hello")
n = later n**3 # n remains lazy
# ... more stuff
print(n) # actually evaluate the cube of length
On Sat, Jun 18, 2022, 10:52 AM Stephen J. Turnbull <
stephenjturnbull@gmail.com> wrote:
def foo(arg_with_new_empty_list_default=None):
if arg_with_new_empty_list_default is None:
arg_with_new_empty_list_default = []
# do stuff
I can't really guess how useful the "use point" version would be. It's not
a pattern I've used, I use a zero-argument function very occasionally but I
can't recall a case where I used a lambda
The bar for adding a new hard keyword to Python is very high.
Likewise for new syntax.
I would suggest less so, provided that it was previously illegal,
because it's backward-compatible, unlike a new keyword.
The suggestion is to add a new keyword to a PEP which absolutely doesn't
need it,
*shrug*
Well, I have heard many times how high the bar is, so I'm surprised that
suggesting a completely unnecessary one gets no more from you than a shrug.
The match...case statement didn't "need" keywords either, we could have
picked symbols instead if we wanted to look like APL. Remember that
keywords have advantages as well as disadvantages. Given the existence
of community support for keywords, the PEP should make the case that
symbols are better in this case.
Even if that's only "a majority prefer symbols".
I did mention the readability issue towards the end of my post.
on the grounds that it **might** (**not** would - we can't know
without a spec) give compatibility with some fictional vapourware which
- for all people keep talking about it - hasn't happened in years, isn't
happening (AFAIK nobody is working on it), doesn't have anything
remotely close to even an outline specification (people disagree as to
what it should do), very likely never will happen, and at best won't
happen for years.
I think that is broadly accurate. Harsh but fair: nobody has a concrete
plan for generalising "defer" keyword would do. It is still vapourware.
Thank you.
def f(x = later -y):
Is that a late-bound default of -y? Bad luck; it's already legal syntax
for an early-bound default of `later` minus `y`.
Good catch.
Thank you.
Late-bound defaults are meant to be evaluated at function
call time (and in particular, not some way down in the function body when
the parameter gets used).
Not necessarily.
I don't recall if this has been raised in this thread before, but it is
possible to delay the evaluation of the default value until it is
actually needed. I believe that this is how Haskell operates pretty much
everywhere. (Haskell experts: do I have that correct?)
Sorry, but what on earth is the relevance of this? Python is not Haskell.
And it can be vital that the evaluation of late-default values is *not*
delayed (it should be guaranteed if PEP 671 is accepted).
Example 1:
def f(startTime => timeNow()):
< 10-minute calculation >
endTime = timeNow()
print('Elapsed time:', endTime-startTime) # Help, why is this
On 18/06/2022 03:28, Steven D'Aprano wrote:
printing "Elapsed time: -0.001"?
Example 2:
def g(x, y=>x**2):
x = x+1
print(y)
Best wishes
Rob Cliffe
Sorry, but I think all this talk about lazy evaluation is a big red herring:
(1) Python is not Haskell or Dask.
(2) Lazy evaluation is something Python doesn't have, and would be
a HUGE amount of work for Chris (or anyone) to implement (much more, I
would think, than he has already put into his reference implementation
of PEP 671). It's effectively asking for an implementation of that
deferred evaluation vapourware that people keep talking about. It's not
going to happen in a foreseeable time frame. And in the unlikely event
that Chris (or someone) DID implement it, I expect there would be a
chorus of "No, no, that's not how (I think) it should work at all".
(3) Late-bound defaults that are evaluated at function call time,
as per PEP 671, give you an easy way of doing something that at present
needs one of a number of workarounds (such as using sentinel values) all
of which have their drawbacks or awkward points.
(4) The guarantee that a late-bound default WILL be executed at
function call time, can be useful, even essential (it could be
time-dependent or it could depend on the values - default or otherwise -
of other parameters whose values might be changed in the function
body). Sure, I appreciate that there are times when you might want to
defer the evaluation because it is expensive and might not be needed, but:
(5) If you really want deferred evaluation of a parameter default,
you can achieve that by explicitly evaluating it, *at the point you want
it*, in the function body. Explicit is better than implicit.
Sorry again, but IMO discussing any model except one where late-bound
defaults are evaluated at function call time is just adding FUD.
I suppose I'll be accused again of trying to censor this thread for
saying that. Well, as Chris said (as far as I recall ATM) there is a
difference between discussing variants of a proposal and discussing
orthogonal proposals (which could be raised in a separate thread). And
IMO lazy evaluation IS a different, orthogonal proposal. There's
nothing in the PEP about it.
Best wishes
Rob Cliffe
On 18/06/2022 16:42, David Mertz, Ph.D. wrote:
I guess Chris will say 0 and David will say 1, but I might be wrong
about either of them.
This is correct. And even though using a (soft) keyword like this gets
me to -0, the semantics I want indeed are different. I only want the
binding to be evaluated when it is referenced. If it never gets
referenced, the compassion time asked side effects are skipped.
Of course, the first line of the function body could be `n = n`. But
adding that line isn't so much different from starting with `if n is
None: n = ... `
As to Stephen's comments on not having used the "evaluate on
reference" pattern, that's pretty much not having used dask.deferred.
The difference is that with Dask (or Haskell) everything stays lazy
until you explicitly call `.compute()` on something in the DAG of
operations. I'd prefer not to need that.
But then my not-a-proposal would need a way to have "a reference that
isn't a reference". I think the same keywords works.
def func(items=[], n=later len(items)):
items.append("Hello")
n = later n**3 # n remains lazy
# ... more stuff
print(n) # actually evaluate the cube of length
On Sat, Jun 18, 2022, 10:52 AM Stephen J. Turnbull
wrote:
Chris wants syntax for the common pattern
def foo(arg_with_new_empty_list_default=None):
if arg_with_new_empty_list_default is None:
arg_with_new_empty_list_default = []
# do stuff
I can't really guess how useful the "use point" version would be.
It's not a pattern I've used, I use a zero-argument function very
occasionally but I can't recall a case where I used a lambda
Sorry again, but IMO discussing any model except one where late-bound
defaults are evaluated at function call time is just adding FUD.
It's definitely rude to repeatedly state that anyone who's opinion is
different from yours is "adding FUD" and doesn't belong in the thread.
Stephen, and Steven, and Paul, and I all perfectly well understand what
"evaluated at function call time" means.
It's a way to spell `if arg is sentinel: arg = ...` using slightly fewer
characters, and moving an expression from the body to the signature.
I won't stoop to saying that advocating what you do is FUD. I can even
understand why someone would want that in Python.
I'm still -1 because I don't think the purpose alone is close to worth the
cost of new syntax... And especially not using sigils that are confusing to
read in code.
The topic of "late binding in function signatures" simply isn't
*orthogonal* to "late binding in the general sense." Yes, they are
distinct, but very closely adjacent.
Sorry again, but IMO discussing any model except one where late-bound defaults are evaluated at function call time is just adding FUD.
It's definitely rude to repeatedly state that anyone who's opinion is different from yours is "adding FUD" and doesn't belong in the thread.
Stephen, and Steven, and Paul, and I all perfectly well understand what "evaluated at function call time" means.
It's a way to spell `if arg is sentinel: arg = ...` using slightly fewer characters, and moving an expression from the body to the signature.
I won't stoop to saying that advocating what you do is FUD. I can even understand why someone would want that in Python.
I'm still -1 because I don't think the purpose alone is close to worth the cost of new syntax... And especially not using sigils that are confusing to read in code.
The topic of "late binding in function signatures" simply isn't *orthogonal* to "late binding in the general sense." Yes, they are distinct, but very closely adjacent.
Every argument you've just made is ALSO an argument against function
defaults in general. Do you think that they aren't worth syntax
either?
ChrisA
I'm still -1 because I don't think the purpose alone is close to worth
the cost of new syntax... And especially not using sigils that are
confusing to read in code.
The topic of "late binding in function signatures" simply isn't
*orthogonal* to "late binding in the general sense." Yes, they are
distinct, but very closely adjacent.
Every argument you've just made is ALSO an argument against function
defaults in general. Do you think that they aren't worth syntax either?
I honestly can find no connection between what I've argued and "function
defaults in general." It feels like a non sequitur, but I accept that they
somehow connect in your mind.
Way back last Oct when this discussion had it's first round, someone
(probably Steven, but maybe it was someone else) did a survey of a numerous
programming languages, and whether they use early binding or late binding
of default function arguments.
The result was that MOST languages do late (call time) binding, albeit many
of those are compiled languages where early binding doesn't really make
sense. Some, like JavaScript don't have defaults at all, so the approach is
actually similar to Python:
const myfunc = (a, b, c) => {
if (typeof c === 'undefined') { c = ... }
}
That's a sentinel, but worse than Python in several ways. Others like Perl
don't really have named arguments at all, but that weird implicit list you
can pop from.
The one thing that there were ZERO examples of elsewhere was what you
propose: slightly different syntax to allow both early-bound and
late-bound. Admittedly, if Python had done late-bound from 1.0, probably no
one would now have a PEP to add an early-bound spelling variant.
If Python had always been late-bound, I'm sure I'd be perfectly happy with
it today. But sentinels are really easy, and obscure sigils are really
hard. So adding the new spelling just makes Python worse. Beginners have
something needless to learn, and experienced developers have just a little
more cognitive burden all the time.
I'm still -1 because I don't think the purpose alone is close to worth the cost of new syntax... And especially not using sigils that are confusing to read in code.
The topic of "late binding in function signatures" simply isn't *orthogonal* to "late binding in the general sense." Yes, they are distinct, but very closely adjacent.
Every argument you've just made is ALSO an argument against function defaults in general. Do you think that they aren't worth syntax either?
I honestly can find no connection between what I've argued and "function defaults in general." It feels like a non sequitur, but I accept that they somehow connect in your mind.
I'm still -1 because I don't think the purpose alone is close to worth the cost of new syntax...
What purpose? Being able to define the function's signature in its
signature, including the meaning of omitted arguments? That's
definitely the purpose of ALL default arguments, not just late-bound.
And especially not using sigils that are confusing to read in code.
The sigils used are very similar here: "a=b" and "a=>b". Are you
really trying to tell me that "=" is not confusing to read in code,
but "=>" is, or are all such sigils equally confusing? Function
signatures already have several symbols other than those used for
defaults, including *, /, and :, not to mention ( ) and -> which
aren't inside the argument list, but are still part of the function
signature. Programmers are accustomed to working with symbols (with
the possible exception of COBOL and DeScribe Macro Language
programmers), and your argument would work just as well against any
sort of argument defaults.
The topic of "late binding in function signatures" simply isn't *orthogonal* to "late binding in the general sense." Yes, they are distinct, but very closely adjacent.
This one, you specifically mention late binding, so that's the one exception.
Other than that, every one of your arguments could be used to show
that we shouldn't have ANY function argument defaults. Not one of them
justifies having default values without default expressions.
Way back last Oct when this discussion had it's first round, someone (probably Steven, but maybe it was someone else) did a survey of a numerous programming languages, and whether they use early binding or late binding of default function arguments.
The result was that MOST languages do late (call time) binding, albeit many of those are compiled languages where early binding doesn't really make sense. Some, like JavaScript don't have defaults at all, so the approach is actually similar to Python:
const myfunc = (a, b, c) => {
if (typeof c === 'undefined') { c = ... }
}
Actually JavaScript does have argument defaults. (Also, I would write
this as simply "if (c === undefined)", since there's no situation in
which that would fail here.) The obvious syntax is equivalent to what
you wrote:
const myfunc = (a, b, c=...) => ...
That's a sentinel, but worse than Python in several ways.
Not sure what you mean by "several", but it is indeed a simple
sentinel, which has the downside that you can pass the special value
undefined to the function. So I would say it's exactly equivalent to
Python and the use of None, except that it's late-bound.
Others like Perl don't really have named arguments at all, but that weird implicit list you can pop from.
JavaScript, being JavaScript, has that too - which is actually good,
because it lets you distinguish between omitted arguments and those
passed as undefined.
I want to do *better* than those languages, not to use them as a pass mark.
The one thing that there were ZERO examples of elsewhere was what you propose: slightly different syntax to allow both early-bound and late-bound. Admittedly, if Python had done late-bound from 1.0, probably no one would now have a PEP to add an early-bound spelling variant.
And yet early-bound argument defaults ARE of value. Maybe they
wouldn't have been worth adding syntax for, but I would be highly
confident that someone would have made a decorator for it:
@freeze_defaults(x=1)
def foo(x): ...
And, since it's hard for a decorator to remap arguments perfectly, it
would have done an imperfect job, but it might have been good enough
to not ask for the feature.
If Python had always been late-bound, I'm sure I'd be perfectly happy with it today. But sentinels are really easy, and obscure sigils are really hard. So adding the new spelling just makes Python worse. Beginners have something needless to learn, and experienced developers have just a little more cognitive burden all the time.
Sentinels are only easy when you're accustomed to them AND you're
never running into the problems that they introduce. When they do,
there are many different ways around the problem, and not one of them
is perfect. Which means that those reading the code have to figure it
out piece by piece, instead of simply reading the function signature
and understanding it.
So your entire argument is "we don't have it, so we don't need it",
aside from this one small consideration, that no other language has
ever offered both options. Finding a good syntax that is sufficiently
similar to the existing "x=dflt" syntax is the key here. People should
be able to read a signature as a whole, without having to also read
several lines of decorator and/or the beginning of the function body,
and immediately know how to go about calling the function.
Python is the only language I know of that offers both eager and lazy
ways to build a list from another list using a transformation
expression. Most languages that have a map() function will give either
eager or lazy, but not both. Python lets you write a list comp or a
genexp, and the only distinction is the type of bracket used to
surround them. Is that a bad thing? No! It's a very good thing! Python
offers more power, with a very simple way to select which one you
want. Even if literally zero other languages offer that feature, it's
still of value to Python.
Python IS allowed to be the first to do something.
ChrisA
Unfortunately, in relation to this PEP, I find your arguments tend to be
sophistical. They ate not generally so in other threads, but for whatever
reason your attachment to this has a different quality.
There's an interesting point you raise though. You seem to feel that
closely related meanings should have similar looking sigils. I'm not sure
my opinion is *opposite *, but it's definitely more that subtly different
semantics should not be marked by easily visually confusable sigils.
Under PEP 671, a single line of a function signature might contain '=',
':=', '=>', '->', ':', and '=='. Obviously it can have other symbols as
well. But those are the ones the most blur into each other visually. 4 of
them have closely related meanings. You don't even need to be contrived to
have such examples.
I guess '>=' also looks "confusable", but it's far less common in
signatures, and the meaning is further away.
Below you seem to try another unconvincing reductio ad absurdum to suggest
that I must either want a language with no symbols or support this PEP.
I think the cognitive complexity of a line with sigils is somewhere around
quadratic or cubic on the number of distinct sigils. But when several look
similar, it quickly tends toward the higher end. And when several have
related meanings, it's harder still to read.
When I write an expression like 'a - b * c / d**e + f' that also has a
bunch of symbols. But they are symbols that:
- look strongly distinct
- have meanings familiar from childhood
- have strongly different meanings (albeit all related to arithmetic)
Yes, I probably have to think for a while about operator precedence to make
sure I understand it. Probably I'll add some redundant parens if I can
edit the code. But the cognitive burden of the sigils remains FAR lower
than what would occur with regularity under PEP 671.
I already dislike the walrus operator in signature context for that
reason... although I think it's great for e.g. 'while a := getdata() > 0:
...'
On Sun, Jun 19, 2022, 10:31 AM Chris Angelico wrote:
The topic of "late binding in function signatures" simply isn't
*orthogonal* to "late binding in the general sense." Yes, they are
distinct, but very closely adjacent.
Every argument you've just made is ALSO an argument against function
defaults in general. Do you think that they aren't worth syntax either?
I honestly can find no connection between what I've argued and "function
defaults in general." It feels like a non sequitur, but I accept that they
somehow connect in your mind.
I'm still -1 because I don't think the purpose alone is close to worth
the cost of new syntax...
What purpose? Being able to define the function's signature in its
signature, including the meaning of omitted arguments? That's
definitely the purpose of ALL default arguments, not just late-bound.
And especially not using sigils that are confusing to read in code.
The sigils used are very similar here: "a=b" and "a=>b". Are you
really trying to tell me that "=" is not confusing to read in code,
but "=>" is, or are all such sigils equally confusing? Function
signatures already have several symbols other than those used for
defaults, including *, /, and :, not to mention ( ) and -> which
aren't inside the argument list, but are still part of the function
signature. Programmers are accustomed to working with symbols (with
the possible exception of COBOL and DeScribe Macro Language
programmers), and your argument would work just as well against any
sort of argument defaults.
The topic of "late binding in function signatures" simply isn't
*orthogonal* to "late binding in the general sense." Yes, they are
distinct, but very closely adjacent.
This one, you specifically mention late binding, so that's the one
exception.
Other than that, every one of your arguments could be used to show
that we shouldn't have ANY function argument defaults. Not one of them
justifies having default values without default expressions.
Way back last Oct when this discussion had it's first round, someone
(probably Steven, but maybe it was someone else) did a survey of a numerous
programming languages, and whether they use early binding or late binding
of default function arguments.
The result was that MOST languages do late (call time) binding, albeit
many of those are compiled languages where early binding doesn't really
make sense. Some, like JavaScript don't have defaults at all, so the
approach is actually similar to Python:
const myfunc = (a, b, c) => {
if (typeof c === 'undefined') { c = ... }
}
Actually JavaScript does have argument defaults. (Also, I would write
this as simply "if (c === undefined)", since there's no situation in
which that would fail here.) The obvious syntax is equivalent to what
you wrote:
That's a sentinel, but worse than Python in several ways.
Not sure what you mean by "several", but it is indeed a simple
sentinel, which has the downside that you can pass the special value
undefined to the function. So I would say it's exactly equivalent to
Python and the use of None, except that it's late-bound.
Others like Perl don't really have named arguments at all, but that
weird implicit list you can pop from.
JavaScript, being JavaScript, has that too - which is actually good,
because it lets you distinguish between omitted arguments and those
passed as undefined.
I want to do *better* than those languages, not to use them as a pass mark.
The one thing that there were ZERO examples of elsewhere was what you
propose: slightly different syntax to allow both early-bound and
late-bound. Admittedly, if Python had done late-bound from 1.0, probably no
one would now have a PEP to add an early-bound spelling variant.
And yet early-bound argument defaults ARE of value. Maybe they
wouldn't have been worth adding syntax for, but I would be highly
confident that someone would have made a decorator for it:
@freeze_defaults(x=1)
def foo(x): ...
And, since it's hard for a decorator to remap arguments perfectly, it
would have done an imperfect job, but it might have been good enough
to not ask for the feature.
If Python had always been late-bound, I'm sure I'd be perfectly happy
with it today. But sentinels are really easy, and obscure sigils are really
hard. So adding the new spelling just makes Python worse. Beginners have
something needless to learn, and experienced developers have just a little
more cognitive burden all the time.
Sentinels are only easy when you're accustomed to them AND you're
never running into the problems that they introduce. When they do,
there are many different ways around the problem, and not one of them
is perfect. Which means that those reading the code have to figure it
out piece by piece, instead of simply reading the function signature
and understanding it.
So your entire argument is "we don't have it, so we don't need it",
aside from this one small consideration, that no other language has
ever offered both options. Finding a good syntax that is sufficiently
similar to the existing "x=dflt" syntax is the key here. People should
be able to read a signature as a whole, without having to also read
several lines of decorator and/or the beginning of the function body,
and immediately know how to go about calling the function.
Python is the only language I know of that offers both eager and lazy
ways to build a list from another list using a transformation
expression. Most languages that have a map() function will give either
eager or lazy, but not both. Python lets you write a list comp or a
genexp, and the only distinction is the type of bracket used to
surround them. Is that a bad thing? No! It's a very good thing! Python
offers more power, with a very simple way to select which one you
want. Even if literally zero other languages offer that feature, it's
still of value to Python.
Python IS allowed to be the first to do something.
Unfortunately, in relation to this PEP, I find your arguments tend to be sophistical. They ate not generally so in other threads, but for whatever reason your attachment to this has a different quality.
I've no idea what you mean by "sophistical" here. Please explain?
There's an interesting point you raise though. You seem to feel that closely related meanings should have similar looking sigils. I'm not sure my opinion is *opposite *, but it's definitely more that subtly different semantics should not be marked by easily visually confusable sigils.
Under PEP 671, a single line of a function signature might contain '=', ':=', '=>', '->', ':', and '=='. Obviously it can have other symbols as well. But those are the ones the most blur into each other visually. 4 of them have closely related meanings. You don't even need to be contrived to have such examples.
Yes, but only some of those are actually part of the signature itself.
I mean, if you put a lambda function into the default value, you could
have even more colons:
def f(x:int=lambda x: x+1, y:object=(_default:=object())): ...
But that's just what happens when syntax is nestable. We almost never
restrict code to using an arbitrary subset of Python syntax just
because otherwise there'd be similar symbols in the same vicinity.
Even when it's a problem for the parser, the solution is often just
parentheses.
Below you seem to try another unconvincing reductio ad absurdum to suggest that I must either want a language with no symbols or support this PEP.
I think the cognitive complexity of a line with sigils is somewhere around quadratic or cubic on the number of distinct sigils. But when several look similar, it quickly tends toward the higher end. And when several have related meanings, it's harder still to read.
It shouldn't need to be. Once you know how expressions are built up,
it should give linear complexity.
Did the introduction of the @ (matrix multiplication) operator to
Python increase the language's complexity multiplicatively, or
additively? Be honest now: are your programs more confusing to read
because @ could have been *, because @ could have been +, because @
could have been ==, etc etc etc, or is it only that @ is one new
operator, additively with the other operators?
Even when syntactically it's multiplicative, it's often possible to be
merely additive in cognitive burden. Augmented assignment broadly
functions as a single feature - "x #= y" parallels "x = x # y" for any
operator where that's legal.
When I write an expression like 'a - b * c / d**e + f' that also has a bunch of symbols. But they are symbols that:
- look strongly distinct
- have meanings familiar from childhood
- have strongly different meanings (albeit all related to arithmetic)
The double asterisk wasn't one that I used in my childhood, yet in
programming, I simply learned it and started using it. What happens is
that known concepts are made use of to teach others. It's the same
with every feature, and this one is no different: if you already
understand "def f(x=1)", it is only a small step to "def f(x=>[])",
and everything you already know is still valid.
Yes, I probably have to think for a while about operator precedence to make sure I understand it. Probably I'll add some redundant parens if I can edit the code. But the cognitive burden of the sigils remains FAR lower than what would occur with regularity under PEP 671.
Is that simply because you already are familiar with those operators,
or is there something inherently different about them? Would it really
be any different?
I already dislike the walrus operator in signature context for that reason... although I think it's great for e.g. 'while a := getdata() > 0: ...'
Then don't use it in a signature. That's fine. Personally, I've never
used the "def f(x=_sentinel:=object())" trick, because it has very
little value - it makes the function header carry information that
actually isn't part of the function signature (that the object is also
in the surrounding context as "_sentinel" - the function's caller
can't use that information), and doesn't have any real advantages over
just putting it a line above.
But that's because of what the feature does, not how it's spelled. An
expression is an expression. Any competent programmer should be able
to read them as such.
ChrisA
I've no idea what you mean by "sophistical" here. Please explain?
1. of or characteristic of sophists or sophistry
2. clever and plausible, but unsound and tending to mislead
a sophistical argument
3. using sophistry
Plato's dialog _The Sophist_ is good background.
I guess '>=' also looks "confusable", but it's far less common in
signatures, and the meaning is further away.
It's no less valid than your other examples, nor less common (why
would you have "==" in a function signature, for instance?).
I guess I probably use `==` more often in function calls and signatures, on
reflection. In call, I use `==` quite often to pass some boolean switch
value, and `>=` much less often. Obviously, I am aware that `>=` also
produces a boolean result, and YMMV on how often comparing for equality and
inequality expresses the flag you want.
In signature, I'd really only use it, I reckon, as a "default default." E.g.
def frobnicate(data, verbose=os.environ.get('LEVEL')==loglevel.DEBUG):
...
This supposes I have an environmental setting for verbosity that I usually
want to use, but might override that on a particular call.
I think the cognitive complexity of a line with sigils is somewhere
around quadratic or cubic on the number of distinct sigils. But when
several look similar, it quickly tends toward the higher end. And when
several have related meanings, it's harder still to read.
It shouldn't need to be. Once you know how expressions are built up, it
should give linear complexity.
I'm not talking about the big-O running time of some particular engine that
parses a BNF grammar here. I'm talking about actual human brains, which
work differently.
I don't have data for my quadratic and cubic guesses. Just 40 years of my
own programming experience, and about the same amount of time watching
other programmers.
It would be relatively easy to measure if one wanted to. But it's a
cognitive psychology experiment. You need to get a bunch of people in
rooms, and show them lots of code lines. Then measure error rates and
response times in their answers. That sort of thing. The protocol for
this experiment would need to be specified more carefully, of course. But
it *is* the kind of thing that can be measured in human beings.
So my (very strong) belief is that a human being parsing a line with 5
sigils in it will require MUCH MORE than 25% more effort than parsing a
line with 4 sigils in it. As in, going from 4 to 5 distinct sigils in the
same line roughly DOUBLES cognitive load. Here the distinctness is
important; it's not at all hard to read:
a + b + c + d + e + f + g + h + i
And in the weeds, the particular sigils involved (and even the font they're
rendered in) will make a difference too. As well it matters the "semantic
proximity" of the various operators. And other factors too I'm sure.
Did the introduction of the @ (matrix multiplication) operator to
Python increase the language's complexity multiplicatively, or
additively? Be honest now: are your programs more confusing to read
because @ could have been *, because @ could have been +, because @
could have been ==, etc etc etc, or is it only that @ is one new
operator, additively with the other operators?
I'm not sure how much you know about the background of this in the NumPy
world. While other libraries have also now used that operator, NumPy was
the driving force.
In the old days, if I wanted to do a matrix multiply, I would either do:
A_matrix = np.matrix(A)
B_matrix = np.matrix(B)
result = A_matrix * B_matrix
Or alternately:
result = np.dot(A, B)
Neither of those approaches are terrible, but in more complex expressions
where the dot product is only part of the expression, indeed `A @ B` reads
better.
And yes, expressions on NumPy arrays will often use a number of those
arithmetic operators I learned in grade school as well as `@`. But
generally, the mathematics expressed in NumPy code is irreducible
complexity. It's not necessarily easy to parse visually, but it *IS* the
underlying mathematics.
When I write an expression like 'a - b * c / d**e + f' that also has a
bunch of symbols. But they are symbols that:
- look strongly distinct
- have meanings familiar from childhood
- have strongly different meanings (albeit all related to arithmetic)
The double asterisk wasn't one that I used in my childhood, yet in
programming, I simply learned it and started using it. What happens is
that known concepts are made use of to teach others.
I didn't learn the double asterisk in school either. That I had to learn
in programming languages. I actually prefer those programming languages
that use `^` for exponentiation (in that one aspect, not overall more than
Python), because it's more reminiscent of superscript.
Is that simply because you already are familiar with those operators,
or is there something inherently different about them? Would it really
be any different?
It's a mixture of familiarity and actual visual distinctness. `/` and `+`
really do just *look different*. In contrast `:=` and `=` just really look
similar.
Then don't use it in a signature. That's fine. Personally, I've never
used the "def f(x=_sentinel:=object())" trick, because it has very
little value
I agree with you here. I am pretty sure I've never used it either. But
most of the code I read isn't code I wrote myself.
In the case of the walrus, I'm not even saying that I think it should have
been prohibited in that context. Just discouraged in style guides. While
I understand the handful of cases where walrus-in-signature has a certain
utility, I would be happy enough to forgo those. But my concern has more
to do with not limiting expressions/symbols/keywords to special one-off
contexts. That's a relative thing, obviously, for example `@deco` can
really only happen in one specific place, and I like decorators quite a
lot. But where possible, symbols or words that can occur in expressions
should be available to all kinds of program contexts, and have *pretty
much* the same meaning in all of them. And yes, you can find other
exceptions to this principle in Python.
This actually circles back to why I would greatly prefer `def
myfunc(a=later some_expression())` as a way to express late binding of a
default argument. Even though you don't like a more generalized deferred
computation, and a version of PEP 671 that used a soft keyword would not
automatically create such broader use, in my mind the option of later more
general use is left open by that approach.
- it makes the function header carry information that
actually isn't part of the function signature (that the object is also
in the surrounding context as "_sentinel" - the function's caller
can't use that information), and doesn't have any real advantages over
just putting it a line above.
I do think some of this comes down to something I find somewhat mythical.
99%+ of the time that I want to use a sentinel, `None` is a great one. Yes
I understand that a different one is required occasionally. But basically,
`arg=None` means "late binding" in almost all cases. So that information
is ALREADY in the header of almost all the functions I deal with.
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
I guess '>=' also looks "confusable", but it's far less common in signatures, and the meaning is further away.
It's no less valid than your other examples, nor less common (why
would you have "==" in a function signature, for instance?).
I guess I probably use `==` more often in function calls and signatures, on reflection. In call, I use `==` quite often to pass some boolean switch value, and `>=` much less often. Obviously, I am aware that `>=` also produces a boolean result, and YMMV on how often comparing for equality and inequality expresses the flag you want.
In signature, I'd really only use it, I reckon, as a "default default." E.g.
This supposes I have an environmental setting for verbosity that I usually want to use, but might override that on a particular call.
Okay. When it comes to finding causes for the difficulty of reading
your function signatures, I think this is a far greater one *in a
vacuum* than having support for late-bound defaults. Maybe it's not a
problem *for you* because you're accustomed to reading that as a
single token, but that can be true of any sequence.
Is there any value in not putting that into a global constant?
I think the cognitive complexity of a line with sigils is somewhere around quadratic or cubic on the number of distinct sigils. But when several look similar, it quickly tends toward the higher end. And when several have related meanings, it's harder still to read.
It shouldn't need to be. Once you know how expressions are built up, it should give linear complexity.
I'm not talking about the big-O running time of some particular engine that parses a BNF grammar here. I'm talking about actual human brains, which work differently.
Yes, and human brains are fine with adding more options, as long as
they work the same way that other options do.
I don't have data for my quadratic and cubic guesses. Just 40 years of my own programming experience, and about the same amount of time watching other programmers.
It would be relatively easy to measure if one wanted to. But it's a cognitive psychology experiment. You need to get a bunch of people in rooms, and show them lots of code lines. Then measure error rates and response times in their answers. That sort of thing. The protocol for this experiment would need to be specified more carefully, of course. But it *is* the kind of thing that can be measured in human beings.
So my (very strong) belief is that a human being parsing a line with 5 sigils in it will require MUCH MORE than 25% more effort than parsing a line with 4 sigils in it. As in, going from 4 to 5 distinct sigils in the same line roughly DOUBLES cognitive load. Here the distinctness is important; it's not at all hard to read:
a + b + c + d + e + f + g + h + i
Okay. You run a test, and let's see how it goes. At the moment, all
you have is a single data point: yourself. Specifically, yourself with
all your current knowledge. I think it's a little biased. :)
Did the introduction of the @ (matrix multiplication) operator to
Python increase the language's complexity multiplicatively, or
additively? Be honest now: are your programs more confusing to read
because @ could have been *, because @ could have been +, because @
could have been ==, etc etc etc, or is it only that @ is one new
operator, additively with the other operators?
I'm not sure how much you know about the background of this in the NumPy world. While other libraries have also now used that operator, NumPy was the driving force.
In the old days, if I wanted to do a matrix multiply, I would either do:
A_matrix = np.matrix(A)
B_matrix = np.matrix(B)
result = A_matrix * B_matrix
Or alternately:
result = np.dot(A, B)
Neither of those approaches are terrible, but in more complex expressions where the dot product is only part of the expression, indeed `A @ B` reads better.
Regardless, the @ operator is now available *everywhere* in Python.
Does it quadratically increase cognitive load?
When I write an expression like 'a - b * c / d**e + f' that also has a bunch of symbols. But they are symbols that:
- look strongly distinct
- have meanings familiar from childhood
- have strongly different meanings (albeit all related to arithmetic)
The double asterisk wasn't one that I used in my childhood, yet in
programming, I simply learned it and started using it. What happens is
that known concepts are made use of to teach others.
I didn't learn the double asterisk in school either. That I had to learn in programming languages. I actually prefer those programming languages that use `^` for exponentiation (in that one aspect, not overall more than Python), because it's more reminiscent of superscript.
In other words: you had to learn it. Just like everything else. So
"have meanings familiar from childhood" only takes you so far; we have
to learn everything.
Is that simply because you already are familiar with those operators,
or is there something inherently different about them? Would it really
be any different?
It's a mixture of familiarity and actual visual distinctness. `/` and `+` really do just *look different*. In contrast `:=` and `=` just really look similar.
That's because they ARE similar. Assignment is assignment. It's the
same as how "x *= 4" looks like a combination of multiplication and
assignment, because it is. Cognitive load is linear when the brain can
find patterns.
This actually circles back to why I would greatly prefer `def myfunc(a=later some_expression())` as a way to express late binding of a default argument. Even though you don't like a more generalized deferred computation, and a version of PEP 671 that used a soft keyword would not automatically create such broader use, in my mind the option of later more general use is left open by that approach.
So, you prefer a spelling that makes it less likely that people will
use it. In other words, you hate the idea, and are asking me to worsen
my own idea to make it less useful. Is that right?
Then, no. I will continue to promote the "=>" form, since it doesn't
give the false impression that "later some_expression()" could stand
on its own.
- it makes the function header carry information that
actually isn't part of the function signature (that the object is also
in the surrounding context as "_sentinel" - the function's caller
can't use that information), and doesn't have any real advantages over
just putting it a line above.
I do think some of this comes down to something I find somewhat mythical. 99%+ of the time that I want to use a sentinel, `None` is a great one. Yes I understand that a different one is required occasionally. But basically, `arg=None` means "late binding" in almost all cases. So that information is ALREADY in the header of almost all the functions I deal with.
As with the loglevel example above, you're accustomed to seeing
"arg=None" as "late binding". But:
1) You don't get the information about WHAT it late-binds to
2) This is only true in your own code, and other code might actually
use None in a completely different way
In a function signature "def bisect(stuff, lo=0, hi=None)", you don't
know what the hi value actually defaults to. Even if it's obvious that
it is late-bound, that is actually the least relevant piece of
information that could be given! Instead of saying what the default
is, you simply say "oh and hey, this has a default that I'm not
telling you about". It would be far more useful to have the default
there, but then have some extra tag somewhere else that says whether
it's late-bound or early-bound, than to have it this way around.
ChrisA
Regardless, the @ operator is now available *everywhere* in Python. Does
it quadratically increase cognitive load?
Yeah, probably about that much. Other than NumPy or closely related array
libraries, I don't know that many other uses. I think I saw something on
PyPI that used it as an email thing, where obviously it has some
familiarity. But in that case, the lines it occurs on probably have no
more than one or two other sigils.
In the numeric stuff, if I have:
newarray = (A @ B) | (C / D) + (E - F)
That's @, |, /, +, and -. So 5 operators, and 25 "complexity points". If
I added one more operator, 36 "complexity points" seems reasonable. And if
I removed one of those operators, 16 "complexity points" feels about right.
In a function signature "def bisect(stuff, lo=0, hi=None)", you don't
know what the hi value actually defaults to. Even if it's obvious that
it is late-bound
Sure, knowing what `hi` defaults to *could be useful*. I'm sure if I used
that function I would often want to know... and also often just assume the
default is "something sensible." I just don't think that "could be useful"
as a benefit is nearly as valuable as the cost of a new sigil and a new
semantics adding to the cognitive load of Python.
For example, it also "could be useful" to have syntax that indicated the
(expected) big-O complexity of that function. But whatever that syntax
was, I really doubt it would be worth the extra complexity in the language
vs. just putting that info in the docstring.
Let's look at a function that has a lot of late-bound default arguments:
pd.read_csv(
filepath_or_buffer: 'FilePath | ReadCsvBuffer[bytes] |
ReadCsvBuffer[str]',
sep=,
delimiter=None,
header='infer',
names=,
index_col=None,
usecols=None,
squeeze=None,
prefix=,
mangle_dupe_cols=True,
dtype: 'DtypeArg | None' = None,
engine: 'CSVEngine | None' = None,
converters=None,
true_values=None,
false_values=None,
skipinitialspace=False,
skiprows=None,
skipfooter=0,
nrows=None,
na_values=None,
keep_default_na=True,
na_filter=True,
verbose=False,
skip_blank_lines=True,
parse_dates=None,
infer_datetime_format=False,
keep_date_col=False,
date_parser=None,
dayfirst=False,
cache_dates=True,
iterator=False,
chunksize=None,
compression: 'CompressionOptions' = 'infer',
thousands=None,
decimal: 'str' = '.',
lineterminator=None,
quotechar='"',
quoting=0,
doublequote=True,
escapechar=None,
comment=None,
encoding=None,
encoding_errors: 'str | None' = 'strict',
dialect=None,
error_bad_lines=None,
warn_bad_lines=None,
on_bad_lines=None,
delim_whitespace=False,
low_memory=True,
memory_map=False,
float_precision=None,
storage_options: 'StorageOptions' = None,
I'd have to look through the implementation, but my guess is that quite a
few of the 25 late-bound defaults require calculations to set that take
more than one line of code. I really don't WANT to know more than "this
parameter is calculated according to some logic, perhaps complex logic" ...
well, unless I think it pertains to something I genuinely want to
configure, in which case I'll read the docs.
--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
Is there any value in not putting that into a global constant?
Probably not. I was just inventing an ad hoc example to show what I meant. I didn't search any actual repos I work on for real-life examples.
Ah okay. Well, if that WERE a real example, I would recommend giving
it a name. (Also, it's probably going to end up using >= rather than
==, so that the verbosity of any function can be set to a minimum
level, so there'd be more complexity, thus making it even more useful
to make it some sort of constant.)
Regardless, the @ operator is now available *everywhere* in Python. Does it quadratically increase cognitive load?
Yeah, probably about that much. Other than NumPy or closely related array libraries, I don't know that many other uses. I think I saw something on PyPI that used it as an email thing, where obviously it has some familiarity. But in that case, the lines it occurs on probably have no more than one or two other sigils.
In the numeric stuff, if I have:
newarray = (A @ B) | (C / D) + (E - F)
That's @, |, /, +, and -. So 5 operators, and 25 "complexity points". If I added one more operator, 36 "complexity points" seems reasonable. And if I removed one of those operators, 16 "complexity points" feels about right.
For my part, I would say that it's quite the opposite. This is three
parenthesized tokens, each of which contains two things combined in a
particular way. That's six 'things' combined in particular ways.
Cognitive load is very close to this version:
newarray = (A * B) + (C * D) + (E * F)
even though this uses a mere two operators. It's slightly more, but
not multiplicatively so. (The exact number of "complexity points" will
depend on what A through F represent, but the difference between "all
multiplying and adding" and "five distinct operators" is only about
three points.)
So unless you have a study showing this, I would say we each have a
single data point - ourselves - and it's basically useless data.
In a function signature "def bisect(stuff, lo=0, hi=None)", you don't
know what the hi value actually defaults to. Even if it's obvious that
it is late-bound
Sure, knowing what `hi` defaults to *could be useful*. I'm sure if I used that function I would often want to know... and also often just assume the default is "something sensible." I just don't think that "could be useful" as a benefit is nearly as valuable as the cost of a new sigil and a new semantics adding to the cognitive load of Python.
Yes, but "something sensible" could be "len(stuff)", "len(stuff)-1",
or various other things. Knowing exactly which of those will tell you
exactly how to use the function.
Would you say that knowing that lo defaults to 0 is useful
information? You could just have a function signature that merely says
which arguments are mandatory and which are optional, and force people
to use the documentation to determine the behaviour of omitted
arguments. If you accept that showing "lo=0" gives useful information
beyond simply that lo is optional, then is it so hard to accept that
"hi=>len(stuff)" is also immensely valuable?
For example, it also "could be useful" to have syntax that indicated the (expected) big-O complexity of that function. But whatever that syntax was, I really doubt it would be worth the extra complexity in the language vs. just putting that info in the docstring.
That's true; there's always a lot more that could go into a function's
docstring than can fit into its signature. Perhaps, if it's of value
to your project, it would be useful to use a function decorator and
then redefine the return value annotation to (also or instead) inform
you of the complexity. But for information about a single argument,
the only useful place to put it is on the argument itself - either in
the signature, or in a duplicated block in the docstring. And function
defaults are a lot broader in value than algorithmic complexity, which
is irrelevant to a huge number of functions.
I'd have to look through the implementation, but my guess is that quite a few of the 25 late-bound defaults require calculations to set that take more than one line of code. I really don't WANT to know more than "this parameter is calculated according to some logic, perhaps complex logic" ... well, unless I think it pertains to something I genuinely want to configure, in which case I'll read the docs.
Actually, I would guess that most of these default to something that's
set elsewhere. Judging only by the documentation, not actually reading
the source, here's what I can say:
delimiter=>sep, # It's an alias for sep
engine=>???, # seems the default is set elsewhere
na_values=>_DEFAULT_NA_VALUES, # there is a default in the docs
on_bad_lines='error' # seems this has a simple default
For the rest, though, these _do not have_ defaults. Not default
values, not default expressions. There is no code that could be placed
at the top of the function to assign behaviour to them. The None
default value actually means something different from passing in some
other value - for instance, "callable or None" means it actually won't
be calling any function if None is provided.
This function isn't a good showcase of PEP 671 - neither its strengths
nor its weaknesses - because it simply doesn't work with argument
defaults in that way. It might be able to take advantage of it for a
couple of them, but it's certainly not going to change the sheer
number of None-default arguments that it has.
Maybe I'm wrong on that, and maybe you could show the lines of code at
the top of the function that could potentially be converted into
argument defaults, but otherwise, this is simply a function that
potentially does a lot of stuff, and only does the stuff for the
arguments you pass in.
(It could potentially benefit from a way to know whether the argument
was passed or not, but since None is a fine sentinel for all of these
args, there wouldn't be much to gain.)
ChrisA
@Chris
My bottom line, as I wrote before, is that even if this were
introduced, I probably will continue to default to
def foo(arg=None):
if arg is None:
arg = default
in my own code until I start seeing "def foo(arg=>default)" in a lot
of code I read. Since Mailman generally supports about 4 Python
versions, that means I won't see it in Mailman until 2027 or so.
But I'm not George Bush to say "Read my lips: no new (syn)taxes!"
Unless somebody comes up with some new really interesting use case, I
think the suggestion somebody (sorry to somebody!) made earlier to
"Just Do It" and submit to the SC is the right one. Both David and I
are convinced that there is value-added in late binding for new
mutables and defaults that are computed from actual arguments, even if
we're not convinced it's enough. The proposal has plenty of fans, who
*are* convinced and *will* use it. I don't see a prospect for that
new really interesting use case, at least not here on Python-Ideas,
the discussion is just variations on the same themes. On the other
hand, a PEP under consideration may get a little more interest from
the Python-Dev crowd, and obviously the SC itself. They may have use
cases or other improvements to offer.
"Now is better than never." The SC will let you know if the companion
koan is applicable. ;-)
@Chris You may or may not want to read my variations on the themes. ;-)
Chris Angelico writes:
That's @, |, /, +, and -. So 5 operators, and 25 "complexity
points". If I added one more operator, 36 "complexity points"
seems reasonable. And if I removed one of those operators, 16
"complexity points" feels about right.
For my part, I would say that it's quite the opposite. This is three
parenthesized tokens, each of which contains two things combined in a
particular way. That's six 'things' combined in particular ways.
Cognitive load is very close to this version:
newarray = (A * B) + (C * D) + (E * F)
I don't have the studies offhand, but "7 plus or minus 2" is famous
enough, google that and you'll find plenty. I'll bet you even find
"cognitive complexity of mathematical formulae" in the education
literature. (And if not, we should sue all the Departments of
Education in the world for fraud. ;-)
I do have the words: "this is a sum of binary products". This
basically reduces the cognitive complexity to two concepts plus a scan
of the list of variables. Given that they're actually in alphabetical
order, "first variable is A" is enough to reproduce the expression.
That's much simpler than trying to describe David's 5-operator case
with any degree of specificity. Or even just try to reproduce his
formula without a lot of effort to memorize it! Also, just from the
regularity of the form and its expression as an algebraic formula, I
can deduce that almost certainly A, C, and E have the same type, and
B, D, and F have the same type, and very likely those two types are
the same. Not so for the five-operator case, where I would be
surprised if less than 3 types were involved.
Of course, this type information is probably redundant. I probably
remember not only the types, but lots of other attributes of A
through F. But this kind of redundancy is good! It reinforces my
understanding of the expression and the program that surrounds it.
even though this uses a mere two operators. It's slightly more, but
not multiplicatively so. (The exact number of "complexity points" will
depend on what A through F represent, but the difference between "all
multiplying and adding" and "five distinct operators" is only about
three points.)
That may be true for you, but it's definitely not true for my
economics graduate students.
Sure, knowing what `hi` defaults to *could be useful*. I'm sure
if I used that function I would often want to know... and also
often just assume the default is "something sensible." I just
don't think that "could be useful" as a benefit is nearly as
valuable as the cost of a new sigil and a new semantics adding to
the cognitive load of Python.
Yes, but "something sensible" could be "len(stuff)", "len(stuff)-1",
or various other things. Knowing exactly which of those will tell you
exactly how to use the function.
@David: I find the "hi=len(stuff)" along with the "lst=[]" examples
fairly persuasive (maybe moves me to +/- 0).
@Chris: It would be a lot more persuasive if you had a plausible
explicit list of "various other things". Even "len(stuff) - 1" is
kind of implausible, given Python's consistent 0-based indexing
and closed-open ranges (yeah, I know some people like to use the
largest value in the range rather than the least upper bound not
in the range, but I consider that bad style in Python, and they
denote the same semantics). And "len(stuff)" itself is "the
obvious" default. How often is the computed default either
"unobvious" or "has multiple frequently useful values"?
If you accept that showing "lo=0" gives useful information
beyond simply that lo is optional, then is it so hard to accept that
"hi=>len(stuff)" is also immensely valuable?
"lo=0" is not only useful, it's also very close to the minimal
notation for "lo is optional" (and in fact my bet is that Python would
express mere optionality with a keyword such as opt, as in
"bisect(stuff, opt lo, opt hi)", which makes "lo=0" shorter than a
very plausible alternative).
For example, it also "could be useful" to have syntax that
indicated the (expected) big-O complexity of that function.
@David: I think this is a bit sophistical. Big-O is occasionally
relevant to the decision to use a function. But when it is, it's a
very big deal indeed, which justifies at least the amount of effort to
read the source, identify the algorithm, and look it up in Knuth :-).
All you've really proven is that there is some information of positive
value but infrequently useful enough that nobody in this thread would
favor adding syntax for it.
Chris's case is that the value of a defaulted argument is something
you need to know *every* time you use many functions with defaults.
That's considerable, even if we are going to end up concluding "not
worth syntax" or maybe "not quite enough with the proposed syntax,
keep trying".
I'd have to look through the implementation, but my guess is that
quite a few of the 25 late-bound defaults require calculations to
set that take more than one line of code. I really don't WANT to
know more than "this parameter is calculated according to some
logic, perhaps complex logic" ... well, unless I think it
pertains to something I genuinely want to configure, in which
case I'll read the docs.
I agree, and I don't think this "don't care" attitude for defaultable
parameters is limited to this example or even this kind of example
(shall we call it an "Alice's Restaurant parameter list"?)
This function isn't a good showcase of PEP 671 - neither its strengths
nor its weaknesses
I agree with the "set elsewhere" guess, but I think that significantly
*reduces* the number of cases where a late-bound default provides a
substantial improvement over None.
Sorry, but I think all this talk about lazy evaluation is a big red herring:
(1) Python is not Haskell or Dask.
Python is not Haskell, but we stole list comprehensions and pattern
matching from it. Python steals concepts from many languages.
And Python might not be Dask, but Dask is Python.
https://www.dask.org/
(2) Lazy evaluation is something Python doesn't have,
Python has lazily evaluated sequences (potentially infinite sequences)
via generators and iterators. We also have short-circuit evaluation,
which is a form of lazy evaluation. There may be other examples as well.
We may also get lazy importing soon:
https://peps.python.org/pep-0690/
At last one of Python's direct competitors in the scientific community,
R, has lazy evaluation built in.
and would be
a HUGE amount of work for Chris (or anyone) to implement
I don't know how hard it is to implement lazy evaluation, but speaking
with the confidence of the ignorant, I expect not that hard if you don't
care too much about making it super efficient. A lazy expression, or
thunk, is basically just a zero-argument function that the interpreter
knows to call.
If you don't care about getting Haskell levels of efficiency, that's
probably pretty simple to implement.
Rewriting Python from the ground up to be completely lazy like Haskell
would be a huge amount of work. Adding some sort of optional and
explicit laziness, like R and F# and other languages use, would possibly
be little more work than just adding late-bound defaults.
Maybe.
And in the unlikely event
that Chris (or someone) DID implement it, I expect there would be a
chorus of "No, no, that's not how (I think) it should work at all".
The idea is that you plan your feature's semantics before writing an
implementation. Even if you plan to "write one to throw away", and do
exploratory coding, you should still have at least a vague idea of the
desired semantics before you write a single line of code.
(3) Late-bound defaults that are evaluated at function call time,
as per PEP 671, give you an easy way of doing something that at present
needs one of a number of workarounds (such as using sentinel values) all
of which have their drawbacks or awkward points.
Yes, we've read the PEP thank you :-)
Late-bound defaults also have their own drawbacks. It is not a question
of whether this PEP has any advantages. It clearly does! The question is
where the balance of pros versus cons falls.
(4) The guarantee that a late-bound default WILL be executed at
function call time, can be useful, even essential (it could be
time-dependent or it could depend on the values - default or otherwise -
of other parameters whose values might be changed in the function
body).
Okay. But a generalised lazy evaluation mechanism can be used to
implement PEP 671 style evaluation.
Let me see if I can give a good analogy... generalised lazy evaluation
is like having a car that can drive anywhere there is a road, at any
time of the day or night. Late-bound defaults is like having a car that
can only drive to the local mall and back, and only on Thursdays.
That's okay if you want to drive to the local mall on Thursdays, but if
you could only have one option, which would be more useful?
Sure, I appreciate that there are times when you might want to
defer the evaluation because it is expensive and might not be needed, but:
(5) If you really want deferred evaluation of a parameter default,
you can achieve that by explicitly evaluating it, *at the point you want
it*, in the function body. Explicit is better than implicit.
That's not really how lazy evaluation works or why people want it.
The point of lazy evaluation is that computations are transparently and
automatically delayed until you actually need them. Lazy evaluation is
kind of doing the same thing for CPUs as garbage collection does for
memory. GC kinda sorta lets you pretend you have infinite memory (so
long as you don't actually try to use it all at once...). Lazy
evaluation kinda sorta lets you pretend your CPU is infinitely fast (so
long as you don't try to actually do too much all at once).
If you think about the differences between generators and lists, that
might help. A generator isn't really like a list that you just evaluate
a few lines later. Its a completely different way of thinking about
code, and often (but not always) better.
(4) The guarantee that a late-bound default WILL be executed at
function call time, can be useful, even essential (it could be
time-dependent or it could depend on the values - default or otherwise -
of other parameters whose values might be changed in the function
body).
Okay. But a generalised lazy evaluation mechanism can be used to
implement PEP 671 style evaluation.
Let me see if I can give a good analogy... generalised lazy evaluation
is like having a car that can drive anywhere there is a road, at any
time of the day or night. Late-bound defaults is like having a car that
can only drive to the local mall and back, and only on Thursdays.
That's okay if you want to drive to the local mall on Thursdays, but if
you could only have one option, which would be more useful?
Nice analogy. It doesn't hold up.
Consider this function:
def f(stuff, max=>len(stuff)):
stuff.append(1)
print(max)
f([1,2,3])
How would you use lazy evaluation to *guarantee* the behaviour here?
The only way I can imagine doing it is basically the same as I'm
doing: that late-bound argument defaults *have special syntax and
meaning to the compiler*. If they were implemented with some sort of
lazy evaluation object, they would need (a) access to the execution
context, so you can't just use a function; (b) guaranteed evaluation
on function entry, regardless of when - if ever - it gets referred to;
and (c) the ability to put it in the function header. The only one of
those that overlaps with lazy evaluation is (c).
Please stop arguing this point. It is a false analogy and until you
can demonstrate *with code* that there is value in doing it, it is a
massive red herring.
Even if Python does later on grow a generalized lazy evaluation
feature, it will only change the *implementation* of late-bound
argument defaults, not their specification.
ChrisA
Sorry again, but IMO discussing any model except one where
late-bound defaults are evaluated at function call time is just
adding FUD.
It's definitely rude to repeatedly state that anyone who's opinion is
different from yours is "adding FUD" and doesn't belong in the thread.
I was not talking about people whose opinion was different from mine. I
was talking about people who obscured the discussion of a proposal by
talking about a different proposal. And that, IMO, would be rude if it
were done deliberately, though I accept that it wasn't.
Stephen, and Steven, and Paul, and I all perfectly well understand
what "evaluated at function call time" means.
I should jolly well hope so too. I certainly did not intend to suggest
On 19/06/2022 04:42, David Mertz, Ph.D. wrote:
that any of you or anyone else do not understand it. And I can't see
anything in any of my posts that suggests that I did intend that. Do
you think that I did? If so, why? (Please quote where appropriate.)
If I did somehow suggest that, I sincerely apologise.
It's a way to spell `if arg is sentinel: arg = ...` using slightly
fewer characters, and moving an expression from the body to the signature.
Yes, if you want to simplify a bit, basically it is. But it avoids the
trap of the sentinel value being a possible parameter value. And it
would answer a number of Stack Overflow posts on the lines of "Why
doesn't this work [as I expected]?" I don't think that anyone,
including Chris, would say that it allows you to do something that you
can't do already (though I might be wrong, but I believe Python is
already Turing-complete 😁). The virtue of the PEP is that it adds some
convenience and some clarity and some concision. (Concision *is* a
virtue, ceteribus paribus - which often they are not.)
I'm still -1 because I don't think the purpose alone is close to worth
the cost of new syntax... And especially not using sigils that are
confusing to read in code.
You complain about sigils. Do you accept my point that more **words**
(and words that can, perhaps a trifle unkindly, be classed as
boilerplate rather than genuine content) can also make stuff harder to read?
The topic of "late binding in function signatures" simply isn't
*orthogonal* to "late binding in the general sense." Yes, they are
distinct, but very closely adjacent.
We disagree about that. *Please consider the */_**IMPLEMENTATIONS**_/*of
each. I respectfully suggest that you may conclude that they are not so
close after all.*
Best wishes
Rob Cliffe
Sorry again, but IMO discussing any model except one where
late-bound defaults are evaluated at function call time is just
adding FUD.
It's definitely rude to repeatedly state that anyone who's opinion is
different from yours is "adding FUD" and doesn't belong in the thread.
I was not talking about people whose opinion was different from mine. I
was talking about people who obscured the discussion of a proposal by
talking about a different proposal. And that, IMO, would be rude if it
were done deliberately, though I accept that it wasn't.
The topic of "late binding in function signatures" simply isn't
*orthogonal* to "late binding in the general sense." Yes, they are
distinct, but very closely adjacent.
We disagree about that. *Please consider the */_**IMPLEMENTATIONS**_/*of
each. I respectfully suggest that you may conclude that they are not so
close after all.
*PS In my support may I quote from a post from Chris:
[Steven D'Aprano] Chris may choose to reject this generalised lazy
evaluation idea, but if
so it needs to go into a Rejected Ideas section. Or he may decide that
actually having a generalised lazy evaluation idea is *brilliant* and
much nicer than making defaults a special case.
[Chris] It's an almost completely orthogonal proposal. I used to have a
reference to it in the PEP but removed it because it was unhelpful.
Rob Cliffe
Sorry again, but IMO discussing any model except one where late-bound defaults are evaluated at function call time is just adding FUD.
It's definitely rude to repeatedly state that anyone who's opinion is different from yours is "adding FUD" and doesn't belong in the thread.
I was not talking about people whose opinion was different from mine. I was talking about people who obscured the discussion of a proposal by talking about a different proposal. And that, IMO, would be rude if it were done deliberately, though I accept that it wasn't.
The topic of "late binding in function signatures" simply isn't *orthogonal* to "late binding in the general sense." Yes, they are distinct, but very closely adjacent.
We disagree about that. Please consider the *IMPLEMENTATIONS* of each. I respectfully suggest that you may conclude that they are not so close after all.
PS In my support may I quote from a post from Chris:
[Steven D'Aprano] Chris may choose to reject this generalised lazy evaluation idea, but if
so it needs to go into a Rejected Ideas section. Or he may decide that
actually having a generalised lazy evaluation idea is *brilliant* and
much nicer than making defaults a special case.
[Chris] It's an almost completely orthogonal proposal. I used to have a
reference to it in the PEP but removed it because it was unhelpful.
Since it appears to matter to people, I've readded a mention of it.
It's just freshly pushed so you might not see it instantly, but within
a few minutes (or browse the source code on GitHub), you should see
deferred evaluation mentioned in PEP 671.
ChrisA
This raises another choice: should lazy defaults be evaluated before
entering the body of the function, or at the point where the parameter
is used? Which would be more useful?
<img src=both_both_both.gif/>
Both are potentially useful.
Yes, both *ARE* potentially useful. *ABSOLUTELY*. I don't think anyone
would deny that. Certainly not I.
Let's call "I want late-bound defaults and Python doesn't have them"
Problem A.
Let's call "I want my default value not to be evaluated until needed"
Problem B.
(Of course, you may not consider either to be a problem, but let's
assume that you think that at least one of them is.)
Chris is offering a PEP and an implementation which addresses Problem A.
If Problem B is a problem for you, you can (currently) use a sentinel
value, and explicitly evaluate the default when you want it. Not so
terrible. In fact arguably, more often than not, better, because it's
explicit. And more flexible (you can evaluate a different expression,
or in a different scope, in different places in the function body).
Chris/PEP 671 is not attempting to provide a better way of doing that.
He is not offering Deferred Evaluation Objects (DEOs). Maybe in 5 years
or so someone will offer an implementation of that, and everyone can be
happy. 😁
Meanwhile, *Chris is offering a solution of Problem A*. He is *NOT
addressing Problem B* - someone else is welcome to try that.
Let's be honest: PEP 671 does not allow you to do anything you can't
already. What is adds is some more convenience, some more concision,
and arguably (and I *would* argue, in appropriate cases) some more
readability.
Why do people keep obscuring the discussion of a PEP which addresses
Problem A by throwing in discussion of the (unrelated) Problem B?
(Chris, and I, have stated, ad nauseam, that these *are* unrelated
problems. If you don't agree, I can only ask you to consider the
implementations necessary to solve each. If that doesn't change your
mind, I have to throw my hands in the air and say "We'll have to agree
to differ".)
*Surely solving one "problem" is better than dithering about which
"problem" to solve.*
I've been accused of trying to censor this thread, but really - I'm just
frustrated when people are invited to comment on PEP 671, and they don't
comment on PEP 671, but on something else.
BTW Thank you Stephen Turnbull, for your measured comments to this thread.
Best wishes
Rob Cliffe
Why do people keep obscuring the discussion of a PEP which addresses
Problem A by throwing in discussion of the (unrelated) Problem B?
(Chris, and I, have stated, ad nauseam, that these *are* unrelated
problems.
Chris says:
"Even if Python does later on grow a generalized lazy evaluation
feature, it will only change the *implementation* of late-bound
argument defaults, not their specification."
So you are mistaken that they are unrelated.
Chris could end this debate (and start a whole new one!) by going to the
Python-Dev mailing list and asking for a sponsor, and if he gets one,
for the Steering Council to make a ruling on the PEP. He doesn't *need*
consensus on Python-Ideas. (Truth is, we should not expect 100%
agreement on any new feature.)
But any arguments, questions and criticisms here which aren't resolved
will just have to be re-hashed when the core devs and the Steering
Council read the PEP. They can't be swept under the carpet.
--
Steve
How would you use lazy evaluation to *guarantee* the behaviour here?
By "the behaviour" I presume you want `max` evaluated before the body of
the function is entered, rather than at its point of use.
Same way your implementation does: ensure that the interpreter
fully evaluates `max` before entering the body of the function.
The only way I can imagine doing it is basically the same as I'm
doing: that late-bound argument defaults *have special syntax and
meaning to the compiler*. If they were implemented with some sort of
lazy evaluation object, they would need (a) access to the execution
context, so you can't just use a function;
Obviously you can't just compile the default expression as a function
*and do nothing else* and have late bound defaults magically appear from
nowhere.
Comprehensions are implemented as functions. Inside comprehensions, the
walrus operator binds to the caller's scope, not the comprehension scope.
>>> def frob(items):
... thunk = ((w:=len(items)) for x in (None,))
... next(thunk)
... return ('w' in locals(), w)
...
>>> frob([1, 2, 3, 4, 5])
(True, 5)
That seems to be exactly the behaviour needed for lazy evaluation
thunks, except of course we don't need all the other goodies that
generators provide (e.g. send and throw methods).
One obvious difference is that currently if we moved that comprehension
into the function signature, it would use the `items` from the
surrounding scope (because of early binding). It has to be set up in
such a way that items comes from the correct scope too.
If we were willing to give up fast locals, I think that the normal LEGB
lookup will do the trick. That works for locals inside classes, so I
expect it should work here too.
If that's the behaviour that people prefer, sure. Functions would need
to know which parameters were:
1. defined with a lazy default;
2. and not passed an argument by the caller (i.e. actually using
the default)
and for that subset of parameters, evaluate them, before entering the
body of the function. That's kinda what you already do, isn't it?
One interesting feature here is that you don't have to compile the
default expressions into the body of the function. You can stick them in
the code object, as distinct, introspectable thunks with a useful repr.
Potentially, the only extra code that needs go inside the function body
is a single byte-code to instantiate the late-bound defaults.
Even that might not need to go in the function body, it could be part of
the CALL_FUNCTION and CALL_FUNCTION_KW op codes (or whatever we use).
Well sure. But if we have syntax for a lazily evaluated expression it
would be an expression, right? So we can put it anywhere an expression
can go. Like parameter defaults in a function header.
The point is, Rob thought (and possibly still does, for all I know) that
lazy evaluation is completely orthogonal to late-bound defaults. The PEP
makes that claim too, even though it is not correct. With a couple of
tweaks that we have to do anyway, and perhaps a change of syntax (and
maybe not even that!) we can get late-bound defaults *almost* for free
if we had lazy evaluation.
That suggests that the amount of work to get *both* is not that much
more than the work needed to get just one. Why have a car that only
drives to the mall on Thursdays when you can get a car that can drive
anywhere, anytime, and use it to drive to the mall on Thursday as well?
Please stop arguing this point. It is a false analogy and until you
can demonstrate *with code* that there is value in doing it, it is a
massive red herring.
You can make further debate moot at any point by asking Python-Dev for a
sponsor for your PEP as it stands right now. If you think your PEP is
as strong as it can possibly be, you should do that.
(You probably want to fix the broken ReST first.)
Chris, you have been involved in the PEP process for long enough, as
both a participant of discussions and writer of PEPs, that you know damn
well that there is no requirement that all PEPs must have a working
implementation before being accepted, let alone being *considered* by
the community.
Yes, we're all very impressed that you are a competent C programmer who
can write an initial implementation of your preferred design. But your
repeated gate-keeping efforts to shut down debate by wrongly insisting
that only a working implementation may be discussed is completely out of
line, and I think you know it.
Being a C programmer with a working knowledge of the CPython internals
is not, and never has been, a prerequisite for raising ideas here.
I do feel some sympathy for you. I can't imagine the frustration you may
be feeling -- your intent is a really tightly focused narrow feature,
late-bound defaults. And not only do some people refuse to see what a
fantastic idea it is, and keep raising weaknesses of the PEP itself, but
some of them are more interested in a superset of the feature!
But I also can't help but feel that some of this is self-inflicted. It
looks to me that you jumped the gun on not only writing a PEP but an
implementation as well, *long* before there was even close to a
consensus on Python-Ideas (let alone in the broader community) that
late-bound defaults are worth the additional syntax, or what their
precise behaviour should be. And ever since then (it seems to me) you
have been trying to shut down any debate that isn't narrowly focused on
your design.
Of course it is your right to code whatever you want, whenever you like.
But you don't then get to complain about people wanting to consider a
larger feature set when they never agreed that they wanted late-bound
defaults alone in the first place.
Even if Python does later on grow a generalized lazy evaluation
feature, it will only change the *implementation* of late-bound
argument defaults, not their specification.
Great, we're in agreement that late-bound defaults can be implemented
via lazy evaluation. You should fix the PEP that wrongly describes them
as unrelated.
--
Steve
Why do people keep obscuring the discussion of a PEP which addresses
Problem A by throwing in discussion of the (unrelated) Problem B?
(Chris, and I, have stated, ad nauseam, that these *are* unrelated
problems.
Chris says:
"Even if Python does later on grow a generalized lazy evaluation
feature, it will only change the *implementation* of late-bound
argument defaults, not their specification."
So you are mistaken that they are unrelated.
*facepalm*
I'm not offering you a way to put C code in your Python function defaults.
However, there is a large amount of C code in the implementation of
them, at least in my reference implementation.
So I guess the features of late-bound defaults and C code in function
defaults aren't unrelated either, and I should stop working on this
and start working on that.
Seriously? Are you unable to distinguish implementation from
specification? What are you even doing on this mailing list?
ChrisA
How would you use lazy evaluation to *guarantee* the behaviour here?
By "the behaviour" I presume you want `max` evaluated before the body of
the function is entered, rather than at its point of use.
Same way your implementation does: ensure that the interpreter
fully evaluates `max` before entering the body of the function.
YES! Which means that.... guess what! It's NOT the same as having a
default which is a deferred evaluation object! It would be *buggy
behaviour* if you set the default to be a deferred evaluation object,
and the interpreter evaluated it on entering the body.
The only way I can imagine doing it is basically the same as I'm
doing: that late-bound argument defaults *have special syntax and
meaning to the compiler*. If they were implemented with some sort of
lazy evaluation object, they would need (a) access to the execution
context, so you can't just use a function;
Obviously you can't just compile the default expression as a function
*and do nothing else* and have late bound defaults magically appear from
nowhere.
Comprehensions are implemented as functions. Inside comprehensions, the
walrus operator binds to the caller's scope, not the comprehension scope.
>>> def frob(items):
... thunk = ((w:=len(items)) for x in (None,))
... next(thunk)
... return ('w' in locals(), w)
...
>>> frob([1, 2, 3, 4, 5])
(True, 5)
That seems to be exactly the behaviour needed for lazy evaluation
thunks, except of course we don't need all the other goodies that
generators provide (e.g. send and throw methods).
GO AND IMPLEMENT IT. I'm done arguing this. Write the code. You'll
find it's a LOT more problematic than you claim.
One obvious difference is that currently if we moved that comprehension
into the function signature, it would use the `items` from the
surrounding scope (because of early binding). It has to be set up in
such a way that items comes from the correct scope too.
If we were willing to give up fast locals, I think that the normal LEGB
lookup will do the trick. That works for locals inside classes, so I
expect it should work here too.
Well sure. But if we have syntax for a lazily evaluated expression it
would be an expression, right? So we can put it anywhere an expression
can go. Like parameter defaults in a function header.
Yes. See? You could do it as a completely separate proposal, like
we've been saying.
The point is, Rob thought (and possibly still does, for all I know) that
lazy evaluation is completely orthogonal to late-bound defaults. The PEP
makes that claim too, even though it is not correct. With a couple of
tweaks that we have to do anyway, and perhaps a change of syntax (and
maybe not even that!) we can get late-bound defaults *almost* for free
if we had lazy evaluation.
That suggests that the amount of work to get *both* is not that much
more than the work needed to get just one. Why have a car that only
drives to the mall on Thursdays when you can get a car that can drive
anywhere, anytime, and use it to drive to the mall on Thursday as well?
Please stop arguing this point. It is a false analogy and until you
can demonstrate *with code* that there is value in doing it, it is a
massive red herring.
You can make further debate moot at any point by asking Python-Dev for a
sponsor for your PEP as it stands right now. If you think your PEP is
as strong as it can possibly be, you should do that.
(You probably want to fix the broken ReST first.)
Chris, you have been involved in the PEP process for long enough, as
both a participant of discussions and writer of PEPs, that you know damn
well that there is no requirement that all PEPs must have a working
implementation before being accepted, let alone being *considered* by
the community.
Yes, we're all very impressed that you are a competent C programmer who
can write an initial implementation of your preferred design. But your
repeated gate-keeping efforts to shut down debate by wrongly insisting
that only a working implementation may be discussed is completely out of
line, and I think you know it.
Yes, I know that a fully-working implementation isn't a prerequisite.
I also know that you are arguing out of zero experience of what it
actually takes to make this happen, and you have yet to overcome MANY
of the problems that deferred evaluation creates.
You cannot use your ignorance as an excuse to shut down other people's ideas.
I do feel some sympathy for you. I can't imagine the frustration you may
be feeling -- your intent is a really tightly focused narrow feature,
late-bound defaults. And not only do some people refuse to see what a
fantastic idea it is, and keep raising weaknesses of the PEP itself, but
some of them are more interested in a superset of the feature!
It is not a superset. I have proven this from the perspective of
specification alone. You keep insisting that it is a superset. Go and
write the code.
But I also can't help but feel that some of this is self-inflicted. It
looks to me that you jumped the gun on not only writing a PEP but an
implementation as well, *long* before there was even close to a
consensus on Python-Ideas (let alone in the broader community) that
late-bound defaults are worth the additional syntax, or what their
precise behaviour should be. And ever since then (it seems to me) you
have been trying to shut down any debate that isn't narrowly focused on
your design.
Of course it is your right to code whatever you want, whenever you like.
But you don't then get to complain about people wanting to consider a
larger feature set when they never agreed that they wanted late-bound
defaults alone in the first place.
Even if Python does later on grow a generalized lazy evaluation
feature, it will only change the *implementation* of late-bound
argument defaults, not their specification.
Great, we're in agreement that late-bound defaults can be implemented
via lazy evaluation. You should fix the PEP that wrongly describes them
as unrelated.
*facepalm*
Okay, here's a compromise.
Go and write a full and detailed specification of the Python-visible
semantics of deferred evaluation objects, including how they would be
used to implement late-bound argument defaults.
Go and actually do some real work on your pet feature, instead of
using the vapourware to try to shut down the one I've been working on.
Go and actually do something useful instead of just arguing that "it
must be possible".
Once again, you're getting very very close to being killfiled.
ChrisA
The point is, Rob thought (and possibly still does, for all I know) that
lazy evaluation is completely orthogonal to late-bound defaults. The PEP
makes that claim too, even though it is not correct. With a couple of
tweaks that we have to do anyway, and perhaps a change of syntax (and
maybe not even that!) we can get late-bound defaults *almost* for free
if we had lazy evaluation.
That depends of lazy evaluation spec, if lazy expression would ever
become a thing in python, it may be defined to have syntax like `lazy
<expr>` which would be rough equivalent off `LazyObject(lambda:
<expr>)` that would evaluate that lambda at most once, plus some
interpreter tweaks to make LazyObject transparent to python code.
so for this code (`??` replaced with different combination of
early/late and not lazy/lazy)
```
x = []
def f(x, y, z ?? len(x)):
x.append(y)
print(z, end = ' ')
x.append(0)
f([1, 2, 3], 4)
x.append(0)
f([1, 2, 3, 4], 4)
```
I expect that
for `??` = `=` I get `0 0 `
for `??` = `=>` I get `3 4 `
for `??` = `= lazy` I get `1 1`
for '??' = `=> lazy` I get `4 5 `
That would be completely orthogonal.
Go and write a full and detailed specification of the Python-visible
semantics of deferred evaluation objects, including how they would be
used to implement late-bound argument defaults.
Go and actually do some real work on your pet feature, instead of
using the vapourware to try to shut down the one I've been working on.
Go and actually do something useful instead of just arguing that "it
must be possible".
I'm not the person you're replying to, but just a reminder here: there
is one alternative proposal that already has a fully functioning
implementation, namely the current behavior. Your arguments against the
deferred-evaluation proposal seem to constantly be reiterating that
there is no concrete deferred-evaluation proposal. You are right. But
your arguments also seem to be insinuating that if there is no such
proposal, then opposition to the PEP is somehow misguided, and that is
incorrect. There doesn't need to be any concrete alternative proposal
other than "leave everything as it is and wait until we think of
something better".
It is perfectly valid to oppose your PEP even on the basis that maybe a
deferred-evaluation proposal has a remote possibility of being better in
the future --- because it is perfectly valid to oppose your PEP even if
such a proposal has NO possibility of being better in the future. There
is no urgency or need for the behavior described in your PEP. I am fine
with the current behavior of Python in this regard. It is not necessary
to provide any alternative proposal, concrete or handwavy, to argue that
the PEP is a bad idea. I believe the PEP is a bad idea because the
current behavior of Python is actually better than what it would be if
the PEP were adopted. I believe it is better to wait until we think of
a better idea than to implement this PEP, and, if we never think of a
better idea, then never change the existing argument-default behavior of
Python.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
Go and write a full and detailed specification of the Python-visible
semantics of deferred evaluation objects, including how they would be
used to implement late-bound argument defaults.
Go and actually do some real work on your pet feature, instead of
using the vapourware to try to shut down the one I've been working on.
Go and actually do something useful instead of just arguing that "it
must be possible".
I'm not the person you're replying to, but just a reminder here: there
is one alternative proposal that already has a fully functioning
implementation, namely the current behavior. Your arguments against the
deferred-evaluation proposal seem to constantly be reiterating that
there is no concrete deferred-evaluation proposal. You are right. But
your arguments also seem to be insinuating that if there is no such
proposal, then opposition to the PEP is somehow misguided, and that is
incorrect. There doesn't need to be any concrete alternative proposal
other than "leave everything as it is and wait until we think of
something better".
It is perfectly valid to oppose your PEP even on the basis that maybe a
deferred-evaluation proposal has a remote possibility of being better in
the future --- because it is perfectly valid to oppose your PEP even if
such a proposal has NO possibility of being better in the future. There
is no urgency or need for the behavior described in your PEP. I am fine
with the current behavior of Python in this regard. It is not necessary
to provide any alternative proposal, concrete or handwavy, to argue that
the PEP is a bad idea. I believe the PEP is a bad idea because the
current behavior of Python is actually better than what it would be if
the PEP were adopted. I believe it is better to wait until we think of
a better idea than to implement this PEP, and, if we never think of a
better idea, then never change the existing argument-default behavior of
Python.
I have laid out, multiple times, how a deferred evaluation feature is
completely distinct from late-bound argument defaults. So have others.
Steven continues to assert that, just because it MIGHT be possible to
use them in the implementation, we should stop working on this and
start working on that. He would, of course, be very welcome to work on
deferred evaluation himself, but he chooses to hide behind his own
ignorance of C to avoid doing any such work, and then still argues
that we should stop working on this because, in his opinion solely, it
would be more useful to have deferred evaluation.
And then he calls me a liar for saying in the PEP the same thing that
I've been saying here, yet he won't even write up a full specification
for deferred evaluation.
You are welcome to dislike the PEP on the basis that the existing
language is better than it would be with this feature. I personally
disagree, but that's what opinions are. But to object on the mere
basis that something MIGHT, despite demonstrated evidence, be better?
That is unfair and unhelpful.
ChrisA
Go and write a full and detailed specification of the Python-visible
semantics of deferred evaluation objects, including how they would be
used to implement late-bound argument defaults.
I'm going to ignore all the rhetoric here, as it's not helpful. I
understand that you're frustrated, and that you feel like you're not
getting your point across. Part of that (IMO) is *because* you're
getting too frustrated, and so not explaining your point well. This is
a case in point.
Deferred evaluation doesn't need to be implemented to be a valid
counter-proposal. Your repeated demands that someone produce an
implementation are a distraction, allowing people to argue that you're
wrong on that point, while ignoring the more important point. Which is
that we don't really have a definition of how deferred evaluation
would work.
I'm not talking about a "full and detailed specification". That's
*still* more than is needed for a valid debate (at this point). But
what *is* needed is a more complete explanation of how deferred
evaluation would work, and some plausible (not set in stone, just
plausible) syntax. With that, it would be possible to write down two
versions of the same code and judge between them. We've not yet had a
sufficiently clear (IMO) definition of the semantics of deferred
evaluation yet (or if we have, it's been lost in the arguments), and
it would help a lot if someone could provide one. I'm thinking
specifically about the rules for variable capture, scoping,
interaction with assignment expressions in terms of introducing names,
etc., as well as how evaluation is "triggered" and what ability there
is to explicitly say "evaluate this now".
Go and actually do some real work on your pet feature, instead of
using the vapourware to try to shut down the one I've been working on.
This is rhetoric again. Asking for more concrete examples of the
proposed alternative is reasonable. Getting frustrated when they are
not provided is understandable, but doesn't help. Calling the proposed
alternative "vapourware" just doubles down on the uncompromising
"implement it or I'll ignore you" stance. And replying with
increasingly frustrated posts that end up at a point where people like
me can't even work out how we'd go looking to see whether anyone *had*
provided concrete examples of deferred evaluation just makes things
worse. All of which could have been avoided by simply including an
early argument posted here in the PEP, under rejected alternatives,
with a link to the post and a statement that "this was proposed as an
alternative, but there's not enough detail provided to confirm how it
would replace the existing proposal". Then anyone who disagrees has a
clear understanding of what you want, and how to provide it.
Once again, you're getting very very close to being killfiled.
This whole discussion is close to that point for me. But believe it or
not, I still have a vague hope that the proposal can be strengthened
by people working together, rather than just ending up with a "this is
what I think, take it or leave it" PEP.
Paul
I'm not talking about a "full and detailed specification". That's
*still* more than is needed for a valid debate (at this point). But
what *is* needed is a more complete explanation of how deferred
evaluation would work, and some plausible (not set in stone, just
plausible) syntax. With that, it would be possible to write down two
versions of the same code and judge between them. We've not yet had a
sufficiently clear (IMO) definition of the semantics of deferred
evaluation yet (or if we have, it's been lost in the arguments), and
it would help a lot if someone could provide one. I'm thinking
specifically about the rules for variable capture, scoping,
interaction with assignment expressions in terms of introducing names,
etc., as well as how evaluation is "triggered" and what ability there
is to explicitly say "evaluate this now".
That's what I mean by a full specification. Even without code, that
would be enough to start talking about it. But those arguing "don't go
for lazy evaluation, go for deferreds" don't seem to want to actually
push that proposal forward.
That's why I call it vapourware.
ChrisA
I'm not talking about a "full and detailed specification". That's
*still* more than is needed for a valid debate (at this point). But
what *is* needed is a more complete explanation of how deferred
evaluation would work, and some plausible (not set in stone, just
plausible) syntax. With that, it would be possible to write down two
versions of the same code and judge between them. We've not yet had a
sufficiently clear (IMO) definition of the semantics of deferred
evaluation yet (or if we have, it's been lost in the arguments), and
it would help a lot if someone could provide one. I'm thinking
specifically about the rules for variable capture, scoping,
interaction with assignment expressions in terms of introducing names,
etc., as well as how evaluation is "triggered" and what ability there
is to explicitly say "evaluate this now".
That's what I mean by a full specification. Even without code, that
would be enough to start talking about it. But those arguing "don't go
for lazy evaluation, go for deferreds" don't seem to want to actually
push that proposal forward.
That's why I call it vapourware.
OK, so maybe if you were a little less aggressive in your replies, we
could see if anyone wants to respond. But frankly, I imagine it's hard
to muster up any enthusiasm for writing up the semantics unless you're
willing to show some sign that you might modify the PEP as a result.
I'm not talking about rewriting your PEP to be a "deferred evaluation"
PEP, but simply to modify it to make it more compatible with the
future that the "deferred evaluation" people imagine.
On the other hand, if the "deferred evaluation" supporters genuinely
have nothing to offer other than "we don't want this PEP *at all*
because what we have right now is sufficient in the short term, and
longer term maybe something else (deferred evaluation being the
possibility we can think of right now) will provide a different
solution" then that's also OK. It's just a -1 vote, and should be
recorded in the PEP as such - "A number of contributors on
python-ideas were against this proposal because they didn't believe it
offered enough benefit over the status quo, and they preferred to wait
for a more general solution such as deferred evaluation (on the basis
of the Zen "never is often better than right now")". That can still go
in the "rejected ideas" section, under a heading of "Do Nothing".
But unless we can reduce the level of conflict here, we're never going
to know which alternative the deferred evaluation supporters want, and
it won't be possible to accurately represent their views in the PEP.
Which is bad for them (as they'll feel ignored) and for you (as your
PEP won't fairly represent the views of the people who contributed to
the discussion).
Paul
PS To be clear, my objections to the PEP aren't based on deferred
evaluation. So I'm an impartial 3rd party on this matter. I *do* have
problems with the PEP, so I have an interest in seeing the PEP fairly
reflect the lack of consensus, and accurately represent the concerns
people are raising, but I don't have a preference for any specific
outcome in the matter of deferred evaluation.
PS To be clear, my objections to the PEP aren't based on deferred
evaluation. So I'm an impartial 3rd party on this matter. I *do* have
problems with the PEP, so I have an interest in seeing the PEP fairly
reflect the lack of consensus, and accurately represent the concerns
people are raising, but I don't have a preference for any specific
outcome in the matter of deferred evaluation.
Thinking some more about this, my comments are pretty much what I'd be
saying if I were a sponsor for this PEP. I don't think a PEP sponsor
should be someone who doesn't agree with the PEP, otherwise I'd offer
to take on the role (assuming you need a sponsor). But please take my
comments in that vein. (And if you do ever manage to convert me to
support of the PEP, remind me of this comment and I'll be the sponsor
;-))
Paul
I will post to a different thread for my actual semi-proposal. But to
answer the claim that late-bound arg defaults and late-bound
everything-else are unrelated, and to start with an early version of what
I'd actually want, I wrote this:
https://github.com/DavidMertz/peps/blob/master/pep-9999.rst
I could definitely use help making it better from anyone who is sympathetic
with the idea of generalized deferred computation in Python.
On Tue, Jun 21, 2022 at 5:12 AM Paul Moore
PS To be clear, my objections to the PEP aren't based on deferred
evaluation. So I'm an impartial 3rd party on this matter. I *do* have
problems with the PEP, so I have an interest in seeing the PEP fairly
reflect the lack of consensus, and accurately represent the concerns
people are raising, but I don't have a preference for any specific
outcome in the matter of deferred evaluation.
Thinking some more about this, my comments are pretty much what I'd be
saying if I were a sponsor for this PEP. I don't think a PEP sponsor
should be someone who doesn't agree with the PEP, otherwise I'd offer
to take on the role (assuming you need a sponsor). But please take my
comments in that vein. (And if you do ever manage to convert me to
support of the PEP, remind me of this comment and I'll be the sponsor
;-))
--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons. Intellectual property is
to the 21st century what the slave trade was to the 16th.
You are welcome to dislike the PEP on the basis that the existing
language is better than it would be with this feature. I personally
disagree, but that's what opinions are. But to object on the mere
basis that something MIGHT, despite demonstrated evidence, be better?
That is unfair and unhelpful.
Again, I disagree, and I think that stance is unreasonable. There is
nothing wrong with saying "Let's not do this. Instead let's wait and
explore other ideas and see if we come up with something better." There
is no requirement to provide any alternative proposal whatsoever ---
whether vague or specific, real or potential, conceptual or concrete ---
merely to justify opposing this PEP.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
1) If this feature existed in Python 3.11 exactly as described, would
you use it?
Definitely not. It significantly increases cognitive burden on the reader. If python didn't have pattern matching, walrus, ternary expressions, etc -- then maybe I would consider this as something harmless. But right now we are at risk of slowly turning a great language into a monstrosity. We already have a tool (an if statement) that solves this problem just as well, is a widely known pattern, and is a lot easier to read.
2) Independently: Is the syntactic distinction between "=" and "=>" a
cognitive burden?
Yes, but a slight one.
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
defer keyword that Andre proposed is a better option, in my opinion. However, I don't like the entire idea so I would argue against that as well. Don't take it as something bad -- it's a good idea in general but I believe that it's a step in the wrong direction specifically for python as it is.
5) Do you know how to compile CPython from source, and would you be
willing to try this out?
I do know how to compile it but I believe that the examples from your pep are great and paint a realistic enough picture of how it's going to feel so I'm afraid I have to refuse to test it. Sorry :)
4) If "no" to question 1, is there some other spelling or other small
change that WOULD mean you would use it? (Some examples in the PEP.)
defer keyword that Andre proposed is a better option, in my opinion. However, I don't like the entire idea so I would argue against that as well. Don't take it as something bad -- it's a good idea in general but I believe that it's a step in the wrong direction specifically for python as it is.
Except that the defer keyword *simply doesn't work* for this use-case.
In my opinion, that's a bit of a downside.
I posted examples of the problems in the other thread, but there were
no replies.
ChrisA
Very late addition to these long threads -- I'm loving the proposal. I'm rather surprised though that the typing advantages of the pep have not been emphasized enough. I have a lot of code that in order to get correct typing in a strict mode does something like this:
```
def move_pitches(pitches: list[Pitch]|None = None):
non_none_pitches: list[Pitch]
if pitches is None:
non_none_pitches = []
else:
non_none_pitches = pitches
```
with the PEP, the typing becomes:
```
def move_pitches(pitches: list[Pitch] => []):
...
```
and not only is the variable typed properly and more simply from the start, but I get to use my preferred variable name throughout.
Since I do about 30% of my work in TypeScript, the idea that `=>` implies that the right side will be evaluated in some way and returned is a cognitive lightening not burden.
I do think that the PEP should be stricter in defining whether if an implementation chooses to do the evaluating in two passes whether normal '=' arguments to the right can or cannot be referenced by late-bound arguments. I initially read the PEP as saying that you might or might not be able to reference rightward normal arguments depending on the implementation.
- Michael Scott Asato Cuthbert (music21)
Very late addition to these long threads -- I'm loving the proposal. I'm rather surprised though that the typing advantages of the pep have not been emphasized enough.
That would be because I personally don't use that kind of strict
typing, so it's not something I'm really qualified to talk about.
Would you like to write up a paragraph or two about it? I could
incorporate it verbatim, if you like.
That would be because I personally don't use that kind of strict
typing, so it's not something I'm really qualified to talk about.
Would you like to write up a paragraph or two about it? I could
incorporate it verbatim, if you like.
Sure!
* * *
Late-bound arg defaults also help with proper typing, especially for established code-bases that are incrementally adding typing to their code. For instance, take this untyped example that might take a list of musical pitches and put them in a particular order:
def order_pitches(pitches=None):
if pitches is None:
pitches = []
... # do reordering referencing "pitches" a lot
pitches.sort(key=lambda p: (p.octave, p.name))
return pitches
When the editor moves to a typed version, something like this would seem reasonable:
def order_pitches(pitches: list[Pitch]|None = None) -> list[Pitch]:
... # same code as above.
However, some type checkers (for instance, that of PyCharm 2022) will continue to reveal the type as "list[Pitch]|None" even after type narrowing. (Mypy will correctly type narrow in this example but not in code that checks for `if not hasattr(pitches, '__iter__')` and in many other more complex examples).
In this case, authors sometimes need to resort to rewriting code with a new variable name:
def order_pitches(pitches: list[Pitch]|None = None) -> list[Pitch]:
non_none_pitches: list[Pitch]
if pitches is None:
non_none_pitches = []
else:
non_none_pitches = cast(list[Pitch], pitches)
... # rest of code must be rewritten to use "non_none_pitches"
The typed definition also seems to imply that "None" is an acceptable calling type for the function rather than just being a stand-in for an omitted call. A type-checker will allow `order_pitches(None)` to pass, perhaps preventing later refactoring to use a sentinel such as:
def order_pitches(pitches: list[Pitch]|MISSING = MISSING) -> list[Pitch]:
if pitches is MISSING:
pitches = []
...
With the PEP, the process of adding typing does not impact the code, nor imply that "None" is a fine calling signature.
def order_pitches(pitches: list[Pitch] => []) -> list[Pitch]:
...
# pitches is always list[Pitch] and needs no narrowing
pitches.sort(...)
return pitches
* * *
It is true that the tools for automatic type-narrowing continue to get better (in looking up code that broke the type-narrowing with a "no_none..." variable, I found that 2 of 4 places I found where I had done this workaround two years ago no longer needed it for either mypy or PyCharm), but there are still numerous cases where the process of typing around a sentinel type that will be replaced by the correct type for computing still unnecessarily exist.
That would be because I personally don't use that kind of strict
typing, so it's not something I'm really qualified to talk about.
Would you like to write up a paragraph or two about it? I could
incorporate it verbatim, if you like.
When the editor moves to a typed version, something like this would seem reasonable:
"Editor" could mean a person or a program. Which do you mean? I want
to guess you mean the person making the edit, but it's not quite
clear.
Otherwise, looks good.
ChrisA
Is the => syntax needed? as far as I can think of, the only time where late evaluation is needed is when the expression references the other arguments. So the rule “if the expression reference other arguments it will get evaluated at function call time” should suffice right?
In effect:
def foo(a, b=len(a), c=max(b, d), d=15):
gets translated into
def foo(a, b=None, c=None, d=15):
if b is None:
b = len(a)
if c is None:
c = max(b, d)
I’m not sure if this is already brought up in previous emails, I tried my best to search for it but can’t find any reference.
Also, I think the sepc should not leave any ambiguous behavior otherwise it creates subtle incompatibilities when people use different implementations. This goes for whether all other argument can be referenced or only some argument can be. Or if the use of outer scope variables should get evaluated at def time, something like foo(a, b=max(a, outer.length())) should length() be ran once or on every function call. Stuff like these create an functional effect to the user so they ought to be well defined and not implementation specific.
Is the => syntax needed? as far as I can think of, the only time where
late evaluation is needed is when the expression references the other
arguments.
You are missing the most common case, the motivating case, for
late-bound defaults: mutable defaults.
def spam(x, y=>[]):
pass
Here the intention is to have y's default be a *different* list each
time you call spam(x), instead of the same list each time.
The ability for default values to refer to other parameters is a Nice To
Have, not a Must Have. It has been a very long time since I have read
the PEP, and I don't remember whether it reviews other languages to see
what functionality they provide for defaults, but I don't think many
other languages allow you to set the default of one parameter to be
another parameter.
--
Steve
Is the => syntax needed? as far as I can think of, the only time where
late evaluation is needed is when the expression references the other
arguments.
You are missing the most common case, the motivating case, for
late-bound defaults: mutable defaults.
def spam(x, y=>[]):
pass
Exactly - this is the most important reason. So the true reason for
the => syntax is: in order to gain late-bound defaults, we have to
distinguish them from early-bound defaults (because changing ALL
function default arguments to late-bound would be a massive breaking
change). There's no particular reason for it to be "=>" specifically,
and other syntax options have been considered, but it does need to be
something other than "=".
The ability for default values to refer to other parameters is a Nice To
Have, not a Must Have. It has been a very long time since I have read
the PEP, and I don't remember whether it reviews other languages to see
what functionality they provide for defaults, but I don't think many
other languages allow you to set the default of one parameter to be
another parameter.
JavaScript does, by nature of its extremely simplistic definition of
argument defaults.
function f(x=1, y=x) {console.log("--> x = ", x, ", y = ", y);}
f(5)
--> x = 5 , y = 5
f()
--> x = 1 , y = 1
f(42, undefined)
--> x = 42 , y = 42
f(undefined, 123)
--> x = 1 , y = 123
Ruby does:
$ irb
irb(main):001:1* def f(x=1, y=x)
irb(main):002:1* puts "x = #{x}, y = #{y}"
irb(main):003:0> end
=> :f
irb(main):004:0> f()
x = 1, y = 1
=> nil
irb(main):005:0> f(123)
x = 123, y = 123
=> nil
I suspect that, in each case, the rule is quite simple: the argument
default is evaluated in the context of the function's body. Exactly
the same as PEP 671 proposes.
Any other languages to test?
ChrisA
I didn't realize def foo(x, y=[]) had this strange artifact but it totally makes sense, TIL. I did not get the right idea reading the PEP though, since currently the motivation reads:
Optional function arguments, if omitted, often have some sort of logical default value. When this value depends on other arguments, or needs to be reevaluated each function call, there is currently no clean way to state this in the function header.
and I kinda glossed over the second use-case. I feel like more emphasis can be added since that part is what necessitates the new syntax.
I do think that being able to reference other arguments is very useful in it's own right and would go a long way in helping to solve the None check problem brought up in PEP 505 even more cleanly.
I didn't realize def foo(x, y=[]) had this strange artifact but it totally makes sense, TIL. I did not get the right idea reading the PEP though, since currently the motivation reads:
Optional function arguments, if omitted, often have some sort of logical default value. When this value depends on other arguments, or needs to be reevaluated each function call, there is currently no clean way to state this in the function header.
and I kinda glossed over the second use-case. I feel like more emphasis can be added since that part is what necessitates the new syntax.
I do think that being able to reference other arguments is very useful in it's own right and would go a long way in helping to solve the None check problem brought up in PEP 505 even more cleanly.
Yeah, they're both useful features, and both handled by the simple
rule of "evaluate late-bound defaults in the context of the function
body".
ChrisA
Ad 4) Wouldn't "<=" be a little more logical than "=>"? The perceived direction of the "flow" of the default value is exactly opposite, i.e., the default value is always evaluated and then put *into* the argument.
For example:
def bisect_right(a, x, lo=0, hi<=len(a), *, key=None):
Ad 4) Wouldn't "<=" be a little more logical than "=>"? The perceived direction of the "flow" of the default value is exactly opposite, i.e., the default value is always evaluated and then put *into* the argument.
Using arrows to represent information flow has been done, for example
in APL and R, but it's generally not been a significant benefit. C++
uses flow operators in a delightfully cute way that gets old after
about the second time you actually use it. (Python has done similarly
cute things with certain operators, with mixed results. I think
Pathlib has been quite successful, but there are others that are less
so.) Flow direction simply isn't a valuable-enough piece of
information to be worth reusing an existing operator and thus risking
ambiguity.
ChrisA
1. Not if it is exactly as described in PEP.
2. No.
3. -
4. Couple of points here. One check and one orthogonal idea, which would make this PEP very attractive to me.
I would definitely like use this functionality if both of below points were satisfied/achievable. If any of those weren't satisfied I might just default to current syntax as I like to have a default pattern, which I know is flexible enough to cover most if not all the cases that I encounter. Currently, I manage this particular area (which your PEP is concerned with) with `None` and `unittest.Sentinel` defaults and deal with them in function's body.
A.------------------------------------------------
Currently, if I write:
```
def foo(bar=A()):
pass
class A:
pass
```
I get an error. That is why having `bar=None` is advantageous. This works ok:
```
def foo(bar=None):
if bar is None:
bar = A()
class A:
pass
```
If PEP is aiming to replace the latter example, then it would be great if it kept all of its advantages. I.e. not having to change the definition order in the module, which could be preferred as it is for other reasons.
My best guess is that it works the same as the latter example, (given the expression can contain other arguments, which are not there yet) but just wanted to double check.
B.------------------------------------------------
And also to come back to my previous notice that there is no way to enforce the default in case of function chain with cascading arguments. You said it is a known limitation. Is there no easy & sensible approach to not have it? E.g.:
a) Any object which has certain dunder attribute, which evaluates to True?
b) NotGiven sentinel value which does exactly that.
c) A special constant, which, if present, at lower level makes things behave the same way as the argument wasn’t provided at all. Such constant could be very useful outside the scope of this PEP as well. Could be a great place to introduce such constant? And to me it seems it could be a well justified one, given it actually is special and does not fall under umbrella of generic sentinel values.
It would be great if it was to retain all the benefits of the latter example. Then (at least from my POV) this PEP would be an excellent addition, and I am most likely be using it now if it existed.
Regards,
DG
If PEP is aiming to replace the latter example, then it would be great if it kept all of its advantages. I.e. not having to change the definition order in the module, which could be preferred as it is for other reasons.
Well, yes, it would work, but I still wouldn't recommend it. This is
confusing to read. But since the default would be evaluated at call
time, it would behave exactly as you describe.
And also to come back to my previous notice that there is no way to enforce the default in case of function chain with cascading arguments. You said it is a known limitation. Is there no easy & sensible approach to not have it? E.g.:
a) Any object which has certain dunder attribute, which evaluates to True?
b) NotGiven sentinel value which does exactly that.
c) A special constant, which, if present, at lower level makes things behave the same way as the argument wasn’t provided at all. Such constant could be very useful outside the scope of this PEP as well. Could be a great place to introduce such constant? And to me it seems it could be a well justified one, given it actually is special and does not fall under umbrella of generic sentinel values.
It would be great if it was to retain all the benefits of the latter example. Then (at least from my POV) this PEP would be an excellent addition, and I am most likely be using it now if it existed.
There is no way to have a value that isn't a value, in Python. The
concept doesn't make sense and would break all kinds of things.
(That's why, when a special function like __getitem__ has to be able
to return literally anything, it signals "nothing to return" by
raising an exception.)
The only way to not pass an argument in Python is to not pass it. That
means, at best, something like *a or **kw, where the sequence/dict
either has something or doesn't, depending on whether you want to pass
the argument. None of this is changed by PEP 671 and I don't think
there's a lot of point trying to, as it would only cause more problems
elsewhere.
ChrisA
There is no way to have a value that isn't a value, in Python. The
concept doesn't make sense and would break all kinds of things.
(That's why, when a special function like __getitem__ has to be able
to return literally anything, it signals "nothing to return" by
raising an exception.)
I accept that it might be difficult to implement.
I see that it would break things at cpython.
Will definitely break some of the stdlib. E.g. inspect stuff.
It wouldn’t break any of the existing python code.
So yes, might not be a minor change.
Could it be done nicely and easily by someone with relevant experience? I don’t know.
But I fail to see why it doesn’t make sense - that’s a strong statement.
It would be a value, just a value that is treated with exception in this particular case.
There is definitely code at that level - resolving args, kwargs, dealing with “/" and “*” in relation to arguments provided, etc.
It would take effect only on keyword arguments with defaults, if so then fairly contained matter.
It could be a default of a keyword argument itself, would have a type and everything as any other object.
There is no way to have a value that isn't a value, in Python. The
concept doesn't make sense and would break all kinds of things.
(That's why, when a special function like __getitem__ has to be able
to return literally anything, it signals "nothing to return" by
raising an exception.)
I accept that it might be difficult to implement.
I see that it would break things at cpython.
Will definitely break some of the stdlib. E.g. inspect stuff.
It wouldn’t break any of the existing python code.
So yes, might not be a minor change.
Could it be done nicely and easily by someone with relevant experience? I don’t know.
But I fail to see why it doesn’t make sense - that’s a strong statement.
It would be a value, just a value that is treated with exception in this particular case.
There is definitely code at that level - resolving args, kwargs, dealing with “/" and “*” in relation to arguments provided, etc.
It would take effect only on keyword arguments with defaults, if so then fairly contained matter.
It could be a default of a keyword argument itself, would have a type and everything as any other object.
Okay. You now have an object that you can't do anything with, because
it can't be a function argument. So...
Show me how you would put this value into a dictionary.
Show me how you would find out the type of this value.
Show me how you would refer to this in an exception.
Show me how you would access an attribute of this object.
Show me how you would do ANYTHING WHATSOEVER with this object.
It does not make sense to have an object that isn't an object. And in
Python, every value *is* an object.
ChrisA
IT IS AN OBJECT. Never said otherwise.
`inspect.getcallargs` can seemingly be modified for such behaviour. I just wrote a decorator, which does what I proposed using `inspect` module for a chosen sentinel value. The issue is that it would be a bottleneck if used on any callable, which is continuously used. `inspect.getcallargs`, `signature`, `getfullargspec` are very expensive.
If that can be done, theoretically it should be able to be done at lower level as well. After all, behaviour of it should be modelled after what is happening at the function call.
There is no way to have a value that isn't a value, in Python. The
concept doesn't make sense and would break all kinds of things.
(That's why, when a special function like __getitem__ has to be able
to return literally anything, it signals "nothing to return" by
raising an exception.)
I accept that it might be difficult to implement.
I see that it would break things at cpython.
Will definitely break some of the stdlib. E.g. inspect stuff.
It wouldn’t break any of the existing python code.
So yes, might not be a minor change.
Could it be done nicely and easily by someone with relevant experience? I don’t know.
But I fail to see why it doesn’t make sense - that’s a strong statement.
It would be a value, just a value that is treated with exception in this particular case.
There is definitely code at that level - resolving args, kwargs, dealing with “/" and “*” in relation to arguments provided, etc.
It would take effect only on keyword arguments with defaults, if so then fairly contained matter.
It could be a default of a keyword argument itself, would have a type and everything as any other object.
Okay. You now have an object that you can't do anything with, because
it can't be a function argument. So...
Show me how you would put this value into a dictionary.
Show me how you would find out the type of this value.
Show me how you would refer to this in an exception.
Show me how you would access an attribute of this object.
Show me how you would do ANYTHING WHATSOEVER with this object.
It does not make sense to have an object that isn't an object. And in
Python, every value *is* an object.
`inspect.getcallargs` can seemingly be modified for such behaviour. I just wrote a decorator, which does what I proposed using `inspect` module for a chosen sentinel value. The issue is that it would be a bottleneck if used on any callable, which is continuously used. `inspect.getcallargs`, `signature`, `getfullargspec` are very expensive.
If that can be done, theoretically it should be able to be done at lower level as well. After all, behaviour of it should be modelled after what is happening at the function call.
Since you clearly are not listening to the discussion, I will leave
you with one final recommendation: Write some code. Don't just claim
that it's possible; write actual code that makes it happen. You will
discover exactly how hard it is. If I am wrong, you will be able to
PROVE that I am wrong, instead of merely claiming it.
ChrisA
Or maybe you are not listening to what I am saying.
It would only take effect for arguments, which have default value.
So.
dict().__setitem__(key, NotGiven)
type(NotGiven)
Exception(NotGiven)
getattr(NotGiven, name)
Ok, maybe there are some crucial functions, which have argument defaults. But again, there is another solution then, implement efficient decorator in standard library, maybe as part of sentinels, maybe as part of your PEP, maybe separately.
I never said that it isn’t hard to write low-level python code. I will do that when the time is right for me. And I am not claiming that it is possible. If you read my language, all of my doubts are clearly expressed. You just seem to read everything as black and white even when it is not so.
However, such statements are in no way valid arguments in such discussion. If my idea is not optimal, not necessary, no-one needs it or any other valid responses, why it is not a good one - I can accept it.
But commenting that “it makes no sense” without properly explaining to me why after giving it some thought - it’s not fair.
Also, if you looked at things from a bit more positive perspective, maybe you could come up with some nice new ways how it could be done. Maybe not exactly what I am proposing, but some alternative, which would make it work. Most likely something much better than I am proposing.
After all, I am trying to see how your PEP can be improved, because if it could be used in all cases, where None can be, then at least to me, it would be a no-brainer to use it instead and to adapt it as best practice.
DG
`inspect.getcallargs` can seemingly be modified for such behaviour. I just wrote a decorator, which does what I proposed using `inspect` module for a chosen sentinel value. The issue is that it would be a bottleneck if used on any callable, which is continuously used. `inspect.getcallargs`, `signature`, `getfullargspec` are very expensive.
If that can be done, theoretically it should be able to be done at lower level as well. After all, behaviour of it should be modelled after what is happening at the function call.
Since you clearly are not listening to the discussion, I will leave
you with one final recommendation: Write some code. Don't just claim
that it's possible; write actual code that makes it happen. You will
discover exactly how hard it is. If I am wrong, you will be able to
PROVE that I am wrong, instead of merely claiming it.
I think the question is, how is that fundamentally different than the
value "None". Once you create this "special value", some function will
decide to make it have a meaning as a passed in value, and then need a
DIFFERENT "special value" as a default.
On 7/23/23 1:13 AM, Dom Grigonis wrote:
Ok, maybe there are some crucial functions, which have argument defaults. But again, there is another solution then, implement efficient decorator in standard library, maybe as part of sentinels, maybe as part of your PEP, maybe separately.
I never said that it isn’t hard to write low-level python code. I will do that when the time is right for me. And I am not claiming that it is possible. If you read my language, all of my doubts are clearly expressed. You just seem to read everything as black and white even when it is not so.
However, such statements are in no way valid arguments in such discussion. If my idea is not optimal, not necessary, no-one needs it or any other valid responses, why it is not a good one - I can accept it.
But commenting that “it makes no sense” without properly explaining to me why after giving it some thought - it’s not fair.
Also, if you looked at things from a bit more positive perspective, maybe you could come up with some nice new ways how it could be done. Maybe not exactly what I am proposing, but some alternative, which would make it work. Most likely something much better than I am proposing.
After all, I am trying to see how your PEP can be improved, because if it could be used in all cases, where None can be, then at least to me, it would be a no-brainer to use it instead and to adapt it as best practice.
`inspect.getcallargs` can seemingly be modified for such behaviour. I just wrote a decorator, which does what I proposed using `inspect` module for a chosen sentinel value. The issue is that it would be a bottleneck if used on any callable, which is continuously used. `inspect.getcallargs`, `signature`, `getfullargspec` are very expensive.
If that can be done, theoretically it should be able to be done at lower level as well. After all, behaviour of it should be modelled after what is happening at the function call.
Since you clearly are not listening to the discussion, I will leave
you with one final recommendation: Write some code. Don't just claim
that it's possible; write actual code that makes it happen. You will
discover exactly how hard it is. If I am wrong, you will be able to
PROVE that I am wrong, instead of merely claiming it.
It’s 2 options:
a) It is a special constant - the only one with such property. Then it would be used solely for this purpose. I don’t think you can prevent someone from using it for something else. But I don’t think this case of misusage is a big exception to the way things are. As this would be only such constant, it then should be used only for this purpose. Standard sentinels would be used for everything else and None could be left for cross-language cases, such as json null value and similar.
But as of now, it is not possible to do what I am referring to with either sentinels or None. The standard is to override default such as None with itself, witch means ‘NotProvided', but it actually is. Then that value is checked within the function’s body and if it is ’NotProvided’, then default is set. I appreciate that None value is singleton and technically it is the same one, but special value is useful if one was to have a more complex default argument and wanted to enforce it from outside, while retaining ability to specify it as well. It is simply cleaner than what is being done now. And without this PEP, I don’t think there would be a big case for it, but with this PEP, it seems more compelling as it seems to encourage more complex defaults.
So no, I don’t think it is likely that another special value would be needed. All the other “special values” can be handled with sentinels. If this case is not special enough, then (b) could work.
b) Property of a callable to specify an object which is treated in such manner as special constant in (a) or a decorator which emulates such behaviour. This might be even better approach as long as it is performant and reliable.
Decorator could be a good option, but that depends on how performant it is. If it’s the same as e.g. `getcallargs` then, at least to me, not a very attractive option.
But I think there might be case for special value as well.
But again, only my opinion, I don’t really have an issue with the way things currently are with function arguments. Just answered questionnaire regarding this PEP and this is my opinion.
——————
I am more excited about deferred-eval, which if was implemented with neat syntax, would improve things for me in many different places in one go and from what I have seen would benefit many different areas and provide more neat solutions for several cases that came to this group & fairly elegant (although maybe not as pythonic) ones for several PEPs that were deferred or rejected.
DG
I think the question is, how is that fundamentally different than the value "None". Once you create this "special value", some function will decide to make it have a meaning as a passed in value, and then need a DIFFERENT "special value" as a default.
Ok, maybe there are some crucial functions, which have argument defaults. But again, there is another solution then, implement efficient decorator in standard library, maybe as part of sentinels, maybe as part of your PEP, maybe separately.
I never said that it isn’t hard to write low-level python code. I will do that when the time is right for me. And I am not claiming that it is possible. If you read my language, all of my doubts are clearly expressed. You just seem to read everything as black and white even when it is not so.
However, such statements are in no way valid arguments in such discussion. If my idea is not optimal, not necessary, no-one needs it or any other valid responses, why it is not a good one - I can accept it.
But commenting that “it makes no sense” without properly explaining to me why after giving it some thought - it’s not fair.
Also, if you looked at things from a bit more positive perspective, maybe you could come up with some nice new ways how it could be done. Maybe not exactly what I am proposing, but some alternative, which would make it work. Most likely something much better than I am proposing.
After all, I am trying to see how your PEP can be improved, because if it could be used in all cases, where None can be, then at least to me, it would be a no-brainer to use it instead and to adapt it as best practice.
`inspect.getcallargs` can seemingly be modified for such behaviour. I just wrote a decorator, which does what I proposed using `inspect` module for a chosen sentinel value. The issue is that it would be a bottleneck if used on any callable, which is continuously used. `inspect.getcallargs`, `signature`, `getfullargspec` are very expensive.
If that can be done, theoretically it should be able to be done at lower level as well. After all, behaviour of it should be modelled after what is happening at the function call.
Since you clearly are not listening to the discussion, I will leave
you with one final recommendation: Write some code. Don't just claim
that it's possible; write actual code that makes it happen. You will
discover exactly how hard it is. If I am wrong, you will be able to
PROVE that I am wrong, instead of merely claiming it.