Keyword arguments self-assignment
I'm opening this thread to discuss and collect feedback on a language change to support keyword arguments to be self-assigned using variables names. Proposal -------- Taking the syntax from [bpo-36817](https://bugs.python.org/issue36817) which just [made it to Python 3.8](https://docs.python.org/3/whatsnew/3.8.html#f-strings-support-for-self-docum...) the `<keyword name>=` syntax would be valid and have the the same effect as `<keyword name>=<keyword name>`, so these two statements would be equivalent: ```python foo(bar=bar, qux=qux) foo(bar=, qux=) ``` This syntax would only be valid inside functions calls so all of the following would remain invalid syntax: ```python x = 42 x= # SyntaxError ``` ```python x: Any [x= for x in foo] # SyntaxError ``` ```python x = 42 def foo(x=): ... # SyntaxError ``` The self-documenting variables in f-strings would also remain unchanged as it is not related to function calls: ```python x = "foo" f"{x=}" # x=foo ``` ```python x = "..." f"{foo(x=)=}" # foo(x=)=... ``` Also, this proposal does not intent to extend this syntax in any form to the dictionary literal declaration syntax. Although possible, language changes to support these are not being discussed here: ```python x = "..." foo["x"] = # just as stated previously, SyntaxError still ``` ```python x, y = 1, 2 foo = { x=, y=, } # SyntaxError ``` Worth noting that the following would now be possible with the proposed syntax: ```python x, y = 1, 2 foo = dict( x=, y=, ) ``` Rationale --------- Keyword arguments are useful for giving better semantics to parameters being passed onto a function and to lift the responsibility from the caller to know arguments order (and possibly avoid mismatching parameters due to wrong order). Said that, keyword and keyword-only arguments are extensively used across Python code and it is quite common to find code that forwards keyword parameters having to re-state keyword arguments names: ```python def foo(a, b, c): d = ... bar(a=a, b=b, c=c, d=d) ``` Of course if `bar` arguments are not keyword-only one could just omit keywords, i.e. `bar(a, b, c, d)`. Though now the caller has to be aware of arguments ordering again. And yet, if arguments are keyword-only then there is nothing you can do. Except for relying on varkwargs: ```python def foo(**kwargs): kwargs["d"] = ... bar(**kwargs) ``` Which, beyond having its own disadvantages, defeats its own purpose of avoiding repeating or reassigning the keys real quickly if one has to work with these arguments in `foo` before calling `bar`: ```python def foo(**kwargs): kwargs["d"] = kwargs["a"] * kwargs["b"] / kwargs["c"] bar(**kwargs) ``` With all that's being said, introducing self-assigned keywords would be useful for a wide range of Python users while being little intrusive on the Python syntax and backwards compatible. Best regards, Rodrigo Martins de Oliveira
On 16/04/2020 16:23, oliveira.rodrigo.m@gmail.com wrote:
I'm opening this thread to discuss and collect feedback on a language change to support keyword arguments to be self-assigned using variables names.
Proposal --------
Taking the syntax from [bpo-36817](https://bugs.python.org/issue36817) which just [made it to Python 3.8](https://docs.python.org/3/whatsnew/3.8.html#f-strings-support-for-self-docum...) the `<keyword name>=` syntax would be valid and have the the same effect as `<keyword name>=<keyword name>`, so these two statements would be equivalent:
```python foo(bar=bar, qux=qux) foo(bar=, qux=) ```
I wasn't in favour of the original proposal, and that at least had the excuse of just being for debugging. Imagine how much less I am enthused by this. Explicit is better than implicit. -- Rhodri James *-* Kynesim Ltd
This (or something really similar) has been brought up recently on this list. Please go look for that, and see how it was resolved at the time. But for now: -1 -- this is not THAT common a pattern, and to the extent that it is, this would encourage people to over-use it, and lead to errors. I find that newbies are already confused enough about scope, and why the "x" in one place is not the same as the "x" in another. This would just blur that line even more. -CHB On Thu, Apr 16, 2020 at 8:49 AM Rhodri James <rhodri@kynesim.co.uk> wrote:
I'm opening this thread to discuss and collect feedback on a language change to support keyword arguments to be self-assigned using variables names.
Proposal --------
Taking the syntax from [bpo-36817](https://bugs.python.org/issue36817) which just [made it to Python 3.8]( https://docs.python.org/3/whatsnew/3.8.html#f-strings-support-for-self-docum...)
On 16/04/2020 16:23, oliveira.rodrigo.m@gmail.com wrote: the `<keyword name>=` syntax would be valid and have the the same effect as `<keyword name>=<keyword name>`, so these two statements would be equivalent:
```python foo(bar=bar, qux=qux) foo(bar=, qux=) ```
I wasn't in favour of the original proposal, and that at least had the excuse of just being for debugging. Imagine how much less I am enthused by this.
Explicit is better than implicit.
-- Rhodri James *-* Kynesim Ltd _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RCTOW4... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Here's a similar thread: https://mail.python.org/archives/list/python-ideas@python.org/thread/SQKZ273... Personally I write code like this all the time and would love this feature, but I can see how it would be a mess for beginners to learn. On Thu, Apr 16, 2020 at 6:36 PM Christopher Barker <pythonchb@gmail.com> wrote:
This (or something really similar) has been brought up recently on this list.
Please go look for that, and see how it was resolved at the time.
But for now: -1 -- this is not THAT common a pattern, and to the extent that it is, this would encourage people to over-use it, and lead to errors.
I find that newbies are already confused enough about scope, and why the "x" in one place is not the same as the "x" in another. This would just blur that line even more.
-CHB
On Thu, Apr 16, 2020 at 8:49 AM Rhodri James <rhodri@kynesim.co.uk> wrote:
I'm opening this thread to discuss and collect feedback on a language change to support keyword arguments to be self-assigned using variables names.
Proposal --------
Taking the syntax from [bpo-36817](https://bugs.python.org/issue36817) which just [made it to Python 3.8]( https://docs.python.org/3/whatsnew/3.8.html#f-strings-support-for-self-docum...)
On 16/04/2020 16:23, oliveira.rodrigo.m@gmail.com wrote: the `<keyword name>=` syntax would be valid and have the the same effect as `<keyword name>=<keyword name>`, so these two statements would be equivalent:
```python foo(bar=bar, qux=qux) foo(bar=, qux=) ```
I wasn't in favour of the original proposal, and that at least had the excuse of just being for debugging. Imagine how much less I am enthused by this.
Explicit is better than implicit.
-- Rhodri James *-* Kynesim Ltd _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RCTOW4... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD
Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/Y3V57V... Code of Conduct: http://python.org/psf/codeofconduct/
Thanks for pointing the previous discussion @ChristopherBarker Proposals are similar but the scope here is limited to function calls which does prevent a bunch of issues already pointed out in the previous discussion.
Do any other languages already have this feature? Can you show some actual real-life code that would benefit from this, as opposed to pretend code like: foo(bar=, qux=) I think that if I had seen this syntax as a beginner, I would have had absolutely no idea how to interpret it. I probably would have decided that Python was an unreadably cryptic language and gone on to learn something else. Of course, with 20+ years of experience reading and writing code, I know better now. I would interpret it as setting bar and qux to some kind of Undefined value. I am very sympathetic to the rationale: "it is quite common to find code that forwards keyword parameters having to re-state keyword arguments names" and I've discussed similar/related issues myself, e.g. here: https://mail.python.org/pipermail/python-list/2018-February/881615.html But I am not convinced that adding magic syntax to implicitly guess the value wanted as argument if it happens to match the parameter name is a good design feature. Is being explicit about the value that you are passing to a parameter really such a burden that we need special syntax to avoid stating what value we are using as the argument? I don't think it is. And I would far prefer to read explicit code like this: # Slightly modified from actual code. self.do_something( meta=meta, dunder=dunder, private=private, invert=invert, ignorecase=ignorecase, ) over the implicit version: # Looks like code I haven't finished writing :-( self.do_something(meta=, dunder=, private=, invert=, ignorecase=) Out of the millions of possible values we might pass, I don't think that the func(spam=spam) case is so special that we want to give it special syntax. -- Steven
@StevenDAprano and this goes for @RhodriJames , thank you for sharing your point of view. Indeed the proposed syntax is obscure and would not be that readable for beginners. Couldn't we work around this so? The concept is still good for me just the syntax that is obscure, maybe something like this would work: ```python # '*' character delimits that subsequent passed parameters will be passed # as keyword arguments with the same name of the variables used self.do_something(positional, *, keyword) # this would be equivalent to: self.do_something(positional, keyword=keyword) ``` I believe this is readable even if you don't know Python: `positional` and `keyword` are being passed as parameters, the `*` character is mysterious at first but so it is in `def foo(a, *, b)` and it doesn't get into the way of basic readability. Rodrigo Martins de Oliveira
On 16/04/2020 17:57, oliveira.rodrigo.m@gmail.com wrote:
@StevenDAprano and this goes for @RhodriJames , thank you for sharing your point of view. Indeed the proposed syntax is obscure and would not be that readable for beginners.
Couldn't we work around this so? The concept is still good for me just the syntax that is obscure, maybe something like this would work:
```python # '*' character delimits that subsequent passed parameters will be passed # as keyword arguments with the same name of the variables used self.do_something(positional, *, keyword) # this would be equivalent to: self.do_something(positional, keyword=keyword) ``` I believe this is readable even if you don't know Python: `positional` and `keyword` are being passed as parameters, the `*` character is mysterious at first but so it is in `def foo(a, *, b)` and it doesn't get into the way of basic readability.
I beg to differ. I do find "def foo(a, *, b)" gets in the way of readability. -- Rhodri James *-* Kynesim Ltd
On 4/16/2020 1:30 PM, Rhodri James wrote:
On 16/04/2020 17:57, oliveira.rodrigo.m@gmail.com wrote:
@StevenDAprano and this goes for @RhodriJames , thank you for sharing your point of view. Indeed the proposed syntax is obscure and would not be that readable for beginners.
Couldn't we work around this so? The concept is still good for me just the syntax that is obscure, maybe something like this would work:
```python # '*' character delimits that subsequent passed parameters will be passed # as keyword arguments with the same name of the variables used self.do_something(positional, *, keyword) # this would be equivalent to: self.do_something(positional, keyword=keyword) ``` I believe this is readable even if you don't know Python: `positional` and `keyword` are being passed as parameters, the `*` character is mysterious at first but so it is in `def foo(a, *, b)` and it doesn't get into the way of basic readability.
I beg to differ. I do find "def foo(a, *, b)" gets in the way of readability.
And what would you do if you wanted to call: self.do_something(positional, keyword=keyword, keyword1=somethingelse, keyword2=keyword2) ? Eric
I'm really open to discuss how we can achieve this feature with clearer syntax than the first proposed version. If any of you have more ideas please share :) @EricVSmith I didn't thought it through about the syntax with the `*` character but your case is well covered: ``` self.do_something( positional, keyword1=somethingelse, *, keyword, keyword2, ) ``` Rodrigo Martins de Oliveira
And what would you do if you wanted to call:
self.do_something(positional, keyword=keyword, keyword1=somethingelse, keyword2=keyword2)
?
Eric
I think this is still pretty clear: self.do_something(positional, *, keyword, keyword1=somethingelse, keyword2) but if you don't like that you can easily add a restriction that no explicit keywords are allowed after *, so: self.do_something(positional, keyword1=somethingelse, *, keyword2, keyword)
On Apr 16, 2020, at 11:04, Alex Hall <alex.mojaki@gmail.com> wrote:
And what would you do if you wanted to call:
self.do_something(positional, keyword=keyword, keyword1=somethingelse, keyword2=keyword2)
?
Eric
I think this is still pretty clear:
self.do_something(positional, *, keyword, keyword1=somethingelse, keyword2)
This kind of reminds me of C++ lambda capture specs, which sound like they’d be terribly confusing when you read about them, but in practice, capturing [name, count=count+1, values, parent] turns out to be something you can usually write without thinking about it, understand when you come back to it later, update to values=move(values) when you realize you need that even later, etc. without ever having to sit down and parse it all out. I’m not sure if this would work out the same way or not. And even if it does, that hurdle of describing the syntax in a way that people won’t get confused the way they do when they first learn the feature in C++ might be hard to overcome. But it’s at least plausible that it could be readable and learnable enough.
I’m not sure if this would work out the same way or not. And even if it does, that hurdle of describing the syntax in a way that people won’t get confused the way they do when they first learn the feature in C++ might be hard to overcome. But it’s at least plausible that it could be readable and learnable enough.
I'm not sure what's hard to explain. """ When you see a function call like this: foo(bar, *, spam, baz) with an asterisk (*) sitting on its own looking like an argument, it's a shorthand for this: foo(bar, spam=spam, baz=baz) Every argument after the asterisk gets turned into a keyword argument where the argument name is the same as the variable name, i.e. <name>=<name>. In a function *definition* (as opposed to a call), an asterisk sitting on its own between parameters has a different meaning, but there are similarities: every parameter after the asterisk is a keyword parameter. """
+1 on the idea, -1 on the syntax. i'm probably not a very skilled developer but i have found myself repeating variable names often enough that i've felt annoyed by it. alex hall's syntax suggested syntax seems nice. would be fun to be able to write:
a=1 b=2 dict(*, a, b) {'a': 1, 'b': 2}
I think this is still pretty clear:
self.do_something(positional, *, keyword, keyword1=somethingelse, keyword2)
but if you don't like that you can easily add a restriction that no explicit keywords are allowed after *, so:
self.do_something(positional, keyword1=somethingelse, *, keyword2, keyword)
One note about that: since kwarg dictionaries are now officially ordered, it would be a little bit of a problem to disallow this: def f(*args, **kwargs): pass f(pos1, pos2, *, kw1, kw2, kw3=other) ...and require this instead: f(pos1, pos2, kw3=other, *, kw1, kw2) ...because then the syntax is forcing the order in the **kwargs dictionary to change. now you can't use the feature at all if the order is important.
On Thu, Apr 16, 2020 at 07:50:30PM +0200, Alex Hall wrote:
And what would you do if you wanted to call:
self.do_something(positional, keyword=keyword, keyword1=somethingelse, keyword2=keyword2)
?
Eric
I think this is still pretty clear:
self.do_something(positional, *, keyword, keyword1=somethingelse, keyword2)
It's not clear to me. It's currently (3.8) illegal syntax, and I have no idea what the `*` would mean in the function call. I know what it means in function definitions, but somehow we seem to have (accidentally?) flipped from talking about Rhodi's dislike of the `*` in function *definitions* to an (imaginary? proposed?) use of `*` in function *calls*. Have I missed something? -- Steven
@StevenDAprano we are discussing alternative syntaxes to enhance readability and in place of blank assignments `k=` we could possibly use a sole `*` character as indicative that the following parameters are all to be passed as keywords. In this syntax, the keyword to which the parameter will be assigned can be explicit as in `f(keyword=something)`, i.e. `something` is assigned to the `keyword` argument; or implicitly as in `f(*, something)`, i.e. `something` is assigned to the argument with same name of the parameter. This thread is actually addressed to this reply: oliveira.rodrigo.m＠gmail.com wrote:
@StevenDAprano and this goes for @RhodriJames , thank you for sharing your point of view. Indeed the proposed syntax is obscure and would not be that readable for beginners. Couldn't we work around this so? The concept is still good for me just the syntax that is obscure, maybe something like this would work: # '*' character delimits that subsequent passed parameters will be passed # as keyword arguments with the same name of the variables used self.do_something(positional, *, keyword) # this would be equivalent to: self.do_something(positional, keyword=keyword)
I believe this is readable even if you don't know Python: positional and keyword are being passed as parameters, the * character is mysterious at first but so it is in def foo(a, *, b) and it doesn't get into the way of basic readability. Rodrigo Martins de Oliveira
I believe some of us just clicked "Reply" on Mailman 3 and started nesting threads. Sorry.
On 17/04/2020 04:04, Steven D'Aprano wrote:
I know what it means in function definitions, but somehow we seem to have (accidentally?) flipped from talking about Rhodi's dislike of the `*` in function *definitions* to an (imaginary? proposed?) use of `*` in function *calls*.
Have I missed something?
Someone was proposing using '*' to mean "the following are keyword parameters, take their values from variables of the same name", citing the use of '*' in function definitions as looking nice and being obvious in meaning. I was disputing both points. We do seem to have got off-track somewhat ;-) -- Rhodri James *-* Kynesim Ltd
On Fri, Apr 17, 2020 at 5:08 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Thu, Apr 16, 2020 at 07:50:30PM +0200, Alex Hall wrote:
And what would you do if you wanted to call:
self.do_something(positional, keyword=keyword, keyword1=somethingelse, keyword2=keyword2)
?
Eric
I think this is still pretty clear:
self.do_something(positional, *, keyword, keyword1=somethingelse, keyword2)
It's not clear to me. It's currently (3.8) illegal syntax, and I have no idea what the `*` would mean in the function call.
Of course it's currently illegal syntax, that's the point. I don't think you really need to know what it means to read the code for most purposes. You look at the function call and you can see a bunch of names being passed to self.do_something. If the function call has 'keyword=keyword' in it instead of just 'keyword', that's not adding much information. The technical details of how an argument is being passed are usually not important to readers.
I know what it means in function definitions, but somehow we seem to have (accidentally?) flipped from talking about Rhodi's dislike of the `*` in function *definitions* to an (imaginary? proposed?) use of `*` in function *calls*.
Have I missed something?
Something weird seems to have happened in this thread. Rodrigo first proposed the magic asterisk syntax here: https://mail.python.org/archives/list/python-ideas@python.org/message/N2ZY5N... There were some replies discussing that proposal, including objections, to which I responded: https://mail.python.org/archives/list/python-ideas@python.org/message/KTJBGO... For some reason Ricky called it my suggestion when it was Rodrigo's: https://mail.python.org/archives/list/python-ideas@python.org/message/QYC6F4... And now it looks like you missed it too.
On Fri, Apr 17, 2020 at 11:25:15AM +0200, Alex Hall wrote:
I don't think you really need to know what it means to read the code for most purposes.
*blink*
You look at the function call and you can see a bunch of names being passed to self.do_something. If the function call has 'keyword=keyword' in it instead of just 'keyword', that's not adding much information.
Of course it adds important information! It is adding the critical information: which parameter gets assigned the specified value. Put yourself in the position of someone who doesn't know the meaning of the * in a function call. What's the difference between these two calls? f(x) f(*, x) It's not good enough to merely say that you're passing an argument `x`. That's true of both calls. There must be a difference between the two, otherwise why have the star? Positional arguments tell us that values are assigned to parameters from left to right: function(spam, # first parameter eggs, # second parameter cheese, # third parameter ) but we have no clue at all what the names of those parameters are. That's the down-side of positional arguments, and one of the reasons why positional arguments don't scale. Knowing the names of the parameters is often important. Keyword arguments fill in the missing critical information: function(alpha=spam, beta=eggs, gamma=cheese, ) and we can shuffle the order around and the mapping of argument value to parameter name is still clear.
The technical details of how an argument is being passed are usually not important to readers.
I don't care about the argument passing implementation. I care about the meaning of the code. Here is a real-world case: open('example.txt', encoding, newline, errors, mode) open('example.txt', *, encoding, newline, errors, mode) Would you agree that the two of those have *radically* different meanings? The first will, if you are lucky, fail immediately; the second may succeed. But to the poor unfortunate reader who doesn't know what the star does, the difference is puzzling. This holds even more strongly if the various parameters take the same type: # a real signature from one of my functions def function( meta:bool, dunder:bool, private:bool, ignorecase:bool, invert:bool) function(dunder, invert, private, meta, ignorecase) function(*, dunder, invert, private, meta, ignorecase) [...]
Have I missed something?
Something weird seems to have happened in this thread. Rodrigo first proposed the magic asterisk syntax here: https://mail.python.org/archives/list/python-ideas@python.org/message/N2ZY5N...
Ah, that's the bit I missed. -- Steven
open('example.txt', encoding, newline, errors, mode)
open('example.txt', *, encoding, newline, errors, mode)
Would you agree that the two of those have *radically* different meanings? The first will, if you are lucky, fail immediately; the second may succeed. But to the poor unfortunate reader who doesn't know what the star does, the difference is puzzling.
This holds even more strongly if the various parameters take the same type:
# a real signature from one of my functions def function( meta:bool, dunder:bool, private:bool, ignorecase:bool, invert:bool)
function(dunder, invert, private, meta, ignorecase) function(*, dunder, invert, private, meta, ignorecase)
[...]
Have I missed something?
Something weird seems to have happened in this thread. Rodrigo first proposed the magic asterisk syntax here:
https://mail.python.org/archives/list/python-ideas@python.org/message/N2ZY5N...
Ah, that's the bit I missed.
I missed that message too. But even having missed the original proposal, it was immediately obvious to me what the code was saying with very little intro from Alex Hall, because of its similarity to keyword-only function definition syntax. It feels like the meaning is obvious. However I'll agree with Steve D' that it could easily lead to some confusing reading. The example I quoted above is compelling. At first I was tempted to say: "hey now! That code was written in a puppet obfuscatory way! Why would someone change the order of the arguments like that?" But the answer came to me: because of the proposed syntax I am feeling very free to just splatter all the arguments the function asked for in any order-- I'm not going to scroll up to read the docs to mimic the safe order especially if the parameters are keyword only. But this means the reader could miss the star, especially with a very large function call over multiple lines, and if that reader happens to use that particular function A LOT and know the parameter order without having to look they would pretty easily believe the arguments are doing something different than what is actually happening. It's an interesting problem with the syntax. I'm not sure if it moves me in the negative on the proposal or not.
But this means the reader could miss the star, especially with a very large function call over multiple lines, and if that reader happens to use that particular function A LOT and know the parameter order without having to look they would pretty easily believe the arguments are doing something different than what is actually happening.
Thank you for giving an actual scenario explaining how confusion could occur. Personally I think it's a very unlikely edge case (particularly someone comparing argument order to their memory), and someone falsely thinking that correct code is buggy is not a major problem anyway. I propose using two asterisks instead of one as the magical argument separator. `**` is more closely associated with keyword arguments, it's harder to visually miss, and it avoids the problem mentioned [here]( https://mail.python.org/archives/list/python-ideas@python.org/message/XFZ5VH...) which I think was a valid point. So a call would look like: function(**, dunder, invert, private, meta, ignorecase)
On 17.04.20 12:49, Alex Hall wrote:
But this means the reader could miss the star, especially with a very large function call over multiple lines, and if that reader happens to use that particular function A LOT and know the parameter order without having to look they would pretty easily believe the arguments are doing something different than what is actually happening.
Thank you for giving an actual scenario explaining how confusion could occur. Personally I think it's a very unlikely edge case (particularly someone comparing argument order to their memory), and someone falsely thinking that correct code is buggy is not a major problem anyway.
I propose using two asterisks instead of one as the magical argument separator. `**` is more closely associated with keyword arguments, it's harder to visually miss, and it avoids the problem mentioned [here](https://mail.python.org/archives/list/python-ideas@python.org/message/XFZ5VH...) which I think was a valid point. So a call would look like:
function(**, dunder, invert, private, meta, ignorecase)
In that case, would you also allow `**kwargs` unpacking to serve as a separator? That is: function(**kwargs, dunder, invert, private, meta, ignorecase) Currently this is a SyntaxError. I think it would fit the symmetry with respect to `def func(*args, bar)` vs. `def func(*, bar)`; whether or not there is something to unpack, what follows after it remains unaffected.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WJYJBO... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Apr 17, 2020 at 8:52 PM Alex Hall <alex.mojaki@gmail.com> wrote:
But this means the reader could miss the star, especially with a very large function call over multiple lines, and if that reader happens to use that particular function A LOT and know the parameter order without having to look they would pretty easily believe the arguments are doing something different than what is actually happening.
Thank you for giving an actual scenario explaining how confusion could occur. Personally I think it's a very unlikely edge case (particularly someone comparing argument order to their memory), and someone falsely thinking that correct code is buggy is not a major problem anyway.
I propose using two asterisks instead of one as the magical argument separator. `**` is more closely associated with keyword arguments, it's harder to visually miss, and it avoids the problem mentioned [here](https://mail.python.org/archives/list/python-ideas@python.org/message/XFZ5VH...) which I think was a valid point. So a call would look like:
function(**, dunder, invert, private, meta, ignorecase)
All of these mode-switch proposals have the fundamental problem that you then cannot mix shorthand and longhand forms - once you go shorthand, suddenly everything has to be shorthand. I don't like that. The original proposal was entirely self-contained and has a very simple interpretation. Consider this example of actual production code (same one as I posted earlier): return render_template("index.html", twitter=twitter, username=user["display_name"], channel=channel, channelid=channelid, error=error, setups=database.list_setups(channelid), sched_tz=sched_tz, schedule=schedule, sched_tweet=sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=tweets, ) Aside from the first parameter, everything is keyword args, and there is no particular order to them. The render_template function doesn't define a set of parameters - it takes whatever it's given and passes it along to the template itself. If I could use the "name=" shorthand, I could write this thus: return render_template("index.html", twitter=, username=user["display_name"], channel=, channelid=, error=, setups=database.list_setups(channelid), sched_tz=, schedule=, sched_tweet=, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=, ) Each individual entry can use the shorthand, and it has a well-defined meaning. The order doesn't define which ones can and can't use shorthand. What's the advantage of a mode switch? This seems perfectly clear to me without any sort of magical cutoff. ChrisA
On 4/17/2020 9:28 AM, Chris Angelico wrote:
On Fri, Apr 17, 2020 at 8:52 PM Alex Hall <alex.mojaki@gmail.com> wrote:
But this means the reader could miss the star, especially with a very large function call over multiple lines, and if that reader happens to use that particular function A LOT and know the parameter order without having to look they would pretty easily believe the arguments are doing something different than what is actually happening.
Thank you for giving an actual scenario explaining how confusion could occur. Personally I think it's a very unlikely edge case (particularly someone comparing argument order to their memory), and someone falsely thinking that correct code is buggy is not a major problem anyway.
I propose using two asterisks instead of one as the magical argument separator. `**` is more closely associated with keyword arguments, it's harder to visually miss, and it avoids the problem mentioned [here](https://mail.python.org/archives/list/python-ideas@python.org/message/XFZ5VH...) which I think was a valid point. So a call would look like:
function(**, dunder, invert, private, meta, ignorecase)
All of these mode-switch proposals have the fundamental problem that you then cannot mix shorthand and longhand forms - once you go shorthand, suddenly everything has to be shorthand. I don't like that. The original proposal was entirely self-contained and has a very simple interpretation. Consider this example of actual production code (same one as I posted earlier):
return render_template("index.html", twitter=twitter, username=user["display_name"], channel=channel, channelid=channelid, error=error, setups=database.list_setups(channelid), sched_tz=sched_tz, schedule=schedule, sched_tweet=sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=tweets, )
Aside from the first parameter, everything is keyword args, and there is no particular order to them. The render_template function doesn't define a set of parameters - it takes whatever it's given and passes it along to the template itself. If I could use the "name=" shorthand, I could write this thus:
return render_template("index.html", twitter=, username=user["display_name"], channel=, channelid=, error=, setups=database.list_setups(channelid), sched_tz=, schedule=, sched_tweet=, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=, )
Each individual entry can use the shorthand, and it has a well-defined meaning. The order doesn't define which ones can and can't use shorthand.
What's the advantage of a mode switch? This seems perfectly clear to me without any sort of magical cutoff.
I agree with Chris. I'm not a fan of the original proposal with the "=" (because I don't think this is a problem that needs solving), but at least it made more sense than a mode-switch among the list of parameters. Eric
What's the advantage of a mode switch? This seems perfectly clear to
me without any sort of magical cutoff.
I agree with Chris. I'm not a fan of the original proposal with the "=" (because I don't think this is a problem that needs solving), but at least it made more sense than a mode-switch among the list of parameters.
Eric
What's the advantage of a mode switch? This seems perfectly clear to
me without any sort of magical cutoff.
ChrisA
Here's a specific advantage I had in mind (it might not be considered very significant for many and that's ok): copying and pasting. If I have a function I want to call, sometimes I'll often hyper-click through to the function and copy the signature instead of typing it all out. I do this to save a little typing but also in large part so that I have the entire function signature right in front of me while I type out the function call, and that helps prevent little errors-- forgetting a parameter or calling one incorrectly, that sort of thing. So if you had a function like this: def f(a, b, *, c=None, d=None, e=None, f=None, g=None): ... And wanted to call it like this: f(a=a, b=b, c=c, g=g) An easy way to reliably type the call is to copy the function signature, delete the parts you aren't going to use, and then type all of the "=a", "=b", etc parts. A nice thing about the mode switch syntax could be that it makes this routine faster (assuming there are no type hints!!!): f(*, a, b, c, g) All you have to do is add the *, delete the parts you don't need, and you're done. This is a small thing. But it got me excited about the proposed syntax.
What's the advantage of a mode switch? This seems perfectly clear to
me without any sort of magical cutoff.
I agree with Chris. I'm not a fan of the original proposal with the "=" (because I don't think this is a problem that needs solving), but at least it made more sense than a mode-switch among the list of parameters.
Eric
What's the advantage of a mode switch? This seems perfectly clear to
me without any sort of magical cutoff.
ChrisA
Here's a specific advantage I had in mind (it might not be considered very significant for many and that's ok): copying and pasting.
If I have a function I want to call, sometimes I'll often hyper-click through to the function and copy the signature instead of typing it all out.
I do this to save a little typing but also in large part so that I have the entire function signature right in front of me while I type out the function call, and that helps prevent little errors-- forgetting a parameter or calling one incorrectly, that sort of thing.
So if you had a function like this:
def f(a, b, *, c=None, d=None, e=None, f=None, g=None): ...
And wanted to call it like this:
f(a=a, b=b, c=c, g=g)
An easy way to reliably type the call is to copy the function signature, delete the parts you aren't going to use, and then type all of the "=a", "=b", etc parts.
A nice thing about the mode switch syntax could be that it makes this routine faster (assuming there are no type hints!!!):
f(*, a, b, c, g)
All you have to do is add the *, delete the parts you don't need, and you're done.
This is a small thing. But it got me excited about the proposed syntax.
Sorry: for clarity, I understand the keyword-only parts are the only ones you currently *MUST* type "=c", "=g" for. I should have not said "=a" and "=b".
Ricky Teachey writes:
A nice thing about the mode switch syntax could be that it makes [the convert prototype to call] routine faster (assuming there are no type hints!!!):
f(*, a, b, c, g)
The trailing equals is faster (as long as you have defaults as in your example), because you don't have to type the "*," part. I don't think it's worth the loss of call flexibility (although a lot of that is because I use keyboard macros a lot, and this is eminently macro-able, or even a simple editor extension). Steve
@ChrisAngelico I do prefer the original proposal, though I do see the point of it being harder for beginner to understand. The mode-switch proposal though would not impede one to mix shorthand and longhand forms. This should be valid syntax: ```python return render_template("index.html", *, twitter, username=user["display_name"], channel, channelid, error, setups=database.list_setups(channelid), sched_tz, schedule, sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets, ) ``` I'll wait the weekend is through to then assess if we can reach consensus or just reject the proposal. Rodrigo Martins de Oliveira
On Fri, Apr 17, 2020 at 11:57 PM <oliveira.rodrigo.m@gmail.com> wrote:
@ChrisAngelico I do prefer the original proposal, though I do see the point of it being harder for beginner to understand.
The mode-switch proposal though would not impede one to mix shorthand and longhand forms. This should be valid syntax:
```python return render_template("index.html", *, twitter, username=user["display_name"], channel, channelid, error, setups=database.list_setups(channelid), sched_tz, schedule, sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets, ) ```
I'll wait the weekend is through to then assess if we can reach consensus or just reject the proposal.
Hmm, I see what you mean. It's not a modeswitch to shorthand, it's a modeswitch to keyword-only parameters. I think it's still vulnerable to the problem of near-identical syntax having extremely different semantics, but it's at least less annoying that way. But I still definitely prefer the original proposal. ChrisA
On Fri, Apr 17, 2020 at 9:57 AM <oliveira.rodrigo.m@gmail.com> wrote:
The mode-switch proposal though would not impede one to mix shorthand and longhand forms. This should be valid syntax:
```python return render_template("index.html", *, twitter, username=user["display_name"], channel, channelid, error, setups=database.list_setups(channelid), sched_tz, schedule, sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets, ) ```
I definitely hate the above version. Intermixing auto-named values with bound values is super-confusing and a huge bug magnet. However, the following does not look bad ONLY if the mode-switch is strictly to bare-names-only after the switch: render_template("index.html", username=user["display_name"], setups=database.list_setups(channelid), checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), **, twitter, channel, channelid, error, sched_tz, schedule, sched_tweet, tweets, **more_kwargs) Putting the named parameters strictly first gives a hint to the fact that the rest are "special named parameters" (with auto-naming). -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Fri, Apr 17, 2020 at 6:09 PM David Mertz <mertz@gnosis.cx> wrote:
On Fri, Apr 17, 2020 at 9:57 AM <oliveira.rodrigo.m@gmail.com> wrote:
The mode-switch proposal though would not impede one to mix shorthand and longhand forms. This should be valid syntax:
```python return render_template("index.html", *, twitter, username=user["display_name"], channel, channelid, error, setups=database.list_setups(channelid), sched_tz, schedule, sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets, ) ```
I definitely hate the above version. Intermixing auto-named values with bound values is super-confusing and a huge bug magnet.
How is it confusing? How is it a bug magnet? I do think that example looks like a mess, but if each parameter is on its own line, I think it looks fine: ``` render_template( "index.html", *, twitter, username=user["display_name"], channel, channelid, error, setups=database.list_setups(channelid), sched_tz, schedule, sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets, ) ```
However, the following does not look bad ONLY if the mode-switch is strictly to bare-names-only after the switch:
render_template("index.html", username=user["display_name"], setups=database.list_setups(channelid), checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), **, twitter, channel, channelid, error, sched_tz, schedule, sched_tweet, tweets, **more_kwargs)
Putting the named parameters strictly first gives a hint to the fact that the rest are "special named parameters" (with auto-naming).
Why does that need emphasising? Are you thinking like Ricky and Steven D'Aprano that people might sometimes think that they're looking at positional arguments and get confused?
On 18/04/20 1:56 am, oliveira.rodrigo.m@gmail.com wrote:
This should be valid syntax:
```python return render_template("index.html", *, twitter, username=user["display_name"], channel, channelid, error, setups=database.list_setups(channelid), sched_tz, schedule, sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets, )
Not sure I like that. The only thing telling you that the args without = are keyword args rather than positional args is the "*", which is easy to miss amidst all that other stuff. -- Greg
On Sat, Apr 18, 2020 at 10:21:46AM +1200, Greg Ewing wrote:
On 18/04/20 1:56 am, oliveira.rodrigo.m@gmail.com wrote:
This should be valid syntax:
```python return render_template("index.html", *, twitter, username=user["display_name"], channel, channelid, error, setups=database.list_setups(channelid), sched_tz, schedule, sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets, )
Not sure I like that. The only thing telling you that the args without = are keyword args rather than positional args is the "*", which is easy to miss amidst all that other stuff.
I agree with Greg here. This is a modal interface: the use of a `*` shifts from "positional argument mode" to "auto-fill keyword mode". Modal interfaces are generally harmful and should be minimized. This doesn't just apply to GUIs but to text interfaces too, and code is an interface between what I, the coder, wants and the computer. https://wiki.inkscape.org/wiki/index.php/Modal_interfaces Modes are okay when they are really big (e.g. "I'm in Python programming mode now") but otherwise should be minimized, with an obvious beginning and end. If you have to have a mode, they should be *really obvious*, and this isn't. There's a really fine difference between modes: my_super_function_with_too_many_parameters( args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, kwargs, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, *, pass_fds, encoding, errors, text, file, mode, buffering, newline, closefd, opener, meta, private, dunder, invert, ignorecase, ascii_only, seed, bindings, callback, log, font, size, style, justify, pumpkin, ) If you miss the star in the middle of the call, there's no hint in the rest of the call to clue you in to the fact that you changed modes. In function definitions we can get away with it, because it is in one place, the function signature, but even there it is easy to miss unless you look carefully: https://docs.python.org/3.8/library/subprocess.html#subprocess.Popen But we call functions far more often than we define them, and here a mode shift is too easy to miss, or neglect. -- Steven
Hmmm... I disagree with Chris. I'm definitely -1 on a magic dangling 'foo=' after variable names. And something less than -1 on the even more magic "Lisp symbol that isn't a symbol" ':foo'. Those are just ugly and mysterious. However, I don't HATE the "mode switch" use of '*' or '**' in function calls. I've certainly written plenty of code where I use the same variable name in the calling scope as I bind in the call. Moreover, function *definitions* have an an analogous mode switch with an isolated '*'. I'm only -0 on the mode switch style. Another thing to learn isn't really work the characters saved. But it's not awful. On Fri, Apr 17, 2020, 9:29 AM Chris Angelico <rosuav@gmail.com> wrote:
But this means the reader could miss the star, especially with a very
large function call over multiple lines, and if that reader happens to use
On Fri, Apr 17, 2020 at 8:52 PM Alex Hall <alex.mojaki@gmail.com> wrote: that particular function A LOT and know the parameter order without having to look they would pretty easily believe the arguments are doing something different than what is actually happening.
Thank you for giving an actual scenario explaining how confusion could
occur. Personally I think it's a very unlikely edge case (particularly someone comparing argument order to their memory), and someone falsely thinking that correct code is buggy is not a major problem anyway.
I propose using two asterisks instead of one as the magical argument
separator. `**` is more closely associated with keyword arguments, it's harder to visually miss, and it avoids the problem mentioned [here]( https://mail.python.org/archives/list/python-ideas@python.org/message/XFZ5VH...) which I think was a valid point. So a call would look like:
function(**, dunder, invert, private, meta, ignorecase)
All of these mode-switch proposals have the fundamental problem that you then cannot mix shorthand and longhand forms - once you go shorthand, suddenly everything has to be shorthand. I don't like that. The original proposal was entirely self-contained and has a very simple interpretation. Consider this example of actual production code (same one as I posted earlier):
return render_template("index.html", twitter=twitter, username=user["display_name"], channel=channel, channelid=channelid, error=error, setups=database.list_setups(channelid), sched_tz=sched_tz, schedule=schedule, sched_tweet=sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=tweets, )
Aside from the first parameter, everything is keyword args, and there is no particular order to them. The render_template function doesn't define a set of parameters - it takes whatever it's given and passes it along to the template itself. If I could use the "name=" shorthand, I could write this thus:
return render_template("index.html", twitter=, username=user["display_name"], channel=, channelid=, error=, setups=database.list_setups(channelid), sched_tz=, schedule=, sched_tweet=, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=, )
Each individual entry can use the shorthand, and it has a well-defined meaning. The order doesn't define which ones can and can't use shorthand.
What's the advantage of a mode switch? This seems perfectly clear to me without any sort of magical cutoff.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/3HZIDW... Code of Conduct: http://python.org/psf/codeofconduct/
On 18/04/20 1:28 am, Chris Angelico wrote:
What's the advantage of a mode switch?
I don't particularly like the mode switch either. Personally I'd be happy with either f(arg=) or f(=arg) The latter is maybe more suggestive that "arg" is to be evaluated in the local scope. -- Greg
Here's another idea for the bikeshed: f(spam, pass eggs, ham) equivalent to f(spam, eggs=eggs, ham=ham) -- Greg
On Sat, Apr 18, 2020 at 12:22:36AM +1200, Greg Ewing wrote:
Here's another idea for the bikeshed:
f(spam, pass eggs, ham)
How is "pass" meaningful here? To me this looks like a choice of a random keyword: f(spam, import eggs, ham) I don't see the link between a keyword that means "do nothing" and a feature that means "swap parameters to keyword-self-assignment mode". -- Steven
On Fri, Apr 17, 2020 at 11:59 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Apr 18, 2020 at 12:22:36AM +1200, Greg Ewing wrote:
Here's another idea for the bikeshed:
f(spam, pass eggs, ham)
How is "pass" meaningful here? To me this looks like a choice of a random keyword:
f(spam, import eggs, ham)
I don't see the link between a keyword that means "do nothing" and a feature that means "swap parameters to keyword-self-assignment mode".
Oh but Steven, Steven, Steven, how can you pass up an opportunity to reignite the fires of "pass by value" and "pass by reference"? This is CLEARLY the way to represent pass-by-reference where the reference is to a mythical value... ChrisA
On Sat, Apr 18, 2020 at 12:01:55AM +1000, Chris Angelico wrote:
Oh but Steven, Steven, Steven, how can you pass up an opportunity to reignite the fires of "pass by value" and "pass by reference"? This is CLEARLY the way to represent pass-by-reference where the reference is to a mythical value...
Oh lord, for a second there I read that in the exact tones of Ranting Rick's posts. Time to step away from the computer, I think. -- Steven
On Sat, Apr 18, 2020 at 1:35 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Apr 18, 2020 at 12:01:55AM +1000, Chris Angelico wrote:
Oh but Steven, Steven, Steven, how can you pass up an opportunity to reignite the fires of "pass by value" and "pass by reference"? This is CLEARLY the way to represent pass-by-reference where the reference is to a mythical value...
Oh lord, for a second there I read that in the exact tones of Ranting Rick's posts.
Time to step away from the computer, I think.
Haha, that was the kind of thing I was parodying :) On Sat, Apr 18, 2020 at 1:54 AM David Mertz <mertz@gnosis.cx> wrote:
Hmmm... I disagree with Chris.
I'm definitely -1 on a magic dangling 'foo=' after variable names. And something less than -1 on the even more magic "Lisp symbol that isn't a symbol" ':foo'.
Those are just ugly and mysterious.
However, I don't HATE the "mode switch" use of '*' or '**' in function calls. I've certainly written plenty of code where I use the same variable name in the calling scope as I bind in the call. Moreover, function *definitions* have an an analogous mode switch with an isolated '*'.
It sounds to me like there's a lot of weak support or weak opposition, with some of it spread between the proposal itself and the individual spellings. Rodrigo, it may be time to start thinking about writing a PEP. If the Steering Council approves, I would be willing to be a (non-core-dev) sponsor; alternatively, there may be others who'd be willing to sponsor it. A PEP will gather all the different syntax options and the arguments for/against each, and will mean we're not going round and round on the same discussion points all the times. ChrisA
On 4/17/2020 12:28 PM, Chris Angelico wrote:
On Sat, Apr 18, 2020 at 1:54 AM David Mertz <mertz@gnosis.cx> wrote:
Hmmm... I disagree with Chris.
I'm definitely -1 on a magic dangling 'foo=' after variable names. And something less than -1 on the even more magic "Lisp symbol that isn't a symbol" ':foo'.
Those are just ugly and mysterious.
However, I don't HATE the "mode switch" use of '*' or '**' in function calls. I've certainly written plenty of code where I use the same variable name in the calling scope as I bind in the call. Moreover, function *definitions* have an an analogous mode switch with an isolated '*'.
It sounds to me like there's a lot of weak support or weak opposition, with some of it spread between the proposal itself and the individual spellings.
Rodrigo, it may be time to start thinking about writing a PEP. If the Steering Council approves, I would be willing to be a (non-core-dev) sponsor; alternatively, there may be others who'd be willing to sponsor it. A PEP will gather all the different syntax options and the arguments for/against each, and will mean we're not going round and round on the same discussion points all the times.
I've been around for a while, and I can't imagine that any of these proposals would be accepted (but I've been accused of having a bad imagination). I'm saying that not to dissuade anyone from writing a PEP: far from it. I think it would be useful to have this on paper and accepted or rejected, either way. I'm saying this to set expectations: a PEP is a ton of work, and it can be disheartening to put in so much work for something that is rejected. So, I'd be willing to sponsor such a PEP, but I'd be arguing that it get rejected. And I say this as someone who has maybe 20 hours of work left on a PEP of my own that I think has less than a 50% chance of success. I already probably have 10 to 15 hours invested in it already. Eric
On Sat, Apr 18, 2020 at 2:41 AM Eric V. Smith <eric@trueblade.com> wrote:
On 4/17/2020 12:28 PM, Chris Angelico wrote:
On Sat, Apr 18, 2020 at 1:54 AM David Mertz <mertz@gnosis.cx> wrote:
Hmmm... I disagree with Chris.
I'm definitely -1 on a magic dangling 'foo=' after variable names. And something less than -1 on the even more magic "Lisp symbol that isn't a symbol" ':foo'.
Those are just ugly and mysterious.
However, I don't HATE the "mode switch" use of '*' or '**' in function calls. I've certainly written plenty of code where I use the same variable name in the calling scope as I bind in the call. Moreover, function *definitions* have an an analogous mode switch with an isolated '*'.
It sounds to me like there's a lot of weak support or weak opposition, with some of it spread between the proposal itself and the individual spellings.
Rodrigo, it may be time to start thinking about writing a PEP. If the Steering Council approves, I would be willing to be a (non-core-dev) sponsor; alternatively, there may be others who'd be willing to sponsor it. A PEP will gather all the different syntax options and the arguments for/against each, and will mean we're not going round and round on the same discussion points all the times.
I've been around for a while, and I can't imagine that any of these proposals would be accepted (but I've been accused of having a bad imagination). I'm saying that not to dissuade anyone from writing a PEP: far from it. I think it would be useful to have this on paper and accepted or rejected, either way. I'm saying this to set expectations: a PEP is a ton of work, and it can be disheartening to put in so much work for something that is rejected.
So, I'd be willing to sponsor such a PEP, but I'd be arguing that it get rejected.
Given the history of previous PEPs with surprising results, I don't want to bet on whether this would be accepted or not, but either way, yes, it will definitely be useful to have it all written down. Rodrigo, if you're willing to write the PEP, I'll happily help you with the technical side of things. ChrisA
@ChrisAngelico @EricVSmith Thank you for be willing to sponsor the PEP. I think it may be best to start writing it already, as I see it, the proposal wasn't a clear no (nor a clear yes but if it was to be, probably someone else would already have proposed something in these lines and we wouldn't be discussing this right now). I've thought of this proposal for 2 years from now and I still think to date that it would be a really nice feature. Like @ChrisAngelico, I have written and reviewed so many different code that would benefit from such a feature that I do believe this deserves consideration from the community. If this ends up on we rejecting the idea that's fine, at least we give closure to this. I do wish to carry on with writing a PEP with your help guys. What's the next step?
On Sat, Apr 18, 2020 at 6:30 AM <oliveira.rodrigo.m@gmail.com> wrote:
I do wish to carry on with writing a PEP with your help guys. What's the next step?
Your primary reference is here: https://www.python.org/dev/peps/pep-0001/#pep-workflow Since you have a sponsor, the next step is to fork the PEPs repo on GitHub and start writing PEP 9999 (the standard placeholder number). You'll find PEP 12 helpful here: https://www.python.org/dev/peps/pep-0012/ Once you've done some keyboard hammering and have yourself a PEP, create a GitHub pull request to propose it for inclusion. At that point, you'll be assigned a PEP number and can both rename your file and change its headers accordingly. And after that, post your PEP to python-ideas and let the bikeshedding commence! I mean, continue! Err, resume with full speed? Something like that anyhow! ChrisA
Thanks @ChrisAngelico! I will get to it. Once a first draft is ready I'll share the github link in here.
I think you should hold off a little bit until there is a bit more consensus on syntax. Or at least delay the parts that are specific to syntax. I think it would help your case if you avoided toy examples like the one you started with: foo(baz=baz, qux=qux) and concentrated on real examples, including from the standard library. -- Steven
Perhaps an easier next step would be to get better data about people's opinions with a simple poll? Is there a standard way to vote on things in this list? I say we do a simple approval vote. Everyone ticks all the syntaxes that they think would be acceptable to include in the language. It's not very precise but it's easy and may inform what to do next. Here is a script to generate a bunch of options: ``` for template in [ 'foo(bar, baz="thing", %s)', '{"baz": "thing", %s}', ]: for affix in ["=", ":", "::"]: for rest in [ f'spam{affix}, stuff{affix}', f'{affix}spam, {affix}stuff', ]: print(template % rest) for template in [ 'foo(bar, baz="thing", %s, spam, stuff)', '{"baz": "thing", %s, spam, stuff}', ]: for separator in ["*", "**", ":", "::", "="]: print(template % separator) # These options are currently valid syntax that are guaranteed to fail at runtime print("""\ foo(bar, baz="thing", **(spam, stuff)) foo(bar, baz="thing", **{spam, stuff}) {{"baz": "thing", spam, stuff}} """) ``` The output: ``` foo(bar, baz="thing", spam=, stuff=) foo(bar, baz="thing", =spam, =stuff) foo(bar, baz="thing", spam:, stuff:) foo(bar, baz="thing", :spam, :stuff) foo(bar, baz="thing", spam::, stuff::) foo(bar, baz="thing", ::spam, ::stuff) {"baz": "thing", spam=, stuff=} {"baz": "thing", =spam, =stuff} {"baz": "thing", spam:, stuff:} {"baz": "thing", :spam, :stuff} {"baz": "thing", spam::, stuff::} {"baz": "thing", ::spam, ::stuff} foo(bar, baz="thing", *, spam, stuff) foo(bar, baz="thing", **, spam, stuff) foo(bar, baz="thing", :, spam, stuff) foo(bar, baz="thing", ::, spam, stuff) foo(bar, baz="thing", =, spam, stuff) {"baz": "thing", *, spam, stuff} {"baz": "thing", **, spam, stuff} {"baz": "thing", :, spam, stuff} {"baz": "thing", ::, spam, stuff} {"baz": "thing", =, spam, stuff} foo(bar, baz="thing", **(spam, stuff)) foo(bar, baz="thing", **{spam, stuff}) {{"baz": "thing", spam, stuff}} ``` Are there any other options to consider? If people think this list looks complete I can create an online poll with them. I'm open to suggestions about preferred software/websites. On Fri, Apr 17, 2020 at 6:41 PM Eric V. Smith <eric@trueblade.com> wrote:
On 4/17/2020 12:28 PM, Chris Angelico wrote:
On Sat, Apr 18, 2020 at 1:54 AM David Mertz <mertz@gnosis.cx> wrote:
Hmmm... I disagree with Chris.
I'm definitely -1 on a magic dangling 'foo=' after variable names. And something less than -1 on the even more magic "Lisp symbol that isn't a symbol" ':foo'.
Those are just ugly and mysterious.
However, I don't HATE the "mode switch" use of '*' or '**' in function calls. I've certainly written plenty of code where I use the same variable name in the calling scope as I bind in the call. Moreover, function *definitions* have an an analogous mode switch with an isolated '*'.
It sounds to me like there's a lot of weak support or weak opposition, with some of it spread between the proposal itself and the individual spellings.
Rodrigo, it may be time to start thinking about writing a PEP. If the Steering Council approves, I would be willing to be a (non-core-dev) sponsor; alternatively, there may be others who'd be willing to sponsor it. A PEP will gather all the different syntax options and the arguments for/against each, and will mean we're not going round and round on the same discussion points all the times.
I've been around for a while, and I can't imagine that any of these proposals would be accepted (but I've been accused of having a bad imagination). I'm saying that not to dissuade anyone from writing a PEP: far from it. I think it would be useful to have this on paper and accepted or rejected, either way. I'm saying this to set expectations: a PEP is a ton of work, and it can be disheartening to put in so much work for something that is rejected.
So, I'd be willing to sponsor such a PEP, but I'd be arguing that it get rejected.
And I say this as someone who has maybe 20 hours of work left on a PEP of my own that I think has less than a 50% chance of success. I already probably have 10 to 15 hours invested in it already.
Eric _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/K46TLU... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, 17 Apr 2020, Alex Hall wrote:
Perhaps an easier next step would be to get better data about people's opinions with a simple poll? Is there a standard way to vote on things in this but it's easy and may inform what to do next.
For what it's worth, I'm a strong -1 on this whole thing, regardless of syntax. I think passing a lot of same-named parameters is an anti-pattern, that should be discouraged, not made easier. Passing an occasional x=x to so some function no disaster; if it happens often enough to be a problem, IMNSHO, you should look to change your coding style, not the language. /Paul
On Sat, Apr 18, 2020 at 4:41 AM Paul Svensson <paul-python@svensson.org> wrote:
On Fri, 17 Apr 2020, Alex Hall wrote:
Perhaps an easier next step would be to get better data about people's opinions with a simple poll? Is there a standard way to vote on things in this but it's easy and may inform what to do next.
For what it's worth, I'm a strong -1 on this whole thing, regardless of syntax. I think passing a lot of same-named parameters is an anti-pattern, that should be discouraged, not made easier. Passing an occasional x=x to so some function no disaster; if it happens often enough to be a problem, IMNSHO, you should look to change your coding style, not the language.
Look at the real-world examples that I posted and tell me what I should be doing differently, if it's an anti-pattern to be discouraged. ChrisA
Chris Angelico writes:
Look at the real-world examples that I posted and tell me what I should be doing differently, if it's an anti-pattern to be discouraged.
IIRC, you didn't post real-world examples relevant to Paul's comment because you posted only the call sites. So, for example, if a function call you posted were refactored to a local function (perhaps using a nonlocal declaration, though cases where that's necessary should be rare), you wouldn't need to pass those arguments at all. Also, see my earlier post for one kind of style change (specifically, taking advantage of comprehensions rather than writing a marshalling function) I've made that means recently I do a lot less same naming. That one's explicitly related to the dict subthread, but it's also true that in my code marshalling functions used to be a substantial source of same name game (though I typically called them with positional parameters by copying the prototype a la Rick Teachey, and so avoided the kind of code you posted). Granted, it's not so easy to avoid that with third party functions, but I personally have rarely had that kind of issue with library functions, whether from the stdlib or from PyPI. Steve
On Sat, Apr 18, 2020 at 11:19 PM Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Chris Angelico writes:
Look at the real-world examples that I posted and tell me what I should be doing differently, if it's an anti-pattern to be discouraged.
IIRC, you didn't post real-world examples relevant to Paul's comment because you posted only the call sites.
So, for example, if a function call you posted were refactored to a local function (perhaps using a nonlocal declaration, though cases where that's necessary should be rare), you wouldn't need to pass those arguments at all.
Except that render_template isn't one of my own functions. It's part of the templating engine (this is a Flask web app). There is no refactoring to do - that IS the correct way to call it. The only way to avoid that would be to do something silly like **locals() and we already know from the f-string discussion that that's a bad idea. ChrisA
Chris Angelico writes:
Except that render_template isn't one of my own functions. It's part of the templating engine (this is a Flask web app). There is no refactoring to do - that IS the correct way to call it.
So NOW you tell me! ;-) Just kidding: of course I recognized that as a library call. Of course cases where one calls a library function with a plethora of keyword arguments and large fraction of them are going to take same name variables as keyword arguments are going to exist. The question is are they frequent enough to justify an IMO ugly syntax change. There are a lot of other factors involved, such as why there are so many randomly-assorted variables that are nevertheless related by this function call, whether this particular call is repeated with the same objects (so that it might be worth wrapping it in a marshalling function with appropriate defaults), if it would be conceptually useful to have a library manager object that collects all of the assorted variables, and so on. I'm arguing that multiple changes in Python, and additional experience for me, mean that I run into same name arguments much less frequently nowadays than I once did, frameworks like Flask notwithstanding. There are several reasons for that, including: 1. Variable naming that refers to the role in the application, rather than the role in the function called. 2. Constructs like comprehensions and zip() that make writing helper functions that call for same name arguments less attractive. 3. Using local rather than global functions in helper roles, eliminating same name arguments in favor of nonlocal access. None of those is a panacea; 2 & 3 are completely inapplicable to the render_template example, and 1 is a matter of style that I think is becoming widespread in the Python community (at least in the code I have to read frequently :-) but is certainly your choice, and any programmer's choice, not mine. And it will vary with the particular library: if you have to call constructors for a bunch of library facilities and do nothing with them but pass them to library functions, the argument names are the obvious variable names. I recognize that. But all of them together, including some not mentioned here, have the effect that this issue is much less salient for me than it used to be, so I'm a lot less sympathetic to this syntactic sugar than I would have been five years ago. I don't know how common this experience is, but I think it's reasonable to present it, and to suggest that it may be quite common. Steve
On Sun, Apr 19, 2020 at 3:48 PM Stephen J. Turnbull <turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
I'm arguing that multiple changes in Python, and additional experience for me, mean that I run into same name arguments much less frequently nowadays than I once did, frameworks like Flask notwithstanding. There are several reasons for that, including:
1. Variable naming that refers to the role in the application, rather than the role in the function called. 2. Constructs like comprehensions and zip() that make writing helper functions that call for same name arguments less attractive. 3. Using local rather than global functions in helper roles, eliminating same name arguments in favor of nonlocal access.
None of those is a panacea; 2 & 3 are completely inapplicable to the render_template example, and 1 is a matter of style that I think is becoming widespread in the Python community (at least in the code I have to read frequently :-) but is certainly your choice, and any programmer's choice, not mine.
In the case of render_template, I effectively define both ends of it, since the template is under my control. So the variable names refer to both the role in the calling function AND the role in the template, and will very frequently correspond. So all three of these concerns are inapplicable here. ChrisA
On 4/19/20 1:48 AM, Stephen J. Turnbull wrote:
Chris Angelico writes:
Except that render_template isn't one of my own functions. It's part of the templating engine (this is a Flask web app). There is no refactoring to do - that IS the correct way to call it.
So NOW you tell me! ;-) Just kidding: of course I recognized that as a library call. Of course cases where one calls a library function with a plethora of keyword arguments and large fraction of them are going to take same name variables as keyword arguments are going to exist. The question is are they frequent enough to justify an IMO ugly syntax change. There are a lot of other factors involved, such as why there are so many randomly-assorted variables that are nevertheless related by this function call, whether this particular call is repeated with the same objects (so that it might be worth wrapping it in a marshalling function with appropriate defaults), if it would be conceptually useful to have a library manager object that collects all of the assorted variables, and so on.
One thing that came to mind as I think about this proposal that may be something to think about. One of the key motivations of this proposal is to make nicer a call with a lot of key word arguments where the local variable name happens (intentionally) to match the keyword parameter name. IF we do something to make this type of call nicer, then there is now an incentive to make your local variables that are going to be passed into functions match the name of the keyword parameter. This incentive might push us from using what be a more descriptive name, which describes what we are doing in our particular case, to match the more generic name from the library. There is also the issue that if we are building a function that might be used with another function, we will have an incentive to name our keyword parameters that there is a reasonable chance would also be passed to that other function with the same keyword name, even if that might not be the most descriptive name. This will cause a slight reduction in readability of code, as cases of foo = phoo, start to stand out and get changed to just foo (by renaming the variable phoo). It is a natural outcome of you get what you make special cases for. -- Richard Damon
On 4/19/2020 7:23 AM, Richard Damon wrote:
Chris Angelico writes:
Except that render_template isn't one of my own functions. It's part of the templating engine (this is a Flask web app). There is no refactoring to do - that IS the correct way to call it.
So NOW you tell me! ;-) Just kidding: of course I recognized that as a library call. Of course cases where one calls a library function with a plethora of keyword arguments and large fraction of them are going to take same name variables as keyword arguments are going to exist. The question is are they frequent enough to justify an IMO ugly syntax change. There are a lot of other factors involved, such as why there are so many randomly-assorted variables that are nevertheless related by this function call, whether this particular call is repeated with the same objects (so that it might be worth wrapping it in a marshalling function with appropriate defaults), if it would be conceptually useful to have a library manager object that collects all of the assorted variables, and so on. One thing that came to mind as I think about this proposal that may be something to think about. One of the key motivations of this proposal is to make nicer a call with a lot of key word arguments where the local variable name happens (intentionally) to match the keyword parameter name. IF we do something to make this type of call nicer, then there is now an incentive to make your local variables that are going to be
On 4/19/20 1:48 AM, Stephen J. Turnbull wrote: passed into functions match the name of the keyword parameter. This incentive might push us from using what be a more descriptive name, which describes what we are doing in our particular case, to match the more generic name from the library.
There is also the issue that if we are building a function that might be used with another function, we will have an incentive to name our keyword parameters that there is a reasonable chance would also be passed to that other function with the same keyword name, even if that might not be the most descriptive name.
This will cause a slight reduction in readability of code, as cases of foo = phoo, start to stand out and get changed to just foo (by renaming the variable phoo). It is a natural outcome of you get what you make special cases for.
I agree: this is part of why I consider this whole proposal to be an anti-pattern. I'd expect a PEP to mention the above issues. Thanks for highlighting them. Eric
On Sun, Apr 19, 2020 at 09:28:28AM -0400, Eric V. Smith wrote: [...]
This will cause a slight reduction in readability of code, as cases of foo = phoo, start to stand out and get changed to just foo (by renaming the variable phoo). It is a natural outcome of you get what you make special cases for.
I agree: this is part of why I consider this whole proposal to be an anti-pattern. I'd expect a PEP to mention the above issues. Thanks for highlighting them.
If it is an anti-pattern for one method to duplicate the parameter names of another method, is it a pattern to intentionally differentiate the parameter names by using synonyms? I regularly -- not frequently, but often enough that it becomes a pain point -- have a situation where I have one or more public functions which call other functions with the same parameters. Here's a real signature from one of my functions where this occurs. def inspect(obj=_SENTINEL, pattern=None, *, dunder=True, meta=False, private=True, ignorecase=False, invert=False) It ends up calling a ton of methods that accept some or all of the same parameter names, using the `dunder=dunder` idiom. Today I learned this is an antipattern. For the record, earlier in this thread I considered making this same argument against this proposal because it would encourage people to rename their parameters to match those of other functions, but I deleted it. At least I hope I deleted it, because seeing it actually written down shows me that it's a really weak argument. Consistency of parameter names is more often a good thing than a bad thing. I don't know if that counts as a point in favour of this proposal, but I'm pretty sure it shouldn't count as a point against it. -- Steven
On Sun, Apr 19, 2020, 7:24 AM Richard Damon <Richard@damon-family.org> wrote:
One of the key motivations of this proposal is to make nicer a call with a lot of key word arguments where the local variable name happens (intentionally) to match the keyword parameter name.
I think Stephen said that this "same name across scopes" is an anti-pattern. I mostly disagree, though obviously agree that it varies between different code. He mentioned something like: process_account(name=name, email=email) And that would be better with email_john and email_jane as names in outer scope. I've worked with large code bases where the equivalent complex object is used in many places. Not necessarily as a continuous life of an actual object, but sometimes sent as JSON, other times read from database, other times calculated dynamically, etc. For example, maybe there is a 'person_record' that has as attributes/keys/whatever name and email. But in this existing code, a different name is used through a call chain and in different corners of the code. E.g. we read json_record, which calls something naming it dynamic_record, which eventually arrives at a function that saves db_record. But these are all actually the same object layout or even the same object. When it comes time to refactor—now we need to rename 'telephone' as 'home_phone' and add 'cell_phone'—finding all the locations to change is a PITA. If only we could grep 'person_record' globally, it would have been easy.
On 4/19/20 11:28 AM, David Mertz wrote:
On Sun, Apr 19, 2020, 7:24 AM Richard Damon <Richard@damon-family.org <mailto:Richard@damon-family.org>> wrote:
One of the key motivations of this proposal is to make nicer a call with a lot of key word arguments where the local variable name happens (intentionally) to match the keyword parameter name.
I think Stephen said that this "same name across scopes" is an anti-pattern. I mostly disagree, though obviously agree that it varies between different code. He mentioned something like:
process_account(name=name, email=email)
And that would be better with email_john and email_jane as names in outer scope.
I've worked with large code bases where the equivalent complex object is used in many places. Not necessarily as a continuous life of an actual object, but sometimes sent as JSON, other times read from database, other times calculated dynamically, etc.
For example, maybe there is a 'person_record' that has as attributes/keys/whatever name and email. But in this existing code, a different name is used through a call chain and in different corners of the code. E.g. we read json_record, which calls something naming it dynamic_record, which eventually arrives at a function that saves db_record. But these are all actually the same object layout or even the same object.
When it comes time to refactor—now we need to rename 'telephone' as 'home_phone' and add 'cell_phone'—finding all the locations to change is a PITA. If only we could grep 'person_record' globally, it would have been easy.
As with most anti-patterns, there tend to be cases where they actually help (which is why some think of them as a useful pattern), but then it get applied beyond the cases where it is actually useful, and then becomes the anti-pattern. My personal thought on the case you are presenting, is that I would be tempted to grep for telephone, to change that because the change likely affects not just this given structure, but may also impact related structures which also assumed a single phone number. -- Richard Damon
On Sun, Apr 19, 2020 at 11:54 AM Richard Damon <Richard@damon-family.org> wrote:
When it comes time to refactor—now we need to rename 'telephone' as 'home_phone' and add 'cell_phone'—finding all the locations to change is a PITA. If only we could grep 'person_record' globally, it would have been easy.
My personal thought on the case you are presenting, is that I would be tempted to grep for telephone, to change that because the change likely affects not just this given structure, but may also impact related structures which also assumed a single phone number.
Sure. But what I gave was a simple case. There are all kinds of complications like this person_record being passed around from call to call without actually accessing `.telephone` in a particular scope. Or doing something dynamic with the attributes/keys that won't show up in a `grep telephone`. Or the string telephone occurring lots of times in unrelated structures/objects. Or lots of other cases where lots of name changes makes refactoring more difficult. I mean, I've DONE it, and I'm sure you have as well. Clearly refactoring isn't impossible with different names across scopes... and this goal is one of several in picking names, not the single predominant one.
-- Richard Damon _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PFSCBT... Code of Conduct: http://python.org/psf/codeofconduct/
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 4/19/20 12:04 PM, David Mertz wrote:
Sure. But what I gave was a simple case. There are all kinds of complications like this person_record being passed around from call to call without actually accessing `.telephone` in a particular scope. Or doing something dynamic with the attributes/keys that won't show up in a `grep telephone`. Or the string telephone occurring lots of times in unrelated structures/objects. Or lots of other cases where lots of name changes makes refactoring more difficult. I mean, I've DONE it, and I'm sure you have as well. Clearly refactoring isn't impossible with different names across scopes... and this goal is one of several in picking names, not the single predominant one.
The fact that the name of the variable changes shouldn't affect the refactoring. If the scope doesn't reference the telephone attribute, then it shouldn't be affected by the refactoring. Yes, the dynamic cases says we need to look for the string telephone as well as the direct reference of the attribute telephone. The fact that we get telephone showing up in unrelated structures is likely a plus, as if we find that it isn't good enough to store just a single phone number in this sort of record, we likely should be thinking about how we did it elsewhere, especially the way it was described as the data flowing through various forms and not just a single structure. In some ways this show the danger of coercing the programmer to reuse names for unrelated things, it encourages bad names. The mere fact that the record had a single field called 'telephone', as even decades ago many people had more than 1 phone number, they at least had a home_phone and and work_phone (and later a cell_phone). -- Richard Damon
On Sun, Apr 19, 2020 at 8:29 AM David Mertz <mertz@gnosis.cx> wrote:
On Sun, Apr 19, 2020, 7:24 AM Richard Damon <Richard@damon-family.org> wrote:
One of the key motivations of this proposal is to make nicer a call with a lot of key word arguments where the local variable name happens (intentionally) to match the keyword parameter name.
I think Stephen said that this "same name across scopes" is an anti-pattern. I mostly disagree, though obviously agree that it varies between different code.
In some sense, I think the "anti-pattern" here is not so much in the naming of variables, arguments, and parameters, but in ending up on the wring side of the "data" vs. "code" divide. e.g.: "data" might be a dict of key"value pairs, and "code" might be an object with attributes (or a bunch of keyword parameters in a function call) In Python, this is a very blurry line, as you can easily do things like use **kwargs (oassing data into code) and accessing __dict__ (getting data from code), not to mention fancier meta-programming techniques. But if you find yourself passing a lot of same-names variables around, maybe you should jsut be passing around a dict? -CHB I've worked with large code bases where the equivalent complex object is
used in many places. Not necessarily as a continuous life of an actual object, but sometimes sent as JSON, other times read from database, other times calculated dynamically, etc.
The JSON - related example is a good one -- JSON maps well to "data" in Python, dicts and lists of numbers and strings. If you find yourself converting a bunch of variable names to/from JSON, you probably should be simply using a dict, and passing that around anyway. and, of course,with **kwargs, you CAN make the transition from a dict to an object and back quite easily. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Apr 20, 2020, at 11:01, Christopher Barker <pythonchb@gmail.com> wrote:
The JSON - related example is a good one -- JSON maps well to "data" in Python, dicts and lists of numbers and strings. If you find yourself converting a bunch of variable names to/from JSON, you probably should be simply using a dict, and passing that around anyway.
A lot of JSON is designed to be consumed by JavaScript, where there is no real line (there is no dict type; objects have both dict-style and dot-style access). So in practice, a lot of JSON maps well to data, a lot maps well to objects, and some is mushily designed and doesn’t quite fit either way, because in JS they all look the same anyway. The example code for an API often shows you doing `result.movies[0].title.en`, because in JS you can. And in other languages, sometimes it is worth writing (or auto-generating) the code for Movie, etc. classes and serializing them to/from JSON so you can do the same. This is really the same point as “sometimes ORMs are useful”, which I don’t think is that controversial. But, maybe even more importantly: even if you _do_ decide it makes more sense to stick to data for this API, you have the parallel `{'country': country, 'year': year}` issue, which is just as repetitive and verbose. The `{::country, ::year}` syntax obviously solves that dict key issue just as easily as it does for keywords. But most of the other variant proposals solve it at least indirectly via dict constructor calls—`dict(**, country, year)`, `dict(country=, year=)`, `dict(**{country, year})`, which isn’t quite as beautiful, but is still better than repeating yourself if the list of members or query conditions gets long.
On Mon, Apr 20, 2020 at 12:17 PM Andrew Barnert <abarnert@yahoo.com> wrote:
On Apr 20, 2020, at 11:01, Christopher Barker <pythonchb@gmail.com> wrote:
The JSON - related example is a good one -- JSON maps well to "data" in Python, dicts and lists of numbers and strings. If you find yourself converting a bunch of variable names to/from JSON, you probably should be simply using a dict, and passing that around anyway.
A lot of JSON is designed to be consumed by JavaScript, where there is no real line (there is no dict type; objects have both dict-style and dot-style access). So in practice, a lot of JSON maps well to data, a lot maps well to objects, and some is mushily designed and doesn’t quite fit either way, because in JS they all look the same anyway.
Well, sure. Though JSON itself is declarative data. In Python, you need to decide how you want to work with it, either as an object with attributes or a dict. But if you are getting it from JSON, it's a dict to begin with. So you can keep it as a dict, or populate an object with it. B ut populating that object can be automated: an_instance = MyObject(**the_dict_from_JSON) And then: do_something_with(an_instance.name) It's not always that simple, but you sure shouldn't have to do anything like: name = the_dict_from_JSON['name'] ... Myobject(name=name) Which is what I'm getting at: why are all those ever variables in a current namespace? it may be worth rethinking. -CHB
But, maybe even more importantly: even if you _do_ decide it makes more sense to stick to data for this API, you have the parallel `{'country': country, 'year': year}` issue, which is just as repetitive and verbose.
only if you have those as local variables -- why are they ? I'm not saying it never comes up in well designed code -- it sure does, but if there's a LOT of that, then maybe some refactoring is in order. -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Apr 20, 2020, at 13:42, Christopher Barker <pythonchb@gmail.com> wrote:
On Mon, Apr 20, 2020 at 12:17 PM Andrew Barnert <abarnert@yahoo.com> wrote:
A lot of JSON is designed to be consumed by JavaScript, where there is no real line (there is no dict type; objects have both dict-style and dot-style access). So in practice, a lot of JSON maps well to data, a lot maps well to objects, and some is mushily designed and doesn’t quite fit either way, because in JS they all look the same anyway.
Well, sure. Though JSON itself is declarative data.
Sure, it’s a declarative format, it’s just that often it’s intended to be understood as representing an object graph.
In Python, you need to decide how you want to work with it, either as an object with attributes or a dict. But if you are getting it from JSON, it's a dict to begin with. So you can keep it as a dict, or populate an object with it. B ut populating that object can be automated:
an_instance = MyObject(**the_dict_from_JSON)
But unless your object graph is flat, this doesn’t work. A MyObject doesn’t just have strings and numbers, it also has a list of MySubObjects; if you just ** the JSON dict, you get subobjs=[{… d1 … }, { … d2 … }], when what you actually wanted was subobjs=[MySubObject(**d) for d in …]. It’s not like it’s _hard_ to write code to serialize and deserialize object graphs as JSON (although it’s hard enough that people keep proposing a __json__ method to go one way and then realizing they don’t have a proposal to go the other way…), but it’s not as trivial as just ** the dict into keywords.
But, maybe even more importantly: even if you _do_ decide it makes more sense to stick to data for this API, you have the parallel `{'country': country, 'year': year}` issue, which is just as repetitive and verbose.
only if you have those as local variables -- why are they ?
Apologies for my overly-fragmentary toy example. Let’s say you have a function that makes an API request to some video site to get the local-language names of all movies of the user-chosen genre in the current year. If you’ve built an object model, it’ll look something like this: query = api.Query(genre=genre, year=datetime.date.today().year) response = api.query_movies(query) result = [movie.name[language] for movie in response.movies] If you’re treating the JSON as data instead, it’ll look something like this: query = {'query': {'genre': genre, 'year': datetime.date.today().year}} response = requests.post(api.query_movies_url, json=query).json result = [movie['name'][language] for movie in response.movies] Either way, the problem is in that first line, and it’s the same problem. (And the existence of ** unpacking and the dict() constructor from keywords means that solving either one very likely solves the other nearly for free.) Here I’ve got one local, `genre`. (I also included one global representing a global setting, just to show that they _can_ be reasonable as well, although I think a lot less often than locals, so ignore that.) I think it’s pretty reasonable that the local variable has the same name as the selector key/keyword. If I ask “why do I have to repeat myself with genre=genre or 'genre': genre”, what’s the answer? If I have 38 locals for all 38 selectors in the API—or, worse, a dynamically-chosen subset of them—then “get rid of those locals” is almost surely the answer, but with just 1? Probably not. And maybe 3 or 4 is reasonable too—a function that select by genre, subgenre, and mood seems like a reasonable thing. (If it isn’t… well, then I was stupid to pick an application area I haven’t done much work in… but you definitely don’t want to just select subgenre without genre in many music APIs, because your user rarely wants to hear both hardcore punk and hardcore techno.) And it’s clearly not an accident that the local and the selector have the same name. So, I think that case is real, and not dismissible.
I'm not saying it never comes up in well designed code -- it sure does, but if there's a LOT of that, then maybe some refactoring is in order.
Yes. And now that you point that out, thinking of how many people go to StackOverflow and python-list and so on looking for help with exactly that antipattern when they shouldn’t be doing it in the first place, there is definitely a risk that making this syntax easier could be an antipattern magnet. So, it’s not just whether the cases with 4 locals are important enough to overcome the cost of making Python syntax more complicated; the benefit has to _also_ overcome the cost of being a potential antipattern magnet. For me, this proposal is right on the border of being worth it (and I’m not sure which side it falls on), so that could be enough to change the answer, so… good thing you brought it up. But I don’t think it eliminates the rationale for the proposal, or even the rationale for using it with JSON-related stuff in particular.
On Mon, Apr 20, 2020 at 3:13 PM Andrew Barnert <abarnert@yahoo.com> wrote:
Sure, it’s a declarative format, it’s just that often it’s intended to be understood as representing an object graph.
I"m not sure the point here -- I was not getting onto detail nor expalingnoi myself well, but I think there are (kind of) three ways to "name" just one piece of data that came from a bunch of JSON: - a key, as in a dict `data['this']` - an attribute of an object: `data.this` - a local variable: `this` what I was getting at is that there may be a fine line between the dsta version and the object version, but that can go between those easily without typing all the names. It's only when you have it in a local variable that this whole idea starts to matter.
In Python, you need to decide how you want to work with it, either as an object with attributes or a dict. But if you are getting it from JSON, it's a dict to begin with. So you can keep it as a dict, or populate an object with it. B ut populating that object can be automated:
an_instance = MyObject(**the_dict_from_JSON)
But unless your object graph is flat, this doesn’t work.
It’s not like it’s _hard_ to write code to serialize and deserialize object graphs as JSON (although it’s hard enough that people keep proposing a __json__ method to go one way and then realizing they don’t have a
Of course not -- that was just the most trivial example. but it's also trivial to unpack a whole graph, as long as you only want the standard types that JSON can store. Even if you want custom types, you can still build those from a dict of teh standrad tyupes. And I've written (as have many others, I"m sure) code that does just that. proposal to go the other way…), but it’s not as trivial as just ** the dict into keywords. sure. But what I'm suggesting is that even if you have a complex object graph, you should write (or use a library) code to do that unpacking for you rather than, say: blob = json.load(...) people = [] for person in blob['people'] name = person['name'] phone_number = person['phone_number'] # a bunch more people.append(Person(name=name, phone_number=phone_number, ... )) When you could write: blob = json.load(...) people = [Person(**person) for person in blob['people']] Granted, I hope no one would write it quite that badly, but the point is that the proposal in hand is only helpful if the information is in local variables, and this kind of use case, that's probably not a good way to structure the code.
Apologies for my overly-fragmentary toy example.
well, better than my no example :-) -- but I wasn't referring to any particular example, I meant it generally -- again, this proposal is about using local variables to fill in function calls (or dict construction) with the same names, and I'm suggesting that if that is "data", then it probably shouldn't be in local variables anyway. And if it it came from JSON (or some such), then it took effort to put it in local variables ...
Let’s say you have a function that makes an API request to some video site to get the local-language names of all movies of the user-chosen genre in the current year.
If you’ve built an object model, it’ll look something like this:
If you’re treating the JSON as data instead, it’ll look something like
query = api.Query(genre=genre, year=datetime.date.today().year) response = api.query_movies(query) result = [movie.name[language] for movie in response.movies] this: query = {'query': {'genre': genre, 'year': datetime.date.today().year}} response = requests.post(api.query_movies_url, json=query).json result = [movie['name'][language] for movie in response.movies] well, the query params and the result are not really the same, on the data vs code continuum. But...
Either way, the problem is in that first line, and it’s the same problem. (And the existence of ** unpacking and the dict() constructor from keywords means that solving either one very likely solves the other nearly for free.)
Here I’ve got one local, `genre`. (I also included one global representing a global setting, just to show that they _can_ be reasonable as well, although I think a lot less often than locals, so ignore that.) I
Agreed. I do think that if this is going to happen at all, it should be a dict display feature (which would help with both "data" and "code"), not a function calling feature. think it’s pretty reasonable that the local variable has the same name as the selector key/keyword. If I ask “why do I have to repeat myself with genre=genre or 'genre': genre”, what’s the answer? Sure, but treating the result as data does not mean you have to treat the query as data as well, and you may have a query_params class that holds all that anyway :-) -- and even if they are locals -- where did they come from? (read from a config file? gotten from user input?) if you were doing a full on JSON API, they may not have ever had to be in variables in the first place.
If I have 38 locals for all 38 selectors in the API—or, worse, a dynamically-chosen subset of them—then “get rid of those locals” is almost surely the answer, but with just 1? Probably not. And maybe 3 or 4 is reasonable too—
right. but I don't think anyone is suggesting a language change for 1, or even 3-4 names (maybe 4...)
And it’s clearly not an accident that the local and the selector have the same name. So, I think that case is real, and not dismissible.
I wasn't trying to dismiss it. I'm not saying it never comes up in well designed code -- it sure does, but if there's a LOT of that, then maybe some refactoring is in order.
Yes. And now that you point that out, thinking of how many people go to StackOverflow and python-list and so on looking for help with exactly that anti-pattern when they shouldn’t be doing it in the first place ...
And if we go back a couple messages in this thread, I was suggesting that
there is definitely a risk that making this syntax easier could be an antipattern magnet. So, it’s not just whether the cases with 4 locals are important enough to overcome the cost of making Python syntax more complicated; the benefit has to _also_ overcome the cost of being a
But I don’t think it eliminates the rationale for the proposal, or even
Right. I think the data vs code distinction is a tough one in Python (maybe a tiny bit less than JS?) In more static languages, you can kind of decide based on what a pain it is to write the code -- if it's a serious pain, it's probably data :-) the anti-pattern was not using the same names in calling and function scope, but rather, using local names, when it really should be data: in a dict, or even a special object. potential antipattern magnet. For me, this proposal is right on the border of being worth it (and I’m not sure which side it falls on), so that could be enough to change the answer, Good point. the rationale for using it with JSON-related stuff in particular. Nor do I. However, as I think about it, where this may make the most sense might be for cases when you're making a complex call that has SOME arguments in the local namespace, with the some with other names, and some specified as literals. That's pretty common pattern in the call to setup() in setup.py files, for example. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Apr 20, 2020, at 16:46, Christopher Barker <pythonchb@gmail.com> wrote:
On Mon, Apr 20, 2020 at 3:13 PM Andrew Barnert <abarnert@yahoo.com> wrote:
Sure, it’s a declarative format, it’s just that often it’s intended to be understood as representing an object graph.
I"m not sure the point here -- I was not getting onto detail nor expalingnoi myself well, but I think there are (kind of) three ways to "name" just one piece of data that came from a bunch of JSON:
- a key, as in a dict `data['this']` - an attribute of an object: `data.this` - a local variable: `this`
what I was getting at is that there may be a fine line between the dsta version and the object version, but that can go between those easily without typing all the names.
OK, I thought you were saying that line is a serious problem for this proposal, so I was arguing that the same problems actually arise either way, and the same proposal helps both. Since you weren’t saying that and I misinterpreted you, that whole part of the message is irrelevant. So I’ll strip all the irrelevant bits down to this quote from you that I agree with 100%:
It's only when you have it in a local variable that this whole idea starts to matter.
And I think we also agree that it would be better to make this a dict display feature, and a bunch of other bits. But here’s the big issue:
If I have 38 locals for all 38 selectors in the API—or, worse, a dynamically-chosen subset of them—then “get rid of those locals” is almost surely the answer, but with just 1? Probably not. And maybe 3 or 4 is reasonable too—
right. but I don't think anyone is suggesting a language change for 1, or even 3-4 names (maybe 4...)
The original post only had 2 arguments. Other people came up with examples like the popen one, which has something insane like 19 arguments, but most of them were either constants or computed values not worth storing; only 4 of them near the end we’re copied from locals. Steven’s example had 5. The “but JavaScript lets me do it” post has 3. I think someone suggested the same setup.py example you came up with later in this same example, and it had 3 or 4. So I think people really are suggesting this for around 4 names. And I agree that’s kind of a marginal benefit. That’s why I think the whole proposal is marginal. It’s almost never going to be a huge win—but it may be a small win in so many places that it adds up to being worth it.
On 2020-04-19 07:23, Richard Damon wrote:
There is also the issue that if we are building a function that might be used with another function, we will have an incentive to name our keyword parameters that there is a reasonable chance would also be passed to that other function with the same keyword name, even if that might not be the most descriptive name.
Many of the function keyword parameters I deal with are data property names; so it makes sense that the data has the same name throughout the codebase. The incentive to align our variable names would be a good thing. Consider pymysql, and the connect parameters
connect( host=host, port=port, user=username, passwd=password )
With the proposal, right, or wrong, there would be an incentive for me to write the caller to use pymysql property names, and the callers of that caller to also use the same property names. This will spread until the application has a standard name for username and password: There is less guessing about the property names. I have done this in ES6 code, and it looks nice. Maybe aligning variable names with function keyword parameteres is an anti-pattern, but I have not seen it. I reviewed my code: of 20,360 keyword arguments, 804 (4%) are have the x=x format. I do not know if this is enough to justify such a proposal, but I would suggest that is a minimum: Currently there is no incentive to have identical names for identical things through a call chain; an incentive will only increase the use of this pattern.
On Thu, 23 Apr 2020 14:47:48 -0400 Kyle Lahnakoski <klahnakoski@mozilla.com> wrote:
On 2020-04-19 07:23, Richard Damon wrote:
There is also the issue that if we are building a function that might be used with another function, we will have an incentive to name our keyword parameters that there is a reasonable chance would also be passed to that other function with the same keyword name, even if that might not be the most descriptive name.
Many of the function keyword parameters I deal with are data property names; so it makes sense that the data has the same name throughout the codebase. The incentive to align our variable names would be a good thing. Consider pymysql, and the connect parameters
connect( host=host, port=port, user=username, passwd=password )
With the proposal, right, or wrong, there would be an incentive for me to write the caller to use pymysql property names, and the callers of that caller to also use the same property names. This will spread until the application has a standard name for username and password: There is less guessing about the property names. I have done this in ES6 code, and it looks nice.
(We'll assume, for the sake of discussion, that you meant either (a) to use the same name for user and username and for passwd and password, or (2) to use host and port in your explanatory paragraph. IMO, either way, you're disproving your own point.)
Maybe aligning variable names with function keyword parameteres is an anti-pattern, but I have not seen it.
Sure, that works great. Until you decide to change from pymysql to pywhizbangdb, and its connect function looks like this: connect(uri, user_id, password) Yes, there will be "layers," or "functional blocks," or whatever your architectural units might be called this week, inside of which it may make sense to have a unified name for certain properties. But the whole application? That sounds like a recipe for future inflexibility. -- “Atoms are not things.” – Werner Heisenberg Dan Sommers, http://www.tombstonezero.net/dan
On Thu, Apr 23, 2020 at 04:24:18PM -0400, Dan Sommers wrote:
On Thu, 23 Apr 2020 14:47:48 -0400 Kyle Lahnakoski <klahnakoski@mozilla.com> wrote:
Many of the function keyword parameters I deal with are data property names; so it makes sense that the data has the same name throughout the codebase. The incentive to align our variable names would be a good thing. Consider pymysql, and the connect parameters
connect( host=host, port=port, user=username, passwd=password )
With the proposal, right, or wrong, there would be an incentive for me to write the caller to use pymysql property names, and the callers of that caller to also use the same property names. This will spread until the application has a standard name for username and password: There is less guessing about the property names. I have done this in ES6 code, and it looks nice.
(We'll assume, for the sake of discussion, that you meant either (a) to use the same name for user and username and for passwd and password, or (2) to use host and port in your explanatory paragraph. IMO, either way, you're disproving your own point.)
Kyle is explicitly discussing how this proposal will encourage names to be aligned in the future, where today they are pointlessly using synonyms. The point is that user/username and passwd/password are not currently aligned, but this proposal will encourage them to become so. So, no, you should not be assuming that Kyle made a mistake with those two pairs of names, as that would defeat the purpose of his comment. As for your second point, host and port are already aligned. How does the existance of aligned names today disprove the point that unaligned names will, in the future, become aligned? Kyle: "Yesterday I ate lunch. Tomorrow I intend to eat lunch." You: "You ate lunch yesterday? That disproves your claim that you will eat lunch tomorrow."
Maybe aligning variable names with function keyword parameteres is an anti-pattern, but I have not seen it.
Sure, that works great. Until you decide to change from pymysql to pywhizbangdb, and its connect function looks like this:
This counter-point would be more credible if pywhizbangdb actually existed, but okay, let's assume it exists.
connect(uri, user_id, password)
Sounds to me that this is actually a point in favour of aligning variables. Then changing to pywhizbangdb would be a relatively simple "change variable name" refactoring for "username" to "user_id". (There are refactoring tools in IDEs such as PyCharm that will do this for you, rather than needing to do a search and replace yourself. But I haven't used them and I cannot tell you how good they are.) Whereas the change from host+port to uri is a more difficult (in the relative sense, not in any absolute sense) two line change: * use host and port to generate a new variable `uri` * change the call to connect to use `uri` instead of host + port. Either way, this doesn't seem like a particularly onerous migration to me. If only all migrations of the backend were that simple!
Yes, there will be "layers," or "functional blocks," or whatever your architectural units might be called this week, inside of which it may make sense to have a unified name for certain properties. But the whole application? That sounds like a recipe for future inflexibility.
Fortunately, this proposal doesn't make it mandatory for people to use the same variable name throughout their entire application, and the function call syntax `func(somename=anothername)` will remain valid. -- Steven
On Fri, 24 Apr 2020 07:46:43 +1000 Steven D'Aprano <steve@pearwood.info> wrote:
On Thu, Apr 23, 2020 at 04:24:18PM -0400, Dan Sommers wrote:
On Thu, 23 Apr 2020 14:47:48 -0400 Kyle Lahnakoski <klahnakoski@mozilla.com> wrote:
Many of the function keyword parameters I deal with are data property names; so it makes sense that the data has the same name throughout the codebase. The incentive to align our variable names would be a good thing. Consider pymysql, and the connect parameters
connect( host=host, port=port, user=username, passwd=password )
With the proposal, right, or wrong, there would be an incentive for me to write the caller to use pymysql property names, and the callers of that caller to also use the same property names. This will spread until the application has a standard name for username and password: There is less guessing about the property names. I have done this in ES6 code, and it looks nice.
(We'll assume, for the sake of discussion, that you meant either (a) to use the same name for user and username and for passwd and password, or (2) to use host and port in your explanatory paragraph. IMO, either way, you're disproving your own point.)
Kyle is explicitly discussing how this proposal will encourage names to be aligned in the future, where today they are pointlessly using synonyms. The point is that user/username and passwd/password are not currently aligned, but this proposal will encourage them to become so.
Hold that thought. My point is that aligning them for the sake of aligning them leads to more work in the future, not less.
So, no, you should not be assuming that Kyle made a mistake with those two pairs of names, as that would defeat the purpose of his comment.
Fair enough. I'm glad I said something, because Kyle's point was apparently not as clear to me as it was to you.
Sure, that works great. Until you decide to change from pymysql to pywhizbangdb, and its connect function looks like this:
This counter-point would be more credible if pywhizbangdb actually existed, but okay, let's assume it exists.
connect(uri, user_id, password)
Sounds to me that this is actually a point in favour of aligning variables. Then changing to pywhizbangdb would be a relatively simple "change variable name" refactoring for "username" to "user_id".
Given the definition of pymysql.connect and pywhizbangdb.connect, what would I call the name of the user? user_id or user? When I switch back ends, why should I refactor/rename anything? And what happens when I have to support both back ends, rather than simply changing from one to the other once for all time? (I probably should have presented this use case first.)
(There are refactoring tools in IDEs such as PyCharm that will do this for you, rather than needing to do a search and replace yourself. But I haven't used them and I cannot tell you how good they are.)
Yeah, they made me use Java at my last paying job, and IMO all such tools do is encourage pointless renaming and huge commits where you can't tell what's been changed vs. what's only been renamed. Sorry, I digress.
Whereas the change from host+port to uri is a more difficult (in the relative sense, not in any absolute sense) two line change:
* use host and port to generate a new variable `uri` * change the call to connect to use `uri` instead of host + port.
Either way, this doesn't seem like a particularly onerous migration to me. If only all migrations of the backend were that simple!
We agree that it's not a particularly onerous migration. My point remains that by adopting a style that propagates any particular back end's naming conventions into the rest of your application, any migration is more work than it has to be.
Yes, there will be "layers," or "functional blocks," or whatever your architectural units might be called this week, inside of which it may make sense to have a unified name for certain properties. But the whole application? That sounds like a recipe for future inflexibility.
Fortunately, this proposal doesn't make it mandatory for people to use the same variable name throughout their entire application, and the function call syntax `func(somename=anothername)` will remain valid.
Absolutely, but it seemed to me that Kyle claimed that using the same name throughout an entire application "looks nice," and I pointed out that it does look nice until it makes what should be a tiny change into a larger one. -- “Atoms are not things.” – Werner Heisenberg Dan Sommers, http://www.tombstonezero.net/dan
Kyle Lahnakoski writes:
Maybe aligning variable names with function keyword parameteres is an anti-pattern, but I have not seen it.
I consider this an anti-pattern for my use case. I review a lot of different students' similar code for different applications. They borrow from each other, cargo cult fashion, which I consider a good thing. However, it's only a good thing to a point, because at some high enough level they're doing different things. I want variable names to reflect those differences, pretty much down to the level of the computational algorithms or network protocols. Despite Steven d'Aprano, I also agree with Dan Sommers:
Sure, that works great. Until you decide to change from pymysql to pywhizbangdb, and its connect function looks like this:
The reason I disagree with Steven is that almost certainly pymysql and pywhizbangdb *already have copied the APIs of the libraries they wrap*. So the abbreviated keyword arguments do not go "all the way down", and *never will*, because the underlying libraries aren't implemented in Python. Nor will the implementers be thinking "we should choose the names our callers are using" -- for one thing, at that point there won't be any callers yet! So the coordination can only go so far. Then consider something like Mailman. In Mailman 2, there is no abstract concept of "user". There are email addresses, there are lists, and there is the relation "subscription" between addresses and lists. Mailman 2 APIs frequently use the term "member" to indicate a subscriber (ie, email address), and this is not ambiguous: member = subscriber. In Mailman 3, this simple structure has become unsupportable. We now have concepts of user, who may have multiple email addresses, and role, where users may fulfil multiple roles (subscriber via one of their addresses, moderator, owner) in a list. For reasons I don't know, in Mailman 3 a member is any user associated with a list, who need not be subscribed. (Nonmembers are email addresses that post to the list but do not have proper users associated with them.) This confused me as a Mailman developer, let alone list owners who never thought about these internals until they upgraded. I don't think either definition ("subscriber" vs. "associated user") is "unnatural", but they are different, this is confusing, and yet given the history I don't see how the term "member" can be avoided. So I see situations (such as the proverbial 3-line wrapper) where coordinating names is natural, obvious, and to be encouraged, others where it's impossible (Python wrappers of external libraries), and still others where thinking about names should be encouraged, and it's an antipattern to encourage coordinating them with function arguments by allowing abbreviated actual arguments.
On Thu, 2020-04-23 at 14:47 -0400, Kyle Lahnakoski wrote: <snip>
I reviewed my code: of 20,360 keyword arguments, 804 (4%) are have the x=x format. I do not know if this is enough to justify such a proposal,
Is that script somewhere? I got a bit curious and wasted some time on making my own script to search projects (link at end), and then applied it on cpython and the scientific python stack. Likely I simply missed an earlier email that already did this, and hopefully it is all correct. For cpython I got: Scanned 985 python files. Total kwargs: 10105 out which identical: 1407 Thus 13.9% are identical. Table of most common identical kwargs: name | value ---------------------|------ file | 43 encoding | 42 context | 36 loop | 33 argname | 31 name | 31 errors | 24 limit | 21 For the scientific python stack (NumPy, SciPy, pandas, astropy, sklearn, matplotlib, skimage), I got: Overall Scanned 1884 python files. Total kwargs: 39381 out which identical: 12229 Thus 31.1% are identical. Table of most common identical kwargs: name | value ---------------------|------ axis | 606 dtype | 471 copy | 296 out | 224 name | 205 mode | 122 fill_value | 115 random_state | 114 sample_weight | 109 verbose | 106 For example including tests, etc. reduces the percentages considerably to 10.5% and 15.1% respectively. These are focusing on the main namespace(s) for the scientific python projects, which exaggerates things by ~7% as well (your numbers will vary depending on which files to scan/analyze). Many of the projects have around 30% in their main namespace, pandas has 40% outside of tests. Since I somewhat liked the arguments for `=arg` compared to `arg=` and I noticed that it is not uncommon to have something like: function(..., dtype=arr.dtype) I included such patterns for fun. The percentages increase to 18.9% or 36.6% (again no tests, etc.) respectively. Not really suggesting it here, just thought it was interesting :). It would be interesting how common the patterns are for dictionary literals if this discussion continues (although I may have missed it, maybe it was already done). Another more trickier but very interesting thing, would be to see how many positional arguments of identical name are passed. If such an argument is not the first or second, it may be cleaner as kwargs, but someone opted to not use them, maybe due to being verbose. Anyway, I am not advocating anything, I was mainly curious. Personally, I am not yet convinced that any of the syntax proposals are nice when considering the additional noise of having more (less-common) syntax. Cheers, Sebastian PS: My hacky script is at: https://gist.github.com/seberg/548a2fa9187739ff33ec406e933fa8a4
but I would suggest that is a minimum: Currently there is no incentive to have identical names for identical things through a call chain; an incentive will only increase the use of this pattern.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/UQGVKM... Code of Conduct: http://python.org/psf/codeofconduct/
Here's a script to find places where this proposal could be useful. Try it on a directory. ```python import ast import linecache import sys from pathlib import Path root = Path(sys.argv[1]) def main(): for path in root.rglob("**/*.py"): source = path.read_text() try: tree = ast.parse(source) except SyntaxError: continue for node in ast.walk(tree): if isinstance(node, ast.Call): def is_same_name(keyword: ast.keyword): return ( isinstance(keyword.value, ast.Name) and keyword.value.id == keyword.arg ) args = node.keywords elif isinstance(node, ast.Dict): def is_same_name(pair): key, value = pair return ( isinstance(value, ast.Name) and isinstance(key, (ast.Constant, ast.Str)) and value.id == key.s ) args = zip(node.keys, node.values) else: continue threshold = 3 if sum(map(is_same_name, args)) < threshold: continue print(f'File "{path}", line {node.lineno}') for lineno in range(node.lineno, node.end_lineno + 1): print(linecache.getline(str(path), lineno), end="") print() main() ``` On Fri, Apr 17, 2020 at 8:40 PM Paul Svensson <paul-python@svensson.org> wrote:
On Fri, 17 Apr 2020, Alex Hall wrote:
Perhaps an easier next step would be to get better data about people's opinions with a simple poll? Is there a standard way to vote on things in this but it's easy and may inform what to do next.
For what it's worth, I'm a strong -1 on this whole thing, regardless of syntax. I think passing a lot of same-named parameters is an anti-pattern, that should be discouraged, not made easier. Passing an occasional x=x to so some function no disaster; if it happens often enough to be a problem, IMNSHO, you should look to change your coding style, not the language.
/Paul _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/B3R63F... Code of Conduct: http://python.org/psf/codeofconduct/
I'm kinda leaning -0.5 even on the form that I think is least bad (the mode switch approach). If typing the same variable from the caller to use in the parameter is really too much repetition, you could maybe just do this:
render_template("index.html", ... username="display_name", ... setups="setups", ... **Q("x y z")) ('index.html',) {'username': 'display_name', 'setups': 'setups', 'x': 1, 'y': 2, 'z': 3}
Perhaps the spelling of `Q` might be something else. But in terms of character count, it's not worse than other proposals. And luckily all you need to do this is get a version of Python later than 1.4 or something like that. :-)
def Q(names): ... import sys ... caller = sys._getframe(1) ... dct = {} ... for name in names.split(): ... dct[name] = eval(name, globals(), caller.f_locals) ... return dct
On Fri, Apr 17, 2020 at 2:41 PM Paul Svensson <paul-python@svensson.org> wrote:
For what it's worth, I'm a strong -1 on this whole thing, regardless of syntax. I think passing a lot of same-named parameters is an anti-pattern, that should be discouraged, not made easier. Passing an occasional x=x to so some function no disaster; if it happens often enough to be a problem, IMNSHO, you should look to change your coding style, not the language.
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 4/17/20 3:04 PM, Andrew Barnert wrote:
I can't think of any either... You know, an ellipsis is about the farthest thing from an *unmarked* elision that you can get [without brackets and reinsertion of a more verbose version of the thing you wanted to elide -ed].
Anyone who has read Celine...knows that ellipses...often are inline operators. Not just for aposiopesis.
Multiple people in this thread have criticized the proposal without any of them stumbling on what { :spam, :eggs } is intended to mean, at least one person noticed that it’s similar to Lisp symbols, one other person proposed the exact same thing, etc., and as far as I know all of those people are native (at least certainly fluent) speakers of a LTR-written language (as were the designers of Lisp, most of the C committee, etc.).
I think I'm the first person to mention Lisp symbols. Maybe not though. I have a very different expectation about what `:spam` would mean based on that analogy than the intended meaning. I could learn the semantics, of course. However, proposals for symbols in Python *do* pop up from time to time, so this would perhaps make such a thing harder if it ever becomes desired (which is unlikely, but possible). My first reading when I see the syntax, however, is something like "that is a set taken from enumerated values" (per the symbol meaning). Being a magic dictionary would be somewhere down the list of the guesses I would make if I had not seen this discussion... of course, if I was suddenly given a copy of Python 5.2, transported from the distant future, I would really just type it in the REPL and probably see a representation that cued me in. On Fri, Apr 17, 2020 at 3:59 PM David Mertz <mertz@gnosis.cx> wrote:
I'm kinda leaning -0.5 even on the form that I think is least bad (the mode switch approach).
If typing the same variable from the caller to use in the parameter is really too much repetition, you could maybe just do this:
render_template("index.html", ... username="display_name", ... setups="setups", ... **Q("x y z")) ('index.html',) {'username': 'display_name', 'setups': 'setups', 'x': 1, 'y': 2, 'z': 3}
Perhaps the spelling of `Q` might be something else. But in terms of character count, it's not worse than other proposals.
And luckily all you need to do this is get a version of Python later than 1.4 or something like that. :-)
def Q(names): ... import sys ... caller = sys._getframe(1) ... dct = {} ... for name in names.split(): ... dct[name] = eval(name, globals(), caller.f_locals) ... return dct
On Fri, Apr 17, 2020 at 2:41 PM Paul Svensson <paul-python@svensson.org> wrote:
For what it's worth, I'm a strong -1 on this whole thing, regardless of syntax. I think passing a lot of same-named parameters is an anti-pattern, that should be discouraged, not made easier. Passing an occasional x=x to so some function no disaster; if it happens often enough to be a problem, IMNSHO, you should look to change your coding style, not the language.
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Apr 17, 2020, at 13:12, David Mertz <mertz@gnosis.cx> wrote:
However, proposals for symbols in Python *do* pop up from time to time, so this would perhaps make such a thing harder if it ever becomes desired (which is unlikely, but possible).
Sure. It would also conflict with Nick Coghlan’s version of the conciser-lambdas idea, where `{ :a, :b }` would mean a set of two nullary lambdas. And the * mode switch would conflict with the proposal to use * as a special positional argument meaning “I’m not passing anything here (just as if I’d left an optional positional off the end, but here I’m doing it in the middle) so use your default value”. And I’m sure there have been other proposals for things :value in a dict display, keyword= in a call, * in a call, etc. could mean that I just don’t happen to remember. But unless one of those other proposals are likely to happen, or should happen, who cares? Assuming one of the proposals in this set of threads has sufficient traction, it would be silly to say “Let’s not do this thing that people want because it would make it harder to do a thing they don’t want”.
At the (online) language summit yesterday, our erstwhile BDFL responded to a suggestion to use '?' in annotations. He said that many ideas arose over time to us the question mark for various things, but he felt none so far were compelling enough to exclude some future better use. I actually find the LOOK of ':var' attractive enough and could easily imagine wanting to use it. E.g. maybe someday we'll get a nice deferred evaluation system; I think that could look very nice as a way to indicate such a deferred term. This purpose of just not typing a name twice in a dict display isn't very important to me. And I just showed a silly hack to do basically the same thing with a function that constructs a dictionary. If we add new syntax, it should do something important rather than just be syntax sugar... Yes, f-strings are arguably a counter argument. On Fri, Apr 17, 2020, 5:59 PM Andrew Barnert <abarnert@yahoo.com> wrote:
On Apr 17, 2020, at 13:12, David Mertz <mertz@gnosis.cx> wrote:
However, proposals for symbols in Python *do* pop up from time to time, so this would perhaps make such a thing harder if it ever becomes desired (which is unlikely, but possible).
Sure. It would also conflict with Nick Coghlan’s version of the conciser-lambdas idea, where `{ :a, :b }` would mean a set of two nullary lambdas.
And the * mode switch would conflict with the proposal to use * as a special positional argument meaning “I’m not passing anything here (just as if I’d left an optional positional off the end, but here I’m doing it in the middle) so use your default value”.
And I’m sure there have been other proposals for things :value in a dict display, keyword= in a call, * in a call, etc. could mean that I just don’t happen to remember.
But unless one of those other proposals are likely to happen, or should happen, who cares? Assuming one of the proposals in this set of threads has sufficient traction, it would be silly to say “Let’s not do this thing that people want because it would make it harder to do a thing they don’t want”.
If typing the same variable from the caller to use in the parameter is really too much repetition, you could maybe just do this:
render_template("index.html", ... username="display_name", ... setups="setups", ... **Q("x y z")) ('index.html',) {'username': 'display_name', 'setups': 'setups', 'x': 1, 'y': 2, 'z': 3}
This doesn't work with tools, so I would be very annoyed if I had to work with this. An IDE friendly implementation can be found here: https://github.com/alexmojaki/sorcery#dict_of https://github.com/alexmojaki/sorcery/blob/master/sorcery/spells.py#L245 https://github.com/alexmojaki/sorcery/blob/master/sorcery/spells.py#L515 Despite having written these, I've never actually used them. Such magic would not be OK at work or in a library. But I've wanted to use them so many times, and a method blessed by the Python language and community would be so great.
On Apr 17, 2020, at 13:07, David Mertz <mertz@gnosis.cx> wrote:
I'm kinda leaning -0.5 even on the form that I think is least bad (the mode switch approach).
If typing the same variable from the caller to use in the parameter is really too much repetition, you could maybe just do this:
render_template("index.html", ... username="display_name", ... setups="setups", ... **Q("x y z")) ('index.html',) {'username': 'display_name', 'setups': 'setups', 'x': 1, 'y': 2, 'z': 3}
This is somewhat reminiscent of the argument that we don’t need any new magic for str.format because you can (and people did) just use the existing version that takes keywords and **locals() into it. The consensus there was that, even though that makes the magic explicit, it makes it too magical, and should be considered and anti-pattern, and we should try even harder for a better way to eliminate it, and ultimately that’s where f-strings came from. And the explicit magic didn’t even require frame-hacking in that case. I’m not sure how much that parallel means (or even what it would imply if it did mean a lot). I think I’m still -0.5, and the fact that you can get the same effect if you’re willing to use CPython frame hacks doesn’t make me either more or less negative.
On Fri, 17 Apr 2020 at 17:06, David Mertz <mertz@gnosis.cx> wrote:
I'm kinda leaning -0.5 even on the form that I think is least bad (the mode switch approach).
If typing the same variable from the caller to use in the parameter is really too much repetition, you could maybe just do this:
For what matters, when I stopped to think about this, I came out with this exactly design! Maybe this is a 'one obvious way' to deal with it, without changing any syntax?
render_template("index.html", ... username="display_name", ... setups="setups", ... **Q("x y z")) ('index.html',) {'username': 'display_name', 'setups': 'setups', 'x': 1, 'y': 2, 'z': 3}
Perhaps the spelling of `Q` might be something else. But in terms of character count, it's not worse than other proposals.
And luckily all you need to do this is get a version of Python later than 1.4 or something like that. :-)
def Q(names): ... import sys ... caller = sys._getframe(1) ... dct = {} ... for name in names.split(): ... dct[name] = eval(name, globals(), caller.f_locals) ... return dct
On Fri, Apr 17, 2020 at 2:41 PM Paul Svensson <paul-python@svensson.org> wrote:
For what it's worth, I'm a strong -1 on this whole thing, regardless of syntax. I think passing a lot of same-named parameters is an anti-pattern, that should be discouraged, not made easier. Passing an occasional x=x to so some function no disaster; if it happens often enough to be a problem, IMNSHO, you should look to change your coding style, not the language.
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IXARPT... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Apr 17, 2020 at 08:12:26PM -0300, Joao S. O. Bueno wrote:
For what matters, when I stopped to think about this, I came out with this exactly design!
Maybe this is a 'one obvious way' to deal with it, without changing any syntax?
There's nothing "obvious" about using a private implementation detail of CPython that probably won't work in other Pythons.
... caller = sys._getframe(1)
Outsider: "Python needs proper private variables." The Python community: "We don't need that, everyone knows not to use names starting with a single leading underscore because they're private." Also the Python community: "Let's use `_getframe`!" *wink* -- Steven
On Fri, Apr 17, 2020 at 10:29 AM Alex Hall <alex.mojaki@gmail.com> wrote:
Perhaps an easier next step would be to get better data about people's opinions with a simple poll? Is there a standard way to vote on things in this list?
No, but if you want to create a poll I would strongly advise that you create a poll at https://discuss.python.org/c/ideas/6 and post the link here (Discourse has built-in poll support). Otherwise you will get a swarm of votes intertwined with comments and keeping track of it all will be difficult. -Brett
I say we do a simple approval vote. Everyone ticks all the syntaxes that they think would be acceptable to include in the language. It's not very precise but it's easy and may inform what to do next.
Here is a script to generate a bunch of options:
``` for template in [ 'foo(bar, baz="thing", %s)', '{"baz": "thing", %s}', ]: for affix in ["=", ":", "::"]: for rest in [ f'spam{affix}, stuff{affix}', f'{affix}spam, {affix}stuff', ]: print(template % rest)
for template in [ 'foo(bar, baz="thing", %s, spam, stuff)', '{"baz": "thing", %s, spam, stuff}', ]: for separator in ["*", "**", ":", "::", "="]: print(template % separator)
# These options are currently valid syntax that are guaranteed to fail at runtime print("""\ foo(bar, baz="thing", **(spam, stuff)) foo(bar, baz="thing", **{spam, stuff}) {{"baz": "thing", spam, stuff}} """) ```
The output:
``` foo(bar, baz="thing", spam=, stuff=) foo(bar, baz="thing", =spam, =stuff) foo(bar, baz="thing", spam:, stuff:) foo(bar, baz="thing", :spam, :stuff) foo(bar, baz="thing", spam::, stuff::) foo(bar, baz="thing", ::spam, ::stuff) {"baz": "thing", spam=, stuff=} {"baz": "thing", =spam, =stuff} {"baz": "thing", spam:, stuff:} {"baz": "thing", :spam, :stuff} {"baz": "thing", spam::, stuff::} {"baz": "thing", ::spam, ::stuff} foo(bar, baz="thing", *, spam, stuff) foo(bar, baz="thing", **, spam, stuff) foo(bar, baz="thing", :, spam, stuff) foo(bar, baz="thing", ::, spam, stuff) foo(bar, baz="thing", =, spam, stuff) {"baz": "thing", *, spam, stuff} {"baz": "thing", **, spam, stuff} {"baz": "thing", :, spam, stuff} {"baz": "thing", ::, spam, stuff} {"baz": "thing", =, spam, stuff} foo(bar, baz="thing", **(spam, stuff)) foo(bar, baz="thing", **{spam, stuff}) {{"baz": "thing", spam, stuff}} ```
Are there any other options to consider? If people think this list looks complete I can create an online poll with them. I'm open to suggestions about preferred software/websites.
On Fri, Apr 17, 2020 at 6:41 PM Eric V. Smith <eric@trueblade.com> wrote:
On 4/17/2020 12:28 PM, Chris Angelico wrote:
On Sat, Apr 18, 2020 at 1:54 AM David Mertz <mertz@gnosis.cx> wrote:
Hmmm... I disagree with Chris.
I'm definitely -1 on a magic dangling 'foo=' after variable names. And something less than -1 on the even more magic "Lisp symbol that isn't a symbol" ':foo'.
Those are just ugly and mysterious.
However, I don't HATE the "mode switch" use of '*' or '**' in function calls. I've certainly written plenty of code where I use the same variable name in the calling scope as I bind in the call. Moreover, function *definitions* have an an analogous mode switch with an isolated '*'.
It sounds to me like there's a lot of weak support or weak opposition, with some of it spread between the proposal itself and the individual spellings.
Rodrigo, it may be time to start thinking about writing a PEP. If the Steering Council approves, I would be willing to be a (non-core-dev) sponsor; alternatively, there may be others who'd be willing to sponsor it. A PEP will gather all the different syntax options and the arguments for/against each, and will mean we're not going round and round on the same discussion points all the times.
I've been around for a while, and I can't imagine that any of these proposals would be accepted (but I've been accused of having a bad imagination). I'm saying that not to dissuade anyone from writing a PEP: far from it. I think it would be useful to have this on paper and accepted or rejected, either way. I'm saying this to set expectations: a PEP is a ton of work, and it can be disheartening to put in so much work for something that is rejected.
So, I'd be willing to sponsor such a PEP, but I'd be arguing that it get rejected.
And I say this as someone who has maybe 20 hours of work left on a PEP of my own that I think has less than a 50% chance of success. I already probably have 10 to 15 hours invested in it already.
Eric _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/K46TLU... Code of Conduct: http://python.org/psf/codeofconduct/
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/TVPJ5C... Code of Conduct: http://python.org/psf/codeofconduct/
The rush to push this to a PEP is unseemly. This has only been two days and I am sure that there will be many people who could be interested but haven't had a chance to look at the thread yet due to other committments. (Even in this time of Covid-19 lockdowns, some people have work and other committments.) Choosing good syntax should require more than just enumerating all the options and having a popularity contest. For starters, it would be nice if the options actually had some connection to the thing we are trying to do, rather than just randomly picking characters :-) -- Steven
On Sat, Apr 18, 2020 at 1:04 PM Steven D'Aprano <steve@pearwood.info> wrote:
The rush to push this to a PEP is unseemly. This has only been two days and I am sure that there will be many people who could be interested but haven't had a chance to look at the thread yet due to other committments.
All the more reason to have a coherent document. There've been a number of different syntaxes proposed, and a number of objections, some to the specific syntaxes and some to the proposal as a whole, and it's unclear at times which is which. If someone's going to catch up on the thread after a delay, wouldn't it be far better to simply read a PEP than to try to delve through the entire thread and figure out which parts to reply to? ChrisA
On Sat, Apr 18, 2020 at 01:09:32PM +1000, Chris Angelico wrote:
On Sat, Apr 18, 2020 at 1:04 PM Steven D'Aprano <steve@pearwood.info> wrote:
The rush to push this to a PEP is unseemly. This has only been two days and I am sure that there will be many people who could be interested but haven't had a chance to look at the thread yet due to other committments.
All the more reason to have a coherent document. There've been a number of different syntaxes proposed, and a number of objections, some to the specific syntaxes and some to the proposal as a whole, and it's unclear at times which is which. If someone's going to catch up on the thread after a delay, wouldn't it be far better to simply read a PEP than to try to delve through the entire thread and figure out which parts to reply to?
Depends on how much the care about the feature, how much time they are willing to spend, and how well the PEP summaries the entire thread. So far the discussion has been neither excessively huge nor excessively low signal-to-noise, so I'm not sure why the rush to move to a formal PEP. In the old days, anyone willing to make a PEP could just do so, without permission, so it's not like I'm saying that we shouldn't have a PEP. But it just seems premature to go from half-formed ideas on the list to a concrete proposal. -- Steven
On 18/04/20 1:53 am, Steven D'Aprano wrote:
On Sat, Apr 18, 2020 at 12:22:36AM +1200, Greg Ewing wrote:
Here's another idea for the bikeshed:
f(spam, pass eggs, ham)
How is "pass" meaningful here?
It means "pass these things on as they are". Think football pass, not quiz show pass. -- Greg
On Sat, Apr 18, 2020 at 10:23:59AM +1200, Greg Ewing wrote:
On 18/04/20 1:53 am, Steven D'Aprano wrote:
On Sat, Apr 18, 2020 at 12:22:36AM +1200, Greg Ewing wrote:
Here's another idea for the bikeshed:
f(spam, pass eggs, ham)
How is "pass" meaningful here?
It means "pass these things on as they are".
You mean, without changing them in any way? As in a no-op? I don't see how that helps. The first argument `spam` is passed on as it is with no changes. The second argument `eggs` is not passed on as it is, it is converted to keyword form `eggs=eggs` at least semantically, if not in the byte-code.
Think football pass, not quiz show pass.
That analogy doesn't help me, especially since I don't know which game of football you are thinking of (soccer, rugby league, rugby union, Gaelic, Australian Rules, Canadian football, gridiron, I probably missed a few...) so I don't know what the consequences of passing the ball will be. Buf for the two I am familar with, passing the ball doesn't change the ball at all. (Except it's position in space, I guess.) -- Steven
On Apr 17, 2020, at 21:01, Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Apr 18, 2020 at 10:23:59AM +1200, Greg Ewing wrote:
Think football pass, not quiz show pass.
That analogy doesn't help me, especially since I don't know which game of football you are thinking of (soccer, rugby league, rugby union, Gaelic, Australian Rules, Canadian football, gridiron, I probably missed a few...) so I don't know what the consequences of passing the ball will be.
You’re being deliberately obtuse here, and I don’t know why you do this. I think you have a legitimate argument to make against the proposal, but working this hard to find ways to pretend not to understand obvious things just so you can raise irrelevant arguments against every individual sentence makes it hard to see what actual substantive argument against the actual proposal is. The analogy is obvious—the `pass` statement means not doing anything when you have a chance to do something, like passing your turn in a quiz show; Greg’s proposed `pass spam` argument syntax means giving spam to the callable, like passing the ball to a teammate in soccer, and gridiron, and the many minor variations on those games, and the raft of similar games, and even wildly different games like basketball. If “pass the ball to a teammate” means anything at all in a team ball game, it always means giving control of the ball to the teammate by moving the ball to them. And you know that. And you also know that there is no chance that Greg has found some obscure game also called “football” where “pass the ball to a teammate” actually means “write the teammate’s name on the ball and bury it at a full moon”, and he’s decided to refer to that game as just “football” so he can trick you into thinking he means soccer or gridiron so you’ll make the wrong argument, because then he’ll win an automatic victory and we’ll all have to agree to his proposal even if we hate it, because that’s not how human discussions work. If you hadn’t pretended not to understand the obvious analogy, you could have used it to make your actual counterpoint (or at least what I’m _guessing_ your counterpoint is): OK, so the `pass spam` argument syntax is like football passing, giving spam to the callable. But just using `spam` as an argument already means that—in fact, we already call it “passing spam to the callable”. So it’s just being more explicit about the exact same thing we’re already saying. What does that add? How is anyone supposed to understand `pass spam` as “don’t just pass spam like usual, pass it as a keyword argument with name 'spam'”? At best, that sounds like the `new` keyword in JavaScript (where `new Spam(eggs)` means “don’t just create a new Spam by calling its constructor with eggs, also give it a blank object for the hidden this parameter”), which most people have a hard time learning and understanding but eventually sort of get the hang of doing, which is not usually what you’re aiming for with a language feature.
On Fri, Apr 17, 2020 at 11:39:41PM -0700, Andrew Barnert wrote:
On Apr 17, 2020, at 21:01, Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Apr 18, 2020 at 10:23:59AM +1200, Greg Ewing wrote:
Think football pass, not quiz show pass.
That analogy doesn't help me, especially since I don't know which game of football you are thinking of (soccer, rugby league, rugby union, Gaelic, Australian Rules, Canadian football, gridiron, I probably missed a few...) so I don't know what the consequences of passing the ball will be.
You’re being deliberately obtuse here, and I don’t know why you do this.
I really am not.
I think you have a legitimate argument to make against the proposal, but working this hard to find ways to pretend not to understand obvious things
I understand the proposal. I even said I was sympathetic to it. I don't understand the use of "pass" as a keyword to mean "auto-fill the argument value". It makes no sense to me, and Greg's analogy just makes it seem even more nonsensical. I think that "import" would make a better keyword, since we could at least say "you import the parameter's value from the surrounding scopes". (Note: I am not actually proposing "import" here as a good keyword to use. It's a terrible, awful keyword to use here. But at least there's some connection, even if it is awfully weak.) We are proposing something which actively modifies the meaning of an identifier in a function call. It changes the bare identifier (which would otherwise be evaluated and passed as a positional argument) into a keyword argument with value taken from the surrounding scope(s). How is this even remotely similar to any meaning of the word "pass", whether you are passing your turn in a quiz show, or passing a football, or a `pass` statement in Python, or some other meaning of the word? This is not a rhetorical question. American gridiron is, from the perspective of people outside of North America, a bizarre game: https://qz.com/150577/an-average-nfl-game-more-than-100-commercials-and-just... For all I know, they change the ball from one kind of ball to a different kind of ball after throwing it, for reasons. But I don't actually know that Greg had gridiron in mind. Maybe he's a gaelic football fanatic and there's some rule in gaelic football that applies after a pass. Who knows? Not me, I'm not a mind reader. The analogy doesn't make sense to me, but giving Greg the benefit of the doubt here, rather than dismissing it as a worthless non-sequitor, I'm allowing for the possibility that it does actually make sense from his perspective, and if I knew what he is thinking of it might make sense to me too. That's how analogies work. But honestly, being attacked by you like this, I'm kinda wishing I had just dismissed Greg's suggestion as too stupid for words instead of trying to encourage him (in my clumsy manner) to explain it.
just so you can raise irrelevant arguments against every individual sentence makes it hard to see what actual substantive argument against the actual proposal is.
Am I doing that?
The analogy is obvious—the `pass` statement means not doing anything when you have a chance to do something, like passing your turn in a quiz show;
Given that Greg says this is nothing like passing in a quiz show, that doesn't really matter. But for the record: Passing in a quiz show is an explicit action. If you want to do nothing, you can just sit still and say nothing until your time runs out. Passing means to explicitly hand over the turn to the next person. Or to request the next question without giving an answer to the current question. It depends on the quiz show.
Greg’s proposed `pass spam` argument syntax means giving spam to the callable,
All arguments are given to the callable. This is another explanation that explains nothing. Greg's example was `f(spam, pass eggs, ham)`. So what's the difference between giving spam to the callable, and giving eggs to the callable? They are both passed to the callable. I already made this observation in my earlier response. I trust you have read it, since that's the email you are responding to.
like passing the ball to a teammate in soccer, and gridiron, and the many minor variations on those games, and the raft of similar games, and even wildly different games like basketball. If “pass the ball to a teammate” means anything at all in a team ball game, it always means giving control of the ball to the teammate by moving the ball to them. And you know that.
Yes, I know that. And I still don't see how it is analogous to what we are proposing here. We pass spam to f, we pass eggs to f, we pass ham to f, but somehow only eggs justifies the keyword "pass" and you claim that it is obvious why and that I'm feigning confusion and being "deliberately obtuse".
And you also know that there is no chance that Greg has found some obscure game also called “football” where “pass the ball to a teammate” actually means “write the teammate’s name on the ball and bury it at a full moon”,
That's some mighty impressive straw-manning there. What I actually said was "I don't know which game of football you are thinking of" (which is true), "so I don't know what the consequences of passing the ball will be" (which is also true). In netball, when you pass the ball, the person who receives it has to immediately stop dead. In Australian Rules, once you have passed the ball, anyone tackling you has to immediately release you. In Rugby, you can only pass the ball backwards. There are all sorts of rules that might apply. How am I supposed to know what rule Greg is thinking of when I don't even know which code of football he is thinking of?
and he’s decided to refer to that game as just “football” so he can trick you into thinking he means soccer or gridiron so you’ll make the wrong argument, because then he’ll win an automatic victory and we’ll all have to agree to his proposal even if we hate it, because that’s not how human discussions work.
That's an impressive rant. Maybe you should settle down a bit and consider that I'm not doing what you accuse me of, that my confusion isn't malicious play acting, but I *genuinely* do not understand this analogy, that it isn't as "obvious" as you imagine.
If you hadn’t pretended not to understand the obvious analogy, you could have used it to make your actual counterpoint (or at least what I’m _guessing_ your counterpoint is): OK, so the `pass spam` argument syntax is like football passing, giving spam to the callable. But just using `spam` as an argument already means that—in fact, we already call it “passing spam to the callable”.
Dammit, I wish I had made that argument! Oh wait, *I did*. -- Steven
On Fri, Apr 17, 2020 at 11:57 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Apr 17, 2020 at 11:25:15AM +0200, Alex Hall wrote:
I don't think you really need to know what it means to read the code for most purposes.
*blink*
You look at the function call and you can see a bunch of names being passed to self.do_something. If the function call has 'keyword=keyword' in it instead of just 'keyword', that's not adding much information.
Of course it adds important information! It is adding the critical information: which parameter gets assigned the specified value.
Of course it's adding some information, but it's not information a reader usually has to think about.
Put yourself in the position of someone who doesn't know the meaning of the * in a function call. What's the difference between these two calls?
f(x) f(*, x)
It's not good enough to merely say that you're passing an argument `x`. That's true of both calls. There must be a difference between the two, otherwise why have the star?
Readers don't generally have to compare two similar looking pieces of code. If they do, it's generally in a conversation with someone (e.g. when reviewing a PR) and they can ask that person what the asterisk means. So "What's the difference between these two calls?" is not usually a relevant question. Positional arguments tell us that values are assigned to parameters from
left to right:
function(spam, # first parameter eggs, # second parameter cheese, # third parameter )
but we have no clue at all what the names of those parameters are. That's the down-side of positional arguments, and one of the reasons why positional arguments don't scale. Knowing the names of the parameters is often important.
Keyword arguments fill in the missing critical information:
function(alpha=spam, beta=eggs, gamma=cheese, )
and we can shuffle the order around and the mapping of argument value to parameter name is still clear.
That doesn't matter here because the parameter and variable names have to match to use the shortcut.
The technical details of how an argument is being passed are usually not important to readers.
I don't care about the argument passing implementation. I care about the meaning of the code. Here is a real-world case:
open('example.txt', encoding, newline, errors, mode)
open('example.txt', *, encoding, newline, errors, mode)
Would you agree that the two of those have *radically* different meanings? The first will, if you are lucky, fail immediately; the second may succeed. But to the poor unfortunate reader who doesn't know what the star does, the difference is puzzling.
This holds even more strongly if the various parameters take the same type:
# a real signature from one of my functions def function( meta:bool, dunder:bool, private:bool, ignorecase:bool, invert:bool)
function(dunder, invert, private, meta, ignorecase) function(*, dunder, invert, private, meta, ignorecase)
No reader will ever have to think about the difference. They will simply see the second version and know which arguments are being passed. Also, your examples are clearly demonstrating that using the shortcut makes it easier to avoid mistakes. I'm not really sure what you're actually concerned about. Can you give a hypothetical scenario explaining how someone would be confused in practice? Here's the worst case scenario I can imagine: A reader is trying to debug some code that isn't behaving right. The call `function(*, dunder, invert, private, meta, ignorecase)` isn't doing what they expect. if they put a breakpoint or some prints inside `function` they would probably see that the variables `dunder`, `invert`, etc. have the correct/obvious values inside, but they don't. Instead they visually inspect the function call and the function definition. They don't know what '*' means, and they don't bother to find out, so they think arguments are being passed positionally. They compare the positions of the arguments and parameters in the call and definition and see they don't match, and they think that's the problem. They try to fix the call by rearranging the arguments or switching to explicit keyword arguments. That doesn't change the behaviour of the program so they spend some time being confused about that.
This holds even more strongly if the various parameters take the same type:
# a real signature from one of my functions def function( meta:bool, dunder:bool, private:bool, ignorecase:bool, invert:bool)
function(dunder, invert, private, meta, ignorecase) function(*, dunder, invert, private, meta, ignorecase)
No reader will ever have to think about the difference. They will simply see the second version and know which arguments are being passed. Also, your examples are clearly demonstrating that using the shortcut makes it easier to avoid mistakes.
It's true that they don't have to think any the DIFFERENCE (since only one version will be read), but they do have to think about the difference between what they might believe it does and what it does. I'm not really sure what you're actually concerned about. Can you give a
hypothetical scenario explaining how someone would be confused in practice?
I tried to explain one: the reader knows the parameter order well and misses or ignores the star and thinks they have understood the function call correctly when they haven't. Again, I'm not sure it's enough to nix the whole idea, and I agree the syntax can just as easily prevent mistakes as you said.
On 17/04/2020 11:37, Alex Hall wrote:
On Fri, Apr 17, 2020 at 11:57 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Fri, Apr 17, 2020 at 11:25:15AM +0200, Alex Hall wrote:
You look at the function call and you can see a bunch of names being passed to self.do_something. If the function call has 'keyword=keyword' in it instead of just 'keyword', that's not adding much information.
Of course it adds important information! It is adding the critical information: which parameter gets assigned the specified value.
Of course it's adding some information, but it's not information a reader usually has to think about.
Uh, it's information that as a reader I consider critical.
Readers don't generally have to compare two similar looking pieces of code. If they do, it's generally in a conversation with someone (e.g. when reviewing a PR) and they can ask that person what the asterisk means. So "What's the difference between these two calls?" is not usually a relevant question.
Actually if the difference is only a couple of characters, it is relevant. In another thread, it was pointed out that having both get_default() and get_defaults() methods on a class was asking for comedy moments (well, comedy from everyone else's point of view). Similarly, if you look at "f(*, x)", don't understand the star or otherwise slide over it and consequently make mistaken assumptions about what x is, well, it matters quite a lot.
The technical details of how an argument is being passed are usually not important to readers.
I don't care about the argument passing implementation. I care about the meaning of the code. Here is a real-world case:
open('example.txt', encoding, newline, errors, mode)
open('example.txt', *, encoding, newline, errors, mode)
Would you agree that the two of those have *radically* different meanings? The first will, if you are lucky, fail immediately; the second may succeed. But to the poor unfortunate reader who doesn't know what the star does, the difference is puzzling.
This holds even more strongly if the various parameters take the same type:
# a real signature from one of my functions def function( meta:bool, dunder:bool, private:bool, ignorecase:bool, invert:bool)
function(dunder, invert, private, meta, ignorecase) function(*, dunder, invert, private, meta, ignorecase)
No reader will ever have to think about the difference. They will simply see the second version and know which arguments are being passed.
I seem to be immune to this magical knowledge.
Also, your examples are clearly demonstrating that using the shortcut makes it easier to avoid mistakes.
It shows that using positional parameters makes it easier to make mistakes, but that's not news. -- Rhodri James *-* Kynesim Ltd
function(*, dunder, invert, private, meta, ignorecase)
No reader will ever have to think about the difference. They will simply see the second version and know which arguments are being passed.
I seem to be immune to this magical knowledge.
Sorry, what? How is there any doubt that the arguments being passed are dunder, invert, private, meta, and ignorecase? They're right there. Now, which parameters those arguments are bound to is less obvious, but: 1. When you read the function call, you're thinking about the arguments, not the parameters. You can see which information goes into the function, and the first question you should ask yourself is 'is that the right information?'. 2. Even if you have no idea what the parameters of the function are, you can already reasonably guess that they match the argument names, and that guess is correct! If you have some idea what the parameter names are, you can be more confident in that guess. Or, if you didn't know the parameter names, but you know what the `**` separator means, now you know the names. 3. You probably only start to think about parameter binding when you open up the function definition, and when you do that, binding is still probably not the first thing you look at. You're probably thinking about what the function does in the body. 4. If there are just a few arguments, you're less likely to miss or ignore the `**`. 5. If there are many arguments, you're less likely to notice any mismatch between argument and parameter positions that might falsely make you think something is wrong. That is, you're less likely to either remember the parameter order or to go through the trouble of inspecting and comparing the orders. 6. If you're thinking about parameter binding and argument order, you're inspecting the arguments at least somewhat closely, and will almost certainly notice that the `**` is present. If you know what it means, problem solved. If you don't, you're at least likely to think about it and try looking it up or ask someone. It takes a very specific kind of skepticism/suspicion to think "the previous programmer messed up the argument order so these parameters are definitely bound wrong, and also that weird `**` that I don't know the meaning of has no bearing on that, and I'm not going to check with a print() or a debugger". In summary, I think that the vast majority of the time, the new syntax will cause no confusion (even when it potentially could have) because people will skim and follow their intuition. Occasionally, it might cause brief, mild confusion, and that will mostly only be one or two times for each programmer as they learn the new syntax. I can't see it causing serious confusion outside of very specific and rare circumstances.
Also, your examples are clearly demonstrating that using the shortcut makes
it easier to avoid mistakes.
It shows that using positional parameters makes it easier to make mistakes, but that's not news.
Right, but people use them anyway, sometimes even to solve the exact problem we're talking about. You're responding to me responding to Steven D'Aprano, here's what he said just slightly earlier: We're not talking about positional arguments here. If you want to
include positional arguments in the analysis, then we already have a perfectly good way to write function calls without repeating outselves: # Instead of this: function(spam=spam, eggs=eggs, cheese=cheese) # Just do this: function(spam, eggs, cheese) And we're done, the problem is solved, and no new syntax is needed.
That's bad, and we should discourage that. Encouraging people to safely pass named arguments instead of abusing positional arguments would improve the readability and correctness of code overall. Below are some cases from the CPython repo of people passing many positional arguments. They could be improved by using keyword arguments more, but with the currently available syntax, that would be very verbose and undesirable, so it's understandable that it wasn't written that way in the first place and hasn't been changed since then. ``` File "Lib/subprocess.py", line 947 self._execute_child(args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, restore_signals, gid, gids, uid, umask, start_new_session) File "Lib/subprocess.py", line 1752 self.pid = _posixsubprocess.fork_exec( args, executable_list, close_fds, tuple(sorted(map(int, fds_to_keep))), cwd, env_list, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, errpipe_read, errpipe_write, restore_signals, start_new_session, gid, gids, uid, umask, preexec_fn) File "Lib/distutils/ccompiler.py", line 692 self.link(CCompiler.SHARED_LIBRARY, objects, self.library_filename(output_libname, lib_type='shared'), output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang) File "Lib/distutils/ccompiler.py", line 713 self.link(CCompiler.SHARED_OBJECT, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, export_symbols, debug, extra_preargs, extra_postargs, build_temp, target_lang) File "Lib/distutils/ccompiler.py", line 731 self.link(CCompiler.EXECUTABLE, objects, self.executable_filename(output_progname), output_dir, libraries, library_dirs, runtime_library_dirs, None, debug, extra_preargs, extra_postargs, None, target_lang) File "Lib/distutils/cygwinccompiler.py", line 243 UnixCCompiler.link(self, target_desc, objects, output_filename, output_dir, libraries, library_dirs, runtime_library_dirs, None, # export_symbols, we do this in our def-file debug, extra_preargs, extra_postargs, build_temp, target_lang) File "Lib/http/cookiejar.py", line 1563 return Cookie(version, name, value, port, port_specified, domain, domain_specified, domain_initial_dot, path, path_specified, secure, expires, discard, comment, comment_url, rest) File "Lib/http/cookiejar.py", line 2057 c = Cookie(0, name, value, None, False, domain, domain_specified, initial_dot, path, False, secure, expires, discard, None, None, {}) ```
And we're done, the problem is solved, and no new syntax is needed.
That's bad, and we should discourage that. Encouraging people to safely pass named arguments instead of abusing positional arguments would improve the readability and correctness of code overall.
I have nothing significant to add, but want to emphasize this. I think it is potentially a large point in favor of the syntax change: nudging people away from abusing positional arguments. --- Ricky. "I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler
On 19/04/2020 17:06, Alex Hall wrote:
function(*, dunder, invert, private, meta, ignorecase)
No reader will ever have to think about the difference. They will simply see the second version and know which arguments are being passed.
I seem to be immune to this magical knowledge.
Sorry, what? How is there any doubt that the arguments being passed are dunder, invert, private, meta, and ignorecase? They're right there.
Oh, so that's what you meant. But how is this different from function(dunder, invert, private, meta, ignorecase) if you trust yourself to get the argument order right, or function(dunder=dunder, invert=invert, private=private, meta=meta, ignorecase=ignorecase) if you don't? I still don't get why you think that last form is a problem.
Now, which parameters those arguments are bound to is less obvious, but:
1. When you read the function call, you're thinking about the arguments, not the parameters. You can see which information goes into the function, and the first question you should ask yourself is 'is that the right information?'.
Bitter experience has taught me to think about both the arguments and the parameters. You can't answer the question "is that the right information?" until you know what the right information is.
2. Even if you have no idea what the parameters of the function are, you can already reasonably guess that they match the argument names, and that guess is correct! If you have some idea what the parameter names are, you can be more confident in that guess. Or, if you didn't know the parameter names, but you know what the `**` separator means, now you know the names.
All I can say is that I doubt I would make that association. I certainly don't when similar things come up in function definitions.
3. You probably only start to think about parameter binding when you open up the function definition, and when you do that, binding is still probably not the first thing you look at. You're probably thinking about what the function does in the body.
Well, no, that's not how I work at all.
4. If there are just a few arguments, you're less likely to miss or ignore the `**`.
True. But on the other hand, you have less excuse not to be explicit about which names are bound to which.
5. If there are many arguments, you're less likely to notice any mismatch between argument and parameter positions that might falsely make you think something is wrong. That is, you're less likely to either remember the parameter order or to go through the trouble of inspecting and comparing the orders.
In my experience, the more arguments there are, the more likely it is that something has been got wrong, so I'm actually more likely to go to the trouble of inspecting and comparing the orders. Maybe I'm paranoid, but I've caught enough bugs that way to feel justified in my paranoia.
6. If you're thinking about parameter binding and argument order, you're inspecting the arguments at least somewhat closely, and will almost certainly notice that the `**` is present. If you know what it means, problem solved. If you don't, you're at least likely to think about it and try looking it up or ask someone. It takes a very specific kind of skepticism/suspicion to think "the previous programmer messed up the argument order so these parameters are definitely bound wrong, and also that weird `**` that I don't know the meaning of has no bearing on that, and I'm not going to check with a print() or a debugger".
If I have to go away and look some syntax up, that syntax has slowed me down. This doesn't seem like a win to me. -- Rhodri James *-* Kynesim Ltd
On Sun, Apr 19, 2020 at 06:06:50PM +0200, Alex Hall wrote:
function(*, dunder, invert, private, meta, ignorecase)
No reader will ever have to think about the difference. They will simply see the second version and know which arguments are being passed.
I seem to be immune to this magical knowledge.
Sorry, what? How is there any doubt that the arguments being passed are dunder, invert, private, meta, and ignorecase? They're right there.
That tells us the meaning of the arguments in the *caller's* context. It doesn't tell us the meaning of the arguments in the *callee's* context, which is critical information. There is a huge difference between these: process_files(delete=obsolete_files, archive=irrelevent_files) process_files(archive=obsolete_files, delete=irrelevent_files) even though both calls receive the same arguments, so it is critical to know the callee's context, i.e. the parameters those arguments get bound to.
Now, which parameters those arguments are bound to is less obvious, but:
1. When you read the function call, you're thinking about the arguments, not the parameters. You can see which information goes into the function, and the first question you should ask yourself is 'is that the right information?'.
And the only way to know that is to think about the parameters. I trust that you will agree that we should care about the difference between (let's say): pow(8, 2) # 64 pow(2, 8) # 256 and that it's critical to match the arguments to the parameters in the right order or you will get the wrong answer. I can't tell you how many times I messed up list.insert() calls because I got the arguments in the wrong order. So for the reader who doesn't know what this star argument `*` does, it is not enough for them to say "Oh that's fine then, we're passing the right arguments, it doesn't matter what parameters they get matched to". The parameters are absolutely critical. -- Steven
On 20/04/20 2:57 pm, Steven D'Aprano wrote:
process_files(delete=obsolete_files, archive=irrelevent_files) process_files(archive=obsolete_files, delete=irrelevent_files)
That's not a use case for the proposed feature. The intended use cases are more like def fancy_file_processing(delete, archive): extra_processing_for_deleted(delete) other_processing_for_archived(archive) basic_file_processing(delete=delete, archive=archive) Since fancy_file_processing is a wrapper around basic_file_processing, it makes sense to name the corresponding arguments the same way. Now maybe someone will misuse it in a situation that doesn't involve wrapping and warp their local names to fit. But any language feature can be abused. The blame for abuse lies with the abuser, not the feature. -- Greg
On Mon, Apr 20, 2020 at 5:04 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Sun, Apr 19, 2020 at 06:06:50PM +0200, Alex Hall wrote:
Sorry, what? How is there any doubt that the arguments being passed are dunder, invert, private, meta, and ignorecase? They're right there.
That tells us the meaning of the arguments in the *caller's* context.
[...] it is critical to know the callee's context, i.e. the parameters those arguments get bound to.
Now, which parameters those arguments are bound to is less obvious, but:
Steven, what happened above? Immediately after your objection ends, you quoted me handling that objection. It feels like you're not reading what I say.
And the only way to know that is to think about the parameters. I trust that you will agree that we should care about the difference between (let's say):
pow(8, 2) # 64 pow(2, 8) # 256
and that it's critical to match the arguments to the parameters in the right order or you will get the wrong answer.
I can't tell you how many times I messed up list.insert() calls because I got the arguments in the wrong order.
None of your examples (delete_files, pow, list.insert) are relevant to the proposal. The hypothetical situation is that we have a call like: function(**, dunder, invert, private, meta, ignorecase) and a function definition: def function(dunder, invert, private, meta, ignorecase): ... My claim is that you'll probably quickly notice that the names are the same. Then you'll guess that they probably match up. Checking that the positions match seems a bit paranoid to me when you're reading someone else's 'complete' code - if they messed that up, they probably would have noticed. Anyway, talking about this has made me feel not so convinced of my own argument. Especially if the list of parameters is long, it's hard to mentally do something like `set(argument_names) <= set(parameter_names)` as there's too many names to remember. You're more likely to do something like `for argument_name, parameter_name in zip(argument_names, parameter_names): assert argument_name == parameter_name` and then notice that something is suspiciously wrong. So let's assume that a beginner is likely to notice arguments out of order and think that something is wrong, and focus on what happens from there. I think it's fair to say that the main concern is that they will have to look up and learn the syntax, and that is significant. But that (and anything else that happens during this time of initial confusion) is essentially an O(1) cost. The code and coders who use the syntax benefit from it for their lifetime, which is O(n). Exactly what the benefits of the syntax are, I will have to leave that for another email...
On Mon, Apr 20, 2020 at 1:35 PM Alex Hall <alex.mojaki@gmail.com> wrote:
So let's assume that a beginner is likely to notice arguments out of order and think that something is wrong, and focus on what happens from there. I think it's fair to say that the main concern is that they will have to look up and learn the syntax, and that is significant.
As far as beginners are concerned, yes, it's one more thing to learn of a already complex interface. But that's not my larger concen: I'm found that newbies often get confused about scope -- that "x" when you call a function is different than "x" in side that function, even though they may sometimes be the same name (same value). And if we make it even easier, and more common, to implicitly use the same names, they will only be more confused. Python has become much less of a newbie friendly language over the years, and that's fine, but I'd like it to be considered. -CHB
But that (and anything else that happens during this time of initial confusion) is essentially an O(1) cost. The code and coders who use the syntax benefit from it for their lifetime, which is O(n).
Exactly what the benefits of the syntax are, I will have to leave that for another email... _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/HHJ6XX... Code of Conduct: http://python.org/psf/codeofconduct/
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Mon, Apr 20, 2020 at 10:46 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Mon, Apr 20, 2020 at 1:35 PM Alex Hall <alex.mojaki@gmail.com> wrote:
So let's assume that a beginner is likely to notice arguments out of order and think that something is wrong, and focus on what happens from there. I think it's fair to say that the main concern is that they will have to look up and learn the syntax, and that is significant.
As far as beginners are concerned, yes, it's one more thing to learn of a already complex interface. But that's not my larger concen: I'm found that newbies often get confused about scope -- that "x" when you call a function is different than "x" in side that function, even though they may sometimes be the same name (same value).
And if we make it even easier, and more common, to implicitly use the same names, they will only be more confused.
Python has become much less of a newbie friendly language over the years, and that's fine, but I'd like it to be considered.
-CHB
This and another post about the complexity of function call syntax has convinced me that it's not worth adding more features to function calls. Let's only consider dicts. Unfortunately the most common use case is still functions and keyword arguments, which means immediately unpacking a dict. I was playing around with refactoring this call into the new syntax: ``` return _copytree( entries=entries, src=src, dst=dst, symlinks=symlinks, ignore=ignore, copy_function=copy_function, ignore_dangling_symlinks=ignore_dangling_symlinks, dirs_exist_ok=dirs_exist_ok, ) ``` Here it is with the function call syntax which sadly I don't want anymore, even though it looks so pretty: ``` return _copytree( **, entries, src, dst, symlinks, ignore, copy_function, ignore_dangling_symlinks, dirs_exist_ok, ) ``` Here it is with the dict syntax I've been advocating for: ``` return _copytree(**{**, entries, src, dst, symlinks, ignore, copy_function, ignore_dangling_symlinks, dirs_exist_ok, }) ``` A couple of things strike me about this: 1. `(**{**,` is an ugly, complicated sequence of punctuation. 2. It's really not obvious what the best way to format this is. I hate trying to decide between squeezing too much into one line and using too many lines. 3. It's probably quite easy to forget the comma after the `**` and thus accidentally unpack the `entries` value. It'd be like accidental implicit string concatenation. We would probably need to make doing that a SyntaxWarning or SyntaxError, and ensure that there's a helpful message that makes it easy to find the problem. We can deal with (3) by using a different separator, e.g. `:`, `::`, or `=`. But that still leaves (1) and (2). And it means we don't get to just switch on the mode with a normal `**kwargs`. Another possibility I'm considering is to not have any kind of syntax that exists only for this purpose. The rule could be simply "a lone name in a dict is an auto-named pair". It just needs to be clear that you're using a dict and not a set, so there needs to be at least one colon or `**` present. Then the call would look like: ``` return _copytree(**{ "entries": entries, src, dst, symlinks, ignore, copy_function, ignore_dangling_symlinks, dirs_exist_ok, }) ``` That looks pretty weird, right? At the same time it sort of feels like after the first pair it's saying "and so on and so forth, you get the idea". One thought I have about this, which also somewhat applies to any of the mode-switch syntaxes for dicts, is that it's quite easy to accidentally transform a dict into a set if part of the dict gets deleted. Not sure how much of a concern that is.
On Mon, Apr 20, 2020 at 10:23:29PM +0200, Alex Hall wrote:
On Mon, Apr 20, 2020 at 5:04 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Sun, Apr 19, 2020 at 06:06:50PM +0200, Alex Hall wrote:
Sorry, what? How is there any doubt that the arguments being passed are dunder, invert, private, meta, and ignorecase? They're right there.
That tells us the meaning of the arguments in the *caller's* context.
[...] it is critical to know the callee's context, i.e. the parameters those arguments get bound to.
Now, which parameters those arguments are bound to is less obvious, but:
Steven, what happened above? Immediately after your objection ends, you quoted me handling that objection. It feels like you're not reading what I say.
Trust me Alex, I am reading what you say. What I don't know is whether you mean what you say, because frankly I think your position that the caller of a function doesn't need to care too much, if at all, about the parameters their arguments are applied to (except under special circumstances, such as when reading the source code of the function), is so obviously wrong that I don't understand why you are standing by it. This thread is long enough, so I'm not going to drag it out further with more tedious "but you said..." quotes. I'm sticking with my response, you are welcome to stick with yours, let's move on. -- Steven
On Thu, Apr 16, 2020 at 01:41:42PM -0400, Eric V. Smith wrote:
On 4/16/2020 1:30 PM, Rhodri James wrote:
I beg to differ. I do find "def foo(a, *, b)" gets in the way of readability.
And what would you do if you wanted to call:
self.do_something(positional, keyword=keyword, keyword1=somethingelse, keyword2=keyword2)
Sorry, am I missing something? Why wouldn't you just call it precisely as you said? self.do_something(positional, keyword=keyword, keyword1=somethingelse, keyword2=keyword2) is syntax that goes back to Python 1.x. You don't need the `*` in the function definition for it to work. -- Steven
I beg to differ. I do find "def foo(a, *, b)" gets in the way of readability.
-- Rhodri James *-* Kynesim Ltd
In what way? In any case, focusing on the calling syntax being proposed, is there anything unreadable about: foo(a, *, b) compared to foo(a, b=b) ? I think in the proposed syntax it's quite easy to understand that we're passing arguments 'a' and 'b' even if we have no idea what the '*' means, and it's small enough that it's fairly easy to mentally filter out.
For function definitions, the introduction of `*` to mark keyword-only parameters was consistent with existing syntax in a sense that `def foo(*args, bar)` had `args` consume all positional arguments, so `bar` can only be passed via keyword. Now using `def foo(*, bar)` just omits the positional argument part, but leaves `bar` unchanged. However for function calls you can have positional arguments following argument unpacking: def foo(a, b): print(a, b) a, b = 2, 1 foo(*[], b, a) # prints "1 2" Now omitting the unpacking part would change the meaning of what follows, namely that it is to be interpreted as keyword arguments: foo(*, b, a) # prints "2 1" Sure you could argue that a lonely `*` in a function call has to have some effect, so it must change what comes after it, but this slight asymmetry between definition and calling of a function could be confusing. On 16.04.20 19:54, Alex Hall wrote:
I beg to differ. I do find "def foo(a, *, b)" gets in the way of readability.
-- Rhodri James *-* Kynesim Ltd
In what way?
In any case, focusing on the calling syntax being proposed, is there anything unreadable about:
foo(a, *, b)
compared to
foo(a, b=b)
? I think in the proposed syntax it's quite easy to understand that we're passing arguments 'a' and 'b' even if we have no idea what the '*' means, and it's small enough that it's fairly easy to mentally filter out.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/NWJTHD... Code of Conduct: http://python.org/psf/codeofconduct/
You have a fair point @DominikVilsmeier though I still believe this can workout there are other alternatives up for discussion such as using the `=` character. You see: after a keyword parameter you cannot define other positional ones, so the new syntax can assume all parameters following a first keyword parameter are to be captured if not explicitly declared. So we can have a syntax in which these three statements are the same: ```python foo(a, b=b, c=c) foo(a, b=b, c) foo(a, =, b, c) ``` But in this option I find `foo(a, b=b, c)` obscure that `c` will be captured as keyword argument. For now I still believe we can live with the `*` being slightly asymmetric. For the record, the positional-only specifier `/` already has no other real meaning other than delimiting different types of arguments.
I'm not sure if this is doable from the compiler perspective, but what about allowing tuples after `**` unpacking: requests.post(url, **(data, params)) # similar to requests.post(url, data=data, params=params) Probably some magic would need to happen in order to merge the names with their values, but the same is true for the `post(url, data=, params=)` syntax. On 16.04.20 18:57, oliveira.rodrigo.m@gmail.com wrote:
@StevenDAprano and this goes for @RhodriJames , thank you for sharing your point of view. Indeed the proposed syntax is obscure and would not be that readable for beginners.
Couldn't we work around this so? The concept is still good for me just the syntax that is obscure, maybe something like this would work:
```python # '*' character delimits that subsequent passed parameters will be passed # as keyword arguments with the same name of the variables used self.do_something(positional, *, keyword) # this would be equivalent to: self.do_something(positional, keyword=keyword) ``` I believe this is readable even if you don't know Python: `positional` and `keyword` are being passed as parameters, the `*` character is mysterious at first but so it is in `def foo(a, *, b)` and it doesn't get into the way of basic readability.
Rodrigo Martins de Oliveira _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/N2ZY5N... Code of Conduct: http://python.org/psf/codeofconduct/
Dominik Vilsmeier wrote:
I'm not sure if this is doable from the compiler perspective, but what about allowing tuples after `**` unpacking:
requests.post(url, **(data, params))
# similar to requests.post(url, data=data, params=params)
+1. I can see the practical utility of the feature, but was strongly against the other syntax proposals so far. IMO, the above alternative does a great job of using an existing feature, and I think it would be rather easy to explain how it works. On Thu, Apr 16, 2020 at 3:06 PM Dominik Vilsmeier <dominik.vilsmeier@gmx.de> wrote:
I'm not sure if this is doable from the compiler perspective, but what about allowing tuples after `**` unpacking:
requests.post(url, **(data, params))
# similar to requests.post(url, data=data, params=params)
Probably some magic would need to happen in order to merge the names with their values, but the same is true for the `post(url, data=, params=)` syntax.
On 16.04.20 18:57, oliveira.rodrigo.m@gmail.com wrote:
@StevenDAprano and this goes for @RhodriJames , thank you for sharing your point of view. Indeed the proposed syntax is obscure and would not be that readable for beginners.
Couldn't we work around this so? The concept is still good for me just the syntax that is obscure, maybe something like this would work:
```python # '*' character delimits that subsequent passed parameters will be passed # as keyword arguments with the same name of the variables used self.do_something(positional, *, keyword) # this would be equivalent to: self.do_something(positional, keyword=keyword) ``` I believe this is readable even if you don't know Python: `positional` and `keyword` are being passed as parameters, the `*` character is mysterious at first but so it is in `def foo(a, *, b)` and it doesn't get into the way of basic readability.
Rodrigo Martins de Oliveira _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/N2ZY5N... Code of Conduct: http://python.org/psf/codeofconduct/
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/S44VDM... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Apr 16, 2020 at 10:13 PM Kyle Stanley <aeros167@gmail.com> wrote:
Dominik Vilsmeier wrote:
I'm not sure if this is doable from the compiler perspective, but what about allowing tuples after `**` unpacking:
requests.post(url, **(data, params))
# similar to requests.post(url, data=data, params=params)
+1. I can see the practical utility of the feature, but was strongly against the other syntax proposals so far. IMO, the above alternative does a great job of using an existing feature, and I think it would be rather easy to explain how it works.
If we go in that direction, I'd prefer curly braces instead so that it's more reminiscient of a dict instead of a tuple, although technically it will look like a set literal. Some other possible syntaxes for a dict (which would have to be unpacked in a function call) with string keys equal to the variable name, i.e. {"foo": foo, "bar": bar}: {*, foo, bar} {**, foo, bar} {:, foo, bar} {{ foo, bar }} {* foo, bar *} {: foo, bar :} {: foo, bar} Personally in these cases I usually write dict(foo=foo, bar=bar) instead of a dict literal because I don't like the quotes, but even then I'm sad that I have to write the word 'dict'. So I would prefer that we covered raw dicts rather than function calls, or both.
On Thu, Apr 16, 2020 at 10:13 PM Kyle Stanley <aeros167@gmail.com <mailto:aeros167@gmail.com>> wrote:
Dominik Vilsmeier wrote: > I'm not sure if this is doable from the compiler perspective, but what > about allowing tuples after `**` unpacking: > > requests.post(url, **(data, params)) > > # similar to > requests.post(url, data=data, params=params)
+1. I can see the practical utility of the feature, but was strongly against the other syntax proposals so far. IMO, the above alternative does a great job of using an existing feature, and I think it would be rather easy to explain how it works.
If we go in that direction, I'd prefer curly braces instead so that it's more reminiscient of a dict instead of a tuple, although technically it will look like a set literal. Do you intend this "shortcut" syntax to also work in other contexts? Because indeed if it looks like a set literal it would be confusing if it emerges as a dict.
Some other possible syntaxes for a dict (which would have to be unpacked in a function call) with string keys equal to the variable name, i.e. {"foo": foo, "bar": bar}:
{*, foo, bar} This looks like someone forgot an iterable after the `*`. {**, foo, bar} This resembles **kwargs but using it to unpack keyword arguments it looks weird: `func(**{**, foo, bar})`. {:, foo, bar} {{ foo, bar }} This is already valid syntax and attempts to store a set inside another set. {* foo, bar *} The `*foo` part is already valid syntax and the `bar *` looks like someone forgot the second operand for the binary multiply. {: foo, bar :} {: foo, bar}
Personally in these cases I usually write dict(foo=foo, bar=bar) instead of a dict literal because I don't like the quotes, but even then I'm sad that I have to write the word 'dict'. So I would prefer that we covered raw dicts rather than function calls, or both. If at all, I'd prefer something like {:foo, :bar}. But anyway this takes
On 16.04.20 22:28, Alex Hall wrote: the proposal in a different direction.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SATYP4... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Apr 16, 2020 at 10:47 PM Dominik Vilsmeier <dominik.vilsmeier@gmx.de> wrote:
On 16.04.20 22:28, Alex Hall wrote:
On Thu, Apr 16, 2020 at 10:13 PM Kyle Stanley <aeros167@gmail.com> wrote:
Dominik Vilsmeier wrote:
I'm not sure if this is doable from the compiler perspective, but what about allowing tuples after `**` unpacking:
requests.post(url, **(data, params))
# similar to requests.post(url, data=data, params=params)
+1. I can see the practical utility of the feature, but was strongly against the other syntax proposals so far. IMO, the above alternative does a great job of using an existing feature, and I think it would be rather easy to explain how it works.
If we go in that direction, I'd prefer curly braces instead so that it's more reminiscient of a dict instead of a tuple, although technically it will look like a set literal.
Do you intend this "shortcut" syntax to also work in other contexts? Because indeed if it looks like a set literal it would be confusing if it emerges as a dict.
My intended semantics are the same as yours, just with different brackets. Of your proposal I could equally say that it looks like a tuple literal and it would be confusing if it emerges as a dict.
{*, foo, bar}
This looks like someone forgot an iterable after the `*`.
I'm not sure how that can be an issue unless someone is not sure whether the code they're looking at runs without a syntax error.
{**, foo, bar}
This resembles **kwargs but using it to unpack keyword arguments it looks weird: `func(**{**, foo, bar})`.
In that special case the inner ** can be optional, and then it just looks like my variation of your proposal. Or we can have two similar looking syntaxes: {**, foo, bar} func(**, foo, bar) If you're familiar with one of them then the other should become naturally intuitive. The function call syntax is essentially the earlier proposal in this thread, just with two stars. {:, foo, bar}
{{ foo, bar }}
This is already valid syntax and attempts to store a set inside another set.
Your quoting is a bit weird, what about the {:, foo, bar} option? Your proposal `requests.post(url, **(data, params))` is also valid syntax, and has some support, so I was taking that as precedent. Both are guaranteed to fail at runtime under current semantics.
If at all, I'd prefer something like {:foo, :bar}.
I like that too, but it was originally proposed in the old thread so I'm guessing it's not popular enough.
But anyway this takes the proposal in a different direction.
I think a productive discussion about this topic necessarily has to at least keep dict literals in mind and consider them as a possible way to achieve the goal.
On Apr 16, 2020, at 12:05, Dominik Vilsmeier <dominik.vilsmeier@gmx.de> wrote:
I'm not sure if this is doable from the compiler perspective, but what about allowing tuples after `**` unpacking:
requests.post(url, **(data, params))
# similar to requests.post(url, data=data, params=params)
Is this actually a tuple? So the items can be any expression at all? Then what’s the keyword for, say, `requests.post(url, **(data, timeout*2))`? Also, could you use **tuple in other contexts where **unpacking is allowed, like dict displays, or only in calls? If we’re going to build on ** unpacking, maybe instead of coming up with special syntax that works everywhere **unpacking does, or that only works in some places **unpacking does, it would be simpler/cleaner to come up with special syntax just for dict displays—which automatically gives you the ability to **unpack in calls and anywhere else? Here’s a straw man. A one-liner grammar change is all you need, and restricting it to just identifiers is easy: key_datum ::= expression ":" expression | "**" or_expr | ":" identifier And the semantics of the new third form would be that the key is the name of the identifier as a string and the value is the value of the identifier as an expression. And here’s what it looks like: requests.post(url, **{:data, :params}) The obvious way to mix normal and magic keywords just works; you can of course write this: requests.post(url, headers=makeheaders(), **{:data, :params}) … but if headers has to come after data for some reason you can do this: requests.post(url, **{:data, "headers": makeheaders(), :params}) … or, if you prefer, this: requests.post(url, **{:data}, headers=makeheaders(), **{:params}) … because we’re just using the already-working, well-understood rules for **unpacking. And it can also be used outside of calls: params = {:name, :email, "phone": formatphone(phone)} request.post(url, json=params) … which also means you could refactor a really hairy call in an obvious way: kw = { :data, "headers": makeheaders(), :params } requests.post(url, **kw) Extending it beyond identifiers might be useful, but there are enough potential problems and questions (e.g., is the name of :spam.eggs “spam.eggs” or “eggs”?) that it would almost certainly be worth putting off to a future version after people have gotten experience with the basic feature. I’m not sure if I like this syntax, but I do like it better than the magic * separator syntax and the magic **-unpack-a-tuple syntax. (And I like the overall idea in this thread if someone can come up with good enough syntax.)
On Fri, Apr 17, 2020 at 3:03 AM <oliveira.rodrigo.m@gmail.com> wrote:
(Steven D'Aprano)
Do any other languages already have this feature?
JavaScript ES6 has similar feature:
```javascript x = 1 y = 2 do_something({ x, y }) ```
Very few other languages even *have* keyword arguments, so the nearest you'll get is something like this, where it's actually being wrapped up into a dict-like object. Here are a few examples gleaned by skimming a single file in one of my projects: r = requests.request(method, "https://api.twitch.tv/" + endpoint, params=params, data=data, headers={ "Accept": "application/vnd.twitchtv.v5+json", "Client-ID": config.CLIENT_ID, "Authorization": auth, }) # Recursion (the token mess is scheduled for change anyway): return query(endpoint, token="bearer" if token == "bearer" else "oauth", method=method, params=params, data=data, auto_refresh=False) # This pattern is VERY common. After doing the work, we pass a # whole bunch of info to the template renderer. return render_template("index.html", twitter=twitter, username=user["display_name"], channel=channel, channelid=channelid, error=error, setups=database.list_setups(channelid), sched_tz=sched_tz, schedule=schedule, sched_tweet=sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=tweets, ) # There are also cases where I *could* use kwargs, but I don't, # because it would be clunkier in syntax. This mixes positional # and keyword just so that I don't need channelid=channelid. database.update_timer_details(channelid, id, title=request.form["title"], delta=parse_time(request.form["delta"]), maxtime=parse_time(request.form["maxtime"]), styling=request.form.get("styling", ""), ) So yes, this definitely does happen in the real world. I think it would be nice to have a shorthand syntax, but it's not a killer feature. I'm +0.75 on adding this. ChrisA
On Thu, Apr 16, 2020 at 10:02 AM <oliveira.rodrigo.m@gmail.com> wrote:
@
Do any other languages already have this feature?
JavaScript ES6 has similar feature:
```javascript x = 1 y = 2 do_something({ x, y }) ```
Rust also has a similar feature when instantiating structs which are always keyword-based: https://doc.rust-lang.org/book/ch05-01-defining-structs.html#using-the-field... .
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/64SUXR... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Apr 16, 2020 at 05:02:03PM -0000, oliveira.rodrigo.m@gmail.com wrote:
@
Do any other languages already have this feature?
JavaScript ES6 has similar feature:
```javascript x = 1 y = 2 do_something({ x, y }) ```
That certainly makes your suggested syntax look comparatively nicer :-) I don't know what `{}` does in Javascript, but it looks like it is a set, or a list, or some other sequence or collection. It doesn't look like a mapping, since a mapping requires the elements come in key+value pairs. The syntax above doesn't look like the elements are pairs. Rust has the same issue (thanks Brett for the link): User { email: email, username: username, active: true, sign_in_count: 1, } defines a struct with fields email etc. With the field init shortcut we get: User { email, username, active, sign_in_count, } whioh again looks like a single entity `email` etc, not a pair (email key, email value). So I will grant you that a least your proposal looks like a pair: param = <blank> where the value of <blank> is implied. So better than the Javascript/Rust syntax. -- Steven
@Steven D'Aprano see that it doesn't actually have to look like a pair, it doesn't need to be one at all. Just like in: ```python def f(a): ... f(x) ``` `x` is implicitly assigned with `a`, i.e. this is `x = a` under the hood and there is no need to think of a key-value pair `x: a`. The same would go for the discussed assignment of keyword args using the `*` separator syntax (which seems more appealing to the ones in this thread than the original proposal): ```python def f(a, b): ... f(*, b, a) ``` The `*` would indicate that the following parameters are all keyword parameters, where the order doesn't matter and, unless explicitly defined, each passed parameter will be assigned to the argument with same name. So, under the hood you get `a = a` and `b = b`. In this syntax one can choose whether or not to explicitly defined the value of a keyword, so these statements would be equivalent: ```python f(positional, *, keyword0, keyword1=explicit, keyword2) f(positional, keyword0=keyword0, keyword1=explicit, keyword2=keyword2) ``` I'm against subverting dictionary literals declaration to support implicit pairs for the sake of readability. If that was the way we're going to implement this feature than I make your point mine, something like this just looks like a weird set not a dict: ```python kw = { , x, y, } f(**kw) ``` In Javascript ES6 they don't have sets built like python so `{}` always refers to objects being constructed. It does indeed support implicit key: value pairs, so in ES6 `{ a: a, b: x, c: c }` is equivalent to `{ a, b: x, c }`. This is okay for Javascript users because they would not thought it as sets and the only obvious assumption to make is that parameters are being implicitly assigned to members. This is not the case in Python so I would refrain from changing dictionary literals syntax.
On Apr 16, 2020, at 20:48, oliveira.rodrigo.m@gmail.com wrote:
In Javascript ES6 they don't have sets built like python so `{}` always refers to objects being constructed. It does indeed support implicit key: value pairs, so in ES6 `{ a: a, b: x, c: c }` is equivalent to `{ a, b: x, c }`. This is okay for Javascript users because they would not thought it as sets and the only obvious assumption to make is that parameters are being implicitly assigned to members. This is not the case in Python so I would refrain from changing dictionary literals syntax.
Obviously the exact same syntax as ES6 doesn’t work in Python, because it would be not just confusing but ineradicably ambiguous with sets. But I don’t see why that rules out the “bare colon” form that I and someone else apparently both proposed in separate sub threads of this thread: { :a, "b": x, :c } as shorthand for: { "a": a, "b": x, "c": c } There’s no problem for the parser. Make a trivial change to the grammar to add a `":" identifier` alternative to dict display items, and nothing becomes ambiguous. And I don’t think it would be confusing to a human reader. It can’t possibly be a set, because colons are what make a dict display not a set display, and there are colons. And I think extending dict display syntax is more powerful and less disruptive than extending call syntax. It means you can refactor the {…} in a **{…} if the call gets too unwieldy; it means calls and other places with ** unpacking remain consistent; etc.
I believe this is a different feature, non-exclusive to the one proposed here, that would also make it possible not to re-declare keywords. But implementing this change with the argument of making function calls less repetitive or verbose when having redundant named keywords and variables doesn't sell it to me. See, function calls would still suffer to be less redundant if we go with this: ```python def foo(a, b, **kwargs): c = ... bar(**{:a, :b, :c, d: kwargs["d"]}) # this just got worse ``` ```python def foo(a, b, **kwargs): c = ... # all parameters definition is away from the function call, not a fan # one can possibly overwrite some key on kwarg without knowing kwargs.update({:a, :b, :c}) bar(**kwargs) ``` ```python def foo(a, b, **kwargs): c = ... bar(**(kwargs | {:a, :b, :c})) # a little better but one can still overwrite some key on kwarg without knowing ``` Using a "magical" separator does the job and has little interactions with other syntaxes, using the `*` character seems better than just picking another random one (like we did with `/`). Comparing with all the above excerpts, this is still more appealing and clearer for me: ```python def foo(a, b, **kwargs): c = ... bar(*, a, b, c, **kwargs) # also, if any of `a`, `b` or `c` is in `kwargs` we get a proper error ``` Rodrigo Martins de Oliveira
I agree that introducing a new way for creating implicit dict literals only for the purpose of saving on keyword arguments seems too much of a change. Although it would be an elegant solution as it builds on already existing structures. And I don't think it hurts readability for function calls, since all your examples could be written as: bar(**{:a, :b, :c}, **kwargs) This doesn't have the problem of accidentally overriding keys as it is similar to: bar(a=a, b=b, c=c, **kwargs) On 17.04.20 06:41, oliveira.rodrigo.m@gmail.com wrote:
I believe this is a different feature, non-exclusive to the one proposed here, that would also make it possible not to re-declare keywords.
But implementing this change with the argument of making function calls less repetitive or verbose when having redundant named keywords and variables doesn't sell it to me.
See, function calls would still suffer to be less redundant if we go with this:
```python def foo(a, b, **kwargs): c = ... bar(**{:a, :b, :c, d: kwargs["d"]}) # this just got worse ```
```python def foo(a, b, **kwargs): c = ... # all parameters definition is away from the function call, not a fan # one can possibly overwrite some key on kwarg without knowing kwargs.update({:a, :b, :c}) bar(**kwargs) ```
```python def foo(a, b, **kwargs): c = ... bar(**(kwargs | {:a, :b, :c})) # a little better but one can still overwrite some key on kwarg without knowing ```
Using a "magical" separator does the job and has little interactions with other syntaxes, using the `*` character seems better than just picking another random one (like we did with `/`). Comparing with all the above excerpts, this is still more appealing and clearer for me:
```python def foo(a, b, **kwargs): c = ... bar(*, a, b, c, **kwargs) # also, if any of `a`, `b` or `c` is in `kwargs` we get a proper error ```
Rodrigo Martins de Oliveira _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ATCTNM... Code of Conduct: http://python.org/psf/codeofconduct/
On Apr 16, 2020, at 21:42, oliveira.rodrigo.m@gmail.com wrote:
I believe this is a different feature, non-exclusive to the one proposed here, that would also make it possible not to re-declare keywords.
Yes, it is a different feature from magic call syntax. And yes, theoretically they’re non-exclusive and we could add both, but practically it would eliminate the need for the magic call syntax feature (as well as having other uses), so we almost surely wouldn’t add both. That’s the point of describing it in this thread: it solves the same problem in a different, and I think better, way.
But implementing this change with the argument of making function calls less repetitive or verbose when having redundant named keywords and variables doesn't sell it to me.
See, function calls would still suffer to be less redundant if we go with this:
```python def foo(a, b, **kwargs): c = ... bar(**{:a, :b, :c, d: kwargs["d"]}) # this just got worse ```
Nobody’s forcing you to put all keywords into the dict. Today, you can mix **dict and normal keywords in a function call (and in a dict literal, and anywhere else **unpacking is allowed), and that isn’t going to go away, so you’d just write: bar(**(:a, :b, :c), d=kwargs['d']) Note that this is different from any magic call syntax. If you want to allow normal keyword arguments to mix with magic ones in a magic call syntax, you have to specify some additional syntax for doing that. With my proposal, you don’t, because the existing call syntax and **unpacking syntax already handle that today, and I’m not changing them.
```python def foo(a, b, **kwargs): c = ... # all parameters definition is away from the function call, not a fan # one can possibly overwrite some key on kwarg without knowing kwargs.update({:a, :b, :c}) bar(**kwargs) ```
Sure, if you want to write that you can, or you can also write this: bar(**kwargs, **{:a, :b, :c}) And again, this is exactly the way things already work. If you have two separate dicts of keyword arguments, you can ** them both, or you can merge them and ** the result, and there are advantages and disadvantages of each that people already choose between, and nothing about that changes with this proposal. If you’re not a fan of one way of doing it, you already presumably use the other, and that won’t change. The only thing that changes is that your dict display, wherever you choose to put it, gets shorter.
```python def foo(a, b, **kwargs): c = ... bar(**(kwargs | {:a, :b, :c})) # a little better but one can still overwrite some key on kwarg without knowing ```
Yes, 3.8 gives you this third option, so people already use it when appropriate today and hopefully don’t use it when it isn’t. And that also won’t change just because the second dict becomes easier to construct.
Using a "magical" separator does the job and has little interactions with other syntaxes,
And adding magic to dict displays also does the job and also has little interaction with other syntaxes. In fact, it has even less interaction with other syntaxes, because the dict display is completely independent of the call syntax and therefore can be refactored out of the call, like any other dict. bar(**{:a, :b, :c}) kw = {:a, :b, :c} bar(**kw) … or used anywhere else a dict can be **’d, like another dict display. Or even in a place where there’s no ** going on: userlist.append({:name, :email, :phone}) json.dump(userlistfile, userlist)
using the `*` character seems better than just picking another random one (like we did with `/`). Comparing with all the above excerpts, this is still more appealing and clearer for me:
```python def foo(a, b, **kwargs): c = ... bar(*, a, b, c, **kwargs) # also, if any of `a`, `b` or `c` is in `kwargs` we get a proper error ```
Well, this doesn’t do the same thing as your earlier examples. And if you want this with my syntax: bar(**{:a, :b, :c}, **kwargs) … you get that exact same benefit of an error if kwargs has an a in it. And you even get that benefit if you refactor the dict into a temporary variable because the call got too long or complex to read, or because you needed that dict for some other purpose (maybe just a debug print), or whatever. Because, again, my proposal changes nothing about call syntax or ** unpacking, and therefore all the things that already work—like detecting repeated keywords from separate unpackings—continue to work. And that’s the advantage of this proposal in a nutshell—I didn’t think about repeated keyword detection until you brought it up, so I didn’t work out a way to make it work, but that’s fine. Because I’m not adding anything new to calls, everything that needs to be thought through already has been thought through, designed, implemented, taught, and used in the field for years, as part of the existing call and **unpacking syntax and semantics that already work. The magic * has few interactions with the rest of call, but changing dict displays has zero interactions, which is even better than few.
@AndrewBarnert yes I should have thought it better that there is no need to use `kwargs` forcibly. My bad. I am a little more convinced of this being a solid candidate solution to the problem. Thanks for talking me through! Rodrigo Martins de Oliveira
On Thu, Apr 16, 2020 at 09:21:05PM -0700, Andrew Barnert via Python-ideas wrote:
But I don’t see why that rules out the “bare colon” form that I and someone else apparently both proposed in separate sub threads of this thread:
{ :a, "b": x, :c }
as shorthand for:
{ "a": a, "b": x, "c": c }
I did a double-take reading that, because I visually parsed it as: { :a, "b": x, :c } and couldn't work out what was going on. After saving this draft, closing the email, then reopening it, I read the proposed dict the same way. So I don't think it was just a momentary glitch. I think that, as little as I like the original proposal and am not really convinced it is necessary, I think that it is better to have the explicit token (the key/parameter name) on the left, and the implicit token (blank) on the right: key= I suspect it may be because we read left-to-right in English and Python, so having the implicit blank come first is mentally like running into a pothole at high speed :-) Perhaps those who read right-to-left languages may have fewer problems with it. Or maybe it's just me.
There’s no problem for the parser. Make a trivial change to the grammar to add a `":" identifier` alternative to dict display items, and nothing becomes ambiguous.
Except to the human reader, which is the only ambiguity that *really* matter when you get down to it. If human readers are going to parse it differently from your intention, as I did, then this is going to hurt readability far in excess of the benefit in saving a few characters typed. -- Steven
On Apr 17, 2020, at 01:58, Steven D'Aprano <steve@pearwood.info> wrote:
On Thu, Apr 16, 2020 at 09:21:05PM -0700, Andrew Barnert via Python-ideas wrote:
But I don’t see why that rules out the “bare colon” form that I and someone else apparently both proposed in separate sub threads of this thread: { :a, "b": x, :c } as shorthand for: { "a": a, "b": x, "c": c }
I did a double-take reading that, because I visually parsed it as:
{ :a, "b": x, :c }
and couldn't work out what was going on.
After saving this draft, closing the email, then reopening it, I read the proposed dict the same way. So I don't think it was just a momentary glitch.
I honestly think, as you suggested at the end, that this may be just you. You’ve had similar reactions to other syntax that nobody else replicated, and I think that’s happening again here. At least one other person suggested the exact same syntax independently in this set of threads. Multiple people have responded to it, positively or negatively, without any of them having any trouble reading it as intended. As one of the other Steves pointed out, everyone who writes Lisp deals with basically the same syntax (albeit for an only vaguely related, pretty different, feature) and nobody has a problem with it there, and it’s not because most Lisp programmers are native speakers of Arabic or other RTL languages. And a wide variety of languages with very different syntax from Lisp (Ruby is closer to Python than to Lisp; Smalltalk is radically different from both Lisp and Python; etc.) borrowed its symbol syntax (sometimes changing the prefix character), and nobody has any trouble reading it in those languages either. And there’s similar syntax for all kinds of less similar things in all kinds of languages. Because really, this is just prefix syntax, not infix syntax with a piece missing, and people don’t have any problem with prefixes. If anything, prefixes are more common than suffixes in programming languages and math notation and other things designed by western people with LTR scripts (and prepositions and subject-first normal sentence order and whatever else might be relevant). I don’t disbelieve that you parsed this weirdly. Some people do read things idiosyncratically. When C had a proposal for designated initializer syntax, there was one person on the committee who just couldn’t see this: struct Spam spam = { .eggs = 2, .cheese = 3, .ham = 1 }; … without trying to figure out how you can take the `cheese` member of a `2,`. Even when it was split onto multiple lines: struct Spam slam = { .eggs = 2, .cheese = 3, .ham = 1 }; … his brain still insisted that the dot must be an infix operator, not a prefix one. Pointing out that you can parse -4 even though that’s normally an infix operator doesn’t help—understanding something intellectually doesn’t rewrite your mental parse rules. If most C programmers read it the same way as him, the feature would be unusable. Nobody would argue “we can teach the whole world to get used to it, and who cares if C is unreadable until we do?” But it turned out that nobody else had any problem reading it as intended, the feature was added to C11, and people love it. (The Python extending docs were even rewritten to strongly encourage people to use it.) And the same is true here. If a lot of people can’t see :spam as meaning the value of spam with the name of spam, because their brain instead keeps looking for something on the left to attach it to, then the feature is unusable and I’d drop it. But if it’s just one guy with idiosyncratic internal parse rules, and everyone else reads it without even stumbling, I don’t think that should stop an otherwise useful proposal. And I don’t see anyone else stumbling. (Of course there are other problems with it, the biggest one being that I’m not sure any change is needed at all. And there are legitimate differences on whether, if a change is needed, it should be a change to call syntax or **unpacking syntax or dict display syntax. And so on. I’m not sure I’m +1 on the proposal myself.)
I think that, as little as I like the original proposal and am not really convinced it is necessary, I think that it is better to have the explicit token (the key/parameter name) on the left, and the implicit token (blank) on the right:
key=
I suspect it may be because we read left-to-right in English and Python, so having the implicit blank come first is mentally like running into a pothole at high speed :-)
What examples can you think of—in English, Python, other popular languages, math notation, whatever—where there’s an infix-operator-like thing and the right token is elided and inferred implicitly? I’m sure there are some, but nothing comes to mind immediately. Meanwhile, I can think of lots of examples of the opposite, the token on the left being elided and inferred. You can read -2 and infer 2 less than 0, but not 2- to infer 0 less than 2. There’s the C designated initializer syntax, and BASIC with statements, and a variety of similar features across other languages, where you can infer what’s to the left of the dot. In fact, in Python, you can `import .spam` with “this package” implied as the thing to the left of the dot (and you can’t `import spam.` with “all the publics” or any other meaning; you have to rewrite the whole statement as `from spam import *` if you want that), and nobody has any problem inferring the intended meaning.
There’s no problem for the parser. Make a trivial change to the grammar to add a `":" identifier` alternative to dict display items, and nothing becomes ambiguous.
Except to the human reader, which is the only ambiguity that *really* matter when you get down to it.
Why do you do this? You snipped out the very next sentence about human readers, just to take this out of context so you can imply that human readers weren’t taken into account just so you can disagree with not taking human readers into account. And you don’t even really believe this. Obviously *both* ambiguities matter. Something that reads wonderfully but nobody could write a parser for would not be any more useful than something that‘s easy to parse but no human can understand it. There are already enough issues to discuss that fabricating issues that aren’t there just to be contrary to them really isn’t necessary.
On Fri, Apr 17, 2020 at 2:22 PM Andrew Barnert via Python-ideas < python-ideas@python.org> wrote:
What examples can you think of—in English, Python, other popular languages, math notation, whatever—where there’s an infix-operator-like thing and the right token is elided and inferred implicitly? I’m sure there are some, but nothing comes to mind immediately.
I can't think of any either... But there are probably pro- and con- arguments. :-) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Apr 17, 2020, at 11:30, David Mertz <mertz@gnosis.cx> wrote:
I can't think of any either...
You know, an ellipsis is about the farthest thing from an *unmarked* elision that you can get [without brackets and reinsertion of a more verbose version of the thing you wanted to elide -ed]. But the hyphen example works. I said I was sure there must be examples. And anyway, in natural human language, elisions are usually best thought of as from a tree node rather than from a linear order. You wouldn’t really say that pro drop is “prefix elision” just because in its Japanese version the subject is the most commonly dropped pronoun and subjects often but not obligatory come first so the missing phrase would have been on the left if it weren’t missing. So, this wasn’t a great point of mine in the first place. But I think the rest of the argument stands. Multiple people in this thread have criticized the proposal without any of them stumbling on what { :spam, :eggs } is intended to mean, at least one person noticed that it’s similar to Lisp symbols, one other person proposed the exact same thing, etc., and as far as I know all of those people are native (at least certainly fluent) speakers of a LTR-written language (as were the designers of Lisp, most of the C committee, etc.). If Steve has a problem reading it, I think it’s probably just him, but of course if I’m wrong I hope (and expect) others will chime in.
On 17/04/2020 19:21, Andrew Barnert via Python-ideas wrote:
On Apr 17, 2020, at 01:58, Steven D'Aprano<steve@pearwood.info> wrote:
On Thu, Apr 16, 2020 at 09:21:05PM -0700, Andrew Barnert via Python-ideas wrote:
But I don’t see why that rules out the “bare colon” form that I and someone else apparently both proposed in separate sub threads of this thread: { :a, "b": x, :c } as shorthand for: { "a": a, "b": x, "c": c }
I did a double-take reading that, because I visually parsed it as:
{ :a, "b": x, :c }
and couldn't work out what was going on.
After saving this draft, closing the email, then reopening it, I read the proposed dict the same way. So I don't think it was just a momentary glitch. I honestly think, as you suggested at the end, that this may be just you. You’ve had similar reactions to other syntax that nobody else replicated, and I think that’s happening again here.
It's not just Steven. After dusting my monitor to remove flyspecs, I still couldn't find a natural way of reading that example. I didn't visually parse it quite the same, but the excess of punctuation still encourage me to completely miss the '"b": x' part being a unit. -- Rhodri James *-* Kynesim Ltd
I also find the example with :keyword a bit jarring at first glance, so I propose a double colon to alleviate the problem if we go in that direction. Compare: { :a, "b": x, :c } { ::a, "b": x, ::c } On Fri, Apr 17, 2020 at 10:05 PM Rhodri James <rhodri@kynesim.co.uk> wrote:
On 17/04/2020 19:21, Andrew Barnert via Python-ideas wrote:
On Apr 17, 2020, at 01:58, Steven D'Aprano<steve@pearwood.info> wrote:
On Thu, Apr 16, 2020 at 09:21:05PM -0700, Andrew Barnert via Python-ideas wrote:
But I don’t see why that rules out the “bare colon” form that I and someone else apparently both proposed in separate sub threads of this thread: { :a, "b": x, :c } as shorthand for: { "a": a, "b": x, "c": c }
I did a double-take reading that, because I visually parsed it as:
{ :a, "b": x, :c }
and couldn't work out what was going on.
After saving this draft, closing the email, then reopening it, I read the proposed dict the same way. So I don't think it was just a momentary glitch. I honestly think, as you suggested at the end, that this may be just you. You’ve had similar reactions to other syntax that nobody else replicated, and I think that’s happening again here.
It's not just Steven. After dusting my monitor to remove flyspecs, I still couldn't find a natural way of reading that example. I didn't visually parse it quite the same, but the excess of punctuation still encourage me to completely miss the '"b": x' part being a unit.
-- Rhodri James *-* Kynesim Ltd _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7UU4MD... Code of Conduct: http://python.org/psf/codeofconduct/
On Apr 17, 2020, at 13:39, Alex Hall <alex.mojaki@gmail.com> wrote:
I also find the example with :keyword a bit jarring at first glance, so I propose a double colon to alleviate the problem if we go in that direction. Compare:
{ :a, "b": x, :c }
{ ::a, "b": x, ::c }
I can see the point of explicitly calling out the syntax like this, and, while it doesn’t really feel necessary to me, it doesn’t feel at all onerous either. So if there are people who feel strongly against the single colon and like the double colon, I’m fine with the double colon. (I’m still not sure we need to do anything. And I don’t love extending dict displays, and think that if we do need a solution, maybe there’s a better one lurking out there that nobody has hit on. But I definitely dislike extending dict displays less than extending some but not other uses of **unpacking, or having a mode switch in the middle of call syntax, or anything else proposed so far.)
On 17.04.20 23:18, Andrew Barnert via Python-ideas wrote:
On Apr 17, 2020, at 13:39, Alex Hall <alex.mojaki@gmail.com> wrote:
I also find the example with :keyword a bit jarring at first glance, so I propose a double colon to alleviate the problem if we go in that direction. Compare:
{ :a, "b": x, :c } { ::a, "b": x, ::c }
I can see the point of explicitly calling out the syntax like this, and, while it doesn’t really feel necessary to me, it doesn’t feel at all onerous either. So if there are people who feel strongly against the single colon and like the double colon, I’m fine with the double colon.
(I’m still not sure we need to do anything. And I don’t love extending dict displays, and think that if we do need a solution, maybe there’s a better one lurking out there that nobody has hit on. But I definitely dislike extending dict displays less than extending some but not other uses of **unpacking, or having a mode switch in the middle of call syntax, or anything else proposed so far.)
It's been a lot of emails, so I probably missed it, but could you summarize your objections regarding the idea using `**` as a "mode switch". That is func(pos, **, foo, bar) I think it aligns nicely with the `*` in function definitions. `def func(a, *, b, c)` or `def func(a, *args, b, c)` means anything following the `*` part is a keyword-only parameter. Similarly we have today `func(a, **kwargs, b=b, c=c)`, i.e. anything that follows the `**` part has to be provided as a keyword argument. Since this eliminates all ambiguity, why not leave out the parameter names: ` func(a, **kwargs, b, c)`; and since unpacking is not necessarily required, we could have `func(a, **, b, c)`.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DS4HBK... Code of Conduct: http://python.org/psf/codeofconduct/
Andrew Barnert via Python-ideas writes:
On Apr 17, 2020, at 13:39, Alex Hall <alex.mojaki@gmail.com> wrote:
{ :a, "b": x, :c }
{ ::a, "b": x, ::c }
And now Lisp bites me, because '::a' means "the identifier 'a' in the default namespace", which doesn't make sense in Python, because the (similar) namespacing construct would be class attributes, while Python has a more nuanced way of handling the current namespace. I quite dislike the whole idea of abbreviating associations between identifiers spelled the same way in different namespaces, both in dict displays and in keyword arguments. It's really "grit on screen" level notation. I'd be happy to put aside my "dislike" if statistics on the frequency of such duplicate names across namespaces were presented (and they're compellingly large). I myself am rarely annoyed by this issue, with the single exception of "self.foo = foo" in __init__() defs (which can't be handled by these notations). It's usually the case for me that the LHS of a keyword argument refers to the role of the identifier in the function's computation, while the RHS refers to the application domain, so I'd rarely get to use these optimizations anyway. I guess I occasionally see dictionary displays of the form "{'name' : name, 'address' : address, 'email' : email}" in my code, but even that is pretty rare nowadays, as it generally gets spelled fields = ('name', 'address', 'email') [{k: v for k, v in zip(fields, next_tuple)} for next_tuple in input] or something like that, which is less annoying to maintain when I inevitably add fields. I used to have little dictbuilder functions def add_record(db, name, address, email): db.append({'name' : name, 'address' : address, 'email' : email}) all over the place, but then I learned about dict comprehensions and zip got promoted to a builtin. Steve
And now Lisp bites me, because '::a' means ...
And a single colon also means something else in Lisp. Does it matter much what that notation means in a different language? Python will struggle to evolve if it can't conflict with other languages. I myself am rarely annoyed by this issue, with
the single exception of "self.foo = foo" in __init__() defs (which can't be handled by these notations).
It can be handled: ``` self.__dict__.update(**, foo, bar, spam) ``` or ``` self.__dict__.update({::foo, ::bar, ::spam}) ``` Alternatively, this is a utility function that I use sometimes: ``` def setattrs(obj, **kwargs): """ >>> data = SimpleNamespace() >>> setattrs(data, a=1, b=2) # doctest:+ELLIPSIS namespace(a=1, b=2) """ for key, value in kwargs.items(): setattr(obj, key, value) return obj ``` And here is some actual code of mine using it: ``` setattrs(cls, text=text, program=program, messages=messages, hints=hints) ``` which could instead be written ``` setattrs(cls, **, text, program, messages, hints) ```
Alex Hall writes:
And now Lisp bites me, because '::a' means ...
And a single colon also means something else in Lisp.
Yeah, and I already pointed that out myself in this subthread. I don't like these notations, and the conflict with Lisp (which I read a lot of) is part of why. My taste, or the existence of Lisp, doesn't rule the issue, but it's input.
Does it matter much what that notation means in a different language?
Of course it might. That's why we use the usual arithmetic operators for their usual meanings. :: is not a universally used notation, but if we can find an alternative that's even less used, why not use that alternative?
Python will struggle to evolve if it can't conflict with other languages.
Strawman. Nobody said "can't". The question is "better".
I myself am rarely annoyed by this issue, with the single exception of "self.foo = foo" in __init__() defs (which can't be handled by these notations).
It can be handled:
``` self.__dict__.update(**, foo, bar, spam) ```
I hope you're kidding.
And here is some actual code of mine using it:
``` setattrs(cls, text=text, program=program, messages=messages, hints=hints) ```
I once wrote code like that as needed; my point was that I now rarely need it. There are already more concise, clearer, less ugly alternatives available, at least for the cases where *I* wrote code like that. I can't speak for your needs, only guess. But these examples of function *calls* don't give me enough information to decide for myself whether I think there's a need to write them, much less show you how I would write the program without them. You can continue to write such code if you want to. The question is, should Python add new syntax that a lot of people dislike so that you can write code in a style that may not be necessary any more? It's not possible to decide whether it's necessary without seeing the rest of the program. (Of course "necessary" is a value judgment, not an objective and absolute fact. Still need to see much more of the program to decide.) Steve
On Sun, Apr 19, 2020 at 7:58 AM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Alex Hall writes:
And now Lisp bites me, because '::a' means ...
And a single colon also means something else in Lisp.
Yeah, and I already pointed that out myself in this subthread. I don't like these notations, and the conflict with Lisp (which I read a lot of) is part of why. My taste, or the existence of Lisp, doesn't rule the issue, but it's input.
Does it matter much what that notation means in a different language?
Of course it might. That's why we use the usual arithmetic operators for their usual meanings. :: is not a universally used notation, but if we can find an alternative that's even less used, why not use that alternative?
Python will struggle to evolve if it can't conflict with other languages.
Strawman. Nobody said "can't". The question is "better".
OK, that's fair. What about `{foo::}`?
I myself am rarely annoyed by this issue, with
the single exception of "self.foo = foo" in __init__() defs (which can't be handled by these notations).
It can be handled:
``` self.__dict__.update(**, foo, bar, spam) ```
I hope you're kidding.
And here is some actual code of mine using it:
``` setattrs(cls, text=text, program=program, messages=messages, hints=hints) ```
I once wrote code like that as needed; my point was that I now rarely need it. There are already more concise, clearer, less ugly alternatives available, at least for the cases where *I* wrote code like that.
I don't understand what's happening. You clearly said you are still annoyed by `self.foo = foo`, but that this proposal doesn't address that. I pointed out that the proposal does address it. There's one way which is slightly ugly but requires no setup. There's another way that's prettier and just requires introducing one simple function. I would personally be quite happy if I could replace: ``` class A: def __init__(self, foo, bar, spam): self.foo = foo self.spam = spam self.bar = bar ``` with something like: ``` class A: def __init__(self, foo, bar, spam): setattrs(self, **, foo, bar, spam) ``` Wouldn't you? Of course there's also dataclasses, but there's plenty of places where those don't apply so nicely. (the name 'setattrs' may not be the best for the purpose) I can't speak for your needs, only guess. But these
examples of function *calls* don't give me enough information to decide for myself whether I think there's a need to write them, much less show you how I would write the program without them.
I provided a script to help find such cases. Let's make this more concrete. Here is the script again, slightly improved: ``` import ast import linecache import sys from collections import Counter from pathlib import Path root = Path(sys.argv[1]) def main(): counts = Counter() for path in root.rglob("**/*.py"): if 'test' in str(path): continue try: source = path.read_text() tree = ast.parse(source) except (SyntaxError, UnicodeDecodeError): continue for node in ast.walk(tree): if isinstance(node, ast.Call): def is_same_name(keyword: ast.keyword): return ( isinstance(keyword.value, ast.Name) and keyword.value.id == keyword.arg ) args = node.keywords elif isinstance(node, ast.Dict): def is_same_name(pair): key, value = pair return ( isinstance(value, ast.Name) and isinstance(key, (ast.Constant, ast.Str)) and value.id == key.s ) args = zip(node.keys, node.values) else: continue count = sum(map(is_same_name, args)) if count: counts[count] += 1 if count < 7: continue print(f'File "{path}", line {node.lineno}') for lineno in range(node.lineno, node.end_lineno + 1): print(linecache.getline(str(path), lineno), end="") print() print("Counts:", counts) main() ``` I ran this on master of the cpython repo. Here is the output with all the cases with at least 7 same-named values: ``` File "setup.py", line 2232 self.add(Extension('_decimal', include_dirs=include_dirs, libraries=libraries, define_macros=define_macros, undef_macros=undef_macros, extra_compile_args=extra_compile_args, sources=sources, depends=depends)) File "Tools/clinic/clinic.py", line 1083 d = { "docstring_prototype" : docstring_prototype, "docstring_definition" : docstring_definition, "impl_prototype" : impl_prototype, "methoddef_define" : methoddef_define, "parser_prototype" : parser_prototype, "parser_definition" : parser_definition, "impl_definition" : impl_definition, "cpp_if" : cpp_if, "cpp_endif" : cpp_endif, "methoddef_ifndef" : methoddef_ifndef, } File "Lib/shutil.py", line 554 return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, ignore=ignore, copy_function=copy_function, ignore_dangling_symlinks=ignore_dangling_symlinks, dirs_exist_ok=dirs_exist_ok) File "Lib/tempfile.py", line 642 self._TemporaryFileArgs = {'mode': mode, 'buffering': buffering, 'suffix': suffix, 'prefix': prefix, 'encoding': encoding, 'newline': newline, 'dir': dir, 'errors': errors} File "Lib/compileall.py", line 99 results = executor.map(partial(compile_file, ddir=ddir, force=force, rx=rx, quiet=quiet, legacy=legacy, optimize=optimize, invalidation_mode=invalidation_mode, stripdir=stripdir, prependdir=prependdir, limit_sl_dest=limit_sl_dest), File "Lib/argparse.py", line 879 super().__init__( option_strings=_option_strings, dest=dest, nargs=0, default=default, type=type, choices=choices, required=required, help=help, metavar=metavar) File "Lib/argparse.py", line 917 super(_StoreAction, self).__init__( option_strings=option_strings, dest=dest, nargs=nargs, const=const, default=default, type=type, choices=choices, required=required, help=help, metavar=metavar) File "Lib/argparse.py", line 1009 super(_AppendAction, self).__init__( option_strings=option_strings, dest=dest, nargs=nargs, const=const, default=default, type=type, choices=choices, required=required, help=help, metavar=metavar) File "Lib/argparse.py", line 1038 super(_AppendConstAction, self).__init__( option_strings=option_strings, dest=dest, nargs=0, const=const, default=default, required=required, help=help, metavar=metavar) File "Lib/json/__init__.py", line 234 return cls( skipkeys=skipkeys, ensure_ascii=ensure_ascii, check_circular=check_circular, allow_nan=allow_nan, indent=indent, separators=separators, default=default, sort_keys=sort_keys, **kw).encode(obj) File "Lib/json/__init__.py", line 173 iterable = cls(skipkeys=skipkeys, ensure_ascii=ensure_ascii, check_circular=check_circular, allow_nan=allow_nan, indent=indent, separators=separators, default=default, sort_keys=sort_keys, **kw).iterencode(obj) File "Lib/asyncio/base_events.py", line 1254 opts = dict(local_addr=local_addr, remote_addr=remote_addr, family=family, proto=proto, flags=flags, reuse_address=reuse_address, reuse_port=reuse_port, allow_broadcast=allow_broadcast) File "Lib/logging/__init__.py", line 108 _nameToLevel = { 'CRITICAL': CRITICAL, 'FATAL': FATAL, 'ERROR': ERROR, 'WARN': WARNING, 'WARNING': WARNING, 'INFO': INFO, 'DEBUG': DEBUG, 'NOTSET': NOTSET, } ``` Please make a PR showing how you would refactor some of these.
Alex Hall writes:
OK, that's fair. What about `{foo::}`?
I don't like any of them. People who are going to use the syntax should choose it, not those of us who won't.
I would personally be quite happy if I could replace:
``` class A: def __init__(self, foo, bar, spam): self.foo = foo self.spam = spam self.bar = bar ```
with something like:
``` class A: def __init__(self, foo, bar, spam): setattrs(self, **, foo, bar, spam) ```
Wouldn't you?
No, I doubt I would use that. It's not so much the repetition of identifiers alone, it's the combination with the repetition of self: two dimensional redundancy! Probably a holdover from the implicit 'this' in C++. Although, come to think of it, in C++ to assign to a member with the same name as a formal argument you'd have to explicitly use 'this'. I apologize for wasting your time on that comment. It turns out to be incoherent and I shouldn't have mentioned it.
Please make a PR showing how you would refactor some of these.
Again, you must be joking. You need to show that this is useful enough to be worth a syntax change, which is a high bar. A dozen examples in the whole stdlib? I'm not going to do hours of work understanding those modules to refactor perfectly good code. I will outline what I'd look at, though. In general the vertically formatted examples look fine to me, and I see no reason to refactor them. I'd reformat the others vertically. In code not intended to be maintained, I'd likely use positional arguments and just copy the prototypes (deleting defaults and type annotations). The two dicts look fine to me as they are. But it's easy to say that now I would design them using Enums. For example: class LogLevel(IntEnum): CRITICAL = 6 FATAL = 5 ERROR = 4 WARN = 3 WARNING = 3 INFO = 2 DEBUG = 1 NOTSET = 0 # I wouldn't bother with this alias in my own code. _nameToLevel = LogLevel.__members__ There are four calls to super (all in argparse), which seems to be due to heavy use of mixins, and two calls to cls (in json/__init__.py). Within the stdlib, the calls to super and to cls seem to be idiosyncratic to the authors of those particular modules, but if the plethora of same name arguments is typical of these techniques, that would be some support for a syntax change. I know those techniques are commonly used, I just don't have use for them myself. That leaves four various other examples in the whole stdlib, not very many. Although I'm sure you'd find more with a lower limit, I doubt they're as convincing as the ones with many same name arguments. We'll see what the senior committers have to say, but I think you have an uphill battle on your hands. Steve
On Sun, Apr 19, 2020 at 7:06 PM Stephen J. Turnbull < turnbull.stephen.fw@u.tsukuba.ac.jp> wrote:
Alex Hall writes:
OK, that's fair. What about `{foo::}`?
I don't like any of them. People who are going to use the syntax should choose it, not those of us who won't.
First of all, if this proposal went through, even if you didn't write code using it, you'd still have to read it. Secondly these discussions are often not just about how we personally would be affected by a proposal, but others too, especially beginners who haven't seen a particular syntax beforehand. So you don't have to tell us which one is your favourite, but I think it's fair to ask if you have specific objections to a specific syntax, or if one syntax (e.g. `foo::`) resolves the objection you had against another syntax (`::foo`). Of course, if you don't want to spend energy or time explaining, that's understandable too.
I would personally be quite happy if I could replace:
``` class A: def __init__(self, foo, bar, spam): self.foo = foo self.spam = spam self.bar = bar ```
with something like:
``` class A: def __init__(self, foo, bar, spam): setattrs(self, **, foo, bar, spam) ```
Wouldn't you?
No, I doubt I would use that. It's not so much the repetition of identifiers alone, it's the combination with the repetition of self: two dimensional redundancy!
But...this solves exactly that! self went from three appearances to one (not counting the signature). I don't get what you're saying.
Please make a PR showing how you would refactor some of these.
Again, you must be joking. You need to show that this is useful enough to be worth a syntax change, which is a high bar. A dozen examples in the whole stdlib? I'm not going to do hours of work understanding those modules to refactor perfectly good code.
Well, one of your original claims was that code that has these same-names generally could have been written better to avoid it. When we tried to provide counterexamples, you said: I can't speak for your needs, only guess. But these
examples of function *calls* don't give me enough information to decide for myself whether I think there's a need to write them, much less show you how I would write the program without them.
So now I've given you examples where you can see the context and your response is that it's "perfectly good code" that isn't worth the effort to refactor. Even in your descriptions of what you 'would' do it sounds like you'd pretty much leave them as they are in most cases. Therefore I think it's safe to say that we're often stuck with this pattern, and we have reason to try to make the best of it.
*In code not intended to be maintained, I'd likely use positional arguments and just copy the prototypes (deleting defaults and type annotations).*
I'd like to highlight this in relation to my other messages about positional arguments. The current syntax encourages dangerous judgements like "this code probably doesn't need to be maintained, I'll take the easy route". Also I'm hearing that the redundancy bothers you enough that you'd like to avoid it where you can, even at a cost. That suggests that there is a problem worth fixing.
That leaves four various other examples in the whole stdlib, not very many. Although I'm sure you'd find more with a lower limit, I doubt they're as convincing as the ones with many same name arguments. We'll see what the senior committers have to say, but I think you have an uphill battle on your hands.
Well, here's some more data. If I take all the dicts and calls with same-names and put the number of same-names into a Counter, here's what I get for various projects: CPython: Counter({1: 631, 2: 174, 3: 58, 4: 27, 5: 12, 6: 9, 8: 5, 7: 4, 10: 4}) pandas: Counter({1: 2785, 2: 864, 3: 363, 4: 165, 5: 86, 6: 55, 7: 48, 9: 23, 8: 20, 10: 11, 11: 8, 12: 7, 13: 7, 15: 4, 16: 2, 14: 2, 17: 2, 18: 2, 25: 1, 48: 1, 22: 1, 19: 1}) IPython: Counter({1: 1184, 2: 284, 3: 85, 4: 44, 5: 28, 6: 12, 7: 9, 11: 5, 10: 2, 8: 2, 14: 2}) scikit-learn: Counter({1: 817, 2: 188, 3: 81, 4: 30, 5: 27, 7: 15, 6: 14, 10: 10, 8: 10, 11: 7, 13: 5, 9: 5, 14: 4, 15: 4, 12: 3, 23: 3, 17: 3, 19: 3, 22: 1, 16: 1, 21: 1}) Django: Counter({1: 562, 2: 130, 3: 48, 5: 13, 4: 12, 7: 3, 6: 3, 10: 2, 9: 2, 18: 1, 12: 1, 8: 1}) matplotlib: Counter({1: 783, 2: 181, 3: 76, 4: 34, 6: 22, 5: 10, 7: 10, 9: 7, 10: 5, 13: 4, 12: 2, 15: 2, 8: 2, 25: 1, 16: 1, 14: 1, 17: 1}) Regarding the large number of calls with just a few same-names: yes, there is less benefit to changing them to a new syntax, but there's similarly less cost to doing so.
Guys, is it really worth saving a few hits on the auto-complete key by adding even more mysterious twists to the existing Python function call syntax ? The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python: funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] func_body_suite parameters: '(' [typedargslist] ')' # The following definition for typedarglist is equivalent to this set of rules: # # arguments = argument (',' [TYPE_COMMENT] argument)* # argument = tfpdef ['=' test] # kwargs = '**' tfpdef [','] [TYPE_COMMENT] # args = '*' [tfpdef] # kwonly_kwargs = (',' [TYPE_COMMENT] argument)* (TYPE_COMMENT | [',' [TYPE_COMMENT] [kwargs]]) # args_kwonly_kwargs = args kwonly_kwargs | kwargs # poskeyword_args_kwonly_kwargs = arguments ( TYPE_COMMENT | [',' [TYPE_COMMENT] [args_kwonly_kwargs]]) # typedargslist_no_posonly = poskeyword_args_kwonly_kwargs | args_kwonly_kwargs # typedarglist = (arguments ',' [TYPE_COMMENT] '/' [',' [[TYPE_COMMENT] typedargslist_no_posonly]])|(typedargslist_no_posonly)" # # It needs to be fully expanded to allow our LL(1) parser to work on it. typedargslist: ( (tfpdef ['=' test] (',' [TYPE_COMMENT] tfpdef ['=' test])* ',' [TYPE_COMMENT] '/' [',' [ [TYPE_COMMENT] tfpdef ['=' test] ( ',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] [ '*' [tfpdef] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] ['**' tfpdef [','] [TYPE_COMMENT]]]) | '**' tfpdef [','] [TYPE_COMMENT]]]) | '*' [tfpdef] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] ['**' tfpdef [','] [TYPE_COMMENT]]]) | '**' tfpdef [','] [TYPE_COMMENT]]] ) | (tfpdef ['=' test] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] [ '*' [tfpdef] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] ['**' tfpdef [','] [TYPE_COMMENT]]]) | '**' tfpdef [','] [TYPE_COMMENT]]]) | '*' [tfpdef] (',' [TYPE_COMMENT] tfpdef ['=' test])* (TYPE_COMMENT | [',' [TYPE_COMMENT] ['**' tfpdef [','] [TYPE_COMMENT]]]) | '**' tfpdef [','] [TYPE_COMMENT]) ) tfpdef: NAME [':' test] # The following definition for varargslist is equivalent to this set of rules: # # arguments = argument (',' argument )* # argument = vfpdef ['=' test] # kwargs = '**' vfpdef [','] # args = '*' [vfpdef] # kwonly_kwargs = (',' argument )* [',' [kwargs]] # args_kwonly_kwargs = args kwonly_kwargs | kwargs # poskeyword_args_kwonly_kwargs = arguments [',' [args_kwonly_kwargs]] # vararglist_no_posonly = poskeyword_args_kwonly_kwargs | args_kwonly_kwargs # varargslist = arguments ',' '/' [','[(vararglist_no_posonly)]] | (vararglist_no_posonly) # # It needs to be fully expanded to allow our LL(1) parser to work on it. varargslist: vfpdef ['=' test ](',' vfpdef ['=' test])* ',' '/' [',' [ (vfpdef ['=' test] (',' vfpdef ['=' test])* [',' [ '*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]] | '**' vfpdef [',']]] | '*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]] | '**' vfpdef [',']) ]] | (vfpdef ['=' test] (',' vfpdef ['=' test])* [',' [ '*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]] | '**' vfpdef [',']]] | '*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]] | '**' vfpdef [','] ) vfpdef: NAME (https://docs.python.org/3/reference/grammar.html) Today's editors have no problem finding the currently used local variables and adding a key binding to turn a list of variables a, b, c into a=a, b=b, c=c is certainly possible for those who believe that auto-complete is not fast enough. Explicit is better than implicit, remember ?! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 20 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Mon, Apr 20, 2020 at 10:03:50AM +0200, M.-A. Lemburg wrote:
Guys, is it really worth saving a few hits on the auto-complete key by adding even more mysterious twists to the existing Python function call syntax ?
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python:
(I think you pasted the typedarglist rules twice.) Now that Python is moving to a PEG parser, could it be simplified? Might we get sequence-unpacking parameters back again? # Legal in Python 2, illegal in Python 3. py> def func(a, (b, c), d): ... print a, b, c, d ... py> func(1, "ab", 3) 1 a b 3 I know that was a feature that got removed only because it was hard to specify in the grammer, even though it was used by lots of people. -- Steven
On 4/20/2020 7:00 AM, Steven D'Aprano wrote:
On Mon, Apr 20, 2020 at 10:03:50AM +0200, M.-A. Lemburg wrote:
Guys, is it really worth saving a few hits on the auto-complete key by adding even more mysterious twists to the existing Python function call syntax ?
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python: (I think you pasted the typedarglist rules twice.)
Now that Python is moving to a PEG parser, could it be simplified?
Might we get sequence-unpacking parameters back again?
# Legal in Python 2, illegal in Python 3. py> def func(a, (b, c), d): ... print a, b, c, d ... py> func(1, "ab", 3) 1 a b 3
I know that was a feature that got removed only because it was hard to specify in the grammer, even though it was used by lots of people.
See PEP 3113, which doesn't mention parsing. My understanding is that it was the introspection problem that drove this. And on very rare occasions, I really miss this feature. Eric
On Mon, 20 Apr 2020 at 12:19, Eric V. Smith <eric@trueblade.com> wrote:
On 4/20/2020 7:00 AM, Steven D'Aprano wrote:
On Mon, Apr 20, 2020 at 10:03:50AM +0200, M.-A. Lemburg wrote:
Guys, is it really worth saving a few hits on the auto-complete key by adding even more mysterious twists to the existing Python function call syntax ?
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python: (I think you pasted the typedarglist rules twice.)
Now that Python is moving to a PEG parser, could it be simplified?
Might we get sequence-unpacking parameters back again?
# Legal in Python 2, illegal in Python 3. py> def func(a, (b, c), d): ... print a, b, c, d ... py> func(1, "ab", 3) 1 a b 3
I know that was a feature that got removed only because it was hard to specify in the grammer, even though it was used by lots of people.
See PEP 3113, which doesn't mention parsing. My understanding is that it was the introspection problem that drove this.
And on very rare occasions, I really miss this feature.
I migrated a lot of code to Python 3 recently and I really miss this feature in lambdas: sorted(users, key=lambda u: (u[1].lower(), u[0])) is IMO much uglier than sorted(users, key=lambda (id, name): (name.lower(), id)) -- Ivan
On 4/20/2020 7:27 AM, Ivan Levkivskyi wrote:
On Mon, 20 Apr 2020 at 12:19, Eric V. Smith <eric@trueblade.com <mailto:eric@trueblade.com>> wrote:
See PEP 3113, which doesn't mention parsing. My understanding is that it was the introspection problem that drove this.
And on very rare occasions, I really miss this feature.
I migrated a lot of code to Python 3 recently and I really miss this feature in lambdas:
sorted(users, key=lambda u: (u[1].lower(), u[0]))
is IMO much uglier than
sorted(users, key=lambda (id, name): (name.lower(), id))
Yes, that's one place I miss it. Also similar usages with dict.items() key value pairs. Eric
On 20.04.2020 13:00, Steven D'Aprano wrote:
On Mon, Apr 20, 2020 at 10:03:50AM +0200, M.-A. Lemburg wrote:
Guys, is it really worth saving a few hits on the auto-complete key by adding even more mysterious twists to the existing Python function call syntax ?
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python:
(I think you pasted the typedarglist rules twice.)
No, there are two lists: typedargslist and varargslist. The latter is only used in lambda functions, which don't support type annotations (yet another twist to remember when it comes to function calls).
Now that Python is moving to a PEG parser, could it be simplified?
Perhaps, but that's not really the point I wanted to make. If it's difficult to spell the correct syntax in a parser grammar, then humans will have trouble understanding it as well, esp. people new to Python. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 20 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Apr 20, 2020, at 01:06, M.-A. Lemburg <mal@egenix.com> wrote:
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python:
funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] func_body_suite
But nobody’s proposing changing the function definition syntax here, only the function call syntax. Which is a lot less hairy. It is still somewhat hairy, but nowhere near as bad, so this argument doesn’t really apply. Also, you’re lumping all the different proposals here, but they don’t all have the same effect, which makes the argument even weaker. Adding a ** mode switch does make calls significantly more complicated, because it effectively clones half of the call grammar to switch to a similar but new grammar. But allowing keyword= is a simple and local change to one small subpart of the call grammar that I don’t think adds too much burden. And ::value in dict displays doesn’t touch the call syntax at all; it makes only a trivial and local change to a subpart of the much simpler dict display grammar. And **{a,b,c} is by far the most complicated, but the complicated part isn’t in calls (which would just gain one simple alternation); it’s in cloning half of the expression grammar to create a new nonset_expression node; the change to call syntax to use that new node is simple. (I’m assuming this proposal would make **{set display} in a call a syntax error when it’s not a magic set-of-identifiers unpacking, because otherwise I don’t know how you could disambiguate at all.) So, even if you hadn’t mixed up definitions and calls, I don’t think this argument really holds much water. I think your point that “hard to parse means hard to reason about” is a good one, however. That’s part of my rationale for the ::value syntax in dict displays: it’s a simple change to a simple piece of syntax that’s well isolated and consistent everywhere it appears. But I don’t think people would actually have a problem learning, internalizing, and reading keyword= syntax. And I think it may be an argument against the **{a,b,c} syntax, but only in a more subtle way than you’re advancing—people wouldn’t even internalize the right grammar; they’d just think of it as a special use of set displays (in fact Steven, who proposed it, encourages that reading), which is an extra special case to learn. Which can still be acceptable (lots of people get away with thinking of target lists as a special use of tuple displays…); it’s just a really high bar to clear.
On 2020-04-20 18:43, Andrew Barnert via Python-ideas wrote:
On Apr 20, 2020, at 01:06, M.-A. Lemburg <mal@egenix.com> wrote:
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python:
funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] func_body_suite
But nobody’s proposing changing the function definition syntax here, only the function call syntax. Which is a lot less hairy. It is still somewhat hairy, but nowhere near as bad, so this argument doesn’t really apply.
Also, you’re lumping all the different proposals here, but they don’t all have the same effect, which makes the argument even weaker.
Adding a ** mode switch does make calls significantly more complicated, because it effectively clones half of the call grammar to switch to a similar but new grammar.
But allowing keyword= is a simple and local change to one small subpart of the call grammar that I don’t think adds too much burden.
And ::value in dict displays doesn’t touch the call syntax at all; it makes only a trivial and local change to a subpart of the much simpler dict display grammar.
And **{a,b,c} is by far the most complicated, but the complicated part isn’t in calls (which would just gain one simple alternation); it’s in cloning half of the expression grammar to create a new nonset_expression node; the change to call syntax to use that new node is simple. (I’m assuming this proposal would make **{set display} in a call a syntax error when it’s not a magic set-of-identifiers unpacking, because otherwise I don’t know how you could disambiguate at all.)
So, even if you hadn’t mixed up definitions and calls, I don’t think this argument really holds much water.
I think your point that “hard to parse means hard to reason about” is a good one, however. That’s part of my rationale for the ::value syntax in dict displays: it’s a simple change to a simple piece of syntax that’s well isolated and consistent everywhere it appears. But I don’t think people would actually have a problem learning, internalizing, and reading keyword= syntax. And I think it may be an argument against the **{a,b,c} syntax, but only in a more subtle way than you’re advancing—people wouldn’t even internalize the right grammar; they’d just think of it as a special use of set displays (in fact Steven, who proposed it, encourages that reading), which is an extra special case to learn. Which can still be acceptable (lots of people get away with thinking of target lists as a special use of tuple displays…); it’s just a really high bar to clear.
It occurs to me that we could use double braces, which is currently invalid because set displays aren't hashable. For example: {{a, b, c}} would be equivalent to: {'a': a, 'b': b, 'c': c} This would let you replace: setattrs(cls, text=text, program=program, messages=messages, hints=hints) with: setattrs(cls, **{{text, program, messages, hints}})
On 20.04.2020 19:43, Andrew Barnert via Python-ideas wrote:
On Apr 20, 2020, at 01:06, M.-A. Lemburg <mal@egenix.com> wrote:
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python:
funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] func_body_suite
But nobody’s proposing changing the function definition syntax here, only the function call syntax. Which is a lot less hairy. It is still somewhat hairy, but nowhere near as bad, so this argument doesn’t really apply.
True, I quoted the wrong part of the grammar for the argument, sorry. I meant this part: https://docs.python.org/3/reference/expressions.html#calls which is simpler, but not really much, since the devil is in the details. We already have shortcuts in the form of * and ** for function calls, both putting the unpacking on the calling side of the function, rather than inside the function where it would arguably be much cleaner to add. You're now proposing to add yet another shortcut to avoid having to translate local variables to possibly the same keyword arguments used by the function. But arguing that f(a=, b=, c=, d=) is any better than f(a=a, b=b, c=c, d=d) does not really solve anything. The problem is with the code design, not with the syntax. You'd be better off putting your variables combined into a context object or container and pass that around: f(context) In many cases, you can avoid passing around any of these variables, by simply using an object rather than a function oriented approach. The variables would then become object attributes, so calling f() becomes: task.f() and the method f would get it's context straight from the object attributes of task -- without any arguments to pass in. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 21 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Apr 20, 2020, at 16:24, M.-A. Lemburg <mal@egenix.com> wrote:
On 20.04.2020 19:43, Andrew Barnert via Python-ideas wrote:
On Apr 20, 2020, at 01:06, M.-A. Lemburg <mal@egenix.com> wrote:
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python:
funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] func_body_suite
But nobody’s proposing changing the function definition syntax here, only the function call syntax. Which is a lot less hairy. It is still somewhat hairy, but nowhere near as bad, so this argument doesn’t really apply.
True, I quoted the wrong part of the grammar for the argument, sorry. I meant this part:
https://docs.python.org/3/reference/expressions.html#calls
which is simpler, but not really much, since the devil is in the details.
Let’s just take one of the variant proposals under discussion here, adding ::identifier to dict displays. This makes no change to the call grammar, or to any of the call-related bits, or any other horribly complicated piece of grammar. It just changes key_datum (a nonterminal referenced only in dict_display) from this: expression ":" expression | “**” or_expr … to this: expression ":" expression | “::” identifier | “**” or_expr That’s about as simple as any syntax change ever gets. Which is still not nothing. But you’re absolutely right that a big and messy change to function definition grammar would have a higher bar to clear than most syntax proposals—and for the exact same reason, a small and local change to dict display datum grammar has a lower bar than most syntax proposals.
On 21.04.2020 04:25, Andrew Barnert via Python-ideas wrote:
On Apr 20, 2020, at 16:24, M.-A. Lemburg <mal@egenix.com> wrote:
On 20.04.2020 19:43, Andrew Barnert via Python-ideas wrote:
On Apr 20, 2020, at 01:06, M.-A. Lemburg <mal@egenix.com> wrote:
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python:
funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] func_body_suite
But nobody’s proposing changing the function definition syntax here, only the function call syntax. Which is a lot less hairy. It is still somewhat hairy, but nowhere near as bad, so this argument doesn’t really apply.
True, I quoted the wrong part of the grammar for the argument, sorry. I meant this part:
https://docs.python.org/3/reference/expressions.html#calls
which is simpler, but not really much, since the devil is in the details.
Let’s just take one of the variant proposals under discussion here, adding ::identifier to dict displays. This makes no change to the call grammar, or to any of the call-related bits, or any other horribly complicated piece of grammar. It just changes key_datum (a nonterminal referenced only in dict_display) from this:
expression ":" expression | “**” or_expr
… to this:
expression ":" expression | “::” identifier | “**” or_expr
That’s about as simple as any syntax change ever gets.
Which is still not nothing. But you’re absolutely right that a big and messy change to function definition grammar would have a higher bar to clear than most syntax proposals—and for the exact same reason, a small and local change to dict display datum grammar has a lower bar than most syntax proposals.
I think the real issue you would like to resolve is how to get at the variable names used for calling a function, essentially pass-by-reference (in the Python sense, where variable names are references to objects, not pointers as in C) rather than pass-by-value, as is the default for Python functions. The f-string logic addresses a similar need. With a way to get at the variable names used for calling a function from inside a function, you could then write a dict constructor which gives you the subset of vars() you are looking for. If you know that the variable names used in the function are the same as in the calling context, you can already do this using introspection. For the general purpose case, I don't think this is possible without the help of the compiler, since the pass-by-value is implemented using the bytecodes and the stack, AFAIR. Rather than using new syntax, a helper in the compiler would do the trick, though, similar to what e.g. @classmethod does. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 21 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Apr 21, 2020, at 01:27, M.-A. Lemburg <mal@egenix.com> wrote:
On 21.04.2020 04:25, Andrew Barnert via Python-ideas wrote:
On Apr 20, 2020, at 16:24, M.-A. Lemburg <mal@egenix.com> wrote:
On 20.04.2020 19:43, Andrew Barnert via Python-ideas wrote:
On Apr 20, 2020, at 01:06, M.-A. Lemburg <mal@egenix.com> wrote:
The current version already strikes me as way too complex. It's by far the most complex piece of grammar we have in Python:
funcdef: 'def' NAME parameters ['->' test] ':' [TYPE_COMMENT] func_body_suite
But nobody’s proposing changing the function definition syntax here, only the function call syntax. Which is a lot less hairy. It is still somewhat hairy, but nowhere near as bad, so this argument doesn’t really apply.
True, I quoted the wrong part of the grammar for the argument, sorry. I meant this part:
https://docs.python.org/3/reference/expressions.html#calls
which is simpler, but not really much, since the devil is in the details.
Let’s just take one of the variant proposals under discussion here, adding ::identifier to dict displays. This makes no change to the call grammar, or to any of the call-related bits, or any other horribly complicated piece of grammar. It just changes key_datum (a nonterminal referenced only in dict_display) from this:
expression ":" expression | “**” or_expr
… to this:
expression ":" expression | “::” identifier | “**” or_expr
That’s about as simple as any syntax change ever gets.
Which is still not nothing. But you’re absolutely right that a big and messy change to function definition grammar would have a higher bar to clear than most syntax proposals—and for the exact same reason, a small and local change to dict display datum grammar has a lower bar than most syntax proposals.
I think the real issue you would like to resolve is how to get at the variable names used for calling a function, essentially pass-by-reference (in the Python sense, where variable names are references to objects, not pointers as in C) rather than pass-by-value, as is the default for Python functions.
No, nobody’s asking for that either. It wouldn’t directly solve most of the examples in this thread, or even indirectly make them easier to solve. The problem in most cases is that they have to call a function that they can’t change with a big mess of parameters. Any change to help the callee side doesn’t do any good, because the callee is the thing they can’t change. The fix needs to be on the caller side alone. This also wouldn’t give you useful pass-by-reference in the usual sense of “I want to let the callee rebind the variables I pass in”, because a name isn’t a reference in Python without the namespace to look it up in. Even if the callee knew the name the caller used for one of its parameters, how would it know whether that name was a local or a cell or a global? If it’s a local, how would it get at the caller’s local environment without frame hacking? (As people have demonstrated on this thread, frame hacking on its own is enough, without any new changes.) Even if it could get that local environment, how could it rebind the variable when you can’t mutate locals dicts? Also, most arguments in Python do not have names, because arguments are arbitrary expressions. Of course the same thing is true in, say, C++, but that’s fine in C++, because lvalue expressions have perfectly good lvalues even if they don’t have good names. You can pass p->vec[idx+1].spam to a function that wants an int&, and it can modify its parameter and you’ll see the change on your side. How could your proposal handle even the simplest case of passing lst[0]? Even if it could work as advertised, it’s hugely overkill for this problem. A full-blown macro system would let people solve this problem, and half the other things people propose for Python, but that doesn’t mean that half the proposals on this list are requests for a full-blown macro system, or that it’s the right answer for them.
The f-string logic addresses a similar need.
Similar, yes, but the f-string logic (a) runs in the caller’s scope and (b) evaluates code that’s textually part of the caller.
With a way to get at the variable names used for calling a function from inside a function, you could then write a dict constructor which gives you the subset of vars() you are looking for.
Most of the use cases involve “I can’t change the callee, so I need to give it the ugly mess of keyword args that it wants”. So any proposal to allow changing callees is already on the wrong track. But even if this were acceptable, how would it help anything? Take any of the examples from this thread, ignore the “I can’t change the function I’m calling part”, and show how you’d solve it if the callee had the caller’s names for its parameters. Also, “you can solve this with ugly introspection code” does not always mean we don’t need a better solution. In fact, you can solve this today, from the caller side, with something like this (not in front of a computer right now, so probably most of the details are wrong, but hopefully you can get the idea): sig = inspect.signature(spam) spamargs = {k: v for (k, v) in vars().items() if k in sig.kwargs} spam(**spamargs) … but it should be obvious why nobody writes that code rather than: spam(eggs=eggs, cheese=cheese, ham=ham) They want a way to make things less verbose and less error prone, and I just offered them a way to make things more verbose and more error prone, and I doubt any of them will take it. You’re offering them a way to rewrite the function they can’t rewrite so they can have another way to make things more verbose and more error prone and also more unsafe and nonportable, and I doubt that will be any more helpful.
Rather than using new syntax, a helper in the compiler would do the trick, though, similar to what e.g. @classmethod does.
@classmethod doesn’t have any help from the compiler. It’s a trivial use of the same __get__ protocol, which you could write yourself. And it’s not clear what kind of compiler helper you could use for your idea. Remember that when compiling a call, the compiler has no idea what the callable is; it can’t know, because that’s completely dynamic. So anything you want to magically sweep up and package from the caller side has to be passed in to every call. You surely wouldn’t want to pass along the name of every argument in every call? That would be a huge backward compatibility break and a huge performance cost, and also impossible because most arguments don’t have names.
On Tue, Apr 21, 2020 at 9:27 AM M.-A. Lemburg <mal@egenix.com> wrote:
But arguing that f(a=, b=, c=, d=) is any better than f(a=a, b=b, c=c, d=d) does not really solve anything.
The problem is with the code design, not with the syntax.
You'd be better off putting your variables combined into a context object or container and pass that around:
f(context)
In many cases, you can avoid passing around any of these variables, by simply using an object rather than a function oriented approach. The variables would then become object attributes, so calling f() becomes:
task.f()
and the method f would get it's context straight from the object attributes of task -- without any arguments to pass in.
Please explain how this will help the case of render_template and similar, where the function accepts ANY keyword arguments, and then passes them along to the template. Do I need to package everything up into a dictionary? And how would that actually improve things, given that I'd still need to do the exact same thing to construct that dictionary? Every template has its own unique variables, and most of them come straight from the calling function, with the same names. This is NOT going to be a consistent object with the same attributes. That simply is not the use-case here. We already have that. ChrisA
On 21.04.2020 06:56, Chris Angelico wrote:
On Tue, Apr 21, 2020 at 9:27 AM M.-A. Lemburg <mal@egenix.com> wrote:
But arguing that f(a=, b=, c=, d=) is any better than f(a=a, b=b, c=c, d=d) does not really solve anything.
The problem is with the code design, not with the syntax.
You'd be better off putting your variables combined into a context object or container and pass that around:
f(context)
In many cases, you can avoid passing around any of these variables, by simply using an object rather than a function oriented approach. The variables would then become object attributes, so calling f() becomes:
task.f()
and the method f would get it's context straight from the object attributes of task -- without any arguments to pass in.
Please explain how this will help the case of render_template and similar, where the function accepts ANY keyword arguments, and then passes them along to the template. Do I need to package everything up into a dictionary? And how would that actually improve things, given that I'd still need to do the exact same thing to construct that dictionary? Every template has its own unique variables, and most of them come straight from the calling function, with the same names.
This is NOT going to be a consistent object with the same attributes. That simply is not the use-case here. We already have that.
A render_template() function which has **kws as parameter can take any number of keyword parameters, including ones which are not used in the template. Those functions will usually take a namespace object as basis for rendering the final string. In the use case discussed here, that namespace would be locals(), so you could just as well write render_template(**locals()), or better: use a namespace object, fill this with the right values and pass in render_template(namespace). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 21 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Tue, Apr 21, 2020 at 5:41 PM M.-A. Lemburg <mal@egenix.com> wrote:
On 21.04.2020 06:56, Chris Angelico wrote:
On Tue, Apr 21, 2020 at 9:27 AM M.-A. Lemburg <mal@egenix.com> wrote:
But arguing that f(a=, b=, c=, d=) is any better than f(a=a, b=b, c=c, d=d) does not really solve anything.
The problem is with the code design, not with the syntax.
You'd be better off putting your variables combined into a context object or container and pass that around:
f(context)
In many cases, you can avoid passing around any of these variables, by simply using an object rather than a function oriented approach. The variables would then become object attributes, so calling f() becomes:
task.f()
and the method f would get it's context straight from the object attributes of task -- without any arguments to pass in.
Please explain how this will help the case of render_template and similar, where the function accepts ANY keyword arguments, and then passes them along to the template. Do I need to package everything up into a dictionary? And how would that actually improve things, given that I'd still need to do the exact same thing to construct that dictionary? Every template has its own unique variables, and most of them come straight from the calling function, with the same names.
This is NOT going to be a consistent object with the same attributes. That simply is not the use-case here. We already have that.
A render_template() function which has **kws as parameter can take any number of keyword parameters, including ones which are not used in the template.
Those functions will usually take a namespace object as basis for rendering the final string.
In the use case discussed here, that namespace would be locals(), so you could just as well write render_template(**locals()), or better: use a namespace object, fill this with the right values and pass in render_template(namespace).
I don't want to give it all my locals; I want to give it a specific subset of them, and a few others that aren't actually in variables. You've seen the code earlier in this thread. Show me how your proposal would improve it. ChrisA
On Tue, Apr 21, 2020 at 9:45 AM M.-A. Lemburg <mal@egenix.com> wrote:
In the use case discussed here, that namespace would be locals(), so you could just as well write render_template(**locals()),
Doing that gets in the way of tools. At the very least it makes it impossible to highlight accidentally unused variables.
or better: use a namespace object, fill this with the right values and pass in render_template(namespace).
This sounds like it's going to have the exact same amount of same-named keywords. Please clarify how this is different. <http://python.org/psf/codeofconduct/> We went through this discussion already with Stephen J Turnbull. I've given concrete examples from the CPython repo of this pattern, where you can see all the context. Feel free to pick any of them (except the log level thing, an enum is the right solution there) and explain exactly how you would reduce same-naming.
On 21.04.2020 10:07, Alex Hall wrote:
On Tue, Apr 21, 2020 at 9:45 AM M.-A. Lemburg <mal@egenix.com <mailto:mal@egenix.com>> wrote:
In the use case discussed here, that namespace would be locals(), so you could just as well write render_template(**locals()),
Doing that gets in the way of tools. At the very least it makes it impossible to highlight accidentally unused variables.
True, but when rendering a template, you normally don't care about unused variables -- just about undefined ones :-)
or better: use a namespace object, fill this with the right values and pass in render_template(namespace).
This sounds like it's going to have the exact same amount of same-named keywords. Please clarify how this is different.
The difference is that you don't have spell out all the attributes in the function, since the namespace already knows about them. Instead of keeping values in local variables, you store them in the namespace object and because this knows about its attributes you can do a lot more in terms of introspection than what is possible when just relying on local variables in a function call context. See e.g. argparse's various namespace uses for an example in the stdlib.
We went through this discussion already with Stephen J Turnbull. I've given concrete examples from the CPython repo of this pattern, where you can see all the context. Feel free to pick any of them (except the log level thing, an enum is the right solution there) and explain exactly how you would reduce same-naming.
I am aware of the pattern and it's needed in cases where you don't have a way to influence the API, e.g. when using a 3rd party lib which wants to receive configuration via function call parameters only. Where you do have full control, it's pretty easy to get around the need to write var1=var1, var2=var2, etc. by using namespace or context objects, keyword dict manipulation, a better object oriented design or dedicated configuration APIs. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 21 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Tue, Apr 21, 2020 at 11:01 AM M.-A. Lemburg <mal@egenix.com> wrote:
On 21.04.2020 10:07, Alex Hall wrote:
On Tue, Apr 21, 2020 at 9:45 AM M.-A. Lemburg <mal@egenix.com <mailto:mal@egenix.com>> wrote:
In the use case discussed here, that namespace would be locals(), so you could just as well write
render_template(**locals()),
Doing that gets in the way of tools. At the very least it makes it impossible to highlight accidentally unused variables.
True, but when rendering a template, you normally don't care about unused variables -- just about undefined ones :-)
I always care about unused variables, because they tend to be a red flag that I meant to use them somewhere but made a mistake, e.g. I used the wrong variable instead.
We went through this discussion already with Stephen J Turnbull. I've given concrete examples from the CPython repo of this pattern, where you can see all the context. Feel free to pick any of them (except the log level thing, an enum is the right solution there) and explain exactly how you would reduce same-naming.
I am aware of the pattern and it's needed in cases where you don't have a way to influence the API, e.g. when using a 3rd party lib which wants to receive configuration via function call parameters only.
Where you do have full control, it's pretty easy to get around the need to write var1=var1, var2=var2, etc. by using namespace or context objects, keyword dict manipulation, a better object oriented design or dedicated configuration APIs.
We're not talking about a third party lib, and you do have plenty of control. Several of those calls are to internal, protected APIs.
On Tue, Apr 21, 2020 at 7:01 PM M.-A. Lemburg <mal@egenix.com> wrote:
Instead of keeping values in local variables, you store them in the namespace object and because this knows about its attributes you can do a lot more in terms of introspection than what is possible when just relying on local variables in a function call context.
See e.g. argparse's various namespace uses for an example in the stdlib.
Let me make things even clearer here. This is the entire code for the index page handler. @app.route("/") @app.route("/editor/<channelid>") def mainpage(channelid=None): # NOTE: If we've *reduced* the required scopes, this will still force a re-login. # However, it'll be an easy login, as Twitch will recognize the existing auth. if "twitch_token" not in session or session.get("twitch_auth_scopes") != REQUIRED_SCOPES: return render_template("login.html") user = session["twitch_user"] if channelid is None: channelid = user["_id"] try: channelid = str(int(channelid)) except ValueError: # If you go to /editor/somename, redirect to /editor/equivalent-id # Bookmarking the version with the ID will be slightly faster, but # streamers will usually want to share the version with the name. users = query("helix/users", token=None, params={"login": channelid})["data"] # users is either an empty list (bad login) or a list of one. if not users: return redirect("/") return redirect("/editor/" + users[0]["id"]) if not may_edit_channel(user["_id"], channelid): return redirect(url_for("mainpage")) database.create_user(channelid) # Just in case, make sure the database has the basic structure channel = get_channel_setup(channelid) sched_tz, schedule, sched_tweet = database.get_schedule(channelid) if "twitter_oauth" in session: auth = session["twitter_oauth"] username = auth["screen_name"] twitter = "Twitter connected: " + username tweets = list_scheduled_tweets(auth["oauth_token"], auth["oauth_token_secret"], sched_tz) else: twitter = Markup("""<div id="login-twitter"><a href="/login-twitter"><img src="/static/Twitter_Social_Icon_Square_Color.svg" alt="Twitter logo"><div>Connect with Twitter</div></a></div>""") tweets = [] error = session.get("last_error_message", "") session["last_error_message"] = "" return render_template("index.html", twitter=twitter, username=user["display_name"], channel=channel, channelid=channelid, error=error, setups=database.list_setups(channelid), sched_tz=sched_tz, schedule=schedule, sched_tweet=sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=tweets, ) Now, show me which part of this should become a "namespace object". That namespace would be *unique* to this function - it would have no purpose or value outside of this one exact function. Show me how the use of such a namespace object would improve this code. There is no way that you would be able to share it with any other function in the entire program. The original proposal (or at least, one of the early proposals) is an entirely self-contained change that allows a notation like "twitter=" rather than "twitter=twitter", removing the repetition and thus the chance for error (consider if one of these got renamed or had a typo), without mandating any sort of wholesale architectural change. You're insisting that this entire concept is an anti-pattern and that the architectural change would improve it. Prove that. ChrisA
On 4/21/2020 9:51 AM, Chris Angelico wrote:
On Tue, Apr 21, 2020 at 7:01 PM M.-A. Lemburg <mal@egenix.com> wrote:
Instead of keeping values in local variables, you store them in the namespace object and because this knows about its attributes you can do a lot more in terms of introspection than what is possible when just relying on local variables in a function call context.
See e.g. argparse's various namespace uses for an example in the stdlib.
Let me make things even clearer here. This is the entire code for the index page handler.
@app.route("/") @app.route("/editor/<channelid>") def mainpage(channelid=None): # NOTE: If we've *reduced* the required scopes, this will still force a re-login. # However, it'll be an easy login, as Twitch will recognize the existing auth. if "twitch_token" not in session or session.get("twitch_auth_scopes") != REQUIRED_SCOPES: return render_template("login.html") user = session["twitch_user"] if channelid is None: channelid = user["_id"] try: channelid = str(int(channelid)) except ValueError: # If you go to /editor/somename, redirect to /editor/equivalent-id # Bookmarking the version with the ID will be slightly faster, but # streamers will usually want to share the version with the name. users = query("helix/users", token=None, params={"login": channelid})["data"] # users is either an empty list (bad login) or a list of one. if not users: return redirect("/") return redirect("/editor/" + users[0]["id"]) if not may_edit_channel(user["_id"], channelid): return redirect(url_for("mainpage")) database.create_user(channelid) # Just in case, make sure the database has the basic structure channel = get_channel_setup(channelid) sched_tz, schedule, sched_tweet = database.get_schedule(channelid) if "twitter_oauth" in session: auth = session["twitter_oauth"] username = auth["screen_name"] twitter = "Twitter connected: " + username tweets = list_scheduled_tweets(auth["oauth_token"], auth["oauth_token_secret"], sched_tz) else: twitter = Markup("""<div id="login-twitter"><a href="/login-twitter"><img src="/static/Twitter_Social_Icon_Square_Color.svg" alt="Twitter logo"><div>Connect with Twitter</div></a></div>""") tweets = [] error = session.get("last_error_message", "") session["last_error_message"] = "" return render_template("index.html", twitter=twitter, username=user["display_name"], channel=channel, channelid=channelid, error=error, setups=database.list_setups(channelid), sched_tz=sched_tz, schedule=schedule, sched_tweet=sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=tweets, )
Now, show me which part of this should become a "namespace object". That namespace would be *unique* to this function - it would have no purpose or value outside of this one exact function. Show me how the use of such a namespace object would improve this code. There is no way that you would be able to share it with any other function in the entire program.
The original proposal (or at least, one of the early proposals) is an entirely self-contained change that allows a notation like "twitter=" rather than "twitter=twitter", removing the repetition and thus the chance for error (consider if one of these got renamed or had a typo), without mandating any sort of wholesale architectural change. You're insisting that this entire concept is an anti-pattern and that the architectural change would improve it. Prove that.
That's a good example, Chris. Thanks. I also don't see that a namespace object would buy you much, if anything. Going with the tersest proposal (twitter=twitter becomes twitter=), we'd save something like 40 characters in the function call in the return statement. I think making a change isn't worth adding more to the language, but of course reasonable people can disagree. Eric
On Wed, Apr 22, 2020 at 12:06 AM Eric V. Smith <eric@trueblade.com> wrote:
That's a good example, Chris. Thanks. I also don't see that a namespace object would buy you much, if anything.
Going with the tersest proposal (twitter=twitter becomes twitter=), we'd save something like 40 characters in the function call in the return statement. I think making a change isn't worth adding more to the language, but of course reasonable people can disagree.
Thanks. But it's really not about terseness. I've already typed more in this thread than I'll probably save over a lifetime of shorthanding. If I wanted shorthands, I'd just use shorter variable names. Removing duplication removes the possibility of desynchronization. It's far easier to see that "twitter=," is passing twitter with the same name than to check that all the instances of "twitter=twitter" and "channel=channel" and "tweets=tweets" are all perfectly correct. It also becomes a logical idiom for "pass these things to the template, as-is", rather than getting bogged down in the mechanics. Having worked with (many) languages that don't have keyword arguments at all, I've gotten all too accustomed to the hassles of wrapping things up into objects. Consider the difference between positional and keyword arguments in the JavaScript fetch() function [1], where the URL to be fetched is passed positionally, and everything else is... an object passed as the second parameter. That is exactly the sort of fiddliness that I'm trying to avoid. ChrisA [1] https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/f...
On 21/04/2020 15:29, Chris Angelico wrote:
On Wed, Apr 22, 2020 at 12:06 AM Eric V. Smith <eric@trueblade.com> wrote:
That's a good example, Chris. Thanks. I also don't see that a namespace object would buy you much, if anything.
Going with the tersest proposal (twitter=twitter becomes twitter=), we'd save something like 40 characters in the function call in the return statement. I think making a change isn't worth adding more to the language, but of course reasonable people can disagree.
Thanks. But it's really not about terseness. I've already typed more in this thread than I'll probably save over a lifetime of shorthanding. If I wanted shorthands, I'd just use shorter variable names.
Removing duplication removes the possibility of desynchronization.
Now here's the heart of things. We're actually talking about synchronization, which in this case happens to be done by using the same name in different contexts. Sometimes this is a good thing, sometimes not; calling it an anti-pattern is harsh, but it does channel your thinking into tramlines that are not always a good thing. The number of times I've changed "handle=handle" to "handle=medium_handle" as projects have crept... -- Rhodri James *-* Kynesim Ltd
On Wed, Apr 22, 2020 at 1:38 AM Rhodri James <rhodri@kynesim.co.uk> wrote:
On 21/04/2020 15:29, Chris Angelico wrote:
On Wed, Apr 22, 2020 at 12:06 AM Eric V. Smith <eric@trueblade.com> wrote:
That's a good example, Chris. Thanks. I also don't see that a namespace object would buy you much, if anything.
Going with the tersest proposal (twitter=twitter becomes twitter=), we'd save something like 40 characters in the function call in the return statement. I think making a change isn't worth adding more to the language, but of course reasonable people can disagree.
Thanks. But it's really not about terseness. I've already typed more in this thread than I'll probably save over a lifetime of shorthanding. If I wanted shorthands, I'd just use shorter variable names.
Removing duplication removes the possibility of desynchronization.
Now here's the heart of things. We're actually talking about synchronization, which in this case happens to be done by using the same name in different contexts. Sometimes this is a good thing, sometimes not; calling it an anti-pattern is harsh, but it does channel your thinking into tramlines that are not always a good thing. The number of times I've changed "handle=handle" to "handle=medium_handle" as projects have crept...
And if you do, you change it, and it's very clear that you deliberately did so. If it's a change from "handle=" to "handle=medium_handle" then there should be a corresponding change elsewhere in the function, and *no* corresponding change in the API. Conversely, if it's a change from "handle=" to "medium_handle=handle", then you'd expect *no* change in the function, and a change somewhere in the API (maybe a change in the template file or something). Either way, the clear change from shorthand to longhand signals that this was a deliberate change that broke the symmetry, rather than being an accidental failure to rename something. ChrisA
On 21.04.2020 15:51, Chris Angelico wrote:
On Tue, Apr 21, 2020 at 7:01 PM M.-A. Lemburg <mal@egenix.com> wrote:
Instead of keeping values in local variables, you store them in the namespace object and because this knows about its attributes you can do a lot more in terms of introspection than what is possible when just relying on local variables in a function call context.
See e.g. argparse's various namespace uses for an example in the stdlib.
Let me make things even clearer here. This is the entire code for the index page handler.
@app.route("/") @app.route("/editor/<channelid>") def mainpage(channelid=None): # NOTE: If we've *reduced* the required scopes, this will still force a re-login. # However, it'll be an easy login, as Twitch will recognize the existing auth. if "twitch_token" not in session or session.get("twitch_auth_scopes") != REQUIRED_SCOPES: return render_template("login.html") user = session["twitch_user"] if channelid is None: channelid = user["_id"] try: channelid = str(int(channelid)) except ValueError: # If you go to /editor/somename, redirect to /editor/equivalent-id # Bookmarking the version with the ID will be slightly faster, but # streamers will usually want to share the version with the name. users = query("helix/users", token=None, params={"login": channelid})["data"] # users is either an empty list (bad login) or a list of one. if not users: return redirect("/") return redirect("/editor/" + users[0]["id"]) if not may_edit_channel(user["_id"], channelid): return redirect(url_for("mainpage")) database.create_user(channelid) # Just in case, make sure the database has the basic structure channel = get_channel_setup(channelid) sched_tz, schedule, sched_tweet = database.get_schedule(channelid) if "twitter_oauth" in session: auth = session["twitter_oauth"] username = auth["screen_name"] twitter = "Twitter connected: " + username tweets = list_scheduled_tweets(auth["oauth_token"], auth["oauth_token_secret"], sched_tz) else: twitter = Markup("""<div id="login-twitter"><a href="/login-twitter"><img src="/static/Twitter_Social_Icon_Square_Color.svg" alt="Twitter logo"><div>Connect with Twitter</div></a></div>""") tweets = [] error = session.get("last_error_message", "") session["last_error_message"] = "" return render_template("index.html", twitter=twitter, username=user["display_name"], channel=channel, channelid=channelid, error=error, setups=database.list_setups(channelid), sched_tz=sched_tz, schedule=schedule, sched_tweet=sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=tweets, )
Now, show me which part of this should become a "namespace object". That namespace would be *unique* to this function - it would have no purpose or value outside of this one exact function. Show me how the use of such a namespace object would improve this code. There is no way that you would be able to share it with any other function in the entire program.
Instead of saving the variables in the local scope, you'd create a namespace object and write them directly into that object, e.g. ns = Namespace() ns.auth = session["twitter_oauth"] ... ns.tweets = [] return render_template("index.html", ns) Another advantage of this style is that debugging becomes much easier, since you can see the full context used by the rendering function and even provide custom output methods for the namespace content. Your local scope doesn't get cluttered up with all the namespace details. I'll leave this at rest now. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Apr 22 2020)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Thu, Apr 23, 2020 at 6:18 AM M.-A. Lemburg <mal@egenix.com> wrote:
On 21.04.2020 15:51, Chris Angelico wrote:
On Tue, Apr 21, 2020 at 7:01 PM M.-A. Lemburg <mal@egenix.com> wrote:
Instead of keeping values in local variables, you store them in the namespace object and because this knows about its attributes you can do a lot more in terms of introspection than what is possible when just relying on local variables in a function call context.
See e.g. argparse's various namespace uses for an example in the stdlib.
Let me make things even clearer here. This is the entire code for the index page handler.
@app.route("/") @app.route("/editor/<channelid>") def mainpage(channelid=None): # NOTE: If we've *reduced* the required scopes, this will still force a re-login. # However, it'll be an easy login, as Twitch will recognize the existing auth. if "twitch_token" not in session or session.get("twitch_auth_scopes") != REQUIRED_SCOPES: return render_template("login.html") user = session["twitch_user"] if channelid is None: channelid = user["_id"] try: channelid = str(int(channelid)) except ValueError: # If you go to /editor/somename, redirect to /editor/equivalent-id # Bookmarking the version with the ID will be slightly faster, but # streamers will usually want to share the version with the name. users = query("helix/users", token=None, params={"login": channelid})["data"] # users is either an empty list (bad login) or a list of one. if not users: return redirect("/") return redirect("/editor/" + users[0]["id"]) if not may_edit_channel(user["_id"], channelid): return redirect(url_for("mainpage")) database.create_user(channelid) # Just in case, make sure the database has the basic structure channel = get_channel_setup(channelid) sched_tz, schedule, sched_tweet = database.get_schedule(channelid) if "twitter_oauth" in session: auth = session["twitter_oauth"] username = auth["screen_name"] twitter = "Twitter connected: " + username tweets = list_scheduled_tweets(auth["oauth_token"], auth["oauth_token_secret"], sched_tz) else: twitter = Markup("""<div id="login-twitter"><a href="/login-twitter"><img src="/static/Twitter_Social_Icon_Square_Color.svg" alt="Twitter logo"><div>Connect with Twitter</div></a></div>""") tweets = [] error = session.get("last_error_message", "") session["last_error_message"] = "" return render_template("index.html", twitter=twitter, username=user["display_name"], channel=channel, channelid=channelid, error=error, setups=database.list_setups(channelid), sched_tz=sched_tz, schedule=schedule, sched_tweet=sched_tweet, checklist=database.get_checklist(channelid), timers=database.list_timers(channelid), tweets=tweets, )
Now, show me which part of this should become a "namespace object". That namespace would be *unique* to this function - it would have no purpose or value outside of this one exact function. Show me how the use of such a namespace object would improve this code. There is no way that you would be able to share it with any other function in the entire program.
Instead of saving the variables in the local scope, you'd create a namespace object and write them directly into that object, e.g.
ns = Namespace() ns.auth = session["twitter_oauth"] ... ns.tweets = [] return render_template("index.html", ns)
Another advantage of this style is that debugging becomes much easier, since you can see the full context used by the rendering function and even provide custom output methods for the namespace content. Your local scope doesn't get cluttered up with all the namespace details.
Meaning that every *local* use of those same names now has to be adorned with "ns." to access the same thing: ns.channel = get_channel_setup(ns.channelid) Thanks, I think that's even worse than using locals(). Didn't think that was possible. :) ChrisA
On Wed, Apr 22, 2020 at 1:22 PM Chris Angelico <rosuav@gmail.com> wrote:
On Thu, Apr 23, 2020 at 6:18 AM M.-A. Lemburg <mal@egenix.com> wrote:
Instead of keeping values in local variables, you store them in the namespace object Meaning that every *local* use of those same names now has to be adorned with "ns." to access the same thing:
ns.channel = get_channel_setup(ns.channelid)
Thanks, I think that's even worse than using locals(). Didn't think that was possible. :)
Whether it's worse or not will very much be situation dependent, but my experie3nce is that if I'm passing a bunch of keyword arguments along, most of them, ate NOT used directly in the local namespace anyway. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Sun, Apr 19, 2020 at 3:37 AM Alex Hall <alex.mojaki@gmail.com> wrote:
``` File "setup.py", line 2232 self.add(Extension('_decimal', include_dirs=include_dirs, libraries=libraries, define_macros=define_macros, undef_macros=undef_macros, extra_compile_args=extra_compile_args, sources=sources, depends=depends))
I don't know about all the other examples, but in a setup.py that cries out for a more declarative approach: put all that in a dict, and call Extension('_decimal', **build_params). In general, I think setup.py modules should be a lot for declarative anyway, but at a glance, that sure looks like you'd have the same set for multiple extensions. I suspect that quite a few of the cases where folks are doing a lot of same-name kwargs passing could hve been better written with an intermediate dict. I'm not saying I don't do it myself, but honestly, it's a bit out of lazyness -- so I'm thinking that new syntax that encourages this style might be a bad idea. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On Mon, Apr 20, 2020 at 7:37 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Sun, Apr 19, 2020 at 3:37 AM Alex Hall <alex.mojaki@gmail.com> wrote:
``` File "setup.py", line 2232 self.add(Extension('_decimal', include_dirs=include_dirs, libraries=libraries, define_macros=define_macros, undef_macros=undef_macros, extra_compile_args=extra_compile_args, sources=sources, depends=depends))
I don't know about all the other examples, but in a setup.py that cries out for a more declarative approach: put all that in a dict, and call Extension('_decimal', **build_params).
Do you mean essentially writing something like: build_params = dict( include_dirs=include_dirs, libraries=libraries, define_macros=define_macros, undef_macros=undef_macros, extra_compile_args=extra_compile_args, sources=sources, depends=depends ) self.add(Extension('_decimal', **build_params) or something more than that? I'm looking at the file and I can't see what else could be done, and the above just seems detrimental.
On Mon, Apr 20, 2020 at 1:47 PM Alex Hall <alex.mojaki@gmail.com> wrote:
On Mon, Apr 20, 2020 at 7:37 PM Christopher Barker <pythonchb@gmail.com> wrote:
On Sun, Apr 19, 2020 at 3:37 AM Alex Hall <alex.mojaki@gmail.com> wrote:
``` File "setup.py", line 2232 self.add(Extension('_decimal', include_dirs=include_dirs, libraries=libraries, define_macros=define_macros, undef_macros=undef_macros, extra_compile_args=extra_compile_args, sources=sources, depends=depends))
I don't know about all the other examples, but in a setup.py that cries out for a more declarative approach: put all that in a dict, and call Extension('_decimal', **build_params).
Do you mean essentially writing something like:
build_params = dict( include_dirs=include_dirs, libraries=libraries, define_macros=define_macros, undef_macros=undef_macros, extra_compile_args=extra_compile_args, sources=sources, depends=depends ) self.add(Extension('_decimal', **build_params)
no -- that would be essentially the same, of course. And would also be addressed by the "dict uses local names" version of this proposal. Sorry, I was being lazy -- I was more referring to setup.py files in general, which I think should be more declarative, not specifically this one, that I haven't gone to find to see how it's really being used in this case. But in the general case, all those local names needed to be set to something at some point -- it *may* have been just as easy (or easier) to populate a dict, rather than set them to local names. And in that example, those looked al lot to me like a bunch of parameters that would likely be shared among multiple Extensions -- so all the more reason to have them, as a set, in a dict or something. I'm sorry I haven't taken the time to go find that example, maybe that one is as good as it gets, in which case -- <quiet_voice> never mind </quiet_voice> -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Alex Hall writes:
OK, that's fair. What about `{foo::}`?
I don't like any of these, I'm just saying why, when I can. People who are going to use the syntax should choose it.
I would personally be quite happy if I could replace:
``` class A: def __init__(self, foo, bar, spam): self.foo = foo self.spam = spam self.bar = bar ```
with something like:
``` class A: def __init__(self, foo, bar, spam): setattrs(self, **, foo, bar, spam) ```
Wouldn't you?
No, I doubt I would use that. But I apologize for wasting your time on that comment. It turns out to be incoherent and I shouldn't have mentioned it.
Please make a PR showing how you would refactor some of these.
It's not my problem, though. A syntax change is a high bar to clear. As an advocate, you need to show that this is useful enough to be worth it. A dozen examples in the whole stdlib? I'm not going to do hours of work understanding those modules and refactoring perfectly good code. I will outline what I'd look at, though. The two dicts look fine to me as they are. In fact, in general the vertically formatted examples look fine to me, and I see no reason to refactor them. If I were designing the dicts now, I probably would use enums. Something like: class LogLevel(IntEnum): CRITICAL = 6 FATAL = 5 ERROR = 4 WARN = 3 WARNING = 3 INFO = 2 DEBUG = 1 NOTSET = 0 _nameToLevel = { member.name : member for member in LogLevel } There are four calls to super (all in argparse), which seems to be due to heavy use of mixins. I don't use mixins much, so I don't know if it's avoidable or not. I would not redesign to avoid mixins, though. There are the two calls to cls (in json/__init__.py). I doubt the redundancy is avoidable. Both the calls to super and the calls to cls seem to be idiosyncratic to the authors of those particular modules, but if the plethora of same name arguments is typical of these techniques, that would be some support for a syntax change. That leaves four various other examples in the whole stdlib, not very many. Although I'm sure you'd find more with a lower limit, I doubt they're as convincing as the ones with many same name arguments. We'll see what the senior committers have to say, but I think you have an uphill battle on your hands. Steve
Stephen J. Turnbull writes:
_nameToLevel = { member.name : member for member in LogLevel }
Urk, premature expostulation there (not sure how it happened, I know I fixed that line before sending...). Just _nameToLevel = LogLevel.__members__ is fine here (LogLevel is an IntEnum, __members__ is a name : member view). In fact, often better in general: if there were aliases, that would catch all of the names, while the comprehension only catches the canonical names. Which is why I bother correcting myself. Steve
On Fri, Apr 17, 2020 at 11:21:39AM -0700, Andrew Barnert wrote:
On Apr 17, 2020, at 01:58, Steven D'Aprano <steve@pearwood.info> wrote: [...] After saving this draft, closing the email, then reopening it, I read the proposed dict the same way. So I don't think it was just a momentary glitch.
I honestly think, as you suggested at the end, that this may be just you.
You may be right. It might also be a combination of being late at night, unfamiliarity, and short identifiers. Using more meaningful identifiers is less difficult to read: {:ignorecase, 'dunder': True, :reverse} And I appreciate your anecdote about the C struct syntax. [...]
I suspect it may be because we read left-to-right in English and Python, so having the implicit blank come first is mentally like running into a pothole at high speed :-)
What examples can you think of—in English, Python, other popular languages, math notation, whatever—where there’s an infix-operator-like thing and the right token is elided and inferred implicitly? I’m sure there are some, but nothing comes to mind immediately.
Good question. Nothing comes directly to my mind either.
Meanwhile, I can think of lots of examples of the opposite, the token on the left being elided and inferred. You can read -2 and infer 2 less than 0, but not 2- to infer 0 less than 2.
For the record, that's not how negative numbers are taught in Australia. (At least not in my state.) Negative numbers are first taught as signed numbers, on a number line, so that -2 is read as the number two steps to the left rather than two steps to the right of zero. Turning that into subtraction comes later. I think that the consequence of this is that most people (at least most maths students) think of -2 as an entity in its own right, rather than "0 - 2". This is ... good and bad. It may also help to explain why so many people expect -2**2 to be +4 rather than -4 ("but you are squaring -2, right?"). But I digress. [...]
There’s no problem for the parser. Make a trivial change to the grammar to add a `":" identifier` alternative to dict display items, and nothing becomes ambiguous.
Except to the human reader, which is the only ambiguity that *really* matter when you get down to it.
Why do you do this? You snipped out the very next sentence about human readers, just to take this out of context so you can imply that human readers weren’t taken into account just so you can disagree with not taking human readers into account.
Andrew, please calm down and try giving me the benefit of the doubt that I am discussing this in good faith, not maliciously trying to manipulate people. I snipped your next sentence: "And I don’t think it would be confusing to a human reader." because my initial response was to answer it sardonically: "What am I, a teapot?" and I didn't think it would be constructive, so I took it out. Without a response, I didn't think I needed to leave the quote in. Apologies if I was overzealous in snipping relevant text, but no harm was intended. You then went on to make a comment: "It can’t possibly be a set, because colons are what make a dict display not a set display, and there are colons." and I snipped that without comment because I thought your observation was: 1. correct; 2. *obviously* correct, so it didn't need agreement; 3. *trivially* correct, so it didn't need repeating for emphasis; 4. and irrelevant to the point I was making. "Confusion with sets" is not the only possible misreading of the proposed syntax.
And you don’t even really believe this.
Don't I? I was thinking of Harold Abelson's famous quote: “Programs must be written for people to read, and only incidentally for machines to execute.” Perhaps he didn't really believe that either. [...]
There are already enough issues to discuss that fabricating issues that aren’t there just to be contrary to them really isn’t necessary.
Yes indeed. Fortunately, I'm not doing that. Please stop claiming that I am fabricating issues. Please stop straw-manning me. Please stop telling me I don't really believe things without strong evidence. Please stop accusing me of playing dumb over "obvious" analogies. Please stop accusing me of being deliberately obtuse. Please stop treating me as someone intentionally trying to manipulate the discussion in a stupidly obvious way. Please stop treating me as dishonest and a liar. If I say I don't understand an analogy, it's because I don't. Do you really think I'm the kind of guy who wants be seen as "too dumb to get the obvious" as some sort of ploy? If I don't get it, either I *am* that dumb, or it's not so obvious. -- Steven
On 17.04.20 10:53, Steven D'Aprano wrote:
I think that, as little as I like the original proposal and am not really convinced it is necessary, I think that it is better to have the explicit token (the key/parameter name) on the left, and the implicit token (blank) on the right:
key=
I suspect it may be because we read left-to-right in English and Python, so having the implicit blank come first is mentally like running into a pothole at high speed :-)
For function calls, I think it is much easier to infer the parameter name than the argument name. That's because usually functions are meant to generalize, so their parameter names denote a more general concept. Variable names on the other hand refer to some specific data that's used throughout the program. For example imagine the function def call(*, phone_number, ...) # here '...' means any number of additional parameters Then in the code we might have the following variables: phone_number_jane = '123' phone_number_john = '456' Using any of these variables to invoke the `call` function, it is pretty obvious which parameter it should be assigned to (for the human reader at least, not the compiler): call(=phone_number_jane) If on the other hand the parameter was specified it would be ambiguous which variable to use: call(phone_number=) # should we call Jane or John? Now this doesn't relate to the original proposal yet, but imagine this is inside some other function and we happen to have another variable `phone_number` around: def create_conference_call_with_jane_and_john(phone_number): """`phone_number` will be placed in a conference call with Jane and John.""" phone_number_jane = '123' phone_number_john = '456' call(phone_number=) # Whom do we call here? Yes, there is a variable `phone_number` around but why use this specifically? `phone_number_jane` and `phone_number_john` are also phone numbers and `call` only asks for a phone number in general, so it's ambiguous which one to use. If on the other hand I read `call(=phone_number)` then I know `phone_number` is a phone number and `call` expects one for its `phone_number` parameter so it's pretty clear how the assignment should be made. Another example: purchases = load_data('purchases.txt') sales = load_data('sales.txt') data = load_data('user_data.txt') process(data=) # We have multiple data sets available, so in the face of ambiguity, ... # ... refuse the temptation to guess. Functions parameters usually represent a concept and arguments then represent the specific data. Often (not always) specific data can be assigned a concept but the other way round is almost always ambiguous.
On 2020-04-18 01:32, Dominik Vilsmeier wrote:
On 17.04.20 10:53, Steven D'Aprano wrote:
I think that, as little as I like the original proposal and am not really convinced it is necessary, I think that it is better to have the explicit token (the key/parameter name) on the left, and the implicit token (blank) on the right:
key=
I suspect it may be because we read left-to-right in English and Python, so having the implicit blank come first is mentally like running into a pothole at high speed :-)
For function calls, I think it is much easier to infer the parameter name than the argument name. That's because usually functions are meant to generalize, so their parameter names denote a more general concept. Variable names on the other hand refer to some specific data that's used throughout the program. For example imagine the function
def call(*, phone_number, ...) # here '...' means any number of additional parameters
Then in the code we might have the following variables:
phone_number_jane = '123' phone_number_john = '456'
Using any of these variables to invoke the `call` function, it is pretty obvious which parameter it should be assigned to (for the human reader at least, not the compiler):
call(=phone_number_jane)
If on the other hand the parameter was specified it would be ambiguous which variable to use:
call(phone_number=) # should we call Jane or John?
Now this doesn't relate to the original proposal yet, but imagine this is inside some other function and we happen to have another variable `phone_number` around:
def create_conference_call_with_jane_and_john(phone_number): """`phone_number` will be placed in a conference call with Jane and John."""
phone_number_jane = '123' phone_number_john = '456'
call(phone_number=) # Whom do we call here?
Yes, there is a variable `phone_number` around but why use this specifically? `phone_number_jane` and `phone_number_john` are also phone numbers and `call` only asks for a phone number in general, so it's ambiguous which one to use.
If on the other hand I read `call(=phone_number)` then I know `phone_number` is a phone number and `call` expects one for its `phone_number` parameter so it's pretty clear how the assignment should be made.
Another example:
purchases = load_data('purchases.txt') sales = load_data('sales.txt') data = load_data('user_data.txt')
process(data=) # We have multiple data sets available, so in the face of ambiguity, ... # ... refuse the temptation to guess.
Functions parameters usually represent a concept and arguments then represent the specific data. Often (not always) specific data can be assigned a concept but the other way round is almost always ambiguous.
I disagree. The rule is very simple: use the variable with the matching name. You can have only a name before the '=', but normally you can have an expression of arbitrary complexity after it. IMHO, it's simpler to omit the RHS than to omit the LHS whilst also restricting it to a name.
Andrew Barnert via Python-ideas writes:
But I don’t see why that rules out the “bare colon” form that I and someone else apparently both proposed in separate sub threads of this thread:
{ :a, "b": x, :c }
This sort of reminds me of the "leading colon" convention for keywords in Lisp, where a keyword is *defined* to be a symbol whose value is itself. Steve
On Thu, 2020-04-16 at 21:21 -0700, Andrew Barnert via Python-ideas wrote:
On Apr 16, 2020, at 20:48, oliveira.rodrigo.m@gmail.com wrote:
In Javascript ES6 they don't have sets built like python so `{}` always refers to objects being constructed. It does indeed support implicit key: value pairs, so in ES6 `{ a: a, b: x, c: c }` is equivalent to `{ a, b: x, c }`. This is okay for Javascript users because they would not thought it as sets and the only obvious assumption to make is that parameters are being implicitly assigned to members. This is not the case in Python so I would refrain from changing dictionary literals syntax.
Obviously the exact same syntax as ES6 doesn’t work in Python, because it would be not just confusing but ineradicably ambiguous with sets.
But I don’t see why that rules out the “bare colon” form that I and someone else apparently both proposed in separate sub threads of this thread:
{ :a, "b": x, :c }
as shorthand for:
{ "a": a, "b": x, "c": c }
It likely does not matter, but one argument in favor of only applying such syntax to function calls is that in function calls there is no ambiguity that "a" is a string: dict(a=a) cannot possibly be thought to mean: {a: a} instead of: {"a": a} That may be a pretty moot point, but if you have a dictionary that does not use strings as keys things might just a tiny bit strange? - Sebastian
There’s no problem for the parser. Make a trivial change to the grammar to add a `":" identifier` alternative to dict display items, and nothing becomes ambiguous.
And I don’t think it would be confusing to a human reader. It can’t possibly be a set, because colons are what make a dict display not a set display, and there are colons.
And I think extending dict display syntax is more powerful and less disruptive than extending call syntax. It means you can refactor the {…} in a **{…} if the call gets too unwieldy; it means calls and other places with ** unpacking remain consistent; etc.
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/IEPZ7W... Code of Conduct: http://python.org/psf/codeofconduct/
On Fri, Apr 17, 2020 at 03:46:31AM -0000, oliveira.rodrigo.m@gmail.com wrote:
@Steven D'Aprano see that it doesn't actually have to look like a pair, it doesn't need to be one at all. Just like in:
```python def f(a): ... f(x) ```
`x` is implicitly assigned with `a`
We're not talking about positional arguments here. If you want to include positional arguments in the analysis, then we already have a perfectly good way to write function calls without repeating outselves: # Instead of this: function(spam=spam, eggs=eggs, cheese=cheese) # Just do this: function(spam, eggs, cheese) And we're done, the problem is solved, and no new syntax is needed. But using positional arguments misses the point that, for many purposes, keyword arguments have advantages and we want to use them. Using positional syntax may be either undesirable or not possible. So my comment needs to be read with the understanding that we are specifically talking about keyword, not positional, syntax. And in keyword syntax, we need a name=value pair. At least your suggested syntax gives a strong clue that we are dealing with a name=value pair: `name=`. Whereas the Javascript and Rust syntax gives us *no* visual clue at all that we're dealing with a name=value pair, it looks like values are assigned by position: `name`. I think this is bad. Consider a function `def func(spam, eggs)`, and consider the function call `func(eggs, spam)`. It makes a big difference whether the values are assigned using positional rules: eggs --> parameter 0, "spam" spam --> parameter 1, "eggs" or keyword rules: # func(eggs, spam) is shortcut for eggs=eggs, spam=spam. eggs --> parameter "eggs" spam --> parameter "spam" Anyway, I realise that you are not proposing the Javascript/Rust style syntax, so we can move on :-) -- Steven
I could absolutely see a lot of use for this when it comes to rebinding names in callbacks and the like. for i in range(10): callLater(delay, lambda i=: print(i)) But with this being a very common scenario for such a feature being needed, we'd see a lot of confusion with how much that looks like a walrus operator. I am +1 on the feature, but -1 on the syntax. On Thu, Apr 16, 2020 at 12:45 PM Steven D'Aprano <steve@pearwood.info> wrote:
Do any other languages already have this feature?
Can you show some actual real-life code that would benefit from this, as opposed to pretend code like:
foo(bar=, qux=)
I think that if I had seen this syntax as a beginner, I would have had absolutely no idea how to interpret it. I probably would have decided that Python was an unreadably cryptic language and gone on to learn something else.
Of course, with 20+ years of experience reading and writing code, I know better now. I would interpret it as setting bar and qux to some kind of Undefined value.
I am very sympathetic to the rationale:
"it is quite common to find code that forwards keyword parameters having to re-state keyword arguments names"
and I've discussed similar/related issues myself, e.g. here:
https://mail.python.org/pipermail/python-list/2018-February/881615.html
But I am not convinced that adding magic syntax to implicitly guess the value wanted as argument if it happens to match the parameter name is a good design feature.
Is being explicit about the value that you are passing to a parameter really such a burden that we need special syntax to avoid stating what value we are using as the argument?
I don't think it is. And I would far prefer to read explicit code like this:
# Slightly modified from actual code. self.do_something( meta=meta, dunder=dunder, private=private, invert=invert, ignorecase=ignorecase, )
over the implicit version:
# Looks like code I haven't finished writing :-( self.do_something(meta=, dunder=, private=, invert=, ignorecase=)
Out of the millions of possible values we might pass, I don't think that the func(spam=spam) case is so special that we want to give it special syntax.
-- Steven _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VCCQ6B... Code of Conduct: http://python.org/psf/codeofconduct/
-- CALVIN SPEALMAN SENIOR QUALITY ENGINEER cspealma@redhat.com M: +1.336.210.5107 [image: https://red.ht/sig] <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Apr 16, 2020, at 10:20, Calvin Spealman <cspealma@redhat.com> wrote:
I could absolutely see a lot of use for this when it comes to rebinding names in callbacks and the like.
for i in range(10): callLater(delay, lambda i=: print(i))
But that one wouldn’t be handled by the proposal. It’s explicitly only allowing the name= syntax on arguments in a call, not on parameters in a (lambda or def) definition. And if you expanded the proposal to include both your feature and the one in the proposal, it might read ambiguously, or at least confusingly: callLater(lambda i=: I, delay=) Can you tell that one is passing a keyword argument while the other is stashing a default value for a positional-or-keyword parameter, without having to stop and think through the parse? (Even with added parens somewhere?) Maybe with enough familiarity, it wouldn’t be a problem. But maybe that in itself enough reason to just include one or the other new feature, wait until people are familiar with it, and then see if the new one looks confusing or not? (That’s assuming we’re eventually likely to implement both of these, when it’s not actually clear many people want either one of them…)
This proposal is an alternative to Rodrigo's "Keyword arguments self-assignment" thread. Rodrigo, please feel free to mine this for useful nuggets in your PEP. (I don't claim to have invented the syntax -- I think it might have been Alex Hall?) Keyword Unpacking Shortcut -------------------------- Inside function calls, the syntax **{identifier [, ...]} expands to a set of `identifier=identifier` argument bindings. This will be legal anywhere inside a function call that keyword unpacking would be legal. That means that syntactically it will legal to mix this between other keyword arguments: # currently legal func(z=5, **d1, x=0, y=6, **d2, w=7) # propose to allow Keyword Unpacking Shortcut as well func(z=5, **d1, x=0, **{u, v}, y=6, **d2, w=7) # which would be equivalent to: func(z=5, **d1, x=0, u=u, v=v, y=6, **d2, w=7) The order of unpacking should be the same as the order of unpacking regular dict unpacking: func(**{a, b, c}) func(**{'a': a, 'b': b, 'c': c}) should unpack the parameters in the same order. Interpretation -------------- There are at least two ways to interpret the shortcut: 1. It is like unpacking a dict where only the keys (identifiers) are given, and the values are implied to be exactly the same as the keys. 2. It is like unpacking a dict formed from a set of identifiers, used as both the parameter name and evaluated as the argument value. (This does not imply that the shortcut need actually generate either a dict or a set. The actual implementation will depend on the compiler.) Either way, the syntax can be justified: * Why use double-star `**`? Because it is a form of dict unpacking. * Why use curly brackets (braces) `{}`? It's a set of identifiers to be auto-filled with values matching the identifier. Put them together and you get `**{ ... }` as the syntax. Examples -------- The function call: function(**{meta, invert, dunder, private, ignorecase}) expands to: function(meta=meta, invert=invert, dunder=dunder, private=private, ignorecase=ignorecase ) Because the keyword unpacking shortcut can be used together with other unpacking operators, it can be (ab)used for quite complex function calls. `Popen` has one of the most complex signatures in the Python standard library: https://docs.python.org/3.8/library/subprocess.html#subprocess.Popen Here is an example of mixing positional and keyword arguments together with the proposed keyword unpacking shortcut in a single call to Popen. subprocess.Popen( # Positional arguments. args, bufsize, executable, *stdfiles, # Keyword arguments. shell=True, close_fds=False, cwd=where, creationflags=flags, env={'MYVAR': 'some value'}, **textinfo, # Keyword unpacking shortcut. **{preexec_fn, startupinfo, restore_signals, pass_fds, universal_newlines, start_new_session} ) which will expand to: subprocess.Popen( # Positional arguments. args, bufsize, executable, *stdfiles, # Keyword arguments. shell=True, close_fds=False, cwd=where, creationflags=flags, env={'MYVAR': 'some value'}, **textinfo, # Keyword unpacking shortcut expands to: preexec_fn=preexec_fn, startupinfo=startupinfo, restore_signals=restore_signals, pass_fds=pass_fds, universal_newlines=universal_newlines, start_new_session=start_new_session ) Note that plain keyword arguments such as: cwd=where have the advantage that the key and value are shown directly in the function call, but can be excessively verbose when there are many of them, especially when the argument value duplicates the parameter name, e.g. `start_new_session=start_new_session` etc. On the other hand, plain keyword unpacking: **textinfo is terse, but perhaps too terse. Neither the keys nor the values are immediately visible. Instead, one must search the rest of the function or module for the definition of `textinfo` to learn which parameters are being filled in. The keyword unpacking shortcut takes a middle ground. Like explicit keyword arguments, and unlike the regular `**textinfo` unpacking, the parameter names are visible in the function call. But it avoids the repetition and redundancy of repeating the parameter name as in `start_new_session=start_new_session`. Backwards compatibility ----------------------- The syntax is not currently legal so there are no backwards compatibility concerns. Possible points of confusion ---------------------------- Sequence unpacking is currently allowed for sets: func(*{a, b, c}) # Unpacks a, b, c in arbitrary order. Anyone intending to use the keyword unpacking shortcut but accidentally writing a single star instead of a double will likely end up with confusing failures rather than an explicit exception. However, the same applies to dict unpacking: func(*{'meta': True', 'invert': False, 'dunder': True}) which will unpack the keys as positional arguments. Experience strongly suggests this is a rare error in practice, and there is little reason to expect that keywords unpacking shortcut will be any different. This could be handled by a linter. -- Steven
On Apr 17, 2020, at 23:18, Steven D'Aprano <steve@pearwood.info> wrote:
Keyword Unpacking Shortcut --------------------------
Inside function calls, the syntax
**{identifier [, ...]}
expands to a set of `identifier=identifier` argument bindings.
This will be legal anywhere inside a function call that keyword unpacking would be legal.
Which means that you can’t just learn ** unpacking as a single consistent thing that’s usable in multiple contexts with (almost) identical syntax and identical meaning, you have to learn that it has an additional syntax with a different meaning in just one specific context, calls, that’s not legal in the others. Each special case like that makes the language’s syntax a little harder to internalize, and it’s a good thing that Python has a lot fewer such special cases than, say, C. Worse, this exact same syntax is a set display anywhere except in a ** in a call. Not only is that another special case to learn about the differences between set and dict displays, it also means that if you naively copy and paste a subexpression from a call into somewhere else (say, to print the value of that dict), you don’t get what you wanted, or a syntax error, or even a runtime error, you get a perfectly valid but very different value.
On the other hand, plain keyword unpacking:
**textinfo
is terse, but perhaps too terse. Neither the keys nor the values are immediately visible. Instead, one must search the rest of the function or module for the definition of `textinfo` to learn which parameters are being filled in.
You can easily put the dict right before the call, and when you don’t, it’s usually because there was a good reason. And there are good reasons. Ideally you shouldn’t have any function calls that are so hairy that you want to refractor them, but the the existence of libraries you can’t control that are too huge and unwieldy is the entire rationale here. Sometimes it’s worth pulling out a group of related parameters to a “launch_params” or “timeout_and_retry_params” dict, or even to a “build_launch_params” method, not just for readability but sometimes for flexibility (e.g., to use it as a cache or stats key, or to give you somewhere to hook easily in the debugger and swap out the launch_params dict.
Backwards compatibility -----------------------
The syntax is not currently legal so there are no backwards compatibility concerns.
The syntax is perfectly legal today. The syntax for ** unpacking in a call expression takes any legal expression, and a set display is a legal expression. You can see this by calling compile (or, better, dis.dis) on the string 'spam(**{a, b, c})'. The semantics will be a guaranteed TypeError at runtime unless you’ve done something pathological, so almost surely nobody’s deployed any code that depends on the existing semantics. But that’s not the same as the syntax not being legal. And, outside of that trivial backward compatibility nit, this raises a bunch of more serious issues. Running Python 3.9 code in 3.8 would do the wrong thing, but maybe not wrong enough to break your program visibly, which could lead to some fun debugging sessions. That’s not a dealbreaker, but it’s definitely better for new syntax to raise a syntax error in old versions, if possible. And of course existing linters, IDEs, etc. will misunderstand the new syntax (which is worse than failing to parse it) until they’re taught the new special case. This also raises an implementation issue. The grammar rule to disambiguate this will probably either be pretty hairy, or require building a parallel fork of half the expression tree so you can have an “expression except for set displays” node. Or there won’t be one, and it’ll be done as a special case post-parse hack, which Python uses sparingly. But all of that goes right along with the human confusion. If the same syntax can mean two different things in different contexts, it’s harder to internalize a usable approximate version of the grammar. For something important enough, that may be worth it, but I don’t think the benefits of this proposal reach that bar.
On Sat, Apr 18, 2020 at 12:42:10AM -0700, Andrew Barnert wrote:
Inside function calls, the syntax
**{identifier [, ...]}
expands to a set of `identifier=identifier` argument bindings.
This will be legal anywhere inside a function call that keyword unpacking would be legal.
Which means that you can’t just learn ** unpacking as a single consistent thing that’s usable in multiple contexts with (almost) identical syntax and identical meaning, you have to learn that it has an additional syntax with a different meaning in just one specific context, calls, that’s not legal in the others.
Um, yes? I think. I'm afraid your objection is unclear to me. Obviously this would be one more thing to learn, but if the benefit is large enough, it would be worthwhile. It would also be true whether we spell it using the initial suggestion, or using mode-shift, or by adding a new way to create dicts: f(meta, dunder=, reverse=) f(meta, *, dunder, reverse) f(meta, **{:dunder, :reverse}) f(meta, **{dunder, reverse}) I'm not really sure I understand your comment about dict unpacking being "usable in multiple contexts with (almost) identical syntax and identical meaning". Can you give some examples? I know that dict unpacking works in function calls: f(**d) and I know it doesn't work in assignments: a, b = **d # What would this even mean? or in list-displays, etc. It *does* work in dict-displays: d = {'a': None, **mapping} but I either didn't know it, or had forgotten it, until I tested it just now. (It quite surprised me too.) Are there any other contexts where this would work? There's probably no reason why this keyword shortcut couldn't be allowed in dict-displays too: d = {'a': None, **{b, c, d}) or even as a new dict "literal": d = **{meta, dunder, reverse} if there is call for it. Personally, I would be conservative about allowing it in other contexts, as we can always add it later, but it's much harder to remove it if it were a mistake. This proposal is only about allowing it in a single context, function calls. [...]
Worse, this exact same syntax is a set display anywhere except in a ** in a call.
You say "worse", I say "Better!" It's a feature that this looks something like a set: you can read it as "unpack this set of identifiers as parameter:value arguments". It's a feature that it uses the same `**` double star as dict unpacking: you can read it as unpacking a dict where the values are implied. It is hardly unprecedented that things which look similar are not always identical, especially when dealing with something as basic as a sequence of terms in a comma-separated list: math, sys, functools, itertools, os It's a tuple! Except when inside parentheses directly following an expression, or an import statement: import math, sys, functools, itertools, os obj.attribute.method(math, sys, functools, itertools, os)
Not only is that another special case to learn about the differences between set and dict displays, it also means that if you naively copy and paste a subexpression from a call into somewhere else (say, to print the value of that dict), you don’t get what you wanted, or a syntax error, or even a runtime error, you get a perfectly valid but very different value.
If you naively copy and paste the curly bracket part: f(meta, **{dunder, reverse}) print({dunder, reverse}) you get to see the values in a set. Is that such a problem that it should kill the syntax? There's a limit to how naive a user we need to care about in the language. We don't have to care about preventing every possible user error.
On the other hand, plain keyword unpacking:
**textinfo
is terse, but perhaps too terse. Neither the keys nor the values are immediately visible. Instead, one must search the rest of the function or module for the definition of `textinfo` to learn which parameters are being filled in.
You can easily put the dict right before the call, and when you don’t, it’s usually because there was a good reason.
Right. I'm not saying that dict unpacking is a usability disaster. I'm just pointing out that it separates the parameters from where they are being used. Yes, it could be one line away, or it could be buried deeply a thousand lines away, imported from another module, which you don't have the source code to. I intentionally gave a real (or at least, real-ish) example using a real function from the standard library, and real parameter names. Without looking in the docs, can you tell what parameters are supplied by the `**textinfo` unpacking? I know I can't, and I wrote the damn thing! (I had to check the function signature to remind me what they were.) Given: Popen( ..., **textinfo, ...) Popen( ..., **{encoding, errors, text}, ...) I think that the second one is clearly superior in respect to showing the parameter names directly in place where they are used, while the first is clearly superior for brevity and terseness.
And there are good reasons. Ideally you shouldn’t have any function calls that are so hairy that you want to refractor them, but the the existence of libraries you can’t control that are too huge and unwieldy is the entire rationale here.
I wouldn't say the entire rationale. For example, I have come across this a lot: def public_function(meta, reverse, private, dunder): do some pre-processing result = _private_function( meta=meta, reverse=reverse, private=private, dunder=dunder ) do some post-processing return result Changing the signature of `public_function` to take only a kwargs is not an option (for reasons I hope are obvious, but if not I'm happy to explain). Writing it like this instead just moves the pain without eliminating it: d = dict(meta=meta, reverse=reverse, private=private, dunder=dunder ) result = _private_function(**d) So there's a genuine pain point here that regular keyword unpacking doesn't solve. [...]
Backwards compatibility -----------------------
The syntax is not currently legal so there are no backwards compatibility concerns.
The syntax is perfectly legal today.
Okay, I misspoke. Miswrote. It's not currently legal to use a set in dict unpacking. I will re-iterate that this proposal does not construct a set. It just looks a bit like a set, in the same way that all of these have things which look like a bit like tuples but aren't: [a, b, c] func(a, b, c) import sys, os, collections except ValueError, TypeError, ImportError and the same way that subscripting looks a bit like a list: mydict['key'] # Not actually a list ['key'] [...]
Running Python 3.9 code in 3.8 would do the wrong thing, but maybe not wrong enough to break your program visibly, which could lead to some fun debugging sessions. That’s not a dealbreaker, but it’s definitely better for new syntax to raise a syntax error in old versions, if possible.
I don't think so. You would get a TypeError. py> func(**{a, b, c}) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: func() argument after ** must be a mapping, not set
And of course existing linters, IDEs, etc. will misunderstand the new syntax (which is worse than failing to parse it) until they’re taught the new special case.
Do you have any examples?
This also raises an implementation issue. The grammar rule to disambiguate this will probably either be pretty hairy, or require building a parallel fork of half the expression tree so you can have an “expression except for set displays” node. Or there won’t be one, and it’ll be done as a special case post-parse hack, which Python uses sparingly.
Obviously if the implementation is hairy enough, that counts against the proposal. But given that there is no realistic chance of this going into Python 3.9 (feature freeze is not far away), and Python 3.10 will be using the new PEG parser, let's not rule it out *just* yet. -- Steven
On Sat, Apr 18, 2020 at 8:18 AM Steven D'Aprano <steve@pearwood.info> wrote:
This proposal is an alternative to Rodrigo's "Keyword arguments self-assignment" thread.
Rodrigo, please feel free to mine this for useful nuggets in your PEP.
(I don't claim to have invented the syntax -- I think it might have been Alex Hall?)
lol everyone thinks I invented things I didn't... The idea was first suggested by Dominik: https://mail.python.org/archives/list/python-ideas@python.org/message/S44VDM... I just suggested making the brackets curly to make it a bit more dict-like.
Keyword Unpacking Shortcut --------------------------
Inside function calls, the syntax
**{identifier [, ...]}
expands to a set of `identifier=identifier` argument bindings.
This will be legal anywhere inside a function call that keyword unpacking would be legal. That means that syntactically it will legal to mix this between other keyword arguments:
# currently legal func(z=5, **d1, x=0, y=6, **d2, w=7)
# propose to allow Keyword Unpacking Shortcut as well func(z=5, **d1, x=0, **{u, v}, y=6, **d2, w=7)
# which would be equivalent to: func(z=5, **d1, x=0, u=u, v=v, y=6, **d2, w=7)
My issue with this, and maybe it's what Andrew is also trying to say, is that it breaks our usual assumptions about composing expressions. `{u, v}` is an expression, it represents a set, and it always represents that wherever you put it. Under your proposal, these two programs are both valid syntax with different meanings: f(**{u, v}) x = {u, v} f(**x) Is there anything else similar in the language? Obviously there are cases where the same text has different meanings in different contexts, but I don't think you can ever refactor an expression (or text that looks like an expression) into a variable and change its meaning while keeping the program runnable. This proposal makes it harder for beginners to understand how a program is interpreted. It breaks the simple mental model where building blocks are combined in a consistent fashion into larger parts. The example above looks a bit dumb, but maybe users will try: ``` if flag: kwargs = {u, v} else: kwargs = {w, x} f(**kwargs) ``` Which is valid syntax but is wrong. Then they might try changing that to: f(**({u, v} if flag else {w, x})) which is suddenly invalid syntax. This is a very weird user experience. On that note, is this valid? f(**({u, v}))
On Sat, Apr 18, 2020 at 02:13:51PM +0200, Alex Hall wrote:
My issue with this, and maybe it's what Andrew is also trying to say, is that it breaks our usual assumptions about composing expressions. `{u, v}` is an expression, it represents a set, and it always represents that wherever you put it. Under your proposal, these two programs are both valid syntax with different meanings:
f(**{u, v})
x = {u, v} f(**x)
True, and that does count as a (minor?) point against it. But not one that I think should rule it out.
Is there anything else similar in the language? Obviously there are cases where the same text has different meanings in different contexts, but I don't think you can ever refactor an expression (or text that looks like an expression) into a variable and change its meaning while keeping the program runnable.
Of course! There are many ways that this can occur. f(a, b, c) x = a, b, c f(x) are very different things. Here's another one: import name x = name import x Here's a third: del fe, fi, fo, fum x = fe, fi, fo, fum del x Here's an example so obvious (and trivial) that I'm almost embarrassed to include it: seq[index] x = [index] seqx If code is made of composable building blocks, those blocks aren't *characters*. What makes a composable block is dependent on context. To make up for how trivial the previous example was, here's a complicated one: for item in items: if item: continue do_stuff() versus: def block(item): if item: continue do_stuff() for item in items: block() I have often wished I could refactor continue and break into functions, but you can't :-( Although in this case at least you get a syntax error when you try. Here's an example with sequence unpacking: a = [1, 2, *seq] x = 2, *seq a = [1, x] Another example: class MyClass(metaclass=MyMeta) metaclass = MyMeta class MyClass(metaclass) That's just a special case of keyword notation itself: func(x=expr) x = expr func(x) Those are not the same, unless the first positional argument happens to be named `x`. And one final example: class C: def method(self): pass versus: def method(self): pass class C: method This is not an exhaustive list, just the first few things that came to my mind.
This proposal makes it harder for beginners to understand how a program is interpreted. It breaks the simple mental model where building blocks are combined in a consistent fashion into larger parts.
I **LOVE** the ability to reason about code with a simple mental model of building blocks. I would consider it a very important property of syntax. But it is not an absolute requirement in all things. I mean, we wouldn't want to say that function call syntax `f(x, y, z)` is a disaster because it looks like we combined a name with a tuple. I acknowledge that "cannot compose this" is a point against it, but I deny that it should be a flat out disqualification. There are lots of things in Python that cannot be trivially composed.
The example above looks a bit dumb, but maybe users will try:
``` if flag: kwargs = {u, v} else: kwargs = {w, x} f(**kwargs) ```
I think this point will apply to all(?) such syntactic proposals. I don't think this scenario is too different from this: if flag: f(u=expr1, v=expr2) else: f(w=expr3, x=expr4) If you want to refactor that, you can't do this: f((u=expr1, v=expr2) if flag else (w=expr3, x=expr4)) but you can just use regular dict unpacking: d = dict(u=expr1, v=expr2) if flag else dict(w=expr3, x=expr4) f(**d)
Which is valid syntax but is wrong. Then they might try changing that to:
f(**({u, v} if flag else {w, x}))
which is suddenly invalid syntax.
Not invalid syntax, but it's still wrong. You'll get a TypeError when trying to `**` unpack a set instead of a dict.
This is a very weird user experience. On that note, is this valid?
f(**({u, v}))
I would expect that to parse as regular old dict unpacking, and give a TypeError at runtime. -- Steven
On Sun, Apr 19, 2020 at 1:04 AM Steven D'Aprano <steve@pearwood.info> wrote:
And one final example:
class C: def method(self): pass
versus:
def method(self): pass
class C: method
Indeed. However, if you put "method = method", it would actually work (modulo __qualname__ and no-arg super, I think). This is an important distinction: are you attempting to compose syntactic symbols, or self-contained expressions? Programming languages have infinite power but finite complexity precisely because you can compose expressions. Alex referred to refactoring "text that looks like an expression", and on that point, I absolutely agree with Steven here: there are MANY places where "text that looks like an expression" can have vastly different meanings in different contexts. But if it actually truly is an expression in both contexts, then it usually will have the same meaning. (That meaning might be coloured by its context - eg name lookups depend on where you put the code - but the meaning is still "look up this name".) And even when it isn't always an expression, the meanings are often closely related - the expression "spam[ham]" and the statement "spam[ham] = 1" are counterparts. So I think that the false parallel here IS a strike against the proposal.
f(**{u, v})
x = {u, v} f(**x)
Logically, inside the argument list, you have a comma-separated list of expressions. Aside from needing to parenthesize tuples, I'm pretty sure any expression will be valid. And if you put "**" followed by an expression, that means "evaluate this expression and then unpack the result into the args". Yes, I know there are some oddities ('with' statements and parentheses come to mind). But those oddities are friction points for everyone who runs into them, they're inconsistencies, they're nuisances. Not something to seek more of. (Side point: Is class scope the only place in Python where you would logically want to write "method = method" as a statement? It's also legal and meaningful at top-level, and has the equivalent effect of importing a built-in into the current namespace, but I've never actually had the need to do that.) ChrisA
On Sun, Apr 19, 2020 at 01:24:14AM +1000, Chris Angelico wrote: [...]
Alex referred to refactoring "text that looks like an expression", and on that point, I absolutely agree with Steven here: there are MANY places where "text that looks like an expression" can have vastly different meanings in different contexts.
[...]
So I think that the false parallel here IS a strike against the proposal.
I don't deny it. I just hope people can understand that language design should not be "one strike and you're out". Sometimes compromises are needed. It would be good if we had something that was neither excessive verbose nor painfully terse, required nothing but ASCII, that was similar enough to dict unpacking to suggest a connection, but without being confusable to anything else even to newbies, was self-descriptive without needing much or any explanation, cured Covid-19, brought peace to the Middle East, ended poverty, and improved the level of political discourse on social media. But I fear we may have to relax the requirements somewhat :-) My requirements for this syntax are: - it's a form of dict unpacking, so it ought to use `**` rather than overloading some other, unrelated, symbol like (say) `@` or `%`; - it ought to use a simple list of identifiers without needing extra sigils or markers on each one, say: `meta, dunder, private` rather than `:meta, :dunder, :private` or `meta=, dunder=, private=` - it ought to be something "like" an expression, rather than a mode that has to be turned on and then applies to the end of the function call; - if it actually is an expression, that's a nice bonus, but it's not essential; - it ought to look Pythonic, which is subjective; - or at least not look like "line noise", which is still subjective but it's probably easier to get agreement on what is ugly than on what is beautiful :-) It is a lot to ask. If anyone else can think of an alternative, please do speak up! -- Steven
It would be good if we had something that was neither excessive verbose nor painfully terse, required nothing but ASCII, that was similar enough to dict unpacking to suggest a connection, but without being confusable to anything else even to newbies, was self-descriptive without needing much or any explanation, cured Covid-19, brought peace to the Middle East, ended poverty, and improved the level of political discourse on social media.
But I fear we may have to relax the requirements somewhat :-)
My requirements for this syntax are:
...
- it ought to be something "like" an expression, rather than a mode that has to be turned on and then applies to the end of the function call;
- if it actually is an expression, that's a nice bonus, but it's
not essential;
But the problem here is not that your proposal doesn't meet all the requirements, it's that you have different requirements from us. You are requiring something that looks like an expression even if it isn't. Chris, Andrew and I see "something that's like an expression but isn't" as a point against, not a point in favour. Side note: what does this mean in your proposal? I'm guessing it's a SyntaxError? foo(a, b, **{u, v}, c, d)
On Sun, Apr 19, 2020 at 10:19:41AM +0200, Alex Hall wrote:
But the problem here is not that your proposal doesn't meet all the requirements, it's that you have different requirements from us. You are requiring something that looks like an expression even if it isn't.
Please read what I said again, because that's a mischaracterisation of what I said.
Side note: what does this mean in your proposal? I'm guessing it's a SyntaxError?
foo(a, b, **{u, v}, c, d)
Yes, because we still have a restriction that positional arguments come before keyword arguments. If that restriction is ever relaxed, then this may become legal. -- Steven
On Sun, Apr 19, 2020 at 11:49 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Sun, Apr 19, 2020 at 10:19:41AM +0200, Alex Hall wrote:
But the problem here is not that your proposal doesn't meet all the requirements, it's that you have different requirements from us. You are requiring something that looks like an expression even if it isn't.
Please read what I said again, because that's a mischaracterisation of what I said.
I don't think it is. You said 'it ought to be something "like" an expression', and then that actually being an expression is not essential. How is it unfair to rephrase that as "something that looks like an expression even if it isn't"? Note the 'if' - I didn't say "something that looks like an expression but isn't". I did say that shortly after, but that's specifically referring to your current proposal, of which I think it's a fair description.
Side note: what does this mean in your proposal? I'm guessing it's a SyntaxError?
foo(a, b, **{u, v}, c, d)
Yes, because we still have a restriction that positional arguments come before keyword arguments. If that restriction is ever relaxed, then this may become legal.
So, as Dominik pointed out, why keep this restriction? Why not say that every lone name after the `**` is an auto-named keyword? What is the advantage of being able to turn the mode off with `}`? More concretely, what is the advantage of your proposal in these pairs? ``` foo(bar, spam=True, **{thing, stuff}) foo(bar, spam=True, **, thing, stuff) foo(bar, spam=True, **{thing, stuff}, other=False) foo(bar, spam=True, **, thing, stuff, other=False) foo(bar, spam=True, **{thing, stuff}, other=False, **{baz}) foo(bar, spam=True, **, thing, stuff, other=False, baz) ```
On Sun, Apr 19, 2020 at 12:09:54PM +0200, Alex Hall wrote:
Please read what I said again, because that's a mischaracterisation of what I said.
I don't think it is. You said 'it ought to be something "like" an expression', and then that actually being an expression is not essential. How is it unfair to rephrase that as "something that looks like an expression even if it isn't"?
I don't *require* it to look like an expression when it's not. What I said was that it should look like an expression *rather than a mode*. But whatever -- if you want to stick with your wording, I'm not going to argue that point any further. There are more important things to discuss. Namely your side-note:
Side note: what does this mean in your proposal? I'm guessing it's a SyntaxError?
foo(a, b, **{u, v}, c, d)
Yes, because we still have a restriction that positional arguments come before keyword arguments. If that restriction is ever relaxed, then this may become legal.
So, as Dominik pointed out, why keep this restriction?
Because that's the conservative choice that doesn't lock out future language development unnecessarily. Today, we allow positional unpacking after keyword arguments: func(a, b=1, c=2, *args) Perhaps some day we will want to allow regular positional arguments after keywords too. If we don't *need* to rule that out, why do so unnecessarily? All else being equal, we should prefer the syntax that doesn't rule out future development. -- Steven
On Sun, Apr 19, 2020 at 5:46 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Sun, Apr 19, 2020 at 01:24:14AM +1000, Chris Angelico wrote:
[...]
Alex referred to refactoring "text that looks like an expression", and on that point, I absolutely agree with Steven here: there are MANY places where "text that looks like an expression" can have vastly different meanings in different contexts.
[...]
So I think that the false parallel here IS a strike against the proposal.
I don't deny it. I just hope people can understand that language design should not be "one strike and you're out". Sometimes compromises are needed.
Yes, exactly. One strike doesn't mean it's out, and with a proposal like this, it's a matter of looking at a whole lot of imperfect alternatives and seeing which tradeoffs we want to go with. (And status quo is one such alternative, with tradeoffs of "names need to be duplicated".)
It would be good if we had something that was neither excessive verbose nor painfully terse, required nothing but ASCII, that was similar enough to dict unpacking to suggest a connection, but without being confusable to anything else even to newbies, was self-descriptive without needing much or any explanation, cured Covid-19, brought peace to the Middle East, ended poverty, and improved the level of political discourse on social media.
But I fear we may have to relax the requirements somewhat :-)
Agreed. And not everyone will have the same preferences - the "nothing but ASCII" option means more to some than to others, and peace in the Middle East would be really nice but not if it means a 0.001% performance hit in microbenchmarks. :)
My requirements for this syntax are:
- it's a form of dict unpacking, so it ought to use `**` rather than overloading some other, unrelated, symbol like (say) `@` or `%`;
- it ought to use a simple list of identifiers without needing extra sigils or markers on each one, say:
`meta, dunder, private`
rather than
`:meta, :dunder, :private` or `meta=, dunder=, private=`
I'm actually fine with the "meta=, dunder=, private=" myself, but that's because I'm mixing and matching so much.
- it ought to be something "like" an expression, rather than a mode that has to be turned on and then applies to the end of the function call;
- if it actually is an expression, that's a nice bonus, but it's not essential;
Given that the existing keyword argument syntax isn't exactly an expression already, I'm fine with it not being perfectly an expression. We already use the equals sign to indicate the separation between target and value, so it's fine IMO to use that as the notation.
- it ought to look Pythonic, which is subjective;
- or at least not look like "line noise", which is still subjective but it's probably easier to get agreement on what is ugly than on what is beautiful :-)
It is a lot to ask.
If anyone else can think of an alternative, please do speak up!
Indeed. And this is why I think we need a PEP that has the alternatives laid out with the various arguments for and against. If someone thinks of an alternative that isn't listed, it can be added. ChrisA
On Sun, Apr 19, 2020 at 7:00 AM Chris Angelico <rosuav@gmail.com> wrote:
Yes, exactly. One strike doesn't mean it's out, and with a proposal like this, it's a matter of looking at a whole lot of imperfect alternatives and seeing which tradeoffs we want to go with. (And status quo is one such alternative, with tradeoffs of "names need to be duplicated".)
I think you've missed on alternative. Well, it's a variation on "status quo": Use a silly magic helper function like my Q() or Alex' dict_of() in his sorcery library. (Albeit, I tried his library, and it seemed to have a bug I filed an issue on, which might be fixed yesterday; I haven't tried again). But if anyone really wants abbreviated calls with parameters now, they can use: process_record(**Q(email, firstname, lastname, cellphone)) That gets skipping `email=email, ...` if you really want. And it's within a character or two of the length of other proposals. Understand that I'm not really advocating for a magic function like Q(). I think it's important to keep in mind that anyone COULD create such a thing for 20 years, and very few people bothered to. If repeating names was actually such a big pain point, a decent solution has been available for a very long time, but it didn't itch enough for many people to write a few lines with some slight magic to scratch it. In a number of similar discussion, someone has proposed new syntax to do something. And I have often written a little utility function to do generally the same thing, perhaps with somewhat cryptic internals. My 15 minute attempts at magic usually have some bugs or limitations, but that's not really the point. Many of the new syntax ideas COULD be done with an arcane function that only needs to be written once (but better than my 15 minute versions). The fact that such magic functions are not in widespread use, to my mind, argues quite strongly against them actually meriting new syntax. Just feeling like some syntax would be clever or cute should not be enough motivation to add it, which a lot of proposals feel like to me. We need a real world advantage. For that, the commonness of existing workarounds to do near-equivalent things is very germane. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Sun, Apr 19, 2020 at 5:59 PM David Mertz <mertz@gnosis.cx> wrote:
On Sun, Apr 19, 2020 at 7:00 AM Chris Angelico <rosuav@gmail.com> wrote:
Yes, exactly. One strike doesn't mean it's out, and with a proposal like this, it's a matter of looking at a whole lot of imperfect alternatives and seeing which tradeoffs we want to go with. (And status quo is one such alternative, with tradeoffs of "names need to be duplicated".)
I think you've missed on alternative. Well, it's a variation on "status quo": Use a silly magic helper function like my Q() or Alex' dict_of() in his sorcery library. (Albeit, I tried his library, and it seemed to have a bug I filed an issue on, which might be fixed yesterday; I haven't tried again).
But if anyone really wants abbreviated calls with parameters now, they can use:
process_record(**Q(email, firstname, lastname, cellphone))
That gets skipping `email=email, ...` if you really want. And it's within a character or two of the length of other proposals.
Understand that I'm not really advocating for a magic function like Q(). I think it's important to keep in mind that anyone COULD create such a thing for 20 years, and very few people bothered to. If repeating names was actually such a big pain point, a decent solution has been available for a very long time, but it didn't itch enough for many people to write a few lines with some slight magic to scratch it.
I think I'm uniquely qualified to say with certainty that this is 100% not true. A basic version like Q("email firstname lastname") like you wrote is indeed easy to do correctly, but I've said why I wouldn't want to use it and I think others would agree. The real thing, `Q(email, firstname, lastname, cellphone)`, is very hard to do correctly. Q was essentially requested in this question: https://stackoverflow.com/questions/18425225/getting-the-name-of-a-variable-... 1. The question is very popular. People have wanted this feature for a long time. I'd say that's a strong point in favour of these proposals in general. 2. The question was asked in 2013, and went a long time without a satisfying answer. 3. Most answers either say it's not possible or give naive implementations that easily fail. 4. The currently accepted answer (which only appeared in 2019) looks legitimate and some people probably trust it, but it's very shaky. [It uses a regex]( https://github.com/pwwang/python-varname/blob/25785b6aab2cdffc71095642e4d48a...) to parse the Python, in much the same way that one shouldn't parse HTML. [I just told the author to use my library instead]( https://github.com/pwwang/python-varname/issues/3), and he quickly agreed that was much better and made me a collaborator. Writing sorcery was hard. The first version was already hard enough, and [had several limitations]( https://github.com/alexmojaki/sorcery/tree/7c85e5d802de26a435e4d190e02ca9326...). For example, you couldn't use dict_of (i.e. Q()) twice in the same line. Some time later I discovered [icecream](https://github.com/gruns/icecream). It seemed to have solved this problem - it could detect the correct call when there were several on the same line. It had about a thousand stars, 150 commits, and claimed to be well tested. I thought it was the real deal, and clearly other people did too. It had a complicated implementation that analysed the bytecode and did lots of manual parsing. When I tried to incorporate this implementation into my own code, I realised that [it was in fact deeply broken and failed in many different simple cases](https://github.com/gruns/icecream/pull/33). People hadn't actually noticed because they rarely used it inside an expression, [they usually just wrote `ic(x)` on its own line]( https://github.com/gruns/icecream/issues/39#issuecomment-552611756). But I used the idea of analysing bytecode to write my own implementation, [executing](https://github.com/alexmojaki/executing). It was extremely hard. At several points I thought it was unsolvable or that I'd fool myself into thinking I'd solved it when actually it was quietly broken. At one point I was going crazy trying to figure out why the peephole optimizer was behaving inconsistently; I think it was because I hadn't correctly copied locations between AST nodes. Ultimately it's one of my proudest achievements. I'm pretty sure no one else has gotten this far. I think some others (e.g. icecream, varname, and amusingly, [q]( https://github.com/zestyping/q)) got quite far, with significant effort, but still had a long way to go, and I'm not sure how much they're aware of that, let alone their users. I see you've tried while I wrote this, and it's pretty clear it's very far from robust. And despite all that, as you've seen, sorcery still has limitations. It clashes with magic like pytest, IPython, and birdseye, and the only way to deal with that is with special cases where I think they're worth it. It can't work without access to the source code, e.g. in the standard interactive shell, or in .pyc files. And it relies on the implementation details of CPython and PyPy. Nevertheless, in normal cases, I'm 99.9% sure that it would work correctly. And yet I still never use it, and I probably still wouldn't if I was 100% sure. If it became part of the standard library and was blessed by the Python community, I'd use it. Or if this proposal went through, I'd use the new syntax with glee. Note that the use case of icecream and q has now been fulfilled by the new debugging syntax `f'{foo=}'`. Your argument could have equally be made against that syntax, but if it was, it failed, and rightly so because there was no simple function that could correctly replicate that behaviour. If we had nice syntax for keyword arguments though, it'd be trivial to write a function which could be used like `magic_print(**, foo)` and which could easily be customised to format and write its output in all sorts of ways. Having that earlier would have significantly reduced the case for the f-string syntax. So no, this is not a problem that can be solved by a function in current Python. On the other hand, we can change Python to make writing functions like that safe and easy. If code objects held a mapping from bytecode positions to AST nodes (which in turn would ideally know their source code, but that's not essential, e.g. it's not needed for the keyword argument stuff) then writing Q() and other neat magical functions (see the rest of sorcery for examples) would be relatively easy and reliable. And it would have other great applications, like making [tracebacks more detailed]( https://github.com/ipython/ipython/pull/12150). I'd be really excited about that, and I've thought about proposing it here before. But it might be opening Pandora's box, and I expect there'd be plenty of (probably valid) objections.
See this thread where I created M() as successor of Q(). It's really not that hard, I don't think. Probably there are edge cars I haven't addressed, but it's hardly intractable. On Sun, Apr 19, 2020, 4:51 PM Alex Hall <alex.mojaki@gmail.com> wrote:
On Sun, Apr 19, 2020 at 5:59 PM David Mertz <mertz@gnosis.cx> wrote:
On Sun, Apr 19, 2020 at 7:00 AM Chris Angelico <rosuav@gmail.com> wrote:
Yes, exactly. One strike doesn't mean it's out, and with a proposal like this, it's a matter of looking at a whole lot of imperfect alternatives and seeing which tradeoffs we want to go with. (And status quo is one such alternative, with tradeoffs of "names need to be duplicated".)
I think you've missed on alternative. Well, it's a variation on "status quo": Use a silly magic helper function like my Q() or Alex' dict_of() in his sorcery library. (Albeit, I tried his library, and it seemed to have a bug I filed an issue on, which might be fixed yesterday; I haven't tried again).
But if anyone really wants abbreviated calls with parameters now, they can use:
process_record(**Q(email, firstname, lastname, cellphone))
That gets skipping `email=email, ...` if you really want. And it's within a character or two of the length of other proposals.
Understand that I'm not really advocating for a magic function like Q(). I think it's important to keep in mind that anyone COULD create such a thing for 20 years, and very few people bothered to. If repeating names was actually such a big pain point, a decent solution has been available for a very long time, but it didn't itch enough for many people to write a few lines with some slight magic to scratch it.
I think I'm uniquely qualified to say with certainty that this is 100% not true. A basic version like Q("email firstname lastname") like you wrote is indeed easy to do correctly, but I've said why I wouldn't want to use it and I think others would agree. The real thing, `Q(email, firstname, lastname, cellphone)`, is very hard to do correctly.
Q was essentially requested in this question: https://stackoverflow.com/questions/18425225/getting-the-name-of-a-variable-...
1. The question is very popular. People have wanted this feature for a long time. I'd say that's a strong point in favour of these proposals in general. 2. The question was asked in 2013, and went a long time without a satisfying answer. 3. Most answers either say it's not possible or give naive implementations that easily fail. 4. The currently accepted answer (which only appeared in 2019) looks legitimate and some people probably trust it, but it's very shaky. [It uses a regex]( https://github.com/pwwang/python-varname/blob/25785b6aab2cdffc71095642e4d48a...) to parse the Python, in much the same way that one shouldn't parse HTML. [I just told the author to use my library instead]( https://github.com/pwwang/python-varname/issues/3), and he quickly agreed that was much better and made me a collaborator.
Writing sorcery was hard. The first version was already hard enough, and [had several limitations]( https://github.com/alexmojaki/sorcery/tree/7c85e5d802de26a435e4d190e02ca9326...). For example, you couldn't use dict_of (i.e. Q()) twice in the same line.
Some time later I discovered [icecream](https://github.com/gruns/icecream). It seemed to have solved this problem - it could detect the correct call when there were several on the same line. It had about a thousand stars, 150 commits, and claimed to be well tested. I thought it was the real deal, and clearly other people did too. It had a complicated implementation that analysed the bytecode and did lots of manual parsing.
When I tried to incorporate this implementation into my own code, I realised that [it was in fact deeply broken and failed in many different simple cases](https://github.com/gruns/icecream/pull/33). People hadn't actually noticed because they rarely used it inside an expression, [they usually just wrote `ic(x)` on its own line]( https://github.com/gruns/icecream/issues/39#issuecomment-552611756).
But I used the idea of analysing bytecode to write my own implementation, [executing](https://github.com/alexmojaki/executing). It was extremely hard. At several points I thought it was unsolvable or that I'd fool myself into thinking I'd solved it when actually it was quietly broken. At one point I was going crazy trying to figure out why the peephole optimizer was behaving inconsistently; I think it was because I hadn't correctly copied locations between AST nodes. Ultimately it's one of my proudest achievements. I'm pretty sure no one else has gotten this far. I think some others (e.g. icecream, varname, and amusingly, [q]( https://github.com/zestyping/q)) got quite far, with significant effort, but still had a long way to go, and I'm not sure how much they're aware of that, let alone their users. I see you've tried while I wrote this, and it's pretty clear it's very far from robust.
And despite all that, as you've seen, sorcery still has limitations. It clashes with magic like pytest, IPython, and birdseye, and the only way to deal with that is with special cases where I think they're worth it. It can't work without access to the source code, e.g. in the standard interactive shell, or in .pyc files. And it relies on the implementation details of CPython and PyPy.
Nevertheless, in normal cases, I'm 99.9% sure that it would work correctly. And yet I still never use it, and I probably still wouldn't if I was 100% sure. If it became part of the standard library and was blessed by the Python community, I'd use it. Or if this proposal went through, I'd use the new syntax with glee.
Note that the use case of icecream and q has now been fulfilled by the new debugging syntax `f'{foo=}'`. Your argument could have equally be made against that syntax, but if it was, it failed, and rightly so because there was no simple function that could correctly replicate that behaviour. If we had nice syntax for keyword arguments though, it'd be trivial to write a function which could be used like `magic_print(**, foo)` and which could easily be customised to format and write its output in all sorts of ways. Having that earlier would have significantly reduced the case for the f-string syntax.
So no, this is not a problem that can be solved by a function in current Python.
On the other hand, we can change Python to make writing functions like that safe and easy. If code objects held a mapping from bytecode positions to AST nodes (which in turn would ideally know their source code, but that's not essential, e.g. it's not needed for the keyword argument stuff) then writing Q() and other neat magical functions (see the rest of sorcery for examples) would be relatively easy and reliable. And it would have other great applications, like making [tracebacks more detailed]( https://github.com/ipython/ipython/pull/12150). I'd be really excited about that, and I've thought about proposing it here before. But it might be opening Pandora's box, and I expect there'd be plenty of (probably valid) objections.
On Sun, Apr 19, 2020 at 11:00 PM David Mertz <mertz@gnosis.cx> wrote:
See this thread where I created M() as successor of Q().
I saw, and I mentioned it:
I see you've tried while I wrote this, and it's pretty clear it's very far
from robust.
It's really not that hard, I don't think. Probably there are edge cars I
haven't addressed, but it's hardly intractable.
I promise, it really is that hard. It's doable, I've done it, but your version is far from needing to address a few more edge cases. You *can* write a decent version without too much difficulty that will work as long as people follow certain obscure rules (i.e. stay away from the edge cases). That's basically what sorcery's first version was. But that'd be a very unusual user experience. It takes a whole lot more for something robust that can't be accidentally misused. And you will certainly need access to the source code.
On Sun, Apr 19, 2020 at 10:50:55PM +0200, Alex Hall wrote:
I think I'm uniquely qualified to say with certainty that this is 100% not true. A basic version like Q("email firstname lastname") like you wrote is indeed easy to do correctly, but I've said why I wouldn't want to use it and I think others would agree. The real thing, `Q(email, firstname, lastname, cellphone)`, is very hard to do correctly.
Thank you Alex for the very informative response. -- Steven
On Sun, Apr 19, 2020 at 10:50 PM Alex Hall <alex.mojaki@gmail.com> wrote:
On Sun, Apr 19, 2020 at 5:59 PM David Mertz <mertz@gnosis.cx> wrote:
I think you've missed on alternative. Well, it's a variation on "status
quo": Use a silly magic helper function like my Q() or Alex' dict_of() in his sorcery library.
[...]
So no, this is not a problem that can be solved by a function in current Python.
On the other hand, we can change Python to make writing functions like that safe and easy. If code objects held a mapping from bytecode positions to AST nodes (which in turn would ideally know their source code, but that's not essential, e.g. it's not needed for the keyword argument stuff) then writing Q() and other neat magical functions (see the rest of sorcery for examples) would be relatively easy and reliable. And it would have other great applications, like making [tracebacks more detailed]( https://github.com/ipython/ipython/pull/12150). I'd be really excited about that, and I've thought about proposing it here before. But it might be opening Pandora's box, and I expect there'd be plenty of (probably valid) objections.
Exposing the AST is probably overkill, but we could probably come up with a different way to give Python users easy access to the argument names. For example, suppose you could define functions like this: ``` foo = 1 bar = 2 # Note the ***, different from ** def Q(***args_with_names): return args_with_names assert Q(foo, bar) == dict(foo=foo, bar=bar) def debug_print(***args_with_names): print(args_with_names) debug_print(foo, bar) # prints {"foo": 1, "bar": 2} ``` Then you would only need a few small instances of magical syntax. Most users would never need to know what `***` means. If they come across a function call like `func(**Q(foo, bar))`, they can at least tell that keyword arguments are being passed. And the function Q would be easy to google or inspect with help(Q) in the shell. The syntax for calls and dicts wouldn't need to be complicated further.
On Tue, Apr 21, 2020 at 6:56 PM Alex Hall <alex.mojaki@gmail.com> wrote:
Exposing the AST is probably overkill, but we could probably come up with a different way to give Python users easy access to the argument names. For example, suppose you could define functions like this:
``` foo = 1 bar = 2
# Note the ***, different from ** def Q(***args_with_names): return args_with_names
assert Q(foo, bar) == dict(foo=foo, bar=bar) ```
Then you would only need a few small instances of magical syntax. Most users would never need to know what `***` means. If they come across a function call like `func(**Q(foo, bar))`, they can at least tell that keyword arguments are being passed. And the function Q would be easy to google or inspect with help(Q) in the shell. The syntax for calls and dicts wouldn't need to be complicated further.
To make this work, Python would either have to provide those names to every function no matter what (massive overkill and a significant backwards-compatibility change), or would somehow need to know the function's calling convention prior to calling it. Normally, the details of the way parameters are passed is controlled entirely by the call site - these ones are positional, those are keyword - and then the function receives them. For this, you'd need to have something that *looks* positional but *acts* keyword. I think it'd be best to adorn the call site rather than the function. ChrisA
David Mertz wrote:
[...] Many of the new syntax ideas COULD be done with an arcane function that only needs to be written once (but better than my 15 minute versions). The fact that such magic functions are not in widespread use, to my mind, argues quite strongly against them actually meriting new syntax.
People can live without this syntax, the rationale of the proposal ins't that users are actively complaining about this and if people aren't expressing anything about this issue isn't real reason to rule out an idea. Type hints may be an example, I guess... We could keep using comments for that or other hacks, no new syntax needed. But in order for people who would benefit from it to start actually using it to make it as easy as possible was the push this feature needed for gaining wider adoption. Correct me if I'm off here please. The proposal intends to allow for better code and to serve as an incentive for the widespread use of keyword parameters. A utility function `Q` won't have high adherence because it is essentially a hack and no one wants to install a lib and go importing `Q` in every project file. If it's not easy and/or doesn't look nice people won't use it. This is only my humble opinion though. Rodrigo Martins de Oliveira
On Sun, Apr 19, 2020 at 11:58:01AM -0400, David Mertz wrote:
Understand that I'm not really advocating for a magic function like Q(). I think it's important to keep in mind that anyone COULD create such a thing for 20 years, and very few people bothered to.
You don't really know how many people are using such a thing in their private code. Its also a pretty obscure thing to use: frame hacks are not something that most Python programmers know anything about. The fact that your Q() and similar solutions are described as "magic" or "sorcery" demenstrates that even the creators of the functions are aware that they are somewhat unclean hacks that shouldn't be relied on. At least with a language feature, the hack is official, and it's the responsibility of the language devs to keep it working, at which point it ceases to be a hack and becomes just part of the language. With a Python level hack, if something breaks, it's *my* responsibility to fix it, and I'll get zero sympathy from anyone else. So if the only choices are "deal with the middling pain of `name=name` function calls" versus "take on an unknown amount of technical debt by using an unsupported hack", I'll keep writing `name=name` thank you very much :-)
If repeating names was actually such a big pain point, a decent solution has been available for a very long time, but it didn't itch enough for many people to write a few lines with some slight magic to scratch it.
The fact that it requires the use of a private function means that many people will not consider it a decent solution, but technical debt which will have to be paid eventually. When I compare the Python community to (say) the Java community, I am pleasantly surprised about how seriously most Python people are about not using _private APIs. Sometimes I feel that Java programmers spend half their day trying to protect their code from other people using it, and the other half trying to break that protection on other people's code. Whereas the Python community, generally, wouldn't touch someone else's _private API if you paid them. There's another issue. Put yourself in the shoes of someone who *isn't* the author of your Q() function, and you come across this function call: func(arg, **Q(alpha)) It looks like a regular function. Obviously it must return a dict, because it is passed to the dict-unpacking operator. You don't know that there's anything special about Q, it just looks like a plain old Python function, so you know that Q takes the *value* of `alpha`, because that's how Python function works. Would you recognise this as a different spelling of `alpha=alpha`? I doubt it. And if you did, you would probably say "Wait a minute, how the hell does Q know the *name* from the *value* of the variable? Most objects don't have names!" If you *didn't* recognise Q as magic, you might be tempted to change it. Say you need a couple of extra arguments: func(arg, **Q(alpha, 'something', beta+1)) and who the hell knows what that will do! (Cue the Sorceror's Apprentice music.) It might even work by accident, and that would be the worst thing of all, because then it could stop working when we least expect it. Sorry David, when I see your magic Q() I see technical debt, not a viable solution for production code. -- Steven
On Sun, Apr 19, 2020, 10:16 PM Steven D'Aprano <steve@pearwood.info> wrote:
Understand that I'm not really advocating for a magic function like Q(). I think it's important to keep in mind that anyone COULD create such a thing for 20 years, and very few people bothered to.
You don't really know how many people are using such a thing in their private code.
I've read quite a bit of open source code and code at places I've worked. I think "very few" is a safe claim. It's more than "none." At least with a language feature, the hack is official, and it's the
responsibility of the language devs to keep it working, at which point it ceases to be a hack and becomes just part of the language.
I see you're point here. Q(), or the improved M(), are certainly not more magic than namedtuple, or typing, or dataclass, or @total_ordering. But those are parts of the official language, and we can trust that edge cases and bugs will be well addressed in every release. Attrs is pretty magic, and also widely used... But after a while, that motivated dataclass, which is largely similar, to be added to the official language. Popular packages like Django or Pandas have a fair amount of magic in them, but that magic is specific to working with the objects they create. I do recognize that Q() or M() or Alex's dict_of() break the rules of what functions are supposed to do. Python is pass-by-reference not pass-by-name, and these deliberately muck with that. So you've moved me a bit towards the side of wanting a new DICT DISPLAY. Something completely limited to calling form feels artificially limited for no reason. And the addition of f"{foo=}" is very similar magic for similar reasons. So constructing a dictionary like these are only -0 for me (maybe -0.1): dct = {**, foo, bar, baz} dct = **{foo, bar, baz} dct = {foo=, bar=, baz=} With any of those corresponding in function calls: fun(**kws, foo, bar, baz) fun(**, foo, bar, baz) fun(**kws, **{foo, bar, baz}) fun(**kws, foo=, bar=, baz=) The form with the dangling equal signs still strikes me as horribly ugly, but I see the closer analogy with the f-string magic.
On Sat, Apr 18, 2020 at 5:03 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Sat, Apr 18, 2020 at 02:13:51PM +0200, Alex Hall wrote:
My issue with this, and maybe it's what Andrew is also trying to say, is that it breaks our usual assumptions about composing expressions. `{u, v}` is an expression, it represents a set, and it always represents that wherever you put it. Under your proposal, these two programs are both valid syntax with different meanings:
f(**{u, v})
x = {u, v} f(**x)
True, and that does count as a (minor?) point against it. But not one that I think should rule it out.
Is there anything else similar in the language? Obviously there are cases where the same text has different meanings in different contexts, but I don't think you can ever refactor an expression (or text that looks like an expression) into a variable and change its meaning while keeping the program runnable.
Of course! There are many ways that this can occur.
OK, I have to admit I had a bit of a failure of imagination. But there is something about all these examples (plausibility?) that feels distinctly different from your proposal. I can't quite put my finger on it. It might just be that your proposal is new and I need to get used to it the way I'm used to how the rest of Python works. But I'm not convinced of that.
f(a, b, c)
x = a, b, c f(x)
are very different things.
This imagined refactoring doesn't feel as plausible. A complete beginner might think that they can do that, but a programmer who knows what tuples are can reason that it doesn't make sense. On the other hand, an intermediate programmer can be forgiven for thinking that they could refactor out the {u, v} in the proposed syntax. Hypothetically, sets could support the mapping protocol. Understanding that it's impossible requires understanding the guts of the language better and that variable names aren't stored with objects at runtime. Put differently, the proposal could mislead readers into thinking that sets do actually know the names of the values in them, without them trying to refactor out the 'set'.
Here's another one:
import name
x = name import x
This is stretching the idea of "text that looks like an expression". 'name' looks like an expression in complete isolation, but even a beginner who has just learned imports knows (at least at an intuitive, informal level) that 'name' is not an expression that is evaluated to determine what to import. This is generally obvious from the fact that 'name' is usually not defined beforehand. So even someone who just knows the basics of variables and hasn't seen imports before might guess that the refactoring would fail (with a NameError). Another litmus test that it isn't like an expression is that `import (name)` isn't valid.
Here's a third:
del fe, fi, fo, fum
x = fe, fi, fo, fum del x
This could be simplified to del y vs x = y del x This actually is something that a beginner might think would work, and several have probably tried. The workings of variables, references, names, scope, and garbage collection are confusing at first. We shouldn't introduce new features that are similarly confusing.
Here's an example so obvious (and trivial) that I'm almost embarrassed to include it:
seq[index]
x = [index] seqx
Again, '[index]' only looks like a list in complete isolation. If you try to interpret it as a list while it's stuck to 'seq', the code doesn't even begin to make sense. You can't just stick two expressions together, there has to be an operator or something in between. And the refactoring isn't plausible - it's obvious that seqx is a new variable and not the concatenation of seq and x, otherwise it could equally be s|eqx or se|qx or s|e|q|x.
If code is made of composable building blocks, those blocks aren't *characters*. What makes a composable block is dependent on context.
To make up for how trivial the previous example was, here's a complicated one:
for item in items: if item: continue do_stuff()
versus:
def block(item): if item: continue do_stuff()
for item in items: block()
I have often wished I could refactor continue and break into functions, but you can't :-(
Me too :-(
Although in this case at least you get a syntax error when you try.
Therefore it doesn't satisfy my criteria.
Here's an example with sequence unpacking:
a = [1, 2, *seq]
x = 2, *seq a = [1, x]
The problem here isn't related to unpacking. This doesn't work for the same reasons: a = [1, 2, 3] x = 2, 3 a = [1, x] or just: a = [1, 2] x = 1, 2 a = [x] Which shows that it's basically the same as the function call example at the beginning. If you understand how lists and tuples work, you can understand why the refactoring doesn't work. Put differently, it's not just about refactoring out variables. The above refactoring is equivalent to changing: a = [1, 2, 3] to a = [1, (2, 3)] And it's pretty obvious to a beginner that the two expressions are different. It's much less obvious that these are different: f(**{u, v}) f(**({u, v})) Tuples in Python are weird. They're supposed to be defined by the comma, but they're obviously not really, because commas have all sorts of other meanings, and empty tuples don't need commas. They need parentheses sometimes in cases that are hard to summarise. People think that parentheses are what define the tuple and don't realise they need a trailing comma for singletons. It's a mess, and again, we should avoid creating things which are similarly confusing.
Another example:
class MyClass(metaclass=MyMeta)
metaclass = MyMeta class MyClass(metaclass)
No, this hasn't refactored `metaclass=MyMeta` out into a variable, you've just moved it. The correct analogy would be: M = metaclass=MyMeta class MyClass(M) which makes the same point, but still violates the rules because `metaclass=MyMeta` isn't text that looks like an expression.
That's just a special case of keyword notation itself:
func(x=expr)
x = expr func(x)
Indeed it is just a special case, and it has the same problem. x=expr is not text that looks like an expression. As you say below, you cannot for example put it into parentheses.
Those are not the same, unless the first positional argument happens to be named `x`.
And one final example:
class C: def method(self): pass
versus:
def method(self): pass
class C: method
`def method...` is not an expression.
The example above looks a bit dumb, but maybe users will try:
``` if flag: kwargs = {u, v} else: kwargs = {w, x} f(**kwargs) ```
I think this point will apply to all(?) such syntactic proposals. I don't think this scenario is too different from this:
if flag: f(u=expr1, v=expr2) else: f(w=expr3, x=expr4)
If you want to refactor that, you can't do this:
f((u=expr1, v=expr2) if flag else (w=expr3, x=expr4))
No you can't, so thankfully it raises a SyntaxError, unlike your proposal.
Which is valid syntax but is wrong. Then they might try changing that to:
f(**({u, v} if flag else {w, x}))
which is suddenly invalid syntax.
Not invalid syntax, but it's still wrong. You'll get a TypeError when trying to `**` unpack a set instead of a dict.
Yes I got confused and made a mistake here. The fact that it doesn't raise a SyntaxError is still bad, for the same reasons that I argued about in the beginning (whereas previously I thought it was bad for new and different reasons). The point is that the syntax looks and feels like it can be manipulated like an expression.
On 19/04/20 7:17 am, Alex Hall wrote:
there is something about all these examples (plausibility?) that feels distinctly different from your proposal.
To me it seems like an unnecessarily complicated syntax that goes out of its way to look deceptively like something else.
f(a, b, c)
x = a, b, c f(x)
This imagined refactoring doesn't feel as plausible. A complete beginner might think that they can do that, but a programmer who knows what tuples are can reason that it doesn't make sense.
Fun fact -- I gather there was a very early version of Python in which this refactoring *did* work. But it was quickly changed because people found it too confusing! I think what's happening here is that long experience with other languages has ingrained in us the idea that commas separate arguments to a function without creating tuples, so when we see a comma-separated list in the context of a function call we instinctively think "argument list" and not "tuple". But there is no such precedent for the OP's proposal. -- Greg
On Sun, Apr 19, 2020 at 02:30:21PM +1200, Greg Ewing wrote:
On 19/04/20 7:17 am, Alex Hall wrote:
there is something about all these examples (plausibility?) that feels distinctly different from your proposal.
To me it seems like an unnecessarily complicated syntax that goes out of its way to look deceptively like something else.
Are we still talking about `**{identifier}`? There are three tokens there: `**{`, an identifier, and `}`. Adding an optional comma makes four. If this is your idea of "complicated syntax", I cannot imagine how you cope with function definitions in their full generality: def name(arg, /, a:str='', *args, b:float=-0.0, **kw) -> str:
f(a, b, c)
x = a, b, c f(x)
This imagined refactoring doesn't feel as plausible. A complete beginner might think that they can do that, but a programmer who knows what tuples are can reason that it doesn't make sense.
Fun fact -- I gather there was a very early version of Python in which this refactoring *did* work. But it was quickly changed because people found it too confusing!
I can confirm your fun fact is true in Python 0.9.1, at least the first part. I don't know if it was changed because people were confused, or if they just didn't like it, or because it made it troublesome to pass a tuple as argument.
I think what's happening here is that long experience with other languages has ingrained in us the idea that commas separate arguments to a function without creating tuples, so when we see a comma-separated list in the context of a function call we instinctively think "argument list" and not "tuple".
But there is no such precedent for the OP's proposal.
You just spent an entire paragraph describing such a precedent. -- Steven
On 19/04/20 6:58 pm, Steven D'Aprano wrote:
There are three tokens there: `**{`, an identifier, and `}`. Adding an optional comma makes four.
If this is your idea of "complicated syntax", I cannot imagine how you cope with function definitions in their full generality:
What I mean is that much simpler syntaxes have been proposed that achieve the same goal, and that don't look like something they're not. -- Greg
On Mon, Apr 20, 2020 at 02:50:53AM +1200, Greg Ewing wrote:
On 19/04/20 6:58 pm, Steven D'Aprano wrote:
There are three tokens there: `**{`, an identifier, and `}`. Adding an optional comma makes four.
If this is your idea of "complicated syntax", I cannot imagine how you cope with function definitions in their full generality:
What I mean is that much simpler syntaxes have been proposed that achieve the same goal, and that don't look like something they're not.
I'll accept the second part, but what are those "much simpler" syntaxes? I know of these alternatives: **{alpha, beta, gamma} **{:alpha, :beta, :gamma} *, alpha, beta, gamma **, alpha, beta, gamma alpha=, beta=, gamma= although I may have missed some. I'm not seeing "much" difference in complexity between them, syntax-wise. Can we at least try to avoid unnecessary hyperbole in describing ideas we don't like? It's disheartening and frustrating to see microscopic differences blown all out of proportion. -- Steven
On 20/04/20 1:36 pm, Steven D'Aprano wrote:
**{alpha, beta, gamma} **{:alpha, :beta, :gamma} *, alpha, beta, gamma **, alpha, beta, gamma alpha=, beta=, gamma=
although I may have missed some. I'm not seeing "much" difference in complexity between them, syntax-wise.
To my eyes, the last three are a lot simpler than the first two. Not just in the number of characters, but in the amount of mental processing required to figure out what's going on.
Can we at least try to avoid unnecessary hyperbole in describing ideas we don't like?
I don't think it's hyperbole, I really mean what I said. And I have reasons for disliking some of those, which I think I have explained. -- Greg
On Sat, Apr 18, 2020, 8:15 AM Alex Hall <alex.mojaki@gmail.com> wrote:
Under your proposal, these two programs are both valid syntax with different meanings:
(1)
f(**{u, v})
(2)
x = {u, v}
f(**x)
I don't understand what the meaning of (2) would be. Currently it is a TypeError... Is that "valid" because it's not a syntax error?!
On Sat, Apr 18, 2020 at 12:19:30PM -0400, David Mertz wrote:
On Sat, Apr 18, 2020, 8:15 AM Alex Hall <alex.mojaki@gmail.com> wrote:
Under your proposal, these two programs are both valid syntax with different meanings:
(1)
f(**{u, v})
(2)
x = {u, v}
f(**x)
I don't understand what the meaning of (2) would be. Currently it is a TypeError... Is that "valid" because it's not a syntax error?!
Yes, as Andrew pointed out, it is valid syntax: it isn't rejected by the compiler, it generates byte-code, and then when you run it, it has behaves differently from (1). So it has a meaning: "raise TypeError in an inefficient fashion". Not *useful* meaning, to be sure, but still meaning. I think a fundamental point is that `**{identifier}` looks like you are applying `**` unpacking to a set, but you actually aren't, it is a special syntactic form. If that disturbs you, I'm not going to say you are wrong. I'm a little sad about it myself, I just don't give it a very high weighting since we already have plenty of other similar cases: func(a, b, c) # Not a tuple. import a, b, c # Also not a tuple. del a, b, c # Still not a tuple. mapping[key] # And not a single-item list either :-) I think I would be more concerned by this if I wanted this keyword argument shortcut to work everywhere, not just in function calls. E.g. mydict = **{a, b c} # like {'a': a, 'b': b, 'c': c} but I don't care much about that. -- Steven
f(**{u, v})
I think a fundamental point is that `**{identifier}` looks like you are applying `**` unpacking to a set, but you actually aren't, it is a special syntactic form. If that disturbs you, I'm not going to say you are wrong.
The special syntactic form doesn't bother me that much. '**dict' is already a syntax error in many places, but allowed in a few other places. I also do not think the purpose served is important enough to warrant a new special form. The actual same conciseness can be achieved with a function that does just a little bit of magic, e.g.: func(**Q(u, v)) Someone linked to a library that apparently does the magic a bit better than my 5-minute version (which required quoting the variable names). I think it was you who complained that sys._getframe() might be CPython specific, which is true. But probably other implementations could do different magic to make their own Q(), so that seems of little importance. I really doubt I'll ever use a magic function like Q()... but the fact that I could have done so for the entire 20+ years that I've used Python tends to emphasize that the need isn't REALLY that huge. And if the need isn't large, reserving new syntax for it isn't compelling. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On 4/18/2020 2:03 PM, David Mertz wrote:
f(**{u, v})
I think a fundamental point is that `**{identifier}` looks like you are applying `**` unpacking to a set, but you actually aren't, it is a special syntactic form. If that disturbs you, I'm not going to say you are wrong.
The special syntactic form doesn't bother me that much. '**dict' is already a syntax error in many places, but allowed in a few other places.
I also do not think the purpose served is important enough to warrant a new special form. The actual same conciseness can be achieved with a function that does just a little bit of magic, e.g.:
func(**Q(u, v))
Someone linked to a library that apparently does the magic a bit better than my 5-minute version (which required quoting the variable names). I think it was you who complained that sys._getframe() might be CPython specific, which is true. But probably other implementations could do different magic to make their own Q(), so that seems of little importance.
I'd be only -0.5 on any proposal from this thread (as opposed to -1000 as I am now) if it were more general purpose than just function calls. Like David's "Q" function (which is an actual function), if we had some magic that said "create a dict from these names, where the values come from normal python scoping rules". Let's say that magic function is named M() (for magic). We might want it to be special syntax to make it clear it's magic (maybe "!<a, b>": who knows, and it's unimportant here). But that's not the point: the point here is making something general purpose. So, if M() existed, you could say: d = M(telephone, name) func(**d) or func(**M(telephone, name)) Or, you could just use "d" from the first example for your own purposes unrelated to function calling. My point is: We already have a way to pass the items in a dictionary as keyword args: let's not invent another one. Instead, let's focus on a general purpose way of creating a dictionary that meets the requirements of being able to be passed as keyword args. That way we'd me making the language more expressive beyond just function calls. Eric
So, if M() existed, you could say:
d = M(telephone, name) func(**d)
or
func(**M(telephone, name))
Or, you could just use "d" from the first example for your own purposes unrelated to function calling.
My point is: We already have a way to pass the items in a dictionary as
keyword args: let's not invent another one. Instead, let's focus on a general purpose way of creating a dictionary that meets the requirements of being able to be passed as keyword args. That way we'd me making the language more expressive beyond just function calls.
Personally my favourite potential outcome from all this would be allowing the `**` mode separator in both dicts and calls, so you could write: d = {**, telephone, name} func(**d) or func(**, telephone, name) Also I think Dominik has made an excellent point that this would only be needed if there were no kwargs already, so this would also be possible: func(**kwargs, telephone, name) assuming that some relevant kwargs exist. Then there isn't even any new syntax, just a way to interpret something which is currently forbidden.
On Sun, Apr 19, 2020 at 2:37 PM Eric V. Smith <eric@trueblade.com> wrote:
So, if M() existed, you could say:
d = M(telephone, name) func(**d)
or
func(**M(telephone, name))
Per your wish, Eric, the glorious successor of Q() ... named M():
def M(*vals): ... import sys ... import inspect ... caller = sys._getframe(1) ... call = inspect.stack()[1].code_context[0] ... _, call = call.split('M(') ... call = call.strip()[:-1] ... names = [name.strip() for name in call.split(',')] ... dct = {} ... for name in names: ... dct[name] = eval(name, globals(), caller.f_locals) ... return dct ... x, y, z = range(3) M(x, y, z) {'x': 0, 'y': 1, 'z': 2}
OK, it's a little bit fragile in assuming the function must be called M rather than trying to derive its name. And maybe my string version of finding the several args could be made more robust. But anyone is welcome to improve it, and the proof of concept shows that's all we need. Basically, a "dict-builder from local names" is perfectly amenable to writing as a Python function... and we don't need to inspect the underlying source code the way I believe Alex' sorcery module does (other parts of it might need that, but not this). -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
Per your wish, Eric, the glorious successor of Q() ... named M():
def M(*vals): ... import sys ... import inspect ... caller = sys._getframe(1) ... call = inspect.stack()[1].code_context[0] ... _, call = call.split('M(') ... call = call.strip()[:-1] ... names = [name.strip() for name in call.split(',')] ... dct = {} ... for name in names: ... dct[name] = eval(name, globals(), caller.f_locals) ... return dct ... x, y, z = range(3) M(x, y, z) {'x': 0, 'y': 1, 'z': 2}
OK, it's a little bit fragile in assuming the function must be called M rather than trying to derive its name. And maybe my string version of finding the several args could be made more robust. But anyone is welcome to improve it, and the proof of concept shows that's all we need. Basically, a "dict-builder from local names" is perfectly amenable to writing as a Python function... and we don't need to inspect the underlying source code the way I believe Alex' sorcery module does (other parts of it might need that, but not this).
I can lay out all the issues with this if you want me to, but after my previous email I don't think I have to. I'm just wondering why you say it doesn't need to inspect the underlying source code. That's what `code_context` is and that's obviously the only place where a string like `'M('` could be found, unless you want to uncompile bytecode (which is not impossible either, but it's an additional mess).
Nope, that's fine. I reckon it's reasonable to call this inspecting the source code. I thought from your GH issue that you meant you read in a whole module of code. I don't want my Q(), or M(), or whatever letter comes after that, in the standard library. I don't even care about making a repo for it or publishing it on PyPI. Even if you can find a way to break it... which I'm happy to stipulate you can, that doesn't matter a whit to the possible utility... if the need was genuine. If this were in the library I or my project used, whatever limitations and edge cases exist wouldn't really matter. If I REALLY cared about saving a few duplicate names in function calls, I could easily include it with the knowledge that it won't handle such-and-such edge cases. If the improvement really mattered for normal, boring code that makes up 99% of the code I write, I could use it in that 99%. But I don't. And you don't. And no one in this thread does.... so special syntax for something no one actually does is foolish. ... that said, I like your latest suggestion best of what I've seen. I.e. built_dict = {**, foo, bar, baz} my_func(**, foo, bar, baz) my_other_func(**kws, foo, bar) Those all read nicely, and in a consistent way. You could even drop the bare ** by using: my_func(**{}, foo, bar, baz) (if there was a reason someone was OK with self-naming values but not with bare '**'). On Sun, Apr 19, 2020 at 5:01 PM Alex Hall <alex.mojaki@gmail.com> wrote:
Per your wish, Eric, the glorious successor of Q() ... named M():
def M(*vals): ... import sys ... import inspect ... caller = sys._getframe(1) ... call = inspect.stack()[1].code_context[0] ... _, call = call.split('M(') ... call = call.strip()[:-1] ... names = [name.strip() for name in call.split(',')] ... dct = {} ... for name in names: ... dct[name] = eval(name, globals(), caller.f_locals) ... return dct ... x, y, z = range(3) M(x, y, z) {'x': 0, 'y': 1, 'z': 2}
OK, it's a little bit fragile in assuming the function must be called M rather than trying to derive its name. And maybe my string version of finding the several args could be made more robust. But anyone is welcome to improve it, and the proof of concept shows that's all we need. Basically, a "dict-builder from local names" is perfectly amenable to writing as a Python function... and we don't need to inspect the underlying source code the way I believe Alex' sorcery module does (other parts of it might need that, but not this).
I can lay out all the issues with this if you want me to, but after my previous email I don't think I have to. I'm just wondering why you say it doesn't need to inspect the underlying source code. That's what `code_context` is and that's obviously the only place where a string like `'M('` could be found, unless you want to uncompile bytecode (which is not impossible either, but it's an additional mess).
-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Sun, Apr 19, 2020 at 11:18 PM David Mertz <mertz@gnosis.cx> wrote:
If this were in the library I or my project used, whatever limitations and edge cases exist wouldn't really matter. If I REALLY cared about saving a few duplicate names in function calls, I could easily include it with the knowledge that it won't handle such-and-such edge cases.
No you couldn't, it would cause untold pain to anyone who collaborated with you (or even just your future self) who didn't know which edge cases it can't handle and that they have to tiptoe around it. "David this code breaks when I spread it out across multiple lines" "David this code breaks when I collapse it into one line" "David this code breaks when I rename a function" "David this code breaks when I paste it into a shell" "David I really like this M function, i've been using it for debugging like print(M(foo, bar)), but when I changed the line make_payment() to print(M(make_payment())) it made the payment twice"
On Sun, Apr 19, 2020 at 04:14:51PM -0400, David Mertz wrote:
Per your wish, Eric, the glorious successor of Q() ... named M():
def M(*vals): ... import sys ... import inspect ... caller = sys._getframe(1) ... call = inspect.stack()[1].code_context[0] ... _, call = call.split('M(') ... call = call.strip()[:-1] ... names = [name.strip() for name in call.split(',')] ... dct = {} ... for name in names: ... dct[name] = eval(name, globals(), caller.f_locals) ... return dct
When I try it I get this: py> alpha = 'something' py> M(alpha) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 5, in M TypeError: 'NoneType' object is not subscriptable Tried and failed in both 3.5 and 3.8, so I'm not sure what version of Python you are using where it works, or under what conditions. Your example M(x, y, z) fails with the same error. Since this is fundamentally broken and doesn't work for me, I can't test it, but I wonder how well it will cope for things like: func(arg, FunctionM(x, y), M(z)) M(x, M(y)['y']) func(arg, M( x, y, ), z) etc, and how many hours of debugging it will take to get it to work. I'm also looking at that call to eval and wondering if someone smarter than me can use that to trick me into evaluating something I don't like. -- Steven
On Mon, Apr 20, 2020 at 1:39 PM Steven D'Aprano <steve@pearwood.info> wrote:
I'm also looking at that call to eval and wondering if someone smarter than me can use that to trick me into evaluating something I don't like.
I'm very confused by that call. It appears to be evaluating a bare name in a specific context... which should give the value of that variable. But, isn't that going to be... one of the parameters to the function? Hasn't it already been parsed out? It'd be a lot more interesting if people post MacroPy proposals rather than these bizarre source-code-parsing things that are fundamentally broken in so many ways. ChrisA
On Sun, Apr 19, 2020 at 11:40 PM Steven D'Aprano <steve@pearwood.info> wrote: def M(*vals): # ... magic stuff ... return dct
py> alpha = 'something' py> M(alpha) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 5, in M TypeError: 'NoneType' object is not subscriptable
Yep. It works in IPython and it works in a script, but I didn't try the plain Python shell. I'm using Python 3.8, but I imagine the same problem exists between IPython and Python shell in other versions. etc, and how many hours of debugging it will take to get it to work.
Some positive but finite number of hours to deal with the edge cases. For the discussion, can we just stipulate some future L() that is a better version of M()? (base) 551-bin % python auto-dict.py {'alpha': 'something'} args: () kws: {'alpha': 'something', 'beta': 'else', 'gamma': 'again'} (base) 552-bin % cat auto-dict.py from magic import M alpha = 'something' beta = 'else' gamma = 'again' print(M(alpha)) def show(*args, **kws): print('args:', args) print('kws:', kws) show(**M(alpha, beta, gamma))
I'm also looking at that call to eval and wondering if someone smarter than me can use that to trick me into evaluating something I don't like.
Yes... in L() is should be `ast.literal_eval()`. Maybe I'll write a better L() in the next day or two. But I really AM NOT pushing for anyone to use such a thing, just to consider the conceptual space it occupies vs. having new syntax. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.
On Sun, Apr 19, 2020 at 02:34:35PM -0400, Eric V. Smith wrote:
I'd be only -0.5 on any proposal from this thread (as opposed to -1000 as I am now) if it were more general purpose than just function calls. [...] My point is: We already have a way to pass the items in a dictionary as keyword args: let's not invent another one. Instead, let's focus on a general purpose way of creating a dictionary that meets the requirements of being able to be passed as keyword args. That way we'd me making the language more expressive beyond just function calls.
Good point. Maybe we should step back and look at the bigger picture and ask what we're fundamentally trying to do. And that, it seems to me, is solve a problem that beginners often ask: "How do I get the name of a variable?" The answer is, in general, you can't. But what if we could? In low-level languages like C and Pascal, there are usually "address of" unary operators. Would it solve the problem if we had a "name of" operator? Let's call it `$` just for something to call it. {'name': name} {$name: name} Well, that saves us a mere one character. In a function call: func(arg, $name=name) and that *costs* us an unnecessary character, if it even worked, which it probably wouldn't. What if it expanded to name=value? {'name': name} {$name} Hmm, that's terser, but now it looks like a set. If we changed the dollar sign to a colon, we come back to Andrew's suggestion. Maybe I've stepped back too far. Sometimes we can overgeneralise. But either way, it's good food for thought. -- Steven
On Apr 18, 2020, at 05:16, Alex Hall <alex.mojaki@gmail.com> wrote:
Is there anything else similar in the language? Obviously there are cases where the same text has different meanings in different contexts, but I don't think you can ever refactor an expression (or text that looks like an expression) into a variable and change its meaning while keeping the program runnable.
I suppose it depends on where you draw the “looks like an expression”, line, but I think there are cases that fit. It’s just that there are _not many_ of them, and most of them are well motivated. Each exception adds a small cost to learning the language, but Python doesn’t have to be perfectly regular like Lisp or Smalltalk, it just has to be a lot less irregular than C or Perl. Most special cases aren’t special enough, but some are. A subscription looks like a list display, but it’s not. Mixing them up will only give you a syntax error if you use slices, ellipses, or *-unpacking in the wrong one, and often won’t even give you a runtime error. And the parallel isn’t even useful. But this is worth it anyway because subscription is so tremendously important. A target list looks like a tuple display, but it’s not. Mixing them up will only give you a syntax error if you try to use a tuple display with a constant or a complex expression in it as a target list. Mixing them up in other ways will only give you at best an UnboundLocalError or NameError at runtime, and at worst silently wrong behavior. But the parallel here is more helpful than confusing (it’s why multiple-value return looks so natural in Python, for one thing), so it’s worth it. **{a, b, c} is a special case in two ways: **-unpacking is no longer one thing but two different things, although with a very useful and still pretty solid parallel between them, and set display syntax now has two meanings, with a somewhat useful and weaker parallel. Even added together, that’s not as much of a learning burden as subscription looking like list displays. But it also isn’t as important a benefit. The magic ** mode switch only pushes two complicated and already-not-quite-parallel forms a little farther apart, which is less of a cost. The keyword= is similar but even less so, especially since anywhere it could be confused is a syntax error. The dict display ::value doesn’t cause any new exceptions or break any existing parallels at all, so it’s even less of a cost. But there are plenty of other advantages and disadvantages of each of the four (and the minor variations on them in this thread); that’s just one factor of many to weigh.
On 18.04.20 08:14, Steven D'Aprano wrote:
This proposal is an alternative to Rodrigo's "Keyword arguments self-assignment" thread.
Rodrigo, please feel free to mine this for useful nuggets in your PEP.
(I don't claim to have invented the syntax -- I think it might have been Alex Hall?)
Keyword Unpacking Shortcut --------------------------
Inside function calls, the syntax
**{identifier [, ...]}
expands to a set of `identifier=identifier` argument bindings.
I don't see how this proposal is significantly different from the `**` version: func(foo, **, bar) vs. func(foo, **{bar}) It's still a mode switch, only the beginning and end markers have changed. Instead of `**,` (or `**mapping,`) we then have `**{` as the opening marker and instead of `)` (the parenthesis that closes the function call) we have `}` as the closing marker. You have criticized the use of modal interfaces in this message from which I quote (https://mail.python.org/archives/list/python-ideas@python.org/message/TPNFSJ...):
Modes are okay when they are really big (e.g. "I'm in Python programming mode now") but otherwise should be minimized, with an obvious beginning and end. If you have to have a mode, they should be *really obvious*, and this isn't. There's a really fine difference between modes:
my_super_function_with_too_many_parameters( args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, kwargs, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, *, pass_fds, encoding, errors, text, file, mode, buffering, newline, closefd, opener, meta, private, dunder, invert, ignorecase, ascii_only, seed, bindings, callback, log, font, size, style, justify, pumpkin, )
If you miss the star in the middle of the call, there's no hint in the rest of the call to clue you in to the fact that you changed modes.
(The above example uses `*` as the marker, while in the meantime `**` has been proposed.) I'm not convinced that `**{` and `}` make the beginning and end of the mode "really obvious": my_super_function_with_too_many_parameters( args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, kwargs, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, **{pass_fds, encoding, errors, text, file, mode, buffering, newline, closefd, opener, meta, private, dunder, invert, ignorecase, ascii_only, seed, bindings, callback, log, font, size, style, justify, pumpkin}) For really long argument lists you can expect the mode switch to be placed on a separate line for the sake of readability and then it's hard to miss either way: my_super_function_with_too_many_parameters( args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, kwargs, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, **, pass_fds, encoding, errors, text, file, mode, buffering, newline, closefd, opener, meta, private, dunder, invert, ignorecase, ascii_only, seed, bindings, callback, log, font, size, style, justify, pumpkin, ) In addition to that, more and more advanced IDEs are available and those could easily highlight the "autofill" part that follows the `**` (or `**{`) to help readability.
On Sat, Apr 18, 2020 at 09:13:44PM +0200, Dominik Vilsmeier wrote:
func(foo, **, bar) vs. func(foo, **{bar})
It's still a mode switch, only the beginning and end markers have changed. Instead of `**,` (or `**mapping,`) we then have `**{` as the opening marker and instead of `)` (the parenthesis that closes the function call) we have `}` as the closing marker.
How do you define a mode switch? Is a list display a mode? Is a string a mode? Is a float a mode? In some sense, maybe, but to me the critical factor is that nobody talks about "list mode", "string mode", let alone "float mode". Its about the mental model. With `func(foo, **, bar, baz, quux)` if I use `**` as a pseudo-argument, the interpreter switches to "auto-fill" mode and everything that follows that (until the end of the function call) has to be interpreted according to the mode. A few people immediately started describing this as a mode, without prompting. I think it is a very natural way of thinking about it. And we have no way of turning the mode off. So if there is every a proposal to allow positional arguments to follow keyword arguments, it won't be compatible with auto-fill mode. With `func(foo, **{bar, baz, quux})` the mental model is closer to ordinary argument or dict unpacking. Nobody refers to this: func(spam, *[eggs, cheese, aardvark], hovercraft) as "list mode" or "argument unpacking mode". It's just "unpacking a list" or similar. No-one thinks about the interpreter entering a special "collect list mode" even if that's what the parser actually does, in some sense. We read the list as an entity, which then gets unpacked. Likewise for dict unpacking: nobody thinks of `{'a': expr}` as entering "dict mode". You just make a dict, then unpack it. And nobody (I hope...) will think of keyword shortcut as a mode: func(foo, **{bar, baz}, quux=1)` It's just unpacking an autofilled set of parameter names. Not a mode at all. And notice that there is absolutely no difficulty with some future enhancement to allow positional arguments after keyword arguments. -- Steven
On 19.04.20 12:57, Steven D'Aprano wrote:
On Sat, Apr 18, 2020 at 09:13:44PM +0200, Dominik Vilsmeier wrote:
func(foo, **, bar) vs. func(foo, **{bar})
It's still a mode switch, only the beginning and end markers have changed. Instead of `**,` (or `**mapping,`) we then have `**{` as the opening marker and instead of `)` (the parenthesis that closes the function call) we have `}` as the closing marker. How do you define a mode switch? I don't have a clear definition, instead my point was to show that there is no substantial difference between `**` and `**{...}` (though you seem to think differently). Is a list display a mode? Is a string a mode? Is a float a mode? I'd say yes, technically they are. There's a substantial difference between writing `sys.exit()` and `"sys.exit()"`. Same for 'comment mode': `# sys.exit()` , and most IDEs have shortcuts for switching it on and off. However I think a major difference is that all these can exist in different contexts, also in isolation, and hence we think of them as self-contained entities. The modal aspect is only relevant to the compiler. In some sense, maybe, but to me the critical factor is that nobody talks about "list mode", "string mode", let alone "float mode". Its about the mental model.
With `func(foo, **, bar, baz, quux)` if I use `**` as a pseudo-argument, the interpreter switches to "auto-fill" mode and everything that follows that (until the end of the function call) has to be interpreted according to the mode. With your proposal if I use `**{` then the interpreter similarly switches to auto-fill mode and everything that follows until the next `}` is affected by that mode. There's no big difference. The idea behind `**` is that you could also use `**kwargs` instead and `**` is just the case when you don't have anything to unpack (as an analogy to `*args` vs. `*` in a function definition when you don't need to consume varargs). A few people immediately started describing this as a mode, without prompting. I think it is a very natural way of thinking about it. I think what obscures the modal aspect of `**{...}` is the fact that it somehow looks like an expression, but it isn't. It's special syntax that can only be used in function calls, to tell the compiler that it should autofill the parameter names. It's a mode in disguise. And we have no way of turning the mode off. So if there is every a proposal to allow positional arguments to follow keyword arguments, it won't be compatible with auto-fill mode. That is true, it doesn't have an explicit end token. However I'm not convinced that ruling out the positional-after-keyword-arguments option is a relevant argument against it. With `func(foo, **{bar, baz, quux})` the mental model is closer to ordinary argument or dict unpacking. Nobody refers to this:
func(spam, *[eggs, cheese, aardvark], hovercraft)
as "list mode" or "argument unpacking mode". It's just "unpacking a list" or similar. No-one thinks about the interpreter entering a special "collect list mode" even if that's what the parser actually does, in some sense. We read the list as an entity, which then gets unpacked.
In this example `*` and `[eggs, cheese, aardvark]` are distinct entities, the latter can exist without the former and it has the exact same meaning, independent of context. So we think about it as a list that gets unpacked (and the list being a concept that can exist in isolation, without unpacking). With the proposed syntax we have `**{eggs, cheese, aardvark}` and here the `**` and `{...}` parts are inseparable. Even though the latter could exist in isolation but then it means something completely different. In the `**{...}` listing of names these not only refer to their objects as usual but also serve the purpose of identifying keyword parameter names. This unusual extension of identifier meaning is limited by `**{` and `}` and hence I consider it a mode, just like `**,` and `)`.
Likewise for dict unpacking: nobody thinks of `{'a': expr}` as entering "dict mode". You just make a dict, then unpack it.
And nobody (I hope...) will think of keyword shortcut as a mode:
func(foo, **{bar, baz}, quux=1)`
It's just unpacking an autofilled set of parameter names. Not a mode at all. And notice that there is absolutely no difficulty with some future enhancement to allow positional arguments after keyword arguments. "unpacking an autofilled set of parameter names" implies that two distinct actions take place. First the set of parameter names is autofilled and converted into something that can be unpacked, and then it actually gets unpacked. But that's not what is happening, you can't have `**({bar, baz})`. What you *describe* is another proposal, namely using e.g. `{:bar, :baz}` to construct the mapping and then `**` to unpack it; `**({:bar, :baz})` works without problem. So the `**{bar, baz}` syntax is not actually unpacking anything, it's a hint for the compiler to treat `bar, baz` as `bar=bar, baz=baz` and hence it's a mode switch.
And notice that there is absolutely no difficulty with some future enhancement to allow positional arguments after keyword arguments.
We've already discussed in this thread that we shouldn't fear conflicting with other (real or hypothetical) proposals, even if they're likely. As I see it, the chance of allowing positional arguments after keyword arguments is basically zero. The restriction is intentionally there for a good reason. And quoting your next message: All else being equal, we should prefer the syntax that doesn't rule out
future development.
I don't think all else is equal. I think the downside of a pseudo-expression far, far outweighs the downside of conflicting with unlikely hypothetical future proposals. Also, the way you're arguing against possibly conflicting with some future enhancement, I'm not sure why you'd ever support said enhancement, given that it would still potentially conflict with other possible enhancements in the even more distant future.
On Sun, Apr 19, 2020 at 02:10:21PM +0200, Alex Hall wrote:
And notice that there is absolutely no difficulty with some future enhancement to allow positional arguments after keyword arguments.
We've already discussed in this thread that we shouldn't fear conflicting with other (real or hypothetical) proposals, even if they're likely. As I see it, the chance of allowing positional arguments after keyword arguments is basically zero. The restriction is intentionally there for a good reason.
Python already allows positional arguments after keyword arguments: py> sorted(reverse=True, *([1, 4, 2, 3],)) [4, 3, 2, 1]
And quoting your next message:
All else being equal, we should prefer the syntax that doesn't rule out future development.
I don't think all else is equal.
That's a fair opinion. [...]
Also, the way you're arguing against possibly conflicting with some future enhancement, I'm not sure why you'd ever support said enhancement, given that it would still potentially conflict with other possible enhancements in the even more distant future.
There is a qualitative difference between: "Your proposal will rule out this specific thing and should be weighed up in light of that" and "Your proposal could rule out some unknown thing that nobody has thought of, so it should be rejected!" I have an actual, concrete possible enhancement in mind: relaxing the restriction on parameter order. Do you have an actual, concrete future enhancement in mind that relaxing the restriction would conflict with? -- Steven
On Mon, Apr 20, 2020 at 2:48 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Sun, Apr 19, 2020 at 02:10:21PM +0200, Alex Hall wrote:
And notice that there is absolutely no difficulty with some future enhancement to allow positional arguments after keyword arguments.
We've already discussed in this thread that we shouldn't fear conflicting with other (real or hypothetical) proposals, even if they're likely. As I see it, the chance of allowing positional arguments after keyword arguments is basically zero. The restriction is intentionally there for a good reason.
Python already allows positional arguments after keyword arguments:
py> sorted(reverse=True, *([1, 4, 2, 3],)) [4, 3, 2, 1]
Haha, that's very clever. I had to think for a bit about why that's allowed. So let me specify: we don't allow non-variadic positional arguments after keyword arguments, and I don't think we ever will or should.
I have an actual, concrete possible enhancement in mind: relaxing the restriction on parameter order.
What? Do you think that the current restriction is bad, and we should just drop it? Why?
On Mon, Apr 20, 2020 at 11:15:32AM +0200, Alex Hall wrote:
On Mon, Apr 20, 2020 at 2:48 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Sun, Apr 19, 2020 at 02:10:21PM +0200, Alex Hall wrote:
And notice that there is absolutely no difficulty with some future enhancement to allow positional arguments after keyword arguments.
[...]
I have an actual, concrete possible enhancement in mind: relaxing the restriction on parameter order.
What? Do you think that the current restriction is bad, and we should just drop it? Why?
No, I have no opinion at the moment on whether we should relax that restriction. I'm saying that the mode-shift suggestion: func(arg, name=value, *, # change to auto-fill mode alpha, beta, gamma, ) will rule out any further relaxation on that restriction, and that is a point against it. That's a concrete enhancement that we might allow some time. Whether *I personally* want that enhancement is irrelevant. You on the other hand, claim that my suggestion: func(arg, name=value, **{alpha, beta, gamma}, ) will also rule out some unspecified, unknown, unimagined future enhancements. I'm saying that's a weak argument, unless you have a specific enhancement in mind. Note what I am **not** doing: I'm not saying that your bare star argument suggestion is bad because it will rule out using a bare star argument for some other purpose. I'm saying that it will rule out another language change which people may, or may not, prefer in the future. -- Steven
I'm saying that the mode-shift suggestion:
func(arg, name=value, *, # change to auto-fill mode alpha, beta, gamma, )
will rule out any further relaxation on that restriction, and that is a point against it. That's a concrete enhancement that we might allow some time. Whether *I personally* want that enhancement is irrelevant.
It doesn't necessarily have to. There's always the possibility of a second mode switch, perhaps using the same indicator as the one related to positional-only signature syntax: func(arg, name=value, *, # change to auto-fill mode alpha, beta, gamma, /, # change back to normal positional mode delta, epsilon )
On 20.04.20 12:52, Steven D'Aprano wrote:
On Mon, Apr 20, 2020 at 11:15:32AM +0200, Alex Hall wrote:
On Mon, Apr 20, 2020 at 2:48 AM Steven D'Aprano <steve@pearwood.info> wrote:
I have an actual, concrete possible enhancement in mind: relaxing the restriction on parameter order.
What? Do you think that the current restriction is bad, and we should just drop it? Why? No, I have no opinion at the moment on whether we should relax that restriction. I'm saying that the mode-shift suggestion:
func(arg, name=value, *, # change to auto-fill mode alpha, beta, gamma, )
will rule out any further relaxation on that restriction, and that is a point against it. That's a concrete enhancement that we might allow some time. Whether *I personally* want that enhancement is irrelevant.
You on the other hand, claim that my suggestion:
func(arg, name=value, **{alpha, beta, gamma}, )
will also rule out some unspecified, unknown, unimagined future enhancements. I'm saying that's a weak argument, unless you have a specific enhancement in mind.
This rules out the possibility to treat sets as mappings from their elements to `True`. Unlikely, but so are positional arguments following keyword arguments.
On Mon, Apr 20, 2020 at 12:57 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Mon, Apr 20, 2020 at 11:15:32AM +0200, Alex Hall wrote:
On Mon, Apr 20, 2020 at 2:48 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Sun, Apr 19, 2020 at 02:10:21PM +0200, Alex Hall wrote:
And notice that there is absolutely no difficulty with some future enhancement to allow positional arguments after keyword arguments.
[...]
I have an actual, concrete possible enhancement in mind: relaxing the restriction on parameter order.
What? Do you think that the current restriction is bad, and we should just drop it? Why?
No, I have no opinion at the moment on whether we should relax that restriction. I'm saying that the mode-shift suggestion:
func(arg, name=value, *, # change to auto-fill mode alpha, beta, gamma, )
will rule out any further relaxation on that restriction, and that is a point against it. That's a concrete enhancement that we might allow some time. Whether *I personally* want that enhancement is irrelevant.
You on the other hand, claim that my suggestion:
func(arg, name=value, **{alpha, beta, gamma}, )
will also rule out some unspecified, unknown, unimagined future enhancements. I'm saying that's a weak argument, unless you have a specific enhancement in mind.
No I'm not claiming anything close to that. We're misunderstanding each other quite badly. We're both talking about possible enhancements involving allowing non-variadic positional-looking arguments after unpacked keyword arguments: A: make them auto-named B: simply allow them and interpret them as positional arguments C: something we haven't thought of yet D: something else we haven't thought of yet I thought you were against A because it would block out C, and I was saying that following that logic when we eventually think of C it isn't gonna happen either because it will block out D. Since you say that ruling out unimagined future enhancements is a weak argument, I take it to mean that I misunderstood and you weren't arguing against A because of C, but rather because of B. Now that I think we've cleared that up: B is a terrible idea, and if anyone is considering it, I'd support A for no other reason than to prevent B from happening.
participants (25)
-
Alex Hall -
Andrew Barnert -
Brett Cannon -
Calvin Spealman -
Chris Angelico -
Christopher Barker -
Dan Sommers -
David Mertz -
Dominik Vilsmeier -
Eric V. Smith -
Greg Ewing -
Ivan Levkivskyi -
Joao S. O. Bueno -
Kyle Lahnakoski -
Kyle Stanley -
M.-A. Lemburg -
MRAB -
oliveira.rodrigo.m＠gmail.com -
Paul Svensson -
Rhodri James -
Richard Damon -
Ricky Teachey -
Sebastian Berg -
Stephen J. Turnbull -
Steven D'Aprano