Ricky Teachey's email to me on PEP 637, and my responses
SUMMARY: I share with you extracts from a personal email Rick Teachey sent me, on my involvement in PEP 637 (keyword arguments in item access), and my answers to some important questions he asks. BACKGROUND: Last week, prior to my publicly stating my resignation as a co-author of PEP 637, Ricky Teachey sent me a personal email. I found it so deep and perceptive that I asked him for permission to share extracts with you on this list. I'm pleased that he's allowed this, and I'm most grateful to him for this. The rest of this post consists of extracts from his email, and some brief comments from me. RICKY'S EMAIL and MY RESPONSES: Ricky Teachey wrote:
PEP 1 says:
The PEP author is responsible for building consensus within the community and documenting dissenting opinions.
[The PEP] should clearly explain why the existing language specification is
inadequate to address the problem that the PEP solves.
I want keyword arguments in item access to be added to Python, and I want it to be done well. Something that won't cause difficulty 3 to 5 years from now. PEP 1, as above, has good advice that contributes to this goal. These are areas I particularly wanted to contribute to, as a PEP author. I have no wish to use improper methods to obtain what I think is best.
Ricky Teachey wrote:
At least when convenient, I'm all for experiments prior to making hard-to-reverses changes to Python. PEP 1 says:
A basic idea of kwkey is an equivalence between d[1, 2, a=3, b=4] = val X(1, 2, a=3, b=4)[d] = val provided X and item keyword access are given the same semantics. This idea will I think help greatly with the 'teach users' section of the PEP. Observation, experiment and theory are fundamental to science. Astronomy has the observatory. Biology, chemistry and physics have the laboratory. All research scientists publish. We can use kwkey to do experiments, without first having to create a new version of Python. It makes experiments cheap. My opinions result in predictions for the result of these experiments. But rather than discuss opinions, I'd like us to do experiments and share our experience. I've been surprised by the results of my own experiments using kwkey! We can all learn from experiments, particularly if we're observant. Ricky Teachey wrote:
I applaud your suggestion, that the steering council use kwkey as part of making a choice. And I think we're getting closer to having a second PEP. Here's an idea. At present both of d[1:2] d[1:2, 3:4, 5, 6] are valid syntax, but neither of d[(1:2)] d[(1:2, 3:4, 5, 6)] are valid syntax. This is, I think, a bit of an anomaly. I suggest a PEP that extended the syntax to allow (1:2) (1:2, 3:4, 5, 6) to be expressions (to be used as function arguments, with the existing item access semantics). I think this is something which most supporters of keyword arguments in item access could support. And which few opponents of keyword arguments in item access would oppose. It's a small and fairly safe change. And this change would, via constructs such as X((1:2), a=(3:4))[d] = val allow us to get perhaps 75% of the benefits provided by the larger (and more risky) change that would allow d[1:2, a=3:4] = val I think we could get (1:2) etc expressions into the next version of Python (in 2021). And then, if X((1:2), a=(3:4))[d] = val is widely used, and the semantics of X agreed upon, perhaps d[1:2, a=(3:4)] = val could be added in 2022. My big concern is making a big change now, whose problems emerge in 2023 to 2025. I hope this helps. And I thank Ricky both for his thoughtful personal email, and for giving me permission to share it with you. -- Jonathan
Jonathan thanks for your desire to make PEP 637, and kwd arguments in subscripting in general, as good as possible. I am bowing out of the conversation from this point on though. I am very happy you found my comments helpful and if there is a second competing PEP, I'll be very interested to read it. Rick. On Fri, Oct 23, 2020, 11:13 AM Jonathan Fine <jfine2358@gmail.com> wrote:
It's been a while since this was posted, but a thought: Here's an idea. At present both of
indeed -- it's been very much a part of this conversation that: thing[i,j,k] is exactly equivalent to thing[(i,j,k)] because the tuple is "created by" the parentheses. That is, nothing special is going on inside the brackets. After all: In [4]: x = 1, 2, 3 In [5]: y = (1, 2, 3) In [6]: x == y Out[6]: True and thing[1:2, 3:4] does create a tuple, and pass it on to the __*item__ dunder. and thing[slice(1,2), slice(1,2)] is exactly equivalent to thing[(slice(1,2), slice(1,2))] (or course) So the only missing piece here is that that slice syntax can only be used directly inside the square brackets. This, of course, has been the case for the history of Python, and it's worked pretty well, but I do think it becomes more of an issue/constraint if/when we add complexity to the indexing -- e.g. adding keyword arguments. And there are other reasons to allow slice syntax elsewhere anyway. As the colon is used elsewhere in Python, it's not as simple as simply allowing slice syntax as a legal expression that can be used everywhere any other expression can be used. I know I haven't thought it out yet, but I would be supportive of a PEP to extend the "domains" in which slice syntax can be used, if someone does have the energy to think it out and write a PEP. I do think it should be completely separate from PEP 637, but if PEP 637, or something like it, were accepted, it would provide additional motivation for this idea. I suggest a PEP that extended the syntax to allow
One adjustment -- if, e.g. (1:2) becomes a legal expression for creating a slice, then we're done -- nothing special about function arguments, or item access semantics -- it would simply be a "literal" for a slice object, and could be used anywhere any other expression could be used, and would mean whatever a slice "means" in that context. -CHB PS: It's hard to imagine this hasn't been proposed in the past, so the next step is to do the research and find out if indeed it has, and if so, why it didn't happen. A quick googling reveals the rejected PEP 204 https://www.python.org/dev/peps/pep-0204/ which proposed a "range literal", which used slice syntax to create a range -- but that's something different (and, indeed, range objects are different than they were back then as well) So it doesn't look like this ever made it to a PEP, but it probably has been discussed on this list ... -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Christopher Barker writes:
thing[i,j,k] is exactly equivalent to thing[(i,j,k)] because the tuple is "created by" the parentheses.
Is that correct? As I understand it, the tuple is created by the commas, and the parentheses are basically thrown away by the parser, as usual.
But is it exactly equivalent to "thing[1:2, 1:2]"? I believe it is, and if so
So the only missing piece here is that that slice syntax can only be used directly inside the square brackets.
is exactly correct, and somewhat anomolous, since slices are perfectly good objects.
As the colon is used elsewhere in Python,
As a statement clause separator, which becomes ambiguous: if thing: x However, I would guess this you can resolve this with a grammar that says "if thing:" is unconditionally a conditional clause, and that in "if (thing : x)" has to parse "thing : x" to slice syntax first, then look for the terminating colon for the if clause. Similarly for all other compound statement clauses: an unparenthesized colon terminates the clause immediately. And in dict displays, where generalized slices could be used as components, rather than treating ':' merely as a separator. (I write "generalized" because I presume slices are currently specialized to ints). Is there anything else? I think one big bugaboo here is in the word "generalized", because you'd enable errors like myslice = 'key' : 'value' for item in alist[myslice]: foo(item) and I just don't know if that's plausible enough to worry about. Also, slice syntax admits partial slices, so you would probably need to parse "naked" slices like "'key':", and what in the world would "::'some arbitrary string'" mean, and how could you justify ruling it out arbitrarily? (Wouldn't consenting adults apply, so for example def myzip(slicearg, iterables): myiterables = [itertools.islice(it, slicearg.start, slicearg.stop) for it in iterables] return zip(myiterables, strict=slicearg.step) "should" be allowed?)
I do think it should be completely separate from PEP 637
+1
I don't like the requirement for parentheses of a single slice expression, and the second example is presumably not slice syntax, but tuple syntax. I.e., given the requirement exemplified by (1:2), the second example should be written ((1:2), (3:4), 5, 6) which is not as attractive. Except in function syntax, where they are a required operator, elsewhere in Python parentheses are used only for controlling order of evaluation as far as I know, with the partial exception of genexps. I write "partial" because in most places a genexp must be parenthesized, but it's acceptable as the only actual argument to a function without parentheses. And of course a single genexp may contain an arbitrarily large number of clauses, so it's quite different from an n-ary operator with very small n.
I don't think it's that different, both ranges and slices (as they exist now) are mostly used to control iteration. It's just that the iteration controlled by slices is implicit in the creation of a new sequence. I would guess any arguments purely about the syntax would apply here as well. My current opinion is that (1) generalized slices are just too messy, the constructor notation "slice(start, stop, step)" is not at all obnoxious, and (3) the colons in compound statements, dict displays, and sequence slices are not operators in any of the three cases, and can't be treated that way. Specifically, in all three cases they're syntactically separators, but in compound statements they don't separate sequences of objects at all, and in slices and dict display items the objects they separate are of quite different types and the implied sequences are even of different lengths (3 vs. 2). It's messy enough that I'm not going to spend any more skull sweat on this, we're fine as is for slices and slice syntax (IMO).
On Tue, Oct 27, 2020 at 11:24 PM Chris Angelico <rosuav@gmail.com> wrote:
yes, thanks -- that was a typo, and an unfortunate one, as it's kinda the key point :-( -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 2020-10-27 23:10, Stephen J. Turnbull wrote:
Non-int slices are valid and are used for instance by pandas, where you can do things like `mydataframe.loc['some_row':'other_row', 'some_column':'other_column']` . -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
Christopher Barker and Stephen J. Turnbull wrote:
As a statement clause separator, which becomes ambiguous:
if thing: x
Yes. Very good. Well done, both of you. Now consider this. PEP 643 allows for things like obj[a=x:y] and "may open up the open the possibility" of allowing fn(a=x:y) The so-to-say obvious way to allow fn(a=x:y) is to allow a bare x:y to function as an expression. This is because at present in the syntax fn(a=SOMETHING) the SOMETHING is an expression. Why would we want to change that? There's a cost, particularly when we read and write code, in changing that syntactic rule. With me so far. Now consider if x:y:z What could it mean? (I'm assuming bare slices are allowed.) It could be either of if (x:y): z if x: (y:z) Now let's go back to PEP 643 allowing obj[a=x:y] which "may open up the open the possibility" of allowing fn(a=x:y) Here's how I see things. Allowing both of obj[a=x:y] fn(a=x:y) opens up the possibility of if x:y:z which in my opinion is a can of worms. Please, let's not go there. Not convinced? Both if a:b: x = y and if a:b: x = y are valid syntax after 'promoting' some colons into slices. But we have to look ahead to the next line, to figure what to do. For this reason I think it better to forbid the short-cut obj[a=x:y] and allow the explicit statement obj[a=(x:y)] by making (x:y) and similar constructs valid syntax for an slice expression, which can be used anywhere. While we're here def fn(u:v=w): pass is valid syntax for a function where u has type v u had default value w I say allow def fn(u:v=(x:y)): pass def fn(u:v=(:)): pass and forbid def fn(u:v=x:y): pass def fn(u:v=:): pass I think PEP 637 would prefer the other way around. By the way, there's a gap in PEP 637. This PEP talks about slice notation in function calls, but not in function definitions. But default values are values that migrate from a function definition to a function call. I'm warming to the idea of a PEP concerned solely with the syntax for 1. item access 2. function definitions and calls 3. slices Finally, thank you Christopher and Stephen, for your concise and deep contribution. -- Jonathan
Jonathan thanks for your desire to make PEP 637, and kwd arguments in subscripting in general, as good as possible. I am bowing out of the conversation from this point on though. I am very happy you found my comments helpful and if there is a second competing PEP, I'll be very interested to read it. Rick. On Fri, Oct 23, 2020, 11:13 AM Jonathan Fine <jfine2358@gmail.com> wrote:
It's been a while since this was posted, but a thought: Here's an idea. At present both of
indeed -- it's been very much a part of this conversation that: thing[i,j,k] is exactly equivalent to thing[(i,j,k)] because the tuple is "created by" the parentheses. That is, nothing special is going on inside the brackets. After all: In [4]: x = 1, 2, 3 In [5]: y = (1, 2, 3) In [6]: x == y Out[6]: True and thing[1:2, 3:4] does create a tuple, and pass it on to the __*item__ dunder. and thing[slice(1,2), slice(1,2)] is exactly equivalent to thing[(slice(1,2), slice(1,2))] (or course) So the only missing piece here is that that slice syntax can only be used directly inside the square brackets. This, of course, has been the case for the history of Python, and it's worked pretty well, but I do think it becomes more of an issue/constraint if/when we add complexity to the indexing -- e.g. adding keyword arguments. And there are other reasons to allow slice syntax elsewhere anyway. As the colon is used elsewhere in Python, it's not as simple as simply allowing slice syntax as a legal expression that can be used everywhere any other expression can be used. I know I haven't thought it out yet, but I would be supportive of a PEP to extend the "domains" in which slice syntax can be used, if someone does have the energy to think it out and write a PEP. I do think it should be completely separate from PEP 637, but if PEP 637, or something like it, were accepted, it would provide additional motivation for this idea. I suggest a PEP that extended the syntax to allow
One adjustment -- if, e.g. (1:2) becomes a legal expression for creating a slice, then we're done -- nothing special about function arguments, or item access semantics -- it would simply be a "literal" for a slice object, and could be used anywhere any other expression could be used, and would mean whatever a slice "means" in that context. -CHB PS: It's hard to imagine this hasn't been proposed in the past, so the next step is to do the research and find out if indeed it has, and if so, why it didn't happen. A quick googling reveals the rejected PEP 204 https://www.python.org/dev/peps/pep-0204/ which proposed a "range literal", which used slice syntax to create a range -- but that's something different (and, indeed, range objects are different than they were back then as well) So it doesn't look like this ever made it to a PEP, but it probably has been discussed on this list ... -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
Christopher Barker writes:
thing[i,j,k] is exactly equivalent to thing[(i,j,k)] because the tuple is "created by" the parentheses.
Is that correct? As I understand it, the tuple is created by the commas, and the parentheses are basically thrown away by the parser, as usual.
But is it exactly equivalent to "thing[1:2, 1:2]"? I believe it is, and if so
So the only missing piece here is that that slice syntax can only be used directly inside the square brackets.
is exactly correct, and somewhat anomolous, since slices are perfectly good objects.
As the colon is used elsewhere in Python,
As a statement clause separator, which becomes ambiguous: if thing: x However, I would guess this you can resolve this with a grammar that says "if thing:" is unconditionally a conditional clause, and that in "if (thing : x)" has to parse "thing : x" to slice syntax first, then look for the terminating colon for the if clause. Similarly for all other compound statement clauses: an unparenthesized colon terminates the clause immediately. And in dict displays, where generalized slices could be used as components, rather than treating ':' merely as a separator. (I write "generalized" because I presume slices are currently specialized to ints). Is there anything else? I think one big bugaboo here is in the word "generalized", because you'd enable errors like myslice = 'key' : 'value' for item in alist[myslice]: foo(item) and I just don't know if that's plausible enough to worry about. Also, slice syntax admits partial slices, so you would probably need to parse "naked" slices like "'key':", and what in the world would "::'some arbitrary string'" mean, and how could you justify ruling it out arbitrarily? (Wouldn't consenting adults apply, so for example def myzip(slicearg, iterables): myiterables = [itertools.islice(it, slicearg.start, slicearg.stop) for it in iterables] return zip(myiterables, strict=slicearg.step) "should" be allowed?)
I do think it should be completely separate from PEP 637
+1
I don't like the requirement for parentheses of a single slice expression, and the second example is presumably not slice syntax, but tuple syntax. I.e., given the requirement exemplified by (1:2), the second example should be written ((1:2), (3:4), 5, 6) which is not as attractive. Except in function syntax, where they are a required operator, elsewhere in Python parentheses are used only for controlling order of evaluation as far as I know, with the partial exception of genexps. I write "partial" because in most places a genexp must be parenthesized, but it's acceptable as the only actual argument to a function without parentheses. And of course a single genexp may contain an arbitrarily large number of clauses, so it's quite different from an n-ary operator with very small n.
I don't think it's that different, both ranges and slices (as they exist now) are mostly used to control iteration. It's just that the iteration controlled by slices is implicit in the creation of a new sequence. I would guess any arguments purely about the syntax would apply here as well. My current opinion is that (1) generalized slices are just too messy, the constructor notation "slice(start, stop, step)" is not at all obnoxious, and (3) the colons in compound statements, dict displays, and sequence slices are not operators in any of the three cases, and can't be treated that way. Specifically, in all three cases they're syntactically separators, but in compound statements they don't separate sequences of objects at all, and in slices and dict display items the objects they separate are of quite different types and the implied sequences are even of different lengths (3 vs. 2). It's messy enough that I'm not going to spend any more skull sweat on this, we're fine as is for slices and slice syntax (IMO).
On Tue, Oct 27, 2020 at 11:24 PM Chris Angelico <rosuav@gmail.com> wrote:
yes, thanks -- that was a typo, and an unfortunate one, as it's kinda the key point :-( -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
On 2020-10-27 23:10, Stephen J. Turnbull wrote:
Non-int slices are valid and are used for instance by pandas, where you can do things like `mydataframe.loc['some_row':'other_row', 'some_column':'other_column']` . -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown
Christopher Barker and Stephen J. Turnbull wrote:
As a statement clause separator, which becomes ambiguous:
if thing: x
Yes. Very good. Well done, both of you. Now consider this. PEP 643 allows for things like obj[a=x:y] and "may open up the open the possibility" of allowing fn(a=x:y) The so-to-say obvious way to allow fn(a=x:y) is to allow a bare x:y to function as an expression. This is because at present in the syntax fn(a=SOMETHING) the SOMETHING is an expression. Why would we want to change that? There's a cost, particularly when we read and write code, in changing that syntactic rule. With me so far. Now consider if x:y:z What could it mean? (I'm assuming bare slices are allowed.) It could be either of if (x:y): z if x: (y:z) Now let's go back to PEP 643 allowing obj[a=x:y] which "may open up the open the possibility" of allowing fn(a=x:y) Here's how I see things. Allowing both of obj[a=x:y] fn(a=x:y) opens up the possibility of if x:y:z which in my opinion is a can of worms. Please, let's not go there. Not convinced? Both if a:b: x = y and if a:b: x = y are valid syntax after 'promoting' some colons into slices. But we have to look ahead to the next line, to figure what to do. For this reason I think it better to forbid the short-cut obj[a=x:y] and allow the explicit statement obj[a=(x:y)] by making (x:y) and similar constructs valid syntax for an slice expression, which can be used anywhere. While we're here def fn(u:v=w): pass is valid syntax for a function where u has type v u had default value w I say allow def fn(u:v=(x:y)): pass def fn(u:v=(:)): pass and forbid def fn(u:v=x:y): pass def fn(u:v=:): pass I think PEP 637 would prefer the other way around. By the way, there's a gap in PEP 637. This PEP talks about slice notation in function calls, but not in function definitions. But default values are values that migrate from a function definition to a function call. I'm warming to the idea of a PEP concerned solely with the syntax for 1. item access 2. function definitions and calls 3. slices Finally, thank you Christopher and Stephen, for your concise and deep contribution. -- Jonathan
participants (6)
-
Brendan Barnwell -
Chris Angelico -
Christopher Barker -
Jonathan Fine -
Ricky Teachey -
Stephen J. Turnbull