PEP 637 - Support for indexing with keyword arguments: request for feedback for SC submission
Hi all, I would like to request feedback by python-dev on the current implementation of PEP 637 - Support for indexing with keyword arguments. https://www.python.org/dev/peps/pep-0637/ The PEP is ready for SC submission and it has a prototype implementation ready, available here (note, not reviewed, but apparently fully functional) https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-imp... (note: not sure if there's a preference for the link to be to the diff or to the branch, let me know if you prefer I change the PEP link) Thank you for your help. -- Kind regards, Stefano Borini
Anyone? I'm +1 on this PEP as it is, and I imagine a few other core devs are too (Brandt, Steven). If nobody responds before the end of the week I think we can say that there was no disagreement and Stefano can submit PEP 637 to the SC. On Tue, Feb 2, 2021 at 3:43 AM Stefano Borini <stefano.borini@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 2/2/21 12:36 PM, Stefano Borini wrote:
+1 from me. This looks quite useful in certain areas and natural to use. I like how this makes the dunder implementations (usually libraries), rather than users, deal with most of the corner cases -- but still allows libraries that don't need this to not care. The PEP does lack a "How to teach" section. "Corner case 3" concludes that "best practice suggests that keyword subscripts should be flagged as keyword-only when possible": def __getitem__(self, index, *, direction='north'): If the PEP is accepted, this should be mentioned in the __(get|set|del)item__ documentation and shown in all relevant examples. Looking at corner case 1, it would also be useful to nudge people to use positional-only arguments whenever they accept arbitrary keyword ones. (The same goes for function definitions, but tutorials for those are already written): def __getitem__(self, index, /, **named_axes): It would be great if what gets copied to StackOverflow is examples of good practices :)
On Thu, 4 Feb 2021 at 01:21, Petr Viktorin <encukou@gmail.com> wrote:
The PEP does lack a "How to teach" section.
I can add it tonight, and I think it would clarify some corner cases to make them more natural. Do you have something specific you are concerned about, when it comes to "teachability" (stress point, hard to grok, expected common mistakes)? I'd focus only on the use of the feature "from the outside", that is, using the indexing operation as a common user. For implementing the dunders, it's an advanced feature and it's unlikely to be discussed during basic teaching sessions. -- Kind regards, Stefano Borini
On Thu, 4 Feb 2021 at 10:34, Stefano Borini <stefano.borini@gmail.com> wrote:
I think there's value in having a note about how to promote "best practices" in implementing the dunders under "How to teach". A relevant point about the proposal is whether it's too easy for people to fall into traps when implementing the functionality, and this is a good section to discuss that. It should be covered by noting where in the documentation it'll be discussed, so I don't think it's a big deal, but it's worth having. Paul
On 2 Feb 2021, at 12:36, Stefano Borini wrote:
It seems to me, that what complicates the specification is the need for backwards compatibility. If that wasn't an issue, we could make indexing operations behave exactly like function calls. Handling the additional argument for `__setitem__` could be done the same way that passing `self` in a method call is done: By passing an additional positional argument (in this case as the second argument after `self`), So: * `foo[1]` is `type(foo).__getitem__(foo, 1)` * `foo[1, 2]` is `type(foo).__getitem__(foo, 1, 2)` * `foo[(1, 2)]` is `type(foo).__getitem__(foo, (1, 2))` * `foo[1] = 3` is `type(foo).__setitem__(foo, 3, 1)` * `foo[1, 2] = 3` is `type(foo).__setitem__(foo, 3, 1, 2)` * `foo[(1, 2)] = 3` is `type(foo).__setitem__(foo, 3, (1, 2))` But of course this isn't backwards compatible with respect to the treatment of tuple arguments and the argument order in `__setitem__`. However it is **much** easier to remember and to teach. The PEP rejects the idea to implement this approach via a new set of dunder methods (e.g. `__getitem_ex__`, `__setitem_ex__` and `__delitem_ex__`) for performance reasons, but would it make sense, to mark existing `__getitem__`, `__setitem__` and `__delitem__` methods as supporting the new calling convention via a decorator? i.e something like: ```python class X: @newstyle_item def __getitem__(self, x, y, z=42): ... @newstyle_item def __setitem__(self, value, x, y, z=42): ... @newstyle_item def __detitem__(self, x, y, z=42): ... ``` This wouldn't require an additional dictionary lookup, but just a check of a bit in the function object.
Thank you for your help.
Servus, Walter
I missed the discussion around the PEP. I don't mean to start another one, I'd just like a clarification. The PEP describes the new functionality, and the interfaces, and that's all fine. But I didn't see where it discussed where this technology would be used. Would this mainly be used by third-party math libraries (Pandas, NumPy), like the @ operator, or is there a plan to use this in Python's own library or builtin objects? If the latter, can you go into the specifics? I'm guessing the typing module would use it, as illustrated by one of the examples, but beyond that I can't imagine how this would be used by e.g. dicts and lists. Thanks, //arry/ On 2/2/21 3:36 AM, Stefano Borini wrote:
dicts and lists will keep working as before. They will not support keyword arguments (probably ever, as there is no clear semantic for them) and the current implementation simply throws an error if the user tries to. Other classes will take advantage of the syntax for enhanced behavior. We are not specifying what that specific behavior might be (as in the case of the @ operator, which is "kind of bound" to a matmul operation, although it can be used for anything else, like / is used as a join operator in pathlib, even though its typical meaning is a division operator). Any class that is willing to accept keyword arguments for the indexing operation is responsible for documenting which arguments are accepted and what is the meaning of their use. We do not prescribe any rule, although some uses can probably be classified as bad practice. On Fri, 5 Feb 2021 at 10:30, Larry Hastings <larry@hastings.org> wrote:
-- Kind regards, Stefano Borini
Yes. It's already supported in the branch. You can do MyType[whatever=5]. One could possibly change dict to allow both the specification dict[str, int] and dict[key=str, value=int], but this specific change of the dict class is not part of the implementation nor the PEP (nor I expect it to happen, for what matters, and I would not request it). On Sat, 6 Feb 2021 at 11:10, Larry Hastings <larry@hastings.org> wrote:
-- Kind regards, Stefano Borini
Lurking around the discussions I started to wonder whether this syntax actually worths the burden it introduces. As you have commented out earlier, there are no use cases in the built-in types, perhaps a research would be great regarding possible use cases of keyword arguments in and out of the stdlib. Here is an example: https://www.python.org/dev/peps/pep-0572/#examples-from-the-python-standard-... that was done for PEP 572, which actually helped me a lot to understand/comprehend the proposal. Now looking at PEP 637, even with the theoretical examples given, it seems like we are introducing a redundant syntax that has no other value than just complicating the already complex subscript notation and becoming a bad alternative to function call syntax. On Tue, Feb 2, 2021 at 2:40 PM Stefano Borini <stefano.borini@gmail.com> wrote:
On Sat, 6 Feb 2021 at 08:56, Batuhan Taskaya <isidentical@gmail.com> wrote:
Lurking around the discussions I started to wonder whether this syntax actually worths the burden it introduces.
It has been discussed for weeks on py-ideas.
Nor there is for the @ operator, but it is still relevant for external libraries.
you can't do: f(x, y=3) = 5 but you can do a[x, y=3] = 5 pandas has relied on workarounds like iloc() to work around the lack of an indexing operator with keyword arguments, and we need them anyway for specifying types, now that the indexing operator is "abused of notation" all the time for typing. Of course the meaning of a[x, y=3] = 5 is up to the implementation of a. We don't prescribe anything on that.
-- Kind regards, Stefano Borini
Anyone? I'm +1 on this PEP as it is, and I imagine a few other core devs are too (Brandt, Steven). If nobody responds before the end of the week I think we can say that there was no disagreement and Stefano can submit PEP 637 to the SC. On Tue, Feb 2, 2021 at 3:43 AM Stefano Borini <stefano.borini@gmail.com> wrote:
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-c...>
On 2/2/21 12:36 PM, Stefano Borini wrote:
+1 from me. This looks quite useful in certain areas and natural to use. I like how this makes the dunder implementations (usually libraries), rather than users, deal with most of the corner cases -- but still allows libraries that don't need this to not care. The PEP does lack a "How to teach" section. "Corner case 3" concludes that "best practice suggests that keyword subscripts should be flagged as keyword-only when possible": def __getitem__(self, index, *, direction='north'): If the PEP is accepted, this should be mentioned in the __(get|set|del)item__ documentation and shown in all relevant examples. Looking at corner case 1, it would also be useful to nudge people to use positional-only arguments whenever they accept arbitrary keyword ones. (The same goes for function definitions, but tutorials for those are already written): def __getitem__(self, index, /, **named_axes): It would be great if what gets copied to StackOverflow is examples of good practices :)
On Thu, 4 Feb 2021 at 01:21, Petr Viktorin <encukou@gmail.com> wrote:
The PEP does lack a "How to teach" section.
I can add it tonight, and I think it would clarify some corner cases to make them more natural. Do you have something specific you are concerned about, when it comes to "teachability" (stress point, hard to grok, expected common mistakes)? I'd focus only on the use of the feature "from the outside", that is, using the indexing operation as a common user. For implementing the dunders, it's an advanced feature and it's unlikely to be discussed during basic teaching sessions. -- Kind regards, Stefano Borini
On Thu, 4 Feb 2021 at 10:34, Stefano Borini <stefano.borini@gmail.com> wrote:
I think there's value in having a note about how to promote "best practices" in implementing the dunders under "How to teach". A relevant point about the proposal is whether it's too easy for people to fall into traps when implementing the functionality, and this is a good section to discuss that. It should be covered by noting where in the documentation it'll be discussed, so I don't think it's a big deal, but it's worth having. Paul
On 2 Feb 2021, at 12:36, Stefano Borini wrote:
It seems to me, that what complicates the specification is the need for backwards compatibility. If that wasn't an issue, we could make indexing operations behave exactly like function calls. Handling the additional argument for `__setitem__` could be done the same way that passing `self` in a method call is done: By passing an additional positional argument (in this case as the second argument after `self`), So: * `foo[1]` is `type(foo).__getitem__(foo, 1)` * `foo[1, 2]` is `type(foo).__getitem__(foo, 1, 2)` * `foo[(1, 2)]` is `type(foo).__getitem__(foo, (1, 2))` * `foo[1] = 3` is `type(foo).__setitem__(foo, 3, 1)` * `foo[1, 2] = 3` is `type(foo).__setitem__(foo, 3, 1, 2)` * `foo[(1, 2)] = 3` is `type(foo).__setitem__(foo, 3, (1, 2))` But of course this isn't backwards compatible with respect to the treatment of tuple arguments and the argument order in `__setitem__`. However it is **much** easier to remember and to teach. The PEP rejects the idea to implement this approach via a new set of dunder methods (e.g. `__getitem_ex__`, `__setitem_ex__` and `__delitem_ex__`) for performance reasons, but would it make sense, to mark existing `__getitem__`, `__setitem__` and `__delitem__` methods as supporting the new calling convention via a decorator? i.e something like: ```python class X: @newstyle_item def __getitem__(self, x, y, z=42): ... @newstyle_item def __setitem__(self, value, x, y, z=42): ... @newstyle_item def __detitem__(self, x, y, z=42): ... ``` This wouldn't require an additional dictionary lookup, but just a check of a bit in the function object.
Thank you for your help.
Servus, Walter
I missed the discussion around the PEP. I don't mean to start another one, I'd just like a clarification. The PEP describes the new functionality, and the interfaces, and that's all fine. But I didn't see where it discussed where this technology would be used. Would this mainly be used by third-party math libraries (Pandas, NumPy), like the @ operator, or is there a plan to use this in Python's own library or builtin objects? If the latter, can you go into the specifics? I'm guessing the typing module would use it, as illustrated by one of the examples, but beyond that I can't imagine how this would be used by e.g. dicts and lists. Thanks, //arry/ On 2/2/21 3:36 AM, Stefano Borini wrote:
dicts and lists will keep working as before. They will not support keyword arguments (probably ever, as there is no clear semantic for them) and the current implementation simply throws an error if the user tries to. Other classes will take advantage of the syntax for enhanced behavior. We are not specifying what that specific behavior might be (as in the case of the @ operator, which is "kind of bound" to a matmul operation, although it can be used for anything else, like / is used as a join operator in pathlib, even though its typical meaning is a division operator). Any class that is willing to accept keyword arguments for the indexing operation is responsible for documenting which arguments are accepted and what is the meaning of their use. We do not prescribe any rule, although some uses can probably be classified as bad practice. On Fri, 5 Feb 2021 at 10:30, Larry Hastings <larry@hastings.org> wrote:
-- Kind regards, Stefano Borini
Yes. It's already supported in the branch. You can do MyType[whatever=5]. One could possibly change dict to allow both the specification dict[str, int] and dict[key=str, value=int], but this specific change of the dict class is not part of the implementation nor the PEP (nor I expect it to happen, for what matters, and I would not request it). On Sat, 6 Feb 2021 at 11:10, Larry Hastings <larry@hastings.org> wrote:
-- Kind regards, Stefano Borini
Lurking around the discussions I started to wonder whether this syntax actually worths the burden it introduces. As you have commented out earlier, there are no use cases in the built-in types, perhaps a research would be great regarding possible use cases of keyword arguments in and out of the stdlib. Here is an example: https://www.python.org/dev/peps/pep-0572/#examples-from-the-python-standard-... that was done for PEP 572, which actually helped me a lot to understand/comprehend the proposal. Now looking at PEP 637, even with the theoretical examples given, it seems like we are introducing a redundant syntax that has no other value than just complicating the already complex subscript notation and becoming a bad alternative to function call syntax. On Tue, Feb 2, 2021 at 2:40 PM Stefano Borini <stefano.borini@gmail.com> wrote:
On Sat, 6 Feb 2021 at 08:56, Batuhan Taskaya <isidentical@gmail.com> wrote:
Lurking around the discussions I started to wonder whether this syntax actually worths the burden it introduces.
It has been discussed for weeks on py-ideas.
Nor there is for the @ operator, but it is still relevant for external libraries.
you can't do: f(x, y=3) = 5 but you can do a[x, y=3] = 5 pandas has relied on workarounds like iloc() to work around the lack of an indexing operator with keyword arguments, and we need them anyway for specifying types, now that the indexing operator is "abused of notation" all the time for typing. Of course the meaning of a[x, y=3] = 5 is up to the implementation of a. We don't prescribe anything on that.
-- Kind regards, Stefano Borini
participants (8)
-
Batuhan Taskaya -
Guido van Rossum -
Larry Hastings -
Paul Bryan -
Paul Moore -
Petr Viktorin -
Stefano Borini -
Walter Dörwald